Kubernetes Services & Ingress Under the Hood: ClusterIP, Kube-Proxy IPVS & Ingress Controller Routing

1. Introduction: The Ephemeral Pod Dilemma & The Service Abstraction

🎯 Target Audience & Prerequisites Framing

Who this guide is for: CS undergraduates, cloud beginners, and software engineers who understand container fundamentals and basic Kubernetes Pod concepts, but feel puzzled by how packets actually travel from an external client's browser to dynamic, short-lived Pods inside a private cluster.

• Required Prior Knowledge: Fundamentals of TCP/IP networking (IP addresses, ports, DNS), command-line basics, and basic container concepts (e.g. Docker or Kubernetes Pods).
• Concepts Demystified Here: ClusterIP virtual IPs, Linux netfilter/iptables vs IPVS packet translation (DNAT), EndpointSlices, NodePort high-port routing, and Layer 7 Ingress reverse proxying.
• Estimated Reading & Hands-on Time: ~25–30 minutes of layered reading, code experimentation, and interactive packet tracing.

💡 Key Learning Objective: By the end of this guide, you will be able to trace a single HTTP request packet through every hop of the Linux kernel and Kubernetes network stack, write production Service and Ingress manifests, and debug real-world networking outages with surgical precision.

When students first spin up containers in Kubernetes, they discover that every single Pod receives its own private IP address (such as 10.244.1.45) allocated by the Container Network Interface (CNI) plugin. At first glance, networking seems simple: if the frontend web server needs to query the backend database, why shouldn't it just send an HTTP request directly to 10.244.1.45:5432?

In a production cloud cluster, doing this leads directly to catastrophic downtime. In Kubernetes, Pods are mortal and ephemeral. They are created, destroyed, rescheduled, scaled up, and healed dynamically. When a worker node runs out of memory, the node terminates Pods. When a software engineer deploys a new release, old Pods are deleted and brand new Pods boot up on completely different physical servers with completely new IP addresses. If your application hardcoded 10.244.1.45, every single deployment or pod crash would immediately break your entire distributed system.

The Everyday Analogy: The University Department Switchboard

To build an intuitive mental model, imagine a bustling university admissions office:

• The Ephemeral Pods (Rotating Student Advisors): Every semester, the university hires 20 student advisors to answer applicant questions. Each advisor sits at a different temporary desk with a personal extension number that changes every week as shifts rotate. If prospective students tried calling individual advisor desk phones directly, half their calls would ring dead whenever an advisor graduated, took a lunch break, or switched desks.
• The Kubernetes Service (The Main Campus Switchboard): Instead of giving out desk numbers, the university publishes a single permanent, immortal phone number: (800) 555-INFO. When an applicant dials this central number, an intelligent automated switchboard answers immediately. The switchboard maintains a real-time directory of which advisors are currently seated at their desks ready to take calls, and silently forwards the ringing call to an available advisor. The caller never knows or cares which specific desk phone answered.
• The EndpointSlice (The Active Shift Roster): Whenever an advisor clocks in for work, their desk extension is added to the switchboard's active roster. When an advisor clocks out or goes on break, the switchboard instantly removes their extension so no incoming call is ever forwarded to an empty chair.
• The Ingress Controller (The University Front Gate Receptionist): When visitors arrive in person at the campus main gate, a security receptionist checks their destination. Visitors asking for admissions.university.edu/apply are guided to the admissions switchboard, while visitors asking for athletics.university.edu/tickets are directed to the stadium office. That intelligent Layer 7 front desk is your Ingress Controller.

Minimal Working Code: Self-Contained Deployment & Service

Let us prove this abstraction with a minimal, copy-run-modify example. Below are two self-contained manifests: a 3-replica NGINX Deployment and a ClusterIP Service that fronts them.

Save the following YAML as web-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  labels:
    app: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: nginx
        image: nginx:1.25-alpine
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 2
          periodSeconds: 5

Now save the companion Service manifest as web-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  type: ClusterIP
  selector:
    app: web-app
  ports:
  - name: http
    protocol: TCP
    port: 8080        # The stable port clients connect to on the Service VIP
    targetPort: 80    # The internal container port running inside the Pod

Hands-On Verification: Inspecting the Glue

Once you apply these manifests using kubectl, observe how Kubernetes automatically pairs them together through label selectors:

# 1. Apply both resources into your local cluster (Minikube / Kind / K3s)
kubectl apply -f web-deployment.yaml
kubectl apply -f web-service.yaml

# 2. Inspect the Pods and note their ephemeral dynamic IPs
kubectl get pods -l app=web-app -o wide
# OUTPUT:
# NAME                       READY   STATUS    IP           NODE
# web-app-7d4f8b9c-2xkl1     1/1     Running   10.244.1.18  worker-1
# web-app-7d4f8b9c-8mzp4     1/1     Running   10.244.2.42  worker-2
# web-app-7d4f8b9c-q9w1v     1/1     Running   10.244.3.99  worker-3

# 3. Inspect the Service and its static ClusterIP
kubectl get svc web-service
# OUTPUT:
# NAME          TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
# web-service   ClusterIP   10.96.0.45    <none>        8080/TCP   30s

# 4. Inspect the automatically generated EndpointSlice
kubectl get endpointslices -l kubernetes.io/service-name=web-service
# OUTPUT shows: 10.244.1.18:80, 10.244.2.42:80, 10.244.3.99:80

Notice the magic: the Service received a permanent ClusterIP (10.96.0.45:8080). Any other Pod inside the cluster can now make an HTTP request to http://web-service:8080 or http://10.96.0.45:8080. Even if all three Pods crash and restart with new IP addresses 100 times, the Service IP and DNS name never change!

2. Under-the-Hood Mechanics: The Virtual IP Illusion, Netfilter DNAT & IPVS

To truly understand Kubernetes networking, computer science students must grasp a fundamental, mind-bending fact about ClusterIP addresses: a ClusterIP does not actually exist anywhere in the physical or virtual world.

If you run ifconfig, ip addr, or netstat -tlpn on any worker node in your cluster, you will not find the ClusterIP 10.96.0.45 bound to any network interface card (NIC), virtual ethernet pair (veth), bridge, or loopback device. Furthermore, there is no background daemon or server program listening on port 8080. If you try to run ping 10.96.0.45, your terminal will likely hang in complete silence. How, then, can thousands of client microservices successfully open TCP connections to an IP address that does not exist?

The Kube-Proxy Daemon: The Kernel Puppet Master

The secret lies in a system component running on every single worker node in your cluster: kube-proxy. Despite its name, modern kube-proxy is not an application proxy at all. It does not sit in user space intercepting HTTP packets or proxying TCP streams. Instead, kube-proxy is a kernel programming controller.

Kube-proxy continuously watches the Kubernetes API server for two key primitives: Services and EndpointSlices. Whenever a Service is created or Pods are scaled, kube-proxy translates those Kubernetes declarations directly into low-level Linux kernel packet-filtering rules. When a packet leaves a container destined for 10.96.0.45:8080, the packet never reaches a socket. Instead, the Linux kernel intercepts the packet in-flight and rewrites its bytes before it ever touches a network wire!

Destination Network Address Translation (DNAT) Step-by-Step

The Linux kernel subsystem responsible for this magic is Netfilter (the underlying engine of iptables and nftables). Here is the exact millisecond lifecycle of a packet traversing a Kubernetes Service:

1. Packet Injection: A client Pod sends an IP packet with SRC: 10.244.0.5:41234 and DST: 10.96.0.45:8080 (the ClusterIP).
2. The Netfilter PREROUTING Hook: As the packet enters the network stack, Netfilter triggers its PREROUTING hook in the nat table. Kube-proxy has hooked this point with a master jump rule called KUBE-SERVICES.
3. Service Chain Matching: Netfilter scans down the KUBE-SERVICES chain looking for a match on destination IP 10.96.0.45 and destination port 8080. When matched, it jumps to a service-specific chain named KUBE-SVC-<HASH>.
4. Probabilistic Load Balancing: If the Service has 3 healthy Pods, the service chain contains 3 endpoint rules (KUBE-SEP-<HASH>) chained with probabilistic matching:
   • Rule 1 matches with probability 1/3 (33.3%) → routes to Pod 1.
   • Rule 2 matches with probability 1/2 (50% of the remaining 66.7% = 33.3%) → routes to Pod 2.
   • Rule 3 matches with probability 1/1 (100% of remainder = 33.3%) → routes to Pod 3.
5. The DNAT Rewrite: Once an endpoint chain (e.g. KUBE-SEP-1) is selected, Netfilter executes the DNAT target. The kernel physically modifies the packet's IP header: the virtual IP 10.96.0.45:8080 is erased and replaced with the real Pod IP 10.244.1.18:80!
6. Conntrack Registration & The Return Trip: The Linux conntrack (Connection Tracking) module records this translation in an in-memory session table. When Pod 1 replies with [SYN,ACK], its source IP is 10.244.1.18:80. Netfilter intercepts the return packet, references conntrack, and seamlessly performs Un-DNAT, rewriting the source IP back to 10.96.0.45:8080 before handing it back to the client. The client is completely oblivious that any translation ever occurred!

iptables vs. IPVS: The $O(N)$ vs. $O(1)$ Scalability Cliff

While iptables works flawlessly for smaller clusters, it has a fatal algorithmic flaw: it is a linear array of sequential rules. Every incoming packet must evaluate iptables rules one by one until it finds a match ($O(N)$ time complexity). In a cluster with 5,000 services and 40,000 pods, iptables generates over 100,000 sequential rules! The Linux kernel spends up to 45% of its CPU time simply walking linked lists of packet rules, introducing millisecond latencies on every single hop.

To eliminate this bottleneck, modern Kubernetes clusters configure kube-proxy in IPVS (IP Virtual Server) mode. IPVS is a transport-layer load balancing engine built directly into the Linux kernel that uses ipset hash tables ($O(1)$ time complexity). Regardless of whether your cluster has 10 services or 100,000 services, IPVS resolves the target endpoint in constant time (~0.03 microseconds)!

Interactive Packet Flow & Kube-Proxy Routing Visualizer

Use the interactive simulation below to trace client packets across Layer 7 Ingress host routing, observe Linux kernel DNAT header translation, switch between iptables and IPVS modes, and simulate a failing readiness probe to watch EndpointSlices heal the cluster in real time:

EndpointSlices: Taming the etcd Scale Bottleneck

In early versions of Kubernetes, all backend Pod IPs for a Service were packed into a single monolithic API object called Endpoints. If a massive microservice scaled to 5,000 Pods, that single Endpoints resource grew to over 2 Megabytes of raw JSON. Every single time one pod crashed, restarted, or rescheduled, the Kubernetes control plane had to serialize that entire 2MB object and push it across the network to every single kube-proxy daemon on every node!

Kubernetes resolved this architectural flaw by introducing EndpointSlices. Instead of one giant object, endpoints are partitioned into bite-sized "slices" containing a maximum of 100 endpoints each. When a single pod scales or terminates, only that single slice is updated and synchronized, reducing etcd network traffic and control plane CPU utilization by over 98% in production clusters.

3. Exposing Traffic: East-West vs. North-South & Ingress Controllers

In distributed computer systems and cloud architecture, engineers divide all network traffic into two primary dimensions:

• East-West Traffic: Communication between microservices inside the cluster (e.g. your payment service querying the customer database). As we explored in Section 2, East-West traffic is handled seamlessly by internal ClusterIP Services and kernel DNAT.
• North-South Traffic: Traffic entering from or exiting to the outside world (e.g. an end-user clicking "Checkout" on their iPhone browser). Because ClusterIP addresses are non-routable private virtual IPs, external clients cannot reach them directly.

To bridge the gap between the public internet and internal ClusterIPs, Kubernetes provides three outward-facing primitives:

1. NodePort: The High-Port Brute Force Approach

When you declare a Service with type: NodePort, Kubernetes allocates a dedicated high-range port (by default from 30000 to 32767) across every single worker node in your cluster. If your cluster has 50 nodes and Kubernetes assigns port 31422, kube-proxy configures netfilter on all 50 nodes to listen for incoming traffic on NodeIP:31422 and forward it to the Service's backend Pods.

While useful for bare-metal testing, NodePort is almost never used alone in production. It forces clients to connect on ugly non-standard ports (like http://example.com:31422), creates port collision nightmares across teams, and requires clients to maintain an external list of all node IP addresses.

2. LoadBalancer: The Expensive Cloud Bridge

When you declare type: LoadBalancer on AWS (EKS), Google Cloud (GKE), or Azure (AKS), the cloud controller manager intercepts the declaration and automatically provisions a native cloud load balancer (such as an AWS Network Load Balancer). The cloud load balancer receives a public static IP, listens on standard port 80 or 443, and routes traffic directly into your nodes' NodePorts.

The problem is cost and resource bloat: cloud providers charge $18 to $25 per month for every single load balancer. If your company deploys 40 independent microservices, provisioning 40 separate cloud load balancers costs over $10,000 per year in redundant networking hardware!

3. The Modern Solution: The Layer 7 Ingress Controller

To eliminate this cost and provide sophisticated HTTP routing, Kubernetes introduced the Ingress Controller. Under this pattern, you provision only one single cloud LoadBalancer for your entire cluster. That load balancer points directly to a specialized set of Pods running an Ingress Controller (such as NGINX Ingress, Envoy Gateway, or Traefik).

Unlike kube-proxy, an Ingress Controller is not a kernel packet filter. It is an application-level reverse proxy that speaks native HTTP/1.1, HTTP/2, HTTP/3, and gRPC. It can read HTTP request headers, decrypt TLS certificates, inspect URL paths, and make routing decisions based on domain names:

• Requests to https://api.codingpancake.com/v1/* are proxied to backend-service:8080.
• Requests to https://app.codingpancake.com/* are proxied to frontend-service:80.
• Requests with cookie version=beta can be directed to a canary deployment.

Minimal Working Ingress Manifest

Here is a complete, production-ready Ingress resource routing public internet traffic to our earlier web-service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.codingpancake.com
    secretName: api-tls-cert
  rules:
  - host: api.codingpancake.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: web-service
            port:
              number: 8080

4. Production Incident Case Study, Trade-Off Rules & Anti-Patterns

Incident Timeline & Diagnostic Investigation

At 10:14 UTC on Black Friday, cluster alerts began firing across dozens of independent backend services. Frontend API gateways reported cascading connection timeouts when communicating with internal billing and catalog services. System administrators opened node monitoring dashboards and observed a bizarre diagnostic anomaly:

• Application container CPU utilization was completely normal (under 25%).
• Yet overall worker node CPU was pinned at 70%–80%, with over 45% consumed entirely by the Linux kernel (system / sys%)!
• When engineers executed perf top on the worker nodes, the two most congested kernel routines were ipt_do_table() (Netfilter sequential packet table traversal) and xtables_lock (the global kernel mutex lock for iptables rule updates).
• Whenever an autoscaler Pod scaled or crashed, kube-proxy took 11.4 seconds just to synchronize the local iptables rules! During those 11 seconds, the global kernel lock stalled packet processing across all network interfaces.

Root Cause Analysis (RCA)

The root cause was the algorithmic scalability cliff of Linux iptables:

1. Linear Array Complexity: iptables does not index rules using hash tables or search trees. It stores rules in a flat, sequential linked list. In a cluster with 2,500 services and 18,000 endpoints, kube-proxy generated more than 62,000 sequential rules in the Netfilter nat table.
2. $O(N)$ Traversal Tax: Every single incoming packet had to be compared against thousands of rules sequentially until a matching destination was found. At 150,000 packets per second per node, the kernel spent tens of millions of CPU cycles doing nothing but traversing string matches in memory.
3. Full Table Rewrites on Any Pod Event: Whenever a single pod was created or killed, iptables-restore could not update a single rule incrementally; it had to flush and rewrite the entire 62,000-rule table into kernel memory, holding xtables_lock for hundreds of milliseconds and causing massive packet jitter.

The Architectural Resolution: Migrating to IPVS Mode

The engineering team immediately initiated an emergency configuration migration to transition kube-proxy from iptables to IPVS (IP Virtual Server) mode:

# Emergency Kube-Proxy ConfigMap Patch:
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  scheduler: "rr"           # Round-robin load balancing
  syncPeriod: "30s"
  minSyncPeriod: "2s"

IPVS replaces linear rule arrays with Linux ipset kernel hash tables. Instead of traversing 62,000 rules, the kernel performs a single $O(1)$ memory hash lookup. The results were immediate and dramatic:

• Worker node kernel CPU dropped instantly from 45% to 1.2%.
• Inter-service RPC latency collapsed from 68 ms back down to 1.1 ms.
• Endpoint update latency plummeted from 11,400 ms to 9 ms.

When To Use vs. When NOT To Use Decision Matrix

To avoid similar architectural traps, use the explicit decision matrix below when designing your Kubernetes networking topology:

Top 4 Production Failure Modes & Anti-Patterns

ANTI-PATTERN 1: HARDCODING POD IPS (Brittle Topology)
Client Pod → [ Connects directly to 10.244.1.18:8080 ]
                  ↓ (Pod crashes, scales, or rolls out)
Client Pod → [ Connects to 10.244.1.18:8080 ] → CONNECTION REFUSED! (Dead IP)

PROPER PATTERN: STABLE SERVICE VIP
Client Pod → [ Connects to http://web-service:8080 ] → Kernel DNAT → Live Pod
    

• Anti-Pattern 1: Hardcoding Pod IPs: Software engineers migrating from physical servers frequently write configuration files that connect directly to Pod IPs. When Pods restart or reschedule during node maintenance, the IP is recycled and clients fail with Connection Refused. Always use the Service DNS name (e.g. http://order-service.production.svc.cluster.local:8080).
• Anti-Pattern 2: Missing Readiness Probes: When deploying rolling updates, Kubernetes assumes a Pod is ready to receive traffic the instant its container process starts, unless a readiness probe is defined. If your Spring Boot or Django app takes 30 seconds to initialize database connection pools, Kubernetes will immediately add the Pod to the EndpointSlice, blackholing user requests with HTTP 502 Bad Gateway! Always declare realistic readinessProbe configurations.
• Anti-Pattern 3: Wildcard Ingress Path Collisions: Ingress rules evaluate paths based on pathType: Prefix. If Team A declares an Ingress matching / and Team B declares an Ingress matching /api on the same host without proper ingress class isolation, traffic to /api/v1 can be unpredictably routed to Team A's frontend single-page application! Use explicit, longest-prefix matching and separate hostnames.
• Anti-Pattern 4: The externalTrafficPolicy: Cluster SNAT Blindspot: When external traffic arrives at a NodePort or LoadBalancer, by default Kubernetes applies Source NAT (SNAT) to route packets across nodes. This overwrites the client's real public IP with the node's internal IP! Security audits, rate limiters, and geolocation lookups all break. Set spec.externalTrafficPolicy: Local to preserve the client's real IP and eliminate extra inter-node hops.

5. Digestible Code Walkthroughs, Mini-Challenges & FAQ

To master Kubernetes networking, computer science students must bridge abstract theory with production code. Below is a structured, production-grade manifest orchestrating an entire microservice stack: an autoscaling Deployment, a zero-downtime ClusterIP Service, and a TLS-terminated Layer 7 Ingress resource. Rather than dumping an overwhelming 80-line YAML file at once, let us inspect it in digestible 6-to-10 line architectural chunks.

Chunk 1: Deployment Identity & Pod Replicas

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: default
  labels:
    app.kubernetes.io/name: order-service
spec:
  replicas: 3

Code Commentary: We declare a standard Kubernetes Deployment targeting the default namespace. Setting replicas: 3 instructs the ReplicaSet controller to maintain three identical worker Pods across separate cluster nodes.

Chunk 2: The Critical Selector Contract

  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service

Code Commentary: This is the most crucial contract in Kubernetes networking. The spec.selector.matchLabels defined here must exactly match the labels applied to the Pod template below it. In Chunk 4, our Service will query this exact same key-value pair to identify which Pods belong in its EndpointSlice.

Chunk 3: Container Definition & The Mandatory Readiness Probe

    spec:
      containers:
      - name: order-api
        image: ghcr.io/codingpancake/order-api:v2.1
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 3

Code Commentary: Notice the readinessProbe. When a new Pod starts during a rolling update, Kubernetes continuously queries /healthz every 3 seconds. The Pod will not be added to the Service's active routing table until this probe returns HTTP 200, guaranteeing zero dropped requests during deployments.

Chunk 4: The Internal ClusterIP Service

---
apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  type: ClusterIP
  selector:
    app: order-service
  ports:
  - name: http-traffic
    protocol: TCP
    port: 80           # Stable virtual port seen by cluster clients
    targetPort: 8080   # Real container port listening inside the Pod

Code Commentary: We declare our ClusterIP Service. Internal microservices can now communicate by sending requests to http://order-service:80. Kube-proxy captures these packets and performs kernel DNAT translation to rewrite port 80 to container port 8080.

Chunk 5: The Layer 7 TLS Ingress Resource

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: order-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - orders.codingpancake.com
    secretName: orders-tls-secret
  rules:
  - host: orders.codingpancake.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 80

Code Commentary: Our Layer 7 Ingress resource maps the public domain orders.codingpancake.com and URL prefix /v1 directly to our internal Service on port 80. The cert-manager annotation automatically provisions a free Let's Encrypt TLS certificate stored in orders-tls-secret.

# Complete Unified Manifest: Deployment + Service + Ingress
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: default
  labels:
    app.kubernetes.io/name: order-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-api
        image: ghcr.io/codingpancake/order-api:v2.1
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
  name: order-service
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: order-service
  ports:
  - name: http-traffic
    protocol: TCP
    port: 80
    targetPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: order-ingress
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - orders.codingpancake.com
    secretName: orders-tls-secret
  rules:
  - host: orders.codingpancake.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 80

Hands-On Self-Assessment Mini-Challenges

Test your systems engineering troubleshooting skills with these 3 real-world debugging scenarios:

curl: (7) Failed to connect to 10.96.10.88 port 80: Connection refused

kubectl describe svc order-service
# Look at Endpoints field: if it shows 10.244.1.33:80 instead of :3000, targetPort is wrong!

metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - http:
      paths:
      - path: /api/v1/billing(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: billing-service
            port:
              number: 8080