networking

Cilium LoadBalancer L2 Service

The first time I created a Service of type LoadBalancer on bare metal, it sat like this:

$ kubectl get svc
NAME      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
my-app    LoadBalancer   10.96.192.212   <pending>     80:31234/TCP   5m

<pending>, forever.

In a cloud, that is someone else's problem. At home it is yours. For a long time MetalLB was the answer. These days, if Cilium is already your CNI, you do not need another component. Cilium can allocate the IPs and answer for them on the LAN.

Addresses below are examples. Swap in your own LAN ranges as you go.

Three pieces, not one feature

This setup is three separate things clicking together:

LB IPAM makes <pending> go away. It stays dormant until you create a CiliumLoadBalancerIPPool.
L2 announcements make the rest of the network able to find that IP. One node answers ARP for it.
kube-proxy replacement handles the packet once it lands. If kube-proxy is still running, the two will fight over the same Service IP.

Miss any one of them and you get the worst kind of failure: an IP that looks allocated, or looks reachable, but only sometimes.

What you need first

A Kubernetes cluster on kernel 5.10 or newer. See the system requirements if you want the full list.
Cilium as the CNI. I am on 1.19.x; nearby releases behave the same for this.
kube-proxy gone. With kubeadm that is --skip-phases=addon/kube-proxy, or delete the DaemonSet afterwards. On Talos, proxy.disabled: true.
A block of free LAN IPs that DHCP will never hand out. In the examples below the LAN is 192.168.1.0/24, DHCP stops at .199, and Kubernetes gets 192.168.1.200 to 192.168.1.240.

That last one catches people. The LoadBalancer IPs are virtual. They are never configured on an interface. They still have to sit in the same L2 broadcast domain as the nodes, because the whole trick is a node claiming the IP over ARP. An IP from another subnet will allocate happily and then go nowhere.

Install the Cilium CLI

I install the CLI before Cilium itself. It is a single binary that talks to the cluster through your existing kubeconfig.

On macOS:

brew install cilium-cli

On Linux:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

$ cilium version
cilium-cli: v0.19.7 compiled with go1.26.5 on darwin/arm64
cilium image (default): v1.19.5
cilium image (stable): v1.20.0
cilium image (running): unknown. Unable to obtain cilium version. Reason: release: not found

The first line is the CLI. running only resolves once Cilium is installed as a Helm release named cilium. If you installed another way, that line can stay unknown. cilium status still works.

Worth remembering: the CLI version and the Cilium version in the cluster are different numbers. An old CLI against a new cluster can report nonsense rather than fail loudly. And if you juggle clusters, be deliberate:

cilium status --context homelab

Install Cilium with the options that matter

I keep Cilium values in git so upgrades stay boring. Helmfile looks like this:

repositories:
  - name: cilium
    url: https://helm.cilium.io/
releases:
  - name: cilium
    chart: cilium/cilium
    version: v1.19.4
    namespace: kube-system
    values:
      - ./values.yaml

And the values:

cluster:
  name: homelab
devices: eth0
encryption:
  enabled: true
  type: wireguard
ipam:
  mode: kubernetes
  operator:
    clusterPoolIPv4PodCIDRList: 10.244.0.0/16
k8sServiceHost: 192.168.1.10
k8sServicePort: 6443
kubeProxyReplacement: true
l2announcements:
  enabled: true
operator:
  replicas: 1
tunnel: vxlan

helmfile apply

The lines I actually care about for LoadBalancer IPs:

kubeProxyReplacement: true is required. Cilium has to own Service traffic in eBPF.
k8sServiceHost / k8sServicePort are needed once kube-proxy is gone. Point them at the control plane VIP, or a control plane node.
l2announcements.enabled: true turns on the ARP responder.
devices: eth0 is the NIC Cilium binds to. Announcements only work on devices in that list. If interface names differ across nodes, use a regex instead of a literal name.

WireGuard encryption is unrelated to load balancing, but I turn it on anyway. One line of config, and pod-to-pod traffic between nodes is encrypted. Nice when that traffic crosses a switch I do not fully trust. Details are in the WireGuard docs.

Prefer the CLI installer? Same settings:

cilium install --version 1.19.4 \
  --set kubeProxyReplacement=true \
  --set l2announcements.enabled=true \
  --set k8sServiceHost=192.168.1.10 \
  --set k8sServicePort=6443 \
  --set devices=eth0

Make sure it actually took

$ cilium status --wait
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator          Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium                   Running: 3
                       cilium-operator          Running: 1
Cluster Pods:          12/12 managed by Cilium
Image versions         cilium             quay.io/cilium/cilium:v1.19.4: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.19.4: 1

Cilium and Operator should be OK. On current releases Envoy is usually disabled (using embedded mode). That is normal. Hubble and ClusterMesh staying disabled is fine for this setup.

Then check the two features this whole post depends on. They live in different places:

$ kubectl -n kube-system exec ds/cilium -- cilium-dbg config --all | grep EnableL2Announcements
EnableL2Announcements             : true

$ kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement
KubeProxyReplacement:    True   [eth0   192.168.1.21 (Direct Routing)]

If KubeProxyReplacement is False, kube-proxy is probably still around. Fix that before anything else. Every symptom later gets stranger if you do not.

Give Cilium a pool to allocate from

This is the object that wakes up LB IPAM. I keep it small and boring:

apiVersion: "cilium.io/v2"
kind: CiliumLoadBalancerIPPool
metadata:
  name: cilium-lb-ipam
spec:
  blocks:
  - start: "192.168.1.200"
    stop: "192.168.1.240"

$ kubectl apply -f CiliumLoadBalancerIPPool.yaml
ciliumloadbalancerippool.cilium.io/cilium-lb-ipam created
$ kubectl get ippools
NAME             DISABLED   CONFLICTING   IPS AVAILABLE   AGE
cilium-lb-ipam   false      False         41              4s

A few things I learned the hard way:

Current releases use cilium.io/v2 for this CRD. Older ones used v2alpha1. Check the LB IPAM docs for your version before copying a snippet.
start / stop beat cidr on a home LAN. DHCP ranges rarely line up with tidy power-of-two boundaries.
Pools are cluster-scoped. A namespace field is silently ignored.
Overlapping pools do not fail loudly. The newer one gets marked CONFLICTING and quietly stops allocating.

At this point a LoadBalancer Service will get an EXTERNAL-IP. It still will not be reachable. Nothing on the LAN knows where to send it yet.

Announce the IPs on the LAN

That is what the announcement policy is for:

apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: cilium-lb-all-services
spec:
  nodeSelector:
    matchLabels:
      cluster/role: worker
  serviceSelector:
    matchLabels:
      service/public: "true"
  loadBalancerIPs: true

In plain English: for Services labelled service/public: "true", let worker nodes answer ARP for the allocated LoadBalancer IPs.

I am deliberate about both selectors.

nodeSelector keeps control plane nodes out of the pool. Whichever node wins the lease becomes the north/south load balancer for that Service. All inbound traffic for that IP flows through it.

serviceSelector is an opt-in gate. Without it, every LoadBalancer Service in the cluster gets announced. I would rather label the few Services I actually want on the LAN. The trade-off is that an unlabelled Service gets an IP and goes nowhere, which is confusing the first time it happens.

loadBalancerIPs: true is required. Both this and externalIPs default to false, so a policy with neither set does nothing at all.

Unlike the IP pool, this CRD is still cilium.io/v2alpha1. The L2 announcements docs cover the rest of the fields.

kubectl label node worker-1 worker-2 cluster/role=worker
kubectl label svc -n my-app my-app service/public=true

Prove it works

$ kubectl create deployment whoami --image=traefik/whoami
deployment.apps/whoami created
$ kubectl expose deployment whoami --type=LoadBalancer --port=80
service/whoami exposed
$ kubectl label svc whoami service/public=true
service/whoami labeled
$ kubectl get svc whoami
NAME     TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
whoami   LoadBalancer   10.96.104.11    192.168.1.200   80:31782/TCP   8s

From a laptop on the same LAN:

$ curl http://192.168.1.200
Hostname: whoami-7d8f9c6b5d-xk2nq
IP: 127.0.0.1
IP: ::1
IP: 10.244.1.47
RemoteAddr: 192.168.1.50:52314
GET / HTTP/1.1
Host: 192.168.1.200
User-Agent: curl/8.7.1
Accept: */*

If the IP is assigned but nothing answers, ask who is claiming it. Cilium runs leader election per Service. Lease names look like cilium-l2announce-<namespace>-<service>:

$ kubectl -n kube-system get lease | grep -E 'NAME|l2announce'
NAME                                   HOLDER     AGE
cilium-l2announce-default-whoami       worker-2   30s

HOLDER is the node answering ARP. Confirm the network agrees:

$ arp -n 192.168.1.200
? (192.168.1.200) at aa:bb:cc:dd:ee:01 on en0 ifscope [ethernet]

That MAC should belong to the holder. No lease at all usually means your selectors do not match.

Failover is worth testing once. Drain the holder, or pull its power. Another node claims the lease and sends a gratuitous ARP. With defaults that is roughly a 10 to 20 second gap.

The gotcha that will bite you

Leader election generates constant API traffic. The default Cilium client rate limit is 5 QPS with bursts to 10, and it is easy to exceed.

k8sClientRateLimit:
  qps: 50
  burst: 100

Without that, things mostly work, then leases flap under load and connections drop for no obvious reason. I have been there. The sizing guidance is worth a look before you announce a lot of Services.

A few other things to know early:

L2 announcements are still marked beta. Read the current limitations before relying on them in production.
externalTrafficPolicy: Local is incompatible. Leave it on Cluster.
One node answers ARP for a given IP, so there is no real multi-node load balancing on the way in.
Keep the IP pool out of DHCP. Overlaps are miserable to debug.
If you set loadBalancerClass, it has to be io.cilium/l2-announcer for this setup. Anything else and Cilium ignores the Service completely.

What I actually use the CLI for

Once the cluster is up, the CLI earns its place because "is the network fine?" should not start with reading agent logs.

Daily health:

cilium status

--verbose on that command does almost nothing when the cluster is healthy. The deep dump is cilium-dbg inside an agent pod:

$ kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep -A1 '^IPAM:'
IPAM:                    IPv4: 7/254 allocated from 10.244.2.0/24,

$ kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose | grep -A6 '^IPAM:'
IPAM:                   IPv4: 7/254 allocated from 10.244.2.0/24,
Allocated addresses:
  10.244.2.14 (default/whoami-7d8f9c6b5d-xk2nq)
  10.244.2.21 (kube-system/coredns-5d6b9b7f8d-abcd1)
  10.244.2.33 (router)
  10.244.2.47 (health)
  ...

After an upgrade or a Helm change, I check that the features I think I enabled are actually on:

$ cilium features status | grep -E 'kube_proxy_replacement|l2_lb'
Yes      cilium_feature_adv_connect_and_lb_kube_proxy_replacement_enabled   1  1  1
Yes      cilium_feature_adv_connect_and_lb_l2_lb_enabled                     1  1  1

And when I have touched the datapath, I run the connectivity test:

$ cilium connectivity test
...
✅ 69/69 tests successful (0 warnings)

Exact counts change between releases. What matters is that the non-skipped tests pass. Clean up with kubectl delete ns cilium-test.

When a specific connection is failing and I cannot tell whether it is DNS, policy or routing, I stop guessing and use Hubble:

cilium hubble enable --ui
cilium hubble ui

Watching a DROPPED verdict show up with the policy that caused it turns hours of guesswork into a minute of reading. See the Hubble setup docs if you want the full picture.

Stuck enough to ask for help? Grab a sysdump first:

cilium sysdump

Final thoughts

L2 announcements will not cover every LoadBalancer story. If you need IPs on a different subnet from the nodes, or traffic entering through multiple nodes at once, that is BGP territory. I will cover that in the next post.

For a flat home network, though, this is one of those changes that feels bigger than the YAML suggests. A couple of manifests, kube-proxy out of the way, and <pending> turns into an IP you can curl from the laptop.

I still keep MetalLB in my head as the historical answer. I no longer keep it in the cluster.