> ## Content Index > Fetch the complete content index at: https://blog.godlyobeng.com/llms.txt > Use this file to discover other available public pages before exploring further. # Cilium LoadBalancer L2 Service - URL: https://blog.godlyobeng.com/cilium-loadbalancer-l2-service/ - Published: 2026-08-14T13:08:48.000Z - Updated: 2026-08-14T16:24:27.000Z - Author: Godly Obeng Karikari - Tags: networking, kubernetes The first time I created a `Service` of type `LoadBalancer` on bare metal, it sat like this: ``` $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-app LoadBalancer 10.96.192.212 80:31234/TCP 5m ``` ``, forever. In a cloud, that is someone else's problem. At home it is yours. For a long time MetalLB was the answer. These days, if Cilium is already your CNI, you do not need another component. Cilium can allocate the IPs *and* answer for them on the LAN. Addresses below are examples. Swap in your own LAN ranges as you go. ## Three pieces, not one feature This setup is three separate things clicking together: 1. [LB IPAM](https://docs.cilium.io/en/latest/network/lb-ipam/?ref=blog.godlyobeng.com) makes `` go away. It stays dormant until you create a `CiliumLoadBalancerIPPool`. 2. [L2 announcements](https://docs.cilium.io/en/latest/network/l2-announcements/?ref=blog.godlyobeng.com) make the rest of the network able to *find* that IP. One node answers ARP for it. 3. [kube-proxy replacement](https://docs.cilium.io/en/latest/network/kubernetes/kubeproxy-free/?ref=blog.godlyobeng.com) handles the packet once it lands. If kube-proxy is still running, the two will fight over the same Service IP. Miss any one of them and you get the worst kind of failure: an IP that looks allocated, or looks reachable, but only sometimes. ## What you need first - A Kubernetes cluster on kernel 5.10 or newer. See the [system requirements](https://docs.cilium.io/en/latest/operations/system%5Frequirements/?ref=blog.godlyobeng.com) if you want the full list. - Cilium as the CNI. I am on 1.19.x; nearby releases behave the same for this. - kube-proxy gone. With kubeadm that is `--skip-phases=addon/kube-proxy`, or delete the DaemonSet afterwards. On Talos, `proxy.disabled: true`. - A block of free LAN IPs that DHCP will never hand out. In the examples below the LAN is `192.168.1.0/24`, DHCP stops at `.199`, and Kubernetes gets `192.168.1.200` to `192.168.1.240`. That last one catches people. The LoadBalancer IPs are virtual. They are never configured on an interface. They still have to sit in the same L2 broadcast domain as the nodes, because the whole trick is a node claiming the IP over ARP. An IP from another subnet will allocate happily and then go nowhere. ## Install the Cilium CLI I install the CLI before Cilium itself. It is a single binary that talks to the cluster through your existing kubeconfig. On macOS: ```bash brew install cilium-cli ``` On Linux: ```bash CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) CLI_ARCH=amd64 if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all \ https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} ``` ```bash $ cilium version cilium-cli: v0.19.7 compiled with go1.26.5 on darwin/arm64 cilium image (default): v1.19.5 cilium image (stable): v1.20.0 cilium image (running): unknown. Unable to obtain cilium version. Reason: release: not found ``` The first line is the CLI. `running` only resolves once Cilium is installed as a Helm release named `cilium`. If you installed another way, that line can stay unknown. `cilium status` still works. Worth remembering: the CLI version and the Cilium version in the cluster are different numbers. An old CLI against a new cluster can report nonsense rather than fail loudly. And if you juggle clusters, be deliberate: ```bash cilium status --context homelab ``` ## Install Cilium with the options that matter I keep Cilium values in git so upgrades stay boring. Helmfile looks like this: ```yaml repositories: - name: cilium url: https://helm.cilium.io/ releases: - name: cilium chart: cilium/cilium version: v1.19.4 namespace: kube-system values: - ./values.yaml ``` And the values: ```yaml cluster: name: homelab devices: eth0 encryption: enabled: true type: wireguard ipam: mode: kubernetes operator: clusterPoolIPv4PodCIDRList: 10.244.0.0/16 k8sServiceHost: 192.168.1.10 k8sServicePort: 6443 kubeProxyReplacement: true l2announcements: enabled: true operator: replicas: 1 tunnel: vxlan ``` ```bash helmfile apply ``` The lines I actually care about for LoadBalancer IPs: - `kubeProxyReplacement: true` is required. Cilium has to own Service traffic in eBPF. - `k8sServiceHost` / `k8sServicePort` are needed once kube-proxy is gone. Point them at the control plane VIP, or a control plane node. - `l2announcements.enabled: true` turns on the ARP responder. - `devices: eth0` is the NIC Cilium binds to. Announcements only work on devices in that list. If interface names differ across nodes, use a regex instead of a literal name. WireGuard encryption is unrelated to load balancing, but I turn it on anyway. One line of config, and pod-to-pod traffic between nodes is encrypted. Nice when that traffic crosses a switch I do not fully trust. Details are in the [WireGuard docs](https://docs.cilium.io/en/latest/security/network/encryption-wireguard/?ref=blog.godlyobeng.com). Prefer the CLI installer? Same settings: ```bash cilium install --version 1.19.4 \ --set kubeProxyReplacement=true \ --set l2announcements.enabled=true \ --set k8sServiceHost=192.168.1.10 \ --set k8sServicePort=6443 \ --set devices=eth0 ``` ## Make sure it actually took ```bash $ cilium status --wait /¯¯\ /¯¯\__/¯¯\ Cilium: OK \__/¯¯\__/ Operator: OK /¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode) \__/¯¯\__/ Hubble Relay: disabled \__/ ClusterMesh: disabled DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3 Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1 Containers: cilium Running: 3 cilium-operator Running: 1 Cluster Pods: 12/12 managed by Cilium Image versions cilium quay.io/cilium/cilium:v1.19.4: 3 cilium-operator quay.io/cilium/operator-generic:v1.19.4: 1 ``` Cilium and Operator should be `OK`. On current releases Envoy is usually `disabled (using embedded mode)`. That is normal. Hubble and ClusterMesh staying disabled is fine for this setup. Then check the two features this whole post depends on. They live in different places: ```bash $ kubectl -n kube-system exec ds/cilium -- cilium-dbg config --all | grep EnableL2Announcements EnableL2Announcements : true $ kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement KubeProxyReplacement: True [eth0 192.168.1.21 (Direct Routing)] ``` If `KubeProxyReplacement` is `False`, kube-proxy is probably still around. Fix that before anything else. Every symptom later gets stranger if you do not. ## Give Cilium a pool to allocate from This is the object that wakes up LB IPAM. I keep it small and boring: ```yaml apiVersion: "cilium.io/v2" kind: CiliumLoadBalancerIPPool metadata: name: cilium-lb-ipam spec: blocks: - start: "192.168.1.200" stop: "192.168.1.240" ``` ```bash $ kubectl apply -f CiliumLoadBalancerIPPool.yaml ciliumloadbalancerippool.cilium.io/cilium-lb-ipam created $ kubectl get ippools NAME DISABLED CONFLICTING IPS AVAILABLE AGE cilium-lb-ipam false False 41 4s ``` A few things I learned the hard way: - Current releases use `cilium.io/v2` for this CRD. Older ones used `v2alpha1`. Check the [LB IPAM docs](https://docs.cilium.io/en/latest/network/lb-ipam/?ref=blog.godlyobeng.com) for your version before copying a snippet. - `start` / `stop` beat `cidr` on a home LAN. DHCP ranges rarely line up with tidy power-of-two boundaries. - Pools are cluster-scoped. A `namespace` field is silently ignored. - Overlapping pools do not fail loudly. The newer one gets marked `CONFLICTING` and quietly stops allocating. At this point a LoadBalancer Service will get an `EXTERNAL-IP`. It still will not be reachable. Nothing on the LAN knows where to send it yet. ## Announce the IPs on the LAN That is what the announcement policy is for: ```yaml apiVersion: "cilium.io/v2alpha1" kind: CiliumL2AnnouncementPolicy metadata: name: cilium-lb-all-services spec: nodeSelector: matchLabels: cluster/role: worker serviceSelector: matchLabels: service/public: "true" loadBalancerIPs: true ``` In plain English: for Services labelled `service/public: "true"`, let worker nodes answer ARP for the allocated LoadBalancer IPs. I am deliberate about both selectors. `nodeSelector` keeps control plane nodes out of the pool. Whichever node wins the lease becomes the north/south load balancer for that Service. All inbound traffic for that IP flows through it. `serviceSelector` is an opt-in gate. Without it, every LoadBalancer Service in the cluster gets announced. I would rather label the few Services I actually want on the LAN. The trade-off is that an unlabelled Service gets an IP and goes nowhere, which is confusing the first time it happens. `loadBalancerIPs: true` is required. Both this and `externalIPs` default to `false`, so a policy with neither set does nothing at all. Unlike the IP pool, this CRD is still `cilium.io/v2alpha1`. The [L2 announcements docs](https://docs.cilium.io/en/latest/network/l2-announcements/?ref=blog.godlyobeng.com) cover the rest of the fields. ```bash kubectl label node worker-1 worker-2 cluster/role=worker kubectl label svc -n my-app my-app service/public=true ``` ## Prove it works ```bash $ kubectl create deployment whoami --image=traefik/whoami deployment.apps/whoami created $ kubectl expose deployment whoami --type=LoadBalancer --port=80 service/whoami exposed $ kubectl label svc whoami service/public=true service/whoami labeled $ kubectl get svc whoami NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE whoami LoadBalancer 10.96.104.11 192.168.1.200 80:31782/TCP 8s ``` From a laptop on the same LAN: ```bash $ curl http://192.168.1.200 Hostname: whoami-7d8f9c6b5d-xk2nq IP: 127.0.0.1 IP: ::1 IP: 10.244.1.47 RemoteAddr: 192.168.1.50:52314 GET / HTTP/1.1 Host: 192.168.1.200 User-Agent: curl/8.7.1 Accept: */* ``` If the IP is assigned but nothing answers, ask who is claiming it. Cilium runs leader election per Service. Lease names look like `cilium-l2announce--`: ```bash $ kubectl -n kube-system get lease | grep -E 'NAME|l2announce' NAME HOLDER AGE cilium-l2announce-default-whoami worker-2 30s ``` `HOLDER` is the node answering ARP. Confirm the network agrees: ```bash $ arp -n 192.168.1.200 ? (192.168.1.200) at aa:bb:cc:dd:ee:01 on en0 ifscope [ethernet] ``` That MAC should belong to the holder. No lease at all usually means your selectors do not match. Failover is worth testing once. Drain the holder, or pull its power. Another node claims the lease and sends a gratuitous ARP. With defaults that is roughly a 10 to 20 second gap. ## The gotcha that will bite you Leader election generates constant API traffic. The default Cilium client rate limit is 5 QPS with bursts to 10, and it is easy to exceed. ```yaml k8sClientRateLimit: qps: 50 burst: 100 ``` Without that, things mostly work, then leases flap under load and connections drop for no obvious reason. I have been there. The [sizing guidance](https://docs.cilium.io/en/latest/network/l2-announcements/?ref=blog.godlyobeng.com#sizing-client-rate-limit) is worth a look before you announce a lot of Services. A few other things to know early: - L2 announcements are still marked beta. Read the current [limitations](https://docs.cilium.io/en/latest/network/l2-announcements/?ref=blog.godlyobeng.com#limitations) before relying on them in production. - `externalTrafficPolicy: Local` is incompatible. Leave it on `Cluster`. - One node answers ARP for a given IP, so there is no real multi-node load balancing on the way in. - Keep the IP pool out of DHCP. Overlaps are miserable to debug. - If you set `loadBalancerClass`, it has to be `io.cilium/l2-announcer` for this setup. Anything else and Cilium ignores the Service completely. ## What I actually use the CLI for Once the cluster is up, the CLI earns its place because "is the network fine?" should not start with reading agent logs. Daily health: ```bash cilium status ``` `--verbose` on that command does almost nothing when the cluster is healthy. The deep dump is `cilium-dbg` inside an agent pod: ```bash $ kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep -A1 '^IPAM:' IPAM: IPv4: 7/254 allocated from 10.244.2.0/24, $ kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose | grep -A6 '^IPAM:' IPAM: IPv4: 7/254 allocated from 10.244.2.0/24, Allocated addresses: 10.244.2.14 (default/whoami-7d8f9c6b5d-xk2nq) 10.244.2.21 (kube-system/coredns-5d6b9b7f8d-abcd1) 10.244.2.33 (router) 10.244.2.47 (health) ... ``` After an upgrade or a Helm change, I check that the features I think I enabled are actually on: ```bash $ cilium features status | grep -E 'kube_proxy_replacement|l2_lb' Yes cilium_feature_adv_connect_and_lb_kube_proxy_replacement_enabled 1 1 1 Yes cilium_feature_adv_connect_and_lb_l2_lb_enabled 1 1 1 ``` And when I have touched the datapath, I run the connectivity test: ```bash $ cilium connectivity test ... ✅ 69/69 tests successful (0 warnings) ``` Exact counts change between releases. What matters is that the non-skipped tests pass. Clean up with `kubectl delete ns cilium-test`. When a specific connection is failing and I cannot tell whether it is DNS, policy or routing, I stop guessing and use Hubble: ```bash cilium hubble enable --ui cilium hubble ui ``` Watching a `DROPPED` verdict show up with the policy that caused it turns hours of guesswork into a minute of reading. See the [Hubble setup docs](https://docs.cilium.io/en/latest/observability/hubble/setup/?ref=blog.godlyobeng.com) if you want the full picture. Stuck enough to ask for help? Grab a sysdump first: ```bash cilium sysdump ``` ## Final thoughts L2 announcements will not cover every LoadBalancer story. If you need IPs on a different subnet from the nodes, or traffic entering through multiple nodes at once, that is BGP territory. I will cover that in the next post. For a flat home network, though, this is one of those changes that feels bigger than the YAML suggests. A couple of manifests, kube-proxy out of the way, and `` turns into an IP you can curl from the laptop. I still keep MetalLB in my head as the historical answer. I no longer keep it in the cluster.