Skip to content

Kubernetes networking

Every Kubernetes cluster runs one pod network plugin, the CNI, installed by the platform when the cluster is created: Canal or Cilium on request. The choice is made at creation and holds for the life of the cluster. There is no in place switch. A cluster that needs the other CNI is a new cluster.

                      Canal                          Cilium
pod to pod traffic    VXLAN overlay between nodes    VXLAN overlay between nodes
network policy        Kubernetes NetworkPolicy       Kubernetes NetworkPolicy and CiliumNetworkPolicy
service routing       kube-proxy                     eBPF; no kube-proxy pod
flow telemetry        none                           Hubble metrics and the hubble observe CLI
encryption            none                           none
RDMA                  outside the CNI                outside the CNI

The default CNI

Canal is the default on every new cluster: a VXLAN overlay carries pod traffic between nodes and Calico enforces Kubernetes NetworkPolicy. Service traffic goes through kube-proxy. Canal exposes no flow telemetry: there is no per flow record of what a policy allowed or dropped.

Cilium on request

Ask support for Cilium before the cluster is created. The request names the cluster. A Cilium cluster differs from a Canal cluster in the three ways below.

Network policy

Kubernetes NetworkPolicy, CiliumNetworkPolicy and CiliumClusterwideNetworkPolicy are enforced. A policy selects pods by labels and namespaces. Its rules match peers by labels, namespaces or CIDRs. Its rules match ports and protocols. The policies are yours to write (below).

Service routing without kube-proxy

No kube-proxy pod runs. ClusterIP and NodePort services are answered by Cilium's eBPF service map on every node.

Hubble flow telemetry

Hubble runs inside the Cilium agent on every node. Two reads exist.

  • Metrics. Every node's agent serves Hubble's flow, drop, TCP, DNS, ICMP and port distribution counters on its metrics endpoint, port 9965 (hubble_flows_processed_total, hubble_drop_total and the other hubble_* series). The flow counter carries the verdict of each flow. The drop counter carries the reason. The monitoring agent the platform runs in your cluster scrapes the endpoint. Read it with kubectl:
kubectl --context <cluster> -n kube-system port-forward <cilium agent pod> 9965:9965
curl -s localhost:9965/metrics | grep '^hubble_'

Queries of the hubble_* series through metrics_query are not offered. - The flow CLI. hubble observe runs inside each node's agent pod and prints the flows that node saw, live or from its ring buffer:

kubectl --context <cluster> -n kube-system exec <cilium agent pod> -- hubble observe --follow

kubectl --context <cluster> -n kube-system get pods -l k8s-app=cilium -o wide lists the agent pod of each node. --verdict DROPPED shows what a policy refused, with the identities on the flow.

The Hubble relay (one cluster wide stream) and the Hubble UI are not installed. Both are yours to install. The console shows no flows. Hubble is read with kubectl and the metrics reads above.

What you own

Your policies are yours: their design, their tests and their effect on your own workloads. The platform supports the CNI itself: its installation, the health of the agent on every node and the upgrade that comes with each Kubernetes release (Kubernetes versions and maintenance). The platform does not write, review or debug a policy.

A policy that blocks the platform's own pods breaks what the platform can see and run in your cluster. The platform's namespaces are kube-system (DNS and the CNI), kube-public, kube-node-lease, cert-manager, local-path-storage, mpi-operator, jobset-system, kueue-system (the queue), prometheus (the monitoring agent), the GPU and network operator namespaces (kubectl --context <cluster> get ns shows the ones your cluster runs) and every nc-* namespace (the platform's diagnostic pods). Leave those namespaces reachable. The drop counters name the policy that blocked a flow. The fix is yours.

What does not change

  • GPU jobs reach the RDMA rails through the device plugin on every node. RDMA traffic never passes through the CNI, with either CNI. NCCL is unchanged.
  • Pod traffic between nodes is not encrypted, with either CNI.
  • The type: LoadBalancer path is the same on both: the public IP, its release and the client source IP rule (Exposing services).
  • The arrival taint, the registry mirror, the cluster CPU worker and the shared volume are the same on both.
  • The CNI's version follows the cluster's Kubernetes release. An upgrade moves it with the release (Kubernetes versions and maintenance).
  • Every port the CNI needs between your nodes is open; nothing is yours to open. One limit exists. On a cluster whose control plane the workers cannot reach on Cilium's own health and Hubble ports, cilium status on a worker lists the control plane nodes as unreachable. A relay you install lists them as unavailable peers. Their agent metrics are absent. Pod traffic, DNS and every worker to worker flow are unaffected.