# Kubernetes networking

Every Kubernetes cluster runs one pod network plugin, the CNI, installed
by the platform when the cluster is created:
[Canal](#the-default-cni) or [Cilium on request](#cilium-on-request).
The choice is made at creation and holds for the life of the cluster.
There is no in place switch. A cluster that needs the other CNI is a new
cluster.

```text
                      Canal                          Cilium
pod to pod traffic    VXLAN overlay between nodes    VXLAN overlay between nodes
network policy        Kubernetes NetworkPolicy       Kubernetes NetworkPolicy and CiliumNetworkPolicy
service routing       kube-proxy                     eBPF; no kube-proxy pod
flow telemetry        none                           Hubble metrics and the hubble observe CLI
encryption            none                           none
RDMA                  outside the CNI                outside the CNI
```

## The default CNI

Canal is the default on every new cluster: a VXLAN overlay carries pod
traffic between nodes and Calico enforces Kubernetes `NetworkPolicy`.
Service traffic goes through `kube-proxy`. Canal exposes no flow
telemetry: there is no per flow record of what a policy allowed or
dropped.

## Cilium on request

Ask support for Cilium before the cluster is created. The request names
the cluster. A Cilium cluster differs from a Canal cluster in the three
ways below.

### Network policy

Kubernetes `NetworkPolicy`, `CiliumNetworkPolicy` and
`CiliumClusterwideNetworkPolicy` are enforced. A policy selects pods by
labels and namespaces. Its rules match peers by labels, namespaces or
CIDRs. Its rules match ports and protocols. The policies are yours to
write ([below](#what-you-own)).

### Service routing without kube-proxy

No `kube-proxy` pod runs. ClusterIP and NodePort services are answered by
Cilium's eBPF service map on every node.

### Hubble flow telemetry

Hubble runs inside the Cilium agent on every node. Two reads exist.

- **Metrics.** Every node's agent serves Hubble's flow, drop, TCP, DNS,
  ICMP and port distribution counters on its metrics endpoint, port
  9965 (`hubble_flows_processed_total`, `hubble_drop_total` and the
  other `hubble_*` series). The flow counter carries the verdict of each
  flow. The drop counter carries the reason. The monitoring agent the
  platform runs in your cluster scrapes the endpoint. Read it with
  `kubectl`:

  ```sh
  kubectl --context <cluster> -n kube-system port-forward <cilium agent pod> 9965:9965
  curl -s localhost:9965/metrics | grep '^hubble_'
  ```

  Queries of the `hubble_*` series through
  [`metrics_query`](https://docs.nationalcompute.com/api/metrics.md#free-form-queries-metrics_query) are
  not offered.
- **The flow CLI.** `hubble observe` runs inside each node's agent pod and
  prints the flows that node saw, live or from its ring buffer:

  ```sh
  kubectl --context <cluster> -n kube-system exec <cilium agent pod> -- hubble observe --follow
  ```

  `kubectl --context <cluster> -n kube-system get pods -l k8s-app=cilium -o wide`
  lists the agent pod of each node. `--verdict DROPPED` shows what a
  policy refused, with the identities on the flow.

The Hubble relay (one cluster wide stream) and the Hubble UI are not
installed. Both are yours to install. The console shows no flows. Hubble
is read with `kubectl` and the metrics reads above.

## What you own

Your policies are yours: their design, their tests and their effect on
your own workloads. The platform supports the CNI itself: its
installation, the health of the agent on every node and the upgrade that
comes with each Kubernetes release
([Kubernetes versions and maintenance](https://docs.nationalcompute.com/kubernetes-versions-and-maintenance.md)).
The platform does not write, review or debug a policy.

A policy that blocks the platform's own pods breaks what the platform can
see and run in your cluster. The platform's namespaces are `kube-system`
(DNS and the CNI), `kube-public`, `kube-node-lease`, `cert-manager`,
`local-path-storage`, `mpi-operator`, `jobset-system`, `kueue-system`
(the queue), `prometheus` (the monitoring agent), the GPU and network
operator namespaces (`kubectl --context <cluster> get ns` shows the ones
your cluster runs) and every `nc-*` namespace (the platform's diagnostic
pods). Leave those namespaces reachable. The drop counters name the
policy that blocked a flow. The fix is yours.

## What does not change

- GPU jobs reach the RDMA rails through the device plugin on every node.
  RDMA traffic never passes through the CNI, with either CNI. NCCL is
  unchanged.
- Pod traffic between nodes is not encrypted, with either CNI.
- The `type: LoadBalancer` path is the same on both: the public IP, its
  release and the client source IP rule
  ([Exposing services](https://docs.nationalcompute.com/kubernetes.md#exposing-services-type-loadbalancer)).
- The arrival taint, the registry mirror, the cluster CPU worker and the
  shared volume are the same on both.
- The CNI's version follows the cluster's Kubernetes release. An upgrade
  moves it with the release
  ([Kubernetes versions and maintenance](https://docs.nationalcompute.com/kubernetes-versions-and-maintenance.md)).
- Every port the CNI needs between your nodes is open; nothing is yours
  to open. One limit exists. On a cluster whose control plane the
  workers cannot reach on Cilium's own health and Hubble ports,
  `cilium status` on a worker lists the control plane nodes as
  unreachable. A relay you install lists them as unavailable peers.
  Their agent metrics are absent. Pod traffic, DNS and every worker to
  worker flow are unaffected.