Skip to content

Deploy with clabernetes

Zoey hugging a bottle labelled C9S full of network nodes

Clabernetes (c9s) runs containerlab topologies on Kubernetes. With the clabernetes provider, Antimony doesn’t start containers itself. It hands each lab to clabernetes, which spreads the nodes across your cluster’s workers.

Use it when a single Docker host isn’t enough: a whole class deploying labs at the same time, or topologies with many heavy network OS images.

The Antimony server itself still runs outside the cluster, for example with the same Docker Compose setup as the Docker deployment. It talks to the cluster through the Kubernetes API using a kubeconfig.

Antimony with clabernetesThe Antimony server runs outside the cluster and uses a kubeconfig to talk to the Kubernetes API server. It creates a namespace and a Topology resource per lab. The clabernetes manager turns the Topology into pods. Shells, logs, stats and captures reach the pods through the API server.Antimony hostAntimony serverprovider: clabernetes:3000kubeconfig~/.kube/configPostgreSQLlabs, topologies, usersstorage/ run/Kubernetes clusterAPI serverexec · port-forwardlogsHTTPSnamespace c9sclabernetes-managerplanner-poolnamespace c9s-<instance>Topology CRpod: leaf1device containerclabwire sidecarpod: leaf2device containerclabwire sidecarVXLANcreatesshells, logs,stats, capturereconciles
Antimony creates a namespace and a Topology resource per lab. clabernetes turns it into pods. Terminals, logs, statistics and packet captures all travel through the API server.

Antimony is built against clabernetes 0.10.1, which uses the native “direct” runtime. Your cluster must meet the clabernetes 0.10 requirements:

  • Kubernetes 1.31 or newer. The direct runtime uses native sidecar containers.
  • Privileged pods allowed in lab namespaces. Antimony labels every lab namespace with pod-security.kubernetes.io/enforce: privileged. If you run an admission controller such as Kyverno or Gatekeeper, allow privileged pods in c9s-* namespaces.
  • VXLAN and nftables support in the worker kernels.
  • /dev/kvm on the workers if you use VM-based kinds (vrnetlab).
  • NetworkPolicies that allow UDP 14789 and 14790 between device pods, TCP 14791 from the kubelet, and egress to the API server.
  • cgroup v2 on the workers, and device images that contain sh, cat and sleep. Antimony reads node statistics and interfaces by running small shell commands inside the device container.

Start from the Docker deployment. Then:

  1. Install clabernetes 0.10.1 in your cluster.

    Terminal window
    helm upgrade --install --create-namespace --namespace c9s --version 0.10.1 \
    clabernetes oci://ghcr.io/clabernetes/clabernetes/clabernetes
  2. Give the server your kubeconfig. In docker-compose.yml, uncomment this line of the server service:

    docker-compose.yml
    - $HOME/.kube/config:/root/.kube/config:ro

    Antimony uses the kubeconfig’s current context. Its user needs the permissions listed under RBAC.

  3. Switch the provider in config.yml:

    config.yml
    deployment:
    provider: clabernetes
  4. Restart Antimony.

    Terminal window
    docker compose up -d

    The server log shows Using the Clabernetes deployment provider.

Private images, terminals and other details
  • Private registries. The cluster pulls node images itself. Add your pull secrets to the clabernetes configuration with kubectl -n c9s edit configs.c9s.run clabernetes (imagePull.pullSecrets).
  • Terminals log in to nodes over SSH through a Kubernetes port-forward, with the credentials from kinds.conf.yml. If that fails, they fall back to a shell inside the container.
  • Credentials lookup. Antimony uses the in-cluster ServiceAccount if it runs in a pod, otherwise the KUBECONFIG environment variable, otherwise ~/.kube/config. If no client can be created, the server stops with Failed to create clabernetes client.
  • Exec plugins. If your kubeconfig uses a credential plugin, the plugin must also exist inside the server container.
  • Unused settings. The Docker socket and the privileged settings of the server container are only needed for containerlab. You can remove them when you only use clabernetes.
  1. Antimony compiles the topology locally with the clabernetes compiler. Topologies that use features clabernetes doesn’t support, such as macvlan endpoints or custom management networks, fail right away with topology is not supported by clabernetes: …. See the containerlab differences in the clabernetes docs.
  2. It removes all containerlab labels from the definition. Older topologies store their graph layout in labels, and clabernetes rejects some of those values.
  3. It creates a namespace named c9s-<instance name> and a Topology resource (topologies.c9s.run) named after the instance. The instance name is derived from the topology name and kept short enough for a valid namespace name. Nodes are not exposed outside the cluster (exposeType: None); you reach them through Antimony.
  4. It waits up to 10 minutes for the topology to become ready. Progress (Deployment status: 3/5 nodes ready), pod events and image pulls appear in the lab’s deployment log.

You can follow along with kubectl:

Terminal window
kubectl get topologies.c9s.run -A
kubectl -n c9s-<instance name> get pods

When you stop the lab, Antimony force-deletes the node pods and the namespace, and waits up to two minutes for it to disappear.

Action What Antimony does in the cluster
Stop node Labels the clabernetes Node resource with c9s.run/ignoreReconcile: "true" and scales the node’s Deployment to 0.
Start node Removes the label and scales the Deployment back to 1.
Restart node Deletes the node’s pod; clabernetes recreates it. A stopped node is started instead.

Everything goes through the Kubernetes API server. Antimony never connects to pod IPs directly, so it doesn’t need access to the cluster network.

Feature Mechanism
Terminals SSH to port 22 over pods/portforward, falling back to pods/exec with bash or sh
Node logs pods/log of the device container
CPU, memory and interface stats A long-running pods/exec that reads cgroup v2 and /proc
Traffic capture packet-capture in the pod’s clabwire sidecar, over pods/exec

All of these streams pass through the API server, so its load and latency directly affect how responsive terminals and live statistics feel.

Packet capture through Antimony’s SSH server works on clabernetes too, with a few differences:

  • Captures run inside the node pod’s clabwire sidecar. Opening a capture takes a moment longer than on Docker.
  • Only the node’s lab link interfaces can be captured, not the management or c9s0 interfaces.
  • The sidecar limits a single capture to one hour. Antimony chains captures, so your Wireshark session keeps running without interruption.
  • Capture only works for pods created by clabernetes 0.9 or newer. Older pods return a “not supported” error.

There is no ready-made manifest yet. The kubeconfig user (or ServiceAccount) needs a ClusterRole with these permissions. It has to be cluster-wide, because Antimony creates namespaces and watches pods across all namespaces.

antimony-clusterrole.yml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: antimony
rules:
- apiGroups: [""]
resources: [namespaces]
verbs: [create, get, delete]
- apiGroups: [""]
resources: [pods]
verbs: [get, list, watch, deletecollection]
- apiGroups: [""]
resources: [pods/exec, pods/portforward]
verbs: [create, get]
- apiGroups: [""]
resources: [pods/log]
verbs: [get]
- apiGroups: [""]
resources: [events]
verbs: [list]
- apiGroups: [apps]
resources: [deployments]
verbs: [get, list, watch, patch]
- apiGroups: [c9s.run]
resources: [topologies]
verbs: [create, get, update]
- apiGroups: [c9s.run]
resources: [nodes]
verbs: [patch]
  • Fixed timeouts. Deploying times out after 10 minutes, which large images on a cold cluster can hit. Pre-pull big network OS images on the workers.
  • No external node access. Nodes are only reachable through Antimony’s terminals and capture.
Symptom Cause and fix
Server exits with Failed to create clabernetes client No usable kubeconfig in the container. Check the mount and that the API server address is reachable from the container.
topology is not supported by clabernetes: … The topology uses a containerlab feature clabernetes doesn’t support. Remove or replace it.
Deployment fails after 10 minutes Images take too long to pull. Pre-pull them or check the pull secrets.
Capture says it’s not supported The node pod wasn’t created by clabernetes 0.9. Redeploy the lab after upgrading.
Empty statistics or interface list The device image lacks sh, cat or sleep, or the worker doesn’t use cgroup v2.