Deploy with clabernetes
Clabernetes (c9s) runs containerlab topologies on Kubernetes. With the clabernetes provider, Antimony doesn’t start containers itself. It hands each lab to clabernetes, which spreads the nodes across your cluster’s workers.
Use it when a single Docker host isn’t enough: a whole class deploying labs at the same time, or topologies with many heavy network OS images.
The Antimony server itself still runs outside the cluster, for example with the same Docker Compose setup as the Docker deployment. It talks to the cluster through the Kubernetes API using a kubeconfig.
Requirements
Section titled “Requirements”Antimony is built against clabernetes 0.10.1, which uses the native “direct” runtime. Your cluster must meet the clabernetes 0.10 requirements:
- Kubernetes 1.31 or newer. The direct runtime uses native sidecar containers.
- Privileged pods allowed in lab namespaces. Antimony labels every lab namespace with
pod-security.kubernetes.io/enforce: privileged. If you run an admission controller such as Kyverno or Gatekeeper, allow privileged pods inc9s-*namespaces. - VXLAN and nftables support in the worker kernels.
/dev/kvmon the workers if you use VM-based kinds (vrnetlab).- NetworkPolicies that allow UDP 14789 and 14790 between device pods, TCP 14791 from the kubelet, and egress to the API server.
- cgroup v2 on the workers, and device images that contain
sh,catandsleep. Antimony reads node statistics and interfaces by running small shell commands inside the device container.
Set up
Section titled “Set up”Start from the Docker deployment. Then:
-
Install clabernetes 0.10.1 in your cluster.
Terminal window helm upgrade --install --create-namespace --namespace c9s --version 0.10.1 \clabernetes oci://ghcr.io/clabernetes/clabernetes/clabernetes -
Give the server your kubeconfig. In
docker-compose.yml, uncomment this line of theserverservice:docker-compose.yml - $HOME/.kube/config:/root/.kube/config:roAntimony uses the kubeconfig’s current context. Its user needs the permissions listed under RBAC.
-
Switch the provider in
config.yml:config.yml deployment:provider: clabernetes -
Restart Antimony.
Terminal window docker compose up -dThe server log shows
Using the Clabernetes deployment provider.
Private images, terminals and other details
- Private registries. The cluster pulls node images itself. Add your pull secrets to the clabernetes configuration with
kubectl -n c9s edit configs.c9s.run clabernetes(imagePull.pullSecrets). - Terminals log in to nodes over SSH through a Kubernetes port-forward, with the credentials from
kinds.conf.yml. If that fails, they fall back to a shell inside the container. - Credentials lookup. Antimony uses the in-cluster ServiceAccount if it runs in a pod, otherwise the
KUBECONFIGenvironment variable, otherwise~/.kube/config. If no client can be created, the server stops withFailed to create clabernetes client. - Exec plugins. If your kubeconfig uses a credential plugin, the plugin must also exist inside the server container.
- Unused settings. The Docker socket and the privileged settings of the server container are only needed for containerlab. You can remove them when you only use clabernetes.
What happens when you deploy a lab
Section titled “What happens when you deploy a lab”- Antimony compiles the topology locally with the clabernetes compiler. Topologies that use features clabernetes doesn’t support, such as
macvlanendpoints or custom management networks, fail right away withtopology is not supported by clabernetes: …. See the containerlab differences in the clabernetes docs. - It removes all containerlab
labelsfrom the definition. Older topologies store their graph layout in labels, and clabernetes rejects some of those values. - It creates a namespace named
c9s-<instance name>and aTopologyresource (topologies.c9s.run) named after the instance. The instance name is derived from the topology name and kept short enough for a valid namespace name. Nodes are not exposed outside the cluster (exposeType: None); you reach them through Antimony. - It waits up to 10 minutes for the topology to become ready. Progress (
Deployment status: 3/5 nodes ready), pod events and image pulls appear in the lab’s deployment log.
You can follow along with kubectl:
kubectl get topologies.c9s.run -Akubectl -n c9s-<instance name> get podsWhen you stop the lab, Antimony force-deletes the node pods and the namespace, and waits up to two minutes for it to disappear.
Node actions
Section titled “Node actions”| Action | What Antimony does in the cluster |
|---|---|
| Stop node | Labels the clabernetes Node resource with c9s.run/ignoreReconcile: "true" and scales the node’s Deployment to 0. |
| Start node | Removes the label and scales the Deployment back to 1. |
| Restart node | Deletes the node’s pod; clabernetes recreates it. A stopped node is started instead. |
How nodes are reached
Section titled “How nodes are reached”Everything goes through the Kubernetes API server. Antimony never connects to pod IPs directly, so it doesn’t need access to the cluster network.
| Feature | Mechanism |
|---|---|
| Terminals | SSH to port 22 over pods/portforward, falling back to pods/exec with bash or sh |
| Node logs | pods/log of the device container |
| CPU, memory and interface stats | A long-running pods/exec that reads cgroup v2 and /proc |
| Traffic capture | packet-capture in the pod’s clabwire sidecar, over pods/exec |
All of these streams pass through the API server, so its load and latency directly affect how responsive terminals and live statistics feel.
Traffic capture on clabernetes
Section titled “Traffic capture on clabernetes”Packet capture through Antimony’s SSH server works on clabernetes too, with a few differences:
- Captures run inside the node pod’s
clabwiresidecar. Opening a capture takes a moment longer than on Docker. - Only the node’s lab link interfaces can be captured, not the management or
c9s0interfaces. - The sidecar limits a single capture to one hour. Antimony chains captures, so your Wireshark session keeps running without interruption.
- Capture only works for pods created by clabernetes 0.9 or newer. Older pods return a “not supported” error.
There is no ready-made manifest yet. The kubeconfig user (or ServiceAccount) needs a ClusterRole with these permissions. It has to be cluster-wide, because Antimony creates namespaces and watches pods across all namespaces.
apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: antimonyrules: - apiGroups: [""] resources: [namespaces] verbs: [create, get, delete] - apiGroups: [""] resources: [pods] verbs: [get, list, watch, deletecollection] - apiGroups: [""] resources: [pods/exec, pods/portforward] verbs: [create, get] - apiGroups: [""] resources: [pods/log] verbs: [get] - apiGroups: [""] resources: [events] verbs: [list] - apiGroups: [apps] resources: [deployments] verbs: [get, list, watch, patch] - apiGroups: [c9s.run] resources: [topologies] verbs: [create, get, update] - apiGroups: [c9s.run] resources: [nodes] verbs: [patch]Limitations
Section titled “Limitations”- Fixed timeouts. Deploying times out after 10 minutes, which large images on a cold cluster can hit. Pre-pull big network OS images on the workers.
- No external node access. Nodes are only reachable through Antimony’s terminals and capture.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause and fix |
|---|---|
Server exits with Failed to create clabernetes client |
No usable kubeconfig in the container. Check the mount and that the API server address is reachable from the container. |
topology is not supported by clabernetes: … |
The topology uses a containerlab feature clabernetes doesn’t support. Remove or replace it. |
| Deployment fails after 10 minutes | Images take too long to pull. Pre-pull them or check the pull secrets. |
| Capture says it’s not supported | The node pod wasn’t created by clabernetes 0.9. Redeploy the lab after upgrading. |
| Empty statistics or interface list | The device image lacks sh, cat or sleep, or the worker doesn’t use cgroup v2. |