Lab lifecycle
A lab is a deployment of a topology between a start and an end time. While it’s deployed, Antimony attaches an instance to it, which holds the live state: whether it’s running, which nodes it has, their addresses and their state.
States
Section titled “States”| State | Value | Meaning |
|---|---|---|
| Scheduled | -2 |
The lab has no instance and its start time is in the future. The scheduler deploys it when the start time arrives. |
| Deploying | 0 |
The provider is deploying the lab. The deployment log is available. |
| Running | 1 |
The deployment succeeded and Antimony knows all nodes. Node logs and statistics are available. |
| Stopping | 2 |
The provider is destroying the lab. Usually short. |
| Failed | 3 |
The deployment failed. You can redeploy or destroy the lab. |
| Inactive | -1 |
The lab has no instance and its start time has passed: it has ended or was destroyed. Deploying it again starts a new instance. |
The values are what the API returns in instance.state. Scheduled and Inactive are pseudo-states: an instance never has them. A lab without an instance has instance set to null, and the interface shows it as Scheduled or Inactive depending on its start time.
From creation to the end
Section titled “From creation to the end”- Create. When you create a lab, Antimony checks that you may deploy from the topology’s collection, that the name is valid and unique in the collection, that the end time, if there is one, is after the start time and, for users who aren’t administrators, that the topology passes the topology policy. It then gives the lab a unique instance name, copies the topology and its bind files into the lab’s run directory and stores the lab.
- Schedule. The scheduler queues the lab twice: for deployment at the start time and for destruction at the end time. A lab without an end time runs until someone destroys it. Moving a lab to another time updates both queues. A lab can only be edited or moved while it isn’t deployed.
- Deploy. At the start time, or when someone clicks Deploy, the lab enters Deploying and Antimony calls the provider. Afterwards it inspects the result to learn the nodes and their addresses, and the lab becomes Running. If something fails, the lab becomes Failed and the topology is marked with
lastDeployFailed. - Wait for nodes. A container that is up isn’t necessarily ready to use. Antimony checks every node every 2 seconds, for up to 10 minutes, by running
ssh admin@localhost trueinside it. The node counts as ready as soon as its SSH daemon answers, even with a login error, or immediately if the node has no SSH client. Once a node is ready, Antimony reads its interfaces and starts collecting statistics, and the interface enables its terminal. - Destroy. At the end time, or when someone clicks Destroy, the lab enters Stopping. The provider removes the nodes, the instance is removed, and the lab becomes Inactive. If the provider fails to remove the lab, it becomes Failed. Destroying a lab that isn’t deployed fails.
A lab can only be deleted when it isn’t deployed, or when its deployment failed. Deleting it removes its run directory.
Redeploying
Section titled “Redeploying”Redeploy on a running or failed lab deploys it again with the same instance name. containerlab uses containerlab redeploy; clabernetes destroys and deploys again.
The lab’s run directory still holds the topology from when the lab was created. To pick up changes to the topology, create a new lab.
Node states
Section titled “Node states”| State | Value | Meaning |
|---|---|---|
| Stopped | 0 |
The node isn’t running. |
| Starting | 1 |
The node is starting. |
| Running | 2 |
The node is running. |
| Stopping | 3 |
The node is stopping. |
Node lifecycle
Section titled “Node lifecycle”Nodes of a running lab can be started, stopped and restarted individually, and keep their links while they do. containerlab uses containerlab start, stop and restart, which move the node’s link interfaces out of the container before it stops, move them back once it started, and run the node’s exec commands again. clabernetes stops a node by scaling it to zero and restarts it by recreating its pod, which gets its links wired up again.
This doesn’t work for some nodes. Commands for them fail with 5023:
- nodes without a container of their own: the kinds
bridge,ovs-bridge,host,ext-containerandk8s-kind - nodes with
network-mode: hostornetwork-mode: container:<node> - nodes with
auto-remove: true
These settings count wherever the node gets them from, its group, its kind or the defaults.
Restarting the server
Section titled “Restarting the server”Instances live only in memory, but the labs keep running in Docker or Kubernetes while the server is down. When the server starts again, it rebuilds its state:
- It asks the provider for all nodes it deployed (
containerlab inspect --all, or the labelled pods in the cluster). - For every lab in the database:
- If the provider still has nodes for the lab’s instance name, Antimony creates a new instance for it and marks it as recovered. The lab is Running again.
- If not, and the start time is in the future, the lab is scheduled again.
- Otherwise the lab is Inactive.
Recovered labs show “(Recovered)” next to their deployment time. Their deployment log starts empty, because logs from before the restart aren’t kept.