Nodes

This page shows how to make a node stop accepting new Workspaces, isolate a node that is down or unreachable, retire a node, set overcommit ratios, and find or choose the node a Workspace runs on.

• The examples call kibo directly. To set up the connection, see Identity and enrollment.
• For global parameters, exit codes, and JSON output, see kibo CLI reference. For command IDs and expected revisions on write commands, see Command IDs and expected revisions.
• For building and publishing Templates and Platforms, see Build your own Template and Build and publish on a build host.

kibo nodeCopy Anchor Link

Use kibo node to view a node's registration state and to make it stop accepting new Workspaces before maintenance:

node_id=node-a
kibo node list --page-size 20
kibo node get "$node_id"
registration_version=$(kibo node get "$node_id" --json | jq -er '.result.registration_version')
kibo node drain "$node_id" --command-id drain-a --expected-version "$registration_version"
# To wait until the node is empty, add --wait --timeout 10m when submitting the drain.
kibo node cancel-drain "$node_id"

list and getCopy Anchor Link

Both commands return the node registration records in ctrl as is:

The human-readable output of list is a table with one row per node. The result of the --json output uses the API field names. The server handles sorting and paging for list.

kibo node get and the JSON output give the release version, source revision, and registration record version separately. In node get, the registration record version, incarnation, session ID, and reported version are shown only with --wide. The Latest session section also gives the uncertainty of the clock offset and the synchronization status reported by the node host's kernel.

At the start of each control session and at every reconciliation, ctrl measures the clock of the node. When the offset, less its uncertainty, is more than 30 seconds, placement puts no new Workspaces on this node. Workspaces already on it keep running. The node takes new Workspaces again after a later measurement is within 30 seconds, so fix the time synchronization of the host.

drain and cancel-drainCopy Anchor Link

• Both use registration_version as the expected revision. Session changes, resource reports, and management operations all increment this revision.
• A drain does one thing: ctrl stops placing new Workspaces on this node, and the node itself rejects new create requests. Stop, delete, and export of existing Workspaces work as usual. cancel-drain restores placement.
• A drain does not migrate or destroy Workspaces. To empty a node, the Operator handles its Workspaces with kibo workspace stop/delete or kibo transfer export. kibo workspace list --node "$node_id" --all -q lists them.
• The kibo tool does not operate the host for you. The maintenance order is: drain → empty the Workspaces → see DRAINED → stop the node service.

drain --waitCopy Anchor Link

After submitting, it polls at a fixed interval until the Instance and Volume counts both reach zero:

DRAINED means this node no longer runs any Instance and no longer holds the use right of any Volume or retained layer. Objects in a shared storage pool whose use right another node holds do not count. Until both conditions are met, the status stays DRAINING.

isolateCopy Anchor Link

kibo node isolate is the operator's assertion about a node: "this host is powered off, or it can no longer write to any of its storage pools". This is not a fact that ctrl can verify by itself:

kibo node isolate "$node_id" --command-id isolate-a --expected-version "$registration_version" --confirm-host-cannot-write

• --confirm-host-cannot-write is required: the operator asserts this fact and bears the consequences if the assertion is wrong. Declaring isolate on a node that is in fact still writing to storage corrupts the block files it is using.
• As long as the node still has a live control session, isolate is rejected (exit code 13): only an operator outside ctrl who is watching the host can make this assertion responsibly.

On success, ctrl does the following at once:

• Releases the use right of every Volume and retained layer this node holds, in both shared pools and pools only it is attached to.
• Releases all allocations under this node.
• Moves every Workspace running on this node to FAILED with the reason UNAVAILABLE. You can retry starting them.
• Fails the commands sent to this node that have not completed.

The result gives the number of released Volumes, the number of retained layers, and the number of Workspaces that failed as a result.

When these Workspaces can be started again or deleted depends on which pool their objects are in:

• After a restart, the data is the same as after a host power loss: everything already written to the pool is there, and writes still in the old host's page cache may be lost.
• When a shared pool is on NFS, another node can take over correctly only if the mount does not use nocto. See Prerequisites for storage pools on NFS.
• Isolate invalidates the node's current incarnation: reconnecting with this or an earlier incarnation is rejected, and ctrl tells the node at the same time that it has been isolated.
• When it receives this, the node destroys all of its Instances and exits, and systemd restarts it. After the restart it reconnects with a new incarnation, and the isolate clears automatically with no extra command.
• When objects on a shared pool have already been handed to another node, the original node detects this from the holder records in the pool within a few seconds, even without a connection to ctrl, and stops the Instances that are still writing to those objects.
• Isolate deletes no objects, and it does not confirm for you that the host is actually powered off or actually cut off from storage. A wrong declaration has no after-the-fact check, and the consequence is data corruption.

retireCopy Anchor Link

When a host will not come back, retire its node with kibo node retire. Retirement is a final state:

kibo node retire "$node_id"

• It accepts only a node whose status is DRAINED or whose current incarnation has been isolated. Others return failed_precondition with exit code 13.
• You confirm in the terminal before it runs. Without a terminal, add --yes. With an explicit --command-id, you must also give --expected-version. Replaying the same pair returns the same result.

After retirement:

• The node is no longer a member of any storage pool, its pending enrollment grants are revoked, and its control session is closed.
• Its registration is kept, with status RETIRED. Allocations and the previous holders of objects still reference this record.
• It cannot register again. Issuing a grant, draining, or isolating the same node_id returns failed_precondition. A new host uses a new node_id.
• If a node retired after being isolated comes back and connects with the isolated incarnation, it still receives the isolation rejection and destroys all its Instances. Only with a new incarnation is it rejected for not being admitted.
• For the Workspace that last ran on it, kibo workspace cleanup-status shows RUNTIME, VOLUME_ATTACHMENT, and RESIDUAL as RELEASED, with the control-plane state as the source.

Like a retired node, a node revoked with kibo enrollment revoke is not a pool member: it is not listed in the pool's Nodes, and it does not count as "another node attached to the pool" when another node enters DRAINED.

A pool that only this node was attached to has no members after retirement. There are two cases:

• The disk is intact: attach the disk to another host and declare it with the same pool ID. That host's node becomes a member. See Moving a pool to another host.
• The disk is gone too: use kibo storage-pool declare-lost.

overcommitCopy Anchor Link

The node's [resources] states only the CPU and memory the host can give to VMs. Overcommit is expressed by ratios in ctrl: the schedulable amount used for placement equals the amount the node reports times the ratio. Changing a ratio does not require restarting the node:

kibo node overcommit get
kibo node overcommit set-default --cpu 4
kibo node overcommit set "$node_id" --cpu 2.5 --memory 0.9
kibo node overcommit set "$node_id" --cpu default

• Ratios are decimals with at most three decimal places. Both CPU and memory ratios are between 0 and 16.
• With a memory ratio above 1, the memory guests actually use can exceed host memory. When host memory runs out, the kernel kills one of the VMs, and ctrl then stops its Workspace on its own; the caller can start it directly. Writes not yet on disk may be lost. KiboVM guest memory is released only after the process has fully exited, and the kernel may kill another process during that time. Set the memory ratio from actual usage after long runs; see Host usage below.
• When available host memory falls below 10% of total memory, the node considers memory tight: it rejects new Workspaces, and ctrl immediately places them on another node instead. If no node can take them, the request returns resource_exhausted. The node accepts new Workspaces again once available memory is back above 15%. Existing Workspaces are not affected.
• set-default changes the cluster-wide default, and its expected revision is the version shown by get. set changes one node's override, and its expected revision is that node's registration_version. default clears the override for that item and returns it to the cluster-wide default.
• Both commands change only the item you give and keep the other at its current value. When you pass --command-id explicitly, you must also pass --expected-version and both ratios, so the replay input does not depend on values read at run time.
• When a ratio is lowered below the allocated amount, this node stops accepting new Workspaces. Existing ones keep running and are not evicted.
• Until the new value reaches the node and the node confirms it, ctrl does not place Workspaces on this node, usually for less than a second. If the node does not confirm, ctrl drops its control connection and sends the value again after it reconnects.

The Host usage section of kibo node get shows actual usage. It is for viewing only and does not affect placement:

• The node samples every 10 seconds, and ctrl fetches the latest sample at each reconciliation (every 60 seconds), so readings lag by up to about a minute.
• Items that cannot be read show -, not 0. A node that has never started a VM has no VM items.
• The two VM items count the VMs on the whole host. When a host runs only one node, these are that node's VMs.
• Memory a guest has used is not returned to the host, so the longer a VM runs, the closer VM memory gets to the memory it requested. When adjusting the memory ratio from actual usage, look at readings after long runs.

Which node runs a WorkspaceCopy Anchor Link

kibo workspace list lists executing Workspaces by node or by Instance, and kibo workspace get shows the node a Workspace runs on:

kibo workspace list --node node-a
kibo workspace list --instance inst-01k6…
kibo workspace get ws-01k6…

• The table is the same as a plain workspace list, and --wide adds the INSTANCE and NODE columns. Each start gets a new Instance.
• Only Workspaces that are executing now are listed. Stopped, deleted, and not-yet-placed Workspaces have no node, and workspace get shows -.
• An ID that is no longer the current Instance finds no rows, the same as an ID that never existed.
• --node and --instance are mutually exclusive and cannot be combined with filters such as --state and --label. If the node does not exist, the exit code is 12.
• The kibo tool first reads one page of locations, then reads each Workspace. A Workspace deleted between the two reads does not appear.
• -q --all prints only Workspace IDs, one per line, for a script to process one by one.
• When connected to an older ctrl without location queries, the Node of workspace get shows unknown, and the command itself still succeeds.

Placing a Workspace on a specific nodeCopy Anchor Link

--node pins one create or start to one node. kibo workspace move stops a Workspace and starts it on another node:

kibo workspace create --template kibotos-coding --node node-c
kibo workspace start ws-01k6… --node node-b
kibo workspace move ws-01k6… --node node-b

• This one node is the only candidate. As with any placement, it must accept the Workspace: not drained, ready, with enough remaining capacity, and attached to the pools that hold the Workspace's Volumes and writable layer. If it does not qualify, the create or start fails and is not placed on another node instead.
• It applies only to this one placement. The Workspace does not remember this node, and later starts without --node are placed as usual.
• A drain still blocks a pinned placement. To smoke-test a new node, cancel its drain first; in the meantime, placement may also put other Workspaces on it.
• At creation, the Volumes and writable layer created with the Workspace go to this node's default pool. When the objects are in the node's private pool, the Workspace can later start only on this node. To be able to change nodes, use a pool shared by several nodes. See storage-pool.

workspace move does three things in order:

• There is no live migration. Stop and start work as usual, and memory and processes are not kept.
• If the Workspace already runs on the target node, the command succeeds directly without stopping it.
• When --command-id is given, the command IDs of the stop and start are that value plus /stop and /start. Running the same command again continues with the step that did not complete.