node and VMM processes

This page explains how node, VMM, data-plane, and egress processes run under systemd, and the two check commands to run before you replace the node binary. For complete upgrade and uninstall steps, see Reinstall and upgrade.

OverviewCopy Anchor Link

• The host needs systemd, cgroup v2, and KVM. The node runs as root.
• The node starts the VMM of each Instance as a separate systemd transient service. Both backends, Firecracker and KiboVM, run this way.
• A data-plane process holds the connections for exec, terminal, file transfer, and port forwarding. It is also a separate service that the node starts. When you stop, restart, or upgrade the node, the VMMs and data-plane processes keep running.

Process modelCopy Anchor Link

The node serviceCopy Anchor Link

• The node service unit uses Restart=on-failure, RestartSec=5s, KillMode=control-group, and SendSIGKILL=yes. A new node starts only after systemd stops the old node's cgroup and cleans up its helper processes.
• The node service unit sets OOMScoreAdjust=-900. When host memory runs out, the kernel kills VMMs first, not the node. Short-lived tools that the node starts (mkfs, guestfish, and others) inherit this value. If you write your own unit file instead of using the packaged one, add this setting too.
• At startup, the node first checks its own MainPID, its cgroup, and whether leftover processes exist. It then takes the owner lock on the state directory.
• The file system tools guestfish and virt-ls always use LIBGUESTFS_BACKEND=direct. Their helper VMs stay inside the node service's cgroup and are not started through an external libvirt.

VMM serviceCopy Anchor Link

• The VMM service unit uses Type=exec, Restart=no, KillMode=control-group, and OOMPolicy=kill, and sits in kibotos-vmm.slice (parent slice kibotos.slice). With OOMPolicy=kill, when the kernel kills any process in the unit, the whole unit is killed, so helper processes that share guest memory with the VMM do not hold on to that memory. VMMs that were already running before the upgrade keep their old settings. The new settings take effect on their next start.
• The VMM service has no stop dependency on the node service. Stopping or restarting the node does not affect VMMs. When the host shuts down, systemd terminates VMMs in its default shutdown order.
• The Firecracker service starts jailer as root, and jailer then switches to the VMM account. The KiboVM service runs directly as the non-root VMM account, with the supplementary group kvm.
• The KiboVM service sets LimitMEMLOCK=infinity. Newer kernels (such as 7.0) count io_uring ring memory against the owning user's locked memory, and all VMMs run as the same VMM account. With the systemd default of 8 MiB, the whole host can start only twenty-odd VMs. Later VMs fail to start, and kibovm.log reports Out of memory (os error 12). The node sets this limit on the service unit, so you do not need to change the host's limits.conf or the systemd DefaultLimitMEMLOCK. Firecracker does not use io_uring and keeps the default.
• The service uses a network namespace that the node creates in advance, and sets PrivateMounts=no. Each backend's own sandbox isolates files and devices. The jailer no longer creates other cgroups or moves processes into them.

Data-plane processesCopy Anchor Link

A data-plane process listens on the data-plane address in the node configuration and holds all data-plane connections. It runs the same kibotos-node executable. The node starts it, and each start gets an increasing generation number.

• The service unit name is kibotos-dataplane-<scope>-<generation>.service. It sits in kibotos-dataplane.slice and uses Restart=no, KillMode=control-group, and OOMScoreAdjust=-900. It has no stop dependency on the node service. The node's runtime directory determines the scope, so several nodes on the same host do not affect each other.

• List all data-plane processes:

  Copy Text

  systemctl list-units 'kibotos-dataplane-*'

• The data-plane process runs as root, because the guest sockets of both backends are open only to root and the VMM account.

• Its local sockets are under dataplane/ in the socket directory (default /run/kibotos/node). It does not read or write the node's state directory. The node supplies the target, the verification public key, and the TLS certificate.

• The node hands the new target and verification public key to the data-plane process before it replies to ctrl, so a session credential that ctrl issues works as soon as the caller receives it. If the data-plane process does not confirm within 5 seconds, the node replies anyway and logs the warning data-plane generation did not take the node's state before the reply. Until the node resends (at most 15 seconds), requests that use the new target or new public key are rejected.

• The node's /metrics sums the metrics of all data-plane processes. After a data-plane process exits, its counts stay in the counters, so counters never decrease.

At startup, the node acts according to the data-plane process that is running:

Notes:

• At most one old process waits for its existing connections to end. On the next replacement, the node ends the older one, and its remaining connections drop.
• While the node is stopped, the data-plane process keeps serving with the last target the node supplied and still accepts new connections. During this time, stops, deletions, and revocations from ctrl do not reach the node, and issued session credentials stay usable until they expire. These operations take effect after the node comes back. To cut off access immediately, stop all data-plane processes.
• The node renews the data-plane TLS certificate. If the certificate expires while the node is stopped, new connections fail the handshake. Established connections are not affected.
• When a data-plane process exits unexpectedly, the connections it holds drop, and the node starts a new data-plane process within a few seconds. While the node is stopped, nothing starts a new one until the node comes back.
• When a data-plane process stops, it gives existing connections 10 seconds to end on their own, then closes the rest and logs how many it closed.

Egress processesCopy Anchor Link

An egress process forwards TCP connections that a guest sends through a caller's SOCKS5 egress. For Workspaces that reference a caller Resolver, it also forwards DNS queries to that upstream. Like the data-plane process, it runs the same kibotos-node executable. The node starts it and replaces it by generation.

• The service unit name is kibotos-egress-<scope>-<generation>.service. It sits in kibotos-egress.slice, uses Restart=no and KillMode=control-group, and has no stop dependency on the node service.

• It runs only when the network policy of some Workspace on this node references an Egress or a Resolver. For Workspaces that do not reference a Resolver, the kernel forwards DNS queries to the deployment resolver without passing through it. After policies stop referencing an Egress, it keeps holding the connections that are still draining, and the node ends it when the last one ends or expires.

• It runs as the system user kibotos-egress, with no capabilities and no way to gain privileges. Its file system is read-only, it can write only its own socket directory, and it cannot see the node's state directory. The node creates the sockets that need privileges and hands them to it. The node also supplies egress credentials over a local socket, and they stay only in memory.

• Its socket directory sits next to the node's socket directory, with the suffix -egress added (default /run/kibotos/node-egress, owned by kibotos-egress with mode 0700). The generation counter is in <storage.root>/state/egress-generation.

• The node uses its own identity to fetch egress credentials from ctrl by Egress revision. Each revision is one file in <storage.root>/state/egress-credentials/, with directory mode 0700 and file mode 0600, readable only by root. When a Workspace switches to a new revision or leaves this node, files that are no longer referenced are deleted right away. When the node restarts, it restores egress from these files only and does not contact ctrl.

• List all egress processes:

  Copy Text

  systemctl list-units 'kibotos-egress-*'

• For a Resolver upstream that goes direct, the egress process connects from the host network namespace as the kibotos-egress user, and the traffic leaves through the host's routes. If the deployer restricts the node's own outbound traffic on the host, these connections are restricted too. When the upstream is written as a host name, the egress process resolves it with the node's configured resolver list in order, not with the host's /etc/resolv.conf.

• When the node restarts, it adopts the running egress process, and existing connections are not affected. When the node changes to another build on the same release line, it starts a new egress process for new connections and new queries. The old process stops accepting new connections, stops reading new queries, discards its credentials, answers queries it already received within 4 seconds, continues or closes existing connections by the deadline the node supplies, and exits when all of them end. Across release lines, the old process is ended and its connections drop.

• At most two egress generations run at the same time. If you change builds again while the old process from the previous change is still draining, the oldest generation is ended and its connections drop at once. The log egress generation ended has reason set to a newer generation is about to drain. To change builds several times in a row without dropping connections, first use the command above to confirm that only one egress process remains.

• A transparent listener accepts guest connections. The node creates the listener, so it belongs to the cgroup of the node service unit, and this does not change while the listener exists. As a result, systemd IP access restrictions on kibotos-node.service (IPAddressDeny=, IPAddressAllow=) also apply to connections forwarded through egress and block them too. If you need such restrictions, allow ipv4_pool and ipv6_pool. Do not use them to cut the connection between the node and ctrl.

• When an egress process exits unexpectedly, connections through egress drop, and the node starts a new egress process within a few seconds. In the meantime, new connections routed to egress fail and do not fall back to going direct. Queries from Workspaces that reference a Resolver get no answer. Name resolution for other Workspaces is not affected.

Launch records and adoptionCopy Anchor Link

• Before starting a VMM, the node writes systemd.json in the Instance directory. It records the unique ID of this launch, the service unit name, and the expected command. After the launch, the node adds the InvocationID, MainPID, process start time, executable identity, cgroup, and network namespace.
• After the service starts, the Instance is usable only when it also passes the backend and guest readiness checks. The node records the guest it accepted at that point: its boot ID and declared capabilities.
• The launch record also pins the identity of the runtime installation the VMM uses. A runtime installation lives in <storage.root>/installations/sha256-<identity>/, and the installation.json in that directory describes it.
• When the node restarts, it compares these records one by one with the services that are actually running and adopts the matching Instances. If a launch response was lost, it looks up the original launch record. If a record does not match reality, or a service exists with no record, the node aborts its startup.
• Adopting a running Instance does not wait for the guest to respond. Once storage, network, and the VMM are reattached, the Instance runs as usual, and the data plane keeps serving based on the recorded guest:
  • In the background, the node hands the guest its launch configuration and the current DNS again. If the guest does not respond for now, the node retries with backoff indefinitely. Meanwhile the Workspace stays AVAILABLE, and the guest is reported as no longer responding.
  • A guest that restarted while the node was offline is reconfigured with its launch configuration.
  • If the guest responds but refuses, for example because its protocol version is outside the range this version supports, the Instance is marked FAILED and the VMM keeps running.
• When the network is reattached, the node regenerates the Instance's firewall rules from the recorded network policy with the current version and replaces them as a whole in one nftables transaction. Established connections do not drop. Rules of an Instance that the previous node version started may differ from the current version, and adoption rewrites them too. The log slot table rewritten from the recorded network policy records the Instance, the namespace, and the differences.
• In the following cases, the node does not rewrite the rules. It isolates the Instance's network and marks it FAILED, and the VMM keeps running:
  • The Instance's network namespace has no node rule table, or has nftables tables that do not belong to the node.
  • The address sets in the rule table do not match the record or the current configuration, except the resolver addresses. For example, management_cidrs changed, or the uplink address changed.
  • Rules or chains were added by hand.
  • The current version cannot enforce the recorded policy, for example because the node no longer provides public egress.
• An Instance that was still being created at restart is an exception. Its guest never passed the readiness check, so adoption still waits for it to become ready, and creation fails on timeout.
• A newly started Instance always uses the runtime installation bundled with the current version. An Instance that the previous node version started may record an earlier installation. The node adopts it as usual and uses the recorded installation, provided that:
  • The installation directory is complete, and the file digests match installation.json.
  • Its runtime version is within the range the current node can still drive. The range is the runtime pinned by the current version, plus the runtimes pinned by earlier versions on the same release line and by the previous MINOR.
• If these conditions are not met, the node cannot adopt the Instance. The Instance is marked FAILED, and its VMM keeps running. For how to handle this, see Troubleshooting.
• When a stopped Workspace starts again, it switches to the current version's runtime. Each time the node starts, it deletes old installation directories that no Instance uses. Installation directories still recorded by Instances that failed adoption are kept.
• When an Instance is deleted, the node first stops the matching service, confirms that the whole cgroup is empty, and then reclaims mounts, network, and files.
• When you delete an Instance that was not adopted but whose service is still running, the node first verifies ownership: the launch record, the service binding, the node's own Instance record, and the actual service must all point to the same process. If they all agree, the node stops the service and then reclaims. If any one disagrees, the deletion fails and the service keeps running.
• Stopping the node is not the same as deleting Instances, and it does not set or clear drain.

To troubleshoot a launch failure, keep the existing records and look at systemctl show <unit>, journalctl -u <unit>, and the retained console and VMM logs. Do not delete binding records, change PIDs, or end processes in bulk by name. These actions remove the basis the node uses to decide which process belongs to what.

Records of ended InstancesCopy Anchor Link

After an Instance ends, the node keeps its record in <storage.root>/runtime/registry/ until ctrl confirms it is no longer needed.

• Once a minute, ctrl checks the records on each node. When the Instance's Allocation is released, its Workspace has no command in progress, and the Workspace no longer uses it as the current Instance, ctrl first stores the cleanup result the node observed and then has the node delete the record, at most 100 records per node each time.
• After the record is deleted, kibo workspace cleanup-status reads the result stored by ctrl. The status of the runtime items stays the same, and the observation time is the moment the node observed that cleanup finished.
• Volume, writable layer, and Template preparation commands are recorded in volume-commands.json in the same directory. Each entry is kept for 24 hours.

Do not delete records by hand. If you delete one before the cleanup result is stored, the runtime items in that Workspace's cleanup status show as unknown.

Temporary files from interrupted writesCopy Anchor Link

The node writes records, block files, and Transfer archives to temporary files first and renames them when the write completes. If the node exits midway, the temporary files stay in place, and no version reads them. Temporary file names in a pool start with the ID of the node that wrote them. Files written by v0.1.x do not:

The node deletes these files at startup:

Two other kinds in a shared pool need manual deletion:

• Files without a node ID: first stop all nodes that declare this pool.
• Files that start with the ID of a node that was permanently removed: after that node no longer runs.

Console and VMM logsCopy Anchor Link

When an Instance ends, before deleting its runtime directory, the node copies the tail of the guest console and VMM logs to <storage.root>/machine-logs/<instance_id>/ and writes only one line, instance machine exited, to the node log. For what is retained, quotas, and how to read the logs, see Logs.

Pre-replacement check: check-maintenanceCopy Anchor Link

You do not need to stop Workspaces to replace the node binary. VMMs run independently, and the new node adopts them from the records after it starts. This also holds when the target version changes the VM runtime: Instances started with the old runtime keep using the old installation. Only Instances that the target version cannot drive must be stopped first, and the target version's check-upgrade lists them one by one.

After you stop the node service, run the maintenance check with the currently installed node:

sudo kibotos-node check-maintenance --config /etc/kibotos/node/config.toml --operation upgrade
sudo kibotos-node check-maintenance --config /etc/kibotos/node/config.toml --operation remove

Neither operation checks or ends data-plane processes. When you uninstall the node package, the package script ends all data-plane processes after the --operation remove check passes.

Both operations have the same requirements for leftovers on the host: every Instance directory, backend state directory, sandbox directory, network record, and mount must belong to an adopted Instance. Leftovers with no owner make the check fail, because the new binary cannot understand them and nothing is responsible for cleaning them up.

Notes:

• Configuration, PKI, installed runtimes, and rebuildable caches may stay. The shared Firecracker sandbox root directory counts as a cache that may stay only if it is a tmpfs owned by root with mode 0700 and contains only directories of adopted Instances.
• Temporary files from interrupted writes do not count as leftovers, and the check does not fail because of them. See Temporary files from interrupted writes.
• The check reads only the configuration, the owner lock, persistent records, and process information. It does not repair state, increment the launch sequence number, or create directories.
• A missing configuration, a corrupted record, a helper process that is still running, or unclear ownership all return a nonzero exit code.
• Objects in a storage pool whose holder record names another node do not make the check fail, whether or not the pool is shared. A pool that moved to another host can therefore be upgraded normally. See Moving a pool to another host.
• This check answers only "are the current resources quiescent and owned". Whether the new version can read these records is answered by the new version's own check-upgrade.

Checking with the target binary before replacementCopy Anchor Link

check-maintenance runs with the current version, which cannot know whether the new version can read the records on disk. Extract the new version outside the installation directory as described in Preparing the target binaries, and after all services stop, run it with the new version's binary:

sudo /tmp/kibotos-next/usr/bin/kibotos-node check-upgrade --config /etc/kibotos/node/config.toml --runtime-dir /tmp/kibotos-next/usr/lib/kibotos/node

It uses the same quiescence criteria as upgrade, and checks four more things:

• The format versions of the node identity, enrollment record, and launch records are all within the range the new version can read.
• The target runtime directory contains the VM runtime executables the new version requires.
• The new version can keep and drive the runtime installation of every Instance that is still running: either it is the installation the new version will use, or its directory is complete and its runtime version is within the range the new version can drive.
• Every guest that is still running responds to GuestInfo through its own VMM, with a protocol version within the range the new version accepts.

The report lists each blocking object and the reason. If anything blocks, the command returns a nonzero exit code.

Notes:

• A running guest that lacks capabilities added in the new version does not block. Such an Instance is adopted as usual, but operations that use those capabilities fail until it restarts. The report lists these capabilities.
• A guest protocol version out of range blocks. The matching Workspace must be stopped normally first.
• A running Instance that uses an old runtime installation does not block by itself. The report states the Instance, the installation and runtime version it uses, and the installation it switches to on its next start.
• It blocks only when the old installation directory is corrupted or missing, or when its runtime version is outside the range the new version can drive. The matching Workspace must be stopped normally first. When it starts again, it uses the new runtime.
• The check is read-only. It does not create directories, increment the launch sequence number, migrate data, or modify guests.
• The result describes the state only at the moment of that shutdown. If any process that writes state has restarted, you need to shut down and check again.

Previewing with --preview before shutdownCopy Anchor Link

The command above requires the node to be stopped. Like check-maintenance, it needs the node's owner lock, and while the service is still running it only reports that the runtime is already in use. With --preview, it runs on a node that is still running and evaluates the checks that do not depend on whether the host is quiescent: the format versions of local records, the target runtime directory, the runtime installations of running Instances, and the protocol versions and capabilities of running guests. The criteria are exactly the same as after shutdown, so the preview already tells you which Workspaces must be stopped normally first. For when and how to run it, see Checking a node.

The report has three kinds of lines:

The preview does not take the owner lock and writes no files, so you can repeat it at any time. It does not evaluate quiescence or ownership: helper processes, leftovers with no owner, host mounts, staging directories, and the ownership of Volumes and writable layers are not part of its result, and the end of the report repeats this. It does not replace the full check after shutdown. Before replacement, you still need to stop the service and run the same command without --preview.

TroubleshootingCopy Anchor Link