Reinstall and upgrade

This page shows how to reinstall a package of the same version and upgrade to a new version without stopping Workspaces. For additional upgrade steps of each version, see Upgrade from v0.1.x to v0.2.0 and Upgrade from v0.2.x to v0.3.0. For uninstalling, see Uninstall. There are two kinds of upgrades. Within a release line (between the final versions and -rc.N versions of the same MAJOR.MINOR), you replace hosts one at a time, and ctrl and nodes do not need to stop together. Upgrades across a MINOR, and those involving -alpha.N or -beta.N, replace the whole deployment during a shutdown. For a first installation, see Installation overview. For details of the node and VMM process model and the maintenance check commands, see node and VMM processes. When ctrl and the gateways run in containers, the criteria and steps are the same as on this page. For how to carry them out, see Run in containers.

OverviewCopy Anchor Link

The steps that follow are easier to read once you understand the points below.

VMMs run independently of the node. The VMM of each Instance is a separate systemd transient service. Stopping the node service does not stop VMMs. After the new node starts, it adopts them again from the persisted binding records. So replacing the node binary does not require stopping Workspaces. When the target version changes the VM runtime, running VMMs keep using the runtime installation they started with, and a Workspace switches to the new runtime only on its next start.

Data-plane connections are independent of the node. Data-plane processes started by the node hold the connections for exec, terminal, file transfer, and port forwarding. Stopping the node, or replacing the node binary within the same release line, does not drop these connections. Replacing it across release lines drops them. See Data-plane processes.

Mixed versions can run within a MINOR. Starting with v0.1.0, the final versions and -rc.N versions of the same MAJOR.MINOR form a release line. On the same release line, ctrl accepts nodes of a different version, whichever side is newer. Nodes with a different MAJOR.MINOR cannot connect to ctrl. -alpha.N, -beta.N, v0.0.x, and unreleased builds require ctrl and the nodes to have exactly the same source revision.

Stopping a Workspace normally requires the guest's cooperation. The ctrl, node, and guest binaries come from the release set. If the guest version is too old to support the pre-stop preparation interface, the stop command fails explicitly instead of falling back to destroying the VM directly.

Interfaces for external callers have their own commitment. This page covers deployers replacing binaries. Interfaces that external applications call have their own compatibility window and release notes. See Compatibility commitments.

Persistent records have version ranges. Each kind of persistent record the node writes locally declares the range of versions it can be read by, at most two versions. The check before replacing binaries lists records the target version cannot read. There is no conversion back to old formats.

node can take over older guests. The guest protocol version only has to be in the accepted range. A capability the older guest lacks fails only the operations that use it.

Same-version reinstall and restartCopy Anchor Link

A plain systemctl restart kibotos-node follows the process adoption path: VMMs stay in their own services, systemd reclaims the old node's helper processes and then starts the new node, and the owner lock ensures that only one node writes the state directory at a time. For both backends, verify the actual VMMs, helper processes, and owner lock. A VMM process that is still running is not enough to conclude that adoption is complete.

To reinstall a package of the same version:

sudo systemctl stop kibotos-node.service
sudo kibotos-node check-maintenance --config /etc/kibotos/node/config.toml --operation upgrade
sudo apt-get install --reinstall ./kibotos-node_1.2.3_amd64.deb
sudo systemctl start kibotos-node.service

--operation upgrade allows detached Volumes and retained layers to stay.

Notes:

• Use --reinstall. Do not remove or purge first. Once the package is removed, no old binary is left to run the maintenance check, and reinstalling with leftover state is refused.
• During upgrade and reinstall, the package scripts only require the node service to be stopped. They do not check resources.
  • Replacing files does not stop or delete any VMM. The new node adopts them after it starts. Instances it cannot adopt are marked FAILED, and their VMMs keep running.
  • When the package scripts run, only the old binary is on disk, so they cannot answer whether the new version can adopt the Instances. The deployer answers this before replacement: for a same-version reinstall, run the check-maintenance above; across versions, run the target version's check-upgrade. See step 5 of Cross-version upgrades.
  • Uninstall still has the package scripts run check-maintenance --operation remove, and refuses to uninstall if the check fails, because after uninstall no program on the host can manage the remaining VMMs. After the check passes, the package scripts end all data-plane processes.
  • Replacing files does not stop data-plane processes either. The new node adopts or replaces them after it starts. See Data-plane processes.
• For ctrl, likewise stop the service, reinstall, and then start it manually. The package scripts do not restart the service automatically.
• If you previously ran kibo node drain, after the node recovers and syncs its state with ctrl, you also need to run kibo node cancel-drain node-a before it accepts new Workspaces again.
• Stopping the node does not drop exec, terminal, file transfer, or port forwarding connections, and new connections can be made while the node is stopped. Data-plane processes hold these connections, and ctrl keeps issuing sessions based on the data-plane certificate the node last reported, until that certificate expires. The exception is when ctrl itself restarted while the node was stopped: ctrl recognizes the node's certificate again only after the node reconnects.

Reinstalling the gatewaysCopy Anchor Link

Before you replace or reinstall a gateway package, stop the service and confirm that it is inactive with no leftover processes. The package scripts refuse to proceed when the service is running, failed, or in an unknown state. After you remove the cause of a failure, clear the failed state with systemctl reset-failed. For the E2B gateway (for the Exposure gateway, use kibotos-gw instead of kibotos-e2b):

sudo systemctl stop kibotos-e2b.service
sudo apt-get install --reinstall ./kibotos-e2b_1.2.3_amd64.deb
sudo systemctl start kibotos-e2b.service

Notes:

• It is normal for a gateway to still have proxied connections when it stops. The service gives them some time to end on their own and then closes them. The log has one warn line per server stating the number of connections. The service still stops as inactive, and you do not need to clear a failed state before installing the package.
• Perform cross-version upgrades together with ctrl and node, following the shutdown procedure in Cross-version upgrades. Upgrade ctrl before the gateways. A successful package installation does not mean the gateway can read the previous version's state, and the Exposure gateway resolves host names per zone through ctrl: if ctrl does not recognize this interface, the whole zone is unavailable.
• The gateways accept no command-line arguments. All configuration comes from /etc/kibotos/e2b/config.env or /etc/kibotos/gw/config.env. If you overrode ExecStart with systemctl edit, remove the override before upgrading and rewrite the arguments as the matching environment variables. For variable names, see e2b.example.env and gw.example.env.
• The Exposure gateway keeps no state, so existing Exposure records in ctrl are not affected while it is stopped. While it is stopped, all requests to that zone fail. Tunnel configuration and DNS do not change when it is upgraded.
• After a same-version reinstall of the E2B gateway, check the health endpoint and verify once with the SDK.

Preparing the target binariesCopy Anchor Link

The checks before an upgrade run the target version's kibotos-ctrl and kibotos-node while the current version is still installed. Extract the target packages into a new directory outside the installation directory, on each host for the package it runs. On the ctrl host:

set -eu
test ! -e /tmp/kibotos-next
dpkg-deb -x kibotos-ctrl_1.2.4_amd64.deb /tmp/kibotos-next
/tmp/kibotos-next/usr/bin/kibotos-ctrl version --json

On each node host:

set -eu
test ! -e /tmp/kibotos-next
dpkg-deb -x kibotos-node_1.2.4_amd64.deb /tmp/kibotos-next
/tmp/kibotos-next/usr/bin/kibotos-node version --json

• The node package also carries the target VM runtime in /tmp/kibotos-next/usr/lib/kibotos/node. Pass this directory to kibotos-node check-upgrade with --runtime-dir. Without it, the check uses the runtime that is installed now.
• The extracted copies are only for the checks. Delete /tmp/kibotos-next when the upgrade is done. Installation still goes through apt-get install.

Preflight checks before shutdownCopy Anchor Link

The shutdown window should be used only for replacing files. Run the two commands below with the target version's binaries while the services are still running, to find out in advance whether anything blocks the upgrade. Both are read-only and can be repeated at any time. Neither replaces the full check after shutdown.

Checking a nodeCopy Anchor Link

check-maintenance and check-upgrade both need the node's owner lock, so they can run only after the service stops. To avoid discovering during the shutdown window that some Workspace must be stopped first, run a preview on each node that is still running:

sudo /tmp/kibotos-next/usr/bin/kibotos-node check-upgrade --preview --config /etc/kibotos/node/config.toml --runtime-dir /tmp/kibotos-next/usr/lib/kibotos/node

The blocking items it lists are the ones the check after shutdown lists again. Handle them now, and the window is left with only the replacement. For what the preview evaluates and how to read its report, see Previewing with --preview before shutdown.

The file that --config points to must be readable by the target version. If the target version changes how the node configuration is written (for example, Moving pool capacity to ctrl), the running node still uses the old form, and the preview fails while loading the configuration. In that case, save a copy in the target version's form and have the preview read the copy:

sudo cp -p /etc/kibotos/node/config.toml /etc/kibotos/node/config.next.toml
sudoedit /etc/kibotos/node/config.next.toml
sudo /tmp/kibotos-next/usr/bin/kibotos-node check-upgrade --preview --config /etc/kibotos/node/config.next.toml --runtime-dir /tmp/kibotos-next/usr/lib/kibotos/node

The copy is only for the preview. For the check after shutdown (step 5 of Cross-version upgrades), first write the changes into /etc/kibotos/node/config.toml itself, then run it. The maintenance check identifies this node's service unit by the --config path. If you run it with the copy, it treats kibotos-node.service as another node's service and does not verify that all its processes have exited. Keep a copy of the original file before you change it, and switch back to it if the check fails and you need to start the original version.

Checking ctrlCopy Anchor Link

The ctrl side needs no separate switch. The upgrade check for ctrl takes no locks and writes nothing, so you can run it directly against a ctrl that is still serving. The command is exactly the same as after shutdown.

sudo -u kibotos-ctrl /tmp/kibotos-next/usr/bin/kibotos-ctrl check-upgrade --config /etc/kibotos/ctrl/config.toml

How to read the report at this point:

So the exit code of this command before shutdown is usually nonzero, with the cause only in the "Unfinished objects" section. To judge whether the upgrade is blocked, look at the other items. Run it again after the actual shutdown. Only that result is the criterion for replacement.

Rolling upgrades within a MINORCopy Anchor Link

This applies when the current version and the target version are on the same release line: the same MAJOR.MINOR, no earlier than v0.1.0, and both final versions or -rc.N versions, for example v0.1.2 to v0.1.4, or v0.2.0-rc.1 to v0.2.0. In other cases (including replacements involving -alpha.N or -beta.N), shut down and replace as described in Cross-version upgrades below.

Upgrade ctrl first, then the nodes one at a time. The node being replaced runs no new lifecycle operations, but its exec, terminal, file transfer, and port forwarding connections do not drop, and new ones can be made in the meantime (ctrl keeps issuing sessions based on the data-plane certificate the node last reported). After the new node starts, it switches to a new data-plane process, and the old process exits after its existing connections end. See Data-plane processes. Workspaces on other nodes are not affected. While ctrl is being replaced, established data-plane connections stay usable, and new Workspace operations and new data-plane sessions wait until ctrl comes back.

Rolling upgrade stepsCopy Anchor Link

The E2B gateway calls only ctrl's external interfaces. The ctrl interfaces that the Exposure gateway calls only gain additions within a release line. You can replace either at any time after step 2, following its reinstall procedure.

Rolling upgrade notesCopy Anchor Link

• The check in step 4 is the same as in a cross-version upgrade: Workspaces whose guest protocol, runtime driving range, or local record format is outside what the target node supports are listed as blocking. When upgrading to a newer version on the same release line, these ranges only grow, so this does not normally happen.
• You can usually publish the target version's Platform at any step, because old nodes can also start it.
• When the target version changes the VMM runtime, old nodes cannot start the new Platform and cannot adopt Instances started with the new runtime. In this case, publish the Platform after all nodes are replaced, and after a node is replaced it cannot roll back to a version before the runtime change. The "Upgrade notes" of that version's CHANGELOG.md say so.
• Stopping partway through an upgrade is an allowed state: ctrl and the nodes can stay on different versions of the same release line for a long time.
• You can roll back nodes one at a time to an older version on the same release line, with the same steps as the upgrade, except for versions that changed the VMM runtime. Rolling back ctrl does not require rolling back the nodes. However, after step 2 runs a database migration, the old ctrl refuses the newer schema, and the only option is to fix forward.
• If a node fails to start or adopt after replacement, keep it drained and handle that node as described in Rollback. Other nodes are not affected.

Cross-version upgradesCopy Anchor Link

This applies to upgrades between different MAJOR.MINOR versions, and to replacements involving v0.0.x, -alpha.N, -beta.N, and unreleased builds. You shut down and replace ctrl and all nodes together.

An upgrade to v0.2.0 must start from v0.1.x. Upgrade earlier versions to the latest v0.1.x first, then to v0.2.0. The check in step 5 refuses databases and node-local records from earlier versions. See Upgrade from v0.1.x to v0.2.0.

An upgrade replaces binaries, not Workspaces. VMMs and processes in the guest keep running throughout. The new node adopts the same Instance, and the Workspace's generation does not change. Connections for exec, terminal, file transfer, and port forwarding drop. The new node ends the old version's data-plane processes when it starts. When upgrading from v0.1.x, stopping the old node already drops them. They can be reestablished once the new data-plane process starts. Processes and memory in the guest are not affected.

Cross-version upgrade stepsCopy Anchor Link

Run the steps in order. If a step's completion condition is not met, stop at that step and resolve it. Do not skip ahead.

The two commands for step 5:

sudo -u kibotos-ctrl /tmp/kibotos-next/usr/bin/kibotos-ctrl check-upgrade --config /etc/kibotos/ctrl/config.toml
sudo /tmp/kibotos-next/usr/bin/kibotos-node check-upgrade --config /etc/kibotos/node/config.toml --runtime-dir /tmp/kibotos-next/usr/lib/kibotos/node

For the criteria of the node check, see node and VMM processes. For ctrl, see below.

Cross-version upgrade notesCopy Anchor Link

• When the target version brings a new guest, publish and approve the Platform with the target version first, then upgrade the nodes. The guest ships with the Platform, and the guest of an old Platform may fail during readiness on a new node. The failure reason is included in the Instance's error.
• Step 3 only requires commands to complete. It does not require stopping Workspaces. The only Workspaces you need to stop are those the report in step 5 names: Workspaces whose guest protocol version is outside the range the target node supports, and Workspaces whose runtime installation is corrupted or whose runtime version is outside the range the target node can drive. Handle them with the normal stop procedure so that data reaches disk. Do not destroy the VM.
• The runtimes a target node can drive are the version it pins itself, plus the version pinned by the previous version. Workspaces started under the current version can all be adopted. A Workspace started earlier and never stopped since is listed as blocking in step 5 if its runtime is older than the one pinned by the previous version. If you skip the check and replace directly, these Instances become FAILED. For how to handle that, see node and VMM processes.
• After step 4, VMMs are still running. This is the expected state. Do not kill them, stop units in bulk by name prefix, or delete binding files. These actions destroy the process ownership evidence that the check in step 5 relies on.
• It is fine for drain to stay at DRAINING. As long as detached Volumes remain, the state does not reach DRAINED, and the upgrade does not need DRAINED.
• A running guest that lacks capabilities added in the target version does not block the upgrade. Such an Instance is adopted as usual, but operations that use those capabilities fail until it restarts. The check-upgrade report lists the affected capabilities.

RollbackCopy Anchor Link

• If the check fails and the files are not replaced yet: start the original version of ctrl and the nodes, wait for the state to sync, and run cancel-drain. The check itself does not modify any state.
• If the target version fails to start or adopt: keep external writes paused. First stop the affected Workspaces through the target version's control plane, confirm that Volumes are detached and helper processes have exited, then run the same check on the current state with the old version. After it passes, restore the old version's installed files and start them in the same order.
• After the files are replaced in an upgrade from v0.1.x to v0.2.0, you cannot roll back. See Upgrade from v0.1.x to v0.2.0.
• In other cases, if the old version cannot read data written by the target version, preserve the current state and leave it to the deployer. Do not overwrite it directly from a backup.

Upgrade check for ctrlCopy Anchor Link

After you stop the service and before you replace the files, run a read-only check of the existing deployment with the kibotos-ctrl you are about to install. You can also run the same command before stopping ctrl; for how to read it, see Preflight checks before shutdown. The run that serves as the criterion for replacement still happens after the service stops.

Use the binary extracted in Preparing the target binaries:

sudo systemctl stop kibotos-ctrl.service
sudo -u kibotos-ctrl /tmp/kibotos-next/usr/bin/kibotos-ctrl check-upgrade --config /etc/kibotos/ctrl/config.toml

What it checks:

• The configuration parses and passes validation.
• The PKI loads and is self-consistent.
• The database migration version.
• The schema of the Template catalog.
• The Launch artifact each Workspace is pinned to can be read and verified again by the target version.
• Unfinished objects: commands in progress, Transfers not yet finished, executions being prepared or already dispatched, pending artifact finalization tasks, and allocations with unknown results.

The report lists each blocking item and its reason, followed by a summary. The exit code is 0 only when there are no blocking items.

The command is read-only. It does not migrate the schema, create directories, write the PKI or artifact store, start background tasks, or listen on ports. Repeated runs against the same state give the same result.

About the database version:

• A database version lower than the target version requires does not block. The report lists kibotos-ctrl migrate --config /etc/kibotos/ctrl/config.toml as an operation that "must run after replacing files and before starting the service", and marks check items that depend on the new schema as "applies after migration".
• The following cases block and must be handled first: the migration state is dirty, the database version is newer than the target version, or the database version is older than the earliest the target version supports (v0.2.0 requires a v0.1.x database).