Kibotos

Upgrade from v0.1.x to v0.2.0

This page describes what you must do, beyond the general cross-version upgrade steps, when you upgrade from v0.1.x to v0.2.0. "Step N" on this page refers to the step table in Cross-version upgrades.

v0.2.0 reads only state left by v0.1.x. The check in step 5 lists the following objects as blocking. Handle them as the table describes before you replace the files:

ObjectHandling
A ctrl database from a version earlier than v0.1.xFirst upgrade fully to the latest v0.1.x, then upgrade to v0.2.0
A Workspace started under v0.0.3, v0.1.0-rc.1, or v0.1.0-rc.2 and never stopped sinceStop it normally once with the current version. Its runtime is outside the range the target version can drive, and its local records may be in a format the target version cannot read. On its next start, it switches to the target version's runtime
A record of an ended Instance reported as has version 7, this build reads [8 9]After you stop the node, move this file outside storage.root and keep it for reference
volumes/metadata-v1/ and retained-layers/metadata-v2/ in a poolDelete these two directories. They are metadata backups left by the upgrade from v0.0.3, and the target version no longer reads them
An Instance reported as node Instance <id> is active or requires cleanup whose Workspace is FAILEDRun kibo workspace stop --force on this Workspace with the current version, so that the node cleans up its Instance. Handle all FAILED Workspaces before the window starts, so that you do not have to bring the original version back up during the shutdown
A command reported as Workspace command <id>: the command is still processing whose Volume has already been deletedWhile ctrl is stopped, end it as aborted as described below

After you move away the record of an ended Instance, starting, stopping, and deleting its Workspace are not affected. The items related to this Instance in the cleanup status show as unknown.

After the upgrade, ctrl has the existing records of ended Instances deleted batch by batch, and kibotos_node_instances{state="stopped"} and {state="destroyed"} go down accordingly. Records with no matching Allocation stay where they are, because ctrl does not recognize them.

When v0.1.x deletes a Volume, it does not end resize, create, or delete commands on it whose result is unknown. Such a command stays processing indefinitely, resending it does not end it, and the target version's check lists it as an unfinished lifecycle operation. After you confirm that the command in the report belongs to a deleted Volume, run the following as ctrl's database account:

UPDATE kibotos_ctrl_commands
SET status = 'completed', error_code = 10, updated_at = now()
WHERE command_id = '<command ID>'
  AND status = 'processing'
  AND volume_id IN (SELECT volume_id FROM kibotos_ctrl_volumes WHERE volume_state = 5);

The command is handled only when the result is UPDATE 1. UPDATE 0 means it is not this kind of command; handle it according to Upgrade check for ctrl. error_code = 10 is aborted, and volume_state = 5 is DELETED, which is the same result the target version produces when it ends such commands while deleting a Volume.

Before you replace the files, also check the configuration and permissions in the following table:

ItemWhat to doConsequence if not handled
[runtime].socket_root on KiboVM nodesKeep it at 39 bytes or fewer. If it is longer, switch to a shorter directory. Sockets of the new runtime are one level deeper than beforeThe node reports an error at startup
[[storage.pool]] in the node configurationRemove capacity. See Moving pool capacity to ctrlcheck-upgrade and node startup fail
ctrl's database.max_connectionsThe default changes from 2.5 × execution_concurrency + 32 to 3 × execution_concurrency + 32, which is 80 instead of 72 at a concurrency of 16. If you did not set it, confirm that PostgreSQL can accept the extra connections. If you did, it must be at least 3 × execution_concurrency + 16. See Number of database connectionsThe ctrl service refuses to start and states the minimum value it needs
The address that ctrl's database.dsn points toIt must be the PostgreSQL primary, not a cloud database proxy address, a read/write splitting address, or a connection pool. See Connecting to the databaseWhen the proxy replaces the database session, two commands for the same Workspace may run at the same time
ctrl's database.dsn and dsn_fileThe connection string is parsed by libpq rules: the user name and password in a URI are percent-encoded, so @ is written as %40, and IPv6 addresses go in square brackets. Run psql "$DSN" -c 'select 1' with the same connection string. If it connects, no change is neededThe ctrl service cannot connect to the database
ctrl's permissions on the artifact storeIt must be able to list objects under the prefix kibotos-transfer/v1/<namespace>/. See Prepare the bucket and permissionsDeleting a Transfer returns unavailable, failed and expired archives stay in the store, and the ctrl log has transfer archive cleanup did not finish
Log collectionIf you select Kibotos logs by unit name, add the units in kibotos-dataplane.slice. The reference Vector configuration already includes them. See Data-plane processesLogs from data-plane processes are not collected

v0.2.0 changes the VMM runtime of the KiboVM backend, and v0.1.x nodes cannot start this version's KiboVM Platform. This is the opposite of "publish the Platform before upgrading the node" in the notes of Cross-version upgrades: after step 7 and before step 8 resumes writes, publish and approve this version's KiboVM Platform. New nodes start VMs only with the new runtime and do not run the old Platform. Running Workspaces are adopted as usual and are not affected.

Workspaces started by v0.1.x keep running their original runtime and Platform until their next start:

  • A normal stop proceeds as usual, but the disk is written only to the host page cache, as in v0.1.x. For each such stop, the node logs stop did not confirm the host flush with the reason the VMM's runtime release does not flush on stop.
  • On the next start, the Workspace switches to the default of this version's KiboVM Platform. The guest protocol and rootfs contract must be the same. The kernel series may differ.

KiboVM Platforms published by v0.1.x stay in the catalog. If this version does not publish a Platform of the same kernel series, the old default of that series still shows as the default and cannot be withdrawn. Nodes no longer provide the runtime it requires, so neither new Workspaces nor restarted Workspaces select it. No action is needed.

After you upgrade to v0.2.0, you cannot roll back to v0.1.x. A v0.1.x ctrl refuses the migrated database, and a v0.1.x node cannot read the local records and pool metadata written by v0.2.0. Before you replace the files, resolve all blocking items with the check in step 5. If the check fails and the files are not replaced yet, start the original version as usual. See Rollback. Problems that appear after the replacement are fixed on v0.2.x.

Moving pool capacity to ctrl

In the target version, pool capacity is an attribute of the pool on ctrl, and the node configuration no longer declares it. Before you replace the files, remove capacity from every [[storage.pool]] in every node configuration. If the configuration is not changed, check-upgrade in step 5 fails while loading the configuration and suggests using kibo storage-pool set-capacity instead.

kibotos-ctrl migrate in step 6 backfills each pool's capacity from the declarations ctrl recorded before the upgrade. The numbers used for placement do not change:

Declarations before the upgradePool capacity after the upgrade
Some nodes report the pool as availableThe smallest value declared by these nodes
All nodes that declare it report it as unavailableThe smallest of all declarations
No node declares a capacityNot set. The pool accepts no new objects

After the upgrade completes, use kibo storage-pool list to check the CAPACITY of each pool. If it shows - or needs adjusting, set it with kibo storage-pool set-capacity. See Pool capacity.

On this page