Upgrade from v0.1.x to v0.2.0
This page describes what you must do, beyond the general cross-version upgrade steps, when you upgrade from v0.1.x to v0.2.0. "Step N" on this page refers to the step table in Cross-version upgrades.
v0.2.0 reads only state left by v0.1.x. The check in step 5 lists the following objects as blocking. Handle them as the table describes before you replace the files:
| Object | Handling |
|---|---|
A ctrl database from a version earlier than v0.1.x | First upgrade fully to the latest v0.1.x, then upgrade to v0.2.0 |
A Workspace started under v0.0.3, v0.1.0-rc.1, or v0.1.0-rc.2 and never stopped since | Stop it normally once with the current version. Its runtime is outside the range the target version can drive, and its local records may be in a format the target version cannot read. On its next start, it switches to the target version's runtime |
A record of an ended Instance reported as has version 7, this build reads [8 9] | After you stop the node, move this file outside storage.root and keep it for reference |
volumes/metadata-v1/ and retained-layers/metadata-v2/ in a pool | Delete these two directories. They are metadata backups left by the upgrade from v0.0.3, and the target version no longer reads them |
An Instance reported as node Instance <id> is active or requires cleanup whose Workspace is FAILED | Run kibo workspace stop --force on this Workspace with the current version, so that the node cleans up its Instance. Handle all FAILED Workspaces before the window starts, so that you do not have to bring the original version back up during the shutdown |
A command reported as Workspace command <id>: the command is still processing whose Volume has already been deleted | While ctrl is stopped, end it as aborted as described below |
After you move away the record of an ended Instance, starting, stopping, and deleting its Workspace are not affected. The items related to this Instance in the cleanup status show as unknown.
After the upgrade, ctrl has the existing records of ended Instances deleted batch by batch, and kibotos_node_instances{state="stopped"} and {state="destroyed"} go down accordingly. Records with no matching Allocation stay where they are, because ctrl does not recognize them.
When v0.1.x deletes a Volume, it does not end resize, create, or delete commands on it whose result is unknown. Such a command stays processing indefinitely, resending it does not end it, and the target version's check lists it as an unfinished lifecycle operation. After you confirm that the command in the report belongs to a deleted Volume, run the following as ctrl's database account:
UPDATE kibotos_ctrl_commands
SET status = 'completed', error_code = 10, updated_at = now()
WHERE command_id = '<command ID>'
AND status = 'processing'
AND volume_id IN (SELECT volume_id FROM kibotos_ctrl_volumes WHERE volume_state = 5);The command is handled only when the result is UPDATE 1. UPDATE 0 means it is not this kind of command; handle it according to Upgrade check for ctrl. error_code = 10 is aborted, and volume_state = 5 is DELETED, which is the same result the target version produces when it ends such commands while deleting a Volume.
Before you replace the files, also check the configuration and permissions in the following table:
| Item | What to do | Consequence if not handled |
|---|---|---|
[runtime].socket_root on KiboVM nodes | Keep it at 39 bytes or fewer. If it is longer, switch to a shorter directory. Sockets of the new runtime are one level deeper than before | The node reports an error at startup |
[[storage.pool]] in the node configuration | Remove capacity. See Moving pool capacity to ctrl | check-upgrade and node startup fail |
ctrl's database.max_connections | The default changes from 2.5 × execution_concurrency + 32 to 3 × execution_concurrency + 32, which is 80 instead of 72 at a concurrency of 16. If you did not set it, confirm that PostgreSQL can accept the extra connections. If you did, it must be at least 3 × execution_concurrency + 16. See Number of database connections | The ctrl service refuses to start and states the minimum value it needs |
The address that ctrl's database.dsn points to | It must be the PostgreSQL primary, not a cloud database proxy address, a read/write splitting address, or a connection pool. See Connecting to the database | When the proxy replaces the database session, two commands for the same Workspace may run at the same time |
ctrl's database.dsn and dsn_file | The connection string is parsed by libpq rules: the user name and password in a URI are percent-encoded, so @ is written as %40, and IPv6 addresses go in square brackets. Run psql "$DSN" -c 'select 1' with the same connection string. If it connects, no change is needed | The ctrl service cannot connect to the database |
| ctrl's permissions on the artifact store | It must be able to list objects under the prefix kibotos-transfer/v1/<namespace>/. See Prepare the bucket and permissions | Deleting a Transfer returns unavailable, failed and expired archives stay in the store, and the ctrl log has transfer archive cleanup did not finish |
| Log collection | If you select Kibotos logs by unit name, add the units in kibotos-dataplane.slice. The reference Vector configuration already includes them. See Data-plane processes | Logs from data-plane processes are not collected |
v0.2.0 changes the VMM runtime of the KiboVM backend, and v0.1.x nodes cannot start this version's KiboVM Platform. This is the opposite of "publish the Platform before upgrading the node" in the notes of Cross-version upgrades: after step 7 and before step 8 resumes writes, publish and approve this version's KiboVM Platform. New nodes start VMs only with the new runtime and do not run the old Platform. Running Workspaces are adopted as usual and are not affected.
Workspaces started by v0.1.x keep running their original runtime and Platform until their next start:
- A normal stop proceeds as usual, but the disk is written only to the host page cache, as in
v0.1.x. For each such stop, the node logsstop did not confirm the host flushwith the reasonthe VMM's runtime release does not flush on stop. - On the next start, the Workspace switches to the default of this version's KiboVM Platform. The guest protocol and rootfs contract must be the same. The kernel series may differ.
KiboVM Platforms published by v0.1.x stay in the catalog. If this version does not publish a Platform of the same kernel series, the old default of that series still shows as the default and cannot be withdrawn. Nodes no longer provide the runtime it requires, so neither new Workspaces nor restarted Workspaces select it. No action is needed.
After you upgrade to v0.2.0, you cannot roll back to v0.1.x. A v0.1.x ctrl refuses the migrated database, and a v0.1.x node cannot read the local records and pool metadata written by v0.2.0. Before you replace the files, resolve all blocking items with the check in step 5. If the check fails and the files are not replaced yet, start the original version as usual. See Rollback. Problems that appear after the replacement are fixed on v0.2.x.
Moving pool capacity to ctrl
In the target version, pool capacity is an attribute of the pool on ctrl, and the node configuration no longer declares it. Before you replace the files, remove capacity from every [[storage.pool]] in every node configuration. If the configuration is not changed, check-upgrade in step 5 fails while loading the configuration and suggests using kibo storage-pool set-capacity instead.
kibotos-ctrl migrate in step 6 backfills each pool's capacity from the declarations ctrl recorded before the upgrade. The numbers used for placement do not change:
| Declarations before the upgrade | Pool capacity after the upgrade |
|---|---|
| Some nodes report the pool as available | The smallest value declared by these nodes |
| All nodes that declare it report it as unavailable | The smallest of all declarations |
| No node declares a capacity | Not set. The pool accepts no new objects |
After the upgrade completes, use kibo storage-pool list to check the CAPACITY of each pool. If it shows - or needs adjusting, set it with kibo storage-pool set-capacity. See Pool capacity.
Reinstall and upgrade
This page shows how to reinstall a package of the same version and upgrade to a new version without stopping Workspaces. For additional upgrade steps of each version, see Upgrade from v0.1.x to v0.2.0 and Upgrade from v0.2.x to v0.3.0. For uninstalling, see Uninstall. There are two kinds of upgrades. Within a release line (between the final versions and -rc.N versions of the same MAJOR.MINOR), you replace hosts one at a time, and ctrl and nodes do not need to stop together. Upgrades across a MINOR, and those involving -alpha.N or -beta.N, replace the whole deployment during a shutdown. For a first installation, see Installation overview. For details of the node and VMM process model and the maintenance check commands, see node and VMM processes. When ctrl and the gateways run in containers, the criteria and steps are the same as on this page. For how to carry them out, see Run in containers.
Upgrade from v0.2.x to v0.3.0
This page describes what you must do, beyond the general cross-version upgrade steps, when you upgrade from v0.2.x to v0.3.0. "Step N" on this page refers to the step table in Cross-version upgrades.