Upgrade from v0.2.x to v0.3.0

This page describes what you must do, beyond the general cross-version upgrade steps, when you upgrade from v0.2.x to v0.3.0. "Step N" on this page refers to the step table in Cross-version upgrades.

Shut down and replace by following the steps in Cross-version upgrades. v0.3.0 adds caller-provided egress (Egress) and caller-chosen DNS upstreams (Resolver), and guest DNS now goes only to one address on the link. Before and after the replacement, handle the items in the following table:

Item	What to do	Consequence if not handled
ctrl database	In step 6, run `kibotos-ctrl migrate` as the check report says	The ctrl service refuses to start
ctrl's `database.max_connections`	The default changes from `3 × execution_concurrency + 32` to a fixed 32, and the minimum from `3 × execution_concurrency + 16` to 8; neither depends on concurrency any more. An explicit value still applies and can be lowered	None, apart from connections held but not used
Connecting to PostgreSQL through a connection proxy or pool	Now supported, as long as all reads and writes reach the primary. See Connecting to the database. `kibotos-ctrl migrate` must still connect to the primary directly	When migration goes through a transaction-level pool, the migration tool's session-level lock may stay on another session, and the next migration waits indefinitely
Running two ctrl processes	The second one now only answers reads, and writes return a retryable `unavailable`. See Execution ownership. When replacing ctrl, still stop the old one before starting the new one	None
Connection pool metrics	`kibotos_ctrl_db_connections`, `kibotos_ctrl_db_max_connections`, `kibotos_ctrl_db_acquire_waits_total`, and `kibotos_ctrl_db_acquire_seconds_total` no longer carry the `pool` label. Change queries and alerts that filter or group by `pool`, and import the Grafana dashboards again	Queries return no data
Callers' command IDs	Watch whether `kibotos_ctrl_nonconforming_command_ids_total` grows. If it does, the ctrl WARN log `request carries a command ID that v0.4.0 refuses` names the `principal`, `procedure`, and the start of the ID. Have these callers switch to IDs that follow the rules	`v0.3.x` works as before. After upgrading to `v0.4.0`, these requests return `invalid_argument`
ctrl's `storage.root`	The first time the new ctrl version starts, it generates `credential-keys/` here. It encrypts the Egress user names and passwords stored in the database. After startup, add it to your backups together with `pki/`. See Initialize and start	If the keys are lost, stored Egress credentials cannot be decrypted, and callers must write the passwords again with `UpdateEgress`
Kernel on node hosts	Must support the nftables `tproxy` and `socket` expressions (kernel modules `nft_tproxy` and `nft_socket`). See Prerequisites	This node does not announce the egress capability, and Workspaces that reference an Egress are not placed on it
node system account and slice	The node package creates the system account `kibotos-egress` together with `kibotos-egress.slice`. If you manage accounts yourself, keep this account. If you collect logs by unit name, add the units in this slice. See Egress processes	The egress process cannot start, or its logs are not collected
Deployments whose first deployment DNS server is an IPv6 address: Platform	Before step 8 resumes writes, publish and approve this version's Platform, and use `kibo platform set-default` to make it the default revision of every PlatformKey in use. See Catalog. Workspaces switch to the default revision when they start. Deployments whose first deployment DNS server is an IPv4 address do not need this step	Workspaces started after the upgrade get the nameserver `fe80::1%eth0`. The guest in Platforms published before `v0.3.0` rejects addresses with an interface identifier like this, and the Workspace fails to start
Labels and annotations written by the E2B gateway	After it starts, the gateway rewrites on its own the keys of existing sandboxes, Volumes, and Egresses in this deployment from `e2b.kibotos.io/` and `m.e2b.kibotos.io/` to `e2b.kibotos.app/` and `m.e2b.kibotos.app/`. The log `gateway keys migrated` records how many of each kind were rewritten. Change operations scripts that filter resources by the old prefix to use the new prefix	Scripts cannot find the rewritten resources. From `v0.4.0`, the gateway no longer reads the old prefix, and resources that still carry it no longer exist for the gateway
node-local Instance records	Records reported in step 5 as `has version 8, this build reads [9 10]` were written by `v0.1.x` and never rewritten by `v0.2.x`. For a running Instance, stop it normally once with the current version. For an ended one, stop the node and move this file outside `storage.root` to keep for reference. Starting, stopping, and deleting this Workspace are not affected, and the items related to this Instance in the cleanup status show as unknown	`check-upgrade` and node startup fail
Cluster metric values	The `status` of `kibotos_cluster_node_status` gains `retired`, and the `state` of `kibotos_cluster_volumes` gains `lost`. For alerts that sum or enumerate all values, include or exclude these	Retired nodes or lost Volumes are counted as abnormal
Revoked nodes	They are no longer listed as pool members, and no longer count as the other attached node when another node enters `DRAINED`. Before the upgrade, find `REVOKED` nodes with `kibo node list`. If only one unrevoked node remains on a shared pool, that node cannot drain to `DRAINED`, and you must first move or delete the objects in the pool	Drain stays at `DRAINING` indefinitely
Cluster metrics	Change queries and alerts that use `kibotos_cluster_node_allocatable_ephemeral_disk_bytes` and `kibotos_cluster_node_allocated_ephemeral_disk_bytes` to `kibotos_cluster_storage_pool_capacity_bytes` and `kibotos_cluster_storage_pool_reserved_bytes`, and import `grafana-cluster.json` again. See Grafana	These two series are no longer updated, and the ephemeral disk panels of the old dashboard are empty
ctrl's `[metrics]`	For E2B deployments, set `instance_info_labels` to `["e2b.kibotos.app/namespace", "e2b.kibotos.io/namespace"]`, and import the two Grafana dashboards again. See Instance metrics	Without the new key, metrics for all sandboxes are empty arrays. Per-namespace panels of the old dashboards are empty

To tell whether the E2B gateway rewrite is complete: a few minutes after the gateway starts, kibo workspace list -l e2b.kibotos.io/namespace=<namespace> and kibo volume list -l e2b.kibotos.io/namespace=<namespace> should return nothing. If they still return output, look at the gateway log:

gateway key migration incomplete: this pass did not finish, and the gateway tries again in about a minute.
gateway keys left in legacy spelling: ctrl refused the rewrite because the longer keys exceed the total length limit of labels. In v0.3.x the gateway still reads such a resource as usual. Delete this sandbox or Volume before you upgrade to v0.4.0.

The rewrite also causes two visible differences:

A create-sandbox request that was sent with an Idempotency-Key before the upgrade and retried only after the upgrade returns a conflict, because the request generates a different label. Create it again with a different key.
Instance metric samples written before v0.3.0 carry label_e2b_kibotos_io_namespace. When the gateway queries sandbox metrics, it recognizes both labels. The per-namespace panels of the Grafana dashboards show only samples with the new label.

The new node version rewrites Instance records to version 10, which v0.2.x nodes cannot read. Before you replace the files, resolve all blocking items with the check in step 5.

Platform archives, Template manifests, and build results built by v0.2.x use format identifiers under kibotos.io/. This version reads them as usual. From v0.4.0, they are no longer read.

After the upgrade, PUBLIC_INTERNET Workspaces behave differently in one way: new TCP flows that do not start with a SYN are reset. Old connections whose connection tracking state was lost (for example, after the host's connection tracking table was flushed) are therefore reset and stop transferring, and programs in the guest must reconnect.

Guest DNS configuration is fixed when the Workspace starts. During the upgrade, Workspaces fall into two groups:

Workspace	DNS configuration	Can reference a Resolver
Started before the upgrade and still running after it	Unchanged: `/etc/resolv.conf` still lists the deployment DNS servers, and when the node restarts it syncs them from the current node configuration	No. `UpdateWorkspaceNetworkPolicy` returns `failed_precondition`, and neither the desired nor the effective policy changes
Started or restarted after the upgrade	`/etc/resolv.conf` has only the link DNS address: the guest gateway address in the address family of the first deployment DNS server, which for IPv6 is `fe80::1%eth0`. This requires this version's Platform (see the table above)	Yes

For a Workspace that should reference a Resolver, stop it and start it again once after the upgrade.
After the switch to the link DNS address, a Workspace that does not reference a Resolver uses only the first deployment DNS server in the same address family as the link address, and no longer uses the others. The address family of the first server determines the address family of the link address. If you configured several deployment DNS servers and rely on the others as backups, put the most reliable one first.
DNS queries from DENY_ALL Workspaces fail immediately, and UDP that the guest sends directly to a deployment DNS server is rejected.
If the deployer restricts the node's own outbound traffic on the host, Resolver upstreams that go direct are restricted too.