Install ctrl

This page shows how to install and initialize ctrl on a host and issue the first Operator enrollment key.

Run the following commands in the verified release set directory, adjusting the version number to match.

1. Install the package and write the configurationCopy Anchor Link

sudo apt-get install ./kibotos-ctrl_1.2.3_amd64.deb
sudo install -o kibotos-ctrl -g kibotos-ctrl -m 0600 \
  /usr/share/doc/kibotos-ctrl/ctrl.example.toml /etc/kibotos/ctrl/config.toml
sudoedit /etc/kibotos/ctrl/config.toml

In the configuration, fill in the database, the artifact store, identity.host, and the data-plane names. For every table and key, see ctrl configuration.

Notes:

• Only kibotos-ctrl may read the configuration file. When you use dsn_file, that file must also be readable by this account.
• To provide E2B sandbox metrics, fill in the [metrics] section here as well. See Configuring ctrl.
• To export ctrl's own metrics and cluster status, fill in the [telemetry] section. See Listener in Cluster and service metrics. It is unrelated to [metrics].
• The state directory is fixed at /var/lib/kibotos/ctrl, and the service can write only to this directory.
• The package only creates the account and directories and reloads systemd. Initialization, migration, enabling, and starting are done in the steps below.

Connecting to the databaseCopy Anchor Link

database.dsn can point to the primary itself, or to a connection proxy or connection pool, as long as all reads and writes reach the primary.

A proxy or pool affects only performance. Only one ctrl executes, which a row in the database guarantees independently of connections and database sessions. A second ctrl on the same database only answers reads. See Execution ownership.

kibotos-ctrl migrate must connect to the primary directly. The migration tool uses a session-level lock to keep two migrations from running at the same time, and a transaction-level pool leaves that lock on a session that another client uses. For migration, use a configuration file whose database.dsn points to the primary. Other commands are not subject to this restriction.

A database failover invalidates the prepared statements on a connection. Behind a proxy, ctrl's connections are not dropped; each connection fails at most once before it is replaced, and callers see a retryable error.

To check that the DSN reaches the primary, run this a few times. The first column must be f every time:

psql "$DSN" -Atc 'select pg_is_in_recovery()'

Number of database connectionsCopy Anchor Link

At runtime, ctrl uses at most database.max_connections PostgreSQL connections, 32 by default. PostgreSQL's max_connections must accommodate this number, plus other clients and a few short-lived connections from one-off kibotos-ctrl commands (migration, issuing enrollment keys, and so on).

• A running command holds a connection only while it reads or writes the database, so raising execution_concurrency does not require more connections.
• When kibotos_ctrl_db_acquire_waits_total keeps growing and requests slow down, raise max_connections.
• The connection limit can be set only in database.max_connections. pool_max_conns in the DSN is rejected.
• At startup, ctrl logs a database connections line giving the actual concurrency and connection limit.

2. Prepare the bucket and permissionsCopy Anchor Link

Before starting ctrl, the storage administrator creates the bucket named in the configuration and grants access to ctrl's runtime identity.

• For the required permissions (object read and write, delete, multipart upload, and listing by prefix), see the example policy. When deploying, replace the resources with the actual bucket and namespace. Automatic object storage reclamation needs the delete permission.
• The permissions cover two prefixes: kibotos-template/v1/<namespace>/ holds Template artifacts, and kibotos-transfer/v1/<namespace>/ holds Transfer archives.
• Create the bucket with a separate administrative identity. Do not put administrative credentials in the ctrl configuration.
• Transfer archives are data from user Volumes. Enable encryption at rest and access auditing on the bucket, and restrict the identities that can read it to ctrl's runtime identity.

Every time ctrl starts, it checks that storage is usable: before opening the API and starting background tasks, it uses the runtime identity to write one small random object under kibotos-template/v1/<namespace>/staging/ and then one under kibotos-transfer/v1/<namespace>/, reads each back to verify it, and deletes it. The whole check is limited to 10 seconds.

• If any step fails, ctrl does not start. The operation field in the journal tells whether the failure was in writing, reading, verifying, or deleting. The reason field distinguishes a missing bucket, insufficient permissions, a configuration error, and unreachable storage, and gives a suggested fix.
• The check uses only object permissions on these two prefixes. It does not create a bucket or start a multipart upload.
• When the check fails, it tries to delete the objects it wrote. On a timeout or without delete permission, small objects may be left behind; configure a lifecycle rule on the staging prefix to reclaim them.
• This check proves only that basic reads and writes work. Capabilities such as multipart upload are verified the first time a Template is published.
• Publishing failures at runtime are logged in the journal with the stage and the command/session identifiers. Transient storage failures do not make a running ctrl exit.

Stores without S3 conditional writes or additional checksums, such as Alibaba Cloud OSS, also work: ctrl switches to the OSS no-overwrite header and verifies each published blob by reading it back, which costs one extra transfer per blob between storage and ctrl.

Alibaba Cloud OSS supports only virtual-hosted addressing. In the configuration, set path_style = false, use the regional domain without the bucket as endpoint (such as oss-cn-shenzhen.aliyuncs.com), and fill in bucket separately.

3. Initialize, issue the first Operator enrollment key, and startCopy Anchor Link

umask 077
sudo -u kibotos-ctrl kibotos-ctrl init --config /etc/kibotos/ctrl/config.toml
sudo -u kibotos-ctrl kibotos-ctrl migrate --config /etc/kibotos/ctrl/config.toml
sudo -u kibotos-ctrl kibotos-ctrl enrollment create \
  --config /etc/kibotos/ctrl/config.toml --operator operator-a \
  --output /var/lib/kibotos/ctrl/operator-enrollment.key
sudo systemctl enable --now kibotos-ctrl.service
sudo journalctl -u kibotos-ctrl.service --no-pager -n 50

• Run all three kibotos-ctrl commands as the service account with sudo -u kibotos-ctrl. Plain sudo creates pki/ as root, the service account cannot enter it, and the service fails at startup with load managed PKI state: ... permission denied. When init finds that the state directory does not belong to the current account, it refuses and tells you which account to rerun it as. If you already created it as root, hand it back with sudo chown -R kibotos-ctrl:kibotos-ctrl /var/lib/kibotos/ctrl/pki, then run init once more as the service account.
• init can be run repeatedly. When already initialized, it prints unchanged. If any step fails, resolve the error before continuing.
• At boot, PostgreSQL may become ready later than ctrl, for example when the database runs in a local container. When ctrl cannot connect to the database, it exits, and systemd restarts it every 5 seconds until the database is available. This interval does not hit systemd's default start limit (5 starts within 10 seconds), so no manual reset-failed is needed. The log repeatedly shows connect ctrl PostgreSQL: ... connection refused, which stops once the database is ready.
• When the database stays unavailable, ctrl keeps retrying. To have ctrl start only after the database service is ready, use systemctl edit kibotos-ctrl.service to add After= and Wants= for that service.
• The kibotos-ctrl account writes the enrollment key to its own state directory with mode 0600. The administrator reads it and hands it to the Operator over a secure channel; there is no need to loosen the file permissions.
• For PKI renewal and recovery, see Identity, configuration, and connections.

Automatic object storage reclamationCopy Anchor Link

Ten minutes after it starts, ctrl begins reclaiming object storage automatically, and repeats every 24 hours. It deletes only manifests and blobs that meet all conditions: withdrawn more than 7 days ago, and no Workspace, retention period, or unwithdrawn revision in the catalog still references them.

For reclamation records, manual planning and execution, and excluding a Launch from reclamation, see Artifact reclamation.

Next steps: enroll an OperatorCopy Anchor Link

On their own workstation, the Operator uses this key to set up the kibo connection configuration, then issues an enrollment key for each node. For the steps, see Operator setup and node enrollment. The node-enrollment.key used in Install node comes from that step.