Run in containers

Each release of ctrl, the E2B gateway, and the Exposure gateway also ships container images that you can run with Kubernetes or Docker. You still install node on hosts. See Install node. Use these images if you operate a container platform.

This page covers only what differs from host installation. For the meaning of configuration items, see ctrl configuration, Install ctrl, and the variable tables in Install the E2B gateway and Install the Exposure gateway. For upgrade criteria, see Reinstall and upgrade. The example manifests are in deploy/kubernetes/ and deploy/compose/.

ImagesCopy Anchor Link

• Metrics ports open only when a metrics listener is configured. See Listener in Cluster and service metrics.
• The E2B gateway also opens an envd-only listener on 127.0.0.1:18096 inside the container. The examples do not publish it: port 18095 already serves envd requests.
• Tags match the release version, for example v1.2.3 and v1.2.3-rc.1. There is no moving tag such as latest.
• The binaries in the images come from the deb packages of the same version and are byte-for-byte identical.
• Each release has an asset kibotos-images_<version>.txt that lists the three images by digest and is covered by SHA256SUMS. Reference images by digest when deploying, so they can be checked against the downloaded release set.

Use the directory from downloading the release set to see the images for this version, and check the version the image reports:

cat kibotos-releases/v1.2.3/kibotos-images_1.2.3.txt
docker run --rm ghcr.io/felinics/kibotos-ctrl:v1.2.3 version --json

The source repository is not public yet, so pulling images also requires read access to the repository: log in to ghcr.io with a GitHub token that has the read:packages permission, and in Kubernetes, turn it into an imagePullSecret.

Differences from host installationCopy Anchor Link

State on persistent volumesCopy Anchor Link

• The PKI of ctrl and the identities of the gateways are on persistent volumes. The volume is mounted at /var/lib/kibotos, and each service creates its own 0700 subdirectory in it.
• Losing ctrl's volume loses this deployment's identity: all nodes and Operators must enroll again. Back it up as you back up the database.
• Do not mount a Secret or ConfigMap directory as a state or identity directory. They consist of symbolic links and carry group permissions, and the services refuse to load them.
• In Kubernetes, set fsGroup and fsGroupChangePolicy: OnRootMismatch. fsGroup lets the service create its own directory under the volume root. The policy's default, Always, makes kubelet add group read and write to the files in the volume on every mount, and ctrl then refuses to start the next time because the private key permissions are too broad.
• A Docker named volume copies the image's directory on first mount, so ownership is already correct. When you use a host directory instead, first create the service's subdirectory with a command such as install -d -o 10001 -g 10001 -m 0700.

ConfigurationCopy Anchor Link

• The configuration of ctrl can be mounted as a whole directory. Kubernetes mounts a Secret or ConfigMap at /etc/kibotos/ctrl, and Docker bind-mounts a directory. config.toml, dsn_file, and the metrics signing private key file can be symbolic links, but they must ultimately point to regular files that the service account can read.
• Gateway configuration comes only from environment variables, with the same variable names as the package's config.env. For the full lists, see e2b.example.env and gw.example.env. Listen addresses default to loopback; in containers, change them to 0.0.0.0.
• When a required item is missing or a value is invalid, the gateway lists all problematic variables at once and exits, without printing their values.
• Metrics listeners are off by default. For ctrl, write [telemetry] in config.toml. For the two gateways, set KIBOTOS_E2B_TELEMETRY_LISTEN and KIBOTOS_GW_TELEMETRY_LISTEN, also bound to 0.0.0.0 in containers. The example manifests already turn on the listeners of both gateways. For how to collect them, see Kubernetes in Cluster and service metrics.

Only one ctrlCopy Anchor Link

• Run only one ctrl at a time. In Kubernetes, use replicas: 1 and strategy: Recreate. Do not use RollingUpdate, and do not raise the replica count. With Docker, start only one container.
• A second ctrl connected to the same database only answers reads and never writes at the same time as the first. See Execution ownership. The two processes still must not share one storage.root.
• A ReadWriteOnce volume prevents a second Pod scheduled on another host from mounting it, but it does not replace Recreate.
• Do not run init automatically on every start, for example by putting it in an initContainer. When the volume is mounted wrongly or lost, serve should fail to load, but init would generate a new deployment identity.
• When the host running ctrl loses contact, the old Pod stays in Terminating, the volume is not released, the new Pod cannot start, and no ctrl is available in the meantime. Force-delete the old Pod only after confirming that the old host has shut down.

Entry point and hostnameCopy Anchor Link

• Only L4 passthrough can sit in front of ctrl's two ports. Both use end-to-end mTLS. In Kubernetes, use a LoadBalancer or NodePort Service, not an Ingress.
• [identity].host resolves to this entry point. The node's [ctrl].endpoint, kibo, and the gateways all connect to ctrl with this name, because ctrl's certificate is signed only for this name.
  • Inside the cluster, gateways cannot connect to ctrl with the Service's in-cluster name. When cluster DNS cannot resolve [identity].host, use hostAliases in Kubernetes to point it to the ctrl Service's address. The Compose example gives ctrl a network alias with the same name.
• Set the artifact store endpoint to an address that both nodes and Operator workstations can reach directly. The presigned URLs that ctrl issues use the same endpoint; an in-cluster address makes node downloads fail.
• Gateways must be able to resolve and reach the node data-plane hostnames, that is, <node_id>.<host_suffix>.
• Enable time synchronization on the host running ctrl. Containers use the host's clock. For the requirements, see Time synchronization.

Gateways: single instance, self-enrollment on first startCopy Anchor Link

Both gateways are stateless, but each instance needs its own Operator identity. When several instances share one identity directory, each renews on its own. Because ctrl accepts only the current and previous certificates of an identity, a third instance is rejected. The example manifests provide one identity per gateway, so they run one instance of each. More instances each need their own identity, as on hosts. See Identity rules.

• The gateway's identity directory is on a persistent volume. When the volume is empty, the gateway enrolls with the one-time enrollment key that KIBOTOS_E2B_ENROLLMENT_KEY_FILE or KIBOTOS_GW_ENROLLMENT_KEY_FILE points to, and only then starts serving.
• A gateway that has already enrolled no longer reads this file, so a used key can stay in the Secret.
• A key is valid for 1 hour by default and at most 24 hours (--ttl). Start the gateway within the validity period after issuing it.
• Secret files mounted by Kubernetes have group read permission, so the gateway logs a "permissions too broad" warning when enrolling. You can ignore it.
• After a gateway's volume is lost, enrollment fails at startup and the gateway does not start serving. Issue a new key for the same principal, update the Secret, and restart the gateway.
• The Kubernetes examples use strategy: Recreate: the new Pod starts only after the old Pod stops, so the same identity is never used by two processes at once.

KubernetesCopy Anchor Link

The example manifests assume the namespace kibotos and use the following files. Adjust the image versions, hostnames, domains, and storage sizes to your environment before applying them.

The NetworkPolicies of the three manifests allow only the metrics collector to connect to the telemetry port. The collector's manifest is deploy/metrics/kubernetes-vmagent.yaml. See Kubernetes in Cluster and service metrics.

The storage that vmagent writes to can be installed in the same cluster with the Helm chart deploy/helm/kibotos-metrics/. See VictoriaMetrics in the cluster.

Instance metrics storage can be installed in the same cluster with the Helm chart deploy/helm/kibotos-instance-metrics/. See Reference deployment on Kubernetes.

1. Prepare the namespace, pull credentials, and ctrl configurationCopy Anchor Link

Fill in config.toml as in Install the package and write the configuration, keeping [storage].root at the default /var/lib/kibotos/ctrl. It contains database and storage credentials, so put it in a Secret. The whole Secret is mounted at /etc/kibotos/ctrl. Put the dsn_file, the metrics signing private key, and the CA files used by the configuration in the same Secret, with paths written as /etc/kibotos/ctrl/<file name>. To export ctrl's own metrics, add a [telemetry] section to the configuration with listen set to 0.0.0.0:18192. When enabling instance metrics, add one more --from-file=metrics-signing-key.pem=.... For how to generate it, see Write credentials:

set -eu
kubectl create namespace kibotos
kubectl -n kibotos create secret docker-registry ghcr \
  --docker-server=ghcr.io --docker-username=deploy-bot --docker-password="$GHCR_TOKEN"
kubectl -n kibotos patch serviceaccount default -p '{"imagePullSecrets":[{"name":"ghcr"}]}'
kubectl -n kibotos create secret generic kibotos-ctrl-config --from-file=config.toml=./config.toml

Preparing the bucket and permissions is the same as for host installation. See Prepare the bucket and permissions.

2. Initialize ctrlCopy Anchor Link

Create the volume first, then run init and migrate on it in order. After each Job completes, check its log, then delete it:

set -eu
kubectl apply -f deploy/kubernetes/ctrl-state.yaml
for command in init migrate; do
  sed "s/COMMAND/$command/g" deploy/kubernetes/ctrl-command.yaml | kubectl apply -f -
  kubectl -n kibotos wait --for=condition=complete --timeout=10m "job/kibotos-ctrl-$command"
  kubectl -n kibotos logs "job/kibotos-ctrl-$command"
  kubectl -n kibotos delete job "kibotos-ctrl-$command"
done

When already initialized, init prints unchanged and does not change any file.

3. Start ctrl and issue the first Operator enrollment keyCopy Anchor Link

set -eu
kubectl apply -f deploy/kubernetes/ctrl.yaml
kubectl -n kibotos rollout status deploy/kibotos-ctrl --timeout=5m
kubectl -n kibotos get service kibotos-ctrl

Point the DNS record of [identity].host to the Service's external address. Then issue the key inside the ctrl container, copy it to your machine, and delete the copy in the container:

set -eu
umask 077
kubectl -n kibotos exec deploy/kibotos-ctrl -- \
  kibotos-ctrl enrollment create --operator operator-a --output /var/lib/kibotos/ctrl/operator-a.key
kubectl -n kibotos exec deploy/kibotos-ctrl -- cat /var/lib/kibotos/ctrl/operator-a.key > operator-a.key
kubectl -n kibotos exec deploy/kibotos-ctrl -- rm /var/lib/kibotos/ctrl/operator-a.key

Then use this key to enroll kibo as described in Identity and enrollment, and connect nodes as described in Install node.

4. Start the E2B gatewayCopy Anchor Link

First issue an enrollment key for the gateway. Non-sensitive variables go in the ConfigMap in e2b.yaml. The API keys, token keys, and namespace go in a Secret; e2b-secret.env contains only these three variables:

set -eu
umask 077
kibo enrollment create --operator kibotos-e2b --ttl 24h --output kibotos-e2b.key
kubectl -n kibotos create secret generic kibotos-e2b-enrollment --from-file=enrollment.key=kibotos-e2b.key
kubectl -n kibotos create secret generic kibotos-e2b --from-env-file=e2b-secret.env
kubectl -n kibotos create secret tls kibotos-e2b-tls --cert=sandbox.crt --key=sandbox.key
kubectl apply -f deploy/kubernetes/e2b.yaml
kubectl -n kibotos rollout status deploy/kibotos-e2b --timeout=10m
rm kibotos-e2b.key

The Ingress has the same requirements as the reverse proxy on a host. See Configure the reverse proxy: the certificate covers api.<domain> and *.<domain>, the original Host is preserved, WebSocket is allowed, and request and response buffering are off. With ingress-nginx, add the two annotations nginx.ingress.kubernetes.io/proxy-buffering: "off" and nginx.ingress.kubernetes.io/proxy-request-buffering: "off".

5. Start the Exposure gatewayCopy Anchor Link

Fill in [exposure] zone in ctrl's config.toml first. The tunnel credentials file is created by the provider; put it in a Secret:

set -eu
umask 077
kibo enrollment create --operator kibotos-gw --ttl 24h --output kibotos-gw.key
kubectl -n kibotos create secret generic kibotos-gw-enrollment --from-file=enrollment.key=kibotos-gw.key
kubectl -n kibotos create secret generic kibotos-cloudflared --from-file=credentials.json=./tunnel-credentials.json
kubectl apply -f deploy/kubernetes/gw.yaml
kubectl -n kibotos rollout status deploy/kibotos-gw --timeout=10m
rm kibotos-gw.key

• The cloudflared replicas all run the same named tunnel and send the whole zone to the gateway's Service. For the tunnel configuration, see Configure the tunnel.
• Traffic between the gateway and the tunnel is plain HTTP. The example NetworkPolicy allows only cloudflared Pods to connect to the gateway's service port and only the metrics collector to connect to the metrics port. The cluster must have NetworkPolicy enabled.

Docker ComposeCopy Anchor Link

compose.yaml runs ctrl, the E2B gateway, the Exposure gateway, and cloudflared on one Docker host. It reads the following files from the same directory:

• In both environment files, set CTRL_ENDPOINT to https://<[identity].host>:18092: on the Compose network, ctrl carries an alias with this name. The example uses ctrl.kibotos.example; change it to the actual name.
• The ingress in gw/cloudflared.yaml points to http://gw:18097; the rest is the same as in Configure the tunnel. Set credentials-file to /etc/cloudflared/credentials.json.
• If you do not offer Exposure, remove the gw and cloudflared services.
• The metrics listeners of both gateways are already on and reachable only on the Compose network. For ctrl, write [telemetry] in ctrl/config.toml. For how to collect them, see Docker Compose in Cluster and service metrics.

Taking ctrl's configuration as an example, hand the directory and file over to the service account:

set -eu
sudo install -d -o 10001 -g 10001 -m 0700 deploy/compose/ctrl
sudo install -o 10001 -g 10001 -m 0600 config.toml deploy/compose/ctrl/config.toml

Initialize ctrl, start it, and issue the first Operator enrollment key:

set -eu
cd deploy/compose
docker compose run --rm ctrl init
docker compose run --rm ctrl migrate
docker compose up -d ctrl
docker compose exec ctrl kibotos-ctrl enrollment create --operator operator-a --output /var/lib/kibotos/ctrl/operator-a.key
(umask 077; docker compose exec ctrl cat /var/lib/kibotos/ctrl/operator-a.key > operator-a.key)
docker compose exec ctrl rm /var/lib/kibotos/ctrl/operator-a.key

After enrolling kibo with this key, issue keys for the gateways, put them in the locations in the table above, and start the remaining services:

set -eu
cd deploy/compose
docker compose up -d
docker compose ps
docker compose logs e2b gw

The E2B gateway is published on 127.0.0.1:18095, and the HTTPS reverse proxy on the host forwards to it.

One-off commandsCopy Anchor Link

The kibotos-ctrl commands that host installation runs as the service account run as follows in container deployments:

• One-off commands all run as the service account in the image. Do not add --user 0 or switch to root: the service account cannot read files created by root.
• A check-upgrade Job run while ctrl is running must land on the same host as ctrl to mount the volume, so ctrl-command.yaml sets Pod affinity. When it is scheduled elsewhere, the Job stays in ContainerCreating.

UpgradesCopy Anchor Link

The criteria and steps are the same as for host deployments. See Rolling upgrades within a MINOR and Cross-version upgrades.

Upgrading ctrlCopy Anchor Link

The only way to replace ctrl is to stop it, then start it. During the replacement, new Workspace operations and new data-plane sessions wait for it to come back; established data-plane connections are not affected. Taking Kubernetes as an example, upgrading v1.2.3 to v1.2.4:

set -eu
target=v1.2.4
run_ctrl_command() {
  sed -e "s/COMMAND/$1/g" -e "s/kibotos-ctrl:v[^ ]*/kibotos-ctrl:$target/" deploy/kubernetes/ctrl-command.yaml | kubectl apply -f -
  kubectl -n kibotos wait --for=condition=complete --timeout=10m "job/kibotos-ctrl-$1" || true
  kubectl -n kibotos logs "job/kibotos-ctrl-$1"
  kubectl -n kibotos delete job "kibotos-ctrl-$1"
}
kubectl -n kibotos scale deploy/kibotos-ctrl --replicas=0
kubectl -n kibotos wait --for=delete pod -l app.kubernetes.io/name=kibotos-ctrl --timeout=2m
run_ctrl_command check-upgrade

When the report has no blockers, run the migration as the report says, then switch the image and start:

set -eu
target=v1.2.4
sed -e "s/COMMAND/migrate/g" -e "s/kibotos-ctrl:v[^ ]*/kibotos-ctrl:$target/" deploy/kubernetes/ctrl-command.yaml | kubectl apply -f -
kubectl -n kibotos wait --for=condition=complete --timeout=10m job/kibotos-ctrl-migrate
kubectl -n kibotos delete job kibotos-ctrl-migrate
kubectl -n kibotos set image deploy/kibotos-ctrl ctrl="ghcr.io/felinics/kibotos-ctrl:$target"
kubectl -n kibotos scale deploy/kibotos-ctrl --replicas=1
kubectl -n kibotos rollout status deploy/kibotos-ctrl --timeout=5m

• When the report has blockers, do not switch the image. Scale the replicas back to 1, and ctrl resumes service on the original version.
• Docker Compose follows the same order: docker compose stop ctrl, run docker compose run --rm ctrl check-upgrade and migrate with the target version's image, change the version in compose.yaml, then docker compose up -d ctrl.
• Once the migration has run, older versions of ctrl reject the newer schema, and the only way out is to fix forward.

Upgrading the gatewaysCopy Anchor Link

Within the same release line, the gateways can be replaced at any time after ctrl is upgraded:

• Kubernetes: kubectl -n kibotos set image deploy/kibotos-e2b e2b=ghcr.io/felinics/kibotos-e2b:v1.2.4.
• Docker Compose: change the version in compose.yaml, then docker compose up -d e2b.

Each gateway has only one instance, so its entry point is briefly unavailable during the replacement. When the gateway receives the stop signal, it immediately stops accepting new connections, and in-flight requests have 5 seconds to finish. Long-lived connections still open after that (command output streams, terminals, Exposure WebSockets) are closed; clients reconnect.

In a cross-version upgrade, the gateways are replaced as part of the shutdown procedure: upgrade ctrl first, then the gateways.

TroubleshootingCopy Anchor Link