Identity, configuration, and connections

This page describes the identity, certificate, and configuration rules that ctrl, nodes, and clients share. It answers questions such as "what does this configuration item mean" and "where does this certificate come from, and when does it expire".

• For first-time deployment steps, see Installation overview.
• For kibo flags and exit codes, see kibo CLI reference.

Source of truth for configurationCopy Anchor Link

Both ctrl and node read only strict TOML: an unknown field is an error. For the full set of fields, example values, and constraints, the authority is ctrl.example.toml and node.example.toml. The values in the examples are only examples, not the defaults of every field. Every table and key, with the comments of the examples, is listed in ctrl configuration and node configuration.

• The default configuration path is /etc/kibotos/ctrl/config.toml for ctrl and /etc/kibotos/node/config.toml for node. Use --config to give another absolute path.
• Certificates, CAs, and signing private keys are kept in directories managed by ctrl or node, not in the TOML.
• The node ID comes from the client certificate obtained during enrollment. It is not a configuration item.
• The node's [resources] declares the CPU and memory this host can give to VMs: the physical amount minus what the host keeps for itself. It cannot exceed the host's online CPU count and MemTotal, or the node refuses to start. Overcommit is not set here. Operators set the ratios with kibo node overcommit. See Overcommit ratios.
• The Template cache limit is declared with [storage].template_cache_limit. When the cache needs space, the node evicts content not used by any running Instance, least recently used first, including partial files left by interrupted transfers. After you lower the limit, the node evicts down to the limit at startup. Content used by running Instances is kept. If eviction falls short, the node still starts, and preparing new Templates fails for lack of capacity until those Instances stop. The storage capacity for Volumes and retained layers is each pool's capacity on ctrl, set with kibo storage-pool set-capacity. The node does not detect machine capacity automatically.
• Omit [host_network] when guests need no network. For the dual-stack interfaces, routes, and DNS configuration that the public-internet policy needs, see the node configuration example.

Identity and enrollmentCopy Anchor Link

The initialization order of ctrl is kibotos-ctrl init → migrate → local enrollment create → serve.

init does not use the network, connect to the database, or listen on ports. It creates four things: the CA that issues client certificates, the ctrl server CA, the signing key for data-plane session credentials, and the ctrl server private key. When the state is complete, running it again returns unchanged. When the state is incomplete or mismatched, it fails and does not repair anything automatically.

Identity rulesCopy Anchor Link

• The URI of a full identity is spiffe://<trust-domain>/operator/<id> or spiffe://<trust-domain>/node/<id>.
• A node ID must be in lowercase domain-name form, because it becomes part of the data-plane certificate hostname.
• Access rights are decided by the enrollment records in ctrl. Currently enrolled Operators have cluster administration rights and can call the Workspace API and the Operator API. Trusting only the ctrl server certificate (as a browser does) grants no rights.

Enrollment key rulesCopy Anchor Link

• A key is valid for 1 hour by default. --ttl adjusts it between 1 minute and 24 hours.
• The key contains a bootstrap private key and the ctrl server CA. Treat it as a credential: keep it out of command-line arguments, environment variables, logs, and version control.
• Delete the key file after enrollment succeeds.

kibo context enroll and kibotos-node join read the key the same way:

• Without a key flag, they prompt for it in the terminal without echo. When standard input is not a terminal, give exactly one of --key-stdin, --key-fd <n> (n is 3 or higher), or --key-file <path>.
• The key cannot be passed as a command-line argument or in an environment variable.
• The key must be a single token of at most 16 KiB.
• --key-file accepts relative paths, absolute paths, and symbolic links to a regular file. FIFOs, directories, and other special files are rejected.
• When the key file is readable by group or other, the command prints a warning and still reads it. It does not change the file. Give the file mode 0600.
• Permissions on the identity directory and the private key files that enrollment writes are checked strictly.

Neither kibo context enroll nor kibotos-node join overwrites an existing identity directory. When the identity in the directory has expired or is corrupted, stop the process that uses it, delete the directory, then enroll with a new enrollment key.

kibotos-ctrl enrollment create on the ctrl host invalidates that Operator's unused older keys. For replaying kibo enrollment create, see Idempotent replay.

Server and data-plane certificatesCopy Anchor Link

Server certificate of ctrlCopy Anchor Link

During init, ctrl creates the server CA, and serve uses it to issue a server certificate for [identity].host. Every enrollment key carries this CA's certificate: kibo context enroll and kibotos-node join first use it to verify the ctrl hostname, then send the bootstrap identity. On success, they save the CA as server-ca.pem in the identity directory. From then on, both ctrl mTLS ports trust only this CA. They do not read the system root certificates, and they need no public DNS, no ACME, and no change to the system trust store.

Data-plane certificate of a nodeCopy Anchor Link

The node generates the data-plane private key locally, and ctrl issues the certificate based on the node's enrolled identity. The hostnames in the certificate come from three places, with duplicates removed:

1. <node_id>.<host_suffix>, derived by ctrl.
2. [data_plane].default_extra_hosts in the ctrl configuration.
3. --data-plane-extra-host passed when the node's enrollment key was issued.

Rules:

• An extra hostname can be a fully qualified domain name or an IP. Wildcards are not accepted. None of them can equal ctrl's own [identity].host.
• The hostname of [data_plane].advertise in the node configuration must be in the certificate.
• The derived hostname does not need to be registered in DNS. It needs to resolve only when clients use it as the connection address.

For example, with host_suffix = "nodes.kibotos.trial", node ID node-a, and --data-plane-extra-host 127.0.0.1 added when the key is issued, the certificate covers both node-a.nodes.kibotos.trial and 127.0.0.1, and clients on a single machine can connect directly to https://127.0.0.1:<port>.

The node stores the certificate as leaf.key, leaf.pem, and manifest.json under <storage.root>/pki/data-plane/, verifies and reuses it at startup, and reissues it when needed.

Changing the set of hostnames:

• To change a node's extra hostnames, revoke the node, issue a new enrollment key with the new hostnames, delete the old identity directory, and join again.
• Changing ctrl's host_suffix or default_extra_hosts affects only certificates issued afterward. Schedule it together with updates to the nodes' advertise configuration and certificates.

How data-plane clients verifyCopy Anchor Link

Data-plane clients such as kibo and the SDK use a different TLS configuration from the control plane: they trust the system root certificates plus server-ca.pem, and check hostnames by the standard rules. So clients can connect either directly to a node or to a public TLS reverse proxy that the deployer places in front of the node. The reverse proxy must be within the deployer's trust boundary and must forward streaming requests and long-lived connections correctly. This public TLS termination applies only to the data plane. The ctrl mTLS ports always require an enrolled identity and ctrl's own server CA.

Client connectionsCopy Anchor Link

Context directoryCopy Anchor Link

kibo context enroll saves the ctrl address, the Operator identity, the trusted CAs, and the optional forward proxy in one directory, called the context. Every later kibo command reads it.

• The default location is $XDG_CONFIG_HOME/kibo/context, or ~/.config/kibo/context when XDG_CONFIG_HOME is not set. The environment variable KIBO_CONTEXT_DIR can point to another absolute path.
• The directory contains context.json, the current identity-<serial>.pem and identity-<serial>.key, control-client-ca.pem, and server-ca.pem.
• enroll refuses to overwrite an existing context directory. To enroll again, delete the old directory first.
• The global flag --endpoint or the environment variable KIBO_ENDPOINT changes only the address for the current request. The identity and the trusted CAs still come from the context.

Forward proxyCopy Anchor Link

kibo context enroll --proxy and the node's [ctrl].proxy accept http://, https:// (CONNECT), or socks5:// proxies, for example --proxy socks5://proxy.example:1080.

• kibo saves the proxy given to enroll in the context and uses it for every later command, including data-plane connections.
• An explicit setting takes precedence. Without one, the proxy is chosen from HTTPS_PROXY, ALL_PROXY, and NO_PROXY (and their lowercase forms). Connections to loopback addresses do not use proxies from environment variables.
• TLS verification and mTLS happen inside the proxy tunnel. The username and password in the proxy address never appear in output or logs.
• The data plane does not follow redirects, and sends the session credential only after TLS verification succeeds.

Renewal, revocation, and recoveryCopy Anchor Link

LifetimeCopy Anchor Link

• When automatic renewal fails, kibo prints a warning and keeps using the identity until it expires.
• An expired identity cannot be renewed. Enroll again with a new enrollment key.

RevocationCopy Anchor Link

kibo enrollment revoke --operator <id> --yes or --node <id> --yes ends that identity immediately and cannot be undone: unused enrollment keys are invalidated, and later requests, enrollment, and renewal are rejected. Revoking a node also closes its current control connection to ctrl.

Revocation does not affect existing Workspaces or in-flight data-plane connections; they continue under their own rules. To only stop a node from accepting new Workspaces, use kibo node drain.

To bring a revoked Operator or node back, issue a new enrollment key, then run kibo context enroll or kibotos-node join again.

Backup and recoveryCopy Anchor Link

Recovering a deployment requires PostgreSQL data, ctrl PKI, node persistent state, and client identities that all match, plus ctrl and node versions that can connect to each other. Do not delete only the ctrl PKI while keeping the other state.

To rebuild a disposable trial environment, stop all processes first, clear all state as described in the deployment documentation, then initialize from empty directories. Do not edit the database migration version by hand, and do not forge identity files.

Service accounts and state directoriesCopy Anchor Link

• The parent directories of the storage directory and the socket directory must allow kibotos-vmm to enter them.
• The node creates and verifies its own directories. Do not bypass the checks with symbolic links or by loosening private key permissions.

Execution ownershipCopy Anchor Link

Only one ctrl on a database executes: it calls nodes, runs commands and background work, and writes to the database. A row in the database guarantees this, independently of connections and database sessions, so it also holds through a connection proxy or pool.

A ctrl that takes over first marks the observations of all nodes as stale. It places work on a node only after the node reconnects and reports its full inventory.

To replace ctrl, stop the old one first, then start the new one, and wait for the nodes to reconnect and report their full inventory.

Version compatibilityCopy Anchor Link

A node connects to ctrl only when their versions are compatible: