Logs

This page explains how ctrl, node, the E2B gateway, and the Exposure gateway write logs, how to collect and query them, and where the guest console and VMM logs are kept.

OverviewCopy Anchor Link

Each service writes its log to stderr, one JSON record per line. The packaged systemd units hand stderr to journald, and containers hand it to the container runtime. Kibotos does not collect or export logs itself: the deployer runs a collector on each host that sends the records to its own log storage.

The node fetches the records of the Kibotos agent inside the guest and writes them to its own log. The guest console and VMM logs do not enter the log stream. They stay only on the node host. See Console and VMM logs.

Record formatCopy Anchor Link

Every record has the following four fields:

The remaining fields depend on the event. An example node record:

{"time":"2026-10-06T08:12:03.418Z","level":"WARN","msg":"instance cleanup pending","component":"node","instance_id":"inst-06gfvmpnqsv7xcsfgz7ybx77pg","step":"volume_attachments","attempt":3,"error":"volume is still attached","suppressed":5,"failing_since":"2026-10-06T08:11:02.004Z","node_id":"node-a","incarnation":12}

What the levels mean:

Errors made by the caller (invalid arguments, objects that do not exist) are not logged as WARN or ERROR.

Correlation fieldsCopy Anchor Link

The same object uses the same field name in the records of every service. Query by that field to connect the services that one operation passed through:

Duration fields end in _ms and are in milliseconds. error is the error text, and code is the RPC error code. States, reasons, and phases are lowercase, for example running and unavailable.

To trace a Workspace down to its guest:

1. In the ctrl records, find workspace command settled by workspace_id. It carries instance_id, node_id, and command_id.
2. In the node records, query by instance_id to see what that node did. A phrase query on the ctrl command ID also matches the command IDs derived from it in node records.
3. Records under the same instance_id whose component is guest come from the agent inside the guest.

Repeated failuresCopy Anchor Link

When the same thing fails repeatedly, the log does not get a line for every failure:

• The first failure is logged immediately.
• After that, the same event for the same object is logged at most once per minute, with suppressed (the number of failures omitted in that period) and failing_since (when this round of failures started).
• When a retried operation recovers, an INFO record with the same name is logged, with recovered: true and failed_ms (how long it kept failing).

So one line per minute does not mean only one failure. Check suppressed.

Log levelsCopy Anchor Link

An environment variable sets the level. The default is info, and the values are debug, info, warn, and error. With an invalid value, the service refuses to start.

For ctrl and node, set it in the service unit. The following command opens the override file of the node unit:

sudo systemctl edit kibotos-node.service

Write the following two lines in the file that opens, save it, and restart the node:

[Service]
Environment=KIBOTOS_NODE_LOG_LEVEL=debug

The two gateways set it in their own environment files. See e2b.example.env and gw.example.env.

Viewing on the hostCopy Anchor Link

Because journald keeps recent logs, you can query them directly on the host even when the collector is unavailable. The following command shows the node's WARN and ERROR records from the last hour:

journalctl -u kibotos-node --since -1h -o cat | jq -c 'select(.level == "WARN" or .level == "ERROR")'

The following command finds all records of one Instance on this node, including those of the guest agent:

journalctl -u kibotos-node -o cat | jq -c 'select(.instance_id == "inst-06gfvmpnqsv7xcsfgz7ybx77pg")'

In container deployments, read the same records with kubectl logs or docker logs.

Console and VMM logsCopy Anchor Link

The guest console and the VMM's own logs do not enter the log stream. When an Instance ends, the node first retains them and then cleans up the Instance's runtime directory:

• Location: <storage.root>/machine-logs/<instance_id>/, for example /var/lib/kibotos/node/machine-logs/inst-06gfvmpnqsv7xcsfgz7ybx77pg/.
• They are retained however the Instance ends: a requested stop, a forced stop, the VM exiting on its own, or the VMM failing to start. When the same Instance ends again, its files replace the previous ones.
• Only the last 1 MiB of each file is kept. When the retained files exceed 1 GiB in total or 1000 Instances, the oldest Instance directories are deleted.
• The node log has a single instance machine exited record, with the exit code or signal, whether the process was killed by OOM, the uptime, and retained (the path of the retained directory). If systemd has already reclaimed the unit after a requested stop and the exit status cannot be found, exit_status: unobserved replaces the exit code. When the guest resets itself, the node restarts the VMM, and on success writes another record instance machine restarted after guest reset with restart_count and the new guest_boot_id.

Notes:

• The console may contain output from programs inside the Workspace. Only root can read the retained files. Do not hand them to a log collector.
• These files can be read only on the host. Neither ctrl nor the gateways have an interface to read them.

What logs never containCopy Anchor Link

The following never appears in a log at any level:

• Process arguments, environment variables, input and output, file contents, paths and file names given by users, and clipboard and screen contents.
• Tokens, private keys, passwords, enrollment keys, API keys, and session credentials. Where a record needs to tell them apart, it logs only the key ID, the certificate serial number, or the first 8 hexadecimal characters of a digest.
• The random part of an Exposure host name. That part is the credential for accessing the Exposure. Logs record exposure_id, and before the Exposure is resolved, the zone and a prefix of the label digest.
• Full URLs with query strings. The gateways log the route pattern, for example GET /sandboxes/{id}.

Reference deployment: Vector, VictoriaLogs, and GrafanaCopy Anchor Link

The reference deployment collects with Vector on each host, writes to single-node VictoriaLogs, and queries with Grafana. Any collector and storage that can read journald or container output and parse it as JSON can replace them.

VictoriaLogsCopy Anchor Link

VictoriaLogs flags:

VectorCopy Anchor Link

Each host runs one Vector that reads the journald records of the Kibotos services and parses them into fields. The reference configuration is deploy/logging/vector.toml. Change endpoints and it is ready to use.

• Starting with Vector 0.58, environment variables in the configuration are not expanded by default. Put the password in a file that only Vector can read and reference it through Vector's secret file backend, as the reference configuration does.
• Each generation of the node data-plane process is a unit named after its generation number, and all of them are in kibotos-dataplane.slice. The reference configuration selects them by slice and records unit as kibotos-dataplane for all of them. The generation number is in the record's data_plane_generation field.
• The egress process also has one unit per generation, all in kibotos-egress.slice. unit is recorded as kibotos-egress for all of them, and the generation number is in the egress_generation field.
• The stream fields are host, component, and unit. Fields such as instance_id and command_id have very many values, so they are written as ordinary fields. VictoriaLogs can still query by them.
• For container deployments, replace the journald source with Vector's kubernetes_logs or docker_logs source. The parsing part stays the same.

GrafanaCopy Anchor Link

Install the VictoriaLogs data source plugin victoriametrics-logs-datasource in Grafana, and enter the VictoriaLogs address and password when you add the data source. deploy/logging/grafana-dashboard.json is a dashboard that counts records by component and level and lists the most frequent WARN and ERROR records. You can import it directly.

The dashboard's variable filter takes a LogsQL filter that applies to every panel, * by default. The link from the traces dashboard opens this dashboard by its uid kibotos-logs with filter set to trace_id: plus the Trace ID, so keep that uid when you import it.

Common queries (LogsQL):

NotesCopy Anchor Link

• The collectors and storage must be reachable from the Kibotos hosts, and tenants must not reach them: the logs contain object IDs and error messages.
• Log volume grows with the number of Workspace operations. Observe one day of writes on one node first, then estimate disk space from the retention period.