Traces

This page shows how to connect trace storage so that one Workspace command appears as a single timeline in Grafana, and how to configure and troubleshoot each service.

OverviewCopy Anchor Link

Traces show one Workspace command as a single timeline in Grafana: from the gateway or kibo into ctrl, through each executor attempt and each node RPC, down to the guest calls, with the duration and failure point of every hop. Tracing is an optional external dependency that the deployer operates. Without it, no service produces spans, and behavior is the same as without tracing.

• Each service produces its own spans and pushes them in batches to trace storage over OTLP/HTTP: ctrl, node (including its data-plane processes), the E2B gateway, and the Exposure gateway. Spans are not written to disk.
• The agent inside the guest does not produce spans. It reads the trace context from the node's calls and writes it into its own log records.
• Each log record carries the trace_id and span_id current at the time, so you can find logs from a trace and traces from a log. See Log fields.

Traces are for troubleshooting only, not usage data. When storage is unreachable, the queue is full, or the credential expires, spans are dropped and never pushed later. kibotos_traces_dropped_total shows how many were dropped.

How a command forms one traceCopy Anchor Link

All spans of one Workspace command share one trace ID derived from the command ID. A command may span several execution attempts, several ctrl processes, and several nodes. This links them together without storing the trace context anywhere.

• The span of the caller's RPC carries a link to the command journey root. In Grafana, you can open the journey directly from References on this span.
• The trace ID of a command journey is the first 16 bytes of the SHA-256 of the command ID. If you know the command ID, you can compute the trace ID directly. Searching by the command_id tag also finds it.
• Before the command settles, the journey root does not exist yet, but attempt spans can still be opened by trace ID.

SamplingCopy Anchor Link

Sampling is decided once at the entry point, and later calls within the same request follow that decision.

• Ratio sampling is decided by trace ID, so entry points with the same ratio reach the same decision for the same trace. A request the gateway samples is also sampled on the node data plane.
• The traceparent in a caller's request supplies only the trace ID and parent span. The sampling flag is not read, so tenants cannot decide how many traces Kibotos records. tracestate and baggage are neither read nor forwarded.
• Every response from the gateways and the node data plane carries a traceresponse header with the value 00-<trace_id>-<span_id>-<flags>. flags set to 01 means this request was sampled. With this value, you can find the request in Grafana.

CredentialsCopy Anchor Link

• Tokens signed by ctrl have the same form as those for instance metrics storage: a JWT signed with ES256, sub is the node ID or ctrl, kibotos_role is node or ctrl, valid for 1 hour, and renewed when less than half the lifetime remains. They carry one more claim, kibotos_sink: traces, and the kid starts with trc-.
• You can share one private key with instance metrics. The traces proxy checks kibotos_role and kibotos_sink, and the instance metrics proxy refuses tokens that carry kibotos_sink, so neither kind of token works on the other store.
• When ctrl has no [traces] section, it tells nodes not to produce spans. Nodes ask again every 5 minutes after that, so after you add [traces] to ctrl and restart it, nodes start pushing within 5 minutes.
• While ctrl is unreachable, the node keeps pushing with the token it holds until the token expires. After that, spans that end are dropped as no_credential.

Reference deployment: VictoriaTraces and vmauthCopy Anchor Link

The single-node VictoriaTraces is reachable only by vmauth and has no password of its own. vmauth provides the push and query endpoints and handles authentication.

Use vmauth v1.152.0 or later. The reference deployment uses the same version as instance metrics storage. Its configuration has three kinds of users:

The sections below deploy the store on a host or with the Helm chart. Both need the key and tokens from the next section.

Signing key and tokensCopy Anchor Link

On the ctrl host, generate a signing private key and export the public key. Only kibotos-ctrl may read the private key. You can also reuse the instance metrics key. Credentials explains what sharing it means:

sudo install -o kibotos-ctrl -g kibotos-ctrl -m 0600 /dev/null /etc/kibotos/ctrl/traces-signing-key.pem
sudo openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out /etc/kibotos/ctrl/traces-signing-key.pem
sudo openssl pkey -in /etc/kibotos/ctrl/traces-signing-key.pem -pubout > traces-signing-public.pem

Then generate one bearer token per gateway and a password for Grafana. The files must not contain newlines, because vmauth compares them as is:

set -eu
umask 077
for name in e2b gw; do openssl rand -hex 32 | tr -d '\n' > traces-$name-token; done
openssl rand -hex 24 | tr -d '\n' > traces-grafana-password

Put traces-e2b-token and traces-gw-token on the two gateway hosts, as the files that their *_TRACES_TOKEN_FILE names. See Gateway configuration.

HostsCopy Anchor Link

Run single-node VictoriaTraces with the following flags:

Run vmauth with the flags of the instance metrics storage, with -auth.config pointing to the file below. Put the content of traces-signing-public.pem into the two JWT users. vmauth reads the gateway tokens and the Grafana password from its own environment variables KIBOTOS_TRACES_WRITER_E2B_TOKEN, KIBOTOS_TRACES_WRITER_GW_TOKEN, and KIBOTOS_TRACES_READER_GRAFANA_PASSWORD:

users:
- name: kibotos-node
  jwt:
    default_vm_access_claim: {}
    public_keys:
    - |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
      -----END PUBLIC KEY-----
    match_claims:
      kibotos_role: node
      kibotos_sink: traces
  url_map:
  - src_paths: ["/insert/opentelemetry/v1/traces"]
    url_prefix: "http://127.0.0.1:10428/"
- name: kibotos-ctrl
  jwt:
    default_vm_access_claim: {}
    public_keys:
    - |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
      -----END PUBLIC KEY-----
    match_claims:
      kibotos_role: ctrl
      kibotos_sink: traces
  url_map:
  - src_paths: ["/insert/opentelemetry/v1/traces"]
    url_prefix: "http://127.0.0.1:10428/"
- name: e2b
  bearer_token: "%{KIBOTOS_TRACES_WRITER_E2B_TOKEN}"
  url_map:
  - src_paths: ["/insert/opentelemetry/v1/traces"]
    url_prefix: "http://127.0.0.1:10428/"
- name: gw
  bearer_token: "%{KIBOTOS_TRACES_WRITER_GW_TOKEN}"
  url_map:
  - src_paths: ["/insert/opentelemetry/v1/traces"]
    url_prefix: "http://127.0.0.1:10428/"
- name: grafana
  username: grafana
  password: "%{KIBOTOS_TRACES_READER_GRAFANA_PASSWORD}"
  url_map:
  - src_paths: ["/select/jaeger/api/.+"]
    url_prefix: "http://127.0.0.1:10428/"

• Both JWT users need default_vm_access_claim: {}, for the same reason as with instance metrics storage. Both claims must match, so a metrics token signed with the same key, which carries no kibotos_sink, is refused.
• Write the public key directly in the configuration file. vmauth reloads only when the configuration file itself changes.

KubernetesCopy Anchor Link

deploy/helm/kibotos-traces/ installs VictoriaTraces (a StatefulSet with a persistent volume, reachable only inside the cluster, with a NetworkPolicy that allows only vmauth), vmauth, and its Service. The structure is the same as the instance metrics storage chart.

Prerequisite: the cluster's CNI enforces NetworkPolicy.

Create the Secrets from the key and token files of Signing key and tokens, then install. The example uses the namespace kibotos-traces:

set -eu
kubectl create namespace kibotos-traces
kubectl -n kibotos-traces create secret generic kibotos-traces-e2b --from-file=token=traces-e2b-token
kubectl -n kibotos-traces create secret generic kibotos-traces-gw --from-file=token=traces-gw-token
kubectl -n kibotos-traces create secret generic kibotos-traces-grafana --from-file=password=traces-grafana-password
cat > traces.yaml <<EOF
vmauth:
  publicKeys:
    - |
$(sed 's/^/      /' traces-signing-public.pem)
  writers:
    - name: e2b
      secretName: kibotos-traces-e2b
    - name: gw
      secretName: kibotos-traces-gw
  readers:
    - name: grafana
      secretName: kibotos-traces-grafana
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet
EOF
helm install kibotos-traces deploy/helm/kibotos-traces -n kibotos-traces -f traces.yaml --wait
kubectl -n kibotos-traces get service kibotos-traces-vmauth

The EXTERNAL-IP in the last command's output is the storage address.

The keys that the three storage charts share are in Common chart values; in this chart, the store's keys start with victoriaTraces.. The keys and defaults specific to this chart:

• With vmauth.tls.secretName set and a certificate from a private CA, give the CA to ctrl's ca_file and the gateways' *_TRACES_CA_FILE.
• vmauth's own /metrics, /flags, pprof, and configuration reload use the key key of the chart-generated Secret <release>-vmauth-admin as authKey, and are not open to the network.
• Pushes fail during the few seconds vmauth restarts after a configuration change, and that batch of spans is dropped.

CapacityCopy Anchor Link

• Each command has about one root span, one span per attempt, and two spans per node RPC (ctrl's client and the node's server), plus the node's calls to the guest.
• Data-plane requests are sampled by ratio, by default 1 in every 100 requests. Estimate the data-plane request volume before you raise the ratio.
• Each process queues at most 2048 spans and pushes every 5 seconds or whenever 512 accumulate.

Configuring ctrlCopy Anchor Link

Only ctrl has a [traces] section. Without this section, ctrl produces no spans, and neither do nodes. Every key, with its range and default, is in [traces] of the ctrl configuration reference. signing_key_file can be the same file as in [metrics]. Keep sample_ratio equal to the gateways' *_TRACES_SAMPLE_RATIO.

Add the following section to the end of /etc/kibotos/ctrl/config.toml, then restart ctrl. Nodes need no configuration change and no restart. For a complete example, see the end of ctrl.example.toml.

[traces]
endpoint = "https://traces.example.com:8427/insert/opentelemetry/v1/traces"
signing_key_file = "/etc/kibotos/ctrl/traces-signing-key.pem"

• init, enrollment create, check-upgrade, and the service itself all validate [traces] and read the private key and CA files. They fail if a file cannot be read or has the wrong format.
• Once enabled, the ctrl log has a line trace export enabled, and each node log has a line node tracing enabled.

Gateway configurationCopy Anchor Link

The variables of the two gateways have the same form, with the prefixes KIBOTOS_E2B_ and KIBOTOS_GW_. For examples, see the end of e2b.example.env and gw.example.env.

• If the address is not set but a token, CA, or ratio is, the gateway refuses to start.
• The token is read only once at startup. Restart the gateway after you change the token.
• Once enabled, the gateway log has a line trace export enabled.

GrafanaCopy Anchor Link

In Grafana, add a data source of type Jaeger. Set the URL to the vmauth address plus /select/jaeger (inside the cluster, http://kibotos-traces-vmauth.<namespace>.svc:8427/select/jaeger), and set Basic auth to a read-only user: grafana in the host configuration, or a user from vmauth.readers on Kubernetes.

deploy/traces/grafana-command-journey.json, the dashboard "Kibotos 命令旅程" (Kibotos command journey):

• The variable "标签过滤" (tag filter) narrows the scope in logfmt form, for example workspace_id=ws-… or command_id=….
• The two tables at the top list recent traces from ctrl and from the E2B gateway. Click a Trace ID to open the whole trace below.
• The link "同一 trace_id 的日志" (logs with the same trace_id) in the top right opens the logs dashboard, filtered by the current Trace ID.

A pushed span takes a few seconds to become searchable.

Log fieldsCopy Anchor Link

Log records written while handling a request or an execution attempt carry trace_id and span_id, both in lowercase hexadecimal. Records without a trace context do not carry these two fields.

• Query trace_id:<ID> in VictoriaLogs to see the records that ctrl, node, and guest wrote for the same request. Guest records are rewritten by the node and keep these two fields.
• Requests that are not sampled still continue the trace ID, and their records carry trace_id too, so logs can still be correlated. Only the trace is missing from storage.
• Without trace storage, logs do not carry these two fields.

What traces never containCopy Anchor Link

Span attributes are limited to the following kinds. Everything else is removed before pushing:

• Request paths, query strings, host names, ports, peer addresses, User-Agent, Workspace labels and annotations, file names, credentials, and request and response bodies never enter spans.
• Span events and error text are not pushed either. Error text may contain user paths, and the result code is already in the attributes.
• Resource attributes are only service.name and service.version. Nodes also have kibotos.node_id.

SecurityCopy Anchor Link

• The ctrl, node, and gateway services must be able to reach vmauth. Tenants must not: neither the guest network nor the public internet should be able to connect to its address.
• Anyone who can push can forge spans and confuse the timeline seen during troubleshooting, but cannot read. Anyone who can read sees object IDs, method names, and durations, but no user data.
• If the signing private key leaks, its holder can push as any node. If it is shared with instance metrics, rotate both together. The rotation steps are the same as rotating the instance metrics private key; change the public key in this chart's vmauth.publicKeys.

TroubleshootingCopy Anchor Link