Instance metrics storage

This page shows how to connect instance metrics storage so that the E2B SDK can read the CPU, memory, and disk readings of a sandbox, and the native API and kibo workspace metrics can read a Workspace's.

OverviewCopy Anchor Link

Calls such as sandbox.getMetrics() and GetWorkspaceMetrics read these readings. Instance metrics storage is an optional external dependency that the deployer runs. Without it, every other feature works as usual: only the two sandbox metrics endpoints return 501, and GetWorkspaceMetrics returns failed_precondition.

The components divide the work as follows:

• Each node samples every running Instance every 5 seconds and writes the readings with the labels instance_id and node_id. Seven readings come from the guest. The node takes the others on the host, from the VMM's cgroup, the Instance's TAP device, and on Firecracker the VMM's block devices, so they keep being written while the guest does not respond.
• The write address and credentials come from ctrl, so the node has no metrics configuration of its own.
• ctrl writes the instance info series kibotos_instance_info, which maps an Instance to its Workspace and carries the Workspace labels that the deployment selects.
• The E2B gateway queries the guest readings and the instance info, merges them by sandbox, and returns the result. For the endpoint semantics, see Sandbox metrics.
• ctrl queries them the same way for GetWorkspaceMetrics, with the token it writes with, once query_url is set, and also returns the host readings. For the method's semantics, see Reading resource metrics.

Series lists every series written to the storage and who reads it.

The metrics are for observation only. They are not usage or billing data. When the storage is unreachable, the guest does not respond in time, the node restarts, or the write credential expires, the samples for that period are lost and are not written later.

This is separate from the metrics of ctrl, node, and the gateways themselves. Those are scraped by a collector from each service's metrics listener, and the reference deployment uses a separate store for them. See Cluster and service metrics.

Storage requirementsCopy Anchor Link

The reference deployment is single-node VictoriaMetrics with its auth proxy vmauth in front. Any storage that implements both protocols above and can have authentication done in front of it works.

Capacity estimationCopy Anchor Link

• Each running Instance has 21 series, each with one sample every 5 seconds: 7 guest series, host memory, uptime, and the 12 host-side counters. An Instance on the Firecracker backend also has the 4 disk counters, 25 series in all. In addition, ctrl writes one instance info sample for every Instance every minute.
• 1000 Instances running at the same time produce about 4200 samples per second, or about 5000 when they all run on Firecracker. Keeping them for 7 days on VictoriaMetrics takes disk space on the order of 10 GB.
• Every Instance start, including a sandbox resume, creates a new set of series. When Instances start often, estimate the total series count as the number of starts within the retention period times 22 (26 on Firecracker).
• Each node fetches a new token from ctrl every half hour.

Write credentialsCopy Anchor Link

ctrl issues write credentials with a private key that the deployer generates, and nodes store no metrics secrets.

• The token is a JWT signed with ES256. sub is the node ID, or ctrl for ctrl itself. kibotos_role is node or ctrl.
• When ctrl has no [metrics] section, it tells the nodes not to sample. Each node then asks again every 5 minutes, so after you add [metrics] to ctrl and restart it, the nodes start sampling within 5 minutes.
• After a node is revoked, ctrl no longer issues tokens to it. Its writes stop when its current token expires, within one hour at most.
• While ctrl is unreachable, a node writes with the token it holds until the token expires. After that, the samples have a gap, and writes resume when ctrl recovers.

Generate the signing key on the ctrl host and export the public key. The public key is not a secret; it goes into the vmauth configuration:

sudo install -o kibotos-ctrl -g kibotos-ctrl -m 0600 /dev/null /etc/kibotos/ctrl/metrics-signing-key.pem
sudo openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out /etc/kibotos/ctrl/metrics-signing-key.pem
sudo openssl pkey -in /etc/kibotos/ctrl/metrics-signing-key.pem -pubout

• Only kibotos-ctrl may read the private key. Anyone who holds it can write as any node, and can sign a ctrl token that reads the readings of every Workspace.
• Rotate the private key only when it leaks. For the steps, see Rotating the signing key.

SecurityCopy Anchor Link

• The node, ctrl, and gateway services must be able to reach vmauth. Tenants must not: neither the guest network nor the public internet should be able to connect to its address. Anyone who can write can forge readings for any sandbox, and anyone who can query can see the readings of all sandboxes.
• A compromised node can still write until its current token expires. It can corrupt the readings of sandboxes whose IDs it knows, but it cannot query.
• The ctrl token can read the readings of every Workspace. It exists only in the memory of the ctrl process.
• The only Workspace labels that ctrl writes to the storage are those listed in instance_info_labels. For what an E2B deployment lists, see Configuring ctrl.

Reference deployment: VictoriaMetrics and vmauthCopy Anchor Link

VictoriaMetrics listens only on an address that vmauth on the same machine can reach and has no password of its own. vmauth provides the write and query endpoints to the outside and handles authentication:

VictoriaMetrics flags:

Use vmauth v1.152.0 or later. That version fixes an authorization bypass when routes are selected by JWT claim. Flags:

Put the public key you exported in Write credentials into the three JWT users of the vmauth configuration. The gateway's password is read from the environment variable KIBOTOS_E2B_METRICS_PASSWORD of the vmauth process:

users:
- name: kibotos-other-sink
  jwt:
    default_vm_access_claim: {}
    public_keys:
    - |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
      -----END PUBLIC KEY-----
    match_claims:
      kibotos_role: ".+"
      kibotos_sink: ".+"
  url_map:
  - src_paths: [".*"]
    deny_paths: [".*"]
    url_prefix: "http://127.0.0.1:1"
- name: kibotos-node
  jwt:
    default_vm_access_claim: {}
    public_keys:
    - |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
      -----END PUBLIC KEY-----
    match_claims:
      kibotos_role: node
  url_map:
  - src_paths: ["/api/v1/write"]
    url_prefix: "http://127.0.0.1:8428/"
- name: kibotos-ctrl
  jwt:
    default_vm_access_claim: {}
    public_keys:
    - |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
      -----END PUBLIC KEY-----
    match_claims:
      kibotos_role: ctrl
  url_map:
  - src_paths: ["/api/v1/write", "/api/v1/query", "/api/v1/query_range"]
    url_prefix: "http://127.0.0.1:8428/"
- name: kibotos-e2b
  username: kibotos-e2b
  password: "%{KIBOTOS_E2B_METRICS_PASSWORD}"
  url_map:
  - src_paths: ["/api/v1/query", "/api/v1/query_range"]
    url_prefix: "http://127.0.0.1:8428/"

• The node token can only write. The ctrl token can write and query, for GetWorkspaceMetrics. The gateway's password can only query. All other paths are rejected.
• kibotos-other-sink refuses tokens that carry a kibotos_sink claim, which are the trace tokens ctrl signs, with 403. Metrics tokens carry no such claim, and match_claims cannot ask for a claim to be absent, so this user does the refusing: vmauth tries the users with more match_claims entries first, and this one has one more than the other two JWT users. When tracing shares a private key with instance metrics, the two stores are therefore separated both ways.
• All three JWT users need default_vm_access_claim: {}. vmauth only accepts tokens that carry a vm_access claim, or users configured with this setting. Tokens signed by ctrl carry no vm_access, so without this setting every write returns 401.
• Write the public key directly in the configuration file, not in a separate file. vmauth reloads only when the configuration file itself changes.
• While -search.latencyOffset keeps its default of 30s, the gateway's default query delay works. If you raise it, raise the gateway's KIBOTOS_E2B_METRICS_QUERY_DELAY as well. See E2B gateway configuration.

Reference deployment on KubernetesCopy Anchor Link

deploy/helm/kibotos-instance-metrics/ is a Helm chart for the same reference deployment. It installs the following into a Kubernetes cluster:

• Single-node VictoriaMetrics, running in a StatefulSet with its data on a persistent volume. It is reachable only inside the cluster, and a NetworkPolicy admits only the vmauth Pods.
• vmauth, configured with the auth.yml above. The public keys come from values. The gateway password is injected through an environment variable from a Secret that the deployer creates, so the chart never renders it. vmauth's own /metrics, /flags, pprof, and config reload use the key key of the chart-generated Secret <release>-vmauth-admin as authKey, and are not open to the network.
• A Service for vmauth. The ctrl, node, and gateway services run outside the cluster and reach vmauth through this address. All three must be able to reach it, and guests and the public internet must not. Use an internal load balancer or a NodePort, and do not give it a public address.

Prerequisites: you generated the signing key as described in Write credentials, and the cluster's CNI enforces NetworkPolicy. Otherwise any Pod in the same cluster can write to VictoriaMetrics directly.

Save the exported public key as metrics-signing-public.pem, generate the gateway password, create the namespace and the password Secret, and then install. The password file must not contain a newline: vmauth uses the Secret content verbatim as the password, so an extra newline makes the gateway's password mismatch. The example uses the namespace kibotos-instance-metrics and the internal load balancer annotation of Alibaba Cloud ACK. On other clusters, use the matching annotation, or use a NodePort instead:

set -eu
umask 077
openssl rand -hex 24 | tr -d '\n' > metrics-query-password
kubectl create namespace kibotos-instance-metrics
kubectl -n kibotos-instance-metrics create secret generic kibotos-instance-metrics-query --from-file=password=metrics-query-password
cat > instance-metrics.yaml <<EOF
vmauth:
  publicKeys:
    - |
$(sed 's/^/      /' metrics-signing-public.pem)
  query:
    secretName: kibotos-instance-metrics-query
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet
EOF
helm install kibotos-instance-metrics deploy/helm/kibotos-instance-metrics -n kibotos-instance-metrics -f instance-metrics.yaml --wait
kubectl -n kibotos-instance-metrics get service kibotos-instance-metrics-vmauth

The EXTERNAL-IP in the output of the last command is the storage address. Set ctrl's remote_write_url to http:// plus this address plus :8427/api/v1/write, set the gateway's KIBOTOS_E2B_METRICS_URL to http:// plus this address plus :8427, and set KIBOTOS_E2B_METRICS_PASSWORD to the content of metrics-query-password. If you give the address a DNS name, use that name in both ctrl and the gateway.

The keys specific to this chart and its defaults are below. The keys it shares with the other two storage charts are in Common chart values.

• With vmauth.tls.secretName set and a certificate from a private CA, give the CA to ctrl's ca_file and the gateway's KIBOTOS_E2B_METRICS_CA_FILE.
• While vmauth restarts after a configuration change, writes fail for a few seconds, and the samples for that time have a gap. After that, node and ctrl continue writing.
• Steps 2 and 4 of Rotating the signing key here mean changing vmauth.publicKeys and running helm upgrade.

Common chart valuesCopy Anchor Link

The three storage charts share the keys below: this chart, kibotos-metrics for cluster and service metrics, and kibotos-traces for traces. In kibotos-traces, the store's keys start with victoriaTraces. instead of victoriaMetrics.. Each chart's values.yaml describes every key in comments.

• Images are pinned by digest. vmauth is at the same version as vmagent in Cluster and service metrics.
• After changing values, run helm upgrade. A change to the vmauth configuration restarts vmauth.
• helm uninstall does not delete the persistent volume. Delete the PVC once you are sure you no longer need the data.

Configuring ctrlCopy Anchor Link

Only ctrl has a [metrics] section. Without this section, ctrl does not write the instance info series, and nodes do not sample. Every key, with its range and default, is in [metrics] of the ctrl configuration reference.

Add the following to the end of /etc/kibotos/ctrl/config.toml:

[metrics]
remote_write_url = "https://metrics.example.com:8427/api/v1/write"
signing_key_file = "/etc/kibotos/ctrl/metrics-signing-key.pem"
instance_info_labels = ["e2b.kibotos.app/namespace", "e2b.kibotos.io/namespace"]
query_url = "https://metrics.example.com:8427"

Then restart ctrl. The nodes need no configuration change and no restart.

• All files use absolute paths. They can be symbolic links, but the final target must be a regular file that kibotos-ctrl can read.
• Keep retention_seconds equal to the storage's retention and query_delay_seconds no less than its visibility delay, the same rules as for the gateway in E2B gateway configuration. The defaults match the reference deployment.
• An E2B deployment sets instance_info_labels to exactly the two namespace keys: e2b.kibotos.app/namespace, and e2b.kibotos.io/namespace, which gateways before v0.3.0 write and which is removed in v0.4.0. If the first is missing, the gateway finds no Instance for any sandbox, and the metrics of every sandbox are an empty array.
• Do not list the labels generated from E2B sandbox metadata (prefix m.e2b.kibotos.app/). They are the caller's data.
• init, enrollment create, and check-upgrade of kibotos-ctrl, as well as the service itself, all validate [metrics] and read the private key and CA files, as they do with dsn_file. If a file cannot be read or has the wrong format, all of these commands fail. migrate reads only the database connection and does not validate [metrics].
• Once enabled, the ctrl log has a line instance info series enabled, and the log of each node has a line guest metrics sampling enabled. With query_url, the ctrl log also has a line Workspace metrics reads enabled.
• If the guest of a Platform predates the sampling interface, the node does not sample that Instance and logs a line guest does not support resource sampling.

E2B gateway configurationCopy Anchor Link

Set the variables below in /etc/kibotos/e2b/config.env, then restart the gateway. For an example, see the end of e2b.example.env.

• If credentials or a CA are set without an address, the gateway refuses to start.
• The query delay must not be less than the storage's visibility delay, that is, how long a newly written sample takes to appear in query results. The visibility delay of VictoriaMetrics is -search.latencyOffset, 30s by default, the same as the gateway's default. If you raise it, raise the query delay as well. The gateway computes the cutoff with its local clock. If that clock differs from the clock of the host running the storage, leave extra margin.
• When the query delay is less than the visibility delay, the most recent buckets may hold values that the storage filled in: a recently paused sandbox gets extra trailing buckets without samples that disappear later, and the batch endpoint may not find the latest sample.
• Set the query delay to 0 only for storage where samples are queryable immediately after they are written.
• All gateway instances of the same deployment use the same metrics configuration.
• Once enabled, sandbox_metrics is true in the startup log line starting kibotos-e2b gateway.

VerificationCopy Anchor Link

Create a sandbox with the E2B SDK. After it has run for a minute, sandbox.getMetrics() should return a non-empty array. Samples are returned only after the query delay (30 seconds by default) has passed.

You can also query directly with the gateway's password to confirm that both kinds of series are being written. curl prompts for the password:

curl -sS -u kibotos-e2b --get \
  --data-urlencode 'query=count by (__name__) ({__name__=~"kibotos_instance_info|kibotos_guest_cpu_count"})' \
  https://metrics.example.com:8427/api/v1/query

Both kibotos_instance_info and kibotos_guest_cpu_count should appear in the result.

With query_url set, kibo workspace metrics <workspace-id> should show host and guest readings for a Workspace that has run for a minute. If it returns unavailable and the ctrl log has workspace metrics unavailable with an HTTP 400 or 401 from vmauth, the ctrl user of the vmauth configuration lacks the two query paths.

Rotating the signing keyCopy Anchor Link

Rotate the signing key only when it leaks. If you follow this order, writes are not interrupted:

1. Generate a new private key and export the new public key.
2. Add the new public key to public_keys of all three JWT users in the vmauth configuration, and keep the old one. Wait for vmauth to reload the configuration. On Kubernetes, append the new public key to vmauth.publicKeys in values, then run helm upgrade.
3. Replace the file that signing_key_file points to with the new private key, and restart ctrl.
4. After one hour, all tokens signed with the old private key have expired. Remove the old public key from the vmauth configuration.

DashboardsCopy Anchor Link

Instance metrics have two Grafana dashboards. When you import either one, select a Prometheus data source that points to the instance metrics storage. Have the data source use a read-only vmauth user from vmauth.readers in Common chart values, not the gateway's password. Inside the cluster, its URL is http://kibotos-instance-metrics-vmauth.kibotos-instance-metrics.svc:8427 with the namespace from the example, and the user's name and password go under Basic auth. Choose the Prometheus type, not the VictoriaMetrics plugin: the dashboards use only standard PromQL, and the data source variable lists only Prometheus data sources.

• The overview's node and 命名空间 (namespace) variables narrow the scope. A Workspace in a table links to "Kibotos Workspace", carrying the namespace and time range.
• The namespace comes from label_e2b_kibotos_app_namespace on the instance info. If instance_info_labels does not list e2b.kibotos.app/namespace, the per-namespace panels are empty. Samples written before v0.3.0 carry label_e2b_kibotos_io_namespace, and the per-namespace panels do not show them.
• kibotos_instance_host_memory_bytes and kibotos_instance_uptime_seconds are written from the v0.2.0 node onward, so these panels have no data for earlier nodes.
• The dashboards do not show the host-side counters and disk counters yet. The host-side CPU, pressure, and network counters are written from the v0.3.0 node onward.
• When everything is healthy, the lists in "Sampling health" are empty and the counts are 0. A freshly started Instance has only instance info and no guest samples until its second sampling, which is normal.

SeriesCopy Anchor Link

The node writes every series below with the labels instance_id and node_id, one sample every 5 seconds per running Instance. ctrl writes kibotos_instance_info.

Host-side countersCopy Anchor Link

• Times are in seconds. Read the counters with rate() or increase(): the total over a range stays correct when samples are missing.
• Every VMM start gets a new cgroup, so the CPU and pressure counters start again from 0. Queries treat this as a counter reset.
• When the host kernel has PSI turned off (psi=0), the three kinds of pressure counters are not written. The node logs this once, and the other series are written as usual.
• Network directions are the guest's: what the TAP device sends is what the guest receives. The TAP device is the host end of the guest's network interface, so traffic forwarded through an Egress and DNS queries answered through a Resolver also pass it. exec, file transfer, and port forwarding use vsock and are not counted.

Disk countersCopy Anchor Link

• They count what the guest asked for, whether or not the host page cache served it and whether the disks are in a local or an NFS storage pool. The read-only Template base layer, the writable layer, and the Volume are added together.
• Data plane transfers are not counted themselves; an uploaded file counts as a write when it is written to a disk.
• Read them with rate() or increase(). They start from 0 for a new VMM and when a restarted node takes a running VMM over. Queries treat this as a counter reset.
• Only the Firecracker backend writes these 4 counters for now. Before the VMM boots, the node points Firecracker's metrics at a named pipe in the jail, and on every sampling round asks it to flush and reads the pipe. The KiboVM backend does not write them yet.
• A Firecracker VMM started before the upgrade has no metrics configured and writes none until it restarts.
• The host-side cgroup io.stat and /proc/<pid>/io measure the host's storage layer: block files on NFS are not counted in io.stat, reads served from the cache are not counted, and a block written again before writeback counts once. Read those to see how hard the storage is driven, and these 4 counters to see how much IO the Instance did.

TroubleshootingCopy Anchor Link