Cluster and service metrics

This page explains how ctrl, node, the E2B gateway, and the Exposure gateway export Prometheus metrics, and how to collect, store, and view them in Grafana.

OverviewCopy Anchor Link

Each service can open a metrics listener that serves GET /metrics over plain HTTP in the Prometheus text format. Kibotos does not collect or store these metrics itself: the deployer runs a collector that scrapes them periodically and writes them to its own time series storage.

Besides its own process metrics, ctrl reads the state of the whole cluster when it is scraped: node status and resources, storage pools, Workspace and Volume counts, and in-flight commands.

These metrics are separate from instance metrics storage:

• Instance metrics storage holds the CPU, memory, and disk readings of each Instance. Nodes and ctrl write them, and the E2B SDK queries them.
• The metrics on this page describe the service processes and the cluster itself, and a collector scrapes them.

Keep them in separate stores. Do not write these metrics to the instance metrics storage, and do not let the E2B gateway query their store: the two differ in retention, readers, and access permissions.

The collector and the store must be reachable from the Kibotos hosts. Tenants must not reach them: the metrics contain node IDs, cluster size, and error rates.

These metrics may be lost and are not a usage record. Do not base billing, quotas, or any state decision on them.

ListenerCopy Anchor Link

Each reference port is the service's main port plus 100.

• Without this configuration, no listener is opened, and an upgrade does not add a port.
• The listener serves only /metrics, without authentication. Other paths return 404, and methods other than GET and HEAD return 405.
• On a host, bind to the loopback address so that the collector on the same host scrapes it. In a container, bind to 0.0.0.0 and use a NetworkPolicy so that only the collector can connect to this port. See Kubernetes.
• If the address is not in host:port form, configuration validation fails and the service does not start. If the port is already in use, the service fails to start and the log names the address. It does not skip the metrics listener and keep running.

Enabling the listenerCopy Anchor Link

For ctrl and node, add the following section to the configuration file, with 18192 for ctrl and 18191 for node. The example below is for node:

[telemetry]
listen = "127.0.0.1:18191"

For the two gateways, add a line to the environment file. For the Exposure gateway:

echo 'KIBOTOS_GW_TELEMETRY_LISTEN=127.0.0.1:18197' | sudo tee -a /etc/kibotos/gw/config.env >/dev/null

Then restart the service. The following commands restart the node and read its metrics once:

sudo systemctl restart kibotos-node.service
curl -s http://127.0.0.1:18191/metrics | grep '^kibotos_build_info'

Once the listener is open, the ctrl and node logs have a line metrics enabled, and the startup log lines of the two gateways, starting kibotos-e2b gateway and starting kibotos-gw, carry telemetry_address.

Exported metricsCopy Anchor Link

The table below summarizes the metrics by prefix. Every metric with its type, labels, and description is listed in the metrics reference, generated from the source code.

• The only labels are node IDs, storage pool IDs, Connect method names and route patterns, error codes and HTTP status codes, the API method name that a command belongs to, the gateways' upstream operation names, and a few fixed values specific to each metric. IDs of objects such as Workspaces, Instances, and Volumes, Workspace labels, paths, addresses, credentials, and error messages never become labels, so the series count does not grow with the number of Workspaces.
• Metrics that count over a fixed set output every value in the set. Values that never occurred are 0.
• Cluster state (metrics starting with kibotos_cluster_) is read from the database when ctrl is scraped. Scrapes within 15 seconds share one read, and a read takes at most 5 seconds. After a read fails or times out, scrapes within the next 15 seconds do not output these metrics, and kibotos_ctrl_cluster_read_failures_total increases by one per read. All other metrics are output as usual.
• The node outputs the metrics that carry its identity only after it joins the cluster and obtains an identity. kibotos_node_info has value 1, and its label node_id is the node ID. Use it to map node metrics scraped on a host to the node ID. See Common queries.

Common queriesCopy Anchor Link

Reference deployment: vmagent, VictoriaMetrics, and GrafanaCopy Anchor Link

The reference deployment scrapes with vmagent, writes to a single-node VictoriaMetrics that stores only these metrics, and views them in Grafana. vmagent was chosen because it records up for each scrape target, so you can tell when a process cannot be scraped, and it supports discovering targets by Kubernetes Pod. Any collector that scrapes the Prometheus text format and any storage that offers the Prometheus query API can replace them.

The sections below set up the store and vmagent on hosts, on Kubernetes, and with Docker Compose. Grafana applies to all of them.

CapacityCopy Anchor Link

Estimate capacity from the series count. The table below shows series counts measured on a KiboVM deployment after running through the lifecycle operations:

With 2000 nodes per cluster and a scrape every 30 seconds, that is about 1.1 million active series and about 37,000 samples per second, almost all from nodes. A scrape takes less than 100 ms. The longest is ctrl, because it reads the cluster state.

HostsCopy Anchor Link

On hosts, run VictoriaMetrics on one internal machine and one vmagent on every host that runs Kibotos services.

VictoriaMetrics on a hostCopy Anchor Link

This VictoriaMetrics stores only the metrics on this page and is deployed separately from the instance metrics storage. Flags:

vmagent on a hostCopy Anchor Link

Each host that runs Kibotos services runs one vmagent that scrapes the local metrics listeners:

• vmagent.yml is the scrape configuration, with one job per service: kibotos-ctrl, kibotos-node, kibotos-e2b, kibotos-gw. Delete the services that this host does not run: a target that cannot be reached keeps up at 0, and vmagent logs a warning on every scrape.
• kibotos-vmagent.service is the systemd unit. It adds a host label with the host name to every series: every host scrapes at 127.0.0.1, so without this label you cannot tell which host a series came from.
• The password is in a file that only root can read. The unit passes it to vmagent with LoadCredential, so the password does not appear on the command line or in environment variables.

The following commands download vmagent, verify the checksum, and install it to /usr/local/bin/vmagent:

set -eu
version=v1.154.0
base="https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/$version"
curl -fsSLO "$base/vmutils-linux-amd64-$version.tar.gz"
curl -fsSLO "$base/vmutils-linux-amd64-${version}_checksums.txt"
sha256sum --check --ignore-missing "vmutils-linux-amd64-${version}_checksums.txt"
tar xzf "vmutils-linux-amd64-$version.tar.gz" vmagent-prod
sudo install -m 0755 vmagent-prod /usr/local/bin/vmagent

Run the following commands in the root of the source repository to install the scrape configuration, the password file, and the service unit. The two sudoedit commands let you write the password and change -remoteWrite.url in the unit to the VictoriaMetrics address. Also delete the jobs you do not need from the scrape configuration in /etc/vmagent/kibotos.yml:

set -eu
sudo install -d -m 0755 /etc/vmagent
sudo install -m 0644 deploy/metrics/vmagent.yml /etc/vmagent/kibotos.yml
sudo install -m 0600 /dev/null /etc/vmagent/remote-write-password
sudoedit /etc/vmagent/remote-write-password
sudo install -m 0644 deploy/metrics/kibotos-vmagent.service /etc/systemd/system/kibotos-vmagent.service
sudoedit /etc/systemd/system/kibotos-vmagent.service
sudo systemctl daemon-reload
sudo systemctl enable --now kibotos-vmagent.service

The vmagent status page listens only on the loopback address. The following command lists every target. state=up means the most recent scrape succeeded:

curl -s http://127.0.0.1:8429/targets

KubernetesCopy Anchor Link

On Kubernetes, a Helm chart installs the store, a manifest runs vmagent in the kibotos namespace, and the chart can also hand the data source and dashboards to Grafana's sidecar. vmagents on hosts outside the cluster can write to the same store.

VictoriaMetrics in the clusterCopy Anchor Link

deploy/helm/kibotos-metrics/ installs this VictoriaMetrics into a Kubernetes cluster. Its structure is the same as the instance metrics storage chart:

• Single-node VictoriaMetrics, running in a StatefulSet with its data on a persistent volume. It is reachable only inside the cluster, and a NetworkPolicy admits only the vmauth Pods.
• vmauth in front for authentication. Write users can call only /api/v1/write, for vmagent. Read-only users can call only the Prometheus query API, for Grafana. All passwords are in Secrets that the deployer creates and reach vmauth through environment variables, so the chart never renders them.
• A Service for vmauth. The vmagents on the hosts write through this address from outside the cluster. The Kibotos hosts must be able to reach it, and guests and the public internet must not. Use an internal load balancer or a NodePort, and do not give it a public address.
• Optionally, a data source and dashboards for the Grafana sidecar to load. See Grafana sidecar.

Prerequisite: the cluster's CNI enforces NetworkPolicy. Otherwise any Pod in the same cluster can read and write VictoriaMetrics directly.

The following commands generate the write password and Grafana's read-only password, create the namespace and the Secrets, and then install. The password files must not contain a newline: vmauth compares the Secret content verbatim as the password. The example uses the namespace kibotos-metrics and the internal load balancer annotation of Alibaba Cloud ACK. On other clusters, use the matching annotation, or use a NodePort instead:

set -eu
umask 077
openssl rand -hex 24 | tr -d '\n' > remote-write-password
openssl rand -hex 24 | tr -d '\n' > metrics-grafana-password
kubectl create namespace kibotos-metrics
kubectl -n kibotos-metrics create secret generic kibotos-metrics-writer --from-file=password=remote-write-password
kubectl -n kibotos-metrics create secret generic kibotos-metrics-grafana --from-file=password=metrics-grafana-password
cat > metrics.yaml <<EOF
vmauth:
  writers:
    - name: kibotos
      secretName: kibotos-metrics-writer
  readers:
    - name: grafana
      secretName: kibotos-metrics-grafana
  service:
    type: LoadBalancer
    annotations:
      service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: intranet
EOF
helm install kibotos-metrics deploy/helm/kibotos-metrics -n kibotos-metrics -f metrics.yaml --wait
kubectl -n kibotos-metrics get service kibotos-metrics-vmauth

The EXTERNAL-IP in the output of the last command is the storage address. The vmagent on each host sets -remoteWrite.url to http:// plus this address plus :8427/api/v1/write, with the username kibotos and the content of remote-write-password as the password. See vmagent on a host. A vmagent in the same cluster uses the in-cluster address http://kibotos-metrics-vmauth.kibotos-metrics.svc:8427/api/v1/write. See vmagent in the cluster.

The keys that the three Kibotos storage charts share, such as vmauth.readers, vmauth.tls.secretName, vmauth.service.*, and the storage size, are described in Common chart values. The keys and defaults specific to this chart:

• vmauth's own /metrics, /flags, pprof, and config reload use the key key of the chart-generated Secret kibotos-metrics-vmauth-admin as authKey, and are not open to the network.
• While vmauth restarts after a configuration change, writes fail for a few seconds. vmagent buffers the samples locally and writes them later.

vmagent in the clusterCopy Anchor Link

• kubernetes-vmagent.yaml runs one vmagent in the kibotos namespace. It discovers targets by Pod and scrapes only container ports named telemetry. job comes from the Pod's app.kubernetes.io/name label, matching the job names on hosts. host is the node the Pod runs on, and pod is the Pod name. Its ServiceAccount can read only the Pods in this namespace.

• The example manifests ctrl.yaml, e2b.yaml, and gw.yaml all declare a port named telemetry. The ConfigMaps of the two gateways already set the listeners to 0.0.0.0:18195 and 0.0.0.0:18197. To open the ctrl listener, add the following section to ctrl's config.toml. Without it, the ctrl target keeps up at 0:

  Copy Text

  [telemetry]
  listen = "0.0.0.0:18192"

• Each of the three manifests includes a NetworkPolicy: the service ports stay open as usual, and only the vmagent Pod may connect to the telemetry port. The cluster must enforce NetworkPolicy. Otherwise any Pod in the same cluster can read these metrics.

• The Services do not expose the telemetry port. vmagent connects to the Pods directly.

The -remoteWrite.url in the manifest is the in-cluster address after installing VictoriaMetrics in the cluster as in the example, with the username kibotos. If the storage is elsewhere, change it to that address. Put the write password in a Secret, then apply:

set -eu
kubectl -n kibotos create secret generic kibotos-vmagent --from-file=password=./remote-write-password
kubectl apply -f deploy/metrics/kubernetes-vmagent.yaml
kubectl -n kibotos rollout status deploy/kibotos-vmagent --timeout=5m

The vmagent status page listens only inside the Pod. The following command forwards it to your machine. Then open http://127.0.0.1:8429/targets to see the scrape status:

kubectl -n kibotos port-forward deploy/kibotos-vmagent 8429:8429

Grafana sidecarCopy Anchor Link

When Grafana runs on Kubernetes with its sidecars enabled, the kibotos-metrics chart can render the data source and dashboards, so you do not have to add and import them by hand:

• grafana.datasource.enabled: renders a ConfigMap labeled grafana_datasource: "1" that contains the provisioning file of a Prometheus data source named Kibotos 指标 with uid kibotos-metrics. It queries through vmauth as the read-only user named by grafana.datasource.reader. The password is not written into the ConfigMap. The file contains $__env{KIBOTOS_METRICS_GRAFANA_PASSWORD}, which Grafana reads from its own environment variables.
• grafana.dashboards.enabled: renders one ConfigMap labeled grafana_dashboard: "1" for each of the two dashboards. The dashboards' data source variable preselects the data source whose uid is grafana.datasource.uid, so when the same Grafana also has other Prometheus data sources, such as the instance metrics storage, the dashboards query this store by default. If you add the data source by hand, set its uid to the same value.
• grafana.namespace: the namespace of the ConfigMaps, by default the chart's namespace. Grafana's sidecar must search this namespace. You can change the labels and the dashboard annotations with grafana.datasource.labels, grafana.dashboards.labels, and grafana.dashboards.annotations.

The following example uses the grafana chart from grafana-community, with Grafana installed in the namespace monitoring. First, put the read-only password in a Secret in Grafana's namespace:

kubectl -n monitoring create secret generic kibotos-metrics-grafana --from-file=password=metrics-grafana-password

In Grafana's values, turn on both sidecars, pass the password to Grafana as an environment variable, and create folders from the annotation grafana_folder:

envValueFrom:
  KIBOTOS_METRICS_GRAFANA_PASSWORD:
    secretKeyRef:
      name: kibotos-metrics-grafana
      key: password
sidecar:
  datasources:
    enabled: true
  dashboards:
    enabled: true
    folderAnnotation: grafana_folder
    provider:
      foldersFromFilesStructure: true

Add the following to the kibotos-metrics values in the metrics.yaml from earlier:

grafana:
  namespace: monitoring
  datasource:
    enabled: true
  dashboards:
    enabled: true
    annotations:
      grafana_folder: Kibotos

Then run helm upgrade kibotos-metrics deploy/helm/kibotos-metrics -n kibotos-metrics -f metrics.yaml. A few seconds later, the data source Kibotos 指标 and the two dashboards in the folder Kibotos appear in Grafana.

• Data sources and dashboards loaded by the sidecar are read-only. Changes made in Grafana cannot be saved. To change a dashboard, change its JSON in deploy/metrics/, then run helm upgrade. The two dashboards in the chart are symbolic links to these files.
• If the storage has TLS enabled with a certificate issued by a private CA, Grafana must trust that CA, or you add the data source by hand and provide the CA.

Docker ComposeCopy Anchor Link

compose.yaml already opens the metrics listeners of the two gateways. They are reachable only inside the Compose network and are not published to the host. To open the ctrl listener, set listen = "0.0.0.0:18192" in the [telemetry] section of ctrl/config.toml; it stays unpublished as well.

Add a vmagent service to the same Compose project to scrape them. Write its scrape configuration based on vmagent.yml, with the target addresses changed to ctrl:18192, e2b:18195, and gw:18197:

  vmagent:
    image: docker.io/victoriametrics/vmagent:v1.154.0@sha256:2a939a9df9c8591aa5447efcd21af8c4f2ecc60ca6a8e742ffd46b9c76ef7f17
    restart: unless-stopped
    command:
      - -promscrape.config=/etc/vmagent/kibotos.yml
      - -remoteWrite.url=http://10.0.0.7:8428/api/v1/write
      - -remoteWrite.basicAuth.username=kibotos
      - -remoteWrite.basicAuth.passwordFile=/etc/vmagent/remote-write-password
      - -remoteWrite.label=host=docker-a
      - -remoteWrite.tmpDataPath=/var/lib/vmagent
    volumes:
      - ./vmagent/kibotos.yml:/etc/vmagent/kibotos.yml:ro
      - ./vmagent/remote-write-password:/etc/vmagent/remote-write-password:ro
      - vmagent-data:/var/lib/vmagent

Set host to the name of this Docker host, and add vmagent-data to the volumes at the end of the file.

GrafanaCopy Anchor Link

Add a Prometheus data source: set the URL to the VictoriaMetrics address http://10.0.0.7:8428, turn on Basic auth, and enter the username and password. If you installed the storage with the chart, set the URL to vmauth (inside the cluster, http://kibotos-metrics-vmauth.kibotos-metrics.svc:8427) and use the username and password of a read-only user. Then import the two dashboards below, selecting this data source when you import:

The instance metrics storage has two more dashboards, "Kibotos 实例概览" (instance overview) and "Kibotos Workspace", whose data source points to that storage. See Instance metrics storage dashboards.

"Kibotos 服务" narrows the scope with three variables: job, host, and node. Its node panels use kibotos_node_info to map the series scraped on hosts to node IDs and filter by node.

TroubleshootingCopy Anchor Link

Next stepsCopy Anchor Link

• Logs: collect and query logs on the same hosts.
• Instance metrics storage: let the E2B SDK read the CPU, memory, and disk readings of a sandbox.
• Run in containers: example manifests for Kubernetes and Docker Compose.