Metrics
Metrics
This page lists every metric the Kibotos processes serve at GET /metrics, generated from the source code that creates them. For which process serves which metrics and how to collect them, see Metrics.
kibotos_build_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_build_info | gauge | component, version, revision | Always 1; the labels identify the running executable. |
kibotos_cluster_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_cluster_command_oldest_processing_seconds | gauge | method | Age on ctrl's clock of the oldest command of the method that has not completed. |
kibotos_cluster_commands_processing | gauge | method | Commands of the method that have not completed; absent for a method with none. |
kibotos_cluster_node_allocated_cpu_cores | gauge | node_id | CPUs allocated to Instances as of the node's latest inventory. |
kibotos_cluster_node_allocated_memory_bytes | gauge | node_id | Memory allocated to Instances as of the node's latest inventory. |
kibotos_cluster_node_certificate_expiry_timestamp_seconds | gauge | node_id, purpose | Unix time the node's control or data-plane certificate expires. |
kibotos_cluster_node_clock_offset_seconds | gauge | node_id | The node's clock minus ctrl's, as ctrl last measured it on the live session. |
kibotos_cluster_node_cpu_idle_ratio | gauge | node_id | Share of the node host's CPU time spent idle over its last sampling interval, 0 to 1. |
kibotos_cluster_node_instances | gauge | node_id | Instances in the node's latest inventory. |
kibotos_cluster_node_memory_available_bytes | gauge | node_id | The node host's available memory. |
kibotos_cluster_node_memory_tight | gauge | node_id | 1 while the node's live session reports its host memory too tight for new Instances, 0 otherwise. |
kibotos_cluster_node_pressure_ratio | gauge | node_id, resource, kind | The node host's 60-second pressure stall average, 0 to 1. |
kibotos_cluster_node_schedulable_cpu_cores | gauge | node_id | CPUs placement admits against on the node: its allocatable CPUs times the overcommit ratio. |
kibotos_cluster_node_schedulable_memory_bytes | gauge | node_id | Memory placement admits against on the node: its allocatable memory times the overcommit ratio. |
kibotos_cluster_node_status | gauge | node_id, status | 1 for the node's current status: syncing, ready, draining, drained, unavailable, revoked or retired. |
kibotos_cluster_node_unready | gauge | node_id, reason | 1 with the reason ctrl does not place on an admitted node now; absent while the node takes placement. |
kibotos_cluster_node_vm_cpu_cores | gauge | node_id | CPUs every VM on the node used over the node's last sampling interval. |
kibotos_cluster_node_vm_memory_bytes | gauge | node_id | Guest memory every VM on the node holds. |
kibotos_cluster_node_volumes | gauge | node_id | Volumes in the node's latest inventory. |
kibotos_cluster_storage_pool_capacity_bytes | gauge | pool_id | Capacity the operator set on the storage pool; absent while it has none. |
kibotos_cluster_storage_pool_reserved_bytes | gauge | pool_id | Declared capacity of the live Volumes and retained layers in the storage pool. |
kibotos_cluster_storage_pool_retained_layers | gauge | pool_id | Retained layers in the storage pool. |
kibotos_cluster_storage_pool_volumes | gauge | pool_id | Live Volumes in the storage pool. |
kibotos_cluster_volumes | gauge | state | Volumes in each state; deleted ones are not counted. |
kibotos_cluster_workspaces | gauge | state | Workspaces in each state; deleted ones are not counted. |
kibotos_ctrl_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_ctrl_cluster_read_failures_total | counter | Reads of the cluster state for its metrics that failed or timed out; the scrapes such a read answers have no kibotos_cluster_ series. | |
kibotos_ctrl_command_attempt_failures_total | counter | method | Attempts of an accepted command that the executor ran and that failed; the command is attempted again. |
kibotos_ctrl_command_exhaustions_total | counter | method, reason | Commands the executor gave up on because their execution budget ran out, by which budget. |
kibotos_ctrl_command_settlements_total | counter | method, code | Background commands settled, by method and by the code of the result every caller gets: ok or a Connect code name. |
kibotos_ctrl_db_acquire_seconds_total | counter | Time spent acquiring connections from the ctrl PostgreSQL pool. | |
kibotos_ctrl_db_acquire_waits_total | counter | Connection acquisitions that found no idle connection and waited for one to be released or opened. | |
kibotos_ctrl_db_connections | gauge | state | Connections in the ctrl PostgreSQL pool, by whether a caller holds them. |
kibotos_ctrl_db_max_connections | gauge | The most connections the ctrl PostgreSQL pool opens. | |
kibotos_ctrl_node_session_ends_total | counter | cause | Node control sessions that ended, by why they ended. |
kibotos_ctrl_nonconforming_command_ids_total | counter | Command IDs on write requests that break the rule ctrl enforces from v0.4.0. |
kibotos_gateway_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_gateway_auth_rejections_total | counter | surface, reason | API keys, envd tokens and traffic tokens the gateway refused. |
kibotos_gateway_proxy_stream_aborts_total | counter | reason | Proxied connections that ended before the exchange on them completed. |
kibotos_gateway_upstream_failures_total | counter | operation, code | Calls to ctrl or a node that failed for a reason other than the caller's own request. |
kibotos_gw_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_gw_active_streams | gauge | Proxied connections in progress. | |
kibotos_gw_stream_bytes_total | counter | direction | Bytes proxied connections carried to and from Workspaces. |
kibotos_gw_stream_refusals_total | counter | scope | Connections refused because a concurrency budget was used up. |
kibotos_http_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_http_server_duration_seconds | histogram | route | Time from receiving an HTTP request to its handler returning; an upgraded or proxied stream counts its whole life. |
kibotos_http_server_requests_total | counter | route, code | HTTP requests served, counted when the handler returns. |
kibotos_instance_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_instance_metrics_dropped_samples_total | counter | reason | Instance metric samples lost before the metrics store accepted them, by reason. |
kibotos_instance_metrics_queued_samples | gauge | Instance metric samples waiting to be sent to the metrics store. |
kibotos_node_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_node_control_connected | gauge | 1 while ctrl has admitted the node's control session, otherwise 0. | |
kibotos_node_control_connection_failures_total | counter | phase | Control connections that failed or ended, by the phase they reached. |
kibotos_node_dataplane_credential_rejections_total | counter | reason | Data-plane requests whose credential the node refused, by reason. |
kibotos_node_dataplane_streams | gauge | kind | Data-plane streams open now, by what they carry: process output, a file transfer, a directory watch, or a service connection. |
kibotos_node_guest_log_dropped_records_total | counter | reason | Guest agent records that never reached the node's log: dropped from the guest's buffer before the node read them, or over the Instance's rate. |
kibotos_node_guest_memory_saturated_instances | gauge | Running Instances whose guest reports its managed processes held at their memory limit. | |
kibotos_node_guest_probe_failures_total | counter | Guest probes that timed out or found no connection to the guest. | |
kibotos_node_guest_rpc_duration_seconds | histogram | procedure | Time a unary Connect call to the guest agent took to return. |
kibotos_node_guest_rpc_requests_total | counter | procedure, code | Unary Connect calls made to the guest agent, counted when the call returns. |
kibotos_node_guest_samples_skipped_total | counter | reason | Readings of a ready guest's resource counters, or of one host-side source of its Instance, that produced no samples, by reason. |
kibotos_node_guest_unresponsive_instances | gauge | Running Instances whose guest agent stopped answering probes. | |
kibotos_node_info | gauge | node_id | Always 1; node_id names the node that exports these metrics. |
kibotos_node_instances | gauge | state | Instances this node holds, by state. |
kibotos_node_pending_cleanups | gauge | Instances whose release could not finish and waits to be tried again. | |
kibotos_node_storage_pool_available | gauge | pool_id | 1 when the storage pool's marker verified the last time the node checked it, otherwise 0. |
kibotos_node_template_cache_entries | gauge | Prepared launches the Template cache holds. | |
kibotos_node_template_cache_max_bytes | gauge | Bytes the Template cache may hold. | |
kibotos_node_template_cache_used_bytes | gauge | Bytes the Template cache holds. |
kibotos_rpc_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_rpc_server_duration_seconds | histogram | procedure | Time from receiving a unary Connect call to returning it. Streams are not observed. |
kibotos_rpc_server_requests_total | counter | procedure, code | Connect calls served, counted when the call ends; a stream counts when it closes. |
kibotos_traces_*
| Metric | Type | Labels | Description |
|---|---|---|---|
kibotos_traces_dropped_total | counter | reason | Ended spans that were not pushed to the trace store, by reason. |
node configuration
This page lists every table and key of the node configuration file, generated from the annotated example node.example.toml, which the node tests check against the keys node reads. A table or key the example leaves commented out is optional.
Capability names
This page lists the capability names the server advertises in GetServerInfo, grouped by object, and the rules they follow. Before using what a name covers, check that the name is in the list. See Querying capabilities first.