Resources and health

This page shows how to change the CPU, memory, and disk of a Workspace, which task and memory limits apply inside the guest, how to tell whether the guest still responds or is stuck on memory, and how to read resource metrics over time.

Adjusting resourcesCopy Anchor Link

kibo workspace update changes the resources a WorkspaceA complete Linux environment for an AI Agent, running in its own MicroVM. Agents read and write files, run processes, and start services in it without affecting the host or other Workspaces. requests. Give at least one of --cpu, --memory, and --disk. Values use the same format as create and are final values, not increments.

Stop the Workspace, change its resources, and start it again:

kibo workspace stop "$workspace_id"
kibo workspace update "$workspace_id" --cpu 2 --memory 4G --disk 40G
kibo workspace start "$workspace_id"

• --cpu and --memory only update the configuration and take effect at the next start. A successful command means the new values are saved; it does not guarantee enough node capacity at the next start. If a start fails for lack of capacity, lower them with this command and try again.
• --disk enlarges the writable layer right away, keeping its content. When the storage that holds the layer cannot fit the new size, the command fails with resource_exhausted and the capacity is unchanged.

To change the network policy, use kibo workspace network. See Network policy.

When you can updateCopy Anchor Link

The Workspace must hold no execution. The table shows what to do first in each case:

CPU and disk valuesCopy Anchor Link

• The number of vCPUs an instance gets is the millicores divided by a thousand, rounded up: 1.5 gets 2 vCPUs, and 0.5 gets 1. The node that hosts the instance decides at start time how many vCPUs a single instance can get.
• --disk sets the capacity of the writable layerThe writable part of a Workspace's root file system, holding changes such as installed packages; Operator pages call it the retained layer. It is kept on a best-effort basis, and only the Volume is guaranteed to keep data.. It can only grow, and all existing content is kept. A value equal to the current capacity does nothing. A value below the current capacity returns invalid_argument.
• For a Workspace that has never started, the disk capacity is only recorded, and the writable layer is created with it at the first start. The Disk line of kibo workspace get says created at first start while the writable layer does not exist yet, and shows both values when the requested and actual capacities differ.

Partial resultsCopy Anchor Link

On the server, memory, CPU, and the writable layer are three independent commands. No atomic operation changes several at once. update submits only the items you give, in the order memory, CPU, disk:

Revisions and replaysCopy Anchor Link

CPU and memory are checked against one shared resources revision; disk is not. The rules:

• When --expected-revision is not passed, kibo reads the current revision before submitting. If the revision has changed by submission time, update returns a conflict and does not retry automatically.
• Setting the same value does not change the revision.
• For a script that needs to replay, pass --command-id together with --expected-revision. Each item's command ID is that value plus /memory, /cpu, or /disk.

For the API methods behind each item, see Changing resources.

Limits inside the guestCopy Anchor Link

The guestThe Linux system that runs inside the MicroVM of a Workspace. The guest agent in it performs file and process operations for the node. reserves part of the task count and memory for the kernel and platform processes. When user processes use up their share, file operations, process management, and stopping keep working.

TasksCopy Anchor Link

Each process and each thread counts as one task. Two limits apply:

MemoryCopy Anchor Link

• The total memory all processes in the Workspace can use is somewhat below the memory available when the guest boots. The difference, 1/16 of memory kept between 64 MiB and 256 MiB, is reserved for guest programs and the kernel.
• When the total is used up, the kernel ends the process using the most memory, and other processes keep running.
• When processes stay stalled waiting for memory, the guest also ends the largest one. See Memory saturation.

System filesCopy Anchor Link

The guest rewrites /etc/hostname, /etc/hosts, and /etc/resolv.conf at every boot. Changes to them made in the guest, including added immutable or append-only attributes, do not persist across boots.

Responsiveness of the guestCopy Anchor Link

A running Workspace carries a guest observation. The node that runs it probes the guest periodically. Several timeouts or connection failures in a row mark the guest as no longer responding, and one response clears it.

List the Workspaces whose guest no longer responds to its node, then look at one of them:

kibo workspace list --guest-unresponsive
kibo workspace get "$workspace_id"

• The GUEST NOT RESPONDING section at the end of the list output gives the time each guest stopped responding and its latest error.
• The Guest line of get shows the same observation. If the guest has reset itself, it also shows the restart count and the time of the latest reset.

What the observation meansCopy Anchor Link

• It is an observation, not a state. When the guest stops responding, the Workspace is still AVAILABLE. Admission and expiration reclamation ignore the observation, and the guest may recover on its own. Data-plane requests keep timing out in the meantime. The same holds if the guest happens not to respond when the node restarts.
• When it is absent. A Workspace with no running instance (for example, STOPPED) does not carry it. It is also absent before the node has reported and while the guest is rebooting itself.
• After the guest reboots itself. Once the guest answers again, the observation says that the guest rebooted. The restart count, the number of times in this execution that the guest reset itself and the node replaced the VMM, goes up by 1, and the time of the latest reset is the time of this one. The next execution starts the count from 0.
• How soon you see it. It usually shows up within 2 minutes of the guest getting stuck.

Recovering an unresponsive guestCopy Anchor Link

The guest may recover on its own. For a guest that never recovers, reclaim it with a force stop, then start it:

kibo workspace stop --force "$workspace_id"
kibo workspace start "$workspace_id"

Memory saturationCopy Anchor Link

A responsive guest does not mean its processes are doing work. The guest has no swap. When processes together reach the memory limit, the kernel can always reclaim a little from the processes' own file pages (code and files they read). It keeps reclaiming and reading back, and the OOM killer never fires. Processes nearly stall, possibly for hours, while the guest still answers probes.

The guest observation reports this case as a memory condition. The Guest memory line of kibo workspace get shows it:

Detection and killing ship with the Platform. A Workspace has them only after it uses a Platform with this report; before that, the memory condition is unknown, and the guest does not end processes. The Firecracker guest kernel tracks memory pressure only when booted with psi=1, and the Firecracker Platform in this release already includes it.

How saturation is judgedCopy Anchor Link

• The measure. The guest uses the share of the last 60 seconds during which all processes were stalled waiting for memory at the same time. At 25% or more, the guest marks it saturated. Below 10%, it clears.
• What does not count. A short memory peak does not fill the window and is not marked as saturation. When memory is full but processes are not stalled waiting for it, as in an idle Workspace, the share is close to 0.
• It is also an observation. Saturation does not change the Workspace's state, admission, or expiration reclamation. It clears on its own once demand drops. If it never clears, the guest ends processes by the rules in the next section.
• How soon you see it. It usually shows up within 2 minutes of processes starting to stall. Clearing also waits for the stall share in the window to drop.

Killing processes under saturationCopy Anchor Link

When saturation persists, the guest sends SIGKILL to the process using the most memory so the remaining processes can continue.

• When it kills. All three conditions hold:
  • Saturation has lasted 90 seconds.
  • The share in the current 60-second window is still at least 25%. If the share has fallen below 25%, the guest takes no action: it is still reported as saturated but can make progress.
  • Processes actually hit the memory limit during this time. A stall without hitting the limit means the whole guest is short of memory, and the kernel's OOM killer handles it.
• Which process. The managed process with the largest resident memory (RSS). The guest agent and platform processes are not candidates and are never chosen. Only this one process is ended. Its children and other processes in the same session are not affected unless they exit along with it.
• Afterward. No action is taken during a 60-second cooldown. If the three conditions above still hold when the cooldown ends, the next process is ended by the same rules.
• How to tell. The ended process exits as killed by signal 9, which looks the same as a kernel OOM kill. kibo workspace get shows an extra Guest memory kills line with the count, the latest ended process, and the reason. The guest log and the node log each have a record.
• How long the count lasts. The kill count and the latest ended process reset when the guest reboots. They are kept while the guest does not respond, and disappear after the Workspace stops.

Cases killing cannot fixCopy Anchor Link

• Memory in tmpfs (/tmp, /run, /dev/shm) belongs to no process, and ending processes does not free it.
• Thrashing whose share stays below 25% is not marked as saturation, so no action is taken.

Recover from either case with a force stop and a start, or increase memory.

Reading resource metricsCopy Anchor Link

kibo workspace metrics shows a Workspace's resource readings over time, by default for the last hour. The deployment must have connected instance metrics storage for reads. Otherwise kibo stops before reading; see Resource metrics for the capability it checks.

Read the last 24 hours, or a fixed range as JSON:

kibo workspace metrics ws-0123 --since 24h
kibo workspace metrics ws-0123 --start 2026-11-06T08:00:00Z --end 2026-11-06T09:00:00Z --json

Host and guest readingsCopy Anchor Link

The readings come in two series, each in buckets of one step:

BucketsCopy Anchor Link

ctrl picks the step from the length of the range, end − start:

• Rates (cpu_cores, the pressure ratios, the network fields) are averages over the bucket: a rate times the step is the bucket's total. Levels (memory_bytes and every guest field) are the bucket's highest value.
• Every execution of the Workspace is merged into one series, also when it was stopped and started on another node. Nothing is filled in between two executions.
• A host point carries only the fields the storage has samples for. A guest point is returned only when all seven values are present.
• A range that would need more than 4,000 buckets is refused with invalid_argument. Narrow it with --start and --end.

Retention and gapsCopy Anchor Link

• How far back readings reach depends on the storage's retention. Do not assume a fixed length; copy readings elsewhere if you need a longer history.
• Readings are observations, not usage records. They have gaps when the storage is unreachable, the guest does not answer, or a node restarts.
• A deleted Workspace stays readable while the storage still holds its readings. A Workspace with no readings returns two empty series.

With the APICopy Anchor Link

This section lists the methods, fields, and capabilities behind the kibo commands above.

Changing resourcesCopy Anchor Link

kibo workspace update maps to three methods: UpdateWorkspaceMemory, UpdateWorkspaceCpu, and ResizeWorkspaceWritableLayer. The rules in Adjusting resources apply to each of them.

• Memory and CPU use optimistic locking and share one resource revision, resources_revision.
• Growing the writable layer has no expected revision, only a command ID and a target capacity.

Guest observationCopy Anchor Link

A Workspace's guest_observation carries the observation described in Responsiveness of the guest and Memory saturation. Its capability name is workspace.guest_observation. For every field, see GuestObservation.

• condition is GUEST_CONDITION_RESPONDING or GUEST_CONDITION_UNRESPONSIVE. rebooted, restart_count, and last_reset_at describe the guest's own reboots.
• memory_condition is GUEST_MEMORY_CONDITION_NORMAL, GUEST_MEMORY_CONDITION_SATURATED, or GUEST_MEMORY_CONDITION_UNSPECIFIED, which kibo shows as normal, saturated at the memory limit since …, and unknown.
• memory_kill_count and latest_memory_kill describe the processes the guest ended under saturation.

guest_condition on ListWorkspaces returns only Workspaces whose current observation has that value:

• You can combine it with states and label_selector.
• When paging, it must match the request that produced the cursor.
• Older servers ignore guest_condition and return all Workspaces. Confirm that the capability workspace.guest_observation exists before filtering.

Resource metricsCopy Anchor Link

kibo workspace metrics maps to WorkspaceMetricsService/GetWorkspaceMetrics. The deployment must connect instance metrics storage for reads and announce workspace.metrics.get; kibo checks this capability before calling the method. For the request and response, see WorkspaceMetricsService.

• The response carries the step it used, and earliest_time, the earliest time the storage still holds.
• When the storage does not answer in time, the method returns unavailable.
• When too many reads are in progress, it returns resource_exhausted at once. Other methods are not affected.
• An ID ctrl has never known returns not_found.

TroubleshootingCopy Anchor Link

Data-plane requests keep timing outCopy Anchor Link

• Symptom: The Workspace is AVAILABLE, but running commands and reading or writing files time out.
• Cause: The guest no longer responds. Confirm with kibo workspace list --guest-unresponsive or kibo workspace get.
• Fix: The guest may recover on its own. If it never recovers, force stop and then start.

Processes nearly stalledCopy Anchor Link

• Symptom: The guest responds and commands run, but application processes make almost no progress for a long time, or they are ended by signal 9.
• Cause: Processes together reached the memory limit. The Guest memory line of kibo workspace get shows saturated…, or a Guest memory kills line appears.
• Fix: Reduce the load, or stop the Workspace and increase memory. If tmpfs fills memory, clean up files in /tmp, /run, and /dev/shm. If commands no longer run either, force stop and then start.

Next stepsCopy Anchor Link

• Lifecycle: force stops and reboots from inside the guest.
• Commands and terminals: running processes in the guest.
• Instance metrics storage: connecting the storage that kibo workspace metrics reads.
• API overview: calling these methods from your own program.