Lifecycle

You stop, start, and delete a Workspace with the Go SDK, the E2B SDK, or kibo. In the Go examples, settleOptions resends a command while ctrl says it is still executing, as in examples/go/quickstart. This page also covers force stops, checking what a stop or deletion released, recovering a FAILED Workspace, and the cases where the server stops or restarts a Workspace on its own.

Operations by stateCopy Anchor Link

What you can do with a Workspace depends on its state. For what each state means and how states change, see Workspace states.

• Resizing CPU, memory, or the writable layer and rebinding Volumes all require a stopped Workspace.
• Labels and annotations can be changed in every state except DELETED. See Labels and annotations.
• When the expiration time arrives, the system stops or deletes the Workspace as configured. There is no default expiration behavior. See Expiration.

StopCopy Anchor Link

Stop a Workspace normally:

current, err := client.Workspaces().GetWorkspace(ctx, connect.NewRequest(&ctrlv1.GetWorkspaceRequest{WorkspaceId: workspaceID}))
if err != nil {
	log.Fatal(err)
}
request := &ctrlv1.StopWorkspaceRequest{
	Metadata:                   &ctrlv1.CommandMetadata{CommandId: kibotos.NewCommandID("example")},
	WorkspaceId:                workspaceID,
	ExpectedExpirationRevision: current.Msg.GetWorkspace().GetExpiration().GetRevision(),
}
stopped, err := kibotos.Settle(ctx, settleOptions, func(ctx context.Context) (*connect.Response[ctrlv1.StopWorkspaceResponse], error) {
	return client.Workspaces().StopWorkspace(ctx, connect.NewRequest(request))
})
if err != nil {
	log.Fatal(err)
}

In the E2B SDK a stop is a pause. See Pause and resume.

A normal stop first closes the data plane, waits for writes in the guest to finish, and syncs the file systems. Then it stops the Instance and releases resources.

• Writes the application has already handed to the kernel are synced. The application must flush data that is still in its own buffers.
• Bound Volumes and the retained layer of the root file system are kept. Memory and processes are not.
• Data-plane session credentials issued for the Workspace become invalid, and all its Exposures are revoked. After the next start, request new credentials and create new Exposures; see Public URLs.
• If the sync fails or its result is unknown, a normal stop does not force the instance down. Query the result of this stop. If the Workspace does not stop, use a force stop instead.

StartCopy Anchor Link

Start a stopped Workspace:

commandID := kibotos.NewCommandID("example")
started, err := client.Workspaces().StartWorkspace(ctx, connect.NewRequest(&ctrlv1.StartWorkspaceRequest{
	Metadata:                   &ctrlv1.CommandMetadata{CommandId: commandID},
	WorkspaceId:                workspaceID,
	ExpectedExpirationRevision: stopped.Msg.GetWorkspace().GetExpiration().GetRevision(),
}))
if err != nil {
	log.Fatal(err)
}
workspace, err := client.WaitWorkspace(ctx, started.Msg.GetWorkspace(), commandID)
if err != nil {
	log.Fatal(err)
}

A start creates a new instance. It does not restore earlier processes or memory.

• It accepts a STOPPED Workspace. It also accepts a FAILED Workspace that holds no execution and has never begun deletion. See Recovering from FAILED.
• You can also start a Workspace whose creation failed before the instance was built. If the cause of the failure is still there, the start fails for the same reason.
• If creation failed before the Workspace's own Volume was created, the start creates it. The Volume ID, storage pool, and capacity stay the same.

Force stopCopy Anchor Link

A force stop handles cases a normal stop cannot complete, including a stuck stop and a FAILED Workspace that still holds an instance:

kibo workspace stop --force "$workspace_id"

It does not wait for the guest to stop writing. It asks the guest for one short sync, then stops the Instance and releases resources. The Workspace moves to STOPPED with its Volumes and writable layer kept, and you can start it afterward.

• kibo asks for confirmation before a force stop. --yes skips it.
• Like a normal stop, a force stop invalidates the Workspace's data-plane session credentials and revokes all its Exposures.
• If the instance's VMM has already exited, there is no machine to stop. The force stop releases the Volumes, network, and writable-layer mounts the instance still holds, and the Workspace also moves to STOPPED.
• When a normal stop or an expiration stop fails, the server does not switch to a force stop on its own.

When no other command is in progress, a force stop accepts only the Workspaces below. It rejects all others and leaves the Workspace unchanged:

DeleteCopy Anchor Link

Delete a Workspace and wait until it is DELETED:

commandID := kibotos.NewCommandID("example")
request := &ctrlv1.DeleteWorkspaceRequest{
	Metadata:                   &ctrlv1.CommandMetadata{CommandId: commandID},
	WorkspaceId:                workspaceID,
	ExpectedExpirationRevision: workspace.GetExpiration().GetRevision(),
}
if _, err := kibotos.Settle(ctx, settleOptions, func(ctx context.Context) (*connect.Response[ctrlv1.DeleteWorkspaceResponse], error) {
	return client.Workspaces().DeleteWorkspace(ctx, connect.NewRequest(request))
}); err != nil {
	log.Fatal(err)
}
if err := client.WaitWorkspaceDeleted(ctx, workspaceID, commandID); err != nil {
	log.Fatal(err)
}

Before it asks for confirmation, delete tells you whether the bound Volume is deleted with the Workspace or kept. --yes skips the confirmation and is required when there is no terminal to prompt in. Like stop, delete waits for up to one hour by default, and ending the wait does not cancel the deletion. See Waiting for results.

What is deletedCopy Anchor Link

Whether a Volume is deleted with its Workspace is a setting of the binding, not a kind of Volume. You set it with --delete-with-workspace when you create the Workspace or bind the Volume. See Binding, rebinding, and unbinding Volumes.

When you can deleteCopy Anchor Link

• You can delete a Workspace directly in any state except DELETING and DELETED.
• Deleting a CREATING Workspace cancels the in-flight creation.
• Once a delete has been accepted, the Workspace rejects starts, force stops, and configuration changes. To finish a deletion that has begun, delete again. While the Workspace is DELETING, wait for it to finish.
• An expiration time with the action delete deletes the Workspace when it arrives. See Expiration.

After deletionCopy Anchor Link

You can still query a DELETED Workspace by ID with kibo workspace get. Lists leave deleted Workspaces out unless you pass --include-deleted. See Filtering lists.

To check item by item whether the runtime environment, the Volume, the writable layer, and other parts have been released, see Checking cleanup status.

Checking cleanup statusCopy Anchor Link

kibo workspace cleanup-status shows which parts of a Workspace's execution have been released and which are still held. Use it after a stop or delete to confirm that the resources were actually released:

kibo workspace cleanup-status "$workspace_id"

The command is read-only. It does not trigger release or change state.

The output has three parts:

Reading the resultCopy Anchor Link

• The times in the Release section are diagnostic information, not deadlines. Release is still being retried, and the Components section tells which parts are still held.
• When the observation itself is incomplete, the result's completeness is PARTIAL or UNKNOWN, and the exit code is 17 or 18 accordingly. Do not treat unknown parts as released.
• When RETAINED_LAYER or VOLUME_DELETION is PENDING with the control-plane state as the source, the Workspace has been deleted, and the files are deleted once the storage pool has an available node. The overall status is PROCESSING in the meantime.
• After a storage pool is declared lost, these two items are RELEASED: the pool no longer holds any readable data.

Recovering from FAILEDCopy Anchor Link

"Holding an execution" means the Workspace still has an unreleased Instance and resource reservation on some node. No response field reports this directly, so you do not need to check it first. Recover in this order:

1. Force stop. On success the Workspace moves to STOPPED.
2. Start. Do this step even if step 1 was rejected: the Workspace held no execution in the first place.
3. If both steps are rejected, read the Workspace again and act on its current state.

Common FAILED cases and whether they hold an execution:

When the guest is stuck but the VMM is still running, the Workspace stays AVAILABLE and does not move to FAILED. See Responsiveness of the guest.

Stops and restarts handled by the serverCopy Anchor Link

The server handles a host running out of memory without manual intervention. Whether it also handles a reboot or shutdown from inside the guest depends on the backend:

• The force stops after a guest shutdown and on host out of memory are the only force stops the server starts on its own. Their effect is the same as a caller's force stop: Volumes and the writable layer are kept, and the last writes may not have reached the disk.
• The server does not start a stop when the Workspace has begun deletion or another command is in progress.
• To stop or restart a Workspace, use the stop and start operations rather than shutting down or rebooting inside the guest.

Reboot from inside the guestCopy Anchor Link

On the Firecracker backend, when a reset such as reboot -f or echo b > /proc/sysrq-trigger runs in the guest, the node starts a new VMM for the same Instance on the same node. The Workspace stays AVAILABLE throughout, and the generation and data-plane credentials do not change.

• Volumes and the writable layer are kept. Memory and processes are not. Writes that have not reached the disk may be lost.
• Until the guest comes back, new data-plane requests return a retryable unavailable, and established streams end with the old VMM.
• The Guest line of kibo workspace get shows the restart count and the time of the latest reset.
• After more than 3 resets within 10 minutes, the node stops restarting it. The Workspace moves to FAILED, and the failure message states the reset count. Handle it as in Recovering from FAILED.
• If the new VMM fails to start, or the node itself restarts partway through the restart, the Workspace also moves to FAILED, and the failure message states the cause.

Shutdown from inside the guestCopy Anchor Link

On the KiboVM backend, when the guest runs poweroff -f or calls reboot(2) with POWER_OFF, the server force stops the Workspace, and it moves to STOPPED. Start it to continue using it.

• Until the stop completes, the Workspace is still AVAILABLE, and data-plane requests are rejected during this time.
• The expiration policy follows the rules for a stop.
• If the Workspace has begun deletion or another command is in progress, the server waits for the next report before deciding.

Host out of memoryCopy Anchor Link

When the host runs out of memory, the kernel kills one VM on the host. The server force stops the Workspace of that VM, and it moves to STOPPED. Start it to continue using it.

Until the server finishes the stop, the Workspace is briefly FAILED, and the failure message states that the host ran out of memory.

With the APICopy Anchor Link

• Query the result of a stop by its command ID through GetWorkspaceCommand, including when a normal stop's sync failed or its result is unknown.
• A rejected force stop returns failed_precondition with ErrorDetail.reason set to CONFLICT.
• A FAILED caused by an unexpected VMM exit does not set failure.stage, because the failure did not happen during any operation.

Stop durabilityCopy Anchor Link

When the stop completes, the durability of the command result tells whether writes were confirmed on disk:

Stop reasonsCopy Anchor Link

In STOPPING and STOPPED, a Workspace carries last_stop, which tells which stop brought it there. It is no longer returned once StartWorkspace or DeleteWorkspace is accepted.

Command IDs of server-started stops carry a fixed prefix:

• guest-shutdown/ for a guest shutdown.
• host-memory/ for host memory exhaustion.

Callers cannot use these two prefixes in their command IDs; doing so returns invalid_argument. Moving to STOPPED after a guest shutdown corresponds to the capability workspace.stop.guest_shutdown.

TroubleshootingCopy Anchor Link

Start rejectedCopy Anchor Link

• Symptom: Starting a FAILED Workspace returns an error.
• Cause: The Workspace still holds an execution, and the old instance may still hold storage.
• Fix: Force stop first, then start. See Recovering from FAILED.

Next stepsCopy Anchor Link

• Volumes, export, and restore: keep a Volume when its Workspace is deleted, or export the data first.
• Expiration: stop or delete a Workspace automatically.
• Resources and health: resize a stopped Workspace and check whether the guest responds.
• kibo CLI: how kibo waits for stop, start, and delete.