Kibotos
Scenarios

Batch creation and reclamation

This page creates many Workspaces so that a restart creates no duplicates, finds them through a label, and deletes them. The complete program is examples/go/fleet.

Prerequisites

Steps

1. Derive each command ID from your own key

The same batch and index always name the same command. If your program restarts halfway, it sends the same commands again, and ctrl replays them instead of creating more Workspaces:

commandID := kibotos.DeriveCommandID("fleet", "create", batch, strconv.Itoa(index))

2. Create with a label and an expiration time

The label finds the batch later. The expiration time removes the Workspaces if your platform never comes back:

request := &ctrlv1.CreateWorkspaceRequest{
	Metadata: &ctrlv1.CommandMetadata{CommandId: commandID},
	Spec: &ctrlv1.WorkspaceCreateSpec{
		Template:  &ctrlv1.TemplateSelector{TemplateId: templateID},
		Resources: &ctrlv1.ResourceRequest{CpuMillis: 500, MemoryBytes: 256 << 20},
	},
	Labels: map[string]string{"example.com/batch": batch},
	Expiration: &ctrlv1.WorkspaceExpirationPolicy{
		Timeout: durationpb.New(30 * time.Minute),
		Action:  ctrlv1.WorkspaceExpirationAction_WORKSPACE_EXPIRATION_ACTION_DELETE,
	},
}

Bound how many creates run at the same time. The program runs four at most.

3. Handle insufficient capacity

When no node has room, waiting for the Workspace returns the reason RESOURCE_EXHAUSTED. Queue the work and try again:

_, err = client.WaitWorkspace(ctx, created.Msg.GetWorkspace(), commandID)
if kibotos.Reason(err) == ctrlv1.FailureReason_FAILURE_REASON_RESOURCE_EXHAUSTED {
	return fmt.Errorf("no capacity: %w", err)
}

4. Find the batch through its label

List every page of the Workspaces that carry the label. Your platform does not need to keep a list of IDs:

page, err := client.Workspaces().ListWorkspaces(ctx, connect.NewRequest(&ctrlv1.ListWorkspacesRequest{
	LabelSelector: map[string]string{"example.com/batch": batch},
	PageToken:     token,
}))

5. Delete and confirm

Delete each Workspace with the expiration revision the list returned, wait for each deletion, and list again to confirm that none is left:

request := &ctrlv1.DeleteWorkspaceRequest{
	Metadata:                   &ctrlv1.CommandMetadata{CommandId: commandID},
	WorkspaceId:                workspaceID,
	ExpectedExpirationRevision: workspace.GetExpiration().GetRevision(),
}

6. Run the program

A batch name can be used once: running it again replays the finished commands, and their Workspaces are already deleted. Without -batch, the program names the batch after the current time:

go run ./examples/go/fleet -endpoint https://ctrl.example.com:18092 \
  -identity "$HOME/.local/state/kibotos-fleet" -enrollment-key-file fleet.key \
  -batch nightly-2026-11-09 -count 3
batch nightly-2026-11-09 has 3 Workspaces
  ws-06gvz1hxyxyv3crnhyq9pj6848 WORKSPACE_STATE_AVAILABLE
  ws-06gvz1hxz5v3z1466hj4s08ec4 WORKSPACE_STATE_AVAILABLE
  ws-06gvz1hxz9yvd3b4w4n86em0km WORKSPACE_STATE_AVAILABLE
reclaimed 3 Workspaces

With the E2B SDK

Mark the sandboxes with metadata, list them through it, and kill them:

import { Sandbox } from 'e2b';

const batch = 'nightly-2026-11-09';
await Promise.all([0, 1, 2].map((worker) =>
  Sandbox.create('kibotos-minimal', {
    timeoutMs: 30 * 60_000,
    metadata: { batch, worker: String(worker) },
  })));

const paginator = Sandbox.list({ query: { metadata: { batch } } });
while (paginator.hasNext) {
  for (const { sandboxId } of await paginator.nextItems()) {
    await Sandbox.kill(sandboxId);
  }
}

Notes

  • Lists are not a snapshot of one moment. Resources created or deleted while you read the pages may or may not appear. See Pages.
  • Resending a command ID with a different request returns already_exists, so keep each request the same for its key.
  • Deleting a Workspace also deletes a Volume created with it. See What is deleted.

Next steps

On this page