Operating a desktop

An Agent that operates a graphical desktop runs one loop: take a screenshot, decide what to do, send keyboard and mouse input, and look again. This page shows that loop. The complete program is examples/go/computer-use.

PrerequisitesCopy Anchor Link

• Go 1.27 or later, and a checkout of the Kibotos repository.
• The ctrl address, and an enrollment key for this program from kibo enrollment create --operator. See Identity and enrollment.
• A Template with a desktop, such as kibotos-desktop, and a server with the workspace.desktop capability.

StepsCopy Anchor Link

1. Create the Workspace and a desktop sessionCopy Anchor Link

A desktop and a browser need more than the smallest resources. This session can only see and operate the desktop, so a mistake in the Agent's tools cannot run commands:

session, err := dataplane.Open(client, workspaceID,
	dataplane.WithCommandPrefix("computer-use"),
	dataplane.WithScopes(ctrlv1.DataPlaneScope_DATA_PLANE_SCOPE_DESKTOP))
if err != nil {
	return err
}
defer session.Close()
desktop, err := session.Desktop(ctx)

2. Wait for the desktop to drawCopy Anchor Link

The first call starts the desktop, and a new desktop's first screenshot is usually blank. The program takes screenshots until the screen changes, then gives the desktop time to settle. See waitForDrawing in the program.

3. Take a screenshotCopy Anchor Link

response, err := desktop.Screenshot(ctx, connect.NewRequest(&ctrlv1.ScreenshotRequest{}))
if err != nil {
	return err
}
image := response.Msg.GetImage()

Send the image to your model, together with GetWidth() and GetHeight(), so that it can return coordinates.

4. Send the actionsCopy Anchor Link

Turn the model's answer into actions. A wait inside the request lets the next screenshot show the result:

result, err := desktop.PerformActions(ctx, connect.NewRequest(&ctrlv1.PerformActionsRequest{
	Actions: []*ctrlv1.DesktopAction{
		{Action: &ctrlv1.DesktopAction_ClickPointer{ClickPointer: &ctrlv1.ClickPointer{
			Button:   ctrlv1.PointerButton_POINTER_BUTTON_RIGHT,
			Count:    1,
			Position: &ctrlv1.ScreenPosition{X: 640, Y: 400},
		}}},
		{Action: &ctrlv1.DesktopAction_Wait{Wait: &ctrlv1.WaitInput{DurationMs: 1000}}},
	},
}))

The server checks the whole list before it delivers any action. For the limits, see With the API.

5. Run the programCopy Anchor Link

The program stands in for the model: it opens a menu with a right-click, closes it with escape, and saves the screen of each step:

go run ./examples/go/computer-use -endpoint https://ctrl.example.com:18092 \
  -identity "$HOME/.local/state/kibotos-computer-use" -enrollment-key-file computer-use.key \
  -template kibotos-desktop -output screens

created Workspace ws-06gvz1jkrhy4h7cahwqck873d0
step 0: 1280x960 screen saved to screens/step-0.png
step 0: pointer at 640,480
step 1: 1280x960 screen saved to screens/step-1.png
step 1: pointer at 640,480
step 2: 1280x960 screen saved to screens/step-2.png
deleted Workspace ws-06gvz1jkrhy4h7cahwqck873d0

The screen size comes from the Template, so it may differ in your deployment.

NotesCopy Anchor Link

• A person can watch or take over at the same time with kibo workspace desktop view. Neither takes the desktop from the other. See Viewing the desktop in a browser.
• Screenshots may contain whatever the desktop shows, including secrets typed into it. Handle them as you handle the Agent's other data.
• This scenario has no E2B SDK version: @e2b/desktop has not been tested on Kibotos. See Migrate from E2B.

Next stepsCopy Anchor Link

• Desktop: every input action, key names, and limits.
• Agent platform integration: identity, session mapping, and failure handling.
• DesktopService: every method and message.