Storage pools

This page shows how to inspect storage pools and the objects in them, set pool capacity, move objects between pools, and declare a pool lost. For the storage pool concept, declaration, and initialization, see Storage pools on NFS.

For command IDs and expected revisions on write commands, see Command IDs and expected revisions.

Viewing and setting capacityCopy Anchor Link

Use kibo storage-pool list to see every storage pool that ctrl knows about, kibo storage-pool get <pool> to see the details of one pool, kibo storage-pool objects <pool> to list the objects in a pool, and kibo storage-pool set-capacity <pool> <capacity> to set pool capacity:

kibo storage-pool list --page-size 20
kibo storage-pool get shared
kibo storage-pool objects shared --all
kibo storage-pool set-capacity shared 4T

list prints one pool per row:

In addition to the items above, get shows the remaining capacity and Awaiting removal, and lists each node that declares the pool on its own line. Awaiting removal is the number of copies still left in the pool, waiting for a node to delete them: source copies of objects that have moved away, and partial data written to the target pool by a failed move. They take up storage space but do not count toward the reservation.

Nodes
  node-a (default, holds 12): available
  node-b: available
  node-c (holds 3): unavailable: storage pool marker is missing

• default means this node uses it as its default pool. All nodes of a shared pool should declare the same default pool.
• holds N is the number of objects this node holds use rights for, that is, the Volumes and writable layers it is using. It is hidden when zero. It drops to zero after the node is isolated.
• The text after unavailable is the reason the node reports.
• The Storage pools section of kibo node get <node> gives the same information from the node's side.

get also shows the pool's Version. set-capacity compares against it:

• Without --expected-version, the command reads the current version first and then submits. If someone else has changed the version, it exits with 13 and does not retry. Check the pool again, then set the capacity.
• If the pool does not exist, it exits with 12. Only pools that some node has already declared are known to ctrl.
• An explicit --command-id requires --expected-version as well. Replaying the same pair returns the current pool without changing it again.
• On success, the output is the same view as get. The next placement uses the new capacity; no node restart is needed.

When Awaiting removal is nonzero and the pool has no available node, the copies are deleted automatically once a node connects. When you delete a Workspace or Volume while the pool has no available node and no node holds use rights for the object, the deletion completes immediately, the block files count toward this number, and kibo workspace cleanup-status reports these objects as PENDING. Once a node is available, it deletes the files and the status changes to RELEASED.

Pool capacityCopy Anchor Link

Pool capacity is an attribute of the pool on ctrl, not of the node configuration. Set it with set-capacity as shown above.

• Until its capacity is set, the pool accepts no new Volumes or retained layers.
• A shared pool has only this one capacity, and every node connected to it uses it.
• Growing and shrinking use the same command. If you set it below the reserved amount, existing objects keep working, and the pool only stops accepting new objects.

On ctrl, storage use is accounted per pool. Volumes and retained layers count against their pool at their declared capacity from creation until the object is deleted. Stopping a Workspace or detaching a Volume does not release this capacity.

• Block files are sparse, but what counts is the declared capacity, not the blocks actually used. Set the capacity to the space the pool's storage is willing to commit.
• When a request does not specify a capacity, the node's default storage capacity counts, matching the size the node actually creates.

Objects in a poolCopy Anchor Link

objects prints one object per row:

The number of rows equals the sum of Volumes and Retained layers in get. An object moving in is listed in the source pool until the move settles.

Moving objects between poolsCopy Anchor Link

The pool of a Volume or retained layer is fixed at its first placement. To move it to another pool, for example to move objects from a node's private pool into a shared pool after enabling one, use the operator move commands. The object ID, content, and capacity stay the same, and so do the Volume's attachments. Afterward, the Workspace starts on a node that can connect to the new pool.

kibo storage-pool objects local-a --all
kibo workspace stop ws-01k6…
kibo workspace move-layer ws-01k6… --storage-pool shared
kibo volume move vol-01k6… --storage-pool shared

RequirementsCopy Anchor Link

If any of the following is not met, the move is rejected and the object stays where it is:

• The object is not in use: the Volume is AVAILABLE, and the Workspace that owns it or has it attached has stopped (STOPPED, or FAILED after a failed start with its execution released).
• At least one available node connects to both the source pool and the target pool. The copy runs on that node.
• After the move, there is still a node that can connect to both the Volume and the writable layer of this Workspace. Otherwise the Workspace can never start again. When the two objects are in different pools, move first the one that brings them back onto the same node.
• The remaining capacity of the target pool can hold the object's declared capacity. During the move, the object holds a reservation in both pools.

If another command is being processed for the object or its Workspace, the move returns a retryable unavailable. Move it after that command ends.

ProcedureCopy Anchor Link

1. In the target pool, ctrl reserves capacity and records the object as moving.
2. The node doing the copy first replays any journal entries in the source file system that have not been replayed, then copies only the used blocks according to the block bitmap into a temporary file in the target pool, verifies it, and renames it into place.
3. Next, ctrl changes the object's location to the target pool, and the Volume's storage_pool changes with it.
4. A node connected to the source pool then deletes the old copy in the source pool. Awaiting removal in kibo storage-pool get is the number of copies not yet deleted. Until they are deleted, objects cannot move back to that pool.

The copy time grows only with the data in use, not with the declared capacity. On an NFSv3 link that writes about 385 MB/s, 10 GiB of used data takes about 22 seconds.

WaitingCopy Anchor Link

• Both move commands wait until the move settles. Each time, ctrl waits at most 1 minute, and kibo sends again with the same command ID until the move settles or --wait-timeout (2 hours by default) is reached.
• Ctrl-C or --wait-timeout only ends the wait and does not cancel the move. kibo exits with 16. Run it again with the same --command-id to keep waiting for it to settle.
• If the object is already in the target pool, the command succeeds directly without copying.

During a move, the following operations return a retryable unavailable whose message says the object is moving: starting the Workspace that owns it or has it attached, attaching or detaching the Volume, expanding, deleting the Volume or the Workspace, and exporting.

FailuresCopy Anchor Link

Declaring a pool lostCopy Anchor Link

When a pool's storage can never be recovered, declare it lost with kibo storage-pool declare-lost.

kibo storage-pool declare-lost local-a

• The request is accepted only when the current incarnation of every member of the pool has been isolated, or the pool has no members. Otherwise it returns failed_precondition with exit code 13, and the error message lists the remaining members. Isolate them, or have them stop declaring this pool, then declare again.
• The command asks for confirmation in a terminal. Without a terminal, add --yes. Replaying the same --command-id returns the same result.

After the declaration:

The pool ID is never used again. Even if the old disk is recovered with the same marker, it stays unavailable. Use a new pool ID for new storage.