Storage pools on NFS

This page explains whether a node's storage pool can live on an NFS mount, what the server and client each need, and what it costs in latency.

OverviewCopy Anchor Link

A node keeps Volume block files and retained layers in storage pools. A storage pool is a directory with two fixed subdirectories and a marker file that proves its identity:

A node declares one or more storage pools in its configuration with [[storage.pool]] (id, path, default, shared; see Configuration). A pool directory can live on NFS: the whole directory is mounted as one NFS export, and volumes/, retained-layers/, and the marker file are all inside the export. When the node creates a block file, it only truncates it to the capacity and formats it. After that, it writes only the blocks the guest actually uses. It does not use hole punching, preallocation, or reflinks, and it does not advertise DISCARD to the guest. This has been verified on NFSv3, v4.0, v4.1, and v4.2: both backends can create a Workspace, write, stop, expand, restart, and keep the data. For the differences between versions, see NFS versions.

The other directories under storage.root belong to no storage pool and always stay on the local disk:

<storage.root>/
├── installations/      # runtime installations
├── runtime/            # runtime files and owner lock; storage-pools/<pool-id>/ holds this node's own attachment records
├── state/              # node's own persistent state
├── template-cache/     # Template cache, limited separately by template_cache_limit
├── network/            # host network state
├── pki/                # node's managed identity
└── machine-logs/       # tail of console and VMM logs of ended Instances

The rest of this page covers only storage pool directories.

Directory layoutCopy Anchor Link

Each of the two subdirectories of a pool directory has a fixed shape. File names are object IDs and contain no node identifier. Two nodes mounting the same storage use the same paths. This is by design: ctrl's use rights provide mutual exclusion.

<pool>/
├── .kibotos-storage-pool        # marker file
├── volumes/
│   ├── data/                    # <volume_id>.ext4    sparse block file, length equals capacity
│   ├── metadata/                # <volume_id>.json    spec, actual capacity, whether the last handoff was clean
│   ├── holders/                 # <volume_id>.json    holder: node_id and epoch
│   ├── releases/                # <volume_id>.json    release receipt: the holder that stopped writing
│   ├── locks/                   # <volume_id>.lock    flock, mutual exclusion within this machine only
│   └── transfer-staging/        # staging files for export and import
└── retained-layers/
    ├── data/                    # <opaque_ref>.ext4
    ├── metadata/                # <opaque_ref>.json   continuity, lower identity, capacity
    ├── holders/                 # same as above
    ├── releases/                # same as above
    └── locks/                   # <opaque_ref>.lock

transfer-staging/ holds only staging files during export and import, and they are deleted when the command ends. The Transfer archive itself is not on the node. It is kept in ctrl's artifact store, so an import can run on any node connected to the target pool.

The pool records only facts about the objects themselves, not which Instance has them attached. Attachments are recorded in a local directory of the node that attaches them (runtime/storage-pools/ under storage.root, above), and other nodes never read it.

Each object has three kinds of records in the pool, each in its own file. Every write goes to a temporary file first, then fsync, then an atomic rename:

Two hosts never rewrite the same file, and each record is reopened every time it is read, without caching. Decisions across hosts therefore do not depend on file locks, and they still hold with a nolock mount.

The marker file is a piece of JSON:

{
  "version": 1,
  "pool_id": "shared",
  "instance": "bf54d7bdc3836824676f7ffef11ebc4d",
  "created_at": "2026-09-17T20:20:23.084208280Z"
}

instance is generated once at initialization. It identifies the storage itself, not the name given to it. In a shared pool, all nodes read the same file, so they report the same instance to ctrl. If someone gives another directory the same pool_id, its instance differs, and ctrl rejects the one that joins later. In that case, kibo storage-pool get <pool> lists the later one as unavailable with the reason, and the ctrl log records it too. The most common cause is several nodes using the same id for their local pools.

File names and object IDsCopy Anchor Link

You can recognize objects by looking at the directory, and you can go straight from ctrl's records to the files. IDs can be used directly as file names because of their shape:

• The ctrl service guarantees that volume_id and opaque_ref are 1–32 bytes of lowercase letters, digits, and hyphens, and do not start with a hyphen.
• The node checks again before use and rejects any command whose ID does not fit the shape, without writing any file. IDs therefore never contain /, .., or spaces, and never fold into the same name on a case-insensitive file system.
• The real identity is still recorded in the metadata JSON. The provider checks that it matches the file name when reading and rejects the object if it does not.

Versions before v0.1.0 used the SHA-256 of the ID (64 lowercase hex characters) as the file name. After an upgrade, these objects keep working and are renamed to their IDs when they are not attached:

• When the node starts, it renames the objects it holds that are not attached. Attached objects are renamed when they are released. Objects not renamed at startup are renamed before their next attach or deletion.
• In a shared pool, only the current holder renames. After another node takes over an object with an old name, that node renames it.
• The files of one object are renamed in a fixed order, with metadata last. If the node exits in the middle of a rename, the next node that uses the object finishes it: that may be the same node after it restarts, or another node that takes over the object.
• When the same file exists under both names, renaming stops. See below.

The startup log line node storage objects by name shows, for each pool, how many objects were renamed and how many old names remain. To match an object with an old name to its ID, read the metadata JSON or compute it yourself:

printf '%s' vol-06gb22twhsw9h7t04g99e42ndc | sha256sum

Same file under two namesCopy Anchor Link

The node does not decide which copy is correct. Renaming of that object stops until the extra copy is moved away:

• The startup log records node storage objects kept digest names, with the object in the error field.
• Operations that use the object fail with a retryable error, including attach, delete, expand, and the release when a Workspace stops. Meanwhile, ctrl keeps retrying.

This usually comes from restoring from a backup or copying a pool directory by hand. Handle it according to the directory the file is in:

Move the extra copy outside the pool directory first, and delete it after the Workspace is back to normal. Once it is moved away, the next retry finishes the rename. You do not need to restart the node.

Standalone and shared poolsCopy Anchor Link

• When only one node declares a pool, two nodes cannot point their volumes/ and retained-layers/ at the same storage: it is still an ordinary standalone export, and each node uses its own directory.
• With shared = true on a pool, several nodes can declare the same pool ID on the same storage. This is how a stopped Workspace can start on another node. Mutual exclusion across hosts no longer relies on file locks but on use rights granted by ctrl: ctrl gives the use right of a Volume or a retained layer to only one node, and a node without the use right does not open the corresponding block file. A cloud provider NAS mounted with nolock is therefore still safe. See Prerequisites.
• The node itself still uses flock in these two directories to prevent concurrency within the machine. Mutual exclusion across hosts does not depend on it.
• After a node goes down, the Operator declares with kibo node isolate that it can no longer write to storage. After that, a Workspace whose objects are in a shared pool can start or be deleted on another node. A Workspace whose objects are in a standalone pool can be deleted, and its files are cleaned up once the pool has an available node. To start it, wait for that node to return, or attach the disk to another host as described in Moving a pool to another host. If the disk is lost too, declare the pool lost as described in Declaring a pool lost. See Nodes.
• To move existing Volumes and writable layers from a standalone pool into a shared pool, see Moving objects between pools.

Moving a pool to another hostCopy Anchor Link

A pool that only one node connects to can move to another host when the original host is lost but the disk is intact. When the disk is lost as well, declare the pool lost instead:

1. Isolate the original node: kibo node isolate. Confirm that the original host can no longer write to this storage.
2. Retire the original node: kibo node retire.
3. Attach the disk to the new host, and declare it in the new node's configuration with the same pool ID. The marker in the pool directory stays the same.

After the new node registers, it becomes a member of this pool:

• Pending physical cleanup runs on it.
• Existing Workspaces can start on it, and the contents of /data are kept. If the original node was isolated, the first start opens the objects after waiting the maximum time for a release receipt, and the writable layer reports UNKNOWN. If the original node was retired from DRAINED, the objects open directly.
• check-maintenance on the new node does not fail because the pool has holder records naming the original node.

Epochs and takeoverCopy Anchor Link

Every time ctrl hands an object to another node, or grants it again after the holder is isolated, it increments the epoch and sends it to the node with every command that opens the block file. Before opening a block file, the node compares the epoch recorded in the pool's holders/ record with the epoch the command carries:

When taking over, if the previous holder was isolated, the node first waits for its release receipt, for at most 15 seconds (measured on a monotonic clock):

• If the receipt appears in time, the node opens immediately and records that the handoff was clean.
• If there is still no receipt at the time limit, the node opens anyway and records released_cleanly=false in the metadata.

If an isolated node can in fact still reach the pool, its check every 5 seconds finds the newer epoch. It then stops writing, flushes and closes the files, and writes the receipt, so 15 seconds is enough for it to step aside. If the previous holder detached normally, there is no wait. When a Workspace takes over both a Volume and a writable layer, the two receipts are awaited together, also for at most 15 seconds.

On a normal detach, the node flushes and closes the block file and records in the metadata that the release was clean. The holder record stays, so the next holder can see which epoch it succeeds.

PrerequisitesCopy Anchor Link

Server:

• Export the directory with rw,sync,no_subtree_check,no_root_squash. The node runs as root. It changes the volumes/ and retained-layers/ subdirectories to root:kibotos-vmm and 0710 when it creates them, and changes block files to root:kibotos-vmm and 0660 when it creates them. root_squash makes all of these steps fail.
• Use a sync export: data already COMMITted to the server then survives a server power loss, and async removes that guarantee. For what a guest fsync guarantees, see Notes.
• Only one node uses the export directory of a standalone pool. Only a shared pool allows several nodes to mount the same export, and each node's configuration must mark the pool shared = true.

Client (node host):

• Install the NFS client (nfs-common on Debian and Ubuntu).

• State the version in the mount options (vers=4.2, vers=4.1, vers=4.0, or vers=3) rather than relying on negotiation. For the requirements of each version, see NFS versions. For the mount commands of each version, see Mount options for other versions.

• Use hard in the mount options.

• Do not add nocto to the mount of a shared pool. When a node takes over another node's object, it reopens the records the previous holder wrote, and nocto can make it read stale content. The Linux client keeps close-to-open consistency by default, so leave this option out.

• When the server provides NLM locks (a self-hosted Linux nfsd), do not add nolock or local_lock. When the server does not provide NLM locks (most cloud providers' NFSv3), you must add nolock. See Cloud provider NAS.

• The uid and gid are the same on the server and on the node host: block files are stored as root:kibotos-vmm, and NFSv4 idmap must also map these two accounts consistently.

• kibotos-vmm must be able to traverse the pool directory itself. The node manages the owner and permissions only of the volumes/ and retained-layers/ subdirectories and does not modify the pool directory itself. This is the same rule as every directory level from storage.root up to / being traversable by kibotos-vmm (see Identity, configuration, and connections). After mounting, run once:

  Copy Text

  sudo chown root:kibotos-vmm /var/lib/kibotos/pool-a
  sudo chmod 0710 /var/lib/kibotos/pool-a

  For a shared pool, run it once on any one of the hosts: the permissions are stored on the server, and they take effect directly on the other hosts that mount the same export.

• Finish mounting before the node service starts. The pool directory itself must already exist, whether it is a mount point or a local directory. When the node creates volumes/ and retained-layers/ under it, a missing pool directory makes the node fail to start rather than marking the pool unavailable.

ConfigurationCopy Anchor Link

A node declares each pool with [[storage.pool]] in [storage]:

[storage]
root = "/var/lib/kibotos/node"
template_cache_limit = "100G"

[[storage.pool]]
id = "pool-a"
path = "/var/lib/kibotos/pool-a"
default = true

[[storage.pool]]
id = "shared-a"
path = "/var/lib/kibotos/shared-a"
shared = true

Pool capacity is not set here. See Pool capacity.

When several nodes declare the same shared pool, keep the pool's default value the same in each configuration: whether ctrl treats a pool as a node's default follows the value that node declares itself.

Initializing a poolCopy Anchor Link

kibotos-node storage-pool init writes the marker file into the pool directory to prove that this directory is the pool declared in the configuration. The node itself never creates or repairs this file: without it, every operation that uses the pool treats the pool as unavailable. The command does not read or need the node's configuration file. --id and --path must match the values in [[storage.pool]].

For a new, empty pool directory:

sudo kibotos-node storage-pool init --id pool-a --path /var/lib/kibotos/pool-a

If the directory does not exist, it is created. If the directory exists, is not empty, and has no marker file, the command is rejected unless you add --adopt:

sudo kibotos-node storage-pool init --id pool-a --path /var/lib/kibotos/pool-a --adopt

--adopt is for a directory that already holds Volumes or retained layers. It keeps the data and only adds the marker file. Running init again on a pool that is already initialized does not fail. Running it on the same path with a different --id is rejected, and the marker file is never silently overwritten.

Order matters. If you start the node before mounting and initializing the pool directory, the node creates empty volumes/ and retained-layers/ subdirectories in it (the marker file does not exist yet, so the pool stays unavailable). When you then run init, the directory is no longer empty, and you need --adopt to continue. This does not overwrite any data. It only bypasses the empty-directory check.

StepsCopy Anchor Link

The example uses storage.root = /var/lib/kibotos/node, a standalone pool pool-a, and a server nfs.example.com that exports /srv/kibotos-nfs/pool-a.

1. Export the directory on the serverCopy Anchor Link

set -eu
sudo install -d -m 0700 /srv/kibotos-nfs/pool-a
printf '%s\n' '/srv/kibotos-nfs/pool-a 192.0.2.10(rw,sync,no_subtree_check,no_root_squash)' | \
  sudo tee -a /etc/exports >/dev/null
sudo exportfs -ra

192.0.2.10 is the address of the node host.

2. Mount and initialize on the node hostCopy Anchor Link

Stop the node service first, mount the export, set the permissions, and then write the pool marker:

set -eu
sudo systemctl stop kibotos-node.service
sudo install -d -m 0700 /var/lib/kibotos/pool-a
printf '%s\n' \
  'nfs.example.com:/srv/kibotos-nfs/pool-a /var/lib/kibotos/pool-a nfs vers=4.2,hard,_netdev 0 0' | \
  sudo tee -a /etc/fstab >/dev/null
sudo mount /var/lib/kibotos/pool-a
sudo chown root:kibotos-vmm /var/lib/kibotos/pool-a
sudo chmod 0710 /var/lib/kibotos/pool-a
sudo kibotos-node storage-pool init --id pool-a --path /var/lib/kibotos/pool-a

Add the matching [[storage.pool]] section to [storage] in the configuration (see Configuration), then start the node:

sudo systemctl start kibotos-node.service

3. VerifyCopy Anchor Link

Create a Workspace, then look at the block files on the node host:

sudo ls -ls /var/lib/kibotos/pool-a/volumes/data /var/lib/kibotos/pool-a/retained-layers/data

The first column is the number of blocks actually used, which should be much smaller than the file length. Use kibo storage-pool list to confirm the pool's availability from the ctrl side. See Storage pools.

Shared pools: another node mounting the same storageCopy Anchor Link

On the second node host, repeat the mount and permission settings of step 2 (with the same export), but do not run storage-pool init again: the marker file is already in the server directory, and both hosts see the same content. Both configurations must mark the pool shared = true:

[[storage.pool]]
id = "shared-a"
path = "/var/lib/kibotos/shared-a"
shared = true

Cloud provider NASCopy Anchor Link

NFS services hosted by cloud providers (Alibaba Cloud NAS, AWS EFS, and others) replace the self-hosted server above. Their server settings are made in the console, and their protocol versions and lock support differ from a self-hosted nfsd. This section uses Alibaba Cloud NAS as the example. Handle other providers with the same mapping table.

Server (console):

• In the permission group, set Linux user mapping to "no mapping" (no_squash). The node changes the owner of subdirectories and block files as root, so mapping root away makes this fail. This is the equivalent of no_root_squash on a self-hosted server.
• A standalone pool uses its own file system or its own subdirectory, by the same rule as a self-hosted server. Only the nodes of a shared pool point to the same one.
• The product determines the versions: Alibaba Cloud Extreme NAS has only NFSv3, and General-purpose NAS has NFSv3 and NFSv4.0. Neither has v4.1 or v4.2. Choose v4.0 when you can: locking happens inside the protocol and does not depend on a separate NLM lock service. See NFS versions.

On the client (node host), using Alibaba Cloud Extreme NAS as the example, write /etc/fstab like this:

printf '%s\n' \
  'file-system-id.region.extreme.nas.aliyuncs.com:/pool-a /var/lib/kibotos/pool-a nfs vers=3,nolock,noacl,noresvport,hard,_netdev 0 0' | \
  sudo tee -a /etc/fstab >/dev/null

The remaining steps (stop the node, create the mount point, mount, set permissions, initialize, start the node, verify) are the same as in Steps. How each option in the provider's recommended command maps to the requirements on this page:

v3 transfers numeric uids and gids without idmap, so only the node host needs a kibotos-vmm account. The server does not check whether this gid exists.

The option set vers=3,nolock,noacl has been verified against a self-hosted nfsd through the node's full workflow: creating a Workspace, writing, stopping, restarting with the writable layer and Volume data kept, and deleting. noresvport is included only on the provider's recommendation and has not been verified separately.

NFS versionsCopy Anchor Link

All four versions can hold block files. They differ in locking and recovery.

The version choice does not affect flush latency: the measured latencies of all four versions are of the same order. See Notes. The reasons to choose v4.1 or v4.2 are the recovery semantics and one less lock service. nconnect=4 does not improve the latency of a single Workspace and is not needed.

Mount options for other versionsCopy Anchor Link

The example in step 2 uses v4.2. To use another version, change only vers= in /etc/fstab. The other options stay the same.

v4.1:

printf '%s\n' \
  'nfs.example.com:/srv/kibotos-nfs/pool-a /var/lib/kibotos/pool-a nfs vers=4.1,hard,_netdev 0 0' | \
  sudo tee -a /etc/fstab >/dev/null

v4.0:

printf '%s\n' \
  'nfs.example.com:/srv/kibotos-nfs/pool-a /var/lib/kibotos/pool-a nfs vers=4.0,hard,_netdev 0 0' | \
  sudo tee -a /etc/fstab >/dev/null

For v3 with a self-hosted nfsd, first make sure the client's rpc.statd is running, then write fstab (cloud providers' v3 does not follow this path; see Cloud provider NAS):

sudo systemctl enable --now rpc-statd.service
printf '%s\n' \
  'nfs.example.com:/srv/kibotos-nfs/pool-a /var/lib/kibotos/pool-a nfs vers=3,hard,_netdev 0 0' | \
  sudo tee -a /etc/fstab >/dev/null

For v3, the server must also open the ports of rpcbind (111), mountd, lockd, and statd. v4 uses only 2049. Writing vers=4 without a minor version makes the client and server negotiate the minor version, and the result depends on the kernels on both sides. Do not write it that way.

To try a version temporarily, you can mount directly without changing fstab:

sudo mount -t nfs -o vers=3,hard nfs.example.com:/srv/kibotos-nfs/pool-a /var/lib/kibotos/pool-a

After mounting, check the version actually in effect in /proc/mounts:

grep /var/lib/kibotos/pool-a /proc/mounts

vers=4.2, vers=4.1, vers=4.0, or vers=3 should appear in the mount options.

NotesCopy Anchor Link

• Both backends have the same semantics: a guest fsync only guarantees that the data reaches the host page cache, and the host kernel writes it back. Data survives a normal stop and start, a node restart, and a normal host reboot. A host power loss or kernel crash may lose recent writes that were not yet written back. fsync returning does not mean the data has reached the NFS server. This is a boundary, not a defect.

• A guest fsync does not trigger a host fsync of the block file, so on NFS it does not produce a COMMIT round trip each time. The table below shows measurements of the KiboVM backend on one host: average latency of 4 KiB appending writes plus fsync, with the NFS server on another machine and a round trip of about 2 ms between them. Where several rounds were measured at the same location, a range is shown.

  Target	Local disk	NFSv3	NFSv4.0	NFSv4.1	NFSv4.2
  Writable layer	0.4 ms	0.8–1.4 ms	1.7 ms	1.0 ms	0.8–0.9 ms
  Volume	0.5 ms	1.0–1.6 ms	0.7 ms	0.8 ms	0.9–1.0 ms

  The Firecracker backend stays below 1 ms at every location and version.

• Latency jitters. In one NFSv3 round, the writable layer had a single latency of 341 ms, and that round's average reached 8.4 ms. Other load on the host also amplifies latency. When you use NFS for latency-sensitive Workspaces, do not use the node host as a build machine as well.

• Sequential write throughput is mainly limited by network bandwidth. In measurements, 32 MiB sequential writes over a gigabit network were of the same order as on a local disk.

• Block files are opened with buffered I/O. Guest writes go into the host page cache first, and data not yet written back takes up host memory.

• The node limits the dirty pages queued for each NFS mount: after the pool's marker check passes, it sets max_bytes of the mount's writeback device /sys/class/bdi/<major>:<minor>/ to 256 MiB and strict_limit to 1. An existing max_bytes that is already smaller is kept. Without a limit, one VMM that writes continuously can queue GBs of dirty pages on the host: on a slow link, after this node is isolated it takes a long time to flush and write its release receipt, small requests that read the pool queue behind these writes, and the node taking over may open the object before this one stops writing.

  • Several exports on the same NFS server may share one writeback device, and this limit applies to them together.
  • The setting persists after the node stops, until a remount or host reboot. The node sets it again every time it checks the pool.
  • To see the current value: cat /sys/class/bdi/$(mountpoint -d <pool>)/max_bytes.
  • 256 MiB is sized for a gigabit link: at about 110 MB/s, one self-check interval (5 seconds) flushes about 550 MB.

• When expanding, the node runs e2fsck and resize2fs on the whole block file on the host, and the data is read over the network. When the target exceeds the descriptor table the file system reserved, it also first removes resize_inode with tune2fs and checks again. Each of these tools is limited to 10 minutes and the whole expansion to 45 minutes, so a large writable layer or Volume on a slow network may time out.

• After the NFS server restarts, there is a lock recovery period. During it, the node may fail to lock block files, which shows up as a retryable conflict when creating or starting a Workspace. Retry after the recovery period ends. Mounts with nolock have no such period.

• Block files only grow. Deleting files in the guest does not reduce the block file's usage on the server, and the same is true on a local disk.

• On a shared pool, a node's check-maintenance (the check before replacing the binaries; see Reinstall and upgrade) does not fail because the pool holds Volumes or retained layers held by other nodes. It checks only the part it holds itself. Objects whose holder record names another node are not its to check.

TroubleshootingCopy Anchor Link

Next stepsCopy Anchor Link

• For every key of [[storage.pool]], see node configuration.
• Use kibo storage-pool list, get, and objects to see a pool's capacity, reservations, the nodes that declare it, and the objects in it. See Storage pools.
• After a node goes down, use kibo node isolate so that its Workspaces can start again. See Nodes.
• For expanding writable layers and Volumes, see kibo CLI.