Install node

This page shows how to install node on a host and join it to the cluster.

AppArmor prerequisites on Ubuntu nodesCopy Anchor Link

Ubuntu's AppArmor user namespace restrictions prevent KiboVM from creating its own sandbox. The node package provides the kibotos-vmm profile, which applies only to the KiboVM executables that node installs: it allows them to create user namespaces, while file and device isolation is still done by KiboVM's own sandbox.

Checking and tightening sysctlCopy Anchor Link

On hosts with user namespace restrictions enabled, unprivileged processes must also be prevented from switching to the unconfined profile on their own; otherwise ordinary programs could borrow this permission. First check the current values:

sysctl kernel.apparmor_restrict_unprivileged_userns \
  kernel.apparmor_restrict_unprivileged_unconfined

If the first value is 1, the second must also be 1. When the second value is 0, the administrator sets it:

set -euo pipefail
printf '%s\n' 'kernel.apparmor_restrict_unprivileged_unconfined=1' | \
  sudo tee /etc/sysctl.d/60-kibotos-apparmor.conf >/dev/null
sudo sysctl -p /etc/sysctl.d/60-kibotos-apparmor.conf

Notes:

• This setting stops all unprivileged processes from switching profiles on their own. Trusted programs that have a profile attached automatically when executed are not affected.
• If the kernel does not have this sysctl, install the distribution's security updates first.
• The package scripts do not modify global sysctls, and they never disable user namespace restrictions or the KiboVM sandbox to pass the check.

Profile loadingCopy Anchor Link

• The profile file is at /usr/share/kibotos/node/apparmor/kibotos-vmm and is not loaded at installation.
• Before starting, the node service checks the host conditions and loads the profile. If the conditions are not met or parsing fails, the service refuses to start.
• Stopping or restarting the node service does not unload the profile. It is removed only when the node package is uninstalled.
• When AppArmor is not enabled in the kernel, KiboVM's own sandbox is still used. When the kernel state cannot be determined, startup fails.

When storage.root is changedCopy Anchor Link

By default, the profile applies to /var/lib/kibotos/node/installations/sha256-*/kibovm. If you change storage.root in the configuration, add the new root directory to the local fragment maintained by the administrator:

sudo install -d -m 0755 /etc/apparmor.d/local
printf '%s\n' '@{KIBOTOS_NODE_ROOT}+=/srv/kibotos-node' | \
  sudo tee /etc/apparmor.d/local/kibotos-node-roots >/dev/null

These directories and their parents must be controlled by the administrator. They cannot be writable by ordinary users, and broad wildcards are not allowed. The local fragment is kept when the package is uninstalled and is reloaded the next time node starts.

ConfigurationCopy Anchor Link

Decide the following before step 1, where you write them into /etc/kibotos/node/config.toml. For every table and key, see node configuration. This section covers only what that page leaves to you.

Required settingsCopy Anchor Link

• Fill in CPU and memory according to the machine's actual capacity; node does not detect them automatically. Storage pool capacity is not in the configuration. Set it on ctrl after the node joins. See Setting pool capacity.
• Fill in the ctrl address and the externally advertised data-plane address. The advertised hostname must be among the certificate names declared when the node key was issued.
• After choosing firecracker or kibovm, publish a Platform that matches it.
• The storage directory and the socket directory default to /var/lib/kibotos/node and /run/kibotos/node. Their parent directories must allow the kibotos-vmm account to enter. The socket directory path must be short and must not be a symbolic link. When startup fails, the log names the path or device that does not qualify.
• Under [storage], declare at least one storage pool with [[storage.pool]]: a directory holding volumes/, retained-layers/, and a marker file, where Volumes and retained layers are kept. The pool's path can equal storage.root, as in the default example. When only one pool is declared, it is implicitly the default pool. installations/, runtime/, template-cache/, state/, pki/, network/, and machine-logs/ belong to no pool and always stay on the local disk under storage.root. machine-logs/ keeps the console and VMM logs of ended Instances. See Console and VMM logs. After declaring a pool, write the marker file with kibotos-node storage-pool init; node itself never creates it. See Storage pools on NFS.

Network prerequisitesCopy Anchor Link

• When guests need no network, delete the whole [host_network] table. Workspaces on both backends then use the deny-all policy, and VMs have no network interface. The node-to-guest connection goes over vsock and does not depend on network attachment.

• [host_network].address_families must be set explicitly to ["ipv4", "ipv6"], ["ipv4"], or ["ipv6"]. Configure address pools and uplink interfaces only for the enabled families. An IPv6-only example:

  Copy Text

  [host_network]
  address_families = ["ipv6"]
  ipv6_pool = "2001:db8::/48"
  
  [host_network.public_internet]
  ipv6_upstream_interface = "eth0"
  resolvers = ["2606:4700:4700::1111"]

  • The example pool is only for documentation and isolated tests. In a deployment, the IPv6 pool and uplink can use GUA or ULA, and you provide the return route to the node and the actual internet exit. node keeps the guest's IPv6 source address, and the upstream may translate it.
  • With IPv6 enabled, each VM gets a /128 address, forwarded through a link-local gateway and an exact host route. The upstream must route the pool to the node; giving the uplink interface an address is not enough.
  • For IPv4 only, set ["ipv4"] and configure only ipv4_pool, ipv4_upstream_interface, and IPv4 DNS servers.
  • A dual-stack deployment can configure DNS servers of a single family. The A and AAAA record types do not decide which family a DNS query travels over.
  • node advertises the enabled families with the capabilities kibotos.network.ipv4 and kibotos.network.ipv6. kibotos.service.public-internet means the uplink checks passed for every enabled family: when one fails, node withdraws this capability, and disabled families take no part.
  • A node with a network also advertises kibotos.network.destination-rules. ctrl places a Workspace whose policy has allow or deny lists only on such a node.
  • With IPv6 enabled, public-internet also allows the ICMPv6 types that neighbor discovery, path MTU discovery, and related error reporting need.
  • Before changing address families, clean up all network attachments with the original configuration and version, then start with the new configuration. A record the node cannot read makes startup fail.

• For Workspaces to use the public-internet policy, the host itself must be able to forward guest traffic. At startup, node checks the following two items and never modifies them:

  • Forwarding is on for each enabled address family: net.ipv4.ip_forward=1 and net.ipv6.conf.all.forwarding=1. Persist them in /etc/sysctl.d/. Do not rely on other software, such as Docker, turning them on at runtime:

    Copy Text

    printf '%s\n' 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/60-kibotos-forward.conf >/dev/null
    sudo sysctl -p /etc/sysctl.d/60-kibotos-forward.conf

  • Forward chains in the root namespace that do not belong to node have no drop policy.

  • If these are not met, node still starts, but it logs node offers no public internet egress with the reason, and this node does not accept Workspaces that need public internet egress. Fix them, then restart node.

• For Workspace policies to forward through a caller's SOCKS5 Egress, the following are needed on top of working public internet egress. At startup, node checks them by trial-installing the forwarding rules and the transparent listener in a temporary network namespace:

  • The kernel provides nftables TPROXY and socket matching (the nft_tproxy and nft_socket modules, or built in), transparent sockets (IP_TRANSPARENT), and policy routing by fwmark. Default distribution kernels meet these.
  • The system account kibotos-egress exists. The node package creates it. Without the package installation, create a system account with the same name yourself, and do not add it to any group.
  • The socket directory of egress processes is the socket directory's sibling with the -egress suffix, /run/kibotos/node-egress by default. Its parent directory must allow kibotos-egress to enter. The /run/kibotos installed by the package is 0711, which satisfies this. The directory itself is created by node, owned by kibotos-egress with mode 0700.
  • If any item is not met, node still starts and logs node offers no forwarding through an Egress with the reason. This node does not advertise kibotos.network.egress-socks5 and does not accept Workspaces with an Egress. Fix it, then restart node.

• For Workspaces to forward DNS queries to a caller's Resolver, the following are needed on top of working public internet egress:

  • The kibotos-egress account and the egress process socket directory, with the same requirements as the previous item. Kernel TPROXY and socket matching are requirements only for Egress; a Resolver does not need them.
  • At least one entry in resolvers of [host_network.public_internet].
  • If these are not met, the log records node offers no Resolver or node offers no forwarding through an Egress and no Resolver with the reason. This node does not advertise kibotos.network.resolver and does not accept Workspaces that reference a Resolver.

• Guest DNS is determined by resolvers:

  • A newly started Workspace sees a single nameserver, the slot's link DNS address. The address family of the first resolver determines its address family: with IPv4, it is the slot's IPv4 gateway address; with IPv6, it is fe80::1%eth0.
  • When the first resolver is an IPv6 address, Workspaces need a Platform published with v0.3.0 or later. Guests in earlier Platforms reject a nameserver with an interface identifier, and the Workspace fails to start.
  • Workspaces that do not reference a Resolver use only the first resolver; the later entries are not used for Workspace queries. They are used only by egress processes, in order, to resolve upstreams of a direct Resolver that are written as hostnames.
  • When the deployer restricts node's own outbound traffic on the host, the upstreams of a direct Resolver are restricted as well. See Egress processes.

• When the upstream network does no further address translation, destinations see the following source addresses. When the other side allows traffic by source address, allow the IPv4 address of the uplink interface and the whole ipv6_pool:

  • For IPv4, it is the address of node's uplink interface, shared by the Workspaces on this node.
  • For IPv6, it is the guest's own address, taken from ipv6_pool. A Workspace gets a new address each time it starts, and released addresses are later given to other Workspaces, so a single address cannot identify a particular Workspace.

Optional settingsCopy Anchor Link

• By default, guests on both backends cannot see the CPU virtualization extensions (Intel VMX, AMD SVM), so KVM cannot run nested virtual machines inside a Workspace. If you really need it, set nested_virtualization = true in [backend]. It affects only VMs started afterward. Once it is on, the nested virtualization path of the host kernel's KVM is reachable from every guest. This code has had several guest escape vulnerabilities, so keep the host kernel up to date.
• Storage pools can be on NFS. This is the prerequisite for a stopped Workspace to start on another node. For the conditions and costs, see Storage pools on NFS.
• To export node's own metrics, fill in the [telemetry] section. See Listener in Cluster and service metrics.

Data plane behind a reverse proxyCopy Anchor Link

When the data plane sits behind a reverse proxy, the two service connection paths have different requirements for the front layer:

File transfers and process output also require response buffering to be off. When browsers connect directly to the data plane, the front layer must present a publicly trusted certificate: browsers do not trust data-plane certificates issued by the internal CA.

1. Install and join the clusterCopy Anchor Link

Install the package, write the configuration as described in Configuration, initialize the storage pool, then join the cluster and start the service:

sudo apt-get install ./kibotos-node_1.2.3_amd64.deb
sudo install -o root -g root -m 0600 \
  /usr/share/doc/kibotos-node/node.example.toml /etc/kibotos/node/config.toml
sudoedit /etc/kibotos/node/config.toml
sudo kibotos-node storage-pool init --id node-a --path /var/lib/kibotos/node
sudo kibotos-node join --config /etc/kibotos/node/config.toml --key-file node-enrollment.key
rm node-enrollment.key
sudo systemctl enable --now kibotos-node.service
sudo journalctl -u kibotos-node.service --no-pager -n 50

--id and --path of storage-pool init must match id and path of [[storage.pool]] in the configuration. This step writes the marker file that proves the directory's identity, which node itself never creates. Without it, the pool is never available, and no Volume or Workspace can be created.

Use this node's ID as the id of its local pool, that is, the value of --node when the enrollment key was issued. The example above uses node-a; replace it with each node's own ID. Do not give the local pools of all nodes the same name: ctrl takes the first storage registered as the pool for that id, and when a later node declares a different directory with the same id, the pool is rejected. If it is the default pool, the node shows READY but cannot place any Workspace. Only pools shared by several nodes (for example, on NFS) use the same id in each node's configuration. See Storage pools on NFS.

2. Confirm the joinCopy Anchor Link

• On each host, check the binary version with version --json.

• The Operator runs kibo node get node-a --json and confirms that the state is ADMITTED and READY and that the version in the session is as expected.

• In kibo node get node-a, Clock offset should be on the order of milliseconds and Clock sync should be synchronized. Otherwise, see Time synchronization.

• In the Storage pools section of kibo node get node-a, the default pool should be available. When it shows unavailable, the reason follows. If the reason says it is different storage, the id of the local pool clashes with another node's. Because ctrl has never placed anything in the rejected pool, rename it with the following steps:

  Copy Text

  sudo systemctl stop kibotos-node.service
  sudoedit /etc/kibotos/node/config.toml   # change the local pool id to this node's ID
  sudo rm /var/lib/kibotos/node/.kibotos-storage-pool
  sudo kibotos-node storage-pool init --id node-a --path /var/lib/kibotos/node --adopt
  sudo systemctl start kibotos-node.service

  storage-pool init never overwrites an existing marker, so delete the old marker first. After starting, node has already created subdirectories in the pool directory, so add --adopt.

• Past session records cannot prove that the files on disk now and the running service are the same version. Also check MainPID/ExecStart in systemctl show and the service logs.

3. Set pool capacityCopy Anchor Link

After a node declares a pool for the first time, the Operator sets the pool's capacity on ctrl. Until then, the pool accepts no new Volumes or retained layers.

kibo storage-pool set-capacity node-a 200G
kibo storage-pool get node-a

• Set it once per pool. Nodes that join later and declare the same shared pool use the capacity already set.
• Set the capacity to the space the pool's storage is willing to commit. Later, grow or shrink it with the same command; no node restart is needed. For how it is calculated, see Pool capacity.

NotesCopy Anchor Link

• At startup, node verifies the VM runtime from /usr/lib/kibotos/node, then installs it to a location under the storage directory pinned by the code. The node never downloads a runtime.
• When runtime files are missing or corrupted, node fails before acquiring the owner lock and creating state. Reinstall the package; do not replace the files with those from another version.