DNS and Resolvers

This page explains where the guest's DNS queries go, and how to hand them to an upstream the caller chooses with a Resolver.

Link DNS

The guest sends DNS queries only to one address on its own link, and the policy decides where they go from there. The diagram below shows three destinations in turn: no Resolver, a Resolver with path direct, and a Resolver with a path via Egress.

No Resolver: deployment DNS serversResolver, path directResolver, path via Egress

Item	Rule
Link DNS address	The guest gateway address in the address family of the first deployment DNS server; for IPv6 it is `fe80::1`
`/etc/resolv.conf`	Only one `nameserver`: the link DNS address
Port	53, UDP and TCP

Policy	Where queries go
`DENY_ALL`	Fail immediately: UDP gets port unreachable, TCP is reset, and no packet leaves the Workspace
`PUBLIC_INTERNET`, no Resolver	Forwarded to the first deployment DNS server in the same family as the link DNS address; the others are not used
`PUBLIC_INTERNET`, Resolver referenced	Forwarded to the caller's upstream according to the Resolver's protocol and path; see Referencing a Resolver in the policy

allow, deny, and the Egress ranges do not apply to these queries. Only a Resolver can send queries via Egress.
Changing the policy does not change the guest's DNS configuration. Changing the mode, or referencing, replacing, or removing a Resolver, leaves /etc/resolv.conf unchanged.
UDP that the guest sends directly to a deployment DNS server is refused. Only queries to the link DNS address count as DNS.
Workspaces started before an upgrade keep their original DNS configuration until they restart. Their /etc/resolv.conf still lists the deployment DNS servers, and they cannot reference a Resolver: such a change returns failed_precondition with reason set to CONFLICT, and neither the desired nor the applied policy changes. After a stop and start, the Workspace switches to the link DNS address and can reference a Resolver.

Resolver

A Resolver is a DNS upstream the caller chooses. The server must support Resolvers. Without that support, kibo resolver and kibo workspace network commands with a Resolver exit with 14 and send no request. For the capability name, see With the API.

Creating a Resolver

The commands below create a DNS over HTTPS Resolver with a label, show it, and list the Resolvers with that label:

resolver_id=$(kibo resolver create --https https://dns.example.com/dns-query --label team=infra -q)
kibo resolver get "$resolver_id"
kibo resolver list -l team=infra

Create and update must choose one protocol. Without one, they return invalid_argument:

Flag	Rules
`--udp`	An IP address or `IP:port`, port 53 by default; host names are not accepted. Queries the guest sends over TCP go to the same address over TCP
`--tcp`	An IP address or `IP:port`, port 53 by default; host names are not accepted
`--tls`	DNS over TLS. `host` or `host:port`, where host can be a host name; port 853 by default
`--https`	DNS over HTTPS. `https://host[:port]/path`, port 443 by default. The path is required, and the URL is at most 2048 bytes; user info, query strings, and fragments are not accepted. Queries are sent with `POST`
`--label`, `--annotation`	Given only at creation and modified separately afterwards; see Labels and annotations

IP addresses are written the same way as allow and deny entries. An IPv6 address followed by a port goes in square brackets, for example [2001:db8::53]:53.
Malformed addresses are rejected by kibo before the request is sent, with exit code 2.
Reads return exactly what was written, without filling in default ports.
tls and https verify the certificate with the system trust store, using the host as the name. When the host is an IP address, the certificate must include that IP.
Each Resolver has its own revision. The server assigns it: 1 at creation, incremented when the upstream settings change. It is independent of the revision of policies that reference the Resolver.

Warning

A Resolver has no credentials, and reads return all of its content. The URL path is visible to other callers in the same deployment who can read Resolvers, so do not put access tokens in the path.

Referencing a Resolver in the policy

Once a public-internet policy references a Resolver, the guest's DNS queries are forwarded to it. The command below references a Resolver whose queries go directly to the upstream:

kibo workspace network set "$workspace_id" public-internet --resolver "$resolver_id" --resolver-path direct

In a policy that has no Resolver yet, the command below adds an Egress and a Resolver that goes through it in one change:

kibo workspace network add "$workspace_id" --egress "$egress_id" --egress-destination 10.0.0.0/8 --resolver "$resolver_id" --resolver-path "egress:$egress_id"

The command below removes the reference:

kibo workspace network remove "$workspace_id" --resolver "$resolver_id"

A policy references at most one Resolver, and the Resolver must exist. When it does not exist or is deleted, the request returns not_found. The path decides how the connection to the upstream leaves, and has no default:

`--resolver-path`	Connection to the upstream
`direct`	Sent directly to the upstream from the node's uplink address
`egress:<egress-id>`	Sent to the upstream through this Egress. It must be the Egress referenced by the same policy, regardless of whether the ranges cover the upstream. The upstream can be in the caller's internal network. A `udp` Resolver also requires this Egress to have UDP turned on

Command	Effect on the Resolver
`network set`, `workspace create`	`--resolver` with `--resolver-path` writes one entry. Without `--resolver`, the policy references no Resolver
`network add`	When the policy has no Resolver yet, references it. When it has one, only the same ID and path are allowed. To switch Resolvers or change the path, use `set`
`network remove`	`--resolver` removes the reference, and queries go back to the deployment DNS servers. The Egress named by the Resolver path can only be removed together with `--resolver`

When the policy they read already references a Resolver, add and remove also require the Resolver capability.

When the upstream is unavailable, the guest gets SERVFAIL and there is no fallback to the deployment DNS servers. Forwarded queries have these time limits and caps:

Item	Rule
One query	At most 4 seconds from receipt to answer
Resend for a `udp` upstream	Resent once when there is no answer within 2 seconds
Concurrent queries	At most 256 per Workspace; queries beyond that get `SERVFAIL` immediately
Answers the guest receives over UDP	Capped at the size the query declares in EDNS, or 512 bytes without EDNS. Larger answers come back with the TC bit set, and the guest retries over TCP

A direct upstream is checked once when referenced. When the IP address falls in a fixed special-purpose range such as private, loopback, multicast, or documentation, or when the host name resolves once on ctrl to a result that includes such an address, the request returns invalid_argument. Updating a Resolver referenced with a direct path to such an upstream also returns invalid_argument.
The check only catches mistakes early. As with the Egress endpoint, when dialing, the node resolves the host name with the deployment DNS servers, removes disallowed addresses, and connects only to the rest. If all are removed, that query gets SERVFAIL.
Via Egress, the node neither resolves nor checks the upstream. The host name is handed to the endpoint to resolve. A udp upstream goes through UDP ASSOCIATE, and the others through CONNECT. The UDP relay address the endpoint replies with is checked by the same rules as the endpoint address.
After the Egress is updated, queries through it switch to the new revision along with TCP. The grace period applies only to TCP connections.
Invalid combinations return invalid_argument, and the desired policy does not change: DENY_ALL with a Resolver, more than one entry, no path, a path naming an Egress that is not in the policy, a udp Resolver through an Egress without UDP turned on, and a direct upstream in a special-purpose range.
Only nodes that support Resolvers receive them. When no node that advertises the Resolver capability is available at creation or start, the request returns failed_precondition with reason set to CAPABILITY_UNAVAILABLE. The Workspace never runs with a policy that drops the Resolver. When you add a Resolver to a running Workspace whose node does not support it, the request returns the same error and the desired policy does not change.
Referencing, replacing, or removing a Resolver is a content change and increments the revision. The request returns after the new configuration takes effect, and new queries after that are sent according to it.

Updating and deleting a Resolver

The commands below switch the Resolver to DNS over TLS, then remove the reference and delete it:

kibo resolver update "$resolver_id" --tls dns.example.com
kibo workspace network remove "$workspace_id" --resolver "$resolver_id"
kibo resolver delete "$resolver_id"

An update replaces the whole Resolver. When the content is the same, the update succeeds and the revision does not change.
Concurrent updates are refused. Without --expected-revision, kibo reads the current revision first and then submits, and exits with 13 if someone else changed the Resolver in the meantime. When a script replays, pass --command-id and --expected-revision together.
An update returns once saved, without waiting for Workspaces. Running Workspaces that reference the Resolver then switch new queries to the new revision, usually within 30 seconds. Queries already sent complete with the original configuration.
Compare the two revisions to check whether the update is in effect. When the applied revision of the corresponding entry in the Workspace status equals the Resolver revision, the update is in effect; see Confirming the policy is in effect.
While referenced with a direct path, a Resolver cannot be changed to a special-purpose address. See the check rules in Referencing a Resolver in the policy.
A referenced Resolver cannot be deleted. While the policy of any Workspace that is not deleted references it, deletion returns failed_precondition (CONFLICT) and kibo exits with 13. Remove the references first.
A deleted Resolver returns not_found for every read and reference.

With the API

Resolvers are managed with ResolverService, and every write operation carries a command ID; see Command IDs. The flags in Creating a Resolver map to these fields:

kibo flag	API field
`--udp`	`udp.address`
`--tcp`	`tcp.address`
`--tls`	`tls.address`
`--https`	`https.url`
`--label`, `--annotation`	`labels`, `annotations`

revision is assigned by the server: 1 at creation, incremented when spec changes.
An update must carry an expected revision. If that does not match the current revision, the update returns aborted (CONFLICT) and the content does not change.
In the network policy, the Resolver is resolvers. It has at most one entry, and each entry names a Resolver with resolver_id and its path. The path is path.direct for direct and path.egress_id for a path via Egress.
Whether an update is in effect is the applied_revision of the corresponding entry in the Workspace status.

A server that supports Resolvers advertises the capability workspace.network_policy.resolver.

Warning

Older servers silently drop resolvers. Confirm that the capability name exists before writing it.

Next steps

Egress: create the Egress that a Resolver path goes through.
Network troubleshooting: find out why queries get SERVFAIL.
Network policy: destination rules and when a policy change takes effect.

DNS and Resolvers

Link DNSCopy Anchor Link

ResolverCopy Anchor Link

Creating a ResolverCopy Anchor Link

Referencing a Resolver in the policyCopy Anchor Link

Updating and deleting a ResolverCopy Anchor Link

With the APICopy Anchor Link

Next stepsCopy Anchor Link