Network troubleshooting
This page shows how to confirm that changes to the network policy, an Egress, or a Resolver are in effect, and how to tell causes apart when a connection or DNS query fails.
Confirming the policy is in effect
On an AVAILABLE Workspace, a successful policy change means the node has applied the new policy; see When changes take effect. Check what the node reports in these cases:
- The policy change failed or timed out.
- The Workspace was
STOPPEDwhen you changed it. The policy is applied when it next starts. - You updated an Egress or a Resolver. The update returns once it is stored, and each running Workspace that references it switches afterwards.
- Connections of an earlier revision are still within their grace period.
kibo workspace get shows what the node reports as in effect. Add --wide to also see the revisions:
kibo workspace get "$workspace_id"
kibo workspace get "$workspace_id" --wideFor a running Workspace, the output has these lines:
| Line | Contents |
|---|---|
| Network | The desired mode. When it is not yet confirmed as applied, followed by (not confirmed yet). When the applied policy is an older one, followed by (applied: <mode>, change pending). When connections of an older policy revision are still within their grace period, followed by connections of revision N still draining |
| Allow, Deny | The policy's destination rules, one entry per line |
| Egress, Via Egress | The Egress the policy references and its ranges, one per line |
| Egress applied | The Egress revision used by new connections for each Egress. When connections of an older revision are still within their grace period, that revision is added |
| Egress fault | The fault kind and start time; see Egress failures |
| Resolver | The Resolver the policy references and its path |
| Resolver applied | The Resolver revision used by new queries |
| Resolver fault | The fault kind and start time; see Resolver failures |
- The revisions of the desired and applied policies are in the Revisions section of
workspace get --wideand in the JSON output. - An Egress update is in effect when the revision on the
Egress appliedline matches the one fromkibo egress get. - A Resolver update is in effect when the revision on the
Resolver appliedline matches the one fromkibo resolver get.
For the status fields behind these lines, see Policy status.
Troubleshooting
| Symptom | Cause | Resolution |
|---|---|---|
| The connection fails immediately instead of timing out | The destination is in deny, or a direct destination is a private or special-purpose address | Check allow and deny. To reach an address in the caller's internal network, put it in the Egress ranges |
| Connections to destinations inside the Egress ranges are reset | The Egress failed, and connections do not fall back to direct | Check the Egress fault line or egress_observations |
DNS queries get SERVFAIL | The upstream of the referenced Resolver is unavailable | Check the Resolver fault line or resolver_observations. When the path goes via Egress, also check the Egress faults |
| DNS queries fail immediately | The policy is DENY_ALL | Change it to PUBLIC_INTERNET |
ping to the internet gets no reply | A gateway on the uplink translates only TCP and UDP, so it stops the Echo Request. Inside the guest this looks the same as a refusal by the node | Ask the deployer to confirm that the uplink forwards ICMP |
A change returns failed_precondition with reason set to CAPABILITY_UNAVAILABLE | The node running the Workspace does not support destination rules, Egress, or Resolver | Leave that part out, or ask the deployer to upgrade the node |
Referencing a Resolver returns failed_precondition with reason set to CONFLICT | The Workspace started before the upgrade and still uses its original DNS configuration | Stop and start the Workspace, then reference the Resolver again |
Exit code 14 from kibo | The server does not advertise the corresponding capability | See Capability names |
Exit code 13 from kibo | Someone else changed the object in the meantime, or the Egress or Resolver to delete is still referenced | Read it again before deciding. Remove references before deleting |
Egress failures
When an Egress fails, the guest only sees its connections reset, with no fallback to direct. Once the endpoint recovers, the next connection via Egress succeeds and the fault record clears. The diagram below shows both:
The cause is recorded in the Workspace, one entry for each failing Egress. kibo workspace get shows it on the Egress fault line, with the fault kind and when it started:
| Kind | Meaning |
|---|---|
ENDPOINT_UNREACHABLE | The endpoint cannot be reached, or all endpoint addresses are refused |
TLS_VERIFICATION_FAILED | The certificate is not trusted, or does not match the verification name |
HANDSHAKE_FAILED | SOCKS5 negotiation failed, the reply was invalid, or it timed out |
AUTHENTICATION_FAILED | The endpoint explicitly rejected the credentials |
UDP_ASSOCIATION_FAILED | The endpoint rejected the UDP association, the relay address it replied with was refused, or sending to the relay returned an unreachable error |
- It is an observation, not a state. While there is a fault, the Workspace is still
AVAILABLE, and reading it does not contact the node. The content stays the same while the fault persists and clears after the next successful connection via Egress. - The absence of
AUTHENTICATION_FAILEDdoes not mean the credentials are correct. When the endpoint does not answer wrong credentials, the observation isENDPOINT_UNREACHABLEorHANDSHAKE_FAILED. - A proxy reporting that the destination is unreachable or refused the connection is the result of a single connection and produces no observation.
- The kind will gain new values in the future. Treat an unknown value as an exit fault, not as no fault. In
kiboit appears asUNKNOWN_FAULT(n). - How soon you see it: from the start of a fault until ctrl can read it usually takes less than 2 minutes.
Resolver failures
When the upstream fails, the guest's queries get SERVFAIL, with no fallback to the deployment DNS servers. The cause is recorded in the Workspace, one entry for each failing Resolver entry of the policy. kibo workspace get shows it on the Resolver fault line, with the fault kind and when it started:
| Kind | Meaning |
|---|---|
UPSTREAM_UNREACHABLE | The upstream cannot be reached, or with a direct path all upstream addresses are refused |
UPSTREAM_TIMEOUT | No answer within the time limit |
TLS_VERIFICATION_FAILED | The certificate of a tls or https upstream is not trusted, or does not match the host |
HTTP_ERROR_STATUS | An https upstream returned a non-2xx status |
NXDOMAINandSERVFAILanswered normally by the upstream are DNS results and produce no observation.- Via Egress, faults of the Egress itself are recorded as Egress faults and are not repeated here.
- The other rules are the same as in Egress failures. The fault does not change the Workspace state, reading it does not contact the node, and the content stays the same while the fault persists and clears after the next successful query. The kind will gain new values in the future; treat an unknown value as an upstream fault.
With the API
Policy status
The Workspace's network_policy_status is what the node reports as in effect. It is empty when the policy was never confirmed in effect or is no longer in effect:
| Field | Meaning |
|---|---|
applied_revision, mode | The policy revision and mode the node is enforcing |
draining_revision | When connections established under an earlier policy revision are still within their grace period, the earliest of those revisions; 0 when there are none |
egresses | One entry for each Egress the policy references: egress_id; applied_revision is the Egress revision used by new connections; draining_revision is the earliest Egress revision used by connections still within their grace period, 0 when there are none |
resolvers | One entry for each entry in the policy's resolvers, matched by resolver_id and path; applied_revision is the Resolver revision used by new queries |
| What to confirm | How to tell |
|---|---|
| The policy change is in effect | applied_revision equals the policy revision |
| Sent but not yet confirmed | applied_revision is less than the policy revision, and the Workspace is not FAILED |
| The Egress update is in effect | The applied_revision of this Egress in egresses equals the Egress revision |
| The Resolver update is in effect | The applied_revision of the corresponding entry in resolvers equals the Resolver revision |
| All old connections are closed | draining_revision and the draining_revision of every entry in egresses are all 0 |
Fault observations
Egress faults are in the Workspace's egress_observations, one entry for each failing Egress, empty when there are no faults. Resolver faults are in resolver_observations, one entry for each failing list entry, empty when there are no faults. The Egress fault and Resolver fault lines of kibo workspace get come from these fields.
| Field | Meaning |
|---|---|
egress_id (Egress); resolver_id, path (Resolver) | The failing Egress, or the failing list entry |
kind | The fault kind; see the tables in Egress failures and Resolver failures |
failing_since | When this run of consecutive failures started, by the node's clock |
observed_at | When ctrl recorded this report, by ctrl's clock |
Next steps
- Network policy: when a policy change takes effect.
- Egress: how an Egress update switches connections.
- DNS and Resolvers: where the guest's DNS queries go.
DNS and Resolvers
This page explains where the guest's DNS queries go, and how to hand them to an upstream the caller chooses with a Resolver.
Volumes, export, and restore
A Volume stores a Workspace's data and exists independently of any Workspace. This page shows how to choose a Volume when you create a Workspace, create and resize Volumes, rebind them, and export a Workspace's data to restore it as a new Workspace.