Egress

An EgressAn outbound SOCKS5 proxy that the caller runs. When a Workspace's network policy references it, TCP connections to the given ranges leave from the caller's network. is a SOCKS5 endpoint that the caller runs. Once a Workspace's network policy references it, TCP connections to the given ranges leave from the caller's network. This page shows how to create, reference, update, and delete an Egress, and when connections go out through it.

PrerequisitesCopy Anchor Link

The server must support Egress. Without it, kibo egress, and kibo workspace network commands with Egress flags or --drain-seconds, exit with 14 and send no request. For the capability name, see With the API.

Creating an EgressCopy Anchor Link

The commands below create an Egress with a label, show it, and list the Egresses with that label:

egress_id=$(kibo egress create --address proxy.example.com:1080 --username alice --password-file ./proxy-password --label team=infra -q)
kibo egress get "$egress_id"
kibo egress list -l team=infra

Create and update describe the complete endpoint:

• The endpoint is checked once at create and update. When the IP address falls in a fixed special-purpose range such as private, loopback, multicast, or documentation, or when the host name resolves once on ctrl to a result that includes such an address, the request returns invalid_argument. This step only catches mistakes early: the node resolves the name again on every dial, removes disallowed addresses, and connects only to the rest.
• Each Egress has its own revision. The server assigns it: 1 at creation, incremented when the endpoint settings change. It is independent of the revision of policies that reference the Egress.

PasswordsCopy Anchor Link

kibo has no flag that takes the password value directly, so the password stays out of shell history and the process list.

• Read it from standard input with --password-stdin, or from a file with --password-file.
• When you give only --username, kibo prompts for the password on the terminal without echo. Without a terminal, it exits with 2.
• One trailing newline at the end of standard input or the file is removed. All other characters are part of the password.
• The password is write-only. No read, list, command output, or error message returns it. The user name is returned as usual, and the details only say whether credentials are configured.

Routing connections through an EgressCopy Anchor Link

A public-internet policy can reference an Egress and list the destination ranges that go out through it. New TCP connections whose destination falls in these ranges are forwarded via Egress, and everything else goes direct as usual. In the diagram below, the ranges contain only 10.0.0.0/8:

The commands below reference the Egress for all IPv4 and IPv6 destinations, add and then remove the range 10.0.0.0/8, and finally remove the reference:

kibo workspace network set "$workspace_id" public-internet --egress "$egress_id" --egress-destination 0.0.0.0/0 --egress-destination ::/0
kibo workspace network add "$workspace_id" --egress-destination 10.0.0.0/8
kibo workspace network remove "$workspace_id" --egress-destination 10.0.0.0/8
kibo workspace network remove "$workspace_id" --egress "$egress_id"

When add and remove edit --allow and --deny, they keep the existing Egress entry. When the policy they read already references an Egress, add and remove also require the Egress capability.

• A policy references at most one Egress, with 1 to 256 ranges. More than one entry, empty ranges or more than 256, non-canonical entries, and DENY_ALL with an Egress all return invalid_argument, and the desired policy does not change.
• Ranges are written the same way as allow and deny. Each applies only to its own address family.
• The Egress must exist. When it does not exist or is deleted, the request returns not_found.
• Only nodes that support Egress receive it. When no node that advertises the exit capability is available at creation or start, the request returns failed_precondition with reason set to CAPABILITY_UNAVAILABLE. The Workspace never runs with a policy that drops the Egress. When you add an Egress to a running Workspace whose node does not support it, the request returns the same error and the desired policy does not change.
• Changing only the order of ranges is a content change and increments the revision, but it does not change where any connection goes.

Where connections goCopy Anchor Link

New TCP connections whose destination is inside the ranges go to the forwarding service on the node, which sends a CONNECT to the SOCKS5 endpoint for the destination address the guest originally wanted to reach:

A connection is matched in this order:

1. To port 53 of the link DNS address: handled by Link DNS, not forwarded via Egress by destination.
2. In deny and not in allow: refused. Private-network entries apply as well.
3. New TCP whose destination is inside the ranges: forwarded via Egress, without the refusal of private and special-purpose addresses.
4. ping whose destination is inside the ranges: refused immediately with ICMP administratively prohibited.
5. Everything else: goes direct according to Destination matching.

The ranges can include private and platform ranges. Connections via Egress leave from the caller's network and are not subject to the private-address refusal that applies to direct connections. The diagram below shows that the same address is still refused when sent direct:

When the destination is inside the ranges, each kind of traffic goes as follows:

A failed exit does not fall back to direct. When the endpoint cannot be reached, TLS verification fails, or SOCKS5 negotiation or authentication fails, the guest's connection is reset. For the causes, see Egress failures.

Changing where connections goCopy Anchor Link

Changes that alter where connections go (from direct to via Egress, the reverse, or replacing the ranges) do not cut off existing connections immediately:

• New connections are established under the new policy immediately. Old connections keep transferring on their original path and close when the grace period ends. During this time, draining_revision is the old policy revision.
• The grace period is --drain-seconds of this change, from 0 to 3600, default 600. 0 means close immediately at the switch.
• Another change within the grace period does not extend an earlier deadline.
• Connections that the new mode, allow, or deny no longer allow are terminated immediately, without waiting for the grace period.

Updating an EgressCopy Anchor Link

An update returns once ctrl saves it, without waiting for Workspaces. Running Workspaces that reference the Egress then switch new connections to the new revision, and existing connections close according to this update's grace period. The diagram below shows the switch:

The command below updates the Egress and closes connections on the old revision 60 seconds after the switch:

kibo egress update "$egress_id" --address proxy.example.com:1080 --username alice --password-file ./proxy-password --drain-seconds 60

• An update replaces the whole Egress. Flags you leave out of kibo egress update are empty. The password cannot be read back, so give it again on every update.
• Concurrent updates are refused. Without --expected-revision, kibo reads the current revision first and then submits, and exits with 13 if someone else changed the Egress in the meantime. When a script replays, pass --command-id and --expected-revision together.
• An update with the same content succeeds, and the revision does not change.
• The grace period is a parameter of this update. --drain-seconds is from 0 to 3600, default 600. 0 means close immediately at the switch. It is not saved with the Egress.
• When the node cannot get the new configuration for a while, it keeps the original revision. The update is retried by ctrl, and the Workspace state does not change.
• UDP cannot be turned off while a udp Resolver still goes out through the Egress. Such an update returns failed_precondition (CONFLICT), kibo exits with 13, and the content does not change.
• Compare the two revisions to check whether the update is in effect. When this Egress's applied revision in the Workspace status equals the Egress revision, new connections already use the new configuration; see Confirming the policy is in effect.

Deleting an EgressCopy Anchor Link

A referenced Egress cannot be deleted, so the commands below remove the reference first and then delete the Egress:

kibo workspace network remove "$workspace_id" --egress "$egress_id"
kibo egress delete "$egress_id"

• While the policy of any Workspace that is not deleted references the Egress, deletion returns failed_precondition (CONFLICT) and kibo exits with 13.
• After deletion, connections still using the Egress within their grace period close immediately.
• A deleted Egress returns not_found for every read and reference.

With the APICopy Anchor Link

Egresses are managed with EgressService, and every write operation carries a command ID; see Command IDs. The flags in Creating an Egress map to these fields:

• spec.socks5 is the endpoint protocol. Only SOCKS5 exists for now. Create and update without a protocol return invalid_argument.
• revision is assigned by the server: 1 at creation, incremented when spec changes.
• An update must carry an expected revision. If that does not match the current revision, the update returns aborted (reason set to CONFLICT) and the content does not change.
• In the network policy, the Egress is egresses. Each entry has an egress_id and its destinations.
• The grace period of a policy change or an Egress update is drain_seconds.

A server that supports Egress advertises the capability workspace.network_policy.egress.socks5. In the future, each exit protocol will have its own capability name, and callers use it to tell which protocols a deployment supports.

Next stepsCopy Anchor Link

• DNS and Resolvers: send DNS queries through the Egress with a Resolver.
• Network troubleshooting: confirm an update is in effect, and find out why connections via Egress are reset.
• Network policy: destination rules and when a policy change takes effect.