Nodes

A node is a machine that joins your organization and keeps one connection open to Loomplane. It dials out and opens no port, so a laptop behind a router and a server behind a firewall join the same way.

Join a machineCopy Anchor Link

Linux is supported. macOS works for development, and Windows is refused. On Linux the join installs a systemd service, so it needs root.

If you already ran loomctl login, run this on the machine:

sudo loomctl node join

node join uses your saved login (or LOOMPLANE_URL and LOOMPLANE_API_KEY) to mint a single-use join token, then spends it. Only an owner's or admin's write key, or a signed-in session, can mint one. Your API key goes to Loomplane to make the token and nowhere else, and the machine never stores it.

Root usually has no login of its own, and sudo drops LOOMPLANE_* from the environment. So when root has nothing saved, sudo loomctl node join reads the login of the user who ran sudo. It only does that if the file belongs to that user and nobody else can write to it. On a Mac, run loomctl node join without sudo.

The machine may not have loomctl yet. Add a node in the dashboard (Nodes, Add node), or mint a token somewhere you are logged in, and run the command it gives you on the new machine:

curl -fsSL https://loomplane.example.com/install/node.sh | LOOMPLANE_JOIN_TOKEN=lpj_... sh -s -- --allow probe,docker

Use your own instance's address. The script downloads the loomctl release your Loomplane runs, checks it, installs it into /usr/local/bin and runs loomctl node join with the flags you gave it. The token rides in LOOMPLANE_JOIN_TOKEN and not on the command line, so other users on the machine can't read it from the process list while the script runs. On Linux the script asks sudo to keep that variable with --preserve-env. If your sudoers rules won't allow it, run the script as root. The --allow list is what you chose when you made the token.

loomplane node join is the same command from the server binary. It needs either --token, or --url plus --key.

What you can passCopy Anchor Link

If you pass a token, mint it with --cloud, --builder or --edge already set, because the token carries them. The dashboard's Add node panel has a Cloud picker and a Dedicated builder checkbox for the same thing.

Check the result with sudo loomctl node status. It shows the node ID, the key fingerprint, the allowed endpoints and whether the service is running.

Tokens nobody usedCopy Anchor Link

A token you made and never ran still works until it expires, an hour at most. You'll find the open ones at the bottom of the Nodes page, under Unused join tokens, with who made each one, the API key they used if they used one, what it's for and how long it has left. Cancel one there and no machine can join with it.

The API has the same two calls: GET /api/v1/nodes/join-tokens lists them and DELETE /api/v1/nodes/join-tokens/{id} cancels one. Neither ever returns the token itself. You can list and cancel tokens if you can make them, so an owner or an admin, from a session or a write key. An instance edge token takes a signed-in platform admin to cancel, the same as it takes one to make. Both calls land in the audit log, as node.join_token.listed and node.join_token.cancelled.

The node's key and its stateCopy Anchor Link

The node creates its own Ed25519 key when it joins and keeps it in its state directory, /var/lib/loomplane-node on Linux. Loomplane learns the public half. The private half stays on the machine, in a directory owned by the service user, which is why status, allow and leave want sudo.

The join request is signed with the new key, over the token, the node's name and the public key. Loomplane checks that signature before it even looks at the token, and refuses the join without spending it if the signature is wrong. That's what stops someone who holds a token from joining with another machine's public key and keeping the real machine out.

Every join makes a fresh key. If the state directory still holds one from an earlier node, after a leave or a revoke, join moves it aside to identity.key.revoked-<time> and starts over. Loomplane refuses a key it has seen before, revoked nodes included, so a key that may have been stolen can't come back as a new node.

The state directory holds three files that matter:

The service runs /usr/local/lib/loomplane-node/loomctl, a copy that belongs to root. The service user can read allow.json and run that binary, and it can't change either. On macOS all of it belongs to you and the binary sits in the state directory, since the node runs as you there anyway.

Join refuses a directory that already holds files that aren't a node's own, so it can't take over /var/lib. Pass --state-dir an empty absolute path to keep a second node somewhere else.

What Loomplane may useCopy Anchor Link

Each node has an allowlist of endpoints. There are five names:

probe and docker do the work today, and dial lets Loomplane reach the node's own loopback and the containers it runs on Loomplane's networks, nothing else. kube and firecracker are accepted and stored, so you won't have to answer again when they arrive.

An interactive node join shows what it found and asks whether to allow it. Say anything but yes and it allows nothing. Without a terminal, such as a script, a cloud-init run or the curl | sh line, it allows nothing unless you pass --allow. The command the dashboard gives you passes it for you:

sudo loomctl node join --token lpj_... --allow probe,docker

Loomplane can only narrow the list. If you remove docker in the dashboard, the node drops it. No message from the control plane can add an endpoint. Widening happens on the machine, as root on Linux:

sudo loomctl node allow docker

That writes allow.json first and then tells the running node. The node serves only what both allow.json and node.json allow, so code running as the service user can't widen it. The exception is docker, which is covered below. If allow.json is missing, belongs to anyone else, or can be written by other users, the node serves nothing and logs why. Run node allow again to fix it.

The ingress mode is the one setting that comes from the dashboard and could open something up. It decides whether public traffic reaches the node. Today the only mode that does is direct, covered under Domains on a node.

Deploy to a nodeCopy Anchor Link

Join the node with docker allowed and a Cloud, and Loomplane connects its Docker as a server in that Cloud. Most nodes adopt on their own. One that needs a look first shows up on the Cloud's Changes tab with the reason and an Adopt button. The reasons are ports 80 and 443 already in use on the machine, an architecture that doesn't match the Cloud's pools, a server with that name already in the Cloud, or a token whose minter can no longer manage Clouds (or whose API key was revoked or expired). Adopt goes through the same reviewed steps as any server you connect, so you see what changes before it does.

After that the node is an ordinary server target. A pool can use it, a service can place on it, and the dashboard marks it with a Node badge and says it was connected by its owner. You can disconnect it from the Cloud, but you can't delete it, because Loomplane didn't create the machine.

The same Docker daemon can't be connected twice. If it is already in a Cloud over SSH, the node's connection is refused until you disconnect the SSH one, and the other way round. Two connections would mean two fence journals for one daemon, and fencing would stop meaning anything.

Podman works through its Docker-compatible socket. If its API leaves out something Loomplane needs to identify the daemon, the check names what's missing instead of passing. Hosts on cgroup v1 don't report their CPU and memory limits, so their capacity shows as unknown.

A deploy to a node that is offline waits for it for about two minutes. If the node hasn't come back by then, the deploy fails with node NAME is offline since TIME, and you can run it again once the node is back.

Builds on a nodeCopy Anchor Link

Join a node with --builder dedicated and it builds source for the Cloud it joined:

sudo loomctl node join --cloud cloud_... --builder dedicated --allow docker,probe

Use a machine that only builds. Untrusted source builds on a dedicated builder, so keep your services somewhere else.

Domains on a nodeCopy Anchor Link

A node serves a domain directly when it has a public address that answers on port 80. Loomplane confirms that from the outside before DNS points at the address, by fetching a one-time token through the node's own proxy. Until the check passes, the route keeps serving where it was. The node's page shows the address, when it was confirmed, and why not if it wasn't. A check that found nothing runs again every five minutes or so, and straight away when you set the node's ingress mode to direct.

If the node's address changes, Loomplane notices and checks the new one. In a zone Loomplane manages, the record moves once the new address answers. In an external zone, the route shows you the record to set, and you set it. Either way DNS never points at an address that hasn't answered.

A node behind NAT has no address to confirm, so it can't take a direct domain. Relaying traffic to it is the next piece.

Drain a nodeCopy Anchor Link

Draining stops new placements on the node's server. What already runs there keeps running until you move it, each with its own reviewed move.

loomctl node drain NAME

It lists what runs on the node first, and warns about any database with no backup policy: if the machine is lost, so is that data. It then asks. Pass --yes to skip the question. The Drain button on the node's page does the same, with the same review. A draining node shows as Draining.

CommandsCopy Anchor Link

loomctl node and loomplane node have the same subcommands.

A node that Loomplane revoked stops itself and exits with code 77. That holds whether it was connected at the time or finds out at its next connect, so a machine that was off when you revoked it stops when it comes back. A service manager that restarts on failure should treat 77 as final. The systemd unit does.

Leaving a nodeCopy Anchor Link

sudo loomctl node leave removes the node and leaves Docker alone. The containers Loomplane started keep running. To clean up as you go:

sudo loomctl node leave --purge

--purge also removes the containers and networks Loomplane made on the machine's Docker. Volumes stay, because they hold data, and the command prints their names so you can delete them yourself with docker volume rm.

sudo loomctl node leave --purge-volumes

--purge-volumes deletes the volumes too, and everything in them. It asks you to type the node's name first. In a script, pass --yes-delete-volumes=NAME with the name instead. A wrong name changes nothing.

If a deploy says the fence couldn't be establishedCopy Anchor Link

Docker requests from Loomplane run under a fence, so two attempts can't act on one daemon at once. The node keeps the fence's journal in its state directory, under fence/<hash>/state. If the node drops in the middle of a change, the journal stays marked dirty, and the next deploy reports that the runtime effect fence couldn't establish quiescence.

When the mark names an exec, a database readiness check for instance, the next deploy asks Docker about that exec and clears the mark itself once it has finished. A mark that doesn't say what was in flight waits for you. Review it with the node's connection ID:

loomctl fence-repair conn_...

You get the dirty mark, a review digest and every exec still running in a Loomplane container. Nothing changes yet. To clear the mark, pass the digest back:

loomctl fence-repair conn_... --yes --review <digest>

The repair keeps the fence's epochs, so an older attempt still can't write. It's refused while another Loomplane request holds the journal's lock, while an exec in a Loomplane container is still running, or when the journal has changed since you reviewed it. Wait for whatever is running, then review again. A node older than this release can't be reviewed at all, so update it first.

If the node dropped in the middle of a container change, look at the containers on the machine before you repair, because the change may be half done. Never delete the state file: that resets the epochs, and the fence stops protecting you.

Releases and updatesCopy Anchor Link

Releases are signed with cosign keyless, by the Release workflow of github.com/xraph/loomplane. node.sh checks them one of two ways:

• With cosign 2.4 or newer installed, it verifies the signature on the release checksums itself and then checks the archive against them. This doesn't trust your Loomplane at all.
• Without cosign, it checks the archive against a checksum your Loomplane already verified against the same signature. In that case Loomplane is vouching for the sum, and the script says so. Install cosign if you want to take Loomplane out of the chain.

With neither, the script stops. Set LOOMPLANE_SKIP_SIGNATURE=1 to install anyway. The install goes ahead unverified and prints a warning saying so.

To move a running node to another release:

sudo loomctl node update --release v1.9.0

The update downloads the release, verifies it, and only then replaces the binary. It restarts the service and watches it for five seconds. If the service won't start on the new binary, or stops within those five seconds, the update puts the old one back and starts that. A binary that fails later than that isn't caught, so check node status after an update.

node verify-release runs the same check by hand, when you've downloaded the archive, binaries-checksums.txt and its .sigstore.json bundle yourself.

LimitsCopy Anchor Link

Loomplane counts what reaches the node endpoints, so a script gone wrong on one address, or one misbehaving node, can't flood it.

An address over its limit gets a 429 and can try again after a minute. The count lives in the database, so every replica shares it. Each node asks for a challenge every time it reconnects, and all the machines behind one NAT share one address, so if you run more than about a hundred nodes behind a single address, raise nodes.challenge_rate_limit or they'll take a few minutes to all come back after a restart. Set either one to 0 to turn it off.

A connected node only sends a heartbeat every 15 seconds and a few messages when it connects, so it never gets near its limit. One that goes over is disconnected, with a warning in the Loomplane log naming the node, and nothing it sent past the limit is stored. It reconnects on its own.

Several replicasCopy Anchor Link

If you run more than one Loomplane replica, a node is connected to only one of them. A request for that node reaches the right replica over a hop between replicas. Set nodes.advertise_url on each replica to the address the others use to reach it. A single replica needs nothing.

Leave it out with several replicas and nothing fails loudly. Each replica only sees the nodes connected to it and reports the rest offline, and a revoke reaches a node on another replica through its next heartbeat, up to 15 seconds later, not at once. A replica with no advertise URL logs this at startup, and warns the first time it meets a node it can't reach.

Hops need https. On a private network you can allow plain http with nodes.hop_insecure. If your replicas use certificates from a private CA, point nodes.hop_ca_file at its PEM file. Configuration has the variables.

Cutting a node offCopy Anchor Link

There are three ways, and each cuts the node off the same way:

• Revoke it in the dashboard.
• Run loomctl node revoke NAME from anywhere you're logged in.
• Run sudo loomctl node leave on the machine.

The node can't reconnect afterwards. A node that was off at the time learns it was revoked at its next connect and stops. If you revoke an API key as compromised, Loomplane lists the nodes that joined with a token that key made and offers to revoke them too. Take the offer unless you're sure those machines are fine.

Security model covers what a node can and can't do.