Clusters

A cluster runs one app on several app servers at once, behind its load balancer, with its database on a dedicated database server. If a node dies, the load balancer stops sending traffic to it and the rest keep serving. Deploys build once and roll across the nodes one at a time, so there is no moment where every node restarts together.

Clusters are available on the Team plan.

Prerequisites

The Cluster page (in your app's sidebar) walks you through these:

  1. A dedicated database server. Every node connects to the same Postgres over your private network. If the app still uses the database on its own server, move it first with the database move tool.
  2. An active load balancer. The LB is the public entry point; cluster nodes only serve plain HTTP on the private network.
  3. External file storage for uploads. Files written to a node's local disk (including STORAGE_DIR) are not shared between nodes. Apps that accept uploads should store them in S3-compatible storage. See Persistent Storage.

Adding nodes

Attach an existing app server from the Cluster page, or create a fresh one - it must be on the same provider, region, and private network as the app's current server. Most servers that already run apps are ineligible because your app keeps the same two ports on every node; the page tells you exactly why a server can't join.

Attaching a node automatically ships the release your fleet is currently running (the same commit, not your branch's latest). The node starts serving through the load balancer as soon as that sync deploy completes. Adding or removing a node updates the load balancer's configuration, which briefly reconnects open WebSocket connections (see Load Balancers, "Things to Know").

The primary node - the server you created the app on - runs migrations, scheduled tasks, and the default console/logs sessions. You can promote any active node to primary at any time; nothing on the servers changes, only which node those responsibilities follow.

How clustered deploys work

  1. The release builds once, on your build infrastructure.
  2. Node by node, in order: the release ships, the new slot starts, a health check gates it, traffic flips to it, and the old slot drains. Database migrations run exactly once, on the primary, before it flips.
  3. Only when every node runs the new release is the deploy marked successful. The Deployments page shows a per-node progress row for every roll.

Two things follow from rolling deploys that your app should already handle (they are true of blue/green deploys on one server too):

  • Mixed versions serve briefly. During a roll, some nodes run the new release while others still run the previous one. Ship backward-compatible changes: add columns before code that requires them, remove code before dropping what it used.
  • Migrations run against the shared database while old code serves. Same discipline: migrations must not break the release that is still running.

If a node fails mid-roll, the deploy halts. Nodes already updated keep serving the new release; the rest stay on the old one, and the load balancer hides the split from your users. From the Deployments page you can retry (the roll resumes into the same slot) or roll back, which restores every touched node to the previous release.

Day-to-day

  • Environment variables apply on the next deploy, to every node.
  • Logs and console default to the primary; both pages grow a node picker.
  • Start/stop/restart fan out to every serving node and report per-node failures.
  • Node health on the Cluster page comes from the load balancer itself: recent connection failures and in-flight requests per node.

Background jobs (Oban)

Queue processing needs no changes: every node runs your queues, coordinating through your database.

Leader-elected plugins (Cron, Lifeline, Pruner) need one config line. Oban tells its instances apart by BEAM node name, and until you enable BEAM clustering every node in a Potions cluster runs under the same name - so the instances cannot distinguish each other, every one of them concludes it is the leader, and cron jobs run once per node instead of once per cluster. Give each instance its host's name instead:

# runtime.exs
{:ok, hostname} = :inet.gethostname()

config :my_app, Oban,
  node: to_string(hostname)

After deploying, verify from the console: Oban.Peer.leader?(Oban) must return true on exactly one node (use the console's node picker to check each), and a cron job scheduled every minute must produce exactly one oban_jobs row per minute, not one per node.

This line is a temporary requirement: a coming update gives each cluster node a unique identity automatically, after which the config is no longer needed - and harmless to keep.

Two more things worth knowing:

  • Use Oban 2.23.1 or newer. Earlier versions have an additional leadership-renewal bug in multi-node setups.
  • Queue limits are per node. queues: [mailer: 1] means up to one job per node - N nodes can run N mailer jobs at once. Job uniqueness in open-source Oban is best-effort across nodes, not a hard guarantee.

Database connections

Each node opens its own pool: N nodes x POOL_SIZE (default 10), plus one notifier connection per node, and roughly double per node for the short blue/green overlap during deploys. The Cluster page shows this arithmetic against your database server's max_connections and warns when you get close. Raise max_connections on the database server's settings page or lower POOL_SIZE if you approach the ceiling.

Removing nodes and tearing down

Detaching a node drains its traffic at the load balancer, then removes the app from the server entirely - the server itself stays and can host other apps. Detach all nodes before disabling clustering. Servers that run cluster nodes can't be deleted until they're detached (or, for a primary, until you promote another node first).