Clusters

A cluster runs one app on several app servers at once, behind its load balancer, with its database on a dedicated database server. If a node dies, the load balancer stops sending traffic to it and the rest keep serving. Deploys build once and roll across the nodes one at a time, so there is no moment where every node restarts together.

Clusters are available on the Team plan and above.

Prerequisites

The Cluster page (in your app's sidebar) walks you through these:

  1. A dedicated database server. Every node connects to the same Postgres over your private network. If the app still uses the database on its own server, move it first with the database move tool.
  2. An active load balancer. The LB is the public entry point; cluster nodes only serve plain HTTP on the private network.
  3. External file storage for uploads. Files written to a node's local disk (including STORAGE_DIR) are not shared between nodes. Apps that accept uploads should store them in S3-compatible storage. See Persistent Storage.

Adding nodes

Attach an existing app server from the Cluster page, or create a fresh one - it must be on the same provider, region, and private network as the app's current server. Most servers that already run apps are ineligible because your app keeps the same two ports on every node; the page tells you exactly why a server can't join.

Attaching a node automatically ships the release your fleet is currently running (the same commit, not your branch's latest). The node starts serving through the load balancer as soon as that sync deploy completes. Adding or removing a node updates the load balancer's configuration, which briefly reconnects open WebSocket connections (see Load Balancers, "Things to Know").

The primary node - the server you created the app on - runs migrations, scheduled tasks, and the default console/logs sessions. You can promote any active node to primary at any time; nothing on the servers changes, only which node those responsibilities follow.

How clustered deploys work

  1. The release builds once, on your build infrastructure.
  2. Node by node, in order: the release ships, the new slot starts, a health check gates it, traffic flips to it, and the old slot drains. Database migrations run exactly once, on the primary, before it flips.
  3. Only when every node runs the new release is the deploy marked successful. The Deployments page shows a per-node progress row for every roll.

Two things follow from rolling deploys that your app should already handle (they are true of blue/green deploys on one server too):

  • Mixed versions serve briefly. During a roll, some nodes run the new release while others still run the previous one. Ship backward-compatible changes: add columns before code that requires them, remove code before dropping what it used.
  • Migrations run against the shared database while old code serves. Same discipline: migrations must not break the release that is still running.

If a node fails mid-roll, the deploy halts. Nodes already updated keep serving the new release; the rest stay on the old one, and the load balancer hides the split from your users. From the Deployments page you can retry, which redeploys every node into the same slot (nodes already running the new release are restarted in place briefly), or roll back, which restores every touched node to the previous release.

Day-to-day

  • Environment variables apply on the next deploy, to every node.
  • Logs and console default to the primary; both pages grow a node picker.
  • Start/stop/restart fan out to every serving node and report per-node failures.
  • Node health on the Cluster page comes from the load balancer itself: recent connection failures and in-flight requests per node.
  • Multi-tenant apps cluster like any other app. Tenant subdomains and tenant custom domains are terminated at the load balancer, the certificate approval check reaches your app through any healthy node, and switching modes or adding wildcards on a running cluster configures every node and the load balancer for you. One constraint: a server can carry only one multi-tenant app in any role, so a node that hosts another multi-tenant app (as its primary or as a cluster node) can neither join nor be promoted, and multi-tenant mode can't be enabled while a node of this app hosts another multi-tenant app. See Multi-tenant apps.

Background jobs (Oban)

Queue processing needs no changes: every node runs your queues, coordinating through your database.

Leader-elected plugins (Cron, Lifeline, Pruner) also work without configuration: each node's BEAM instance runs under a name that carries its server's private IP, so Oban can tell the instances apart and elects exactly one leader through your database. This does not require enabling BEAM clustering. Apps deployed before this update pick up the unique names on their next deploy.

If you previously set config :my_app, Oban, node: ... while this fix was pending, the line is no longer needed - and harmless to keep.

To verify: Oban.Peer.leader?(Oban) must return true on exactly one node (use the console's node picker to check each), and a cron job scheduled every minute must produce exactly one oban_jobs row per minute, not one per node.

Two more things worth knowing:

  • Use Oban 2.23.1 or newer. Earlier versions have an additional leadership-renewal bug in multi-node setups.
  • Queue limits are per node. queues: [mailer: 1] means up to one job per node - N nodes can run N mailer jobs at once. Job uniqueness in open-source Oban is best-effort across nodes, not a hard guarantee.

Database connections

Each node opens its own pool: N nodes x POOL_SIZE (default 10), plus one notifier connection per node, and roughly double per node for the short blue/green overlap during deploys. The Cluster page shows this arithmetic against your database server's max_connections and warns when you get close. Raise max_connections on the database server's settings page or lower POOL_SIZE if you approach the ceiling.

Removing nodes and tearing down

Detaching a node drains its traffic at the load balancer, then removes the app from the server entirely - the server itself stays and can host other apps. If that cleanup fails for good (the server is unreachable or was already wiped), the node shows Removal failed: retry it, or choose Remove without cleanup to drop the node from the cluster while leaving whatever the app left on that server in place until it is reprovisioned or deleted.

Once every node is detached, Disable clustering on the Cluster page returns the app to a single-server app on its primary. BEAM clustering is turned off with it; the load balancer and database server stay as they are.

Deleting servers: a server that is a cluster node can't be deleted until it is detached (or removed without cleanup). Deleting the primary is not blocked: it deletes the app, like deleting any app server does, along with its database on the database server and its footprint on every node. The other nodes, the load balancer and the database server keep running.