BEAM Clustering

BEAM clustering connects the nodes of a cluster with Erlang distribution. Once connected, Phoenix.PubSub, Presence, and broadcasts work across every node with no further plumbing: a broadcast on one node reaches LiveViews on all of them.

BEAM clustering is optional. Everything else about clusters - rolling deploys, the load balancer, LiveView over WebSockets, Oban background jobs - works without it. Enable it when your app broadcasts between users or uses Presence across the fleet.

What Potions provides

Enabling BEAM clustering on the Cluster page wires up, on each node's next deploy:

  • A shared cluster cookie, generated once and delivered as RELEASE_COOKIE. (Without it, each build's baked-in cookie would differ and nodes could never authenticate.)
  • Node names on private IPs: each node runs as yourapp_blue@10.x.x.x instead of a loopback name, so nodes can reach each other over your private network.
  • Pinned distribution ports, bound to the private interface only, with firewall rules that admit only your private subnet. Nothing is reachable from the public internet.
  • A peer list: the POTIONS_CLUSTER_HOSTS environment variable lists both deploy slots of every node. Your libcluster config reads it.
  • A host EPMD service on every node, so node discovery survives deploys and restarts. Its binary comes from the newest OTP release deployed on the node and is refreshed when a deploy ships a newer one, so EPMD security fixes arrive with your OTP upgrades; the refreshed binary takes over at the service's next restart (a reboot, or the next time a node is attached or BEAM clustering is enabled).

One thing Potions cannot do is add a dependency to your codebase. The connection itself is made by your app, with the snippet below.

BEAM clustering requires OTP 27 or newer (set on the app's Settings page). The host EPMD service runs a binary staged from your release, and OTP 26 no longer receives security fixes.

The snippet your app needs

# mix.exs - pin 3.5 or newer (the hard floor is 3.3.3: older
# versions silently ignore :timeout and connect only once at boot)
{:libcluster, "~> 3.5"}

# application.ex children (before your Endpoint):
topologies = [
  potions: [
    strategy: Cluster.Strategy.Epmd,
    config: [
      hosts:
        "POTIONS_CLUSTER_HOSTS"
        |> System.get_env("")
        |> String.split(",", trim: true)
        |> Enum.map(&String.to_atom/1),
      # do not omit: this makes libcluster retry and reconnect
      # after every deploy
      timeout: 5_000
    ]
  ]
]

{Cluster.Supervisor, [topologies, [name: MyApp.ClusterSupervisor]]}

Two lines here are load-bearing:

  • The version floor. Before libcluster 3.3.3, the Epmd strategy connected once at boot and silently ignored timeout: - a cluster that never heals after the first deploy. If your mix.lock pins an older version, update it.
  • timeout: 5_000. Without it libcluster uses the default of "connect once, never retry". With it, every node re-checks its peer list every 5 seconds, which is what re-forms the mesh after each rolling deploy.

Deploy after adding the snippet, then verify from the Cluster page ("Connected peers" check) or the console: Node.list() on any node should show the other nodes.

Settings your release must not override

Potions delivers the node name, distribution mode, cookie, and VM flags through the environment at start time. Two files in a release are sourced after that environment and win silently, so a value set in either of them disables clustering without an error:

  • rel/env.sh.eex must not export RELEASE_NODE, RELEASE_DISTRIBUTION, RELEASE_COOKIE, or ELIXIR_ERL_OPTIONS. Apps migrated from platforms that set the node name there (Fly's generated env.sh.eex, for example) need those lines removed.
  • rel/vm.args.eex must not set -start_epmd, -erl_epmd_port, -name/-sname, -setcookie, or -kernel inet_dist_*. The same goes for config :kernel, inet_dist_listen_min: ... in your config files.

The symptom of an override is a mesh that never forms: libcluster keeps logging unable to connect for every peer, and the "Connected peers" check shows nothing.

What to expect in your logs

The peer list names both deploy slots of every node, but only one slot per node is alive outside deploys. libcluster logs a warning for each absent name on every retry:

[libcluster:potions] unable to connect to :"yourapp_green@10.0.0.3"

This is normal, and it is the proof that reconnection polling works. On a three-node cluster at the default 5-second timeout, expect roughly 36 such lines per minute across the fleet. If the noise bothers you, either raise timeout: to 15-30 seconds (reconnects after deploys just take proportionally longer), or silence libcluster's warnings in one line at startup:

Logger.put_module_level(Cluster.Logger, :error)

Also worth knowing:

  • Worker nodes join the mesh too. They run the same code and env, and Erlang connections are transitive, so Node.list() shows web and worker nodes of every host. That is normal and useful.
  • Node add/remove reaches the peer list on the next deploy. Attaching or detaching a node updates POTIONS_CLUSTER_HOSTS, which lands on each node at its next deploy.
  • A hard-killed node takes up to a minute to disappear from Node.list() (Erlang's net_ticktime); a cleanly stopped slot disconnects immediately.
  • Debugging from a laptop or a one-off shell? Attach with --hidden so your debug node doesn't become a mesh member. (Potions' own console and scheduled tasks run as hidden nodes and additionally disable the distribution listener with -dist_listen false, since they run on the servers where the port window is pinned.)

Rolling deploys and mixed versions

During a roll, old-release and new-release nodes share the mesh for a short window. This is the same discipline as mixed versions serving HTTP (see Clusters), extended to inter-node messages:

  • PubSub payloads and messages to remote processes cross the version boundary as plain terms. Keep changes to broadcast payloads additive across one release, or tolerate a handler crash during the seconds of overlap.
  • Apps that register singleton processes with :global will see a "name conflict" log and one singleton restart per deploy as the mesh re-forms. That is expected behavior, not a bug.
  • If you configure Phoenix.PubSub with a pool_size, it must be identical on every node. Never derive it from CPU count - your nodes may be differently sized.
  • Erlang distribution is compatible across at most two OTP major versions in either direction. Moving an app more than two majors in one deploy (26 to 29, say) splits the mesh for the duration of the roll; step through the versions instead.

Oban does not need this

Oban coordinates through your database, not through Erlang distribution. Queue processing and leader-elected plugins work on clusters with BEAM clustering off. See Clusters for the details.

Multi-tenant apps

Nothing here is tenant-specific: a multi-tenant app clusters like any other, and BEAM clustering gives it PubSub and Presence across every tenant session regardless of which node serves it. Tenant subdomains and custom domains keep terminating at the load balancer (see Clusters).

Security model

Erlang distribution makes your nodes one trust domain: a process on one node can run code on all of them. That is the feature. It also means a compromised node is a compromised cluster, which is why Potions confines distribution to your private network - node names on private IPs, listeners bound to the private interface, firewall rules admitting only your private subnet, and nothing on the public interface.

Traffic between nodes is not encrypted (your provider's private network isolates it, but does not encrypt it), and all of your servers on the same private network can reach the distribution ports. Both are standard for this deployment shape; if your compliance needs go further, contact us before enabling.

Disabling BEAM clustering revokes the distribution firewall rules right away, and each node leaves the mesh on its next deploy. Node names keep their private-IP form for as long as the app runs on more than one server: host-unique names are part of how clustered apps keep scheduled work firing exactly once (see Clusters). They return to loopback names once the app is back on a single server. The cookie is kept so re-enabling later doesn't require a coordinated cookie change.