HA Cluster

An HA cluster - high availability cluster - is a pair of servers that share one copy of your configuration and data. The mechanism is disk mirroring: a volume is kept byte-for-byte identical on both machines as it is written, so every write goes to both at the moment it is made. That single shared copy is why only one machine runs the services at a time. If that machine is lost, cluster software starts the services on the other one and moves the address your phones and carriers use along with them.

These clusters are built for a site by the Thirdlane services team, using software that is not part of the platform: disk mirroring in the Linux kernel (DRBD) and a cluster manager that starts, stops and moves the services (Pacemaker, with pcs as its command-line tool). The Manager reports what it finds but does not configure or control any of it.

This is not a cluster deployment. The Infrastructure section of the Manager is for spreading one phone system across several servers to handle more traffic - telephony workers, media servers, the cluster’s topology. That answers how much the platform can carry. An HA cluster answers something else: whether one server surviving means the phone system survives. The two are unrelated, are set up separately, and the shared word is unfortunate. See Deployment Options.

If you are choosing how to make a server redundant, read Two ways to run a second server first. The platform’s own arrangement, a replicated standby, is set up from the Manager and needs nothing outside it. Do not configure both on the same servers.

Two ways to run a second server

The platform supports two arrangements for keeping service available when a server is lost. They are built on different mechanisms, they protect against different failures, and neither is a better version of the other.

An HA cluster keeps one copy of configuration and data, written to both machines at the same moment. Only one machine runs the services; cluster software moves them, and the address clients use, to the other machine when it needs to. Because there is a single copy, nothing has to be matched up afterwards.

A replicated standby keeps a second server with its own separate copy, continuously up to date and deliberately unable to serve anyone. Its databases run, because they have to receive the copy as it arrives, and it rebuilds its own telephony configuration from that data as it lands. Everything that answers phones, clients and carriers is held stopped on purpose, and an administrator promotes it. See Standby Server for how it works in full.

Neither arrangement is two servers sharing the work. One server carries all of it and the other is a copy being kept ready, so capacity does not increase and there is nothing to balance across.

What an HA cluster solves

Nothing that was committed is lost. Each write is confirmed by both machines before it completes, so losing one machine loses no data.
Everything stays in step. The database, its logs, voicemail, call recordings and every configuration file are one set of files on one volume, so they cannot disagree with each other after a move.
The move happens without anyone present. The cluster software detects the failure and starts the services on the other machine.

Where it fits: two machines in one data center or one cloud region, on a fast local network, where losing committed data is unacceptable and call recordings have to survive on the servers themselves.

What a replicated standby solves

Damage to the data does not follow. The standby runs its own database and applies changes as a database rather than as disk blocks, so a fault that corrupts data on the active server does not automatically destroy the second copy as well.
You can check that it is ready. Because the copy is arriving continuously, the Standby Server screen can report at any time whether a switchover would succeed - whether replication is current, whether the data is complete, and what would be missing. That is a check on the copy, not a demonstration that the server can carry calls. The only thing that demonstrates that is a promotion, which is why testing one in a maintenance window is worth doing.
Taking over is quick. The standby’s databases are already open and their caches are warm, so promotion starts the client-facing services and little else. A machine whose services have been stopped has to mount the volume, let the database recover from its transaction log and warm its caches first, which on a large database is measured in minutes.
Distance is supported. The two servers can be in different data centers or different regions.

Where it fits: any All-in-One server, including ones in different locations, where a few minutes of administrator-initiated switchover is acceptable and a verifiable standby matters.

Where each one is limited

An HA cluster writes every change to both machines, so a fault in the data itself reaches both copies. Neither arrangement replaces backups, and this is the case that makes the point.
An HA cluster expects both machines close together. Each write waits for the second machine to confirm it, so the time to reach that machine is added to every write. Mirroring over long distances is possible, but only by letting the second copy lag, which gives up the guarantee above.
The address clients use cannot move between networks. It moves within one network, or within one cloud region where the provider can reassign it. Between distant sites, each site needs its own address and the switch is made in DNS, which affects new connections only.
A replicated standby does not copy call recordings. See Where call recordings should live.
A replicated standby copies voicemail every five minutes, so messages left in the last few minutes stay on the server that took them.
A replicated standby is promoted by an administrator, because two servers on their own cannot tell “the other server has stopped” apart from “I cannot currently reach the other server”. Cluster software solves that with a third machine that arbitrates; a pair has nobody to ask.

Do not run both on the same servers

Each arrangement has to be the only thing deciding which machine is in charge. Two mechanisms making that decision independently can both conclude they are in charge at the same moment, which is the situation either one on its own exists to prevent.

On servers that are part of a cluster, the Manager refuses to configure a standby and says why. Everything else in the Manager works normally.

How an HA cluster is put together

Three pieces do the work, and it helps to know which one is responsible for what when something goes wrong.

The mirror keeps one volume identical on both machines. Writes are sent to the local disk and to the other machine at the same time, and a write is not finished until both have it. One machine’s copy is the live one; the other machine holds its copy and does not mount it.

The links into the mirrored volume are what make the platform use it. The volume is mounted in one place, and the directories the platform expects - the Asterisk configuration and spool, the database files, the web server and Manager configuration, chat storage, the SIP proxy configuration - are redirected into it when the machine takes over. Nothing in the platform is aware of this; it reads and writes its usual paths.

The cluster manager decides which machine holds the volume and starts the services on that machine, in order: the mirror first, then the volume, then the database, then telephony, then everything else. Each step depends on the one before it, so a database that will not start stops the sequence rather than leaving half a phone system running.

What the Manager shows you

Navigate to General Settings & Tools > HA Cluster. The menu entry appears only on a server that has a mirror on it, and the screen is read-only - it reports the cluster and offers no way to change it:

A diagram of the two machines, which one currently holds the mirrored volume and runs the services, and whether the mirror is connected.
Whether this machine’s copy of the volume is up to date.
Which machines the cluster considers available, held out of service, or offline.
A note that the arrangement is not managed by the Manager.

Standby Server is a separate screen for the platform’s own arrangement, a replicated standby. On a clustered server it will tell you a standby cannot be paired with this machine and point you back here; it does not report the mirror.

The same facts are in the server’s health report, so whatever you use for monitoring sees them too:

tl-health

The report raises a problem of its own when the mirror stops being a mirror. That matters more than it sounds: when the mirror goes down, the machine carrying the calls keeps carrying them, so everything visible from outside is still correct while the second machine has quietly stopped being a copy of anything. The conditions reported are the mirror not being connected, a copy that is not up to date, a machine held out of service by an administrator, and a machine the cluster can no longer reach.

Services that are stopped on the machine not holding the volume are reported as stopped on purpose rather than as failures. That machine is supposed to be running nothing.

Day-to-day operation

Making configuration changes. Use the Manager on the machine currently running the services, which is the machine the client-facing address points at. It is the only one with the volume mounted.

Restarting a service. Restarting a platform service from the Manager or with systemctl does not hand the phone system to the other machine. The cluster is set up to allow a service to fail and be restarted in place.

Taking a machine out of service for maintenance. This is a cluster operation, not a Manager one, and it is what the services team should be asked for. A machine left out of service is reported as such, and it stays that way - it is a deliberate act with no expiry, and until it is brought back the pair is a single server.

Checking the mirror. The health report covers it. On the machine itself, the cluster’s own tools report the same thing in more detail. Note that a machine which has been away comes back and catches up: until it has, there is a second copy but not yet a usable one.

Updating the platform. Ask the services team before you do. Your data and most of your configuration are one shared copy, so there is no second database to patch - but the installed program files are not on the mirrored volume, so each machine has its own and both have to be brought to the same version. The order the services restart in is also under the cluster’s control rather than the installer’s.

Changing how the cluster is built. Contact the Thirdlane services team. The mirrored path list, the order the services start in, and the way the client-facing address moves are all specific to how your servers are deployed, and none of them is configured from the Manager.

Best practices

Test a takeover before you need one. Most first attempts uncover something. A maintenance window is a cheaper place to find it than an outage.
Bring a machine back into service as soon as its maintenance is done, and confirm the mirror reconnected and both copies are complete.
Keep taking backups. Both arrangements protect against losing a machine. Neither protects against losing the data, which is written to both copies at the same moment by design.
The HA Cluster screen reports; it is not where the cluster is changed. Every change to a cluster is a cluster operation, made with the cluster’s own tools by the Thirdlane services team.

Standby Server - the platform’s own arrangement, a replicated standby set up from the Manager
Server Health - the health report, and how to reach it from outside the server
Deployment Options - All-in-One and distributed deployments
Network Topology - addresses the platform publishes to clients and carriers