Server Health

Every Thirdlane server can report on its own condition: how busy it is, how much disk it has left, which of its services are running, whether the database and chat storage are answering, and - on a redundant pair - which half of the pair it is. The report is available three ways: on the command line, over the REST API, and to a monitoring system you run yourself.

This is a single-server report. It says what one machine knows about itself, and it makes no decisions - see What it does not do.

Why ask the server instead of watching it from outside

A monitoring system that connects to a server from outside can tell you whether a port answers. That is useful, but it cannot tell you whether what it found is correct, because it does not know what the server was supposed to be doing.

The clearest case is a redundant pair. On the server that follows the other one, the call-handling services - Asterisk, Kamailio, the media relay - are deliberately stopped, so that only one of the two machines answers calls. An outside check that tests those ports finds nothing listening and reports a healthy pair as half broken. Every sweep raises the same false alarm, and after a while nobody reads the alerts.

The report solves this by giving two values for every service: the state it is supposed to be in and the state it is actually in. A stopped Asterisk on a server where Asterisk is meant to be stopped is reported as correct. A running Asterisk on that same server is reported as a serious fault, which it is - two machines answering calls with one database between them is how the two copies of your configuration diverge.

The second reason is that the report keeps working when the things behind it do not. Reading it needs no database, no session, and no user account, so “the database has stopped” is one of the answers you can get rather than the reason you get no answer at all.

What the report contains

The server itself - name, Thirdlane version and build, operating system, kernel.
Load and capacity - processor use, memory used and total, load averages, uptime, and every mounted filesystem with its percentage used and free megabytes.
Services - one entry per service the product installs, each with its expected state, its actual state, how long it has been in that state, and how many times it has been restarted. Services that are stopped on purpose are also listed together, so a reader does not have to work out which stopped services are intended.
Services that are answering - the telephony engine, the SIP proxy, the media relay, the chat server, the web front end and the session cache are each asked to do something and report what came back: calls in progress, phones registered, calls being carried. See below for why this is asked separately.
Storage - whether the configuration database answers a query (not merely a connection: a database out of file handles accepts the connection and then fails on the first statement), the database version, and either how replication is doing or how many other servers are copying from this one. Chat and file storage and the session cache are reported the same way.
Redundancy - whether this server is part of a pair, which member it is, whether the call-handling services are switched off here, and whether changes to shared configuration are accepted or refused.
HA cluster - only on a server whose data sits on a mirrored disk shared with a second server: whether the mirror is connected to the other server, whether this server’s copy is up to date, which of the two currently holds the disk and runs the services, and which servers the cluster considers available, held out of service, or offline. A mirror that has stopped copying is reported as a problem, because from outside nothing looks wrong - the server carrying the calls carries on doing so while the second one quietly stops being a copy of anything. See HA Cluster.
Problems - a short list of things this server is in a position to be certain about, each with a severity. There is a summary state for the whole server: ok, warn, or problem.

The problem list is deliberately short. It covers a service that should be running and is not, a service that should be stopped and is not, a service that is running but does not answer, a service that keeps restarting, a database that is not answering, replication that has stopped, an active server that no longer has a standby copying from it, a filesystem with nothing left, and chat storage that is not answering. It does not include a busy processor or a nearly full disk: judging those needs to know what this particular server normally looks like, which the server does not know and your monitoring system does. The numbers are in the report so that whatever reads it can apply its own thresholds.

Running is not the same as working

Asking the operating system whether a service is running tells you a process exists. It does not tell you the process is doing its job, and on a phone system the difference is the whole point: a SIP proxy that accepts connections and answers nothing carries no calls, but every check that looks at its process or its port calls it healthy.

The report therefore asks each of the main services to actually do something - report its uptime, count its sessions, return a status line - and reports a service that is running but not answering as a serious fault.

The same reasoning covers a service that is crashing. A service being restarted every few seconds is running at almost every individual moment you look, so a check that samples it finds it healthy nearly every time. The report includes how long each service has been up alongside its restart count, and warns when a service that has restarted before has only just started. It is a warning rather than a fault because an ordinary restart - a deployment, an administrator - looks identical from a single reading. Something that reads the report repeatedly can tell the two apart, because a crash loop keeps moving the restart count and a deployment does not.

Losing the standby

On a pair, the standby knows whether it is still copying from the active server and says so, so a standby that falls behind or stops is reported by the standby itself.

The active server reports the other side of that: how many servers are connected to copy from it. Zero, on a server that is supposed to have a standby, means the pair is certainly down to one machine - no second copy of the configuration, nothing ready to take over - and is reported as a fault. A single server with no pair is not affected, because it is not expected to have anything copying from it.

A count above zero is weaker evidence than it looks, and it is worth knowing why. The connection that feeds a standby is held open by a database thread that spends its time waiting for new changes to send. If the other machine stops, that thread does not find out until it next tries to send something, which on a quiet server can be a long time. So the count answers “is anything set up to copy from me” reliably, and “is the other machine alive right now” only eventually.

The dependable way to know a pair is still a pair is to read both members’ reports and alert when one of them stops answering. No single server can do this for you: a server that has died does not report that it has died. See the monitoring setup below.

Reading it on the server

tl-health

You get a summary written for a person - state, load, disks, telephony, storage, redundancy, and any problems first.

Three other forms are useful:

tl-health --json

The same report as the API returns, for a script.

tl-health --check

Prints nothing when the server is healthy and exits with status 0; prints the problems and exits 1 when it is not. This is the form to put in a cron job or a Nagios-style check.

tl-health --quick

Skips the checks that ask a service to do something. Everything else - services, disks, load, redundancy - is read from local files and answers immediately.

The command is at /usr/libexec/webmin/asterisk/config_adm/bin/tl-health if it is not on your path.

Reading it over the REST API

GET /api/health

Add ?quick=1 for the shorter form described above.

Two credentials are accepted, and the difference between them matters:

A session or an API key, as with every other endpoint. This requires a server-level administrator account, because the report covers services and filesystems rather than anything belonging to one tenant.
The server’s health token, sent as an X-Health-Token header. This is the form for a monitoring system.

The token exists because an API key is a row in the configuration database. When the database stops, key authentication stops with it - and a server whose database has stopped is exactly the server you most want a report from. The token is a file on the server instead, read at the moment of the request, so it still works. It is created when the product is installed, is readable only by root, and grants exactly one thing: reading this report. It is not a login and it cannot change anything.

To see the token (it is created on first use if the server has none):

tl-health --show-token

Then, from your monitoring system:

curl -s -H "X-Health-Token: <token>" https://pbx.example.com/api/health

Give each server’s token only to the system that monitors it, and treat it as a credential. Anyone holding it can read that server’s service states and disk use.

Setting up monitoring

On each server, run tl-health --show-token and record the token.
In your monitoring system, add a check that requests https://<server>/api/health with the X-Health-Token header, on whatever interval you use elsewhere. Five minutes is a reasonable starting point; the report is cheap to produce and every check inside it has a time limit, so a stalled service cannot make the request hang.
Treat a request that does not complete as the server being unreachable, and treat the problems list as the server’s own findings. Raise one alert per problem rather than one per server, so a stopped database does not hide a stopped replica.
Apply your own thresholds to the numbers - processor, memory, and each filesystem’s percentage used.
Confirm it works before you rely on it. Stop a service that is meant to be running - systemctl stop prosody is harmless enough on a test server - and check that your monitoring system notices and that starting it again clears the alert.

On a redundant pair, do steps 1 and 2 for both members, using each machine’s own name rather than the shared service name. A request to the shared name reaches whichever server currently holds it, which is not what you want when the question is “which of these two is in trouble”.

Then add one check that neither server can do for itself: alert when one member of a pair stops answering while the other is still answering normally. That is the pair losing its redundancy, and it is invisible from either machine - the one that has died says nothing, and the survivor looks perfectly healthy. It is worth treating as seriously as an outage, because the next fault has nowhere to fail over to.

What it does not do

It does not notify anybody. The server produces the report; sending mail, raising a ticket, or paging someone is your monitoring system’s job.
It does not fail over. On a redundant pair, promoting the standby is a deliberate action taken by an administrator or by something you have built for the purpose. See Standby Server.
It does not keep history. Each request produces a report for that moment. Storing and graphing them is your monitoring system’s job.

Best practices

Monitor each server by its own name. On a pair, the shared service name answers from whichever member holds it, so a check against it cannot tell you which machine is degraded.
Alert on the problem list, apply thresholds to the numbers. The two are separated on purpose: the server is certain about the first and has no opinion about the second.
Keep the token out of shared configuration. It is per server, and a monitoring system that carries one token for the whole fleet loses the only thing that limits what a leaked token exposes.
Alert when one member of a pair goes quiet, and watch both replication lines. Neither is something either machine can report for you, and both are the difference between having a standby and believing you have one.
Test the check, not just the endpoint. A monitoring system configured against a URL that returns 200 tells you nothing until you have seen it react to a real fault.

Standby Server - running two servers as a pair, and what promotion involves
HA Cluster - two servers sharing one copy of the data on a mirrored disk, and where its state is reported
Servers - the servers that make up an installation
OpenAPI REST - the full API reference, including this endpoint’s response schema