Monitoring

Nixt Server gives you four ways to see how it is doing: Prometheus metrics, health and readiness endpoints for your load balancer or orchestrator, logs, and doctor, which checks the things that stop mail arriving or being believed.

The metrics and health endpoint

The endpoint is off until you give it an address:

[telemetry]
metrics_bind = "127.0.0.1:9187"

It serves plain HTTP with no authentication, so bind it to a loopback or private address only. Without metrics_bind, metrics are still counted but not served, and the log says no metrics endpoint configured; metrics are kept but not served.

Readiness

/readyz has one check, store, sampled every 15 seconds:

Metrics

No metric label ever carries an address, a mailbox or anything else belonging to one person.

A metric appears once something has been counted in it; a new node may show only some of them.

# HELP vsx_queue_depth Entries waiting in a queue.
# TYPE vsx_queue_depth gauge
vsx_queue_depth{queue="transport"} 3
vsx_queue_depth{queue="delivery"} 0

What to alert on

Alerts to an organisation’s administrators

An organisation’s administrators can be told by mail when something needs them. Each rule names what to watch, the number at which to go off, and whom to tell:

An organisation that has never saved its rules has thirteen, sent to its administrators: passkey-went-back, sending-held, organisation-storage, reported-phishing, approvals, account-recovered, webhook-paused, break-glass-used, not-me, journal-delayed, mail-damaged and lookalike-domain, each at 1, and new-networks at 20. An alert goes off when its number reaches the threshold, and not again until the number has fallen below it and reached it again. Each time, everybody the rule names is sent one plain-text message saying what was watched, the number now, the threshold, and where to look. Every alert is also kept in a history for 90 days.

Write the rules as a file and set them:

{
  "rules": [
    { "when": "queue-backlog", "threshold": 500, "notify": ["postmaster@example.com"] },
    { "when": "dns-problem", "threshold": 1, "notify": ["it@example.com", "on-call@pager.example.net"] }
  ]
}

vsx admin alerts set alerts.json
vsx admin alerts show
vsx admin alerts history

The server’s own alerts

The server has alerts of its own, for what only it can see. They are tenant 0’s, and only its operator sets them:

disk-filling, provider-throttling, backup-failed, backup-test-failed, node-missing, probation-kept and slow-scanner are the server’s alone: an organisation that asks for one is refused. Until the operator saves otherwise, the server has all eleven: disk-filling at 85, bounces-rising at 5, slow-scanner at 10, and the rest at 1. A node that has never been set up to take daily backups is not told about them. They tell nobody by mail until the operator names whom, and each is logged as a warning, and kept in the history, either way. Set them over the local socket, as tenant 0:

vsx admin alerts show --tenant 0
vsx admin alerts set server-alerts.json --tenant 0

The server’s alerts go every way an organisation’s go. As tenant 0, the operator can add a webhook that takes alert events, or export the security log to a SIEM. Each server alert is then sent there as alert.raised, even when it tells nobody by mail. On the console, the operator does this from the Alerts and Webhooks pages. From the command line:

vsx admin webhooks add https://pager.example.com/hooks/mail --events alert --description 'On call' --tenant 0

Blocklists

Once an hour, one node looks the server up on blocklists: each of its sending addresses (what its host name resolves to, leaving out private ones) on the lists of addresses, and each organisation’s domains, up to 1,000, on the lists of domains. The lists are the filter’s own blocklists, leaving out its allowlists. Where the filter names no list of a kind, Spamhaus ZEN is used for addresses and Spamhaus DBL for domains.

The lookups go through the node’s own resolver, as the lists’ terms require. A list asked through a public resolver refuses to answer; a refusal is never counted as a listing, and doctor says which list refused and why.

A rule may name up to 10 addresses, and an organisation may have up to 20 rules. A rule with no addresses is kept in the history only. Administrators set the rules; auditors can see them and the history. Over the API they are GET and PUT /api/v1/tenants/{tenant}/alerts, the PUT with the version the GET gave, and GET /api/v1/tenants/{tenant}/alerts/history.

What is stored

Once a day, one node counts what each organisation stores, by kind:

• message content (each message counted once, however many people hold it)
• message records
• calendars and contacts
• quarantine
• message traces
• the audit log
• reports
• change logs

The count is paced so that it never competes with mail. Each node also records how full its disks are once a day. From 30 days of those records, the server forecasts the day each disk will be full, when it is growing.

versealx-server storage status
versealx-server storage organisations
versealx-server storage kinds --tenant 3

The operator reads the same on the console’s Storage page, beside Usage, and through GET /storage, GET /storage/organisations and GET /storage/kinds. An organisation’s administrators and auditors see their own counts on the What is stored card on the Usage page, with versealx-server admin usage breakdown, or through GET /tenants/{tenant}/storage/breakdown. Counts are kept for 400 days.

When a disk fills

A node never damages mail or bounces it because a disk filled. When a volume holding the store or the blobs reaches the reserve ([storage] reserve_percent, 95% unless you change it), the node stops taking new mail. MAIL FROM on port 25, on submission and from your other premises gets this reply:

452 4.3.1 Insufficient system storage; try again later

That is a temporary refusal. Senders keep the message and try again, and mail apps keep it in their outbox. Everything else goes on. People can read, move and delete mail, and administrators can make room. As soon as the volume is back under the reserve, mail is accepted again without any action from you. The node checks its disks at most every ten seconds.

Well before the reserve, the server’s disk-filling alert tells you a volume is filling (85% unless you change it).

Logs

The server writes its log to standard output. The packaged service sends it to the systemd journal:

journalctl -u versealx-server -f

• Lines are human-readable, with a time, a level and named fields.
• The level is info unless you set the RUST_LOG environment variable, for example RUST_LOG=debug in a systemd drop-in.
• Every line about a message carries its queue id, and no line ever carries what a message says. Relay passwords are never written.

Lines worth knowing

explain

versealx-server explain prints what a node’s configuration will do without starting it: each role, what it listens on and what it sends out of the machine; the outbound pacing; the message ceilings; where data lives; the filter pipeline; and what leaves the premises — certificate requests, metrics and whether the node may deliver to private addresses. See the Command-line reference.

doctor

doctor checks a deployment from the outside in and tells you what to do about each problem. It reads DNS, the certificate, the store and the clock, and changes nothing.

sudo -u versealx versealx-server doctor --config /etc/versealx-server/versealx-server.toml

ok   hostname: mail.example.com
ok   tls certificate: /var/lib/versealx-server/tls/certificate.pem
ok   tls key: /var/lib/versealx-server/tls/key.pem
ok   clock: reads 2026
ok   resolver: passes DNSSEC through; a name that does not exist can be proved not to and the answer kept
ok   reverse DNS: mail.example.com and its 1 address(es) confirm each other
ok   DANE: no _25._tcp.mail.example.com record, so nothing is promised about this host's key
ok   store: answers; 1 domain(s) configured
ok   example.com MX: names this host (mail.example.com)
BAD  example.com SPF: does not name the relay this node sends through: v=spf1 mx -all
         mail leaves from relay.example.net, not from this domain's MX, so every message fails SPF and then DMARC. Publish `example.com. IN TXT "v=spf1 mx include:relay.example.net -all"`
ok   example.com ownership: _versealx-verify.example.com is published, so autoconfig and MTA-STS are answered
ok   example.com DMARC: v=DMARC1; p=none; rua=mailto:dmarc@example.com
…

16 checked, 15 good, 0 to look at, 1 broken

Each line starts with ok, warn (works, but will not keep working, or is not what you meant) or BAD (mail will be lost or refused). A problem’s remedy is on the indented line below it.

Exit codes

The checks

A lookup that fails rather than finding nothing is reported as warn with check the resolver before reading anything else here.