Monitor Everything Under Your VMs

A Ceph cluster is an organization-level object here, identified by its fsid and reported by however many of your hosts run the agent. Monitor a five-node cluster from five agents and you still get one cluster, one health verdict and one incident - not five copies of the same story. The agent reads Ceph with a scoped read-only keyring; it never needs the admin key.

Health, quorum and capacity are always derived from the reporters that are currently fresh, so the most complete one is authoritative and another takes over the moment it goes quiet. A cluster only becomes visible once two separate hosts confirm the same fsid, which is what stops a typo or a half-configured host from inventing a phantom cluster; for a genuine single-node or lab cluster you promote it yourself with one click.

Its own page charts capacity, OSD and placement-group states, monitor quorum, client throughput and IOPS as a read/write butterfly, recovery throughput and the misplaced and degraded object backlog. Below that, a per-pool table ranks pools by usage, and an OSD section names the slowest and the fullest OSDs rather than just counting how many are nearfull.

RabbitMQ is polled over its management API, so it is monitored the same way from a Linux, Windows or macOS host. The alert fires when a queue crosses a depth you set or when it has zero consumers while messages are piling up - the canonical sign your workers died, and the case a depth threshold alone misses on a slow queue. One incident per queue, named for it.

HAProxy is read from the stats socket or the HTTP CSV. By default the alert watches the backend aggregate, so it pages when a whole pool is down; switch it to per-server and it pages the moment any single member fails. Pick the one that matches how you run: whole-pool keeps a rolling deploy quiet, per-server tells you about the member nobody noticed.

Both understand planned work. A backend or server in MAINT or DRAIN freezes its alert rather than resolving it, so draining a node for maintenance never sends the false all-clear that would tell you the outage is over.

If you run Prometheus or VictoriaMetrics, it is the thing watching everything else - which means that when it dies, nothing reports the outage. The collector of outages is the collector that went down, so the failure is silent by construction and you find out when someone asks why the graphs stopped.

Fivenines checks that server as software, from outside it: reachability, flavour and version, series count and storage size, plus scrape targets down, rule evaluation failures, dropped notifications and write-ahead-log corruptions on Prometheus, or free disk space, slow inserts and rows ignored on VictoriaMetrics. Bad credentials or a TLS problem are reported as a configuration error rather than an outage, because those are different work.

The targets alert is deliberately conservative: if the targets API stops answering, it freezes instead of reporting that everything recovered. Losing visibility is not the same as being healthy, and a monitoring tool that confuses the two is worse than none.

Seven triggers cover this layer, and every one of them ships as a one-click workflow template: Ceph cluster degraded, RabbitMQ queue backlog, HAProxy backend down, systemd service down, ZFS pool degraded, monitoring stack unreachable and Prometheus scrape targets down. Pick one from the gallery and it lands fully wired - trigger, condition and notification - and every node stays editable.

Defaults are chosen to page on the thing that matters and stay quiet otherwise. The ZFS alert watches DEGRADED, FAULTED, UNAVAIL and SUSPENDED, with OFFLINE and REMOVED available if you want them, since those are usually an operator's own doing. The systemd alert watches failed by default, with inactive opt-in. The Ceph alert watches HEALTH_ERR, and you can add HEALTH_WARN to page earlier - many clusters live in WARN, so that is your call rather than ours.

Alerts fan out per entity, so two failed units open two incidents, each named for its unit, and you know what to look at before you open anything. Everything else on the platform still applies: business-hours and maintenance gates, cooldowns, rechecks and escalation.

Alerts go where your team works: email, Slack, Microsoft Teams, Telegram, Discord, Pushover, PagerDuty or webhooks.