Rules watch one service each and judge a window of requests. Alerts are what rules produce. Declared incidents are written by people and are kept apart.
Loading alert rules
Alert rules did not load
This page could not reach the rules service (no answer after 10 s). It cannot show rule states right now. That does not mean your services are healthy, and nothing was changed.
kyc-verify has been over its line since 13:55 GMT and fx-gateway since 14:04 GMT. The apache-httpd rule has seen 5 of the 10 requests it needs, so it is not judging. That is not the same as healthy.
All rules last evaluated 14:07 GMTA request is an entry span (server, consumer, or a root span) with health probes excluded. A failed request is a distinct trace with an error span in that service, never more than the requests. Rules, Overview, Services and Query Studio all use this rule.
2Firing
1OK
1Not enough traffic
1Paused
No rules yet, so nothing is watching
Until you add a rule, Onyx cannot raise an alert for you. Silence on this page does not mean your services are healthy. Declared incidents still appear in their own tab.
Rules
One service per rule. Fired in 24 h counts alerts that started.
Rules are evaluated every minute. A rule that cannot judge shows Not enough traffic or Paused, never OK.
No alert rules yet
A rule watches one service and tells you when a signal crosses a line you choose. Pick a starter below to fill in the form for you, or begin with a blank rule. Every rule waits for a minimum of 10 requests by default, so a quiet service shows Not enough traffic instead of a false alarm or a false all-clear.
Starter rules
Failed requests above 5%, 5 min
Catches a service that starts failing. Waits for 10 requests in the window before it judges.
failed_pct(5m) > 5 and requests(5m) >= 10
Latency p95 above 450 ms
Catches a service that is up but slow. Useful at the edge, where customers feel it first.
p95_ms(5m) > 450 and requests(5m) >= 10
Slow leak: failed requests above 2%, 30 min
Catches a small, steady failure rate that a 5 minute rule would miss.
failed_pct(30m) > 2 and requests(30m) >= 10
Alerts from rules
Each alert quotes the window its rule judged, for example the last 5 minutes. Overview and Services quote the last 60 minutes, so the same service can show a different rate. Both use the same counting rule.
Open now
2
kyc-verify: failed requests above 5%
Firing
Started
13:55 GMT
Duration
12 min, still firing
Severity
Critical
Rule
Failed requests above 5%, 5 min
Service
kyc-verify
What we know31 of 118 requests failed in the last 5 minutes (26.3%), against a 5% line. It began 3 minutes after the kyc-verify config change at 13:52 GMT, and the errors are provider timeouts to the identity verifier.
What we knowp95 is 468 ms against a 450 ms line, up from 389 ms at 13:08 GMT. It has risen since 13:52 GMT, because fx-gateway waits on kyc-verify for new customers. The kyc-verify alert above is the likely cause.
What we know8 of 131 requests failed in the 5 minute window (6.1%) during the late-evening lull. It recovered without a deploy or config change and has stayed under 1% since.
Declared incidents are not listed here. They have their own tab and their own count.
No alerts from rules
No rules exist yet, so no rule can produce an alert. This is not an all-clear. Add a rule from the Rules tab.
Declared incidents
Declared incidents are written by people. A rule never produces them, so they are not counted in Alerts from rules, in the sidebar badge on Alert Rules, or in the open alerts on Overview.
INC-0007Declared by a person, not produced by a rule
Paystack payout delay
Declared
By ops at 13:20 GMT, 47 min ago
Severity
Severity 3
Service
Not set by ops
Linked rule
None
Note from opsPayout confirmations from Paystack are arriving late.
No rule is linked, so nothing is watching this incident.
Muting stops notifications for a rule for a set time. The rule keeps evaluating and its alerts stay visible in Alerts from rules. Muted rules would appear here with who muted them and when they switch back on.