Ask Onyx Oracle
Ask about one service at a time. Every answer shows its steps, what each step read, and what was not checked.
Why did kyc-verify start failing?
No investigation yet
Ask a question above and name the service, for example kyc-verify. You will see each step as it runs, what it read, how long it took, and what was not checked.
Answer
Written in step 6Answer
Scoped to kyc-verifykyc-verify started failing at 13:52 GMT, the same minute as a config change on that service. 81 of its 96 failed requests are timeouts calling the identity verifier, so the change is the likely cause but is not confirmed.
The start time, the config change and the main error group agree. It is not High because the change was read by its time, not by its contents, and the identity provider's side was not visible.
Evidence, read from kyc-verify
What I did not check
- The identity provider's own status page. Only timeouts seen from kyc-verify's side were read.
- What the 13:52 config change changed. Its time was read, not its diff.
- Whether one transfer route is hit harder. Failures were not split by route.
- The other 15 failed requests. 9 are invalid documents and 6 are in small groups. This answer does not explain them.
Suggested next actions
-
Create alert rule
Alert on provider timeouts in kyc-verify
The current rule watches all failed requests above 5%. A rule on provider timeouts alone would point at this cause directly.
-
Keep this answer with its evidence
An investigation record holds the steps, the queries and the open questions for the next person on call.
-
Consider rolling back the 13:52 config change Your team's decision
The owner of that change is best placed to decide. Onyx Oracle does not change configuration, and the cause is not confirmed yet.
Partial findings
Partial, not an answerWhat was found, from kyc-verify only
Not known yet
How this was investigated
Cancel stops after the current read and keeps the findings from finished steps.
-
1Confirm scopeDone0.4 s
Resolve the service named in the question, then fix the window and the counting rule.
Scope is kyc-verify only: entry spans, probes excluded, last 60 min. Failures in other services are kept apart.
-
2Read failed requestsDone3.8 s
Read the failed requests in kyc-verify: distinct traces with an error span in this service.
96 of 1,471 requests failed (6.53%).
-
3Find when failures beganDone2.9 s
Count failed and total requests per minute and find where the failure rate leaves its normal level.
Failures began at 13:52 GMT. Before it, 7 of 1,103 requests failed (0.63%). From 13:52, 89 of 368 failed (24.2%).
-
4Check changes near 13:52 GMTDone2.1 s
Look for deploys and config changes on kyc-verify within 5 minutes of 13:52 GMT.
One change on kyc-verify: a config change at 13:52 GMT. The rate-engine v2.14.1 deploy at 13:41 GMT is another service, outside this window, so it is not attributed.
-
5Read error groupsDone4.7 s
Group the 96 failed requests by error type to see whether one cause dominates.
Provider timeout: 81 of 96 (84%). Invalid document: 9. Three small groups: 6.
Step 5 failed. The error-group query timed out after 4.1 s and returned nothing.
Read: steps 1 to 4, kept below. Not read: error groups for the 96 failed requests, so no cause was identified and no answer was written.
-
6Write the answerDone5.6 s
Write the answer from steps 2 to 5. Cite only evidence read from kyc-verify.
Answer written from kyc-verify evidence only. Confidence: Medium.
ReadFindings from steps 2 to 54 queries