Overhear
A QA analyst for voice agents. It listens to every call so you don't have to
Built by Devanshu Chicholikar, software engineer and AI engineer in Boston.
AI QA analyst for voice agents: grades every call a Retell scheduling agent takes. Code checks what the call log proves, and an LLM-as-a-judge checks what the agent said against the clinic's real slots.
Voice agents fail quietly. They offer a slot that does not exist, skip the identity check or book the wrong provider, and the call still sounds fine. Overhear grades every call a Retell healthcare-scheduling agent takes: code checks the call's action log, and an LLM judge checks what the agent said against the clinic's real open slots.
The split is the point. Code reads the call's action log in time order: was the patient verified before anything got booked, did the booking go through, did any tool call fail. Those 3 of 7 rubric dimensions carry 50% of the score and never depend on a model. An LLM-as-a-judge, handed the clinic's real open slots from Postgres, grades the other 4: invented slots, the wrong provider, safety and tone.
On a 39-call golden dataset with 4 planted failure types, it caught 23 of 23 planted failures (macro-F1 0.89) with 0.99 self-consistency across 3 repeat runs. In one earlier run the judge missed an identity check skipped behind medical advice; the code check caught it, matching the gold labels 14 of 14. Adversarial cases showed where the judge over-flags (precision 0.60 and 0.67 on two types), and the eval page says so. Built in 7 days, with 39 commits and 27 merged PRs that week, and 132 tests.
- 23/23
- Planted failures caught
- 0.89
- Macro-F1
- 50%
- Of the score decided in code
- 132
- Vitest cases
4 failure types on a 39-call golden dataset
Small, mostly synthetic set with one labeler
Identity, booking and tool checks read from the call log
38 files over in-memory PGlite databases