AI Hiring in a Live Company: The Agent Filled the Funnel. The AI Interview's Labels Did Not Line Up With Who Is Still Working
Description and use
The hiring agent processed 1,973 applications in 36 days, entering 142 candidates into the CRM and booking 138 human interviews. The AI interview was introduced on day 15, inviting 41 candidates; 22 completed it, yielding 7 HIRE, 12 MAYBE, and 3 NO verdicts. By day 36, 10 candidates were working: 2 from AI HIRE, 6 from AI MAYBE, and 2 from non‑AI interviews. The AI interview did not clearly separate those who stayed from those who left, and a high no‑show rate (37 of 81 held) and waiting delays limited its effectiveness.
Problem
A telemedicine company needed to process ~2,400 job applications for a phone sales role but lacked recruiter capacity to read and triage them, leading to bottlenecks and lost candidates.
Outcome
The hiring agent processed 1,973 applications in 36 days, entering 142 candidates into the CRM and booking 138 human interviews. The AI interview was introduced on day 15, inviting 41 candidates; 22 completed it, yielding 7 HIRE, 12 MAYBE, and 3 NO verdicts. By day 36, 10 candidates were working: 2 from AI HIRE, 6 from AI MAYBE, and 2 from non‑AI interviews. The AI interview did not clearly separate those who stayed from those who left, and a high no‑show rate (37 of 81 held) and waiting delays limited its effectiveness.
Workflow
- 1. AI agent monitors job site and reads incoming applications, including screening answers and work history.
- 2. Agent applies leadership‑written rules to select candidates and enters them into the CRM.
- 3. Agent offers interview times and books slots; candidates move from "picked" to "booked" instantly.
- 4. Leadership changes selection rules eight times in first 15 days; agent applies new rules to subsequent batches.
- 5. On day 15, AI interview replaces human supervisor for first‑round interviews.
- 6. AI interview delivers a 15‑minute voice‑AI conversation, scores transcript into HIRE/MAYBE/NO, and records verdict in CRM.
- 7. Human interviews continue for candidates not sent to AI interview; hiring decisions are made by team lead and recorded in CRM.
- 8. Trial days begin on day 21; candidates with CRM accounts must make a call to be considered working.
- 9. Hiring decisions are logged; AI verdicts alone can lead to rejection without human review.
Evidence
Reading the 1,973 worth a close look would take one recruiter 65 to 100 hours, time this small team did not have.
Over 36 days the agent read 1,973 applications in detail. The CRM received 142 candidates, and 138 human interviews were booked for 130 of them.
The AI interview was meant to replace the human first round. Instead it ran alongside it.
The final verdicts were HIRE 7, MAYBE 12, NO 3.
Of the ten people now on the team, six came through as MAYBE and two as HIRE (Fisher exact test for 2 of 7 against 6 of 12: p ≈ 0.63).
- Record date: 2026-11-03T19:29:07.632Z