I Used an AI Agent to Test an Open-Source TypeScript Tool and Found a Real Bug

Description and use

An AI agent performed a black‑box test of an open‑source TypeScript tool, found a real bug in combined search filters, and produced a reproducible report that was fixed.

Problem

Testing an open‑source TypeScript AI‑agent toolkit manually is time‑consuming; a concise, reproducible bug report is needed.

Outcome

The AI agent discovered a bug where combined search filters returned incorrect results. The maintainer fixed the issue after receiving a concise, reproducible report.

Workflow

• Define a strict black‑box phase so the AI cannot inspect source code before testing the public experience.
• Use an isolated project with synthetic data to avoid exposing sensitive information.
• Require exact evidence: record OS, runtime, package version, commands, exit codes, expected and actual results.
• Preserve failed attempts to provide useful setup feedback.
• Review every conclusion to separate bugs from documentation or design questions.
• Report one confirmed problem at a time for easier review and fixing.

Evidence

The result was a reproducible bug report that the maintainer confirmed and fixed.

Source

The AI agent was instructed to behave like a new external user.

Source