Real-World Project: A Posting Collection & Analysis Agent from Start to Finish

Description and use

An autonomous agent that daily scrapes contest and grant postings, verifies eligibility and term safety, logs failures, and produces a concise digest for quick decision-making.

Problem

Automatically collect contest and grant-program postings daily, filter risky clauses, and produce a one-page digest for quick morning review.

Outcome

The agent reliably collects and filters postings, logs failures, and provides a concise daily digest, enabling the user to focus on decision-making rather than data gathering.

Workflow

• Define a single-sentence goal with completion criteria
• Measure site structure with curl and grep to identify real data points
• Create test fixtures from measured HTML fragments
• Write tests to enforce open status, deadline extraction, and other rules
• Implement collector modules with error handling and failure logging
• Add a terms filter using regex patterns to classify clauses
• Schedule the agent to run daily and generate a digest
• Audit outputs with an independent reviewer and adjust parsing logic
• Review the digest and decide on participation
• Iterate on parser and regexes for new sites or fields

Evidence

The first item in the list is an ad banner, so it always contaminates the parse

Source

Writing tests first narrows the AI's scope of work. Instead of "make something reasonably good," it becomes "make these tests pass."

Source

A system that only shows success counts cannot be trusted. Failures must appear on the same line for a human to believe it.

Source