My job search API returned the same job five times — the apply URL gave away the dedup key

Description and use

Job search API returned duplicates because the same posting appeared on multiple aggregators. Using the apply URL’s ATS ID removed duplicates and improved result freshness.

Problem

Job search API returned duplicate listings for the same requisition, causing confusion and inaccurate results.

Outcome

Deduplication using the apply URL’s requisition ID eliminated duplicate job listings and improved result freshness by prioritizing ATS-hosted postings.

Workflow

• Query job search API with keywords and location
• Collect listings from multiple aggregators and company board
• Extract apply URL from each listing
• Parse canonical requisition ID from the URL (e.g., boards.greenhouse.io/company/jobs/1234567)
• Deduplicate listings using the requisition ID
• For listings without a clear ATS URL, apply fuzzy matching on company name and token‑sorted title
• Rank results, preferring ATS‑hosted apply links for freshness
• Return deduplicated, structured results with title, company, location, seniority, salary, and one direct apply URL per requisition

Evidence

The signal that actually worked was the apply URL. Nearly every listing sits on top of an ATS — Greenhouse, Lever, Ashby — and those URLs carry the canonical requisition ID:

Source

That combination removed almost all duplicates without ever merging two genuinely distinct reqs.

Source

The ATS-native posting vanishes the day a role closes, while aggregators keep serving it for weeks.

Source