My job search API returned the same job five times — the apply URL gave away the dedup key

Описание и применение

При поиске вакансий API возвращал дубли, потому что одно объявление публиковалось через несколько агрегаторов. Использование URL‑ссылки на ATS позволило удалить дубли и улучшить актуальность результатов.

Задача

Job search API returned duplicate listings for the same requisition, causing confusion and inaccurate results.

Результат

Deduplication using the apply URL’s requisition ID eliminated duplicate job listings and improved result freshness by prioritizing ATS-hosted postings.

Рабочий процесс

• Query job search API with keywords and location
• Collect listings from multiple aggregators and company board
• Extract apply URL from each listing
• Parse canonical requisition ID from the URL (e.g., boards.greenhouse.io/company/jobs/1234567)
• Deduplicate listings using the requisition ID
• For listings without a clear ATS URL, apply fuzzy matching on company name and token‑sorted title
• Rank results, preferring ATS‑hosted apply links for freshness
• Return deduplicated, structured results with title, company, location, seniority, salary, and one direct apply URL per requisition

Доказательства

The signal that actually worked was the apply URL. Nearly every listing sits on top of an ATS — Greenhouse, Lever, Ashby — and those URLs carry the canonical requisition ID:

Источник

That combination removed almost all duplicates without ever merging two genuinely distinct reqs.

Источник

The ATS-native posting vanishes the day a role closes, while aggregators keep serving it for weeks.

Источник