My job search API returned the same job five times — the apply URL gave away the dedup key

说明与用途

招聘搜索 API 返回重复列表，因为同一职位在多个聚合器上发布。使用 ATS URL 的 ID 消除了重复并提高了结果的新鲜度。

问题

Job search API returned duplicate listings for the same requisition, causing confusion and inaccurate results.

结果

Deduplication using the apply URL’s requisition ID eliminated duplicate job listings and improved result freshness by prioritizing ATS-hosted postings.

工作流

• Query job search API with keywords and location
• Collect listings from multiple aggregators and company board
• Extract apply URL from each listing
• Parse canonical requisition ID from the URL (e.g., boards.greenhouse.io/company/jobs/1234567)
• Deduplicate listings using the requisition ID
• For listings without a clear ATS URL, apply fuzzy matching on company name and token‑sorted title
• Rank results, preferring ATS‑hosted apply links for freshness
• Return deduplicated, structured results with title, company, location, seniority, salary, and one direct apply URL per requisition

证据

The signal that actually worked was the apply URL. Nearly every listing sits on top of an ATS — Greenhouse, Lever, Ashby — and those URLs carry the canonical requisition ID:

来源

That combination removed almost all duplicates without ever merging two genuinely distinct reqs.

来源

The ATS-native posting vanishes the day a role closes, while aggregators keep serving it for weeks.

来源