The Missing Piece: Adding Link Extraction to a Two-Pass Grading Pipeline

Description and use

The article details adding a link extraction component to a two‑pass grading pipeline, covering GitHub, YouTube, and article links with domain‑specific extractors, LLM‑based metadata enrichment, and LangGraph orchestration.

Problem

The knowledge graph pipeline lacked link content, causing users to manually re‑read shared links and increasing friction.

Outcome

The integrated link scraper captures 40% of message content that was previously missing, reduces user effort by providing concise metadata, and operates efficiently within the existing grading pipeline using domain‑specific extractors and a local LLM.

Workflow

• Identify link type via regex or URL pattern
• For GitHub links: use GitHub REST API to fetch repo data; if rate limited or 403, fallback to Playwright scraping
• For YouTube links: use youtube‑transcript‑api; if transcript missing, fallback to Supadata
• For article links: use Playwright + enhancer to capture page content unless blocked by auth wall; record blocked links with reason
• Enrich extracted data with a local LLM to produce intention, topics, entities, and summary
• Merge enriched link data with first‑pass grading results; run second‑pass grading in parallel; merge final states before ingestion

Evidence

From our data, we found roughly 40% of the total messages contain at least one link embedded in them.

Source

For article data, we are using Playwright, but only if the article does not have a login wall or bot detection.

Source

Using the GitHub REST API for GitHub links, we first verify via regex if it is a GitHub link, and after verification, we use the REST API for extracting the contents of the repository.

Source

For YouTube videos, since the content is in the video, we first rely on the youtube‑transcript‑api library. As a fallback, we use Supadata, which mostly provides the transcript of the video.

Source

We achieved this with the help of LangGraph. We wanted to have persistence, because during our tests, we experienced first‑hand how the whole system had to fire up and start from scratch even when the work was halfway done.

Source