The Missing Piece: Adding Link Extraction to a Two-Pass Grading Pipeline

Описание и применение

В статье описывается добавление к существующей системе оценки сообщений двух проходов компонента, извлекающего контент из ссылок на GitHub, YouTube и статьи. Используются специализированные экстракторы, LLM для обогащения метаданных и LangGraph для параллельной обработки и сохранения промежуточных состояний.

Задача

The knowledge graph pipeline lacked link content, causing users to manually re‑read shared links and increasing friction.

Результат

The integrated link scraper captures 40% of message content that was previously missing, reduces user effort by providing concise metadata, and operates efficiently within the existing grading pipeline using domain‑specific extractors and a local LLM.

Рабочий процесс

• Identify link type via regex or URL pattern
• For GitHub links: use GitHub REST API to fetch repo data; if rate limited or 403, fallback to Playwright scraping
• For YouTube links: use youtube‑transcript‑api; if transcript missing, fallback to Supadata
• For article links: use Playwright + enhancer to capture page content unless blocked by auth wall; record blocked links with reason
• Enrich extracted data with a local LLM to produce intention, topics, entities, and summary
• Merge enriched link data with first‑pass grading results; run second‑pass grading in parallel; merge final states before ingestion

Доказательства

From our data, we found roughly 40% of the total messages contain at least one link embedded in them.

Источник

For article data, we are using Playwright, but only if the article does not have a login wall or bot detection.

Источник

Using the GitHub REST API for GitHub links, we first verify via regex if it is a GitHub link, and after verification, we use the REST API for extracting the contents of the repository.

Источник

For YouTube videos, since the content is in the video, we first rely on the youtube‑transcript‑api library. As a fallback, we use Supadata, which mostly provides the transcript of the video.

Источник

We achieved this with the help of LangGraph. We wanted to have persistence, because during our tests, we experienced first‑hand how the whole system had to fire up and start from scratch even when the work was halfway done.

Источник