Ghostlog: a local séance for dead data pipelines

说明与用途

Ghostlog 将遗忘的 ETL 作业工件转换为离线墓碑页面，保留作业上下文、表线性和叙述语音，帮助未来团队成员理解并维护数据管道。

问题

Data teams often lose context for retired ETL jobs, making it hard to understand what a job did, why it failed, and how it fit into the data pipeline.

结果

Ghostlog turns forgotten job artifacts into a clear, offline tombstone page that preserves job context, lineage, and a narrative voice, enabling future team members to understand and maintain the data pipeline.

工作流

• Point Ghostlog at a log file, Airflow log, or .sql file
• Ghostlog parses the file to extract job name, run nights, start time, SQL statements, and error cause
• Ghostlog uses sqlglot to build a source‑to‑target table lineage
• Ghostlog sends extracted facts to Gemma via Ollama’s local HTTP API, constrained by a JSON schema
• Gemma returns a three‑line testimony and a three‑line epitaph in the job’s voice
• Ghostlog writes a self‑contained HTML tombstone page with the job’s years, epitaph, testimony, and lineage

证据

Ghostlog holds a séance for that job. You point it at the remains, either an old cron or Airflow log or a deprecated SQL file, and it: reads the remains: the job's name, how many nights it ran, its usual start time, every SQL statement it executed, and the error that ended it; rebuilds the job's table lineage with sqlglot; asks Gemma, running locally through Ollama, to let the job speak in the first person; carves an ofrenda-style tombstone page

来源

Everything runs on your own machine. Your logs and SQL stay on your laptop, and it keeps working with the Wi‑Fi off.

来源

My first runs used gemma3:1b. It had plenty of atmosphere but a loose grip on facts: it once said the job ran "for fourteen days each night." So I split the work. Code computes the facts, and the model only supplies the voice.

来源