Ghostlog: a local séance for dead data pipelines

Description and use

Ghostlog turns forgotten ETL job artifacts into an offline tombstone page that preserves job context, table lineage, and a narrative voice, enabling future team members to understand and maintain the data pipeline.

Problem

Data teams often lose context for retired ETL jobs, making it hard to understand what a job did, why it failed, and how it fit into the data pipeline.

Outcome

Ghostlog turns forgotten job artifacts into a clear, offline tombstone page that preserves job context, lineage, and a narrative voice, enabling future team members to understand and maintain the data pipeline.

Workflow

• Point Ghostlog at a log file, Airflow log, or .sql file
• Ghostlog parses the file to extract job name, run nights, start time, SQL statements, and error cause
• Ghostlog uses sqlglot to build a source‑to‑target table lineage
• Ghostlog sends extracted facts to Gemma via Ollama’s local HTTP API, constrained by a JSON schema
• Gemma returns a three‑line testimony and a three‑line epitaph in the job’s voice
• Ghostlog writes a self‑contained HTML tombstone page with the job’s years, epitaph, testimony, and lineage

Evidence

Ghostlog holds a séance for that job. You point it at the remains, either an old cron or Airflow log or a deprecated SQL file, and it: reads the remains: the job's name, how many nights it ran, its usual start time, every SQL statement it executed, and the error that ended it; rebuilds the job's table lineage with sqlglot; asks Gemma, running locally through Ollama, to let the job speak in the first person; carves an ofrenda-style tombstone page

Source

Everything runs on your own machine. Your logs and SQL stay on your laptop, and it keeps working with the Wi‑Fi off.

Source

My first runs used gemma3:1b. It had plenty of atmosphere but a loose grip on facts: it once said the job ran "for fourteen days each night." So I split the work. Code computes the facts, and the model only supplies the voice.

Source