Your Iceberg tables are rotting: we measured it, then built a fix (free, Apache 2.0)

Description and use

Cryo is an Iceberg maintenance tool that grades tables, plans per‑partition operations, admits jobs only when the engine has headroom, runs operations in a fixed order, and logs everything, resulting in successful maintenance, fewer files, and faster queries.

Problem

Iceberg tables accumulate small files, delete files, stale statistics, manifest bloat, snapshot pile‑up, and orphan files, causing slow queries and wasted storage, yet failures are silent and jobs often fail to commit.

Outcome

Across 4 cycles of a mixed fleet and a 100 GiB table, Cryo performed 151/151 successful jobs, reduced file counts from 7,594 to 690, improved query performance up to 4.34×, and maintained table health while keeping user queries unaffected.

Workflow

• Scan catalog metadata to grade tables and partitions
• Plan maintenance per partition based on small‑file count and GB budget
• Admit jobs only when engine has free slots and table is not paused or in cooldown
• Execute operations in order: compaction (convert deletes, rewrite position deletes, optimize), statistics, cleanup (manifests, snapshot expiry, orphans)
• Log audit rows for each operation, including deferred or failed runs

Evidence

We put the scheduled Spark compaction job that most Iceberg teams write on a table taking a MERGE every 200 seconds for two hours. It tried three times. It committed zero times.

Source

Cryo's scanner reads catalog metadata only, never data files. For each table and partition it records file counts and sizes, snapshot and manifest counts, and delete files.

Source

Cryo made 22/22 jobs succeed, and it held the table at 3 to 6 delete files from its second run.

Source

Dashboard-style Trino queries ran while Cryo rewrote 67 GiB. Their p95 stayed between 0.88x and 1.04x of idle, with zero failures.

Source

Across all tests, 151/151 Cryo jobs succeeded and 0 rows were lost or changed.

Source

About 2x the DAG's Spark compute on small tables, and 1.67x hand-run SQL on the 100 GiB table.

Source

Jobs waited a median of about 3 minutes in the queue.

Source