Your Iceberg tables are rotting: we measured it, then built a fix (free, Apache 2.0)

Описание и применение

Cryo — инструмент для обслуживания таблиц Iceberg, который оценивает таблицы, планирует целевые операции по разделам, запускает их только при наличии свободных ресурсов и фиксирует все действия, что приводит к успешному обслуживанию, уменьшению количества файлов и ускорению запросов.

Задача

Iceberg tables accumulate small files, delete files, stale statistics, manifest bloat, snapshot pile‑up, and orphan files, causing slow queries and wasted storage, yet failures are silent and jobs often fail to commit.

Результат

Across 4 cycles of a mixed fleet and a 100 GiB table, Cryo performed 151/151 successful jobs, reduced file counts from 7,594 to 690, improved query performance up to 4.34×, and maintained table health while keeping user queries unaffected.

Рабочий процесс

• Scan catalog metadata to grade tables and partitions
• Plan maintenance per partition based on small‑file count and GB budget
• Admit jobs only when engine has free slots and table is not paused or in cooldown
• Execute operations in order: compaction (convert deletes, rewrite position deletes, optimize), statistics, cleanup (manifests, snapshot expiry, orphans)
• Log audit rows for each operation, including deferred or failed runs

Доказательства

We put the scheduled Spark compaction job that most Iceberg teams write on a table taking a MERGE every 200 seconds for two hours. It tried three times. It committed zero times.

Источник

Cryo's scanner reads catalog metadata only, never data files. For each table and partition it records file counts and sizes, snapshot and manifest counts, and delete files.

Источник

Cryo made 22/22 jobs succeed, and it held the table at 3 to 6 delete files from its second run.

Источник

Dashboard-style Trino queries ran while Cryo rewrote 67 GiB. Their p95 stayed between 0.88x and 1.04x of idle, with zero failures.

Источник

Across all tests, 151/151 Cryo jobs succeeded and 0 rows were lost or changed.

Источник

About 2x the DAG's Spark compute on small tables, and 1.67x hand-run SQL on the 100 GiB table.

Источник

Jobs waited a median of about 3 minutes in the queue.

Источник