The Iceberg Table That Was Never Where We Wrote It

Description and use

Apache Hop now reliably reads and writes Apache Iceberg tables without Spark, fixing path, class‑loader, parallel read, and commit issues.

Problem

Apache Hop pipelines were unable to reliably read and write Apache Iceberg tables when running on Hop’s own engine, leading to data being written to unexpected locations, missing Parquet writers, and inconsistent parallel reads.

Outcome

After addressing path resolution, class‑loader isolation, deterministic parallelism, and commit semantics, Hop can now reliably read and write Iceberg tables on its own engine, matching Spark behavior and passing all tests.

Workflow

• 1. Split Iceberg support into shared plugin, reader, writer, and docs PRs.
• 2. Add per‑folder catalog to ensure tables are written to the requested path.
• 3. Implement thread‑classloader helper to expose Parquet writer classes to Iceberg.
• 4. Replace list‑index based parallel split with deterministic file‑name based assignment.
• 5. Enforce single commit after entire pipeline succeeds, with retry on metadata file conflicts.
• 6. Add tests for Windows paths, case‑insensitive paths, spaces, parallel writers, and commit ordering.
• 7. Merge PRs after reviews by Matt Casters and Bart Maertens.

Evidence

I wrote one: write a table with the Spark engine, then read it with Hop's own engine.

Source

Hop's engine said there was no table at that path.

Source

The fix, #8795, gives each folder its own small catalog that stores tables in that folder, so the table now lands exactly at the path you asked for.

Source

When Iceberg starts up, it looks for the code that knows how to write Parquet files and registers it. It does this once.

Source

A small helper that points the thread at the plugin's classes whenever Hop calls into Iceberg.

Source

Instead of using the position in the list, each piece is assigned to a copy based on its file name and where it starts in the file.

Source

Hop commits only once, after the whole pipeline has finished without errors.

Source