Hosting an Entire Open Lakehouse on Port 8080

Description and use

A fully local lakehouse built with Spark 4.0.1, Iceberg 1.10.0, SeaweedFS, and PostgreSQL, with exact dependency pinning for stable demos and personal data storage.

Problem

Running a fully functional lakehouse with the latest Spark and Iceberg releases is fragile because dependency mismatches break the stack silently between builds and demos.

Outcome

The lakehouse runs reliably on a single machine, enabling rapid iteration of Spark and Iceberg releases, stable demos, and eventual local data storage.

Workflow

• Set up Postgres locally for Iceberg JDBC catalog
• Deploy SeaweedFS locally to provide S3‑compatible object storage
• Configure Spark with Iceberg extensions and JDBC catalog settings
• Pin Hadoop, AWS SDK v1 and v2, and Spark runtime to compatible versions
• Tweak s3a Hadoop settings to point to SeaweedFS and enable plain HTTP
• Iteratively test and resolve ClassNotFoundException/NoSuchMethodError failures
• Create a namespace and table in Iceberg to verify write/read
• Containerize Spark with host networking for easy version swaps
• Use Docker Compose to orchestrate Spark, Postgres, and SeaweedFS

Evidence

Most of what goes wrong when you upgrade a lakehouse isn't the engine; it's the dependency graph around it

Source

Hadoop 3.4.1 → AWS SDK v2.24.6 (not newer!)

Source

spark.hadoop.fs.s3a.endpoint http://localhost:8333

Source

spark-master: image: apache/spark:4.0.1-scala2.13-java17-python3-r-ubuntu network_mode: "host"

Source