Hosting an Entire Open Lakehouse on Port 8080

说明与用途

构建了一个完全本地的 lakehouse，使用 Spark 4.0.1、Iceberg 1.10.0、SeaweedFS 和 PostgreSQL，精确锁定依赖版本以实现稳定演示和个人数据存储。

问题

Running a fully functional lakehouse with the latest Spark and Iceberg releases is fragile because dependency mismatches break the stack silently between builds and demos.

结果

The lakehouse runs reliably on a single machine, enabling rapid iteration of Spark and Iceberg releases, stable demos, and eventual local data storage.

工作流

• Set up Postgres locally for Iceberg JDBC catalog
• Deploy SeaweedFS locally to provide S3‑compatible object storage
• Configure Spark with Iceberg extensions and JDBC catalog settings
• Pin Hadoop, AWS SDK v1 and v2, and Spark runtime to compatible versions
• Tweak s3a Hadoop settings to point to SeaweedFS and enable plain HTTP
• Iteratively test and resolve ClassNotFoundException/NoSuchMethodError failures
• Create a namespace and table in Iceberg to verify write/read
• Containerize Spark with host networking for easy version swaps
• Use Docker Compose to orchestrate Spark, Postgres, and SeaweedFS

证据

Most of what goes wrong when you upgrade a lakehouse isn't the engine; it's the dependency graph around it

来源

Hadoop 3.4.1 → AWS SDK v2.24.6 (not newer!)

来源

spark.hadoop.fs.s3a.endpoint http://localhost:8333

来源

spark-master: image: apache/spark:4.0.1-scala2.13-java17-python3-r-ubuntu network_mode: "host"

来源