Why we stopped using one Elasticsearch index per tenant and built our own

Description and use

SeaSearch replaces per‑tenant Elasticsearch indexes with a shared S3 storage and ownership routing, eliminating cluster state bloat and enabling unlimited tenants.

Problem

Managing thousands of tenant‑specific Elasticsearch indexes caused cluster state bloat, slow master node performance, and limited scalability.

Outcome

SeaSearch removes the need for one index per tenant, reducing cluster state size, improving master node performance, and enabling unlimited tenants while maintaining Elasticsearch compatibility.

Workflow

• 1. Deploy SeaSearch compute nodes that read/write to a shared S3 bucket.
• 2. Store index metadata and partition ownership map in etcd.
• 3. Cluster manager monitors node health and updates partition‑to‑node mapping in etcd.
• 4. Proxy receives client requests, looks up owner node from etcd, and forwards the request.
• 5. Compute nodes cache immutable segments locally; on cache miss, fetch segments from S3 concurrently (parallel warm‑up).
• 6. For large indexes, split queries across multiple compute nodes and aggregate results.
• 7. Use unified_search endpoint to score across multiple indexes consistently.

Evidence

Every index costs at least one shard, and each shard is basically its own mini Lucene database... A few thousand indexes and mapping updates get sluggish.

Source

Compute nodes hold no authoritative data, so failover is a map update rather than a data migration.

Source

Parallel warm‑up. Modern datacenter bandwidth is enormous and a single‑threaded fetch wastes it.

Source

SeaSearch can split a single query across multiple compute nodes.

Source