Kestra 2.0 Engine Rewrite and Deployment Flexibility

Description and use

Kestra 2.0 separates the system into control and data planes, enabling workers to run in isolated networks without database access, improving flexibility and security.

Problem

Organizations struggled with Kestra’s heavy architecture requiring Kafka and Elasticsearch clusters, limiting deployment flexibility and causing security teams to avoid using it.

Outcome

Kestra 2.0 enables secure, scalable orchestration in isolated or multi-cloud environments without requiring heavy infrastructure, improving adoption and reducing operational friction.

Workflow

• 1. Deploy control plane (scheduler, executor, webserver, indexer, worker controller) in desired location.
• 2. Configure queue (e.g., JDBC) and repository (e.g., PostgreSQL) for control plane.
• 3. Deploy workers in target environment (cloud, on-prem, isolated network).
• 4. Workers open persistent gRPC stream to worker controller; authenticate via TLS, client cert or JWT.
• 5. Submit jobs from control plane; workers execute and return results over same stream.
• 6. Monitor via UI or logs; adjust quotas, trigger conditions, or loop constructs as needed.

Evidence

In 1.x, every worker needed a connection to the central database. That single design decision dictated where you were allowed to run tasks.

Source

2.0 splits Kestra into a control plane and a data plane. The executor, the scheduler, the webserver, the indexer and a new worker controller live in the control plane, and none of them run user code.

Source