Architecture

DataKit is a manifest-driven platform for building, publishing, promoting, and running data pipelines. Its architecture has a deliberately small service surface: the long-lived platform runtime is the Kubernetes controller layer, while the CLI and SDK provide packaging, validation, local execution, and promotion utilities.

This section breaks the platform into three views:

• Service Architecture explains the runtime topology, reconciliation loops, deployment path, and failure handling.
• Data Model explains the manifest kinds, CRDs, validation rules, and lifecycle from YAML to workloads.
• Design Decisions summarizes the ADRs and target-state choices that shape the platform.

The architecture is centered on a few core ideas:

• packages are stored as OCI artifacts
• deployments are GitOps-driven through ArgoCD
• pipeline graphs are derived from manifest relationships rather than a separate pipeline.yaml
• cells provide intra-environment isolation for workloads, stores, and tenancy
• the project is pre-release, so the design favors clean replacement over compatibility shims

The pages in this section are written for contributors who need to understand how DataKit behaves in production, where responsibilities stop and start, and which decisions are already settled.

Next Steps

• Start with Service Architecture for the end-to-end runtime flow.
• Continue to Data Model for manifest and CRD relationships.
• Use Design Decisions to check whether a change conflicts with existing ADRs.