Design Decisions

DataKit’s architecture is intentionally constrained. The goal is not to support every historical deployment style; it is to converge on a small, explicit model for packaging, promoting, and running data work.

Why ADRs

ADR 001 adopts Architecture Decision Records so major decisions are recorded in version control with context, alternatives, and consequences. That matters in DataKit because the project already has target-state and vision documents; ADRs add the missing decision trail that explains why a choice exists and whether it is still active.

The practical effect is straightforward:

important architecture changes should check existing ADRs first
accepted ADRs are treated as stable until explicitly superseded
conflicts with target-state documents should trigger a deliberate architecture discussion instead of ad hoc drift

Environment and Cell Model

ADR 002 separates two kinds of isolation:

environments are compute isolation boundaries, typically separate clusters or accounts
cells are workload and tenant isolation boundaries inside an environment

This model keeps deployment routing simple:

environments answer which cluster a package belongs in
cells answer which namespace and store set logical names resolve against

Cells are flat, not nested. That choice keeps store resolution, RBAC, GitOps layout, and naming conventions predictable.

GitOps-Native Deployment

The target-state documents choose GitOps as the deployment control plane.

The rationale is architectural, not cosmetic:

desired state lives in Git, not in imperative controller-specific APIs
promotion becomes a reviewed change to environment or cell state
ArgoCD applies PackageDeployment resources consistently from shared chart output
rollback is a Git operation, not a special platform escape hatch

This also keeps the runtime service narrow. The controller reconciles declared state; it does not become the source of truth for what should exist.

Manifest-Driven Pipelines

DataKit deliberately avoids a separate pipeline manifest.

Instead, the graph is derived from existing objects:

Transform.spec.inputs names upstream datasets
Transform.spec.outputs names downstream datasets
DataSet.spec.store anchors those datasets to infrastructure

That means package authors describe contracts and computation, and the platform derives orchestration relationships from those references. The vision documents reinforce this with the canonical chain Connector -> Store -> DataSet -> Transform.

Multi-Tenancy via Cells

Cells are the platform’s main multi-tenancy primitive within an environment.

ADR 002 makes two important choices here:

the cell is a logical boundary, not a single physical primitive
the Store is what maps that logical boundary to physical addresses

So the same cell identity can map to:

a Kubernetes namespace for compute
a topic suffix for Kafka
a prefix or bucket segment for S3
a database boundary for PostgreSQL

This gives DataKit a single mental model for tenant isolation even though the backing systems differ.

Naming and Derivation by Default

The environment and cell ADR also chooses derivation over repeated configuration.

By default, physical names are derived from namespace, dataset, and cell rather than copied into every manifest. Explicit names are still allowed, but they are treated as exceptions and can be linted.

That decision reduces boilerplate and makes cross-system naming rules enforceable.

OCI as the Package Boundary

The architecture documents consistently treat OCI artifacts as the package transport boundary.

That enables:

immutable package references
digest verification
reuse of Harbor and ORAS-compatible tooling
a clean separation between build/publish and deploy/reconcile

OCI is therefore not just a storage choice; it is the handoff point between authoring and runtime.

Pre-Release Philosophy

DataKit is explicitly pre-release. The project instructions and target-state material bias the platform toward clean replacement over compatibility shims.

Architecturally, that means:

obsolete concepts should be removed rather than indefinitely deprecated
generators and manifest schemas can evolve quickly when the new model is cleaner
the documentation should describe the intended steady-state model, not preserve every interim behavior forever

This is why the docs emphasize the current manifest set, derived pipeline model, and cell-based deployment layout rather than legacy alternatives.