Service Architecture

DataKit has many components, but only one long-lived platform service: the Kubernetes controller layer. The dk CLI and SDK packages are client-side utilities and libraries; they prepare artifacts, validate manifests, and drive promotions, but they do not hold platform state.

System Components

Service Boundaries

Operationally, the controller is the platform service boundary.

• CLI: a utility that authors manifests, publishes OCI artifacts, and updates GitOps state.
• SDK: reusable libraries consumed by the CLI and controller.
• Controller: the only continuously running control-plane service.

Today that controller layer is split into two controller-runtime binaries:

• platform/controller/cmd/main.go starts the main controller manager and registers PackageDeployment and DataSet reconcilers.
• platform/controller/cmd/ch-migration/main.go starts the ClickHouse migration controller manager and registers ClickHouseMigrationReconciler.

That split does not change the architectural boundary: both binaries belong to the controller service tier.

How It Works

A promotion does not deploy workloads directly. Instead, it moves desired state through OCI, Git, ArgoCD, and the controller.

    sequenceDiagram
    participant User as Developer or CI
    participant CLI as dk CLI
    participant Registry as OCI Registry
    participant Git as GitHub
    participant Argo as ArgoCD
    participant API as Kubernetes API
    participant Controller as dk-controller
    participant Workload as Job / CronJob / Deployment

    User->>CLI: dk build / dk publish
    CLI->>Registry: Push OCI artifact
    User->>CLI: dk promote --to <env> --cell <cell>
    CLI->>Git: Update GitOps values and open/refresh PR
    Git-->>Argo: Merged desired state becomes visible
    Argo->>API: Apply PackageDeployment CR
    API-->>Controller: Reconcile event
    Controller->>Registry: Resolve digest and download package metadata
    Controller->>API: Create or update Job/CronJob/Deployment
    API-->>Workload: Start workload in target cell namespace
  

Controller Runtime Behavior

Startup

At startup, the main controller binary:

1. builds a shared runtime scheme for core Kubernetes types and DataKit CRDs
2. configures metrics, health probes, and optional leader election
3. creates an ORAS-backed registry client
4. constructs helpers for store resolution, dataset lookup, and lineage reporting
5. registers reconcilers with the manager
6. starts the controller-runtime manager and its shared caches

The migration binary follows the same pattern, but only wires the ClickHouseMigrationReconciler and a default ClickHouse connector.

Registered Controllers

Reconciliation Loops

PackageDeploymentReconciler is phase-driven:

• Pending: validates package reference and moves to Pulling
• Pulling: resolves the OCI reference and digest
• Ready: dispatches by mode to batch, streaming, or semantic handling
• Running: monitors Jobs or Deployments
• Completed / Failed: terminal handling, with bounded retry for pull failures

DataSetReconciler is intentionally minimal. It only checks whether spec.store resolves in the namespace and reflects that in status.

ClickHouseMigrationReconciler is order-aware. It blocks on lower-order migrations for the same store and database, resolves the Store plus backing Secret, interpolates ${VAR} placeholders from envFrom, executes DDL, and records Applied and Ready conditions.

    sequenceDiagram
    participant API as Kubernetes API
    participant Queue as controller-runtime work queue
    participant Reconciler as PackageDeploymentReconciler
    participant Registry as OCI Registry
    participant K8s as Kubernetes API

    API-->>Queue: PackageDeployment event
    Queue->>Reconciler: Reconcile(name, namespace)
    Reconciler->>K8s: Read PackageDeployment status/spec
    alt phase = Pulling
        Reconciler->>Registry: Resolve ref and digest
        Reconciler->>K8s: Update status to Ready
    else phase = Ready
        Reconciler->>K8s: Create/Update Job, CronJob, or Deployment
        Reconciler->>K8s: Update status to Running
    else phase = Running
        Reconciler->>K8s: Inspect workload state
        Reconciler->>K8s: Update LastRun / terminal phase
    end
  

Request Lifecycle

A package deployment crosses several boundaries:

1. the CLI publishes a package to an OCI registry
2. the CLI updates GitOps state for an environment and cell
3. ArgoCD syncs the shared chart output into the cluster
4. a PackageDeployment CR appears in Kubernetes
5. the controller resolves the referenced package and target cell namespace
6. the controller injects resolved DK_STORE_* environment variables from Store CRs
7. the controller creates a Job, CronJob, Deployment, or semantic resources
8. the controller keeps status current as workloads run or fail

For batch workloads, the controller requeues and monitors the newest matching Job. For streaming workloads, it creates or updates a Deployment and monitors rollout state. For semantic packages, it writes ConfigMaps or Cube-related resources instead of process workloads.

Concurrency Model

DataKit relies on controller-runtime’s standard concurrency model:

• watch events are placed onto a rate-limited work queue
• reconciles are keyed by object name and namespace
• each reconcile is expected to be idempotent because the same object can be requeued repeatedly
• transient failures are represented either by returned errors or explicit RequeueAfter delays
• optional leader election ensures only one manager instance actively reconciles when the flag is enabled

Leader election is configured but disabled by default in both controller binaries. When enabled, only the elected instance processes queue items for that manager.

Failure Modes

OCI registry unreachable

If the registry cannot resolve a package reference, handlePulling sets the Ready condition reason to PullFailed, marks the deployment Failed, and records the failed pull. handleFailed retries that case up to five times, requeueing after 30 seconds each time. After the retry cap, the object stays failed until the spec changes.

ClickHouse unavailable

If the migration controller cannot connect or execute DDL, it sets Applied=False with reason ExecutionFailed, keeps status.applied=false, and requeues after 60 seconds. Store lookup or env interpolation problems also stay non-terminal and are retried after 30 seconds.

Kubernetes API errors

Read, create, update, or status update failures are returned from reconcile when they cannot be handled locally. In that case controller-runtime requeues the key according to its work-queue backoff behavior. For some recoverable cases, the reconcilers also set explicit conditions before requeueing so the CR reflects the current failure state.

See Also

• Data Model
• Design Decisions
• Diagrams