Data Model

DataKit uses two closely related models:

Data Model

Manifest Kinds and Relationships

The core manifest graph is derived, not separately declared.

Kind	Purpose	Key relationship
`Connector`	Technology type such as Postgres, S3, or Kafka	`Store.spec.connector` points at the connector provider identity
`Store`	Named infrastructure instance plus connection data and secrets	a `DataSet` lives in exactly one store
`DataSet`	Data contract for a table, prefix, or topic	`Transform` inputs and outputs reference datasets by name
`DataSetGroup`	Bundle of multiple datasets from one materialization	groups dataset names under one store
`Transform`	Executable computation unit	consumes input datasets and produces output datasets
`SemanticProject`	Semantic-layer package root	references semantic models and produces semantic deployment artifacts

The default dependency chain is:

Connector -> Store -> DataSet -> Transform

Important implications:

there is no pipeline.yaml
the pipeline graph is derived from Transform.spec.inputs and Transform.spec.outputs
stores hold secrets and connection details; datasets and transforms do not
cell selection changes how logical names resolve to physical infrastructure, not the logical graph itself

PackageDeployment is the controller’s execution intent object. It links an OCI package reference to a target cell and an execution mode.

Field	Meaning
`spec.package`	OCI package reference: name, namespace, version, registry, optional digest
`spec.cell`	target `Cell` name; controller resolves the cell to a Kubernetes namespace
`spec.mode`	`batch`, `streaming`, or `semantic`
`spec.schedule`	cron schedule for recurring batch runs
`spec.replicas`	desired replica count for streaming mode
`spec.timeout`	max run duration for batch mode
`spec.resources`	CPU and memory requests/limits
`spec.serviceAccountName`	pod service account
`spec.imagePullSecrets`	registry auth secrets
`spec.env`	deploy-time environment overrides

Field	Meaning
`status.phase`	`Pending`, `Pulling`, `Ready`, `Running`, `Completed`, `Failed`
`status.conditions`	detailed controller observations
`status.package`	resolved OCI digest and fully qualified reference
`status.lastRun`	run id, start/end times, result, records processed
`status.failureCount`	consecutive workload failures
`status.pullRetryCount`	consecutive pull failures
`status.observedGeneration`	last reconciled spec generation

ClickHouseMigration does not use an explicit phase enum. Its lifecycle is inferred from status and conditions.

Conceptual state	Observed fields
Pending	CR exists, `status.applied=false`, required dependencies may not be satisfied yet
Running	controller is actively resolving stores, secrets, env vars, or executing DDL
Completed	`status.applied=true`, `status.appliedAt` set, `Applied=True`, `Ready=True`
Failed	`status.applied=false`, with conditions such as `StoreResolutionFailed`, `EnvResolutionFailed`, or `ExecutionFailed`

Additional lifecycle rules:

migrations are ordered within the same store + database scope using spec.order
the applied SQL hash includes both SQL text and envFrom, so changing either triggers re-application
deleting the CR does not drop tables; deletion is intentionally a no-op for safety

Inside the cluster, the main controller-side relationships are:

Cell maps a logical deployment target to a Kubernetes namespace
Store and DataSet live in that namespace
PackageDeployment.spec.cell selects the namespace where stores resolve and workloads run
ClickHouseMigration.spec.store binds DDL execution to a store in the same namespace

That means the same logical package can be promoted to different cells while resolving different physical stores.

Validation is split between generic validator helpers and kind-specific manifest rules. The most important architecture-level codes are below.

Code	Meaning
`E001`	metadata name is not DNS-safe
`E020`	version is not valid SemVer
`E040`	`spec.runtime` is required
`E041`	`spec.image` is required for generic runtimes
`E200`	connector type is required
`E210`	store connector is required
`E211`	store connection must be non-empty
`E220`	dataset store is required
`E221`	dataset location is required
`E222`	dataset schema is invalid
`E230`	transform inputs must be non-empty
`E231`	transform outputs must be non-empty
`E232`	transform image is required for generic runtimes
`E240`	dataset group store is required
`E241`	dataset group must list datasets
`E310`	`schemaRef` and inline schema are mutually exclusive
`E311`	`schemaRef` format is invalid
`E312`	schema lock entry is missing
`E313`	schema lock checksum failed
`E314`	breaking schema change detected

The data model becomes runtime state in stages:

See the Service Architecture — Reconciliation Loop for the full sequence diagram of this lifecycle.

In other words: