Concepts

This glossary defines every domain term used across the DataKit documentation. Each entry is self-linked so you can navigate between related concepts.

Role table: If you are a…

Data Flow Summary

    flowchart LR
    A[dk init] --> B[Transform]
    B --> C[dk build]
    C --> D[OCI Artifact]
    D --> E[dk promote]
    E --> F[PackageDeployment CR]
    F --> G[Controller]
    G --> H[Job/Deployment]
    H --> I[Store]
  

Transform

A Transform is the primary unit of computation in DataKit. It declares inputs, outputs, a Runtime, and a Mode. Transforms are defined in a dk.yaml Manifest and packaged into an Open Container Initiative (OCI) Artifact for deployment.

A Pipeline is the reactive dependency graph derived from Transform and DataSet relationships — there is no separate pipeline.yaml.

DataSet

A DataSet is a data contract that describes a table, S3 prefix, or Kafka topic with schema, column-level classification, and Lineage metadata. Every DataSet references a Store where its data lives. DataSets can be grouped into a DataSetGroup.

DataSetGroup

A DataSetGroup bundles multiple DataSets that are materialized by a single Transform. It allows one transform to declare multiple outputs in a single Manifest.

Store

A Store is a named infrastructure instance with connection details and secret references. Stores belong to a Cell and resolve at deployment time via the controller’s StoreEnvResolver. Each Store references a Connector type.

Connector

A Connector defines a storage technology type (e.g., postgres, s3, kafka, clickhouse). It is a shared resource managed by the platform team. Stores reference a Connector to declare their technology.

Cell

A Cell is an infrastructure isolation boundary within an Environment. Each Cell has its own set of Stores and Kubernetes namespace. The Cell model enables multi-tenancy — different teams or workloads can operate in isolated Cells within the same Environment.

Cells are cluster-scoped Custom Resources (CRs) managed by the DataKit controller.

Environment

An Environment represents a deployment stage (e.g., us-dev-5, us-prod-1). Each Environment contains one or more Cells. Promotion moves a Data Package from one Environment to another via GitOps.

Naming convention: <region>-<stage>-<instance> (see Architecture Decision Record DK-ADR-002).

Data Package

A Data Package is the self-contained unit of work in DataKit. It contains a Transform, DataSet declarations, and configuration. A package is built into an OCI Artifact and deployed via PackageDeployment.

Manifest

A Manifest is the dk.yaml configuration file at the root of every Data Package. It declares the package kind (Transform, DataSet, DataSetGroup, Connector, Store, or SemanticProject), metadata, and specification.

Runtime

A Runtime is the execution engine for a Transform. Supported runtimes:

Mode

A Mode determines how a Transform executes:

• batch — runs as a Kubernetes Job or CronJob on a schedule
• streaming — runs as a long-lived Kubernetes Deployment consuming events continuously

Pipeline

A Pipeline is a reactive dependency graph derived from Transform and DataSet manifests. There is no pipeline.yaml — the graph is computed automatically by resolving input/output DataSet references across Transforms. Visualize with dk pipeline show --scan-dir ..

Artifact

An Artifact is an OCI (Open Container Initiative) image containing a built Data Package. Artifacts are pushed to a registry (Harbor) via dk publish and pulled by the controller during deployment. Artifacts are immutable — once published, a version cannot be modified.

PackageDeployment

A PackageDeployment is a Kubernetes Custom Resource (CR) that the DataKit controller reconciles. It declares which Artifact to deploy, in which Cell, with what Mode. The controller creates Jobs, CronJobs, or Deployments based on the Mode and Runtime.

ClickHouseMigration

A ClickHouseMigration is a Kubernetes CR that applies Data Definition Language (DDL) statements to a ClickHouse Store. Migrations are dependency-ordered and track execution state (pending → running → completed/failed). The controller resolves the target Store connection and applies SQL.

SemanticProject

A SemanticProject is a versioned analytics layer that turns DataSets into a typed catalog of entities, dimensions, and measures. It compiles into a Business Intelligence (BI)-ready runtime artifact using the atlas-cubejs Runtime.

Lineage

Lineage tracks data flow between DataSets and Transforms using the OpenLineage standard. The controller emits lineage events to Marquez during PackageDeployment reconciliation. View lineage with dk show --lineage.

Promotion

Promotion is the GitOps workflow that moves a Data Package Artifact from one Environment to another. dk promote creates or updates a pull request (PR) in the deployment repository. ArgoCD syncs the promoted PackageDeployment CR to the target cluster.

SDK

The Software Development Kit (SDK) is the shared Go library (github.com/Infoblox-CTO/platform.data.kit/sdk) used by the dk CLI (Command-Line Interface) and controller. Key packages: Manifest parsing, validation, OCI registry, Promotion, Lineage, and schema resolution.

Governance

Governance in DataKit enforces data classification (Personally Identifiable Information (PII) tagging), compliance metadata, and protection policies on DataSets. Classification is required — not optional — for every column in a DataSet schema.

See Also

• Developer Guide — build and test pipelines
• Operator Guide — deploy and troubleshoot
• Architecture — system design and decisions