Target State — Operating Model¶

Two personas, one platform¶

DataKit serves two distinct roles with clear ownership boundaries.

┌─────────────────────────────────────────────────────────────────────────────┐
│                           Platform Engineer                                 │
│                                                                             │
│   connector/                   environments/            policies/           │
│   ├── postgres.yaml            ├── dev.yaml              ├── quality.yaml   │
│   ├── s3.yaml                  ├── stage.yaml            └── versions.yaml  │
│   └── kafka.yaml               └── prod.yaml                               │
│                                                                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                            Data Engineer                                    │
│                                                                             │
│   store/                       dataset/                 transforms/         │
│   ├── warehouse.yaml           ├── users.yaml           └── aws_compliance/ │
│   └── lake-raw.yaml            ├── users-parquet.yaml       └── dk.yaml    │
│                                └── orders.yaml                              │
│                                                                             │
│   models/                                                                   │
│   └── dbt/                                                                  │
│       ├── staging/                                                          │
│       └── marts/                                                            │
└─────────────────────────────────────────────────────────────────────────────┘

Four layers¶

The platform separates concerns into four distinct layers. Each layer references the one above it by name — never by embedding its internals.

Connector ──▶ Store ──▶ DataSet ──▶ Transform ──▶ Binding
  (type)      (instance)  (contract)  (compute)    (infra)

1. Connectors — technology types¶

A Connector is a technology type definition maintained by the platform team. It declares what a storage technology is — Postgres, S3, Kafka, etc. — and which CloudQuery plugin images to use.

# connector/postgres.yaml
apiVersion: datakit.infoblox.dev/v1alpha1
kind: Connector
metadata:
  name: postgres
spec:
  type: postgres
  protocol: postgresql
  capabilities: [source, destination]

Connectors rarely change. They define the technology catalog available to the platform.

2. Stores and DataSets — data contracts¶

A Store is a named instance of a Connector with connection details and credentials, managed by infra/SRE. A DataSet is a named data contract — a table, S3 prefix, or Kafka topic that lives in a Store — created by data engineers.

# dataset/users.yaml
apiVersion: datakit.infoblox.dev/v1alpha1
kind: DataSet
metadata:
  name: users
spec:
  store: warehouse
  table: public.users
  classification: confidential
  schema:
    - name: id
      type: integer
    - name: email
      type: string
      pii: true

DataSets are declarative metadata: schema, classification, and lineage. Validation runs at dk lint time — errors surface before runtime.

3. Pipelines — reactive dependency graph¶

A pipeline is the dependency graph derived from Transform and DataSet manifests. Each Transform declares its inputs and outputs; the graph is built automatically.

# View the full pipeline graph
dk pipeline show

# View the chain leading to a specific DataSet
dk pipeline show --destination event-summary

Transforms are independently deployable. Each declares a trigger (schedule, on-change, manual) and references DataSets by name. There is no separate pipeline manifest — the graph emerges from the individual dk.yaml files.

4. Bindings — per-environment infrastructure¶

Bindings map abstract DataSet references to concrete infrastructure, varying by environment. A pipeline's definition never changes between dev and prod — only its bindings do.

# environments/prod.yaml (excerpt)
bindings:
  snowflake_raw:
    type: snowflake
    account: acme.us-east-1
    database: RAW
    schema: SECURITY
    role: LOADER

Environments and policies¶

Managed environments¶

Platform engineers define environments with allowed connectors, version constraints, approval workflows, and resource quotas.

Policies¶

Declarative YAML policies enforce guardrails at dk validate time:

• versions.yaml — prod requires exact version pins; dev allows ranges
• quality.yaml — gold-tier outputs require tests and documentation
• security.yaml — all outputs require PII classification metadata

Developer workflows¶

Platform engineer¶

# Define connectors and stores
# Edit connector/ and store/ manifests
dk lint

Data engineer¶

dk dataset create users --store warehouse --table public.users
dk dataset create users-parquet --store lake-raw --prefix data/users/
# Edit DataSet schemas and dbt models
dk lint
dk run
dk pipeline show

RACI summary¶

Rule of thumb: platform owns how it runs safely; data owns what runs and what it produces.

Evolution path¶

The target state is reached incrementally through five features, each independently shippable:

The existing dk init / dk build / dk run workflow continues to work throughout — it is progressively refactored under the hood as each feature lands.