Getting Started

DataKit operators deploy two related pieces: the dk-controller Helm chart that runs the controller manager, and the optional dk-argocd chart that creates the ArgoCD AppProject and ApplicationSet resources. The controller watches PackageDeployment and DataSet resources; when chMigration.enabled is on, a sidecar also reconciles ClickHouseMigration resources.

Control-plane bootstrap flow

    flowchart LR
    A[Build/publish controller images] --> B[Install dk-controller chart]
    B --> C[Install dk-argocd chart]
    C --> D[ArgoCD discovers gitops/envs/*/cells/*/apps/*]
    D --> E[Application sync creates PackageDeployment CRs]
    E --> F[dk-controller resolves package, stores, and workloads]
  

1. Deploy the controller Helm chart

The controller chart name and deployment name are both dk-controller, and the main deployment always runs in the namespace selected by values.yaml or the Helm release namespace (charts/dk-controller/templates/_helpers.tpl).

Typical operator sequence:

# Build images locally when testing chart changes
make build-controller-image
make build-ch-migration-image

# Or build/push dev-tagged images for Harbor
make release-controller-dev
make release-ch-migration-dev

# Lint and package the chart
make helm-lint
make build-helm-package

# Install or upgrade the controller
helm upgrade --install dk-controller charts/dk-controller \
  --namespace dk-system --create-namespace

Important chart defaults from charts/dk-controller/values.yaml:

The main controller container runs with --health-probe-bind-address=:8081 and --metrics-bind-address=:8080 from the deployment template; the sidecar uses :8083 for health and :8082 for metrics.

2. Install ArgoCD integration

charts/dk-argocd does not install ArgoCD itself. It only creates the ArgoCD custom resources that tell an existing ArgoCD installation how to watch DataKit repos (charts/dk-argocd/README.md).

Use the repo’s Make targets first:

# Review the generated AppProject/ApplicationSet manifests
make argocd-template

# Install or upgrade the ArgoCD integration layer
make argocd-install

Equivalent Helm command:

helm upgrade --install dk-argocd charts/dk-argocd --namespace dk-system

By default the ApplicationSet watches:

• generator repo: https://github.com/Infoblox-CTO/dk-dataeng.git
• chart repo: https://github.com/Infoblox-CTO/platform.data.kit.git
• discovery path: gitops/envs/*/cells/*/apps/*
• destination namespace pattern: dk-{env}-{cell}

If you enable chMigrations.enabled, a second ApplicationSet discovers migrations/* directories and syncs store manifests plus *.cr.yaml ClickHouse migration resources.

3. Define environments and cells

ADR-002 defines the isolation model: environments are compute boundaries (typically separate clusters/accounts), while cells are flat isolation boundaries within an environment. The fully qualified identity is {env}/{cell}.

Typical layout in the GitOps repo:

gitops/envs/
  dev/cells/c0/apps/<package>/values.yaml
  int/cells/c0/apps/<package>/values.yaml
  prod/cells/canary/apps/<package>/values.yaml

Cell conventions from the CRD and ADR:

• the Cell CR is cluster-scoped
• spec.namespace points to the namespace that contains that cell’s Store CRs
• cell names are unique per cluster, but the same cell name can exist in each environment
• cells are peers, not nested hierarchies

Example cell object shape:

apiVersion: datakit.infoblox.dev/v1alpha1
kind: Cell
metadata:
  name: canary
spec:
  namespace: dk-canary
  labels:
    scope: canary

4. Configure stores for each cell

Packages reference stores by logical name; operators supply the physical connection details by creating Store resources in the cell namespace.

Minimal store shape from the CRD:

apiVersion: datakit.infoblox.dev/v1alpha1
kind: Store
metadata:
  name: warehouse
  namespace: dk-canary
spec:
  connector: postgres
  connection:
    host: pg-rw.analytics.svc.cluster.local
    port: "5432"
    database: dk_canary
    user: dk
  secretRef:
    name: warehouse-secret
    keys:
      password: password

For scaffolding, the CLI command is source-backed in cli/cmd/store_create.go:

dk store create warehouse --connector postgres \
  --secret-name warehouse-secret \
  --secret-key password=password \
  --set host=pg-rw.analytics.svc.cluster.local \
  --set port=5432 \
  --set database=dk_canary \
  --set user=dk

5. Verify the installation

After the controller and ArgoCD layers are installed, validate the cluster state:

kubectl -n dk-system get deployment dk-controller
kubectl -n dk-system get pods -l app.kubernetes.io/component=controller
kubectl get cells
kubectl get stores -A
kubectl get applications -n dk-system

If ClickHouse migrations are enabled:

kubectl -n dk-system get pods -l app.kubernetes.io/component=controller
kubectl get clickhousemigrations -A

A healthy install shows the deployment ready, the Cell objects present, and Store objects resolving inside the correct cell namespaces.

Next Steps

• Configuration
• CLI Reference
• Troubleshooting
• Runbooks