Runbooks

These runbooks assume the cluster already has the DataKit CRDs and the dk-controller deployment. Use them for repeatable operational changes and incident recovery.

Promotion workflow

    sequenceDiagram
    participant Op as Operator
    participant CLI as dk promote
    participant Git as GitOps PR
    participant Argo as ArgoCD
    participant Ctrl as dk-controller
    Op->>CLI: dk promote . vX --to dev --cell c0
    CLI->>Git: update gitops/envs/dev/cells/c0/apps/<pkg>/values.yaml
    Git-->>Argo: merged PR
    Argo->>Ctrl: sync creates/updates PackageDeployment
    Ctrl->>Ctrl: resolve package, stores, workload mode

Procedure

Build and publish the package artifact.
Create the GitOps PR with dk promote.
Verify the PR updates the correct env/cell path.
Merge the PR and watch ArgoCD sync.
Verify the PackageDeployment reaches the expected phase.

Commands:

dk build .
dk publish . --registry <registry>
dk promote . v1.2.3 --to dev --cell c0
dk promote status <pr-number>
kubectl get application -n dk-system
kubectl get packagedeployments -n dk-dev-c0

Success criteria

PR targets gitops/envs/<env>/cells/<cell>/apps/<package>/values.yaml
ArgoCD application syncs successfully
PackageDeployment.status.phase moves through Pulling to Ready/Running

Rollback procedure

Rollback is implemented as a promotion to an older version, not a special controller-side action.

Procedure

# Preview the rollback
dk rollback <package> --to prod --cell canary --to-version v1.2.2 --dry-run

# Execute the rollback
dk rollback <package> --to prod --cell canary --to-version v1.2.2

Then verify:

dk promote status <rollback-pr>
kubectl describe application prod-canary-<package> -n dk-system
kubectl describe packagedeployment <package> -n dk-prod-canary

Notes

--to is required
--to-version is required; automatic previous-version detection is not implemented yet
default cell is c0 when --cell is omitted

Controller restart and recovery

Use this when the controller deployment is unhealthy after an upgrade, config change, or transient control-plane outage.

Procedure

capture current pod status and recent logs
confirm the root cause is fixed (chart values, registry secret, CRD, etc.)
restart the deployment
watch readiness and reconcile backlog

kubectl -n dk-system get deployment dk-controller
kubectl -n dk-system logs deployment/dk-controller --tail=200
kubectl -n dk-system rollout restart deployment/dk-controller
kubectl -n dk-system rollout status deployment/dk-controller --timeout=120s

If the ClickHouse sidecar is enabled, also inspect that container explicitly:

kubectl -n dk-system logs deployment/dk-controller -c ch-migration --tail=200

Escalation checklist

if rollout does not complete, compare the deployed image tag with chart values
if probes fail immediately, verify ports 8080/8081 and sidecar 8082/8083
if leader election blocks progress after scaling, leave one replica until the incident is resolved

ClickHouse migration failure recovery

ClickHouseMigration recovery is normally a data fix plus a requeue. The CR is idempotent because applied SQL is tracked by status.sqlHash.

Procedure

inspect the failed migration condition and message
identify whether the issue is ordering, store resolution, env interpolation, or SQL execution
fix the underlying store/secret/SQL
let the reconciler retry, or touch the resource to force a new reconcile

kubectl get clickhousemigrations -n dk-system
kubectl describe clickhousemigration <name> -n dk-system
kubectl get stores -n dk-system
kubectl get secret <secret> -n dk-system -o yaml

Recovery patterns

Failure pattern	Recovery
Lower-order migration missing	Apply or fix the lower-order CR first
Store missing `host`	Patch the referenced `Store.spec.connection.host`
Secret key missing	Add the key expected by `spec.envFrom` or `secretRef.keys`
SQL rejected by ClickHouse	Fix the CR SQL, commit the corrected manifest, let ArgoCD resync
CR already applied but drift suspected	Compare `spec.sql` and `status.sqlHash`; update SQL if a reapply is required

Store connectivity incident

Use this when packages fail because a store endpoint, credential, or cell mapping is wrong.

Procedure

confirm which cell namespace the workload should resolve against
inspect the Store object and referenced secret
verify the generated DSN fields (host, port, database, user, connector-specific keys)
fix the manifest or secret and re-run the failing workload or wait for reconcile

kubectl get cell <cell>
kubectl get store <store> -n <cell-namespace> -o yaml
kubectl get secret <secret> -n <cell-namespace> -o yaml
kubectl describe packagedeployment <package> -n <cell-namespace>

Connector-specific checks

Postgres: host, port, database/dbname, user, password
ClickHouse: host mandatory; default port=9000, user=default, database=default
S3: bucket, optional region and endpoint
Kafka: brokers or bootstrapServers, optional topic

ArgoCD application drift recovery

When Git has the right files but the cluster has not converged:

kubectl get applications -n dk-system
kubectl describe application <env>-<cell>-<package> -n dk-system
kubectl annotate application <env>-<cell>-<package> -n dk-system argocd.argoproj.io/refresh=hard --overwrite

If the issue affects ClickHouse migrations, inspect the migration ApplicationSet output as well, especially when chMigrations.enabled=true and the repo layout under migrations/* changed.

Next Steps

Troubleshooting