Troubleshooting

Use this page when DataKit is installed but something in the control plane is no longer converging: local prerequisites are missing, ArgoCD is not creating the right objects, the controller is stuck in Failed, or ClickHouse migrations are retrying forever.

Diagnostic flow

    flowchart TD
    A[Problem observed] --> B{Local CLI or cluster-side?}
    B -->|CLI| C[Run dk doctor]
    C --> D{Any failed checks?}
    D -->|Yes| E[Fix prerequisites and rerun]
    D -->|No| F[Inspect command-specific logs or flags]
    B -->|Cluster| G[kubectl get packagedeployments, stores, cells, clickhousemigrations]
    G --> H{Status condition points to root cause?}
    H -->|Yes| I[Follow targeted fix below]
    H -->|No| J[Describe resource, inspect events, restart controller if needed]
  

Start with dk doctor

dk doctor validates the local operator workstation before you chase cluster problems. The current source checks:

• Go version at least 1.25
• Docker or Rancher Desktop availability
• k3d, kubectl, and helm
• OCI registry connectivity
• local dev stack health

Run it first:

dk doctor
dk doctor --verbose

If it fails, fix the local toolchain before investigating higher-level symptoms.

Common symptoms and fixes

Controller troubleshooting

The controller deployment name is always dk-controller (charts/dk-controller/templates/_helpers.tpl).

Inspect controller health

kubectl -n dk-system get deployment dk-controller
kubectl -n dk-system get pods -l app.kubernetes.io/component=controller
kubectl -n dk-system logs deployment/dk-controller --tail=200

The deployment template exposes:

• controller health on port 8081
• controller metrics on port 8080
• sidecar health on 8083 and metrics on 8082 when enabled

Inspect CRD status and events

kubectl get packagedeployments -A
kubectl describe packagedeployment <name> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp | tail -50

Important PackageDeployment.status.phase values from the CRD:

• Pending
• Pulling
• Ready
• Running
• Completed
• Failed

High-signal failure reasons from the reconciler:

Cell and store troubleshooting

Cells are cluster-scoped; stores are namespaced inside the cell namespace. Verify both sides when package resolution fails.

kubectl get cells
kubectl get stores -A
kubectl get stores -n dk-canary
kubectl describe cell canary
kubectl describe store warehouse -n dk-canary

Quick interpretation:

• Cell.status.ready=false usually means the namespace or expected stores are not initialised
• Store.status.ready=false means the connection has not been verified
• a package targeting spec.cell=canary will resolve stores from dk-canary

ClickHouse migration troubleshooting

ClickHouseMigration is reconciled by the optional sidecar. The controller is order-aware within a store + database pair and updates status conditions for each failure mode.

Inspect migration state

kubectl get clickhousemigrations -A
kubectl describe clickhousemigration <name> -n <namespace>
kubectl -n dk-system logs deployment/dk-controller -c ch-migration --tail=200

Common migration reasons

The reconciler defaults missing ClickHouse connection fields to port=9000, user=default, and database=default, but it requires spec.connection.host.

Promotion and rollback troubleshooting

When GitOps changes do not reach the cluster:

1. confirm the PR was created or updated (dk promote, dk promote status, or GitHub UI)
2. inspect the changed gitops/envs/<env>/cells/<cell>/apps/<package>/values.yaml
3. inspect the ArgoCD Application in dk-system
4. confirm the resulting PackageDeployment exists in the destination namespace

Useful checks:

dk promote status <pr-number>
kubectl get applications -n dk-system
kubectl describe application <env>-<cell>-<package> -n dk-system
kubectl get packagedeployments -n dk-<env>-<cell>

When to restart the controller

Restart the controller only after you have fixed the underlying object or secret. A restart helps when:

• the deployment is crash-looping after an image or chart upgrade
• the controller cache is stale after API-server or CRD churn
• the ClickHouse sidecar is stuck after a dependency outage

Use the runbook procedure in the next page rather than deleting pods ad hoc.

Next Steps

• Runbooks
• Configuration
• Getting Started
• CLI Reference

Common Issues

This page covers common issues you may encounter and how to resolve them.

Installation Issues

Command Not Found: dk

Symptom: Running dk returns “command not found”

Cause: The dk binary is not in your PATH

Solution:

# Reinstall to ~/go/bin (default)
make install

# Verify it's on your PATH
which dk

# If ~/go/bin is not on your PATH, add it
echo 'export PATH=$PATH:$HOME/go/bin' >> ~/.zshrc
source ~/.zshrc

# Or install to a directory already on your PATH
make install DESTDIR=/usr/local/bin

Build Fails with Go Errors

Symptom: make build fails with Go compilation errors

Cause: Wrong Go version or missing dependencies

Solution:

# Verify Go version (needs 1.22+)
go version

# Download dependencies
go mod download

# Clear module cache if needed
go clean -modcache
go mod download

Permission Denied on Binary

Symptom: Running dk gives “permission denied”

Solution:

chmod +x "$(which dk)"
dk version

Development Stack Issues

dk dev up Fails

Symptom: dk dev up fails to start services

Common Causes:

1. Docker not running

   # Check Docker
   docker info
   
   # Start Docker Desktop (macOS)
   open -a Docker

2. Port conflicts

   # Check ports in use
   lsof -i :9092  # Kafka
   lsof -i :9000  # MinIO
   lsof -i :5000  # Marquez
   
   # Kill conflicting process
   kill -9 <PID>

3. Previous containers not cleaned up

   # Force cleanup
   dk dev down --volumes
   dk dev up

Kafka Connection Refused

Symptom: Pipeline can’t connect to Kafka at localhost:9092

Solutions:

1. Wait for Kafka to be ready

   # Check health
   dk dev status
   
   # Wait and retry
   sleep 30
   dk run ./my-pipeline

2. Check Kafka logs

   kubectl --context k3d-dk-local logs -l app=redpanda

3. Verify Kafka is accepting connections

   docker exec dk-kafka kafka-broker-api-versions \
     --bootstrap-server localhost:9092

MinIO Access Denied

Symptom: S3 operations fail with access denied

Solution:

# Verify MinIO is running
dk dev status

# Check credentials (default: minioadmin/minioadmin)
mc alias set local http://localhost:9000 minioadmin minioadmin

# Create bucket if it doesn't exist
mc mb local/my-bucket

Marquez Not Showing Lineage

Symptom: Lineage events don’t appear in Marquez UI

Solutions:

1. Verify Marquez is healthy

   curl http://localhost:5000/api/v1/namespaces

2. Check environment variables

   # Ensure this is set
   export OPENLINEAGE_URL=http://localhost:5000/api/v1/lineage

3. Run pipeline and check events

   # Run with debug
   DK_LOG_LEVEL=debug dk run ./my-pipeline

Pipeline Issues

dk lint Fails

Symptom: dk lint returns validation errors

Common Errors:

Example fixes:

# Fix E001/E004 - invalid name
metadata:
  name: my-pipeline  # lowercase, hyphens only

# Fix E010 - add missing store
# store.yaml
apiVersion: datakit.infoblox.dev/v1alpha1
kind: Store
metadata:
  name: local-events
spec:
  type: kafka-topic
  connection:
    brokers: localhost:9092
    topic: events

# Fix E025 - add sensitivity
outputs:
  - name: data
    classification:
      pii: true
      sensitivity: confidential  # Add this

dk run Fails Immediately

Symptom: Pipeline starts but exits immediately

Solutions:

1. Check logs

   dk logs <run-id>

2. Run with debug

   DK_LOG_LEVEL=debug dk run ./my-pipeline

3. Common causes:

   • Missing environment variables
   • Can’t connect to input sources
   • Syntax errors in pipeline code

dk run Timeout

Symptom: Pipeline times out before completing

Solution:

# Increase timeout
dk run ./my-pipeline --timeout 60m

# Or set default in config
# ~/.dk/config.yaml
defaults:
  timeout: 60m

No Data Flowing

Symptom: Pipeline runs but processes no records

Debugging steps:

1. Check input has data

   # For Kafka
   docker exec dk-kafka kafka-console-consumer \
     --bootstrap-server localhost:9092 \
     --topic user-events \
     --from-beginning \
     --max-messages 5

2. Check consumer group offset

   docker exec dk-kafka kafka-consumer-groups \
     --bootstrap-server localhost:9092 \
     --group my-consumer-group \
     --describe

3. Check bindings

   • Verify topic name matches
   • Check consumer group setting
   • Verify offset reset policy

Publishing Issues

Authentication Failed

Symptom: dk publish fails with authentication error

Solution:

# For GitHub Container Registry
echo $GITHUB_TOKEN | docker login ghcr.io -u $GITHUB_USER --password-stdin

# For Docker Hub
docker login

# Set in config
export DK_REGISTRY_USERNAME=myuser
export DK_REGISTRY_TOKEN=mytoken

Push Denied

Symptom: dk publish says push denied

Common Causes:

1. No write access to registry

   • Verify you have push permissions
   • Check repository/package visibility settings

2. Wrong registry URL

   dk publish --registry ghcr.io/correct-org

3. Token expired

   docker logout ghcr.io
   docker login ghcr.io

Package Already Exists

Symptom: Can’t push because version exists

Cause: OCI artifacts are immutable

Solution:

# Use a new version
dk build --tag v1.0.1
dk publish

Promotion Issues

PR Not Created

Symptom: dk promote doesn’t create a PR

Solutions:

1. Check GitHub token

   # Ensure token has repo access
   export GITHUB_TOKEN=ghp_xxx

2. Check GitOps repository

   # ~/.dk/config.yaml
   environments:
     dev:
       gitops: https://github.com/org/gitops.git  # Verify URL

3. Check network connectivity

   curl -I https://github.com

PR Failed CI

Symptom: Promotion PR fails CI checks

Solutions:

1. Check the PR for failure details

2. Run lint locally first:

   dk lint --strict

3. Verify package exists in registry:

   dk versions my-pipeline

Sync Failed in ArgoCD

Symptom: PR merged but deployment not synced

Check:

1. ArgoCD application status
2. Kubernetes cluster connectivity
3. Resource quotas/limits

dk status my-pipeline --env dev
dk logs my-pipeline --env dev --sync

Performance Issues

Pipeline Running Slowly

Optimization strategies:

1. Increase resources

   # dk.yaml
   spec:
     runtime:
       resources:
         memory: "4Gi"
         cpu: "4"

2. Increase batch size

   dk run --env BATCH_SIZE=5000

3. Check I/O bottlenecks

   • Use local SSDs
   • Increase network throughput
   • Use compression

High Memory Usage

Solution:

1. Process in smaller batches

2. Use streaming instead of loading all data

3. Increase memory limits:

   resources:
     limits:
       memory: "8Gi"

Registry Cache Issues (k3d Runtime)

Cache Container Not Starting

Symptom: dev-registry-cache container fails to start

Solutions:

1. Check Docker resources

   docker system df
   docker system prune -f  # Clean up if low on space

2. Check for port conflicts

   lsof -i :5000
   # Kill conflicting process if found

3. Inspect container logs

   docker logs dev-registry-cache

4. Force cache rebuild

   docker rm -f dev-registry-cache
   docker volume rm dev_registry_cache
   dk dev up --runtime=k3d

Image Pulls Still Slow

Symptom: Cache is running but images aren’t being cached

Solutions:

1. Verify cache is being used

   # Check the registries.yaml was created
   cat .cache/registries.yaml

2. Check cache hit rate

   docker logs dev-registry-cache 2>&1 | grep -E "manifest|blob"

3. Ensure cluster uses the registry config

   # Delete cluster and recreate
   dk dev down --runtime=k3d
   dk dev up --runtime=k3d

Cache Not Created in CI

Symptom: Cache doesn’t start in CI environment

Cause: This is expected behavior. The registry cache is automatically skipped in CI to avoid complications.

Detection: The following environment variables trigger CI mode:

• CI=true
• GITHUB_ACTIONS=true
• JENKINS_URL (any value)

Solution: If you need the cache in CI, unset these variables (not recommended).

“Network not found” Error

Symptom: Container fails to start with network error

Solution:

# Create the network manually
docker network create devcache

# Or remove and let it recreate
docker network rm devcache
dk dev up --runtime=k3d

Lineage Issues

Missing Upstream/Downstream

Symptom: Lineage graph shows orphan nodes

Causes:

1. Different namespaces

   • Packages should use the same namespace for linked lineage

2. Binding name mismatch

   # Both packages should reference same binding
   binding: shared/user-events  # Use consistent naming

3. Never ran successfully

   • Only successful runs emit complete lineage

Stale Lineage

Symptom: Lineage shows old data

Solution:

# Planned: dk lineage my-pipeline --refresh
# For now, query Marquez directly:
curl http://localhost:5000/api/v1/namespaces/default/jobs/my-pipeline/runs

Getting More Help

If you can’t resolve your issue:

1. Check debug logs

   DK_LOG_LEVEL=debug dk <command>

2. Search existing issues

   • GitHub Issues

3. Open a new issue

   • Include: command, error message, environment details

4. Contact the team

   • Slack: #datakit-support
   • Email: datakit@example.com

See Also

• FAQ - Frequently asked questions
• CLI Reference - Command documentation
• Configuration - Config options

Frequently Asked Questions

Answers to common questions about DataKit.

General Questions

What is DataKit?

DataKit is a system for building, publishing, and operating data pipelines with built-in governance, lineage tracking, and GitOps-based deployment. It provides:

• Data Packages: Self-contained, versioned bundles for data pipelines
• Local Development: Docker-based development stack
• Lineage Tracking: Automatic OpenLineage integration
• GitOps Deployment: Environment promotion through pull requests

What problem does DataKit solve?

DataKit addresses common challenges in data engineering:

How is DataKit different from Airflow/Dagster/Prefect?

DataKit is complementary to orchestrators, not a replacement:

DataKit packages can be scheduled by any orchestrator.

What languages/runtimes are supported?

DataKit supports any containerized runtime:

• Python (most common)
• Java/Scala (Spark, Flink)
• Go
• Node.js
• Any language that runs in a container

Development Questions

How do I start developing locally?

# 1. Install dk CLI
make build
export PATH=$PATH:$(pwd)/bin

# 2. Start local stack
dk dev up

# 3. Create a Transform package
dk init my-pipeline --runtime generic-python

# 4. Run locally
dk run ./my-pipeline

See the Quickstart for details.

What’s included in the local development stack?

Can I use my own Kafka/S3 locally?

Yes! Override connection details in your Store manifests:

apiVersion: datakit.infoblox.dev/v1alpha1
kind: Store
metadata:
  name: my-kafka
spec:
  connector: kafka
  connection:
    bootstrap-servers: my-kafka:9092

DataSets reference the Store by name — no additional configuration is needed.

How do I add custom services to the dev stack?

Use dk config to customize chart versions and Helm values:

dk config set dev.charts.redpanda.version 25.2.0
dk config set dev.charts.postgres.values.primary.resources.limits.memory 1Gi

How do I persist data between runs?

Data is stored in Docker volumes. To reset:

# Keep data
dk dev down

# Remove data
dk dev down --volumes

Package Questions

What files are in a data package?

The dk.yaml file is a Transform manifest that references DataSets by name. DataSets reference Stores, and Stores reference Connectors.

What runtimes are available?

dk init my-pkg --runtime generic-python
dk init my-pkg --runtime cloudquery

How do I version packages?

Packages use semantic versioning:

# Build with version
dk build --tag v1.0.0

# Increment for changes
v1.0.0 → v1.0.1  # Bug fix
v1.0.0 → v1.1.0  # New feature
v1.0.0 → v2.0.0  # Breaking change

Can I publish private packages?

Yes! Push to a private registry:

# Use private registry
dk publish --registry ghcr.io/my-private-org

# Ensure authentication
docker login ghcr.io

Deployment Questions

How does promotion work?

DataKit uses GitOps for deployment:

1. dk promote creates a PR in the GitOps repository
2. PR is reviewed and approved
3. After merge, ArgoCD syncs to Kubernetes
4. Package runs in the target environment

What are the standard environments?

Can I skip environments?

Not recommended, but possible:

# This will work but triggers a warning
dk promote my-pkg v1.0.0 --to prod
# Warning: Skipping dev and int environments

How do I rollback a deployment?

# Rollback to previous version
dk rollback my-pkg --env prod

# Rollback to specific version
dk rollback my-pkg --to v1.0.0 --env prod

How do I know what version is deployed?

dk status my-pkg

Environment  Version   Status
dev          v1.2.0    Synced
int          v1.1.0    Synced
prod         v1.1.0    Synced

Lineage Questions

What is data lineage?

Lineage tracks:

• Where data comes from (upstream)
• Where data goes to (downstream)
• What transformations were applied
• When the pipeline ran

How does DataKit track lineage?

DataKit automatically emits OpenLineage events:

1. Reads inputs/outputs from dk.yaml
2. Emits START event when pipeline begins
3. Emits COMPLETE/FAIL event when pipeline ends
4. Events sent to Marquez (or configured backend)

Where can I view lineage?

• Local: http://localhost:3000 (Marquez Web UI)
• CLI: dk lineage my-pipeline (not yet implemented)
• Production: Your organization’s lineage backend

Can I use a different lineage backend?

Yes! Configure in ~/.dk/config.yaml:

lineage:
  backend: datahub  # or: marquez, custom
  endpoint: http://datahub:8080/api/lineage

Why isn’t my lineage showing up?

Common causes:

1. Marquez not running: dk dev status
2. Wrong endpoint: Check OPENLINEAGE_URL
3. Pipeline never completed successfully

Governance Questions

What data classification levels are available?

How do I mark data as containing PII?

Use the classification and pii fields on DataSet schemas:

apiVersion: datakit.infoblox.dev/v1alpha1
kind: DataSet
metadata:
  name: customer-data
spec:
  store: warehouse
  table: public.customers
  classification: confidential
  schema:
    - name: email
      type: string
      pii: true
    - name: name
      type: string
      pii: true

Are classification policies enforced?

Yes! dk lint enforces policies:

dk lint
# Error: output 'customer-data': pii=true requires sensitivity level

How do I view what packages handle PII?

dk governance report --filter pii=true

Troubleshooting Questions

dk command not found

Add the binary to your PATH:

export PATH=$PATH:/path/to/datakit/bin

dk dev up fails

1. Check Docker is running: docker info
2. Check port conflicts: lsof -i :9092
3. Clean up: dk dev down --volumes

Pipeline can’t connect to Kafka

1. Wait for Kafka to be ready: dk dev status
2. Check Kafka logs: kubectl --context k3d-dk-local logs -l app=redpanda
3. Verify bootstrap server: localhost:9092

Push to registry fails

1. Check authentication: docker login ghcr.io
2. Check permissions on the registry
3. Check network connectivity

Promotion PR not created

1. Check GITHUB_TOKEN is set
2. Verify GitOps repository URL in config
3. Check network connectivity

More Questions?

If your question isn’t answered here:

1. Check Common Issues
2. Search GitHub Issues
3. Ask in #datakit-support on Slack
4. Open a new issue on GitHub

See Also

• Common Issues - Detailed troubleshooting
• CLI Reference - Command documentation
• Quickstart - Getting started guide