Quickstart

This quickstart walks through the core developer loop for DataKit: install the CLI, scaffold a project, run a package locally, and produce a publishable artifact.

Prerequisites

Make sure the following tools are installed before you start:

Go 1.25+
Docker
k3d
Helm

Install `dk` from source

Build the CLI directly from this repository:

cd cli && go build -o ~/bin/dk ./

If ~/bin is not already on your PATH, add it before continuing.

Workflow overview

    sequenceDiagram
  participant Dev as Developer
  participant DK as dk CLI
  participant Local as Local dev stack
  participant OCI as OCI registry

  Dev->>DK: dk project init myproject
  Dev->>DK: dk init mytransform --runtime cloudquery --mode batch --namespace myns --team myteam
  Dev->>DK: dk dev up
  DK->>Local: Start local services
  Dev->>DK: dk run mytransform
  DK->>Local: Execute package locally
  Dev->>DK: dk build mytransform
  DK->>DK: Bundle OCI artifact
  Dev->>DK: dk publish mytransform
  DK->>OCI: Push artifact

Scaffold a project

Create a workspace for one or more packages:

dk project init myproject
cd myproject

Scaffold a first transform

Create a batch transform package:

dk init mytransform --runtime cloudquery --mode batch --namespace myns --team myteam

This creates a mytransform/ directory with a dk.yaml manifest and runtime-specific starter files.

Start local development services

Bring up the local stack used for development and test runs:

dk dev up

The legacy quickstart documents local services such as Redpanda, LocalStack, PostgreSQL, and Marquez. Keep Docker running while you work.

Run and test your package

Execute the package locally:

dk run <package>

Run the package test entrypoint:

dk test <package>

For repository-level validation, the E2E workflow exercises init, lint, and build --dry-run in tests/e2e/workflow_test.go.

Build and publish

From inside the package directory, build an OCI artifact:

dk build

Publish it to your configured registry:

dk publish

Authenticate to the target registry before publishing if required.