DataKit Documentation

Welcome to the DataKit documentation. DataKit (dk) is a CLI and SDK for building, testing, and deploying data pipelines with built-in governance, lineage tracking, and GitOps deployment.

Choose Your Path

Developer Guide

Build and test data pipelines locally

Operator Guide

Deploy, configure, and troubleshoot DataKit

Architecture

System design, data model, and decisions

Quick Start

# Install dk CLI
cd cli && go build -o ~/bin/dk ./

# Scaffold a new project
dk project init myproject && cd myproject

# Create a transform
dk init mytransform --runtime cloudquery --mode batch --namespace myns --team myteam

# Local development
dk dev up
dk run mytransform/
dk test mytransform/

# Build and publish
dk build mytransform/
dk publish mytransform/

Key Concepts

Before diving in, familiarize yourself with the core concepts that power DataKit:

Data Package — self-contained unit of data processing (Transform + DataSets + config)
Manifest — dk.yaml declaring what a package does (kinds: Transform, DataSet, Store, Connector)
Cell — infrastructure isolation boundary within an environment
Pipeline — reactive dependency graph derived from manifests (no pipeline.yaml)

Documentation Site

This documentation is served at datakit.internal.infoblox.dev.