SDK Reference
The SDK packages under sdk/ are the reusable building blocks behind the CLI.
The most important flow is parse manifests, validate them, run or bundle the
package, then emit metadata for registry, promotion, catalog, and lineage.
flowchart LR
M[manifest] --> V[validate]
V --> R[runner]
V --> G[registry]
V --> P[promotion]
V --> S[schema]
R --> L[lineage]
V --> C[catalog]
sdk/manifest
Purpose: parse dk.yaml and related manifest kinds into concrete contracts types.
Key APIs: DetectKind, Parser, DefaultParser, ParseManifest, ParseManifestFile, Manifest.
Usage from the codebase: ParseManifest is used by registry bundling, local runs, and CLI commands.
m, kind, err := manifest.ParseManifest(dpData)
if err != nil {
return nil, fmt.Errorf("failed to parse dk.yaml: %w", err)
}sdk/validate
Purpose: provide kind-aware validation plus package-level aggregation.
Key APIs: Validator, ValidationResult, ValidationContext, ManifestValidator, AggregateValidator.
Usage from the codebase: NewManifestValidatorFromFile parses dk.yaml, detects the kind, and dispatches to the right validator.
validator, err := NewManifestValidatorFromFile(path)
if err != nil {
return result
}
errs := validator.Validate(ctx)sdk/runner
Purpose: execute transform packages locally, primarily through the Docker runner.
Key APIs: Runner, RunOptions, RunResult, GetRunner, RegisterRunner, DockerRunner.
Usage from the codebase: the runner tests exercise a dry run against a generated dk.yaml.
runner, err := NewDockerRunner()
result, err := runner.Run(context.Background(), RunOptions{
PackageDir: tmpDir,
DryRun: true,
})sdk/registry
Purpose: bundle packages as OCI artifacts and push or pull them through an OCI registry.
Key APIs: Client, Artifact, ArtifactManifest, PushResult, ClientConfig, Bundler, BundleOptions, OrasClient.
Usage from the codebase: bundler tests show the standard artifact creation path.
bundler := NewBundler("v1.0.0")
artifact, err := bundler.Bundle(BundleOptions{
PackageDir: tmpDir,
GitCommit: "abc123",
GitBranch: "main",
})sdk/promotion
Purpose: prepare GitOps promotion changes, sync package files, and create or update PRs.
Key APIs: PromotionRequest, PromotionResult, Service, Promoter, PRClient, CollectPackageFiles.
Usage from the codebase: Promoter.Promote is tested with a real PromotionRequest shape.
result, err := promoter.Promote(context.Background(), &PromotionRequest{
Package: "my-pkg",
Version: "v1.0.0",
TargetEnv: EnvDev,
})sdk/schema
Purpose: resolve schema references, maintain dk.lock, and convert external schema definitions into DataKit fields.
Key APIs: SchemaResolver, ResolvedSchema, SchemaModule, ReadLockFile, WriteLockFile, ResolveLock, SyntheticDataSet, ModuleToSchemaFields.
Usage from the codebase: the lock tests show how a locked schema becomes a synthetic dataset for downstream resolution.
locked := contracts.LockedSchema{Module: "users", Version: "1.2.3", Format: "parquet"}
ds := SyntheticDataSet(locked)sdk/catalog
Purpose: represent datasets and jobs in a catalog and query lineage-oriented metadata.
Key APIs: Client, Record, Schema, Source, Lineage, LineageGraph, NewRecord, FromManifest, MarquezClient.
Usage from the codebase: catalog tests use the in-memory client and strongly typed records.
record := &Record{
ID: "ns/test",
Type: RecordTypeDataset,
Namespace: "ns",
Name: "test",
}sdk/lineage
Purpose: build and emit OpenLineage events to console, Marquez, DataHub, or generic HTTP backends.
Key APIs: Emitter, EmitterConfig, Event, Dataset, EventBuilder, NewEmitterFromConfig, NewEvent, NewDataset.
Usage from the codebase: the factory and event tests cover the default constructors and backend selection.
emitter, err := NewEmitterFromConfig(EmitterConfig{Type: "console"})
e := NewEvent(EventTypeFail, "r", "ns", "j").WithErrorFacet("boom", "")
_ = emitter.Emit(context.Background(), e)