CloudQuery Go Plugin

Build a Go-based CloudQuery source plugin that fetches data from an API and syncs it to file or PostgreSQL destinations.

Time: ~15 minutes Difficulty: Beginner Prerequisites: DK CLI installed, Go 1.25+, Docker running, dk dev up completed

Overview

You’ll learn the full developer lifecycle for a Go CloudQuery plugin:

Scaffold — Generate a working plugin project
Understand — Explore the project structure
Test — Run unit tests locally
Run — Build and deploy to the local k3d cluster
Sync — Extract data to file and PostgreSQL destinations
Develop — Add tables and resolvers for your own data source

1. Scaffold a Go Plugin

dk init my-source --runtime cloudquery
cd my-source

This creates a complete Go plugin project:

my-source/
├── .gitignore                       # Go-specific ignore patterns
├── .datakit/
│   └── Makefile.common              # Managed targets (do not edit)
├── Makefile                         # Project Makefile (add your own targets here)
├── dk.yaml                          # Package manifest
├── go.mod                           # Go module definition
├── main.go                          # Plugin entry point
├── resources/
│   └── plugin/
│       └── plugin.go                # Plugin factory + Configure function
└── internal/
    ├── client/
    │   ├── client.go                # Client (Tables, Sync, Close)
    │   └── spec.go                  # Config deserialization
    └── tables/
        ├── example_resource.go      # Table schema + resolver
        └── example_resource_test.go # Unit tests

2. Understand the Project

dk.yaml — Package Manifest

cat dk.yaml

The manifest declares this as a Transform with the CloudQuery runtime:

apiVersion: datakit.infoblox.dev/v1alpha1
kind: Transform
metadata:
  name: my-source
  namespace: default
  version: 0.1.0
  labels:
    team: my-team
spec:
  runtime: cloudquery
  mode: batch
  inputs:
    - dataset: my-source-source-table
  outputs:
    - dataset: my-source-dest-table
  timeout: 30m

The kind: Transform with runtime: cloudquery tells DKDK that this package is a CloudQuery plugin. The inputs and outputs declare the DataSets this transform reads from and writes to. No container image is required — plugin images come from the Connector manifest.

internal/tables/example_resource.go — Table Definition

Each table defines its schema (columns and Arrow types) and a resolver function that fetches rows:

func ExampleResourceTable() *schema.Table {
    return &schema.Table{
        Name:     "example_resource",
        Resolver: fetchExampleResource,
        Columns: []schema.Column{
            {Name: "id",     Type: arrow.BinaryTypes.String,       Resolver: schema.PathResolver("ID")},
            {Name: "name",   Type: arrow.BinaryTypes.String,       Resolver: schema.PathResolver("Name")},
            {Name: "active", Type: arrow.FixedWidthTypes.Boolean,  Resolver: schema.PathResolver("Active")},
        },
    }
}

func fetchExampleResource(_ context.Context, _ schema.ClientMeta, _ *schema.Resource, res chan<- any) error {
    resources := []ExampleResource{
        {ID: "res-001", Name: "example-1", Active: true},
        {ID: "res-002", Name: "example-2", Active: false},
    }
    for _, r := range resources {
        res <- r
    }
    return nil
}

internal/client/client.go — Client

The client implements plugin.Client and wires together tables, the scheduler, and the sync loop:

func (c *Client) Tables(ctx context.Context, opts plugin.TableOptions) (schema.Tables, error) {
    return []*schema.Table{tables.ExampleResourceTable()}, nil
}

func (c *Client) Sync(ctx context.Context, options plugin.SyncOptions, res chan<- message.SyncMessage) error {
    tt, _ := c.Tables(ctx, plugin.TableOptions{})
    return c.scheduler.Sync(ctx, c, tt, res, ...)
}

resources/plugin/plugin.go — Plugin Factory

func Plugin() *plugin.Plugin {
    return plugin.NewPlugin(Name, Version, Configure,
        plugin.WithKind(Kind),
        plugin.WithTeam(Team),
    )
}

func Configure(ctx context.Context, logger zerolog.Logger, specBytes []byte, opts plugin.NewClientOptions) (plugin.Client, error) {
    return client.New(logger, specBytes, opts), nil
}

Makefile — Common Targets

Every scaffolded project includes a Makefile with standard targets. Run make or make help to see them:

$ make
Usage: make <target>

Targets:
  build                Build the plugin container image
  clean                Remove build artifacts and sync output
  fmt                  Format Go source code
  help                 Show this help message
  lint                 Run dk lint on the package
  run                  Build and deploy to k3d, discover tables
  sync                 Run a full sync to local files
  sync-pg              Run a full sync to PostgreSQL
  test                 Run unit tests
  tidy                 Run go mod tidy
  vet                  Run go vet

The Makefile includes .datakit/Makefile.common which is managed by the dk CLI — do not edit it. It is automatically kept in sync when you run dk build or dk run. Add your own targets to the root Makefile using ## comments so they appear in make help:

# In your Makefile:
my-target: ## My custom description
 echo "hello"

3. Run Tests

Using the DK CLI (recommended)

dk test

For Go plugins, dk test runs go test ./... -v:

------------------------------------------------------------
CloudQuery Unit Tests
------------------------------------------------------------

=== RUN   TestExampleResourceTable
--- PASS: TestExampleResourceTable (0.00s)
PASS

✓ CloudQuery unit tests PASSED

Without the CLI

go test ./... -v

This is exactly what dk test does. No additional setup needed — Go’s toolchain handles everything.

4. Build and Run the Plugin

dk run

This builds a Docker container using a multi-stage Dockerfile (Go 1.25 builder → distroless static runtime), imports it into the k3d cluster, and starts the plugin:

Building CloudQuery plugin image: default/my-source:latest (lang=go)
...
Discovered 1 table(s):

  example_resource
    An example table demonstrating the CloudQuery plugin pattern.
    Columns:
      id      utf8
      name    utf8
      active  bool

✓ CloudQuery plugin is working correctly

5. Sync Data

Sync to local files

dk run --sync

Writes JSON output to ./cq-sync-output/:

✓ Sync completed: my-source → file
  Output directory: ./cq-sync-output/

Sync to PostgreSQL

dk run --sync --destination postgresql

Uses the PostgreSQL instance running in the k3d dev environment:

✓ Sync completed: my-source → postgresql

Verify the data:

kubectl exec -it deploy/dk-postgres-postgres -n dk-local -- \
  psql -U postgres -c "SELECT * FROM example_resource;"

6. Develop Your Own Tables

Add a new table

Create internal/tables/users.go:

package tables

import (
    "context"

    "github.com/apache/arrow-go/v18/arrow"
    "github.com/cloudquery/plugin-sdk/v4/schema"
)

func UsersTable() *schema.Table {
    return &schema.Table{
        Name:        "users",
        Description: "Users from the external API.",
        Resolver:    fetchUsers,
        Columns: []schema.Column{
            {Name: "id",    Type: arrow.BinaryTypes.String,    Resolver: schema.PathResolver("ID")},
            {Name: "email", Type: arrow.BinaryTypes.String,    Resolver: schema.PathResolver("Email")},
        },
    }
}

type User struct {
    ID    string
    Email string
}

func fetchUsers(ctx context.Context, meta schema.ClientMeta, _ *schema.Resource, res chan<- any) error {
    // Replace with actual API calls
    res <- User{ID: "u-001", Email: "alice@example.com"}
    res <- User{ID: "u-002", Email: "bob@example.com"}
    return nil
}

Register it in client.go

func (c *Client) Tables(_ context.Context, _ plugin.TableOptions) (schema.Tables, error) {
    return []*schema.Table{
        tables.ExampleResourceTable(),
        tables.UsersTable(),  // new
    }, nil
}

Add tests

Create internal/tables/users_test.go:

package tables

import "testing"

func TestUsersTable(t *testing.T) {
    table := UsersTable()
    if table.Name != "users" {
        t.Errorf("table name = %q, want %q", table.Name, "users")
    }
    if len(table.Columns) != 2 {
        t.Fatalf("expected 2 columns, got %d", len(table.Columns))
    }
}

Test and run

dk test                              # Run unit tests
dk run                               # Verify table discovery
dk run --sync                        # Sync data to files
dk run --sync --destination postgresql  # Sync to PostgreSQL

Command Reference

Command	What it does	Needs Docker/k3d?
`dk init <name> --runtime cloudquery`	Scaffold a new Go plugin	No
`dk test`	Run `go test ./... -v`	No
`dk run`	Build container, deploy to k3d, discover tables	Yes
`dk run --sync`	Sync data to local JSON files	Yes
`dk run --sync --destination postgresql`	Sync data to PostgreSQL	Yes
`dk test --integration`	Full build + sync integration test	Yes
`make`	Show all available Make targets	No
`make test`	Run `go test ./... -v` (same as `dk test`)	No
`make sync`	Build + sync to local files	Yes

Python vs Go Comparison

Aspect	Go	Python
SDK	`plugin-sdk/v4`	`cloudquery-plugin-sdk`
Table types	Apache Arrow Go	PyArrow
Resolver pattern	`func(ctx, meta, parent, res chan<-)`	`def resolve(self, client, parent)` → yield
Test runner	`go test`	pytest
Build image	`golang:1.25-alpine`	`python:3.11-slim`
Runtime image	`distroless/static-debian12`	`distroless/python3-debian12`
Local deps	Go 1.25+ (auto-managed)	Python 3.12+ + venv (auto-created by `dk test`)

Next Steps

Python CloudQuery Plugin — Same workflow in Python
Promoting Packages — Deploy to environments
CLI Reference — All available commands

Building a Streaming Pipeline CloudQuery Python Plugin