DK-ADR-002: Environments, Cells, and Naming Conventions

Context

DataKit deploys data pipelines across multiple environments and tenant boundaries. The existing docs describe cells and environments but leave several questions unanswered:

• What exactly is an environment vs. a cell?
• Can cells nest?
• How do cell boundaries manifest across Kafka, S3, and PostgreSQL?
• What naming conventions should physical resources follow?
• How should naming be enforced?
• What happens when tenants move between cells?

This ADR records the decisions reached after pressure-testing these questions across REST/RPC, event-based, and object-storage communication patterns.

Decision

1. Environments are compute isolation boundaries

An environment (dev, int, prod) maps to an isolated compute boundary — typically a Kubernetes cluster or cloud account. Environments are segregated by infrastructure: different clusters, different accounts, different credentials.

2. Cells are tenant/workload isolation within an environment

A cell is a named isolation boundary within an environment. It provides infrastructure context (Stores, credentials, namespaces) for a set of deployed packages. The same package can deploy to many cells; the same cell can host many packages.

Cells are flat — no nesting. A cell cannot contain another cell. Isolation within a cell is handled by creating a new peer cell, not a sub-cell.

Cell names are unique per cluster (environment). The same name may appear across environments (e.g., c0 exists in both dev and prod). The fully-qualified identity is {env}/{cell}.

3. Cells have scope labels

Not every cell serves the same purpose. Cells carry a scope label to distinguish their role:

apiVersion: datakit.infoblox.dev/v1alpha1
kind: Cell
metadata:
  name: c0
spec:
  namespace: dk-c0
  labels:
    scope: tenant          # tenant | shared | canary

Scope labels are advisory metadata today. Future policy ADRs may enforce deployment rules based on scope (e.g., cross-cell inputs only from shared cells).

4. Cell = logical boundary, Store = physical mapping

A cell is not a single physical primitive. It manifests differently per infrastructure type. The Store is what maps a cell to physical addressing.

The k8s namespace is the anchor point — Store CRs live in the cell’s namespace and describe the Kafka topics, S3 prefixes, and PG connection strings for that cell.

5. Cell metadata lives in Custom Resources

Clients discover cell routing via Kubernetes Custom Resources (the Cell CR and Store CRs in the cell’s namespace). Not DNS SRV records.

DNS SRV is a poor fit because cell routing metadata (topic names, S3 prefixes) goes beyond host:port. A discovery API can be built on top of the CRs later if external clients need access.

6. DataKit namespace is required

The DataKit namespace (the metadata.namespace field on manifests, distinct from k8s namespace) is a required component of every manifest. It prevents naming collisions and appears in every derived physical name.

• Inherited from the project when running inside a dk project init directory.
• Prompted in interactive mode during dk init.
• Required as --namespace flag in CI / non-interactive mode.
• No default. dk lint errors if namespace is empty.

7. Naming conventions

Physical resource names are derived from three components: {namespace}, {dataset}, and {cell}.

The cell appears in different positions depending on the infrastructure type’s access patterns:

Why the positions differ:

• Kafka — dataset-first (analytics.orders.*) supports topic discovery and schema registry subject grouping. Cell as suffix.
• S3 — cell-first (/c0/...) supports prefix-based IAM policies and efficient ListObjectsV2. Dataset-first listing is a catalog concern, not an S3 listing concern.
• PG — cell provides the database (connection-level isolation). Namespace provides the schema (logical grouping). Table name is the dataset.

8. Physical names are derived by default

The DataSet provides the within-cell logical name. The cell provides the between-cell scoping. The platform assembles the physical address.

# DataSet — no physical name specified, derived at deploy time
apiVersion: datakit.infoblox.dev/v1alpha1
kind: DataSet
metadata:
  name: orders
  namespace: analytics
spec:
  store: warehouse
  # table, topic, prefix omitted — derived from namespace + dataset + cell

When table, topic, or prefix is explicitly set, the platform uses it as-is within the cell’s scope. The explicit field overrides derivation.

dk show --cell <cell> displays the resolved physical names:

$ dk show --cell c0
Resolved stores:
  orders       → dk_c0.analytics.orders (postgres)
  raw-events   → analytics.raw-events.c0 (kafka)
  output       → s3://dk-datalake/c0/analytics/output/ (s3)

9. dk lint enforces naming conventions

When a DataSet specifies an explicit table, topic, or prefix, dk lint validates it against the naming convention. Non-conforming names produce an error:

E350  dataset/legacy-users.yaml
      Table name "public.legacy_users" does not match convention.
      Expected: "analytics.legacy_users" (derived from namespace + dataset name)
      To override, add annotation: dk.infoblox.dev/naming: explicit

Escape hatch: the dk.infoblox.dev/naming: explicit annotation on the manifest suppresses the naming convention check for that resource:

metadata:
  name: legacy-users
  namespace: analytics
  annotations:
    dk.infoblox.dev/naming: explicit
spec:
  store: legacy-db
  table: public.legacy_users       # non-standard, intentionally

Why an annotation and not a flag or ignore file:

• dk lint --skip-naming disables the check globally — too broad.
• .dklintignore separates the exception from the manifest — reviewers don’t see it in the same diff.
• An annotation lives on the manifest, shows up in code review, and is greppable: grep "naming: explicit" finds all overrides.

10. shared is the conventional cell for global data

Reference data, cross-tenant aggregations, and platform-wide pipelines deploy to a cell conventionally named shared. It is a regular cell with scope: shared — no special casing in Store resolution, promotion, or naming.

analytics.country-codes.shared     (Kafka)
s3://dk-datalake/shared/analytics/country-codes/   (S3)
dk_shared.analytics.country_codes  (PG)

The name shared is a convention, not a reserved keyword. Teams can use global, common, or any other name. The scope: shared label is what matters.

11. Tenant migration is pause-drain-flip-resume

Moving a tenant from one cell to another does not require distributed transactions. Data pipelines are batch-oriented or at-least-once, so a brief processing pause is acceptable.

Migration sequence:

1. Pause — stop scheduling new runs in the source cell.
2. Drain — let in-flight jobs complete (or kill and rely on retry).
3. Flip — update cell routing metadata (CellBinding or equivalent).
4. Resume — new runs pick up the target cell’s Stores.

Historical data in the source cell remains accessible via cross-cell DataSet references. A dk cell migrate command may automate this sequence in the future but is not required for v1.

Consequences

Positive

• Clear separation: environments isolate compute, cells isolate tenants/workloads.
• Flat cell model keeps Store resolution, RBAC, and tooling (ArgoCD, OPA) simple.
• Naming conventions are predictable and enforceable. Data ops can write IAM policies and ACLs by pattern.
• Derivation-by-default means devs specify less config for greenfield projects.
• Annotation-based escape hatch is visible in code review and greppable.

Negative

• Cell position varies by infrastructure type (suffix for Kafka, prefix for S3, database for PG). This is technically correct but adds cognitive load. dk show --cell mitigates this.
• Required namespace adds friction for quick experiments. Mitigated by project inheritance and interactive prompts.
• Annotation-based lint suppression can accumulate over time. Teams should periodically audit dk.infoblox.dev/naming: explicit annotations.

Alternatives Considered

Nested cells

Cells containing sub-cells (e.g., c0/team-a). Rejected because k8s namespaces are flat, and every platform tool (ArgoCD, OPA, network policies) would need to understand the hierarchy. Team-level isolation is handled by creating peer cells (c0-team-a, c0-team-b).

DNS SRV for cell discovery

Using DNS SRV records for cell routing metadata. Rejected because SRV records carry host:port only — topic names, S3 prefixes, and connection strings don’t fit. TXT record workarounds are fragile. DNS TTL caching causes stale routing during migrations.

Transactional tenant migration

Distributed transactions across Kafka, S3, and PG during cell moves. Rejected because data pipelines are batch/at-least-once by nature. Pause-drain-flip-resume is sufficient and dramatically simpler.

Dataset-first ordering everywhere

Using {namespace}.{dataset}.{cell} for all systems. Rejected because S3 prefix-based IAM policies and listing require cell-first ordering, and PG connection-level isolation requires cell as the database name. Each system uses the position that plays to its strengths.

Optional namespace with default

Defaulting namespace to default or the transform name when omitted. Rejected because a meaningless default pollutes physical names and creates collisions. Namespace should be a deliberate choice.

Open Questions (future ADRs)

1. Cross-cell access control model — When cell c0 reads from cell c1’s DataSet, what authorizes that access? Implicit, or does c1 explicitly export the DataSet? Security implications for multi-tenant isolation.

2. Streaming cell migration — Batch transforms are retriable, but streaming transforms require a coordinated topic cutover. The pause-drain-flip-resume sequence needs additional detail for streaming.

3. CRD schema for cell routing — Should routing metadata extend the existing Cell CRD (status.endpoints) or live in a separate CellBinding CRD? Affects client discovery patterns.

4. Convention enforcement strictness over time — Should dk lint start in warn mode and graduate to error mode? Or error from day one with the annotation escape hatch?

5. Cell lifecycle management — What happens when a cell is decommissioned? Cascading effects on Stores, topics, S3 data, deployed transforms. Needs a dk cell drain / dk cell delete workflow.