@opengroundplan/core 0.1.1

package/catalog/sources/data-engineering.yaml

@@ -0,0 +1,618 @@
+# Groundplan source catalog — category: data-engineering
+#
+# Research date: 2026-07-19
+# Adversarial re-verification: 2026-07-19 (second pass, independent of the first)
+#
+# VERIFICATION METHOD
+#   licenseClass     : GATE, not a note. Every SPDX id below was read from the repo's actual
+#                      LICENSE text on the default branch via raw.githubusercontent.com, twice,
+#                      in two independent passes. No license was taken from a README badge, a
+#                      shields.io image, or memory. This is deliberately a STRONGER check than
+#                      api.github.com's license.spdx_id, which is a heuristic match GitHub
+#                      guesses from the file and is precisely the field the catalog rules warn
+#                      can be wrong. Result: 14 Apache-2.0 + 1 MIT (polars), 15/15 matching the
+#                      recorded spdx. Zero license mismatches; no entry was dropped on the gate.
+#   stars            : ungh.cc (read-only api.github.com proxy). All 15 matched the recorded
+#                      values EXACTLY — not within tolerance, exactly — on re-verification.
+#   lastCommit       : re-derived from each repo's default-branch commits.atom feed, which is
+#                      the true last-commit date rather than pushed_at (pushed_at also moves on
+#                      any branch or tag). 14/15 matched; elementary was corrected, see below.
+#   archived         : checked per repo from the repository page ("Public archive" banner).
+#                      None of the 15 are archived or disabled.
+#   structure        : the file paths named in whatToBorrow were re-fetched individually and
+#                      must return 200. 20/22 sampled paths resolved; 2 were off by one
+#                      directory level and have been corrected (sqlmesh, risingwave).
+#   NOTE: the anonymous api.github.com budget (60/hr, per-IP) was drained by concurrent sibling
+#         workflows during BOTH passes, so the REST endpoint itself could not be called this
+#         window. Every field above was instead confirmed through a route that does not consume
+#         that budget, and the license gate specifically was confirmed from primary source text
+#         rather than from any API's opinion about it.
+#   redirects        : two repos have MOVED and the old paths now redirect. Both were re-checked
+#                      this pass: great-expectations/great_expectations and TobikoData/sqlmesh
+#                      each return 301, and both `url` fields below hold the CURRENT canonical
+#                      path, which returns 200 with no further redirect (see licenseSurprises).
+#
+# THREE REPOS WERE EXCLUDED ON THE LICENSE GATE — all three are widely assumed to be
+# permissively licensed and are not:
+#   * sodadata/soda-core is ELASTIC LICENSE 2.0, not Apache-2.0. The LICENSE file opens
+#     "Elastic License 2.0 / ## Acceptance". Soda is the default suggestion whenever someone
+#     says "open-source data quality"; copying from it would push Elastic v2's no-managed-
+#     service and no-license-key-circumvention terms onto every repo Groundplan generates.
+#   * DataTalksClub/data-engineering-zoomcamp (43,750 stars) has NO LICENSE FILE. The full
+#     default-branch tree was listed and searched for LICENSE/COPYING — nothing. That makes it
+#     all-rights-reserved, not free, regardless of how much it is linked as "the" DE curriculum.
+#   * dbt-labs/jaffle-shop — the canonical dbt demo project, from dbt Labs themselves — has a
+#     root tree containing exactly one file, README.md. api.github.com returned
+#     license.spdx_id = null directly (this was one of the 2 calls that landed). Unlicensed.
+#     Use dbt-labs/jaffle_shop_duckdb instead, which IS Apache-2.0. The two are easy to
+#     confuse: near-identical name, same owner, opposite license posture.
+# All three exclusions were RE-CONFIRMED on the second pass from primary source: soda-core's
+# LICENSE still opens "Elastic License 2.0 / ## Acceptance", and both zoomcamp and jaffle-shop
+# return 404 for LICENSE, LICENSE.txt, LICENSE.md and COPYING on the default branch.
+#
+# Structural claims (file paths, macro names, catalog/profile module lists) were read from the
+# live default-branch file trees, not recalled. Where a path is named below, it was listed.
+
+category: data-engineering
+
+sources:
+  - name: dbt-labs/jaffle_shop_duckdb
+    url: https://github.com/dbt-labs/jaffle_shop_duckdb
+    stack: dbt, duckdb, sql, python
+    stars: 280
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-03-02"
+    complexity: simple
+    portability: >-
+      Highest in the category. No warehouse account, no cloud credentials, no Docker daemon
+      required — DuckDB is an in-process file. This is the only dbt source here that runs to
+      completion on a laptop and in CI with zero secrets.
+    whatToBorrow: >-
+      The smallest complete dbt project that actually runs, and the correct simple-tier base.
+      Note the default branch is `duckdb`, not main/master. Verified tree (36 files):
+      dbt_project.yml, profiles.yml checked into the repo root (safe precisely because DuckDB
+      needs no credentials — this is the trick that makes a zero-config dbt layer possible),
+      models/staging/stg_customers.sql + stg_orders.sql + stg_payments.sql with
+      models/staging/schema.yml, then models/customers.sql + orders.sql over them — the
+      staging/marts split in its minimal honest form. Also borrow .sqlfluff + .sqlfluffignore
+      for SQL linting defaults, .devcontainer.json, and
+      .github/workflows/validate_on_platforms.yml, which proves the project on multiple OSes.
+    whatToAvoid: >-
+      Seed-driven, so there is no ingestion story at all — do not present it as an end-to-end
+      pipeline. Only two marts and no incremental models, no snapshots, no macros worth
+      keeping. The checked-in profiles.yml pattern is safe ONLY for DuckDB; a generated layer
+      must not carry that habit over to a Snowflake or BigQuery target. images/ and etc/ are
+      ~1MB of PNG documentation that should be stripped when vendoring.
+    proposedLayerIds:
+      - data-dbt
+      - data-dbt-duckdb
+      - data-duckdb
+
+  - name: dbt-labs/dbt-utils
+    url: https://github.com/dbt-labs/dbt-utils
+    stack: dbt, sql, jinja
+    stars: 1779
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-07"
+    complexity: simple
+    portability: >-
+      Cross-warehouse by construction — the whole package exists to paper over dialect
+      differences, so it is the least likely thing here to break when a layer swaps targets.
+    whatToBorrow: >-
+      The generic-test vocabulary that every serious dbt project depends on, and the single
+      highest-leverage dependency to pin in a data-dbt layer. Borrow the packages.yml
+      dependency block and the schema.yml test idioms it unlocks: equal_rowcount,
+      fewer_rows_than, recency, at_least_one, not_constant, cardinality_equality,
+      expression_is_true, unique_combination_of_columns, accepted_range. Together with dbt's
+      four built-ins these are the assertions a generated project should ship with on day one.
+    whatToAvoid: >-
+      It is a dbt package, not a project — it produces nothing on its own and must be layered
+      onto a real dbt_project.yml. Do not vendor the macro source into a generated repo; add it
+      to packages.yml with a pinned version and let dbt deps fetch it, otherwise you inherit
+      maintenance of cross-dialect SQL you did not write. Several older cross-db macros
+      (datediff, dateadd, split_part) migrated into dbt-core adapters and are now shims — use
+      the dbt built-ins directly rather than routing through dbt_utils for those.
+    proposedLayerIds:
+      - data-dbt
+      - data-dbt-tests
+
+  - name: duckdb/dbt-duckdb
+    url: https://github.com/duckdb/dbt-duckdb
+    stack: python, dbt, duckdb, sql
+    stars: 1317
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-14"
+    complexity: standard
+    whatToBorrow: >-
+      The adapter that makes a credential-free dbt layer viable, plus the clearest worked
+      examples of DuckDB-as-lakehouse. The functional test suite doubles as a spec — verified
+      files under tests/functional/adapter/: test_external.py (materializing models straight
+      out to Parquet/CSV rather than into a table — the mechanism behind a file-based gold
+      layer), test_attach.py (ATTACH-ing other databases so one dbt run spans several stores),
+      test_community_extensions.py, test_constraints.py, and a cluster of DuckLake tests —
+      test_ducklake_partitioned_by.py, test_ducklake_sorted_by.py and their _integration
+      variants — which is the current, actively-developed catalog story worth tracking.
+      Borrow the profiles.yml shape (path, extensions, settings, attach blocks).
+    whatToAvoid: >-
+      Single-process and single-writer: concurrent dbt threads against one DuckDB file will
+      contend, so a generated layer should keep threads low rather than copy a warehouse-tuned
+      value. test_concurrency.py exists because this is a real edge. Not a fit for a
+      multi-user production warehouse — position DuckDB layers as local/CI/single-node
+      analytics. Extension auto-install reaches the network on first run, which will fail in
+      an air-gapped or locked-down CI runner unless extensions are pre-baked.
+    proposedLayerIds:
+      - data-dbt-duckdb
+      - data-duckdb
+      - data-lakehouse-ducklake
+
+  - name: dbt-labs/dbt-project-evaluator
+    url: https://github.com/dbt-labs/dbt-project-evaluator
+    stack: dbt, sql, jinja
+    stars: 567
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-06-24"
+    complexity: standard
+    whatToBorrow: >-
+      dbt Labs' own executable opinion about what a well-formed dbt project looks like — it
+      reads the manifest and flags violations, which is exactly the rule set a Groundplan dbt
+      layer should be generated to satisfy from the start. Verified macros/ contents worth
+      adapting: insert_resources_from_graph.sql (the graph-introspection entrypoint),
+      find_all_hard_coded_references.sql, calculate_sql_complexity.sql,
+      calculate_number_lines.sql, get_directory_pattern.sql, and
+      macros/on-run-end/print_dbt_project_evaluator_issues.sql — the on-run-end hook that turns
+      findings into build output. macros/cross_db_shim/ (duckdb_shims.sql,
+      clickhouse_shims.sql, type_string.sql, quote_identifier.sql) is a compact, copyable
+      pattern for supporting several warehouses from one macro set.
+    whatToAvoid: >-
+      It is a package that must run against an already-built project, so it cannot be the
+      starting point of a layer — it is the check applied after. Its conventions are opinionated
+      (fanouts, rejoins, staging-naming) and will emit noise on a deliberately small generated
+      project; ship it with a seeded exceptions config rather than letting first-run output
+      look like failure. It materializes its own models into the target schema, which surprises
+      people on a shared warehouse.
+    proposedLayerIds:
+      - data-dbt-tests
+      - data-dbt-governance
+
+  - name: elementary-data/elementary
+    url: https://github.com/elementary-data/elementary
+    stack: python, dbt, sql, jinja
+    stars: 2379
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-08"
+    complexity: standard
+    whatToBorrow: >-
+      The data-observability layer that rides on dbt artifacts instead of a separate agent —
+      the cheapest credible way to give a generated project anomaly detection and alerting.
+      The whole mechanism is a dbt package inside the repo: verified tree under
+      elementary/monitor/dbt_project/ with its own dbt_project.yml and a macros/alerts/ tree
+      split into population/ (model_alerts.sql, test_alerts.sql, source_freshness_alerts.sql,
+      populate_alerts_table.sql), fetchers/ (get_alerts.sql), and manage/
+      (update_sent_alerts.sql, update_skipped_alerts.sql). That
+      populate -> fetch -> mark-sent state machine is the reusable idea: alert dedup and
+      delivery tracking held in warehouse tables, no external queue. Also borrow
+      macros/base_queries/ (current_tests_run_results_query.sql, owners.sql) for reading
+      run_results without parsing JSON by hand.
+    whatToAvoid: >-
+      Apache-2.0 covers this repo, but Elementary Cloud is the commercial product and parts of
+      the docs assume it — keep a generated layer on the OSS CLI report and Slack path only,
+      and do not wire anything to a hosted endpoint. It writes a sizable schema of its own
+      (elementary schema) and its on-run-end hooks add measurable time to every dbt invocation,
+      so make it opt-in rather than default-on for small projects. Anomaly detection needs
+      history to be meaningful and will be silent or noisy for the first few weeks.
+    proposedLayerIds:
+      - data-dbt-tests
+      - data-observability-elementary
+      - observability-otel
+
+  - name: dlt-hub/dlt
+    url: https://github.com/dlt-hub/dlt
+    stack: python
+    stars: 5629
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-18"
+    complexity: standard
+    portability: >-
+      Runs as a plain Python library in whatever already exists — a script, an Airflow task, a
+      Dagster asset, a Lambda. No server, no scheduler, no separate deployment. That makes it
+      the ingestion choice that composes with every orchestration layer here rather than
+      competing with one.
+    whatToBorrow: >-
+      Schema evolution and incremental loading as library code rather than platform features —
+      the missing ingestion half that jaffle_shop_duckdb has no answer for. docs/examples/ is
+      an unusually large, runnable pattern library (verified): _template/_template.py is the
+      literal scaffold dlt intends you to copy for a new pipeline, and
+      backfill_in_chunks/ is the pattern worth adapting for bounded historical loads. Borrow
+      the @dlt.resource / @dlt.source decorator shape, the write_disposition
+      (append/replace/merge) declaration, and the dlt.sources.incremental cursor — that trio is
+      the whole incremental-ingestion contract and it is small enough to teach in a template.
+    whatToAvoid: >-
+      docs/examples/archive/ is explicitly retired (singer_tap_example.py, rasa_example.py,
+      discord_iterator.py et al) — it still appears in search results and must not be used as a
+      current pattern. Automatic schema inference is a real convenience and a real hazard: a
+      generated layer should pin an explicit schema contract for anything downstream depends
+      on, not let column types drift from whatever the source returned this run. dlt+ is a
+      separate commercial product; keep layers on the OSS package.
+    proposedLayerIds:
+      - data-ingestion-dlt
+      - data-cdc-incremental
+
+  - name: pola-rs/polars
+    url: https://github.com/pola-rs/polars
+    stack: rust, python
+    stars: 39041
+    spdx: MIT
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-18"
+    complexity: standard
+    portability: >-
+      MIT with no reciprocal terms — the least encumbered license in this file. Single
+      dependency, no service, runs anywhere Python or Rust does.
+    whatToBorrow: >-
+      The DataFrame engine for the transform step when the work is genuinely Python rather than
+      SQL. LICENSE was read in full: it is MIT text (Copyright Ritchie Vink, with an NVIDIA
+      portions notice), despite the repo's size making people assume something heavier. Borrow
+      the lazy-frame idiom — scan_parquet/scan_csv -> chained expressions -> collect() — as the
+      default shape for a transform layer, because it is what lets predicate and projection
+      pushdown work; an eager read_parquet template teaches the wrong habit. The Arrow-native
+      memory model is what makes zero-copy handoff to DuckDB, delta-rs and pyiceberg
+      actually free, so a Polars layer composes with the lake layers below rather than
+      duplicating them.
+    whatToAvoid: >-
+      Not an orchestrator and not a storage format — it is one process, one machine, and a
+      generated layer should say so rather than implying distributed capability. The API still
+      moves between minor versions and older Stack Overflow answers are frequently stale, so
+      pin the version in the layer. The NVIDIA GPU engine path has different packaging and
+      hardware assumptions — do not put it in a default template.
+    proposedLayerIds:
+      - data-transform-polars
+      - data-duckdb
+
+  - name: delta-io/delta-rs
+    url: https://github.com/delta-io/delta-rs
+    stack: rust, python
+    stars: 3263
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-17"
+    complexity: standard
+    portability: >-
+      The reason to prefer this over Delta Lake proper: no JVM and no Spark cluster. A Delta
+      table becomes reachable from a plain Python process, which is what makes a
+      medallion-architecture layer feasible at template scale.
+    whatToBorrow: >-
+      ACID table semantics without a cluster — the storage half of a bronze/silver/gold layer.
+      Verified python/deltalake/ modules: table.py (DeltaTable — the read/write/history/vacuum
+      surface), transaction.py (the commit protocol, worth reading before designing any
+      concurrent-writer layer), schema.py, query.py. Borrow the write_deltalake mode contract
+      (append / overwrite / error) and the merge/upsert path, since a medallion silver layer is
+      mostly upserts. Time travel by version or timestamp is the feature to expose in a
+      template, because it is what distinguishes a Delta layer from writing Parquet directly.
+    whatToAvoid: >-
+      Feature parity with the JVM implementation is not complete and the gap moves — check
+      before promising deletion vectors, column mapping, or a given protocol reader/writer
+      version in a generated layer. Concurrent writers from separate processes need a real
+      locking provider on S3; the default local behaviour will not save you and this is the
+      most common way people corrupt a Delta table. vacuum() with a short retention is
+      irreversible and should never be a template default.
+    proposedLayerIds:
+      - data-lakehouse-delta
+      - data-medallion
+
+  - name: apache/iceberg-python
+    url: https://github.com/apache/iceberg-python
+    stack: python
+    stars: 1097
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-19"
+    complexity: professional
+    whatToBorrow: >-
+      The catalog abstraction, which is the part of Iceberg that actually determines whether a
+      template is portable. Verified pyiceberg/catalog/ modules: sql.py, hive.py, glue.py,
+      dynamodb.py, bigquery_metastore.py, memory.py, noop.py (REST lives alongside these). One
+      table-operations API over all of them means a Groundplan layer can ship memory/sql for
+      local development and swap to glue or REST in production by changing config only — that
+      swap is the single most valuable thing to borrow here. Also take the .pyiceberg.yaml /
+      PYICEBERG_ environment-variable configuration convention, which keeps catalog endpoints
+      and credentials out of code by construction.
+    whatToAvoid: >-
+      Iceberg's value is multi-engine interop; if a project only ever reads its own tables from
+      one process, this is real complexity for no return — route those to Delta or plain
+      Parquet. Writer support has historically trailed reader support, so verify the specific
+      write path (partition evolution, row-level deletes) before a layer depends on it. The
+      catalog is operational infrastructure with its own availability and backup story; a
+      template that quietly stands up a Hive metastore or Glue database is making an
+      infrastructure decision on the user's behalf and should say so loudly.
+    proposedLayerIds:
+      - data-lakehouse-iceberg
+      - data-medallion
+
+  - name: SQLMesh/sqlmesh
+    url: https://github.com/SQLMesh/sqlmesh
+    stack: python, sql
+    stars: 3204
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-16"
+    complexity: professional
+    whatToBorrow: >-
+      The strongest available answer to the problem dbt leaves open — knowing what a SQL change
+      actually breaks before it runs. Borrow the plan/apply workflow (column-level lineage
+      diffing a change against the deployed state, then previewing in an isolated virtual
+      environment) as the CI shape for any transformation layer; it maps cleanly onto a
+      pull-request gate. Verified examples/: ibis/ is a complete runnable project —
+      config.yaml, models/full_model.sql, audits/assert_positive_order_ids.sql, macros/ — and
+      is the right size to adapt into a layer. The extension-point pattern worth copying for
+      teams with non-standard load semantics is examples/custom_materializations/, which nests
+      one level deeper than the name suggests: the code sits in
+      examples/custom_materializations/custom_materializations/ (custom_kind.py, full.py,
+      __init__.py) with pyproject.toml at the directory root — it is packaged as an installable
+      distribution, which is itself the point, since custom materializations must be importable. Audits are the built-in test
+      primitive; use them rather than bolting on a separate quality tool.
+    whatToAvoid: >-
+      Repo has MOVED from TobikoData/sqlmesh to SQLMesh/sqlmesh — the old path redirects, but
+      pin the new one in any generated config or docs. Tobiko Cloud is the commercial product
+      and some workflow documentation assumes it; keep layers on the OSS CLI. It is a genuine
+      alternative to dbt, not a companion — do not generate a project that runs both over the
+      same models. The virtual-environment model creates real schemas in the warehouse and its
+      cost profile surprises people on per-query-billed engines.
+    proposedLayerIds:
+      - data-transform-sqlmesh
+      - data-dbt-governance
+      - ci-github-actions
+
+  - name: dagster-io/dagster
+    url: https://github.com/dagster-io/dagster
+    stack: python
+    stars: 15860
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-17"
+    complexity: professional
+    whatToBorrow: >-
+      Asset-centric orchestration — you declare the tables that should exist and the graph is
+      derived, which is a materially better fit for template generation than declaring tasks.
+      examples/ is the richest per-topic starter collection of any orchestrator here (verified
+      READMEs): quickstart_etl/ is the strip-down-ready base;
+      docs_projects/project_etl_tutorial/ ships transform/jdbt/ and dashboard/ subprojects and
+      is the closest thing to a complete reference pipeline;
+      docs_projects/project_dbt/src/project_dbt/analytics/ is the dbt-integration pattern to
+      copy when dbt models should appear as first-class assets rather than one opaque task;
+      project_snowflake_dynamic_tables/, project_multi_workspace_databricks/ and
+      project_serverless_hybrid/ cover the warehouse and deployment-topology variants.
+      Borrow the Definitions object, the resource/config-schema split (which is where
+      environment-specific values belong), and asset checks as the data-quality hook.
+    whatToAvoid: >-
+      Dagster+ is the commercial hosted product; several examples assume it, and
+      project_serverless_hybrid/ in particular is about Dagster+ deployment topology — do not
+      let that leak into an OSS layer. The asset model is a real conceptual shift and a
+      simple-tier layer should not use it; route simple needs to a script or Prefect. The
+      repo's Python packaging is a large multi-package monorepo — take single example
+      directories, never the tree.
+    proposedLayerIds:
+      - data-orchestration-dagster
+      - data-dbt
+      - data-medallion
+      - docker-compose
+
+  - name: astronomer/astronomer-cosmos
+    url: https://github.com/astronomer/astronomer-cosmos
+    stack: python, airflow, dbt
+    stars: 1232
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-17"
+    complexity: professional
+    portability: >-
+      The highest-value-per-line source in this file for a specific reason: it is pure glue
+      between two things a generated project will already have (Airflow, dbt), so it adds
+      capability without adding a new system to operate.
+    whatToBorrow: >-
+      It parses a dbt project and expands each model into its own Airflow task, so failures are
+      per-model instead of one opaque "dbt run failed" — that expansion is the pattern worth
+      taking. The genuinely reusable asset is cosmos/profiles/, which solves credential
+      translation once for every warehouse: verified modules for postgres, snowflake, bigquery,
+      databricks, redshift, athena, trino, clickhouse, duckdb, spark, mysql, oracle, sqlserver,
+      teradata, vertica, exasol, starrocks. Each maps Airflow connection fields to dbt
+      profiles.yml keys, which means a generated layer can hold warehouse credentials in one
+      place (the Airflow connection) rather than in a checked-in profiles.yml — the direct fix
+      for the one habit jaffle_shop_duckdb should not teach. dev/dags/basic_cosmos_dag.py is
+      the minimal working DAG to adapt.
+    whatToAvoid: >-
+      Parsing cost is the known operational trap — naive configuration re-parses the dbt
+      project on every scheduler loop; a generated layer must set a manifest-based load mode,
+      not the default. Requires both Airflow and dbt to be correctly installed and version-
+      compatible in one environment, which is the single most common failure mode; pin both.
+      Astronomer is a commercial vendor and their docs route toward Astro — the package itself
+      is plain Apache-2.0 and works on vanilla Airflow, so keep layers on that path.
+    proposedLayerIds:
+      - data-orchestration-airflow
+      - data-dbt
+      - data-dbt-duckdb
+
+  - name: apache/airflow
+    url: https://github.com/apache/airflow
+    stack: python
+    stars: 46168
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-19"
+    complexity: enterprise
+    whatToBorrow: >-
+      The default answer in regulated and large environments — worth a layer for institutional
+      reasons even where Dagster or Prefect is technically nicer, because it is what audit and
+      platform teams already accept. Borrow the provider-package model: airflow itself stays
+      thin and every integration (AWS, Google, Snowflake, dbt, Postgres, Kubernetes) is a
+      separately versioned apache-airflow-providers-* package, so a generated layer declares
+      exactly the providers it needs instead of pulling a monolith. The constraints-file
+      mechanism (pinned, per-Airflow-version constraint URLs used with pip) is the pattern that
+      makes Airflow installs reproducible at all, and is the thing most hand-rolled Airflow
+      templates get wrong. Take the Connections/Variables abstraction as the credential
+      boundary and the TaskFlow @dag/@task decorators as the modern authoring style.
+    whatToAvoid: >-
+      Heavy: scheduler, metadata database, executor, and workers are separate operational
+      concerns, and a template that hides that is doing the user a disservice — this belongs at
+      enterprise tier only. Do not vendor from the source tree; depend on released packages.
+      Airflow 3 changed enough (task execution interface, asset-oriented scheduling, deprecated
+      SubDagOperator patterns) that pre-3 examples found online will mislead — pin the major
+      version explicitly in the layer. LocalExecutor in docker-compose is a development
+      convenience and should never be presented as a production topology.
+    proposedLayerIds:
+      - data-orchestration-airflow
+      - docker-compose
+      - db-postgres
+      - queue-broker
+
+  - name: fivetran/great_expectations
+    url: https://github.com/fivetran/great_expectations
+    stack: python
+    stars: 11653
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-17"
+    complexity: enterprise
+    whatToBorrow: >-
+      The most complete standalone data-validation vocabulary — the right choice when
+      validation must cover sources dbt never sees (files landing in object storage, API
+      payloads, a pre-load staging area). Borrow the four-object model: Expectation Suites as
+      versioned, checked-in JSON/YAML assertions; Checkpoints as the runnable unit that binds a
+      suite to a batch; Data Docs as generated human-readable validation reports, which is the
+      artifact non-engineers and auditors actually consume; and Data Context as the config
+      root. That suite-as-code-artifact idea is the transferable part even if a project never
+      adopts the library. The expectation catalog itself (nullity, ranges, set membership,
+      regex, distributional, multi-column) is a good checklist for what a quality layer should
+      assert regardless of tool.
+    whatToAvoid: >-
+      NOTE THE MOVE: this repo now lives under fivetran/, not great-expectations/ — the old
+      path redirects but the vendor relationship changed and roadmap risk should be weighed
+      before a layer takes a hard dependency. Historically heavy and famously verbose to
+      configure; the 0.x-to-1.x API break invalidated most tutorials and blog posts still in
+      circulation, so pin the version and do not trust recalled API shapes. For a project that
+      is already all-dbt, dbt tests plus dbt-utils cover most of this at a fraction of the
+      setup cost — reserve Great Expectations for the non-dbt surface area.
+    proposedLayerIds:
+      - data-quality-great-expectations
+      - data-dbt-tests
+      - testing-unit
+
+  - name: debezium/debezium
+    url: https://github.com/debezium/debezium
+    stack: java, kafka
+    stars: 12929
+    spdx: Apache-2.0
+    licenseClass: vendorable
+    maintained: true
+    lastCommit: "2026-07-19"
+    complexity: enterprise
+    whatToBorrow: >-
+      The reference implementation of log-based CDC, and the source of the change-event
+      envelope that most of the ecosystem has standardised on: before / after / source / op /
+      ts_ms. Borrow that envelope shape for any streaming-ingestion layer even if the layer
+      never runs Debezium — downstream consumers and lake formats already expect it. The
+      connector configuration model (per-database connectors for Postgres, MySQL, MongoDB, SQL
+      Server, Oracle, Db2, Cassandra behind one event contract) is the pattern that lets a
+      generated layer swap source databases without rewriting consumers. Debezium Server is the
+      part to point a template at: it runs standalone and emits to Kinesis/Pub-Sub/Pulsar/Redis
+      without requiring a Kafka Connect cluster.
+    whatToAvoid: >-
+      Log-based CDC is a database-privilege and database-configuration problem before it is an
+      application problem — logical replication slots on Postgres, binlog format on MySQL,
+      supplemental logging on Oracle. A template cannot grant those, so it must document them
+      as prerequisites rather than appearing to just work. An abandoned Postgres replication
+      slot will retain WAL until the disk fills; this is the classic production incident and
+      belongs in generated docs. Java/Maven build, so vendoring source into a Python or
+      TypeScript project is not realistic — treat this as a deployed component and a contract
+      to imitate, not a codebase to copy from.
+    proposedLayerIds:
+      - data-cdc-debezium
+      - data-streaming-kafka
+      - queue-broker
+      - db-postgres
+
+droppedCandidates:
+  - >-
+    sodadata/soda-core — EXCLUDED ON LICENSE. LICENSE file reads "Elastic License 2.0 /
+    ## Acceptance", verified from raw.githubusercontent.com. Not Apache-2.0, contrary to
+    near-universal assumption. Elastic v2 forbids providing the software as a managed service
+    and forbids circumventing license keys; those terms would propagate to generated repos.
+    2394 stars, actively maintained, would otherwise have been a strong data-quality entry.
+  - >-
+    DataTalksClub/data-engineering-zoomcamp — EXCLUDED. 43,751 stars, last commit 2026-06-10, and
+    NO LICENSE FILE anywhere in the default-branch tree (searched for LICENSE/COPYING across
+    the full file listing). All rights reserved. Excellent to read and learn from; nothing may
+    be copied into a generated repository.
+  - >-
+    dbt-labs/jaffle-shop — EXCLUDED. api.github.com returned license.spdx_id = null directly
+    (one of only two calls that landed before the quota drained), and the root tree contains
+    exactly one file: README.md. dbt Labs' own flagship demo project is unlicensed. Replaced
+    by dbt-labs/jaffle_shop_duckdb, which is Apache-2.0.
+  - >-
+    calogica/dbt-expectations — DROPPED ON STALENESS, not license (Apache-2.0, 1229 stars).
+    Last push 2024-12-16, roughly nineteen months before this pass. The brief asks for CURRENT
+    2025-2026 sources; a dbt package that has not moved through two dbt-core minor cycles is a
+    liability. elementary + dbt-utils cover the same ground and are both current.
+  - >-
+    risingwavelabs/risingwave — NOT EXCLUDED, deferred (Apache-2.0 verified, 9175 stars,
+    last commit 2026-07-17). Held back only to stay inside the 15-source cap; debezium already
+    covers the CDC/streaming slot. Carries a caveat worth recording before any future layer
+    uses it — see licenseSurprises.
+  - >-
+    kestra-io/kestra (27381 stars, Apache-2.0 verified), datahub-project/datahub (12297,
+    Apache-2.0 verified), meltano/meltano (2565, MIT text verified — now copyright Arch Data,
+    Inc.), PrefectHQ/prefect (23425, Apache-2.0 verified), duckdb/duckdb (39522, MIT text
+    verified). All five passed the license gate and are current; all five were cut for
+    category overlap against stronger entries above, not for any defect. duckdb/duckdb in
+    particular is an engine rather than a template source — dbt-duckdb carries the borrowable
+    patterns.
+
+licenseSurprises:
+  - >-
+    soda-core is Elastic License 2.0, not Apache-2.0. This is the headline finding. Soda is the
+    reflexive recommendation for open-source data quality and is frequently listed alongside
+    Great Expectations as if the two were equivalently licensed. They are not.
+  - >-
+    Two of the most-starred repos in this category have NO LICENSE AT ALL: DataTalksClub's
+    zoomcamp (43,751 stars) and dbt Labs' own jaffle-shop. Star count and vendor pedigree
+    carry no information about licensing, which is exactly the trap the catalog rules warn
+    about — a 50k-star CC-BY-NC-ND collection was the previous pass's version of this.
+  - >-
+    dbt-labs/jaffle-shop (unlicensed) and dbt-labs/jaffle_shop_duckdb (Apache-2.0) are the same
+    owner, near-identical names, opposite outcomes on the gate. Anything resolving "jaffle shop"
+    by name alone will land on the wrong one roughly half the time.
+  - >-
+    risingwavelabs/risingwave is genuinely Apache-2.0 at the root, but the source tree contains
+    a license-key enforcement subsystem: src/license/src/ with feature.rs, key.pub, key.rs,
+    manager.rs, rwu.rs and a feature.json enumerating paid-tier features (TimeTravel,
+    GlueSchemaRegistry, SnowflakeSink, RedisSinkStream). It is a real Cargo crate — src/license/
+    has its own Cargo.toml — so the gating is a first-class component, not a stray file. The LICENSE file is not the whole story — Apache-2.0 code
+    can still gate functionality behind a purchased key. Read the tree, not just the header.
+  - >-
+    Two repos have MOVED, and both old paths still redirect so the breakage is silent:
+    great-expectations/great_expectations -> fivetran/great_expectations (an acquisition, with
+    the governance implications that carries) and TobikoData/sqlmesh -> SQLMesh/sqlmesh. Any
+    recalled URL for either is stale.
+  - >-
+    duckdb, polars and meltano all carry verbatim MIT text with no "MIT License" title line, so
+    a naive header-matching license scanner returns no match and may report them as unlicensed.
+    Match on the "Permission is hereby granted, free of charge" body, not the title.