@sqlanvil/cli 1.20.1 → 1.21.0

package/README.md

@@ -2,7 +2,7 @@
 
 **SQL workflow tool for BigQuery, Postgres, Supabase, and MySQL/MariaDB.**
 
-SQLAnvil is an open-source fork of [Dataform OSS](https://github.com/dataform-co/dataform) (Apache 2.0), extended with first-class PostgreSQL and Supabase support, plus adapters for other data sources. Define your data transformations in SQLX, have SQLAnvil compile them to idiomatic SQL, and run them against your warehouse.
+SQLAnvil is an open-source fork of [Dataform OSS](https://github.com/dataform-co/dataform) (Apache 2.0), extended with first-class PostgreSQL, Supabase, and MySQL/MariaDB support. Define your data transformations in SQLX, have SQLAnvil compile them to idiomatic SQL for your warehouse, and run, validate, and document the whole DAG from one CLI.
 
 > **SQLAnvil is not affiliated with or endorsed by Google.** The Dataform name and related marks are trademarks of Google LLC. See [NOTICE](NOTICE) for attribution.
 
@@ -10,10 +10,21 @@ SQLAnvil is an open-source fork of [Dataform OSS](https://github.com/dataform-co
 
 ## Features
 
-- **BigQuery** — full support: partitioning, clustering, labels, materialized views, `MERGE`-based incremental upserts
-- **PostgreSQL** — idiomatic DDL: native partitioning, `INSERT ... ON CONFLICT` upserts, btree/gin/gist/brin indexes, tablespaces, fillfactor
-- **Supabase** — extends Postgres with RLS policies, Realtime publications, pgvector indexes, and Supabase Wrappers _(coming soon)_
-- **MySQL / MariaDB** — portable MySQL DDL: CTAS tables, `CREATE OR REPLACE VIEW`, `ON DUPLICATE KEY UPDATE` incremental upserts (one adapter, validated against both engines)
+**Four first-class warehouses** — each adapter generates its warehouse's own idiomatic SQL, not translated BigQuery:
+
+- **PostgreSQL** — native partitioning (RANGE/LIST/HASH), `INSERT … ON CONFLICT` upserts, btree/hash/gin/gist/brin indexes (unique, partial, `INCLUDE`, opclasses), tablespaces, fillfactor, materialized views with in-place refresh policies
+- **Supabase** — everything Postgres, plus RLS policies, Realtime publications, pgvector indexes, and Supabase Wrappers (FDW) as declarative actions
+- **MySQL / MariaDB** — one adapter validated against both engines: CTAS tables, `ON DUPLICATE KEY UPDATE` incremental upserts, the full `mysql: {}` config block (engine/charset/collation/rowFormat, secondary indexes incl. `FULLTEXT`/`SPATIAL` and prefix lengths, native partitioning), materialized-view emulation
+- **BigQuery** — full upstream support: partitioning, clustering, labels, materialized views, `MERGE`-based incremental upserts
+
+**Beyond transformation** — the pieces most SQL workflow tools leave outside the pipeline:
+
+- **Cross-warehouse sources** — read BigQuery or MySQL/MariaDB from a Postgres/Supabase warehouse via named `connections:`: a live FDW bridge (Supabase Wrappers) or **runner-extract** (the CLI reads the source at run time and materializes a plain `ref()`-able table — keyless BigQuery supported). `sqlanvil introspect` scaffolds the declarations
+- **File imports & exports** — `type: "import"` loads Parquet/CSV/JSON from `s3://`/`gs://`/local into a `ref()`-able table (DuckDB bridge on Postgres/Supabase, native `LOAD DATA` on BigQuery); `type: "export"` writes query results back out to files
+- **Python script actions** — `python:` in `actions.yaml` runs file-staging and glue scripts as first-class DAG nodes: ordered by dependencies, covered by run history, no warehouse credentials injected (scripts stage files; `import` loads them). SQLAnvil validates the declared environment — interpreter version, `requirements.txt` vs installed packages, syntax — and never installs anything
+- **`sqlanvil validate`** — EXPLAIN/dry-run the *whole DAG* against the live warehouse without executing: an isolated shadow schema of empty stubs lets every downstream `ref()` resolve; results classify as PASS / FAIL / BLOCKED so one real error doesn't cascade
+- **Queryable artifacts** — every compile/run writes a Parquet catalog + run history under `target/`; `sqlanvil query` runs SQL over them, `sqlanvil inspect` summarizes, `sqlanvil docs` renders a self-contained HTML catalog
+- **Release-artifact runs** — `sqlanvil run --graph <file>` executes a stored `compile --json` output exactly as compiled; **named environments** (`--environment`) give per-env schema suffixes and vars
 - **SQLX + YAML + JS** — three authoring modes: SQL with config blocks, `actions.yaml` bulk definitions, or the JavaScript API
 
 ---
@@ -24,23 +35,32 @@ SQLAnvil is an open-source fork of [Dataform OSS](https://github.com/dataform-co
 npm install -g @sqlanvil/cli
 sqlanvil init my-project
 cd my-project
-# edit workflow_settings.yaml to configure your warehouse
+# edit workflow_settings.yaml + .df-credentials.json (see below)
 sqlanvil compile
 sqlanvil run
 ```
 
-**`workflow_settings.yaml` example (Postgres):**
+**`workflow_settings.yaml` (Postgres/Supabase):** the warehouse is a flat string; connection
+secrets never live here.
 
 ```yaml
-warehouse:
-  kind: postgres
-  host: localhost
-  port: 5432
-  database: analytics
-  user: sqlanvil_writer
-  password: ${PG_PASSWORD}
-  ssl: disable
-  defaultSchema: public
+warehouse: postgres            # bigquery | postgres | supabase | mysql
+defaultDataset: public         # the Postgres schema
+defaultAssertionDataset: sqlanvil_assertions
+sqlanvilCoreVersion: 1.20.1
+```
+
+**`.df-credentials.json`** (gitignored) holds the connection:
+
+```json
+{
+  "host": "localhost",
+  "port": 5432,
+  "database": "analytics",
+  "user": "sqlanvil_writer",
+  "password": "...",
+  "sslMode": "disable"
+}
 ```
 
 **First action (`definitions/my_view.sqlx`):**
@@ -58,23 +78,29 @@ SELECT 1 AS id, 'hello' AS greeting
 
 ## Documentation
 
-Full documentation at **[sqlanvil.com](https://sqlanvil.com)**.
+Full documentation at **[sqlanvil.com/docs](https://sqlanvil.com/docs/)**.
+
+- [Getting Started](https://sqlanvil.com/docs/getting-started/)
+- Warehouses: [PostgreSQL](https://sqlanvil.com/docs/guides/postgres/) · [Supabase](https://sqlanvil.com/docs/guides/supabase/) · [MySQL/MariaDB](https://sqlanvil.com/docs/guides/mysql/) · [BigQuery](https://sqlanvil.com/docs/guides/bigquery/)
+- [Cross-warehouse sources](https://sqlanvil.com/docs/guides/foreign-wrappers/) · [File Imports](https://sqlanvil.com/docs/guides/imports/) · [File Exports](https://sqlanvil.com/docs/guides/exports/) · [Python Script Actions](https://sqlanvil.com/docs/guides/python-actions/)
+- [Validate](https://sqlanvil.com/docs/guides/validate/) · [Named Environments](https://sqlanvil.com/docs/guides/environments/) · [Artifacts & Catalog](https://sqlanvil.com/docs/guides/artifacts/)
+- [Configs Reference](https://sqlanvil.com/docs/reference/configs/) · [What's New](https://sqlanvil.com/docs/whats-new/)
 
-- [Getting Started](https://sqlanvil.com/getting-started/)
-- [BigQuery Guide](https://sqlanvil.com/guides/bigquery/)
-- [PostgreSQL Guide](https://sqlanvil.com/guides/postgres/)
-- [Supabase Guide](https://sqlanvil.com/guides/supabase/)
-- [Configs Reference](https://sqlanvil.com/reference/configs/)
+Writing sqlanvil projects with an AI agent? Point it at [`AGENTS.md`](AGENTS.md) — it corrects
+the Dataform/BigQuery priors that otherwise produce broken sqlanvil code. Contributor/design
+docs live in the separate [`sqlanvil/docs`](https://github.com/sqlanvil/docs) repo.
 
 ---
 
 ## Project layout
 
 ```
-core/         Compiler + action types (table/view/incremental/assertion/operation/notebook/declaration)
-cli/          CLI entrypoint and per-adapter glue (cli/api/dbadapters/)
+core/         Compiler + action types (table/view/incremental/assertion/operation/declaration/import/export/script/…)
+cli/          CLI entrypoint and per-adapter glue (cli/api/dbadapters/ — bigquery, postgres, supabase, mysql)
 protos/       Protobuf definitions for core/configs/execution/db_adapter
-examples/     Sample SQLAnvil projects
+tests/        Integration specs against real warehouses (Postgres, Supabase, MySQL/MariaDB, BigQuery)
+tools/        Bazel rules + Docker fixtures for the integration databases
+examples/     Sample SQLAnvil projects (see examples/supabase_bigquery_mailing_list)
 scripts/      ./scripts/run is the CLI entrypoint wrapper
 ```
 
@@ -82,28 +108,22 @@ scripts/      ./scripts/run is the CLI entrypoint wrapper
 
 ## Building from source
 
-SQLAnvil uses [Bazel](https://bazel.build) (via Bazelisk). Native Bazel works on macOS
-and Linux (Bazel 7 + bzlmod):
+SQLAnvil uses [Bazel](https://bazel.build) 7 with bzlmod (via Bazelisk). Native Bazel works on
+macOS and Linux:
 
 ```bash
-# Build the proto layer
-bazel build //protos:sqlanvil_proto
-
 # Run the CLI
 ./scripts/run help
 
 # Run tests
 bazel test //core/... //cli/...
-```
 
-A Docker-based build container (`./scripts/docker-bazel`) remains available as an
-optional hermetic environment. See [contributing.md](contributing.md) for full instructions.
-
----
-
-## Documentation
+# Integration tests against a local Docker Postgres
+./tools/postgres/run-postgres-db.sh
+```
 
-Reference docs and design documents live in the separate [`sqlanvil/docs`](https://github.com/sqlanvil/docs) repo, published at [sqlanvil.com/docs](https://sqlanvil.com/docs/).
+A Docker-based build container (`./scripts/docker-bazel`) remains available as an optional
+hermetic environment. See [contributing.md](contributing.md) for full instructions.
 
 ---
 
@@ -111,4 +131,4 @@ Reference docs and design documents live in the separate [`sqlanvil/docs`](https
 
 Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
 
-SQLAnvil is a derivative of Dataform OSS (originally developed by Dataform Co and contributed to by Google LLC). It has been fully renamed and extended with first-class PostgreSQL/Supabase support. Licensed under Apache License 2.0.
+SQLAnvil is a derivative of Dataform OSS (originally developed by Dataform Co and contributed to by Google LLC). It has been fully renamed and extended with first-class PostgreSQL/Supabase/MySQL support. Licensed under Apache License 2.0.

package/bundle.js

@@ -49392,11 +49392,11 @@ function setOrValidateTableEnumType(table) {
 function prune(compiledGraph, runConfig) {
     compiledGraph.tables.forEach(setOrValidateTableEnumType);
     const includedActionNames = computeIncludedActionNames(compiledGraph, runConfig);
-    return Object.assign(Object.assign({}, compiledGraph), { tables: compiledGraph.tables.filter(action => includedActionNames.has(targetAsReadableString(action.target))), assertions: compiledGraph.assertions.filter(action => includedActionNames.has(targetAsReadableString(action.target))), operations: compiledGraph.operations.filter(action => includedActionNames.has(targetAsReadableString(action.target))), exports: compiledGraph.exports.filter(action => includedActionNames.has(targetAsReadableString(action.target))), imports: compiledGraph.imports.filter(action => includedActionNames.has(targetAsReadableString(action.target))), scripts: (compiledGraph.scripts || []).filter(action => includedActionNames.has(targetAsReadableString(action.target))) });
+    return Object.assign(Object.assign({}, compiledGraph), { tables: compiledGraph.tables.filter(action => includedActionNames.has(targetAsReadableString(action.target))), assertions: compiledGraph.assertions.filter(action => includedActionNames.has(targetAsReadableString(action.target))), operations: compiledGraph.operations.filter(action => includedActionNames.has(targetAsReadableString(action.target))), exports: compiledGraph.exports.filter(action => includedActionNames.has(targetAsReadableString(action.target))), imports: compiledGraph.imports.filter(action => includedActionNames.has(targetAsReadableString(action.target))), extracts: (compiledGraph.extracts || []).filter(action => includedActionNames.has(targetAsReadableString(action.target))), scripts: (compiledGraph.scripts || []).filter(action => includedActionNames.has(targetAsReadableString(action.target))) });
 }
 function computeIncludedActionNames(compiledGraph, runConfig) {
     var _a, _b, _c;
-    const allActions = [].concat(compiledGraph.tables, compiledGraph.operations, compiledGraph.assertions, compiledGraph.exports, compiledGraph.imports, compiledGraph.scripts || []);
+    const allActions = [].concat(compiledGraph.tables, compiledGraph.operations, compiledGraph.assertions, compiledGraph.exports, compiledGraph.imports, compiledGraph.extracts || [], compiledGraph.scripts || []);
     const allActionNames = new Set(allActions.map(action => targetAsReadableString(action.target)));
     const allActionsByName = new Map(allActions.map(action => [targetAsReadableString(action.target), action]));
     const hasActionSelector = ((_a = runConfig.actions) === null || _a === void 0 ? void 0 : _a.length) > 0;
@@ -50606,7 +50606,7 @@ function collectEvaluationQueries(queryOrAction, concatenate, queryModifier = (q
         .filter(validationQuery => !!validationQuery.query);
 }
 
-const version = "1.20.1";
+const version = "1.21.0";
 const dataformVersion = "3.0.61";
 
 async function build(compiledGraph, runConfig, dbadapter) {
@@ -50995,10 +50995,10 @@ node_modules/
 function postgresCredentialsTemplate(warehouse) {
     const isSupabase = warehouse === "supabase";
     const template = {
-        host: isSupabase ? "db.<project-ref>.supabase.co" : "localhost",
+        host: isSupabase ? "aws-1-<region>.pooler.supabase.com" : "localhost",
         port: 5432,
         database: "postgres",
-        user: "postgres",
+        user: isSupabase ? "postgres.<your-project-ref>" : "postgres",
         password: "",
         sslMode: isSupabase ? "require" : "disable",
         defaultSchema: "public"
@@ -51041,7 +51041,9 @@ async function init(projectDir, projectConfig) {
         workflowSettings.defaultProject = projectConfig.defaultDatabase;
         workflowSettings.defaultLocation = projectConfig.defaultLocation;
     }
-    workflowSettings.defaultDataset = projectConfig.defaultSchema || "sqlanvil";
+    const isPostgresLike = warehouse === "postgres" || warehouse === "supabase";
+    workflowSettings.defaultDataset =
+        projectConfig.defaultSchema || (isPostgresLike ? "public" : "sqlanvil");
     workflowSettings.defaultAssertionDataset =
         projectConfig.assertionSchema || "sqlanvil_assertions";
     if (isBigQuery && projectConfig.defaultIcebergConfig) {
@@ -51062,6 +51064,17 @@ async function init(projectDir, projectConfig) {
     if (projectConfig.builtinAssertionNamePrefix) {
         workflowSettings.builtinAssertionNamePrefix = projectConfig.builtinAssertionNamePrefix;
     }
+    if (isPostgresLike) {
+        workflowSettings.connections = {
+            bigquery_public: sqlanvil.ConnectionConfig.create({
+                platform: "bigquery",
+                project: "bigquery-public-data",
+                dataset: "geo_us_boundaries",
+                billingProject: "REPLACE_WITH_YOUR_GCP_PROJECT",
+                mode: "runner-extract"
+            })
+        };
+    }
     fs__namespace$1.writeFileSync(workflowSettingsYamlPath, jsYaml.dump(workflowSettings));
     filesWritten.push(workflowSettingsYamlPath);
     fs__namespace$1.writeFileSync(gitignorePath, gitIgnoreContents);
@@ -51070,39 +51083,140 @@ async function init(projectDir, projectConfig) {
         fs__namespace$1.writeFileSync(path__namespace.join(projectDir, CREDENTIALS_FILENAME), warehouse === "mysql" ? mysqlCredentialsTemplate() : postgresCredentialsTemplate(warehouse));
         filesWritten.push(path__namespace.join(projectDir, CREDENTIALS_FILENAME));
     }
-    const definitionsDir = path__namespace.join(projectDir, "definitions");
-    fs__namespace$1.mkdirSync(definitionsDir);
-    dirsCreated.push(definitionsDir);
-    const sourcesDir = path__namespace.join(definitionsDir, "sources");
-    fs__namespace$1.mkdirSync(sourcesDir);
-    dirsCreated.push(sourcesDir);
-    const ecommerceDir = path__namespace.join(sourcesDir, "ecommerce");
-    fs__namespace$1.mkdirSync(ecommerceDir);
-    dirsCreated.push(ecommerceDir);
-    const intermediateDir = path__namespace.join(definitionsDir, "intermediate");
-    fs__namespace$1.mkdirSync(intermediateDir);
-    dirsCreated.push(intermediateDir);
-    const outputsDir = path__namespace.join(definitionsDir, "outputs");
-    fs__namespace$1.mkdirSync(outputsDir);
-    dirsCreated.push(outputsDir);
-    const salesDir = path__namespace.join(outputsDir, "sales");
-    fs__namespace$1.mkdirSync(salesDir);
-    dirsCreated.push(salesDir);
-    const ordersDir = path__namespace.join(outputsDir, "orders");
-    fs__namespace$1.mkdirSync(ordersDir);
-    dirsCreated.push(ordersDir);
-    const marketingDir = path__namespace.join(outputsDir, "marketing");
-    fs__namespace$1.mkdirSync(marketingDir);
-    dirsCreated.push(marketingDir);
-    const testDir = path__namespace.join(definitionsDir, "test");
-    fs__namespace$1.mkdirSync(testDir);
-    dirsCreated.push(testDir);
-    const extraDir = path__namespace.join(definitionsDir, "extra");
-    fs__namespace$1.mkdirSync(extraDir);
-    dirsCreated.push(extraDir);
-    const includesDir = path__namespace.join(projectDir, "includes");
-    fs__namespace$1.mkdirSync(includesDir);
-    dirsCreated.push(includesDir);
+    const mkdir = (...segments) => {
+        const dir = path__namespace.join(projectDir, ...segments);
+        fs__namespace$1.mkdirSync(dir, { recursive: true });
+        dirsCreated.push(dir);
+        return dir;
+    };
+    const write = (filePath, contents) => {
+        fs__namespace$1.writeFileSync(filePath, contents);
+        filesWritten.push(filePath);
+    };
+    const keep = (dir) => write(path__namespace.join(dir, ".gitkeep"), "");
+    mkdir("definitions");
+    const sourcesDir = mkdir("definitions", "sources");
+    const intermediateDir = mkdir("definitions", "intermediate");
+    const salesDir = mkdir("definitions", "outputs", "sales");
+    const reportingDir = mkdir("definitions", "outputs", "reporting");
+    const testDir = mkdir("definitions", "test");
+    keep(mkdir("includes"));
+    const withBigQuerySource = warehouse !== "mysql";
+    write(path__namespace.join(sourcesDir, "app_orders.sqlx"), `-- A declaration: a table that already exists in your warehouse (created by your
+-- application, an import, or another pipeline). Point name/dataset at a real table —
+-- downstream models read it with \${ref("app_orders")}.
+config {
+  type: "declaration",
+  name: "app_orders",
+  description: "Raw orders table owned by the application — replace with one of your tables."
+}
+`);
+    write(path__namespace.join(intermediateDir, "stg_app_orders.sqlx"), `config {
+  type: "view",
+  description: "Staging view over the raw orders: rename, cast, and filter here — downstream models only read staged data."
+}
+
+SELECT
+  order_id,
+  product,
+  quantity,
+  amount,
+  postal_code,
+  ordered_at
+FROM \${ref("app_orders")}
+WHERE amount IS NOT NULL
+`);
+    write(path__namespace.join(salesDir, "daily_sales.sqlx"), `config {
+  type: "table",
+  description: "Sales by day and product, built from the staged orders."
+}
+
+SELECT
+  CAST(ordered_at AS DATE) AS sale_date,
+  product,
+  SUM(quantity) AS units_sold,
+  SUM(amount) AS revenue
+FROM \${ref("stg_app_orders")}
+GROUP BY CAST(ordered_at AS DATE), product
+`);
+    write(path__namespace.join(reportingDir, "product_revenue.sqlx"), `config {
+  type: "view",
+  description: "Revenue by product, straight off the staged orders."
+}
+
+SELECT
+  product,
+  SUM(quantity) AS units_sold,
+  SUM(amount) AS revenue
+FROM \${ref("stg_app_orders")}
+GROUP BY product
+`);
+    write(path__namespace.join(testDir, "assert_sales_amounts_positive.sqlx"), `config {
+  type: "assertion",
+  description: "Business rule: every sales row has positive units and revenue."
+}
+
+-- An assertion FAILS when its query returns rows — select the violations.
+SELECT *
+FROM \${ref("daily_sales")}
+WHERE units_sold <= 0 OR revenue <= 0
+`);
+    if (withBigQuerySource) {
+        write(path__namespace.join(sourcesDir, "bigquery_zip_codes.sqlx"), isBigQuery
+            ? `-- A declaration for a table in another project: Google's public ZIP geo data.
+-- BigQuery reads it natively — \${ref("zip_codes")} resolves to the full name.
+config {
+  type: "declaration",
+  database: "bigquery-public-data",
+  schema: "geo_us_boundaries",
+  name: "zip_codes",
+  description: "US ZIP codes from BigQuery public data."
+}
+`
+            : `-- A cross-warehouse source: Google's public ZIP geo data in BigQuery, read through
+-- the "bigquery_public" connection (see workflow_settings.yaml). With
+-- mode: runner-extract the CLI reads BigQuery at run time and materializes the rows
+-- as a plain table here — \${ref("zip_codes")} works like any other source.
+-- Scaffold real sources with:  sqlanvil introspect bigquery_public <dataset.table>
+config {
+  type: "declaration",
+  connection: "bigquery_public",
+  name: "zip_codes",
+  description: "US ZIP codes from BigQuery public data, materialized by runner-extract.",
+  columnTypes: {
+    zip_code: "text",
+    city: "text",
+    state_code: "text"
+  }
+}
+`);
+        write(path__namespace.join(intermediateDir, "stg_zip_codes.sqlx"), `config {
+  type: "view",
+  description: "Staged ZIP geography from the BigQuery source."
+}
+
+SELECT
+  zip_code,
+  city,
+  state_code
+FROM \${ref("zip_codes")}
+`);
+        write(path__namespace.join(reportingDir, "orders_by_region.sqlx"), `config {
+  type: "table",
+  description: "Orders and revenue by state/city — joins both staged sources."
+}
+
+SELECT
+  z.state_code,
+  z.city,
+  COUNT(*) AS orders,
+  SUM(o.amount) AS revenue
+FROM \${ref("stg_app_orders")} o
+JOIN \${ref("stg_zip_codes")} z
+  ON z.zip_code = o.postal_code
+GROUP BY z.state_code, z.city
+`);
+    }
     return {
         filesWritten,
         dirsCreated
@@ -56125,7 +56239,7 @@ function runCli() {
             {
                 format: `init [${projectDirOption.name}] [${ProjectConfigOptions.defaultDatabase.name}]` +
                     ` [${ProjectConfigOptions.defaultLocation.name}]`,
-                description: "Create a new sqlanvil project (BigQuery, Postgres, or Supabase).",
+                description: "Create a new sqlanvil project (BigQuery, Postgres, Supabase, or MySQL/MariaDB).",
                 positionalOptions: [
                     projectDirOption,
                     positionalOption(ProjectConfigOptions.defaultDatabase.name, {

package/package.json

@@ -64,7 +64,7 @@
     "bin": {
         "sqlanvil": "bundle.js"
     },
-    "version": "1.20.1",
+    "version": "1.21.0",
     "name": "@sqlanvil/cli",
     "description": "sqlanvil command line interface.",
     "main": "bundle.js"