sqlbuild 0.55.3__tar.gz → 0.55.5__tar.gz

{sqlbuild-0.55.3 → sqlbuild-0.55.5}/Cargo.lock

@@ -572,7 +572,7 @@ checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
 
 [[package]]
 name = "sqlbuild-kata-native"
-version = "0.55.3"
+version = "0.55.5"
 dependencies = [
  "fensu-policy",
  "globset",

{sqlbuild-0.55.3 → sqlbuild-0.55.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sqlbuild
-Version: 0.55.3
+Version: 0.55.5
 Requires-Dist: duckdb>=1.3.2
 Requires-Dist: polyglot-sql>=0.5.10
 Requires-Dist: pydantic-settings>=2,<3
@@ -43,9 +43,9 @@ Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
 
 **Valid isn't the same as correct.** Your SQL compiles, runs, and returns rows; none of that means the number is right, and a silently-wrong number a stakeholder already trusted is the bug that actually hurts.
 
-SQLBuild brings software-engineering rigor to SQL pipelines: catch errors before the warehouse runs them, test your logic locally, and opt into change-aware execution when you need it. It works as a standalone framework or points at your existing dbt project with no migration and no edits to your dbt files.
+SQLBuild brings software-engineering rigor to SQL pipelines: catch errors before the warehouse runs them, test your logic locally, and opt into change-aware execution when you need it. It is a standalone, open-source framework for building SQL and Python data pipelines.
 
-All state is persisted as append-only tables in the warehouse alongside your data: no external state database, no manifest files, no paid add-on. It keeps a low, dbt-like floor for SQL models and adds ingestion, Python nodes, and opt-in virtual environments as your project grows.
+All state is persisted as append-only tables in the warehouse alongside your data: no external state database, no manifest files, no paid add-on. Start with straightforward SQL models, then add ingestion, Python nodes, and opt-in virtual environments as your project grows.
 
 ## Key features
 
@@ -55,28 +55,11 @@ All state is persisted as append-only tables in the warehouse alongside your dat
 - **Audits that block bad data.** Audits run before data reaches the target table. Full table builds materialize into a staging table and only promote if audits pass; incremental models validate each batch before DML.
 - **Deploy reversibly (opt-in).** Virtual environments add instant low-copy branching, partial promotion, rollback, checkpoints, and reconciliation. Opt-in, not a tax you pay upfront.
 - **Opt-in change-aware execution.** Models, seeds, UDFs, and Python nodes are fingerprinted, and source freshness is tracked. In virtual environments, pass `--changes-only` or set `changes_only = true` to skip work that is already current; commands otherwise run the full selected scope.
-- **Works with your existing dbt project.** Point SQLBuild at a dbt project and run ordinary dbt selections alongside SQLBuild models. It reads the manifest and drives the `dbt` CLI as a subprocess; it never edits your dbt files. dbt-native `--state` and `--defer` remain available for production-shaped, state-aware selections. See [dbt compatibility](https://docs.sqlbuild.com/concepts/dbt-compatibility/overview).
 - **Warehouse-native state.** All change-tracking state lives in append-only tables (`_sqlbuild_fingerprints`, `_sqlbuild_source_freshness`, `_sqlbuild_node_results`) in your warehouse schemas. No external state machine, no corruption risk.
 - **Cursor-based incremental processing.** Automatic gap detection and resume, with microbatch mode for large ranges. No external checkpoint to maintain.
 - **Ingestion and Python nodes.** Load external data with Python `@loader` functions, and run `@task`, `@asset`, and `@check` nodes as first-class members of the same DAG as your SQL models.
 
-See the [documentation](https://docs.sqlbuild.com) for the full feature set, including providers, lifecycle hooks, Python macros, UDFs, custom materializations, data diffs, zero-copy cloning, and virtual environments.
-
-## Works with your existing dbt project
-
-Point SQLBuild at a dbt project and run a `sqb dbt` command. The first time, it bootstraps a minimal twin project from your `dbt_project.yml` and profile (reusing your dbt connection), then runs your selection through dbt:
-
-```bash
-sqb dbt build --select path:models/marts
-```
-
-SQLBuild preserves dbt-native state and defer arguments when you need dbt's own state-aware selection:
-
-```bash
-sqb dbt build --state path/to/state --defer --select state:modified+
-```
-
-Ordinary `sqb dbt plan`, `run`, and `build` commands do not fingerprint dbt models or inspect production state automatically. Use dbt-native `--state`/`--defer` for production-shaped comparisons. See [dbt compatibility](https://docs.sqlbuild.com/concepts/dbt-compatibility/overview).
+See the [documentation](https://docs.sqlbuild.com) for the full feature set, including providers, lifecycle hooks, Python macros, UDFs, custom materializations, data diffs, zero-copy cloning, and virtual environments. If you are evaluating a migration from dbt, see the [dbt compatibility guide](https://docs.sqlbuild.com/concepts/dbt-compatibility/overview).
 
 ## Quick start
 
@@ -141,61 +124,6 @@ SELECT 1
 
 See the [documentation](https://docs.sqlbuild.com) for incremental models, scenarios, loaders, and more.
 
-## Concurrent microbatches
-
-Native `delete_insert` microbatch models can opt into parallel batch execution. Serial execution is
-the default and remains recommended unless parallel batches provide a meaningful runtime benefit.
-Enable the project capability and set a per-model ceiling:
-
-```toml
-[settings]
-microbatch_concurrency = true
-microbatch_unaccounted_partition_policy = "synthesize"
-```
-
-```sql
-MODEL (
-  materialized incremental,
-  incremental_mode microbatch,
-  incremental_strategy delete_insert,
-  cursor event_time,
-  cursor_type timestamp,
-  cursor_grain hour,
-  batch_size 1h,
-  batch_concurrency 4,
-);
-```
-
-`batch_concurrency` is a model ceiling, not a separate worker pool. Batches share the build's global
-`concurrency` limit and connection pool with every other DAG node. First-run and full-refresh target
-bootstrap remain serialized. Concurrent batches are rejected for adapters that have not explicitly
-declared same-target concurrent delete/insert support.
-
-Every native microbatch, including serial models and projects where the capability is disabled,
-records successful half-open partitions and their model fingerprints. Standard builds use the
-warehouse `_sqlbuild_microbatches` table. Virtual builds store equivalent events in the configured
-DuckDB or Postgres state backend and scope them to the immutable physical model version. Virtual
-builds renew a physical-version lease while mutating shared data; standard-mode orchestrators must
-prevent overlapping invocations for the same model destination.
-
-Janitor may read microbatch history to protect active physical versions, but it never drops or
-prunes `_sqlbuild_microbatches` or virtual `microbatch_events`. Removing virtual event history is an
-explicit `sqb state reset` operator action, not ordinary retention cleanup.
-
-The append-only history distinguishes physical continuity from fingerprint continuity. Known gaps
-are recovered before new normal work. Automatic `replay_on_change` ranges are durable and
-version-specific, while explicit backfills remain one-shot requests. If destination progress is not
-explained by retained history, `microbatch_unaccounted_partition_policy` controls reconciliation:
-
-- `synthesize` accepts inferred physical coverage and records an unknown fingerprint.
-- `recover_empty` counts candidates in bounded chunks, reruns empty intervals, and synthesizes
-  non-empty intervals.
-- `recover_all` reruns every unaccounted interval without a preliminary count.
-
-Synthetic coverage preserves forward progress but weakens version guarantees and remains visible in
-warnings and JSON output. Target DML and event insertion do not require a cross-system transaction;
-idempotent `delete_insert` recovery handles failures between those operations.
-
 ## Kata SQL architecture checks
 
 Kata is SQLBuild's opt-in, error-only SQL model shape checker. It runs offline over the compiled

{sqlbuild-0.55.3 → sqlbuild-0.55.5}/README.md

@@ -8,9 +8,9 @@
 
 **Valid isn't the same as correct.** Your SQL compiles, runs, and returns rows; none of that means the number is right, and a silently-wrong number a stakeholder already trusted is the bug that actually hurts.
 
-SQLBuild brings software-engineering rigor to SQL pipelines: catch errors before the warehouse runs them, test your logic locally, and opt into change-aware execution when you need it. It works as a standalone framework or points at your existing dbt project with no migration and no edits to your dbt files.
+SQLBuild brings software-engineering rigor to SQL pipelines: catch errors before the warehouse runs them, test your logic locally, and opt into change-aware execution when you need it. It is a standalone, open-source framework for building SQL and Python data pipelines.
 
-All state is persisted as append-only tables in the warehouse alongside your data: no external state database, no manifest files, no paid add-on. It keeps a low, dbt-like floor for SQL models and adds ingestion, Python nodes, and opt-in virtual environments as your project grows.
+All state is persisted as append-only tables in the warehouse alongside your data: no external state database, no manifest files, no paid add-on. Start with straightforward SQL models, then add ingestion, Python nodes, and opt-in virtual environments as your project grows.
 
 ## Key features
 
@@ -20,28 +20,11 @@ All state is persisted as append-only tables in the warehouse alongside your dat
 - **Audits that block bad data.** Audits run before data reaches the target table. Full table builds materialize into a staging table and only promote if audits pass; incremental models validate each batch before DML.
 - **Deploy reversibly (opt-in).** Virtual environments add instant low-copy branching, partial promotion, rollback, checkpoints, and reconciliation. Opt-in, not a tax you pay upfront.
 - **Opt-in change-aware execution.** Models, seeds, UDFs, and Python nodes are fingerprinted, and source freshness is tracked. In virtual environments, pass `--changes-only` or set `changes_only = true` to skip work that is already current; commands otherwise run the full selected scope.
-- **Works with your existing dbt project.** Point SQLBuild at a dbt project and run ordinary dbt selections alongside SQLBuild models. It reads the manifest and drives the `dbt` CLI as a subprocess; it never edits your dbt files. dbt-native `--state` and `--defer` remain available for production-shaped, state-aware selections. See [dbt compatibility](https://docs.sqlbuild.com/concepts/dbt-compatibility/overview).
 - **Warehouse-native state.** All change-tracking state lives in append-only tables (`_sqlbuild_fingerprints`, `_sqlbuild_source_freshness`, `_sqlbuild_node_results`) in your warehouse schemas. No external state machine, no corruption risk.
 - **Cursor-based incremental processing.** Automatic gap detection and resume, with microbatch mode for large ranges. No external checkpoint to maintain.
 - **Ingestion and Python nodes.** Load external data with Python `@loader` functions, and run `@task`, `@asset`, and `@check` nodes as first-class members of the same DAG as your SQL models.
 
-See the [documentation](https://docs.sqlbuild.com) for the full feature set, including providers, lifecycle hooks, Python macros, UDFs, custom materializations, data diffs, zero-copy cloning, and virtual environments.
-
-## Works with your existing dbt project
-
-Point SQLBuild at a dbt project and run a `sqb dbt` command. The first time, it bootstraps a minimal twin project from your `dbt_project.yml` and profile (reusing your dbt connection), then runs your selection through dbt:
-
-```bash
-sqb dbt build --select path:models/marts
-```
-
-SQLBuild preserves dbt-native state and defer arguments when you need dbt's own state-aware selection:
-
-```bash
-sqb dbt build --state path/to/state --defer --select state:modified+
-```
-
-Ordinary `sqb dbt plan`, `run`, and `build` commands do not fingerprint dbt models or inspect production state automatically. Use dbt-native `--state`/`--defer` for production-shaped comparisons. See [dbt compatibility](https://docs.sqlbuild.com/concepts/dbt-compatibility/overview).
+See the [documentation](https://docs.sqlbuild.com) for the full feature set, including providers, lifecycle hooks, Python macros, UDFs, custom materializations, data diffs, zero-copy cloning, and virtual environments. If you are evaluating a migration from dbt, see the [dbt compatibility guide](https://docs.sqlbuild.com/concepts/dbt-compatibility/overview).
 
 ## Quick start
 
@@ -106,61 +89,6 @@ SELECT 1
 
 See the [documentation](https://docs.sqlbuild.com) for incremental models, scenarios, loaders, and more.
 
-## Concurrent microbatches
-
-Native `delete_insert` microbatch models can opt into parallel batch execution. Serial execution is
-the default and remains recommended unless parallel batches provide a meaningful runtime benefit.
-Enable the project capability and set a per-model ceiling:
-
-```toml
-[settings]
-microbatch_concurrency = true
-microbatch_unaccounted_partition_policy = "synthesize"
-```
-
-```sql
-MODEL (
-  materialized incremental,
-  incremental_mode microbatch,
-  incremental_strategy delete_insert,
-  cursor event_time,
-  cursor_type timestamp,
-  cursor_grain hour,
-  batch_size 1h,
-  batch_concurrency 4,
-);
-```
-
-`batch_concurrency` is a model ceiling, not a separate worker pool. Batches share the build's global
-`concurrency` limit and connection pool with every other DAG node. First-run and full-refresh target
-bootstrap remain serialized. Concurrent batches are rejected for adapters that have not explicitly
-declared same-target concurrent delete/insert support.
-
-Every native microbatch, including serial models and projects where the capability is disabled,
-records successful half-open partitions and their model fingerprints. Standard builds use the
-warehouse `_sqlbuild_microbatches` table. Virtual builds store equivalent events in the configured
-DuckDB or Postgres state backend and scope them to the immutable physical model version. Virtual
-builds renew a physical-version lease while mutating shared data; standard-mode orchestrators must
-prevent overlapping invocations for the same model destination.
-
-Janitor may read microbatch history to protect active physical versions, but it never drops or
-prunes `_sqlbuild_microbatches` or virtual `microbatch_events`. Removing virtual event history is an
-explicit `sqb state reset` operator action, not ordinary retention cleanup.
-
-The append-only history distinguishes physical continuity from fingerprint continuity. Known gaps
-are recovered before new normal work. Automatic `replay_on_change` ranges are durable and
-version-specific, while explicit backfills remain one-shot requests. If destination progress is not
-explained by retained history, `microbatch_unaccounted_partition_policy` controls reconciliation:
-
-- `synthesize` accepts inferred physical coverage and records an unknown fingerprint.
-- `recover_empty` counts candidates in bounded chunks, reruns empty intervals, and synthesizes
-  non-empty intervals.
-- `recover_all` reruns every unaccounted interval without a preliminary count.
-
-Synthetic coverage preserves forward progress but weakens version guarantees and remains visible in
-warnings and JSON output. Target DML and event insertion do not require a cross-system transaction;
-idempotent `delete_insert` recovery handles failures between those operations.
-
 ## Kata SQL architecture checks
 
 Kata is SQLBuild's opt-in, error-only SQL model shape checker. It runs offline over the compiled

{sqlbuild-0.55.3 → sqlbuild-0.55.5}/crates/sqlbuild-kata-native/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "sqlbuild-kata-native"
-version = "0.55.3"
+version = "0.55.5"
 edition.workspace = true
 license.workspace = true
 publish.workspace = true

{sqlbuild-0.55.3 → sqlbuild-0.55.5}/pyproject.toml

@@ -7,7 +7,7 @@ build-backend = "maturin"
 
 [project]
 name = "sqlbuild"
-version = "0.55.3"
+version = "0.55.5"
 description = "Typed, test-first SQL pipelines with local E2E testing"
 readme = "README.md"
 requires-python = ">=3.12"

{sqlbuild-0.55.3 → sqlbuild-0.55.5}/src/sqlbuild/compiler/compile/_helpers/assembly/project.py

@@ -98,6 +98,7 @@ from sqlbuild.spec.contracts.models import (
 
 _POLYGLOT_ANALYSIS_WORKERS: int = 2
 _POLYGLOT_PARALLEL_ANALYSIS_MIN_MODELS: int = 32
+_POLYGLOT_PARALLEL_REANALYSIS_MIN_MODELS: int = 2
 
 
 @dataclass(frozen=True)
@@ -400,42 +401,14 @@ def _analyze_model_sql_in_parallel(
         if analysis_cache is not None
         else ({}, {})
     )
-    if len(model_inputs) < _POLYGLOT_PARALLEL_ANALYSIS_MIN_MODELS:
-        analyses: tuple[_ModelSqlAnalysis, ...] = tuple(
-            _analyze_model_sql(
-                request=request,
-                cached_analysis=(
-                    cached_analyses.get(request.cache_key)
-                    if request.cache_key is not None
-                    else None
-                ),
-                column_nullability_by_table=column_nullability_by_table,
-                column_types_by_table=column_types_by_table,
-                inference_profile=inference_profile,
-                allow_compact_analysis=allow_compact_analysis,
-            )
-            for request in requests
-        )
-    else:
-        workers: int = min(_POLYGLOT_ANALYSIS_WORKERS, len(model_inputs))
-        with ThreadPoolExecutor(max_workers=workers) as executor:
-            analyses = tuple(
-                executor.map(
-                    lambda request: _analyze_model_sql(
-                        request=request,
-                        cached_analysis=(
-                            cached_analyses.get(request.cache_key)
-                            if request.cache_key is not None
-                            else None
-                        ),
-                        column_nullability_by_table=column_nullability_by_table,
-                        column_types_by_table=column_types_by_table,
-                        inference_profile=inference_profile,
-                        allow_compact_analysis=allow_compact_analysis,
-                    ),
-                    requests,
-                )
-            )
+    analyses: tuple[_ModelSqlAnalysis, ...] = _analyze_model_sql_requests(
+        requests=requests,
+        cached_analyses=cached_analyses,
+        column_nullability_by_table=column_nullability_by_table,
+        column_types_by_table=column_types_by_table,
+        inference_profile=inference_profile,
+        allow_compact_analysis=allow_compact_analysis,
+    )
     current_analyses_by_name: dict[str, PolyglotAnalysisResult] = {
         _model_name(request.model_input): analysis.polyglot_analysis
         for request, analysis in zip(requests, analyses, strict=True)
@@ -462,23 +435,33 @@ def _analyze_model_sql_in_parallel(
         changed_names=changed_signature_names,
     )
     if invalidated_names:
-        analyses = tuple(
-            (
-                _analyze_model_sql(
-                    request=request,
-                    cached_analysis=None,
+        invalidated_requests: tuple[_ModelSqlAnalysisRequest, ...] = tuple(
+            request
+            for request in requests
+            if (
+                _model_name(request.model_input) in invalidated_names
+                and request.cache_key is not None
+                and request.cache_key in cached_analyses
+            )
+        )
+        invalidated_analyses_by_name: dict[str, _ModelSqlAnalysis] = {
+            _model_name(request.model_input): analysis
+            for request, analysis in zip(
+                invalidated_requests,
+                _analyze_model_sql_requests(
+                    requests=invalidated_requests,
+                    cached_analyses={},
                     column_nullability_by_table=column_nullability_by_table,
                     column_types_by_table=column_types_by_table,
                     inference_profile=inference_profile,
                     allow_compact_analysis=allow_compact_analysis,
-                )
-                if (
-                    _model_name(request.model_input) in invalidated_names
-                    and request.cache_key is not None
-                    and request.cache_key in cached_analyses
-                )
-                else analysis
+                    parallel_min_models=_POLYGLOT_PARALLEL_REANALYSIS_MIN_MODELS,
+                ),
+                strict=True,
             )
+        }
+        analyses = tuple(
+            invalidated_analyses_by_name.get(_model_name(request.model_input), analysis)
             for request, analysis in zip(requests, analyses, strict=True)
         )
         analyses_to_record_by_name.update(
@@ -522,6 +505,35 @@ def _analyze_model_sql_in_parallel(
     }
 
 
+def _analyze_model_sql_requests(
+    *,
+    requests: tuple[_ModelSqlAnalysisRequest, ...],
+    cached_analyses: dict[str, PolyglotAnalysisResult],
+    column_nullability_by_table: dict[str, dict[str, InferredNullability]],
+    column_types_by_table: dict[str, dict[str, str]],
+    inference_profile: ExpressionInferenceProfile,
+    allow_compact_analysis: bool,
+    parallel_min_models: int = _POLYGLOT_PARALLEL_ANALYSIS_MIN_MODELS,
+) -> tuple[_ModelSqlAnalysis, ...]:
+    def analyze(request: _ModelSqlAnalysisRequest) -> _ModelSqlAnalysis:
+        return _analyze_model_sql(
+            request=request,
+            cached_analysis=(
+                cached_analyses.get(request.cache_key) if request.cache_key is not None else None
+            ),
+            column_nullability_by_table=column_nullability_by_table,
+            column_types_by_table=column_types_by_table,
+            inference_profile=inference_profile,
+            allow_compact_analysis=allow_compact_analysis,
+        )
+
+    if len(requests) < parallel_min_models:
+        return tuple(analyze(request) for request in requests)
+    workers: int = min(_POLYGLOT_ANALYSIS_WORKERS, len(requests))
+    with ThreadPoolExecutor(max_workers=workers) as executor:
+        return tuple(executor.map(analyze, requests))
+
+
 def _downstream_model_names(
     *,
     model_inputs: tuple[CompileModelInput, ...],