openstatspec 0.3.0__tar.gz → 0.5.0__tar.gz

{openstatspec-0.3.0 → openstatspec-0.5.0}/.github/workflows/ci.yml

@@ -20,7 +20,7 @@ jobs:
         uses: actions/checkout@v7
         with: &specification-checkout
           repository: OpenStatSpec/specification
-          ref: 79339ec3d8f8aa81789b7e85f6b8afa6f1374e50
+          ref: f2fdf687d8cb32b944ca55a3e9e7215ffc603019
           path: openstatspec-specification
       - name: Checkout required SPSS engine
         uses: actions/checkout@v7
@@ -143,7 +143,7 @@ jobs:
     steps: *sql-test-steps
 
   dolt-integration:
-    name: Dolt ${{ matrix.dolt }} integration
+    name: Dolt ${{ matrix.dolt }} read-only / fail-closed integration
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
@@ -217,7 +217,7 @@ jobs:
                   connection.close()
                   break
           PY
-      - run: python -m pytest -m services
+      - run: python -m pytest -m "services and not candidate_evidence"
       - name: Show Dolt logs on failure and stop server
         if: always()
         run: |

{openstatspec-0.3.0 → openstatspec-0.5.0}/.github/workflows/release.yml

@@ -47,7 +47,7 @@ jobs:
         uses: actions/checkout@v7
         with:
           repository: OpenStatSpec/specification
-          ref: 79339ec3d8f8aa81789b7e85f6b8afa6f1374e50
+          ref: f2fdf687d8cb32b944ca55a3e9e7215ffc603019
           path: openstatspec-specification
       - name: Checkout required SPSS engine
         uses: actions/checkout@v7

openstatspec-0.5.0/CHANGELOG.md

@@ -0,0 +1,191 @@
+# Changelog
+
+All notable changes to this reference implementation are documented here.
+
+## Unreleased
+
+Planned adapter release: `0.5.0`, after lifecycle integration and final specification conformance.
+### Fixed
+
+- Removed the temporary `*_catalog` compatibility schema and made SAV/ZSAV
+  import, validation, export, fidelity reporting, and in-place metadata edits
+  use the normative UUID-keyed OpenStatSpec catalog exclusively.
+- Existing databases that contain former compatibility relations must be
+  remediated manually before further operations.
+- Imports now reject dataset names that collide with normative UUID identifiers,
+  cleanup drops only a physical table actually created by the failing import,
+  and non-preserved SPSS compatible variable names are reported as an explicit
+  loss requiring export consent.
+
+
+### Added
+
+- Added bounded typed expressions with variable references, numeric literals,
+  parentheses, `=`, `<`, `<=`, `>`, `>=`, `AND`, and `OR`. String comparison
+  and v0.2 string assignment fail closed pending exact portable semantics.
+- Added sequential SPSS-like `COMPUTE` and `IF` assignment operations plus
+  `FORMATS`, `VARIABLE LEVEL`, and `EXECUTE`.
+- Added atomic numeric target creation on SQLite and PostgreSQL. MySQL,
+  MariaDB, and Dolt fail closed on `target_mode=create`; their targets must be
+  provisioned physically and in the catalog by a separate versioned stage
+  before this executor applies assignment and metadata operations.
+- Added synthetic exact-program, catalog, failure-boundary, and pre-existing
+  target regression coverage.
+
+### Changed
+
+- Bumped the canonical transformation-plan and SPSS frontend contracts to
+  `v0.2`; canonical JSON and hashes include every sequential operation.
+- In-place apply records variable label, value labels, `F` print/write
+  format, and measurement level only in the normative catalog.
+- Dolt still requires an exact branch, exact HEAD, and clean working set;
+  successful apply leaves an inspectable diff and never calls `DOLT_COMMIT`.
+- Dolt declaration validation is implemented by the Python adapter. The
+  specification remains a language-neutral source of JSON contracts and
+  fixtures and is not installed as a Python distribution.
+
+### Specification basis
+
+- Release validation is pinned to the untagged OpenStatSpec specification
+  release candidate at immutable commit
+  `f2fdf687d8cb32b944ca55a3e9e7215ffc603019`. Its
+  `specification_release` remains null until that commit receives a stable tag.
+
+## 0.4.0 — 2026-07-31
+
+### Added
+
+- Added a public canonical transformation-plan in-place API accepting either
+  typed plan objects or strictly validated mappings.
+- Added apply-plan and install-in-place-schema CLI workflows alongside the
+  compatible apply-spss command.
+- Added explicit audit provenance distinguishing canonical plan documents from
+  SPSS syntax sources.
+- Added a transformation manual covering the canonical core, frontend boundary,
+  execution invariants, CLI/API workflows, and extension rules.
+
+### Changed
+
+- Moved the SPSS parser, binder, compiler, and convenience execution adapter to
+  the dedicated openstatspec.frontends.spss package.
+- Kept openstatspec.transform language-neutral with canonical plan, schema, and
+  live-schema validation modules while preserving previous SPSS import paths.
+- Reserved empty Stata and SAS frontend directories without exposing parsers,
+  capabilities, CLI choices, or support claims.
+- Strengthened pre-mutation target-type checks and target-scoped physical table
+  identity guards. The executor continues to modify the same dataset and table
+  without creating OpenStatSpec rollback, copy, snapshot, or history layers.
+
+### Specification basis
+
+- Conformance and release validation continue to use OpenStatSpec specification
+  release v0.2.0 at exact commit
+  79339ec3d8f8aa81789b7e85f6b8afa6f1374e50.
+
+## 0.3.0 — 2026-07-31
+
+### Added
+
+- Added a bounded SPSS-like `RECODE`, `VARIABLE LABELS`, and `VALUE LABELS`
+  parser with stable diagnostics and conformance fixtures.
+- Added canonical Transformation Plan serialization, RFC 8785 hashes, schema
+  binding, deterministic typed operations, and an agent-facing CLI/API.
+- Added product-neutral in-place execution with same dataset/table identity,
+  direct data and metadata mutation, and compact operation audit. Dolt adds
+  expected branch/HEAD and clean-working-set checks.
+
+### Changed
+
+- Version history, diff, rollback, restoration, and commit remain database
+  responsibilities; the transformer creates no derived/copy/snapshot/recovery
+  layer and performs no `DOLT_COMMIT`.
+- SQL server support now distinguishes conservative family claims from exact
+  CI evidence: PostgreSQL 17.x/18.x at 17.10/18.4, MySQL 8.4.x/9.7.x at
+  8.4.11/9.7.2, and MariaDB 11.4.x/11.8.x/12.3.x at
+  11.4.12/11.8.8/12.3.2. Each service job verifies the live normalized version.
+- Dolt remains an independent core profile, now claiming canonical stable
+  versions `>=2.2.2,<2.3.0` and running its complete service suite at exact
+  versions 2.2.2 and 2.2.3 from immutable container-image pins.
+  SQLite retains its core `>=3.24.0,<4.0.0` and optional workflow
+  `>=3.35.0,<4.0.0` tiers. MSSQL remains unsupported and roadmap-only.
+
+### Specification basis
+
+- CI, release validation, and machine-readable capabilities use OpenStatSpec
+  specification release `v0.2.0` at exact commit
+  `79339ec3d8f8aa81789b7e85f6b8afa6f1374e50`.
+
+## 0.2.0 — 2026-07-30
+
+This release extends the reference adapter while keeping imported source
+datasets immutable and all unsupported paths fail-closed.
+
+### Added
+
+- An optional SQLite-only Transformation Workflow for immutable, versioned,
+  parameterized SELECT definitions; materialized derived datasets; lineage and
+  weights; public catalog APIs; and CLI operations.
+- An independent Dolt core SQL profile for exact product version 2.2.2, with
+  service-backed coverage, conservative adapter limits, complete compensating
+  cleanup, and pre-DDL rejection of unsupported values. Transformation Workflow
+  support on Dolt is not claimed.
+
+### Changed
+
+- SAV and ZSAV validation now rewrites legacy compatible names consistently
+  across type-2 records and subtype-13/subtype-14 long-name metadata.
+- Transformation publication, interrupted-run reconciliation, retirement, and
+  physical-removal recovery now preserve auditable state and fail closed on
+  catalog or relation drift.
+- Release automation builds distributions before test-only source checkouts and
+  supports idempotent rebuilds of an existing release tag.
+
+### Specification basis
+
+- CI and release validation are pinned to OpenStatSpec specification release
+  `v1.0.0-rc.1` at exact commit
+  `fef0dc6f4b17ff7141dad3f49d0524c63efbfed5`.
+
+## 0.1.0 — 2026-07-29
+
+First public reference implementation of the OpenStatSpec strict wide-table
+SPSS profile.
+
+### Included
+
+- Import of unencrypted SAV and ZSAV sources into one dedicated wide SQL table
+  and its metadata catalog.
+- Export of supported dataset semantics to SAV and ZSAV.
+- SQLite, PostgreSQL, MySQL, and MariaDB profiles, including service-backed
+  PostgreSQL 17/18, MySQL 8.4/9.7, and MariaDB 11.4/11.8/12.3 CI coverage.
+- Read-only positive Dolt identity and working-set inspection. Operational
+  Dolt writes use adapter-owned declaration validation and remain fail-closed
+  because the adapter ships no concrete declaration claim. The specification
+  remains language-neutral and is consumed only as JSON contracts and fixtures.
+- Added explicit packaged/directory conformance-source injection and exact
+  single-match binding across active Dolt product version, adapter
+  implementation and version, and specification commit before mutation.
+- Explicit catalog initialization/migration through `initialize_catalog` and
+  `openstatspec init`; data operations never auto-create catalog relations.
+- Read-only Dolt working-set evidence through `dolt_state_snapshot` and
+  `openstatspec dolt-state`; core performs no Dolt version-control mutations.
+- Preflight checks for target profile limits, atomic imports, validation, a
+  command-line interface, and machine-readable capability and loss reports.
+
+### Important boundaries
+
+- This release promises semantic round-trip equivalence only for supported
+  features; it never promises byte-identical SAV/ZSAV output.
+- The adapter requires explicit `allow_loss` consent before exporting known
+  unsupported or lossy dictionary semantics.
+- Multiple-response sets, variable alignment, variable sets, and custom
+  attributes have explicit capability diagnostics; see the SAV profile.
+- Encrypted SPSS files are not supported.
+- Dolt adapter-envelope values are not claims about Dolt server limits. Failure
+  recovery is compensating and reports deterministic residual inventory; a
+  verified catalog receives a best-effort failed-operation audit when cleanup
+  itself fails.
+- Export publication uses a durable, audit-recorded prior-file backup through
+  SQL finalization. Post-success backup retention is reported separately and
+  does not rewrite a successful export as failed; pre-success failures restore
+  the prior destination and close the running operation as failed.

{openstatspec-0.3.0 → openstatspec-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: openstatspec
-Version: 0.3.0
+Version: 0.5.0
 Summary: Reference adapter for the OpenStatSpec relational contract
 Author: OpenStatSpec
 License: Apache-2.0
@@ -30,10 +30,10 @@ For each supported import, one source dataset becomes one dedicated wide SQL
 table. Cases are rows and source variables are physical SQL columns. The
 singular UUID-keyed tables from the specification (`dataset`, `variable`,
 `operation`, `fidelity_event`, and related metadata tables) are the public
-catalog contract. Historical `*_catalog` tables are an internal compatibility
-layer for the current exporter and are not the standard database interface.
-The adapter does not reshape data, create EAV
-or long-form tables, or harmonize studies or waves.
+catalog contract and the only catalog representation created or read by the
+adapter. Databases created with the former `*_catalog` compatibility schema
+must be remediated to the normative schema before use. The adapter does not
+reshape data, create EAV or long-form tables, or harmonize studies or waves.
 
 Unsupported source features, SQL targets, or export paths fail explicitly.
 There is no silent truncation, type conversion, metadata loss, or partial
@@ -45,6 +45,10 @@ import.
   capabilities, and loss reports.
 - `openstatspec.sql`: database connection and wide-table/catalog operations.
 - `openstatspec.spss`: SAV/ZSAV adapter boundary.
+- `openstatspec.transform`: canonical plans, frontend-neutral schema concepts,
+  and plan validation.
+- `openstatspec.frontends.spss`: the SPSS-like syntax frontend and convenience
+  execution adapter.
 
 ## Intended workflow
 
@@ -82,43 +86,24 @@ implemented capability boundary.
 
 ## SPSS-like transformation frontend
 
-The transformation frontend accepts `RECODE`, `VARIABLE LABELS`, and
-`VALUE LABELS`, lowers them to a canonical OpenStatSpec Transformation Plan,
-and mutates the same logical dataset and physical wide table. SQLite,
-PostgreSQL, MySQL, MariaDB, and Dolt connections are allowed. It creates no
-derived dataset, copied table, snapshot, or OpenStatSpec rollback/version layer.
-
-Install the compact operation-audit relation separately, then apply syntax:
-
-```python
-from openstatspec import (
-    apply_spss_in_place,
-    install_in_place_transformation_schema,
-)
-
-database_url = "postgresql+psycopg://user:password@host/database"
-install_in_place_transformation_schema(database_url=database_url)
-result = apply_spss_in_place(
-    database_url=database_url,
-    dataset_id="...",
-    actor="agent@example.org",
-    source_text="""
-      RECODE age (18 THRU 34 = 1) (35 THRU 64 = 2) INTO age_group.
-      VARIABLE LABELS age_group 'Age group'.
-      VALUE LABELS age_group 1 '18-34' 2 '35-64'.
-    """,
-)
-```
-
-Existing-target recodes and metadata mutations use direct DML. SQLite and
-PostgreSQL may also add a target column in the same native transaction. MySQL,
-MariaDB, and Dolt reject such schema-changing plans before mutation because
-their implicit-commit DDL could otherwise leave a partial apply. On Dolt, the
-caller additionally supplies expected branch and HEAD identities, and the
-working set must be clean. The transformer never calls `DOLT_COMMIT`.
-
-See [in-place transformations](docs/in-place-transformation.md) for the full
-execution boundary and CLI form.
+The bounded SPSS-like frontend lowers `RECODE`, sequential `COMPUTE` and `IF`,
+`VARIABLE LABELS`, `VALUE LABELS`, numeric `FORMATS`, `VARIABLE LEVEL`, and
+`EXECUTE` into a language-neutral typed canonical plan. Conditions support
+parentheses, variable and numeric literal operands, comparisons, `AND`, and
+`OR`. String comparison and v0.2 string assignment currently fail closed.
+The in-place path applies the plan to the same logical dataset, physical wide
+table, and normative/compatibility metadata without a derived dataset, copied
+table, snapshot, or hidden history layer. Numeric targets may be created
+atomically on SQLite and PostgreSQL. MySQL, MariaDB, and Dolt fail closed on
+`target_mode=create` and require a separately provisioned physical and catalog
+target before assignment. Dolt requires the caller's exact clean branch/HEAD,
+leaves success as an inspectable working-set diff, and never calls
+`DOLT_COMMIT`.
+
+See the [dataset transformations manual](docs/transformations.md) for schema
+installation, Python and CLI surfaces, database invariants, audit provenance,
+package layout, and extension guidance. Stata and SAS are unimplemented
+placeholders.
 
 ## Current support status
 

{openstatspec-0.3.0 → openstatspec-0.5.0}/README.md

@@ -11,10 +11,10 @@ For each supported import, one source dataset becomes one dedicated wide SQL
 table. Cases are rows and source variables are physical SQL columns. The
 singular UUID-keyed tables from the specification (`dataset`, `variable`,
 `operation`, `fidelity_event`, and related metadata tables) are the public
-catalog contract. Historical `*_catalog` tables are an internal compatibility
-layer for the current exporter and are not the standard database interface.
-The adapter does not reshape data, create EAV
-or long-form tables, or harmonize studies or waves.
+catalog contract and the only catalog representation created or read by the
+adapter. Databases created with the former `*_catalog` compatibility schema
+must be remediated to the normative schema before use. The adapter does not
+reshape data, create EAV or long-form tables, or harmonize studies or waves.
 
 Unsupported source features, SQL targets, or export paths fail explicitly.
 There is no silent truncation, type conversion, metadata loss, or partial
@@ -26,6 +26,10 @@ import.
   capabilities, and loss reports.
 - `openstatspec.sql`: database connection and wide-table/catalog operations.
 - `openstatspec.spss`: SAV/ZSAV adapter boundary.
+- `openstatspec.transform`: canonical plans, frontend-neutral schema concepts,
+  and plan validation.
+- `openstatspec.frontends.spss`: the SPSS-like syntax frontend and convenience
+  execution adapter.
 
 ## Intended workflow
 
@@ -63,43 +67,24 @@ implemented capability boundary.
 
 ## SPSS-like transformation frontend
 
-The transformation frontend accepts `RECODE`, `VARIABLE LABELS`, and
-`VALUE LABELS`, lowers them to a canonical OpenStatSpec Transformation Plan,
-and mutates the same logical dataset and physical wide table. SQLite,
-PostgreSQL, MySQL, MariaDB, and Dolt connections are allowed. It creates no
-derived dataset, copied table, snapshot, or OpenStatSpec rollback/version layer.
-
-Install the compact operation-audit relation separately, then apply syntax:
-
-```python
-from openstatspec import (
-    apply_spss_in_place,
-    install_in_place_transformation_schema,
-)
-
-database_url = "postgresql+psycopg://user:password@host/database"
-install_in_place_transformation_schema(database_url=database_url)
-result = apply_spss_in_place(
-    database_url=database_url,
-    dataset_id="...",
-    actor="agent@example.org",
-    source_text="""
-      RECODE age (18 THRU 34 = 1) (35 THRU 64 = 2) INTO age_group.
-      VARIABLE LABELS age_group 'Age group'.
-      VALUE LABELS age_group 1 '18-34' 2 '35-64'.
-    """,
-)
-```
-
-Existing-target recodes and metadata mutations use direct DML. SQLite and
-PostgreSQL may also add a target column in the same native transaction. MySQL,
-MariaDB, and Dolt reject such schema-changing plans before mutation because
-their implicit-commit DDL could otherwise leave a partial apply. On Dolt, the
-caller additionally supplies expected branch and HEAD identities, and the
-working set must be clean. The transformer never calls `DOLT_COMMIT`.
-
-See [in-place transformations](docs/in-place-transformation.md) for the full
-execution boundary and CLI form.
+The bounded SPSS-like frontend lowers `RECODE`, sequential `COMPUTE` and `IF`,
+`VARIABLE LABELS`, `VALUE LABELS`, numeric `FORMATS`, `VARIABLE LEVEL`, and
+`EXECUTE` into a language-neutral typed canonical plan. Conditions support
+parentheses, variable and numeric literal operands, comparisons, `AND`, and
+`OR`. String comparison and v0.2 string assignment currently fail closed.
+The in-place path applies the plan to the same logical dataset, physical wide
+table, and normative/compatibility metadata without a derived dataset, copied
+table, snapshot, or hidden history layer. Numeric targets may be created
+atomically on SQLite and PostgreSQL. MySQL, MariaDB, and Dolt fail closed on
+`target_mode=create` and require a separately provisioned physical and catalog
+target before assignment. Dolt requires the caller's exact clean branch/HEAD,
+leaves success as an inspectable working-set diff, and never calls
+`DOLT_COMMIT`.
+
+See the [dataset transformations manual](docs/transformations.md) for schema
+installation, Python and CLI surfaces, database invariants, audit provenance,
+package layout, and extension guidance. Stata and SAS are unimplemented
+placeholders.
 
 ## Current support status
 

openstatspec-0.5.0/docs/in-place-transformation.md

@@ -0,0 +1,9 @@
+# In-place transformations
+
+This page has moved to the comprehensive
+[dataset transformations manual](transformations.md).
+
+The manual covers the canonical plan, SPSS-like frontend, executor, audit
+schema, Python and CLI surfaces, database and Dolt invariants, provenance,
+package layout, and future frontend boundary. Stata and SAS remain
+unimplemented placeholders.

{openstatspec-0.3.0 → openstatspec-0.5.0}/docs/release-readiness.md

@@ -1,4 +1,4 @@
-# 0.3.0 release readiness
+# 0.5.0 release readiness
 
 This page records the expected release contract, not a publication event.
 Creating a version tag remains a separate maintainer action.
@@ -56,6 +56,50 @@ otherwise the strict export policy reports and blocks the loss. A caller
 that supplies consent for an available loss code receives the
 machine-readable loss report with the export result.
 
+## Transformation release gates
+
+The canonical transformation core and SPSS syntax frontend are separate public
+surfaces. A release must run the specification-owned canonical-plan and SPSS
+frontend conformance fixtures, then exercise both the generic plan apply API
+and the SPSS compatibility apply path.
+
+The gate must prove that:
+
+- a TransformationPlan object and its strict JSON mapping produce the same
+  plan hash and in-place result;
+- the exact bounded `COMPUTE`/`IF` program compiles to all seven ordered
+  operations without dropping `FORMATS`, `VARIABLE LEVEL`, or `EXECUTE`;
+- boolean data results match the equivalent expression and the target's label,
+  0/1 value labels, `F1.0` print/write format, and nominal level exist in the
+  normative catalog;
+- injected schema, data, catalog, and audit failures leave no partial apply;
+- compensation tracks only newly created targets and never drops or rewrites a
+  pre-existing target;
+- MySQL, MariaDB, and Dolt reject create-target plans before mutation; their
+  service evidence covers assignment to a separately provisioned physical and
+  catalog target without schema DDL;
+- top-level SPSS compiler imports and legacy openstatspec.transform re-exports
+  still load from an installed wheel;
+- install-in-place-schema, apply-plan, and apply-spss execute their documented
+  CLI workflows;
+- invalid or unsupported plans fail before the first data or metadata mutation;
+- successful applies retain the same dataset ID, physical schema/table
+  identity, dataset count, and persistent physical data-table count;
+- audit rows distinguish canonical plans from SPSS syntax, preserve correct
+  source/plan hashes and frontend contract, and contain no copied data;
+- no OpenStatSpec rollback, snapshot, staging, copy, derived-dataset, or
+  parallel history artifacts are created; and
+- Dolt checks expected branch, HEAD, and a clean working set; success changes
+  neither HEAD nor branch and never commits or resets; state is rechecked after
+  the dataset lock and success must leave an inspectable working-set diff;
+  other supported SQL connections remain allowed; and
+- string comparisons and v0.2 string assignments fail closed until exact
+
+The built wheel must contain the generic openstatspec.transform modules and the
+implemented openstatspec.frontends.spss package. Stata and SAS remain empty
+source-tree placeholders and must expose no compiler, apply API, CLI choice,
+capability claim, or implied support.
+
 ## Maintainer checks before tagging
 
 1. Publish the pinned `openstatspec-pyspssio==0.5.1.post2` engine distribution
@@ -68,9 +112,12 @@ machine-readable loss report with the export result.
 4. Build with `python -m build` and install the generated wheel in a clean
    environment.
 5. Confirm `openstatspec capabilities` reflects the intended support boundary.
-6. Confirm the release tag matches the package version and that CI, release
-   fixtures, and capabilities use OpenStatSpec specification release `v0.2.0`
-   at exact commit `79339ec3d8f8aa81789b7e85f6b8afa6f1374e50`.
+6. Confirm CI, release fixtures, and capabilities use the untagged OpenStatSpec
+   specification release candidate at exact commit
+   `f2fdf687d8cb32b944ca55a3e9e7215ffc603019`, publish
+   `specification_status=release_candidate`, and leave
+   `specification_release` null. If that exact commit receives a stable tag
+   before this package is tagged, update the identity and re-run every gate.
 7. Review this document, the README, and CHANGELOG for accurate scope.
 
 The tag-triggered release workflow repeats the non-service test suite, builds