policystrata 0.1.0__tar.gz

policystrata-0.1.0/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+## [Unreleased]
+
+- No changes yet.
+
+## [0.1.0] - 2026-06-25
+
+- Initial public research artifact.
+- Deterministic `support_saas` and `finance_saas` benchmark domains.
+- Seeded and generated mutation suites for cross-layer policy drift.
+- Traces, summaries, baselines, evidence tables, minimized witnesses, scanner fixtures, and Docker
+  PostgreSQL evidence support.
+- Public release files, CI, GitHub Action wrapper, and source distribution manifest coverage.
+- Eval-card governance, scanner regression-case labels, database state assertions, and
+  Inspect/BenchFlow export adapters.
+- Suite provenance, evidence-level, and detector-freeze metadata for future blinded or externally
+  authored suites.
+- `defense_in_depth_stack` baseline and scanner `evidence_exercised` reporting for clean scans.
+- Artifact usability report command for reviewer-facing run, witness, latency, and fixture metrics.
+- arXiv-ready paper source and same-day submission notes under `paper/arxiv`.

policystrata-0.1.0/CITATION.cff

@@ -0,0 +1,10 @@
+cff-version: 1.2.0
+message: "If you use PolicyStrata in research, please cite it."
+title: "PolicyStrata"
+type: software
+version: "0.1.0"
+date-released: "2026-06-25"
+license: MIT
+repository-code: "https://github.com/raintree-technology/policystrata"
+authors:
+  - name: "Raintree Technology"

policystrata-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Contributing
+
+PolicyStrata is a deterministic research artifact. Keep changes reproducible, scoped, and explicit
+about what the evidence does and does not prove.
+
+```bash
+uv sync --extra dev
+uv run pytest
+uv run ruff check .
+uv run mypy src
+```
+
+Optional PostgreSQL tests:
+
+```bash
+docker compose up -d postgres
+POLICYSTRATA_RUN_DB_TESTS=1 uv run pytest tests/test_postgres_integration.py
+```
+
+- Keep the policy oracle independent from the SQL compiler path.
+- Treat constrained generation as a reliability layer, not an authorization boundary.
+- Preserve JSON/YAML trace stability. Add fields compatibly.
+- Keep the built-in `support_saas` domain deterministic and seed-driven.
+- Use adapters for external frameworks. Do not couple core execution to them.
+- Do not require an LLM API key, hosted service, or host `psql` for deterministic tests.
+
+When evidence behavior changes, regenerate the tables:
+
+```bash
+scripts/reproduce-evidence.sh
+```

policystrata-0.1.0/EVAL_CARD.md

@@ -0,0 +1,98 @@
+# PolicyStrata Eval Card
+
+PolicyStrata is a deterministic policy-regression environment for governed LLM data-agent stacks.
+It is not an authorization boundary, a generic LLM leaderboard, or a claim of production incident
+recall.
+
+## Scope
+
+PolicyStrata evaluates whether authorization, semantic, database-containment, and release
+obligations survive translation across policy-bearing surfaces:
+
+- model-visible manifests;
+- grammars and semantic IR;
+- validators;
+- SQL compilers;
+- database controls;
+- output-release checks.
+
+The core artifact uses deterministic semantic plans and traces. It does not require an LLM API key.
+
+## Current Suites
+
+| Suite | Provenance | Boundary |
+| --- | --- | --- |
+| `support_saas` seeded | public hand-authored fixture | regression coverage, not recall |
+| `support_saas` generated | deterministic operator-generated cases | generated from the same public taxonomy |
+| `support_saas` generated_alt_seed | secondary deterministic generated suite | reproducibility evidence, not blinded held-out evidence |
+| `finance_saas` seeded | second synthetic built-in domain | reduces single-domain risk, still synthetic |
+
+The current canonical evidence reports 620/620 killed non-equivalent mutants across these suites.
+That means coverage over the implemented deterministic operators and fixtures. It does not mean
+PolicyStrata detects all real-world policy drift.
+
+Run metadata records each suite's evidence level, provenance, and detector-freeze status. Future
+externally authored, detector-frozen, or incident-reconstruction suites should be reported
+separately from this 620-mutant public deterministic score.
+
+## Scanner Evidence Levels
+
+Scanner findings carry evidence levels:
+
+- `deterministic_fixture`: built-in or explicitly configured fixtures.
+- `property_generated`: generated SQL/IR mutants over configured inputs.
+- `imported_trace`: imported production or representative traces.
+- `real_db`: PostgreSQL fixture or RLS observations through Python adapters.
+- `blinded_suite`: externally authored or detector-frozen suites when provided.
+
+These levels describe what was exercised. They are not confidence intervals for unknown production
+faults.
+
+## Regression Gate Semantics
+
+PolicyStrata scanner traces and state assertions may be labeled:
+
+- `fail_to_pass`: known drift evidence should now be caught or contained.
+- `pass_to_pass`: legitimate behavior should stay clean.
+- `contain_to_contain`: a risky request should remain contained by a later layer.
+- `deny_to_deny`: a forbidden request should remain denied.
+- `allow_to_allow`: an authorized request should remain usable.
+- `unclassified`: legacy or unlabeled imported evidence.
+
+Release gates should not rely only on failing examples. A useful gate includes both
+`fail_to_pass` evidence and `pass_to_pass`/`allow_to_allow` maintenance evidence so fixes do not
+create over-restriction regressions.
+
+## Real Database Boundary
+
+Deterministic benchmark runs simulate database effects. The scanner can optionally prepare a
+Docker/PostgreSQL fixture, execute read-only imported SQL beside canonical compiler SQL, run RLS
+checks, and evaluate state assertions over result rows. Host `psql` is not required.
+
+The current real-DB fixture is a smoke test for containment and SQL behavior. It is not an
+end-to-end dbt/warehouse execution harness and should not be represented as one.
+
+## Benchmark Integrity
+
+Current limitations:
+
+- no blinded externally authored held-out suite is shipped;
+- no verified real incident reconstructions are shipped;
+- synthetic domains may miss organization-specific policy nuance;
+- generated mutants share the public operator taxonomy;
+- baseline comparators are simple observability controls, not independent production test suites;
+- bounded witness reduction is not full delta debugging or source-code root-cause localization.
+
+External validation should follow `docs/external-suite-protocol.md` and, for real incidents,
+`docs/incident-reconstruction-template.md`.
+
+## Model-In-The-Loop Use
+
+Model-mediated experiments are a reachability layer on top of deterministic conformance. They
+should report reliability separately from capability:
+
+- `reachability@k`: at least one of `k` attempts reached a witness.
+- `policy_pass^k`: all `k` independent attempts respected the policy.
+- `release_safe^k`: all `k` independent attempts avoided unsafe release.
+
+Do not mix these with deterministic mutant kill rate.

policystrata-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Raintree Technology
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

policystrata-0.1.0/MANIFEST.in

@@ -0,0 +1,15 @@
+include README.md
+include action.yml
+include EVAL_CARD.md
+include CHANGELOG.md
+include CONTRIBUTING.md
+include SECURITY.md
+include CITATION.cff
+include LICENSE
+include docker-compose.yml
+include uv.lock
+recursive-include docs *.md
+recursive-include examples *.jsonl *.yaml *.yml
+recursive-include scripts *.py *.sh
+recursive-include tests *.py
+recursive-include src/policystrata/domains *.sql *.yaml

policystrata-0.1.0/PKG-INFO

@@ -0,0 +1,324 @@
+Metadata-Version: 2.4
+Name: policystrata
+Version: 0.1.0
+Summary: Cross-layer policy regression testing for LLM data-agent stacks
+Author-email: Raintree Technology <support@raintree.technology>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/raintree-technology/policystrata
+Project-URL: Repository, https://github.com/raintree-technology/policystrata
+Project-URL: Documentation, https://github.com/raintree-technology/policystrata#readme
+Project-URL: Paper, https://raintree.technology/papers
+Project-URL: Changelog, https://github.com/raintree-technology/policystrata/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/raintree-technology/policystrata/issues
+Keywords: llm,text-to-sql,data-agents,policy-testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: psycopg[binary]>=3.2
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: hypothesis>=6.0; extra == "dev"
+Requires-Dist: ruff>=0.8.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: types-pyyaml>=6.0.0; extra == "dev"
+Dynamic: license-file
+
+# PolicyStrata
+
+PolicyStrata is a deterministic regression-testing framework for cross-layer policy drift in LLM
+data-agent stacks.
+
+It generates principals, requests, semantic plans, database states, lowered queries, and release
+decisions; compares each layer against a canonical reference policy; and minimizes failures into
+small reproducible witnesses.
+
+Use it when you are building text-to-SQL agents, BI copilots, internal analytics agents, warehouse
+chat systems, or governed enterprise LLM tools and need to know whether prompts, manifests,
+semantic plans, validators, SQL compilers, database controls, and output filters still agree about
+policy.
+
+PolicyStrata is not an authorization boundary, and it is not another generic text-to-SQL benchmark.
+It is a reproducible research artifact and regression gate for finding reachable disagreements
+between layers.
+
+## Quick Start
+
+From PyPI:
+
+```bash
+uvx policystrata demo
+pipx run policystrata demo
+```
+
+From a source checkout:
+
+```bash
+uv sync --extra dev
+uv run policystrata demo
+```
+
+The demo runs the built-in `support_saas` fixture, writes traces and minimized witnesses to
+`runs/demo`, and prints the drift classes it found. Use `--out` to choose another output directory:
+
+```bash
+uv run policystrata demo --out runs/demo
+```
+
+No LLM API key is required for deterministic tests, benchmark runs, or the built-in demo.
+
+## Install
+
+PolicyStrata is a CLI-first Python package. The public package provides the `policystrata` console
+script and importable Python modules.
+
+```bash
+python -m pip install policystrata
+policystrata demo
+```
+
+For one-off CLI use without managing an environment:
+
+```bash
+uvx policystrata demo
+pipx run policystrata demo
+```
+
+Repository examples under `examples/`, Docker Compose fixtures, and evidence scripts are available
+from a GitHub checkout or source distribution. The wheel installs the runtime package and built-in
+domain fixtures used by `policystrata demo`, `run`, `init-domain`, and `scan`.
+
+## Use As A Template
+
+Click **Use this template** on GitHub, then start with the deterministic fixtures:
+
+```bash
+uv sync --extra dev
+uv run policystrata run --domain support_saas --suite seeded --out runs/example
+uv run policystrata summarize runs/example
+```
+
+To copy a built-in domain fixture into your tree:
+
+```bash
+uv run policystrata init-domain support_saas --out examples/my-policystrata-domain
+```
+
+Keep custom integrations as adapters. The policy oracle should stay independent from SQL compiler
+behavior, external eval frameworks, and model-provider behavior.
+
+## What It Tests
+
+The core failure class is cross-layer policy drift:
+
+```text
+Canonical policy:
+  Analysts may view tenant-scoped aggregate ticket counts, but not customer-level PII.
+
+Model-visible manifest or grammar:
+  Accidentally exposes customer_email as a dimension.
+
+SQL compiler:
+  Accidentally drops the tenant predicate while lowering an authorized aggregate.
+
+Output layer:
+  Releases the result because the final answer looks like a summary.
+
+PolicyStrata result:
+  A minimized witness localizes the violated layer and failed obligation.
+```
+
+PolicyStrata does not assume every layer should behave identically. Each surface has a declared
+responsibility:
+
+- `manifest`: expose model-visible capabilities without stale or forbidden options.
+- `grammar`: parse the declared intent space and preserve untrusted intent for validation.
+- `validator`: authorize semantic queries and bind principal, tenant, time, and budget obligations.
+- `compiler`: lower authorized semantic IR into SQL while preserving metric, tenant, time, and row
+  obligations.
+- `database`: contain row access with RLS and other database-side controls.
+- `release`: withhold contained or unauthorized results.
+
+See [docs/failure-taxonomy.md](docs/failure-taxonomy.md) for how witness classes map to concrete
+policy-drift failures.
+
+## Run Benchmarks
+
+PolicyStrata ships with deterministic `support_saas` and `finance_saas` benchmarks, generated
+mutation suites, minimized witnesses, JSONL traces, baseline comparisons, and evidence tables.
+
+```bash
+uv run policystrata run --domain support_saas --suite seeded --out runs/example
+uv run policystrata run \
+  --domain support_saas \
+  --suite generated \
+  --count 500 \
+  --seed 1729 \
+  --out runs/generated
+uv run policystrata run --domain finance_saas --suite seeded --out runs/finance
+uv run policystrata baselines runs/example
+```
+
+The default `run` command writes:
+
+```text
+runs/<id>/traces.jsonl
+runs/<id>/summary.json
+runs/<id>/metadata.json
+runs/<id>/witnesses/*.json
+```
+
+`metadata.json` records the mutation operator set, suite provenance, evidence level, and
+detector-freeze status. Static suite YAML can declare `suite_metadata` so externally authored,
+detector-frozen, or incident-reconstruction cases stay separate from public/generated benchmark
+scores.
+
+Regenerate paper-style evidence tables with:
+
+```bash
+scripts/reproduce-evidence.sh
+```
+
+Generate reviewer-facing artifact metrics for a run:
+
+```bash
+uv run policystrata artifact-report runs/repro/seeded
+```
+
+Current benchmark details are in [docs/evidence.md](docs/evidence.md), with methodology and claim
+boundaries in [docs/methodology.md](docs/methodology.md) and [EVAL_CARD.md](EVAL_CARD.md).
+
+## Run The Scanner
+
+`policystrata scan` is the production-oriented path. It treats PolicyStrata as a scanner and
+release gate, not as the authorization boundary.
+
+Clean smoke test:
+
+```bash
+uv run policystrata scan --config examples/postgres_dbt/policystrata_clean.yaml --out runs/scan-clean
+```
+
+Intentional gate-failure fixture:
+
+```bash
+uv run policystrata scan --config examples/postgres_dbt/policystrata.yaml --out runs/scan
+```
+
+That fixture should exit `1` because it contains imported traces with known authorization,
+unsafe-release, and tenant-scope findings.
+
+Scanner outputs include:
+
+```text
+runs/scan-clean/scan.json
+runs/scan-clean/findings.jsonl
+runs/scan-clean/summary.json
+runs/scan-clean/report.md
+runs/scan-clean/witnesses/*.json
+runs/scan-clean/scan.sarif  # when sarif: true
+```
+
+For a scanner run that also executes imported SQL beside canonical compiler SQL against the
+Docker/PostgreSQL fixture:
+
+```bash
+docker compose up -d postgres
+uv run policystrata scan --config examples/postgres_dbt/policystrata_real_db_clean.yaml --out runs/scan-real-db-clean
+```
+
+Postgres access goes through Python/`psycopg`; host `psql` is not required. See
+[docs/scanner.md](docs/scanner.md) for scanner configuration, gate behavior, state assertions, and
+real-database fixture details.
+
+## GitHub Action
+
+Use the first-party action to run `policystrata scan` as a pull-request or release gate:
+
+```yaml
+name: PolicyStrata
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  scan:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: raintree-technology/policystrata@v0.1.0
+        with:
+          config: policystrata.yaml
+          out: runs/policystrata
+```
+
+See [docs/github-action.md](docs/github-action.md) for inputs, artifact upload, and database
+fixture guidance.
+
+## Integrations And Exports
+
+PolicyStrata keeps core execution independent from external eval frameworks. Adapter exports are
+available for downstream systems:
+
+```bash
+uv run policystrata export runs/example --format inspect --out runs/example/inspect.jsonl
+uv run policystrata export runs/example --format benchflow --out runs/example/benchflow.json
+```
+
+The repo also includes a small dbt Semantic Layer adapter and fixture:
+
+```bash
+uv run policystrata check-integration dbt-semantic \
+  --domain finance_saas \
+  --path examples/integrations/dbt_semantic/finance_saas/semantic_models.yml
+```
+
+See [docs/trace-interop.md](docs/trace-interop.md) for adapter field mappings.
+
+## Reference Docs
+
+- [docs/benchmark-reference.md](docs/benchmark-reference.md): domains, generated mutants,
+  baselines, and witness shape.
+- [docs/scanner.md](docs/scanner.md): scanner inputs, gates, state assertions, and PostgreSQL
+  fixture use.
+- [docs/github-action.md](docs/github-action.md): CI wrapper for `policystrata scan`.
+- [docs/distribution-roadmap.md](docs/distribution-roadmap.md): CLI, GitHub Action, SDK, MCP, and
+  GitHub CLI extension sequence.
+- [docs/evidence.md](docs/evidence.md): current evidence snapshot and reproduction commands.
+- [docs/methodology.md](docs/methodology.md): claims, limitations, mutant definitions, and witness
+  minimization.
+- [EVAL_CARD.md](EVAL_CARD.md): benchmark provenance, evidence levels, and eval boundaries.
+- [docs/open-source-commercial-strategy.md](docs/open-source-commercial-strategy.md): packaging and
+  product boundary.
+
+## Development
+
+```bash
+uv run pytest
+uv run ruff check .
+uv run mypy src
+```
+
+The built-in `support_saas` domain is deterministic and seed-driven. Preserve JSON/YAML trace
+stability when extending artifacts; add fields compatibly.
+
+## Status
+
+PolicyStrata is an early research artifact. It is useful for reproducing the paper's core failure
+model and for building regression gates around real stacks. It does not prove recall on unknown
+production incidents, and it should not be represented as a production security scanner by itself.