focus-data-toolkit 0.11.0__tar.gz → 0.12.0__tar.gz

{focus_data_toolkit-0.11.0 → focus_data_toolkit-0.12.0}/CHANGELOG.md

@@ -9,6 +9,148 @@ policy.
 
 ## [Unreleased]
 
+## [0.12.0] — 2026-08-21
+
+Back-ports the reviewed FOCUS-Sample-Data conformance fixes into the generator engine.
+
+> This release carries a **deliberate reproducibility break**: every golden fixture was
+> regenerated once, so synthetic output bytes differ from `0.11.0` for identical generation
+> parameters. That is a new byte baseline, not a regression — see
+> [docs/versioning.md](docs/versioning.md); pin an exact version if you need byte stability.
+
+### Fixed
+
+- **Exact cost arithmetic in the generators.** Costs are now exact products of their
+  factors: the unit price (10 dp) and quantity (4 dp) are quantised, never the product,
+  so `ListCost == ListUnitPrice × PricingQuantity` and `ContractedCost ==
+  ContractedUnitPrice × PricingQuantity` hold under exact `Decimal` equality on every
+  priced row (previously ~46% of usage rows carried a ≤ 5e-7 rounding delta). Products
+  are display-trimmed to 6 decimals only when lossless. The Parquet decimal registry
+  default widens from `(38, 12)` to `(38, 14)` to hold these products exactly.
+- **Three prices kept apart on commitment-covered usage.** `ContractedUnitPrice` now
+  carries only the negotiated rate; the commitment discount shows only between
+  `ContractedCost` and `EffectiveCost`, giving the strict `EffectiveCost <
+  ContractedCost ≤ ListCost` ordering on Used rows (previously the commitment rate was
+  folded into the contracted price and `ContractedCost == EffectiveCost`).
+- **FOCUS erratum #3 `ContractApplied` casing.** The 1.3 generators emit the canonical
+  `ContractId` / `ContractCommitmentId` element keys of the 1.3.0.1 rule model, and the
+  1.3 parser accepts **both** casings — legacy pre-erratum `ContractID` /
+  `ContractCommitmentID` input is normalized and surfaced once per conversion as the
+  new catalogued `FDT-CA-001` warning (compatibility is never silent equivalence; mixed
+  casings in one element are rejected as ambiguous). `to_json(..., version="1.3")` emits
+  the canonical casing. Supersedes the 0.2.0 note that recorded the uppercase casing as
+  the fix.
+- **Full 1.2 billing identity on commitment groups.** 1.2 commitment usage rows now copy
+  all seven identity keys from the purchase (as 1.3 already did), so each
+  `BillingAccountId` maps to exactly one `BillingAccountName` and one `InvoiceId`
+  (previously name and invoice diverged within every 1.2 commitment group).
+- **`PricingCurrency` on 1.2 Tax/Credit rows.** Tax and Credit rows carry
+  `PricingCurrency` and `PricingCurrencyEffectiveCost` in 1.2 as in 1.3, and
+  same-currency pricing columns are exact mirrors instead of re-quantised views.
+
+### Added
+
+- **Generated-data conformance suite** (`tests/test_generated_conformance.py`): the
+  upstream FOCUS-Sample-Data checker catalogue (24 assertions per provider for 1.2,
+  36-37 for 1.3) ported onto the toolkit's generators and run against both fresh
+  generation and the committed golden fixtures — exact `Decimal` equality throughout.
+- **Official-validator CI gate** (`scripts/validate_official_samples.py` + the
+  `official-validation` job in `ci.yml`): the nine generated outputs (3 providers ×
+  1.2 Cost and Usage, × 1.3 Cost and Usage + Contract Commitment) are validated by the
+  official FinOps `focus-validator`, pinned to `2.2.1`, with
+  `--applicability-criteria ALL` so the conditional rules for the capabilities these
+  datasets exercise run too, offline via the packaged / SHA-256-pinned release rule
+  models. Each `(version, provider, dataset)` run is compared against its **exact**
+  expected artifact set — per-rule justifications printed on every run, each claim
+  pinned by a data-side conformance test; an unexpected failure *or* a stale
+  allowlist entry (an expected artifact that stops failing) breaks the build. The
+  1.3 Contract Commitment dataset validates with zero artifacts.
+- **`scripts/regenerate_golden_fixtures.py`**: replays the exact golden grid of
+  `tests/test_generator_golden.py`, replacing the ad-hoc regeneration procedure.
+- **CI: `dependency-review` job** in `.github/workflows/security.yml` (PR-only, informational).
+  Now that the repository Dependency Graph is enabled, `actions/dependency-review-action` gates a
+  PR's dependency **diff** against the GitHub Advisory database and fails on a HIGH+ vulnerability.
+  It complements `pip-audit` (which scans the installed Linux/py3.12 environment every run) by
+  covering the full locked graph at PR time — including platform-/version-conditional packages
+  (e.g. the Windows-only `tzdata`) that the single-platform install never exercises — ahead of
+  Dependabot's post-merge alerts. Least privilege (`contents: read`), guarded to `pull_request`
+  events, SHA-pinned; deliberately not a required check.
+
+### Changed
+
+- **Commitment discounts are modelled per charge period and reconcile exactly.** Each
+  commitment now emits whole per-period blocks: a `Recurring` Purchase row
+  (`BilledCost` = the per-period fee, `EffectiveCost` = 0, explicit
+  `CommitmentDiscountQuantity`/`Unit` — committed spend in USD or capacity in the
+  native unit), Used rows for consumed capacity, and one `Unused` row absorbing the
+  use-it-or-lose-it remainder — so `sum(Usage.EffectiveCost) ==
+  sum(Purchase.BilledCost)` holds under exact equality per charge period and per
+  billing period (previously one One-Time all-upfront purchase with a handful of
+  covered hours and no reconciliation). The provider terms are `NoUpfront`
+  accordingly (payment-option metadata, SKU names and purchase descriptions — a
+  recurring fee contradicts all-upfront). Spend commitments price a **monetary
+  block** on their Purchase and Unused rows: `PricingUnit` is the currency, the
+  unit prices are exactly 1.00 and the priced quantity is the committed/unused
+  spend itself (Used rows keep the consuming resource's native pricing).
+  `ContractApplied` is attached to every 1.3 commitment row — Purchase
+  (`ContractCommitmentId == ResourceId` per rule `O-039-C`), Used and Unused alike
+  — with all five element keys always present (rule `O-007-M`) and exactly one
+  metric branch per category: a spend commitment applies a cost alone, a usage
+  commitment applies the measured quantity in its native unit alone, so the
+  quantity branch survives the 1.4 `oneOf` migration instead of being demoted to
+  `x_` custom keys.
+- **The Contract Commitment dataset carries term totals and negotiated terms.** Costs
+  and quantities are the 1-year term totals; Spend commitments leave quantity/unit
+  empty while Usage commitments carry a real quantity in its native unit; the contract
+  period encloses the commitment period by 90 days. Three negotiated non-discount
+  terms per provider (minimum spend, negotiated rate card, usage commitment) share one
+  multi-commitment `ContractId` and are reachable from Cost and Usage **exclusively**
+  through `ContractApplied` — the FOCUS-defined dataset relationship — never via
+  `CommitmentDiscountId` equality. On-demand 1.3 usage rows reference them with
+  cross-dataset unit coherence: the rate card and minimum spend apply a cost on every
+  row (unit-agnostic), while the usage commitment — contracted in Hours — receives
+  quantities only from usage of the commitment-eligible compute service, measured in
+  that same unit (an element applied to a Usage-category commitment always matches
+  its `ContractCommitmentUnit`).
+- **Split Cost Allocation rows are coherent groups.** One shared host charge is fully
+  allocated to 2-3 distinct workloads in a single charge period: `AllocatedRatio`
+  values sum to exactly 1 and every cost column conserves the host amount exactly
+  (quantity shares absorb the residue; each row's costs stay exact unit-price ×
+  quantity products). The residue arithmetic is shared with
+  `generators/scenarios.py` via the new `generators/engine/allocation_math` module.
+- **Docs: `docs/runner.md` rewritten as a step-by-step guide, and several claims corrected** (no
+  runtime code changed). The page now walks a newcomer from `docker pull` to a validated FOCUS 1.4
+  output — prerequisites, a mount-free first run, directory setup, a six-step walkthrough with the
+  real command output, Podman, GitHub Actions / Kubernetes / cron examples, and a troubleshooting
+  table — before the reference sections. Corrections to previously published statements: the disk
+  budgets apply to the **streaming path only** (the eager CSV conversion ignores them and cannot
+  exit 5); the exit-code table is the **`convert`** contract, not a global one; the image ships the
+  `[parquet]` extra only, so `validate --official` and `ui` are unavailable in the container; the
+  image's CycloneDX SBOM is retained as a **workflow artifact**, not attached to the release or
+  pushed to the registry; trivy fails on HIGH/CRITICAL findings **that have a fix available**
+  (`ignore-unfixed: true`). Also documents `FOCUS_TOOLKIT_LOG_LEVEL` (with its current lack of
+  effect in the Runner), the `_run.json` sidecar, `focus-toolkit clean` after a hard kill, and the
+  `TMPDIR` vs `FOCUS_TOOLKIT_WORK_DIR` distinction for `validate-bundle`. Two further scope
+  corrections: the cooperative cancel (SIGTERM → exit 130, nothing published) is a property of the
+  **streaming** path — the eager CSV conversion installs no signal handler, so `docker stop` there
+  terminates it mid-flight (exit 143) and can leave staging behind; and `clean` takes only `--out`,
+  so it does **not** sweep `fdt-<run_id>` scratch orphaned under `FOCUS_TOOLKIT_WORK_DIR` by a
+  killed streaming run. `docs/compatibility.md` now points Windows readers to the Runner for the
+  streaming path.
+- **CI (risk-based audit follow-up; no runtime code changed).** The container SIGTERM smoke
+  is now deterministic: it waits for the conversion to observably start before stopping, and
+  a run that finishes before the signal lands **fails** as inconclusive instead of passing
+  silently. The bounded-memory streaming test (`-m slow`, ~10 min under tracemalloc) now runs
+  automatically in a new `scale.yml` — on streaming-engine PRs, every push to main, and on
+  demand — instead of never. `reproducibility.yml` uses the exact locked build procedure of
+  `release-build.yml` (hash-pinned backend, `--no-isolation`). `release-dry-run.yml` also
+  triggers on PRs touching release-relevant paths (informational). One macOS combo
+  (`macos-15`/3.13) joins the test matrix. pip-audit now also audits the `[validator]` extra,
+  and the `package` job smoke-installs `[all]`. All jobs set `timeout-minutes`; the container
+  workflow gained `concurrency` cancellation; `actions/checkout` and `astral-sh/setup-uv`
+  pins are converged to one version repo-wide. Coverage floor raised 80 → 85. New unit tests
+  cover the `official_validator` subprocess wrapper.
+
 ## [0.11.0] — 2026-07-18
 
 First **stable** release. Same feature set as `0.11.0rc1`, promoted to a final release now that the
@@ -552,7 +694,8 @@ conformance defects.
 
 <!-- Reference links. 0.2.0/0.3.0 were pre-release development milestones and were never tagged
      or published, so only the first public release (0.9.0) has a tag link. -->
-[Unreleased]: https://github.com/guymano/focus-data-toolkit/compare/v0.11.0...HEAD
+[Unreleased]: https://github.com/guymano/focus-data-toolkit/compare/v0.12.0...HEAD
+[0.12.0]: https://github.com/guymano/focus-data-toolkit/compare/v0.11.0...v0.12.0
 [0.11.0]: https://github.com/guymano/focus-data-toolkit/compare/v0.11.0rc1...v0.11.0
 [0.11.0rc1]: https://github.com/guymano/focus-data-toolkit/compare/v0.9.0...v0.11.0rc1
 [0.9.0]: https://github.com/guymano/focus-data-toolkit/releases/tag/v0.9.0

focus_data_toolkit-0.12.0/PKG-INFO

@@ -0,0 +1,196 @@
+Metadata-Version: 2.4
+Name: focus-data-toolkit
+Version: 0.12.0
+Summary: Generate provider-realistic FOCUS 1.2/1.3 sample data (AWS, Azure, GCP) and convert it to the four FOCUS 1.4 datasets — strictly from source facts, completed by client supplements including native AWS/Azure/GCP exports — with schema detection, gap analysis, a cross-dataset validation gate and atomic writes.
+Author: Guy-Hermann Adiko
+License-Expression: MIT AND CC-BY-4.0
+Project-URL: Homepage, https://github.com/guymano/focus-data-toolkit
+Project-URL: Source, https://github.com/guymano/focus-data-toolkit
+Project-URL: Issues, https://github.com/guymano/focus-data-toolkit/issues
+Project-URL: Changelog, https://github.com/guymano/focus-data-toolkit/blob/main/CHANGELOG.md
+Project-URL: Documentation, https://github.com/guymano/focus-data-toolkit#readme
+Keywords: finops,focus,billing,cost,sample-data
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Office/Business :: Financial
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+License-File: LICENSES/CC-BY-4.0.txt
+Provides-Extra: parquet
+Requires-Dist: pyarrow<26,>=15; extra == "parquet"
+Requires-Dist: tzdata>=2024.1; sys_platform == "win32" and extra == "parquet"
+Provides-Extra: scale
+Requires-Dist: focus-data-toolkit[parquet]; extra == "scale"
+Provides-Extra: validator
+Requires-Dist: focus-validator<3,>=2.1; python_version >= "3.12" and extra == "validator"
+Provides-Extra: studio
+Requires-Dist: fastapi<1,>=0.110; extra == "studio"
+Requires-Dist: uvicorn[standard]<1,>=0.29; extra == "studio"
+Requires-Dist: python-multipart<1,>=0.0.9; extra == "studio"
+Provides-Extra: studio-all
+Requires-Dist: focus-data-toolkit[parquet,studio]; extra == "studio-all"
+Provides-Extra: all
+Requires-Dist: focus-data-toolkit[parquet,studio,validator]; extra == "all"
+Provides-Extra: release
+Requires-Dist: build<2,>=1; extra == "release"
+Requires-Dist: twine<8,>=5; extra == "release"
+Provides-Extra: dev
+Requires-Dist: focus-data-toolkit[parquet,release,studio]; extra == "dev"
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-cov<8,>=5; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: mypy<2.4,>=1.11; extra == "dev"
+Requires-Dist: jsonschema<5,>=4.21; extra == "dev"
+Requires-Dist: httpx<1,>=0.27; extra == "dev"
+Dynamic: license-file
+
+# focus-data-toolkit
+
+Generate realistic **FOCUS 1.2 / 1.3** cost & usage data (AWS, Azure, GCP), convert it to
+**FOCUS 1.4**, and validate the result — from a single dependency-free Python core, usable three ways.
+
+[![PyPI](https://img.shields.io/pypi/v/focus-data-toolkit)](https://pypi.org/project/focus-data-toolkit/)
+[![Python](https://img.shields.io/pypi/pyversions/focus-data-toolkit)](https://pypi.org/project/focus-data-toolkit/)
+[![License: MIT AND CC-BY-4.0](https://img.shields.io/badge/license-MIT%20AND%20CC--BY--4.0-blue)](LICENSE)
+[![CI](https://github.com/guymano/focus-data-toolkit/actions/workflows/ci.yml/badge.svg)](https://github.com/guymano/focus-data-toolkit/actions/workflows/ci.yml)
+
+[FOCUS](https://focus.finops.org) is the open standard for cloud cost & usage data. FOCUS 1.4 defines
+four datasets: **Cost and Usage**, **Contract Commitment**, **Billing Period** and **Invoice Detail**.
+This toolkit migrates real data, synthesizes sample data, and checks structure — honestly: a
+structurally valid file is not automatically FOCUS-conformant, and any value it cannot derive from
+the source is either left empty (**strict** mode) or filled only as a clearly-labelled assumption
+(**synthetic** mode).
+
+## Three ways to use it
+
+| Interface | Best for | Get started |
+|---|---|---|
+| **Studio** | FinOps users who prefer a UI | `pip install "focus-data-toolkit[studio]"` → `focus-toolkit ui` |
+| **Runner** | Automation & large volumes | `docker run --rm ghcr.io/guymano/focus-data-toolkit:0.12.0 version` |
+| **CLI & SDK** | Engineers & scripts | `pip install focus-data-toolkit` → `focus-toolkit --help` |
+
+All three drive the **same core**, so their outputs are byte-for-byte identical — same datasets,
+manifest, diagnostics and `SHA256SUMS`. No FOCUS logic is duplicated across interfaces.
+
+- **Studio** — a local, single-user web app: pick or upload a file (or generate one), detect it,
+  convert with live progress, preview a sampled page, and download the results. Binds `127.0.0.1`,
+  token-guarded; no data leaves your machine. → [docs/studio.md](docs/studio.md)
+- **Runner** — a non-root OCI image whose entrypoint *is* the CLI, for batch jobs and CI.
+  → [docs/runner.md](docs/runner.md)
+- **CLI & SDK** — the `focus-toolkit` command and the importable Python API.
+
+## Install
+
+```bash
+pip install focus-data-toolkit
+```
+
+The core is **standard-library only** (Python ≥ 3.11). Optional features live behind extras:
+
+| Extra | Adds |
+|---|---|
+| `parquet` | Parquet I/O — columnar, decimal128, Hive partitioning (alias: `scale`) |
+| `studio` | the local web UI (`focus-toolkit ui`) |
+| `studio-all` | `studio` + `parquet` |
+| `validator` | the official FinOps validator (`--official`; needs Python ≥ 3.12) |
+| `all` | `parquet` + `validator` + `studio` |
+
+```bash
+pipx install focus-data-toolkit                        # isolated CLI
+uv tool install focus-data-toolkit                     # or with uv
+docker pull ghcr.io/guymano/focus-data-toolkit:0.12.0  # container (Runner)
+```
+
+## Quickstart
+
+```bash
+# 1. Generate provider-realistic sample data (aws|azure|gcp, FOCUS 1.2|1.3)
+focus-toolkit generate --provider aws --focus-version 1.3 --rows 1000 --out ./out
+
+# 2. Convert a 1.2/1.3 Cost & Usage file to FOCUS 1.4 (source version auto-detected)
+focus-toolkit convert --cost-and-usage out/focus_1_3_cost_and_usage_aws.csv --out ./focus-1.4
+#    strict conversion of this sample exits 3 (three datasets NOT_PRODUCED, by design) —
+#    add --exit-policy pipeline in `set -e` / CI scripts. Use --mode synthetic to also emit
+#    the other three datasets (clearly labelled), or --stream --output-format parquet for large files.
+
+# 3. Open the Studio web app (needs the studio extra: pip install "focus-data-toolkit[studio]")
+focus-toolkit ui
+```
+
+Use it as a library — the same engine the CLI, Studio and Runner run:
+
+```python
+from focus_data_toolkit import convert_files
+
+convert_files("cost_and_usage.csv", "focus-1.4", mode="strict")
+```
+
+## What it does — and what it doesn't
+
+- **Converts** FOCUS **1.2 / 1.3 Cost & Usage → the four FOCUS 1.4 datasets.** `strict` mode emits
+  only what is genuinely derivable from the source; `synthetic` mode also fills the
+  provider-billing datasets with clearly-labelled assumed data for demos and tests.
+- **Honest by design.** The source is Cost & Usage only, so Contract Commitment / Billing Period /
+  Invoice Detail come **only** from client [supplements](docs/supplements.md) or synthetic mode.
+  There is **no FOCUS 1.4 generator** and **no invalid-data generator**. Generation is in-memory
+  (use the CLI/Runner for very large synthetic sets), and the engine is **single-node**.
+- **Built for real data.** Bounded-memory streaming conversion, atomic/journaled writes, a
+  deterministic manifest with per-column lineage, structured `FDT-*` diagnostics, and a
+  cross-dataset validation gate.
+
+See [docs/compatibility.md](docs/compatibility.md) for the Python / OS / FOCUS support matrix.
+
+## Exit codes
+
+`convert` returns meaningful codes so pipelines can branch on the outcome:
+
+| Code | Meaning |
+|---|---|
+| `0` | success, no assumptions |
+| `1` | lint / validation / write failure |
+| `2` | invalid input or arguments |
+| `3` | strict result intentionally incomplete |
+| `4` | synthetic result contains assumptions |
+| `5` | disk space / budget exhausted (streaming / Runner) |
+| `130` | cancelled (Ctrl-C / SIGTERM) — nothing partial is published |
+
+Code `5` is raised on the bounded-memory path (`--stream`, Parquet, partitioning, or the Runner); the
+eager default CSV conversion reports a disk-full as a write failure (`1`). Add `--exit-policy pipeline`
+to treat the functional-but-incomplete outcomes (`3`, `4`) as `0`.
+
+## Trust & provenance
+
+- **Deterministic:** the same input always produces the same output bytes (no clock, no RNG); the
+  datasets and manifest are listed in `SHA256SUMS` (the non-deterministic `_run.json` run sidecar is
+  intentionally excluded).
+- **Verifiable model:** the embedded FOCUS 1.4 model is extracted from the FinOps source workbook and
+  its provenance is **complete** — hash-pinned and reproduced byte-for-byte.
+  See [docs/model-provenance.md](docs/model-provenance.md).
+- **Supply chain:** releases are built reproducibly, signed (Sigstore/cosign), and ship an SBOM and
+  build attestations. See [docs/releasing.md](docs/releasing.md).
+- **Private:** the core does **no network I/O**, has **no telemetry**, and needs **no credentials**.
+  See [docs/security-model.md](docs/security-model.md).
+
+## Documentation
+
+[Studio](docs/studio.md) · [Runner](docs/runner.md) · [Supplements & gaps](docs/supplements.md) ·
+[Compatibility](docs/compatibility.md) · [Versioning](docs/versioning.md) ·
+[Security model](docs/security-model.md) · [Model provenance](docs/model-provenance.md) ·
+[Releasing](docs/releasing.md) · [Changelog](CHANGELOG.md) · [Contributing](CONTRIBUTING.md) ·
+[Security policy](SECURITY.md)
+
+## License & credits
+
+Code is **MIT** (see [LICENSE](LICENSE)). The embedded FOCUS 1.4 data model is a derivative of the
+FinOps FOCUS data-model workbook, © the FinOps Foundation and licensed **CC-BY-4.0**, redistributed
+with attribution (see [NOTICE](NOTICE)). "FOCUS" and "FinOps" are trademarks of the FinOps
+Foundation; this is an independent community project, not endorsed by the FinOps Foundation. Related
+sample datasets were contributed upstream to
+[FOCUS-Sample-Data](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS-Sample-Data).

focus_data_toolkit-0.12.0/README.md

@@ -0,0 +1,143 @@
+# focus-data-toolkit
+
+Generate realistic **FOCUS 1.2 / 1.3** cost & usage data (AWS, Azure, GCP), convert it to
+**FOCUS 1.4**, and validate the result — from a single dependency-free Python core, usable three ways.
+
+[![PyPI](https://img.shields.io/pypi/v/focus-data-toolkit)](https://pypi.org/project/focus-data-toolkit/)
+[![Python](https://img.shields.io/pypi/pyversions/focus-data-toolkit)](https://pypi.org/project/focus-data-toolkit/)
+[![License: MIT AND CC-BY-4.0](https://img.shields.io/badge/license-MIT%20AND%20CC--BY--4.0-blue)](LICENSE)
+[![CI](https://github.com/guymano/focus-data-toolkit/actions/workflows/ci.yml/badge.svg)](https://github.com/guymano/focus-data-toolkit/actions/workflows/ci.yml)
+
+[FOCUS](https://focus.finops.org) is the open standard for cloud cost & usage data. FOCUS 1.4 defines
+four datasets: **Cost and Usage**, **Contract Commitment**, **Billing Period** and **Invoice Detail**.
+This toolkit migrates real data, synthesizes sample data, and checks structure — honestly: a
+structurally valid file is not automatically FOCUS-conformant, and any value it cannot derive from
+the source is either left empty (**strict** mode) or filled only as a clearly-labelled assumption
+(**synthetic** mode).
+
+## Three ways to use it
+
+| Interface | Best for | Get started |
+|---|---|---|
+| **Studio** | FinOps users who prefer a UI | `pip install "focus-data-toolkit[studio]"` → `focus-toolkit ui` |
+| **Runner** | Automation & large volumes | `docker run --rm ghcr.io/guymano/focus-data-toolkit:0.12.0 version` |
+| **CLI & SDK** | Engineers & scripts | `pip install focus-data-toolkit` → `focus-toolkit --help` |
+
+All three drive the **same core**, so their outputs are byte-for-byte identical — same datasets,
+manifest, diagnostics and `SHA256SUMS`. No FOCUS logic is duplicated across interfaces.
+
+- **Studio** — a local, single-user web app: pick or upload a file (or generate one), detect it,
+  convert with live progress, preview a sampled page, and download the results. Binds `127.0.0.1`,
+  token-guarded; no data leaves your machine. → [docs/studio.md](docs/studio.md)
+- **Runner** — a non-root OCI image whose entrypoint *is* the CLI, for batch jobs and CI.
+  → [docs/runner.md](docs/runner.md)
+- **CLI & SDK** — the `focus-toolkit` command and the importable Python API.
+
+## Install
+
+```bash
+pip install focus-data-toolkit
+```
+
+The core is **standard-library only** (Python ≥ 3.11). Optional features live behind extras:
+
+| Extra | Adds |
+|---|---|
+| `parquet` | Parquet I/O — columnar, decimal128, Hive partitioning (alias: `scale`) |
+| `studio` | the local web UI (`focus-toolkit ui`) |
+| `studio-all` | `studio` + `parquet` |
+| `validator` | the official FinOps validator (`--official`; needs Python ≥ 3.12) |
+| `all` | `parquet` + `validator` + `studio` |
+
+```bash
+pipx install focus-data-toolkit                        # isolated CLI
+uv tool install focus-data-toolkit                     # or with uv
+docker pull ghcr.io/guymano/focus-data-toolkit:0.12.0  # container (Runner)
+```
+
+## Quickstart
+
+```bash
+# 1. Generate provider-realistic sample data (aws|azure|gcp, FOCUS 1.2|1.3)
+focus-toolkit generate --provider aws --focus-version 1.3 --rows 1000 --out ./out
+
+# 2. Convert a 1.2/1.3 Cost & Usage file to FOCUS 1.4 (source version auto-detected)
+focus-toolkit convert --cost-and-usage out/focus_1_3_cost_and_usage_aws.csv --out ./focus-1.4
+#    strict conversion of this sample exits 3 (three datasets NOT_PRODUCED, by design) —
+#    add --exit-policy pipeline in `set -e` / CI scripts. Use --mode synthetic to also emit
+#    the other three datasets (clearly labelled), or --stream --output-format parquet for large files.
+
+# 3. Open the Studio web app (needs the studio extra: pip install "focus-data-toolkit[studio]")
+focus-toolkit ui
+```
+
+Use it as a library — the same engine the CLI, Studio and Runner run:
+
+```python
+from focus_data_toolkit import convert_files
+
+convert_files("cost_and_usage.csv", "focus-1.4", mode="strict")
+```
+
+## What it does — and what it doesn't
+
+- **Converts** FOCUS **1.2 / 1.3 Cost & Usage → the four FOCUS 1.4 datasets.** `strict` mode emits
+  only what is genuinely derivable from the source; `synthetic` mode also fills the
+  provider-billing datasets with clearly-labelled assumed data for demos and tests.
+- **Honest by design.** The source is Cost & Usage only, so Contract Commitment / Billing Period /
+  Invoice Detail come **only** from client [supplements](docs/supplements.md) or synthetic mode.
+  There is **no FOCUS 1.4 generator** and **no invalid-data generator**. Generation is in-memory
+  (use the CLI/Runner for very large synthetic sets), and the engine is **single-node**.
+- **Built for real data.** Bounded-memory streaming conversion, atomic/journaled writes, a
+  deterministic manifest with per-column lineage, structured `FDT-*` diagnostics, and a
+  cross-dataset validation gate.
+
+See [docs/compatibility.md](docs/compatibility.md) for the Python / OS / FOCUS support matrix.
+
+## Exit codes
+
+`convert` returns meaningful codes so pipelines can branch on the outcome:
+
+| Code | Meaning |
+|---|---|
+| `0` | success, no assumptions |
+| `1` | lint / validation / write failure |
+| `2` | invalid input or arguments |
+| `3` | strict result intentionally incomplete |
+| `4` | synthetic result contains assumptions |
+| `5` | disk space / budget exhausted (streaming / Runner) |
+| `130` | cancelled (Ctrl-C / SIGTERM) — nothing partial is published |
+
+Code `5` is raised on the bounded-memory path (`--stream`, Parquet, partitioning, or the Runner); the
+eager default CSV conversion reports a disk-full as a write failure (`1`). Add `--exit-policy pipeline`
+to treat the functional-but-incomplete outcomes (`3`, `4`) as `0`.
+
+## Trust & provenance
+
+- **Deterministic:** the same input always produces the same output bytes (no clock, no RNG); the
+  datasets and manifest are listed in `SHA256SUMS` (the non-deterministic `_run.json` run sidecar is
+  intentionally excluded).
+- **Verifiable model:** the embedded FOCUS 1.4 model is extracted from the FinOps source workbook and
+  its provenance is **complete** — hash-pinned and reproduced byte-for-byte.
+  See [docs/model-provenance.md](docs/model-provenance.md).
+- **Supply chain:** releases are built reproducibly, signed (Sigstore/cosign), and ship an SBOM and
+  build attestations. See [docs/releasing.md](docs/releasing.md).
+- **Private:** the core does **no network I/O**, has **no telemetry**, and needs **no credentials**.
+  See [docs/security-model.md](docs/security-model.md).
+
+## Documentation
+
+[Studio](docs/studio.md) · [Runner](docs/runner.md) · [Supplements & gaps](docs/supplements.md) ·
+[Compatibility](docs/compatibility.md) · [Versioning](docs/versioning.md) ·
+[Security model](docs/security-model.md) · [Model provenance](docs/model-provenance.md) ·
+[Releasing](docs/releasing.md) · [Changelog](CHANGELOG.md) · [Contributing](CONTRIBUTING.md) ·
+[Security policy](SECURITY.md)
+
+## License & credits
+
+Code is **MIT** (see [LICENSE](LICENSE)). The embedded FOCUS 1.4 data model is a derivative of the
+FinOps FOCUS data-model workbook, © the FinOps Foundation and licensed **CC-BY-4.0**, redistributed
+with attribution (see [NOTICE](NOTICE)). "FOCUS" and "FinOps" are trademarks of the FinOps
+Foundation; this is an independent community project, not endorsed by the FinOps Foundation. Related
+sample datasets were contributed upstream to
+[FOCUS-Sample-Data](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS-Sample-Data).

{focus_data_toolkit-0.11.0 → focus_data_toolkit-0.12.0}/SECURITY.md

@@ -87,8 +87,8 @@ fixes; there are no long-term-support branches yet.
 
 | Version    | Supported                                                   |
 | ---------- | ----------------------------------------------------------- |
-| `0.11.x`   | ✅ Current line — security fixes land here.                 |
-| `< 0.11.0` | ❌ Superseded pre-1.0 releases / snapshots; upgrade to 0.11.x. |
+| `0.12.x`   | ✅ Current line — security fixes land here.                 |
+| `< 0.12.0` | ❌ Superseded pre-1.0 releases / snapshots; upgrade to 0.12.x. |
 
 When `1.0.0` is released, this table will be updated with the then-current
 support policy.

{focus_data_toolkit-0.11.0 → focus_data_toolkit-0.12.0}/docs/compatibility.md

@@ -48,6 +48,10 @@ This is tracked as a "Windows streaming / atomic-write hardening" follow-up. On
 Windows, use the eager conversion path; on Linux/macOS, `--stream` gives flat
 memory regardless of row count.
 
+Windows users who need the streaming path can run it through the container
+instead: the Runner image is Linux, so `--stream`, Parquet and partitioned
+output work there from a Windows host. See [docs/runner.md](runner.md).
+
 > Note: because streaming is unsupported on Windows, the POSIX-only path-traversal
 > guard it relies on is not a Windows security control. See
 > [docs/security-model.md](security-model.md).

{focus_data_toolkit-0.11.0 → focus_data_toolkit-0.12.0}/docs/releasing.md

@@ -9,7 +9,7 @@ The release **pipeline** lives in `.github/workflows/`:
 | Workflow | Trigger | What it does |
 | --- | --- | --- |
 | `release-build.yml` | `workflow_call` (reusable) | Build wheel+sdist **once** (locked toolchain, `SOURCE_DATE_EPOCH` = commit date), test them, generate the CycloneDX SBOM + `SHA256SUMS` + a build manifest, run `verify_release.py`, upload the artifacts. No publish scopes. |
-| `release-dry-run.yml` | `workflow_dispatch` | Calls `release-build` and re-verifies — **no** id-token, **no** environment, publishes nothing. Rehearse on any branch. |
+| `release-dry-run.yml` | `workflow_dispatch` + `pull_request` on release-relevant paths | Calls `release-build` and re-verifies — **no** id-token, **no** environment, publishes nothing. Rehearse on any branch; also runs automatically on PRs touching the release workflows, `constraints/`, the release scripts, packaging metadata or the version/CHANGELOG, so a broken release chain is caught at PR time (informational — deliberately not a required check, because a paths-filtered required check would block unrelated PRs). |
 | `release.yml` | push tag `v*` | Calls `release-build`, then **attests** wheel/sdist/SBOM/checksums (GitHub Artifact Attestations, keyless OIDC) and **publishes** to PyPI via Trusted Publishing in the `pypi` environment. The same artifacts flow by digest — nothing is rebuilt to publish. |
 | `reproducibility.yml` | `workflow_dispatch` | Double-builds and compares (see Reproducibility below). |
 
@@ -50,16 +50,27 @@ these are in place:
       and dependency-review can run (see below).
 - [ ] **CODEOWNERS** confirmed and "require review from Code Owners" turned on.
 
+Pre-release manual steps (each release, owner-run):
+
+- [ ] Run `reproducibility.yml` (workflow_dispatch) on the release commit and check the
+      wheel double-build gate is green — it uses the exact locked build procedure of
+      `release-build.yml`, so a green run vouches for the artifact the release will ship.
+- [ ] Studio browser smoke (the JS frontend has no automated browser test — a deliberate
+      trade-off, see `tests/test_studio.py` for the API-level coverage): `focus-toolkit ui`,
+      open the printed tokenized URL, select a file, run a conversion, watch progress,
+      download the result.
+
 ## The security workflows
 
 `.github/workflows/codeql.yml` and `scorecard.yml` run on `push` / `pull_request`
 / `schedule` (plus `branch_protection_rule` for Scorecard). They upload results
 to GitHub **code scanning**, which must be enabled in the repository settings
 (Operationally Ready checklist above) for the uploads to land.
-`actions/dependency-review-action` similarly needs the **Dependency Graph**; a
-`dependency-review` job can be added to `security.yml` once it is enabled
-(dependency CVEs are already covered by `pip-audit` + Dependabot in the
-meantime).
+`.github/workflows/security.yml` also runs a **`dependency-review`** job (pull
+requests only) that gates a PR's dependency **diff** against the GitHub Advisory
+database; it needs the repository **Dependency Graph** (now enabled). It
+complements `pip-audit` (which scans the installed environment) and Dependabot
+(post-merge alerts).
 
 ## Cutting a release