focus-data-toolkit 0.11.0__py3-none-any.whl

focus_data_toolkit-0.11.0.dist-info/METADATA

@@ -0,0 +1,519 @@
+Metadata-Version: 2.4
+Name: focus-data-toolkit
+Version: 0.11.0
+Summary: Generate provider-realistic FOCUS 1.2/1.3 sample data (AWS, Azure, GCP) and convert it to the four FOCUS 1.4 datasets — strictly from source facts, completed by client supplements including native AWS/Azure/GCP exports — with schema detection, gap analysis, a cross-dataset validation gate and atomic writes.
+Author: Guy-Hermann Adiko
+License-Expression: MIT AND CC-BY-4.0
+Project-URL: Homepage, https://github.com/guymano/focus-data-toolkit
+Project-URL: Source, https://github.com/guymano/focus-data-toolkit
+Project-URL: Issues, https://github.com/guymano/focus-data-toolkit/issues
+Project-URL: Changelog, https://github.com/guymano/focus-data-toolkit/blob/main/CHANGELOG.md
+Project-URL: Documentation, https://github.com/guymano/focus-data-toolkit#readme
+Keywords: finops,focus,billing,cost,sample-data
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Office/Business :: Financial
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+License-File: LICENSES/CC-BY-4.0.txt
+Provides-Extra: parquet
+Requires-Dist: pyarrow<26,>=15; extra == "parquet"
+Requires-Dist: tzdata>=2024.1; sys_platform == "win32" and extra == "parquet"
+Provides-Extra: scale
+Requires-Dist: focus-data-toolkit[parquet]; extra == "scale"
+Provides-Extra: validator
+Requires-Dist: focus-validator<3,>=2.1; python_version >= "3.12" and extra == "validator"
+Provides-Extra: studio
+Requires-Dist: fastapi<1,>=0.110; extra == "studio"
+Requires-Dist: uvicorn[standard]<1,>=0.29; extra == "studio"
+Requires-Dist: python-multipart<1,>=0.0.9; extra == "studio"
+Provides-Extra: studio-all
+Requires-Dist: focus-data-toolkit[parquet,studio]; extra == "studio-all"
+Provides-Extra: all
+Requires-Dist: focus-data-toolkit[parquet,studio,validator]; extra == "all"
+Provides-Extra: release
+Requires-Dist: build<2,>=1; extra == "release"
+Requires-Dist: twine<7,>=5; extra == "release"
+Provides-Extra: dev
+Requires-Dist: focus-data-toolkit[parquet,release,studio]; extra == "dev"
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-cov<8,>=5; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: mypy<2.4,>=1.11; extra == "dev"
+Requires-Dist: jsonschema<5,>=4.21; extra == "dev"
+Requires-Dist: httpx<1,>=0.27; extra == "dev"
+Dynamic: license-file
+
+# focus-data-toolkit
+
+Generate provider-realistic **FOCUS 1.2 / 1.3** sample data (AWS, Azure, GCP),
+**migrate** a FOCUS 1.2/1.3 Cost and Usage table to **FOCUS 1.4**, optionally
+**synthesize** the new 1.4 datasets for demos/tests, and **lint** the results —
+in one dependency-free Python toolkit.
+
+[FOCUS](https://focus.finops.org) (FinOps Open Cost and Usage Specification) is
+the open standard for cloud cost and usage data. FOCUS 1.4 (June 2026) defines
+four datasets: **Cost and Usage** (65 columns), **Contract Commitment**
+(30 columns), **Billing Period** (6 columns, new) and **Invoice Detail**
+(22 columns, new).
+
+## What this toolkit does — and does not — claim
+
+The toolkit is explicit about three very different operations:
+
+- **Schema migration** — transforming columns/formats where the mapping is exact
+  or explicitly documented. *Cost and Usage* migrates 1.2/1.3 → 1.4 essentially
+  losslessly (1.4 only adds nullable columns and drops two deprecated ones).
+- **Enrichment** — adding data from a complementary *authoritative* source
+  (e.g. a real invoice). Not yet ingested; the architecture reserves a place for it.
+- **Synthetic projection** — inventing plausible values for demos/tests/learning.
+  These results are **never** presented as real financial facts or as fully
+  FOCUS-conformant.
+
+The three new-in-1.4 datasets (**Billing Period**, **Invoice Detail**, and the
+1.4-expanded **Contract Commitment**) contain **Mandatory columns that are
+provider billing-system facts** — invoice status, payment terms, issuer-assigned
+ids, provider record timestamps, commitment commercial terms. These are **not
+present in, and not derivable from, a Cost and Usage table**. FOCUS 1.4 itself
+treats them as provider-emitted (it adds an *Invoice Reconciliation* feature and
+a *Rounding Variance Tolerance* appendix precisely because the issued invoice
+legitimately differs from summed usage). So an aggregation of `BilledCost` is a
+useful analytical summary — **not** a real invoice line.
+
+A structurally valid file is **not** automatically FOCUS-conformant.
+
+## Modes
+
+| Mode | Behaviour |
+|---|---|
+| **`strict`** (default) | Never invents provider facts. A canonical FOCUS 1.4 dataset is produced only when every Mandatory non-nullable column has a factual lineage (observed / renamed / derived / enriched). From a Cost-and-Usage source that means **only Cost and Usage** is produced; the other three are reported `NOT_PRODUCED` in the manifest, with the exact blocking columns. |
+| **`synthetic`** | Generates assumed values so you get all four datasets for demos/tests. Affected datasets are written with a `synthetic_` filename prefix and marked `PRODUCED_SYNTHETIC` in the manifest; the result is never labelled fully conformant. |
+
+Every conversion writes a deterministic **manifest** (`focus_1_4_manifest.json`)
+recording, per column, how each value was obtained
+(`OBSERVED` / `RENAMED` / `DERIVED` / `ENRICHED` / `ASSUMED` / `UNAVAILABLE`).
+
+The core package is **standard-library only** (Python ≥ 3.11).
+
+## Installation
+
+### Prerequisites
+
+- **Python 3.11, 3.12 or 3.13** (the `validator` extra needs **3.12+**). Linux, macOS and
+  Windows are supported — CI tests Ubuntu and Windows across all three versions (see
+  [docs/compatibility.md](docs/compatibility.md)).
+- `pip` (or [uv](https://docs.astral.sh/uv/)). **No compiler is needed**: the core package is
+  pure Python and standard-library only — zero runtime dependencies.
+
+### Standard install
+
+In a virtual environment (recommended):
+
+```bash
+python -m venv .venv
+source .venv/bin/activate        # Windows: .venv\Scripts\activate
+pip install focus-data-toolkit
+```
+
+> **Until the first PyPI release**, install straight from GitHub instead:
+> `pip install "git+https://github.com/guymano/focus-data-toolkit"`
+> (append `@vX.Y.Z` to pin a tag once releases exist).
+
+This installs the `focus-toolkit` command (alias of `python -m focus_data_toolkit`).
+
+### Optional extras
+
+The core covers generation, 1.2/1.3 → 1.4 conversion, client supplements + the AWS/Azure/GCP
+provider-export adapters, bounded-memory CSV streaming, and every validation gate — all with
+the standard library alone (the streaming state uses built-in `sqlite3`). Extras add:
+
+| Command | Adds | Requires |
+|---|---|---|
+| `pip install "focus-data-toolkit[parquet]"` | Parquet **input and output** (PyArrow; value-exact decimal128, Hive partitioning). On Windows, `tzdata` comes along automatically. | Python ≥ 3.11 |
+| `pip install "focus-data-toolkit[validator]"` | the official FinOps `focus-validator`, used by `focus-toolkit validate --official` (rule models for FOCUS 1.2/1.3) | Python ≥ 3.12 |
+| `pip install "focus-data-toolkit[all]"` | both of the above (on 3.11 the validator part is skipped automatically by its Python marker) | Python ≥ 3.11 |
+
+### With uv
+
+```bash
+uv tool install focus-data-toolkit                     # standalone CLI on your PATH
+uv add "focus-data-toolkit[parquet]"                   # as a project dependency
+uvx --from focus-data-toolkit focus-toolkit --help     # one-off run, nothing installed
+# (--from is needed because the command name differs from the package name)
+```
+
+### From source
+
+```bash
+git clone https://github.com/guymano/focus-data-toolkit
+cd focus-data-toolkit
+pip install .                                # or: pip install ".[all]"
+```
+
+For a development install (editable, with the test/lint toolchain), see
+[Development](#development). Release artifacts (wheel, sdist) attached to GitHub Releases
+ship with `SHA256SUMS`, two SBOM profiles and GitHub Artifact Attestations — verify a
+downloaded wheel and install it directly:
+
+```bash
+# SHA256SUMS covers every release asset; check just the wheel's line:
+grep "focus_data_toolkit-X.Y.Z-py3-none-any.whl" SHA256SUMS | sha256sum -c -
+gh attestation verify focus_data_toolkit-X.Y.Z-py3-none-any.whl --repo guymano/focus-data-toolkit
+pip install ./focus_data_toolkit-X.Y.Z-py3-none-any.whl
+```
+
+(To verify the whole asset set at once, download all files listed in `SHA256SUMS` into one
+directory and run `sha256sum -c SHA256SUMS` there.)
+
+### Verify the installation
+
+```bash
+focus-toolkit --help                                         # CLI is on the PATH
+# venv/project installs only — a `uv tool install` lives in its own isolated
+# environment, so check it via the tool env instead:
+python -c "import focus_data_toolkit as f; print(f.__version__)"
+#   uv tool run --from focus-data-toolkit python -c "import focus_data_toolkit as f; print(f.__version__)"
+# end-to-end smoke test (deterministic, ~1 s):
+focus-toolkit generate --provider aws --focus-version 1.3 --rows 10 --out /tmp/fdt-smoke
+focus-toolkit convert --cost-and-usage /tmp/fdt-smoke/focus_1_3_cost_and_usage_aws.csv \
+  --out /tmp/fdt-smoke/focus-1.4
+# exit code 3 is expected: the strict result is intentionally incomplete without supplements
+```
+
+### Upgrade / uninstall
+
+```bash
+pip install -U focus-data-toolkit
+pip uninstall focus-data-toolkit
+```
+
+## Quickstart
+
+Install first (see [Installation](#installation) above), then:
+
+### 1. Generate FOCUS 1.2/1.3 sample data
+
+```bash
+focus-toolkit generate --provider aws --focus-version 1.3 --rows 1000 --seed 1302 --out ./out
+# -> out/focus_1_3_cost_and_usage_aws.csv (65 columns)
+# -> out/focus_1_3_contract_commitment_aws.csv (13 columns)
+```
+
+Providers: `aws`, `azure`, `gcp`. Versions: `1.2`, `1.3`. Same rows+seed →
+byte-identical output.
+
+### 2. Convert towards FOCUS 1.4
+
+**Strict (default)** — migrate what is genuinely migratable:
+
+```bash
+focus-toolkit convert \
+  --cost-and-usage out/focus_1_3_cost_and_usage_aws.csv \
+  --out ./focus-1.4
+# -> focus-1.4/focus_1_4_cost_and_usage.csv   (65 columns)
+# -> focus-1.4/focus_1_4_manifest.json        (why the other 3 were NOT produced)
+# exit code 3: strict result is intentionally incomplete
+```
+
+**Synthetic** — also generate the provider-emitted datasets for demos/tests:
+
+```bash
+focus-toolkit convert \
+  --cost-and-usage out/focus_1_3_cost_and_usage_aws.csv \
+  --contract-commitment out/focus_1_3_contract_commitment_aws.csv \
+  --out ./focus-1.4 --mode synthetic
+# -> synthetic_focus_1_4_cost_and_usage.csv        (migration + assumed InvoiceDetailId back-link)
+# -> synthetic_focus_1_4_contract_commitment.csv   (assumed terms)
+# -> synthetic_focus_1_4_billing_period.csv        (assumed status/timestamps)
+# -> synthetic_focus_1_4_invoice_detail.csv        (assumed invoice facts)
+# -> focus_1_4_manifest.json
+# exit code 4: synthetic result contains ASSUMED values
+```
+
+In synthetic mode Cost and Usage is also `synthetic_`-prefixed, because its
+`InvoiceDetailId` back-links to the (synthetic) Invoice Detail — every other
+column is a faithful migration, as the manifest's per-column lineage records. In
+strict mode Cost and Usage is emitted unprefixed with its `InvoiceDetailId` left null.
+
+Works the same on your **own** FOCUS 1.2/1.3 exports — the source version is
+detected from the CSV header. CLI exit codes: `0` success without assumptions ·
+`1` lint violation · `2` invalid arguments · `3` incomplete strict result ·
+`4` synthetic result produced with assumptions.
+
+### 3. Lint
+
+```bash
+# built-in FOCUS 1.4 structural + semantic linter (not a full conformance validator)
+focus-toolkit validate focus-1.4/focus_1_4_cost_and_usage.csv --dataset cost-and-usage
+
+# official FinOps validator (optional extra, requires Python >= 3.12; supports 1.2/1.3)
+pip install "focus-data-toolkit[validator]"
+focus-toolkit validate --official --focus-version 1.2.0.1 my_focus_1_2_export.csv
+```
+
+The official FinOps validator ships rule models for FOCUS 1.2/1.3 and does **not**
+yet support 1.4. Until it does, the built-in linter provides a structural +
+semantic check only — it cannot certify full FOCUS 1.4 conformance.
+
+### Working with real client data (0.3.0 / P1)
+
+For consolidated, multi-provider, multi-issuer, multi-currency exports:
+
+- **Schema detection** identifies the dataset *and* version (1.2/1.3/1.4) with a confidence,
+  and reports missing / extension (`x_`) / unknown columns. Strict mode refuses an ambiguous
+  source; force it with `--source-version` / `--source-dataset`.
+- **Grouping keys** use the full billing grain (issuer, invoice, account, currency, period,
+  charge category), so lines from different issuers/accounts/currencies are never merged.
+  Locally generated ids are `x_fdt_…`-prefixed and never presented as issuer-assigned.
+- **Cross-dataset validation** (`validate_dataset_bundle`) checks referential integrity,
+  currency/period/issuer coherence, invoice reconciliation (with a rounding tolerance),
+  Split Cost Allocation, and correction/lifecycle integrity — separately from the
+  per-dataset linter. It runs as a **publication gate** on every conversion (eager and
+  streaming): an ERROR refuses to publish, the outcome is recorded in the manifest's
+  `bundle_validation` section, and `--no-validate` skips it (the skip is recorded too).
+  In the streaming path the checks re-read the staged files in bounded memory, spilling
+  per-key lookup state to a scratch SQLite database past a threshold.
+- **Atomic writes**: output appears only after validation succeeds and checksums + manifest
+  are written; nothing partial is left on error. `--on-exists refuse|replace|version`.
+  A `replace` swap is **journaled**: if the process dies between its two renames, the next
+  publish to the same destination (or `focus-toolkit clean --out DIR`) reads the journal and
+  rolls the fully staged result forward — or the previous result back — so the destination
+  is never left missing. `clean` also removes orphan `.output.tmp-*` / `.trash-*` leftovers.
+
+```python
+from focus_data_toolkit import convert_to_focus_1_4, detect_focus_schema, validate_bundle
+
+detection = detect_focus_schema(headers)          # dataset, version, confidence, ...
+report = validate_bundle({"Cost and Usage": cu_rows, "Invoice Detail": invd_rows})
+print(report.ok, report.counts())                 # cross-dataset findings, JSON-serialisable
+```
+
+### Scale: streaming conversion and Parquet (0.3.0 / P1 Phase B)
+
+For large client files, `convert_files` streams the Cost and Usage file **once** and stages
+Invoice Detail aggregation / Billing Period dedup in a throwaway SQLite database, so peak
+memory stays **flat regardless of row count** — a constant ~64 MB peak process RSS whether
+converting 50k or 300k rows (6× the rows, ×1.05 the memory; `tools/benchmark_streaming.py`).
+Its output is **byte-identical** to the in-memory path — both call the same pure
+per-row/per-group functions and the same manifest assembler.
+
+Sources may be **CSV (gzip ok) or Parquet** — the format is sniffed per file (magic bytes,
+extension as fallback), so `--cost-and-usage export.parquet` works everywhere a CSV does
+(`convert`, `gaps`, `supplements validate`), in both the eager and streaming paths.
+
+```bash
+# bounded-memory streaming to CSV (recommended for large exports)
+focus-toolkit convert --cost-and-usage huge_cost_and_usage.csv.gz --out ./focus-1.4 \
+  --mode synthetic --stream
+
+# Parquet output with exact decimal128 (requires the [parquet] extra)
+focus-toolkit convert --cost-and-usage huge_cost_and_usage.csv --out ./focus-1.4 \
+  --mode synthetic --output-format parquet
+```
+
+Gzip input is auto-detected. Exactness contract: **CSV is byte-exact** (the literal), while
+**Parquet is decimal-value-exact** — decimal columns are written as `decimal128(precision,
+scale)` (never binary float; a value needing more scale than the column allows raises with its
+line number instead of rounding silently), dates as UTC timestamps, JSON/strings verbatim.
+
+**Partitioning & compression** (Parquet): `--partition-by` writes the Cost and Usage dataset as
+a Hive-partitioned tree on low-cardinality String / Date-Time columns; `--compression`
+(snappy default / zstd / gzip / none) and `--target-file-size` (roll to a new part file per
+partition) tune the layout:
+
+```bash
+focus-toolkit convert --cost-and-usage huge_cost_and_usage.csv --out ./focus-1.4 \
+  --mode synthetic --output-format parquet \
+  --partition-by BillingCurrency,BillingPeriodStart --compression zstd --target-file-size 128MB
+# -> synthetic_focus_1_4_cost_and_usage/BillingCurrency=USD/BillingPeriodStart=.../part-0.parquet
+```
+
+The partition columns are stored in the paths (standard Hive), so any `pyarrow.dataset` reader
+reconstructs full rows; a too-high-cardinality key is warned about and, past a hard cap,
+refused (nothing partial is published).
+
+```python
+from focus_data_toolkit import convert_files
+
+out = convert_files("huge_cost_and_usage.csv.gz", "./focus-1.4",
+                    mode="synthetic", output_format="parquet",
+                    partition_by=["BillingCurrency"])  # -> published Path
+```
+
+`pip install "focus-data-toolkit[parquet]"` for Parquet; streaming CSV needs no extra
+(the external state uses the standard-library `sqlite3`).
+
+### Synthetic scenarios (SCA, corrections, billing lifecycle)
+
+Deterministic, provider-agnostic scenario builders produce self-consistent data for tests and
+demos, and typed lifecycle checks validate the full snapshot chains — status transitions plus
+chain structure (id/order uniqueness, `previous_instance_id` resolution, cycle detection,
+`last_updated` monotonicity, closed-instance immutability; `FDT-LIFE-001..006`):
+
+```python
+from focus_data_toolkit.generators.scenarios import (
+    split_allocation_group, correction_set, billing_lifecycle_instances,
+)
+from focus_data_toolkit import check_dataset_instances, validate_bundle
+
+alloc = split_allocation_group("origin-1", "100.00", weights=[3, 2, 1])   # ratios sum to 1,
+validate_bundle({"Cost and Usage": alloc}).ok                             # costs sum to origin
+
+corr = correction_set("chg-1", "100.00", ["-30.00"])   # original + signed Correction line,
+validate_bundle({"Cost and Usage": corr}).ok           # running net recorded in x_NetCharge
+
+check_dataset_instances(billing_lifecycle_instances())  # [] — chains + transitions all valid
+```
+
+### Python API
+
+```python
+from focus_data_toolkit import convert_to_focus_1_4, lint_focus_1_4_structure
+from focus_data_toolkit.convert import read_csv_rows
+from focus_data_toolkit.modes import Mode
+
+result = convert_to_focus_1_4(read_csv_rows("focus_1_3_cost_and_usage.csv"))  # strict
+print(result.coverage)               # ('Cost and Usage',)
+print(result.not_produced)           # ('Contract Commitment', 'Billing Period', 'Invoice Detail')
+print(result.manifest["datasets"]["Invoice Detail"]["blocking_columns"])
+
+result = convert_to_focus_1_4(read_csv_rows("focus_1_3_cost_and_usage.csv"), mode=Mode.SYNTHETIC)
+rows = result.datasets["Invoice Detail"]
+print(lint_focus_1_4_structure("Invoice Detail", rows).levels_passed)
+```
+
+## Studio (local web UI)
+
+For a no-CLI experience, launch the **Studio** — a local web app over the same engine:
+
+```bash
+pip install "focus-data-toolkit[studio]"   # or [studio-all] to also read/write Parquet
+focus-toolkit ui                           # opens http://127.0.0.1:8765/?token=…
+```
+
+- **Local & private:** binds `127.0.0.1` by default, requires a fresh per-start **token** on every
+  request, validates `Host`/`Origin` (anti DNS-rebinding / CSRF), and confines file access to an
+  allowlisted `--root`. No telemetry, no external upload — data never leaves the machine.
+- **Workflow:** pick a file already under `--root` (best for large files), upload a small/medium
+  file (capped), or generate synthetic test data → **Detect** → **Convert** with live per-phase
+  progress and **Cancel** → preview a **sampled** page (the full file is never loaded) → download
+  datasets, the manifest, diagnostics (JSON/CSV), `SHA256SUMS`, and an HTML summary.
+- **Same Core:** every operation drives the same SDK the CLI and Runner use, so the Studio's
+  manifests, diagnostics and checksums are identical to a CLI run.
+- Exposing it on a network needs `--allow-remote` (the token is still required). Generation is
+  capped in the UI (use the CLI/Runner for very large synthetic sets). See
+  [docs/studio.md](docs/studio.md).
+
+## Runner (Docker / OCI)
+
+For production, automation and large volumes, the toolkit ships as a container image whose
+entrypoint **is** the `focus-toolkit` CLI — a container run is exactly a CLI run (same
+manifests, diagnostics, checksums and exit codes; no FOCUS logic is duplicated). It is
+**batch-only** (no HTTP server); status is the exit code, the logs, and the produced files.
+
+```bash
+docker run --rm \
+  --read-only --tmpfs /tmp \
+  -v "$PWD/input:/input:ro" \
+  -v "$PWD/output:/output" \
+  -v fdt-work:/work \
+  ghcr.io/guymano/focus-data-toolkit:0.10.0 \
+  convert --cost-and-usage /input/focus.csv \
+    --stream --output-format parquet --out /output/result \
+    --exit-policy pipeline
+```
+
+- **Non-root** (uid 65532) and **read-only-rootfs compatible** — only `/work` (scratch) and
+  `/output` are written. Because the atomic publish stages next to `--out`, **`/output` must be
+  writable** (mount a writable volume/dir; if you run as the image's non-root uid, ensure the
+  target is writable by it — or use a named volume, which Docker initialises writable for you).
+- `/input` is meant to be mounted **read-only**. `FOCUS_TOOLKIT_WORK_DIR=/work` and `TMPDIR=/work`
+  are preset, so all scratch stays on the `/work` volume — point it at fast local storage for big
+  files, and size it (the SQLite aggregation + bundle spill scale with invoice/period/commitment
+  cardinalities, not with the Cost-and-Usage row count).
+- **Signals**: `docker stop` sends SIGTERM to PID 1 (the CLI), which cancels cleanly — exit code
+  **130**, nothing partial published. Give it a grace period (`--stop-timeout`).
+- **Exit codes** for orchestrators: use `--exit-policy pipeline` so a functional-but-incomplete
+  strict run (3) or a synthetic run (4) reports success (0); genuine failures (1/2/5/130) stay
+  non-zero. Disk budgets: `FOCUS_TOOLKIT_MAX_WORK_BYTES`, `FOCUS_TOOLKIT_MIN_WORK_FREE_BYTES`,
+  `FOCUS_TOOLKIT_MIN_OUTPUT_FREE_BYTES` (a shortfall fails fast with `FDT-IO-005/006`, exit 5).
+
+**Single-node engine — pick the right method by scale:**
+
+| Volume | Recommended method |
+|---|---|
+| Tests / ordinary files | CLI (or the forthcoming Studio) |
+| Large files on one machine | Runner |
+| Hundreds of GB | Runner with fast local `/work` + sized resources; Parquet + partitioning + zstd |
+| Beyond a single node | Partition upstream / orchestrate multiple batches |
+
+The image is not a distributed engine. **Immutable** tags — `<version>` (e.g. `0.10.0`) and
+`sha-<full-commit>` — always identify the same bytes; the `<major>.<minor>` tag (e.g. `0.10`)
+is a **rolling** convenience alias that advances with each patch release. Every release is
+scanned (trivy) **before** its public tags are assigned, carries a CycloneDX SBOM, and is signed
+(cosign) and attested (build provenance). See [docs/runner.md](docs/runner.md) for details.
+
+## What is really migratable
+
+| 1.4 dataset | Real migration? | Notes |
+|---|---|---|
+| Cost and Usage | **Yes** | Column mapping (1.2 lifted to 1.3 shape; deprecated provider columns dropped; nullable 1.4 additions null; `PricingCurrency*` backfilled; `ContractApplied` re-cased 1.3→1.4). |
+| Contract Commitment | Partial → **synthetic** | The 13 source columns migrate; the 14 Mandatory 1.4 commercial terms are provider facts → assumed (synthetic only). |
+| Billing Period | **No** (synthetic) | Period keys derive from Cost and Usage; `BillingPeriodStatus` and the timestamps are provider billing-cycle state. |
+| Invoice Detail | **No** (synthetic) | `BilledCost` aggregates usage, but invoice status/terms/issuer-assigned id/timestamps are provider-issued. |
+
+Conversion is pure (no clock, no RNG): the same input always produces the same
+output bytes, in both modes.
+
+## Development
+
+```bash
+git clone https://github.com/guymano/focus-data-toolkit && cd focus-data-toolkit
+pip install -e .[dev]           # includes pyarrow, so the Parquet suite runs (not skipped)
+pytest -q                       # generators, detection, migration/lint, modes, manifest, CLI,
+                                # streaming, Parquet, split-allocation, corrections, lifecycle
+pytest -m slow                  # large-scale bounded-memory test (excluded by default)
+python tools/benchmark_streaming.py --rows 100000 500000   # throughput + peak RSS
+ruff check src tests
+```
+
+The FOCUS 1.4 model JSON (`src/focus_data_toolkit/model/focus_1_4_model.json`)
+is the artifact of record, extracted from the FinOps Foundation "FOCUS 1.4
+Data Model" workbook with `tools/extract_focus_1_4_model.py` (the workbook
+itself is not redistributed here — download it from
+[focus.finops.org](https://focus.finops.org)). Its machine-readable provenance
+is recorded in `model_provenance.json` and can be checked with
+`python scripts/verify_model_provenance.py`; see
+[docs/model-provenance.md](docs/model-provenance.md).
+
+## Contributing, security & docs
+
+- **Contributing:** [CONTRIBUTING.md](CONTRIBUTING.md)
+- **Security policy / reporting:** [SECURITY.md](SECURITY.md) (private reporting;
+  a FOCUS conformance bug is a normal issue, not a security report)
+- **Versioning & reproducibility:** [docs/versioning.md](docs/versioning.md)
+- **Compatibility (Python / OS / FOCUS):** [docs/compatibility.md](docs/compatibility.md)
+- **Security model:** [docs/security-model.md](docs/security-model.md)
+- **Releasing:** [docs/releasing.md](docs/releasing.md)
+- **Changelog:** [CHANGELOG.md](CHANGELOG.md)
+
+## License and credits
+
+The toolkit code is **MIT** (see [LICENSE](LICENSE)). The embedded FOCUS 1.4 data
+model is a derivative of the FinOps FOCUS specification / data-model workbook,
+which is © the FinOps Foundation and licensed **CC-BY-4.0**; it is redistributed
+here with attribution (see [NOTICE](NOTICE) and
+[docs/model-provenance.md](docs/model-provenance.md)). "FOCUS" and "FinOps" are
+trademarks of the FinOps Foundation; the FOCUS specification, data-model workbook
+and official validator are © the FinOps Foundation — this project is an
+independent community toolkit and is not endorsed by the FinOps Foundation.
+Related sample datasets from the same generators were contributed to
+[FOCUS-Sample-Data](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS-Sample-Data)
+(PRs #6 and #7).

focus_data_toolkit-0.11.0.dist-info/RECORD

@@ -0,0 +1,116 @@
+focus_data_toolkit/__init__.py,sha256=tyd4C3tdJJM7mNU4oeGSc7BaxNwADHnfhtneocDhyxc,2538
+focus_data_toolkit/__main__.py,sha256=BswQRTWsAPifuwP39nW8-evmiw2xKb1pB_uefWJp1-A,191
+focus_data_toolkit/_version.py,sha256=LKhp8tqxoUCYHl3H3Mx2I9TfIZCx2mdjc-v61UYEnLk,323
+focus_data_toolkit/cli.py,sha256=3PdZ8WUMuOfl3nqNzuyp6-4M19C_hDt3HLzTo_6MZU0,38677
+focus_data_toolkit/errors.py,sha256=iiKdOu3YnHODNHQHLaDahUuUIwP22CF7XWXypw3aaz4,5161
+focus_data_toolkit/focus_json.py,sha256=FBWGZzsVAZhjV3J7Q5sf0A5Ya7WKDfsDCe3aOMfrrjE,2806
+focus_data_toolkit/lifecycle.py,sha256=jPwSyJM7vsGjDYAXxpsu90_O_8QeR8NV1t2uJokU3A8,14557
+focus_data_toolkit/manifest.py,sha256=MNFrjmV7aZT5yPwDjvjC1Df8OVGMFPcVbG6Ve-YrvHs,4451
+focus_data_toolkit/modes.py,sha256=fqIsS-Hy4eHSr8g3QeuOeV4Ur5Rhfb-7m-4OitLrJrI,686
+focus_data_toolkit/official_validator.py,sha256=2-GqeucVazyzoe4pvLzNDl66uNIh0wbTBPLZJn27kIc,1987
+focus_data_toolkit/progress.py,sha256=5cxjh0bwEEna4GKFP8rZingZ1k1qBtp6NrCtu_eyTQI,3372
+focus_data_toolkit/provenance.py,sha256=C0wdde3uWPPBdyMlihwHU60IHoVI3b0ek_g8nd-oXf4,4122
+focus_data_toolkit/py.typed,sha256=6s6t40Ie33L-SxfLsfkB0w0Y0WUMke9BmGXW2GdQdIQ,68
+focus_data_toolkit/runtime.py,sha256=Dfse7ROzIJVW_jhl999ULx_f-QJtjHw9lBO3r_aSggs,9814
+focus_data_toolkit/context/__init__.py,sha256=7Bo3atX4ABmPVc5oyY5-KywxZ8mbXi7KhIBgrz_1Hs8,3019
+focus_data_toolkit/context/billing.py,sha256=HoCs4XtHbYp1lseO9oOyZ6V9Gib7p6Jhg_-xcz52FnM,1922
+focus_data_toolkit/context/provider.py,sha256=mdH4cByNwNx40Qxlz_MUCVuE-9bJRzbryMaY812oy18,3834
+focus_data_toolkit/convert/__init__.py,sha256=XePzx4VdbiMB_b-Wsb7b9_mxX4vAHLl-eBI7cchYQ88,30408
+focus_data_toolkit/convert/billing_period.py,sha256=BoJfbTY06se6Vm6Gz7uLBYXlK_88DBpY207Oa2l3S0A,2892
+focus_data_toolkit/convert/contract_applied.py,sha256=58vbVTYH4cSlZtWTT147-a4YHCnVMQvtW55XvTVNOuI,9787
+focus_data_toolkit/convert/contract_commitment.py,sha256=SXMgB9flO-xLApeIj_D_WsFEZNtCwqz1ntFt04fQf4c,8601
+focus_data_toolkit/convert/cost_and_usage.py,sha256=f3O1o8HmRLEeftyAOO_QKENKwPi1OAgpRk57UTd2B-E,7903
+focus_data_toolkit/convert/detect.py,sha256=XoePBgzCGI7AaJi1uroovBoLXZ3huL5_k-CH5Z4mQes,1444
+focus_data_toolkit/convert/invoice_detail.py,sha256=kmzcaeXKZmKfGAt956mO5wrAR9AQbuAwI3OOauNTFeg,8940
+focus_data_toolkit/convert/streaming.py,sha256=KLw48NAQlIYvDBfG0P_uWdraPxXm8cjdqFioRdtOCzA,44030
+focus_data_toolkit/generators/__init__.py,sha256=55H7mKoBdfkyRfjL0c3SCsvw46sZd3CsEBcPPJTvDTI,2591
+focus_data_toolkit/generators/_shim.py,sha256=FdgvnvgnwhPDlbURd6WxOsyvIl0kImEXaSPJrgI15EU,2064
+focus_data_toolkit/generators/generate_aws_focus_1_2.py,sha256=LpjkpGnvNGlV4-bhyHB4AGhbg6ac6SamrvJ5_oFHlW4,768
+focus_data_toolkit/generators/generate_aws_focus_1_3.py,sha256=0_6_7oNLkxE7MPZzCsrOFp3cOYirzGkE7wS-WS4v2bA,845
+focus_data_toolkit/generators/generate_azure_focus_1_2.py,sha256=OFYv6pZOI0qMPEWwhXK5Os1CERaDRAKActlXvtnW_yo,585
+focus_data_toolkit/generators/generate_azure_focus_1_3.py,sha256=vCgnDgRm_anv6G5Po07ntH-EwAAfo5uoyQguOlow5ko,585
+focus_data_toolkit/generators/generate_gcp_focus_1_2.py,sha256=EKIzHtiV283euQRF9pii5CnIUrhFhSCtV_nkZ3eUk28,574
+focus_data_toolkit/generators/generate_gcp_focus_1_3.py,sha256=Q0vw8dSxqs_YHL5DvlWHq_Wifsu-FVQCI70-We2O56Y,574
+focus_data_toolkit/generators/scenarios.py,sha256=HV7zGGb_qaXwD3ORtD25GWinqcABhMAt3AL6Xk0zxjA,7221
+focus_data_toolkit/generators/engine/__init__.py,sha256=KMJ_dIXf1IThG8-ell-9A1_qCwh8bcQC-8Ok8Nta6rI,849
+focus_data_toolkit/generators/engine/context.py,sha256=QSWR8AFmevrmzVmAO4xlrDD3TfjPntgBBYjYj4MLI3M,505
+focus_data_toolkit/generators/engine/determinism.py,sha256=xeiqtOxqpipoS8RjphWbg8q5iF7jIPmh_JCltFQG9EA,4413
+focus_data_toolkit/generators/engine/json_focus.py,sha256=o3j-GZr_V0XdR31eVhOxgdowNGRDoh9Xcld1Pj2YWKw,2567
+focus_data_toolkit/generators/engine/ladder.py,sha256=bC2M2Hp-_dqTU80C6Wzlq7HGENK1CtUwyNSVn8An5gE,2578
+focus_data_toolkit/generators/engine/scenarios_core.py,sha256=VXr0cFzZjoAAmzifnzOqtUPREsd2JL-gI4IQ5mV5UHc,16755
+focus_data_toolkit/generators/engine/serialize.py,sha256=cW8x_VjYYsFlFcHFDBF4nRvWOBpdmnW0bhGsBvZ6sqg,5974
+focus_data_toolkit/generators/providers/__init__.py,sha256=0WEjH3NpHK0hFl11weshUPSH9ADQPE2-YE8oM7mSKnI,892
+focus_data_toolkit/generators/providers/aws.py,sha256=hoyObMVUo2L00NUAIIJq_Fz-zepYa6rPlDGqH2TY7Cw,7360
+focus_data_toolkit/generators/providers/azure.py,sha256=v1HHmepy10et7KCrqyY_nVctXLegtG6Wg0U7M6CxicU,7920
+focus_data_toolkit/generators/providers/gcp.py,sha256=PbHbRuU1GMBTmi76Gf6_GzanyQ37cVTed-_46TPVNq4,7761
+focus_data_toolkit/generators/providers/profile.py,sha256=3KUiNzTAM206xXfBiTsk9s3VqSZjANmOiTMya3MZrfU,4967
+focus_data_toolkit/generators/versions/__init__.py,sha256=tciA-qFbVzVbCKGhBK5chbL7ntfQi6m81lHY_-JMr7Y,798
+focus_data_toolkit/generators/versions/adapter.py,sha256=HkfjLVUNA0f5XOXqq0egtIsUvJsAOOptmE8q3NFU3YU,1687
+focus_data_toolkit/generators/versions/v1_2.py,sha256=QxWqN_cTF44JegCSHdergSyEHBh1QKNPmJpgLxxO-WE,3063
+focus_data_toolkit/generators/versions/v1_3.py,sha256=w7Qirn1qQUfT_F83ns3Cipuing1HwI3lFI0oG10RmvI,4644
+focus_data_toolkit/io/__init__.py,sha256=49aHPWaJ0PN9lYyxYX5pjpmc7UdurqaiSl1mzgG-xvk,51
+focus_data_toolkit/io/atomic_writer.py,sha256=e_eKWMe25Ydwuyil93UPIakJiS1bhn-9d29suxPGi80,20471
+focus_data_toolkit/io/csv_io.py,sha256=w9qOok9uf6ZlICOCfO2j3hAuVo8xS2aZSil5-l3bvU4,5002
+focus_data_toolkit/io/parquet_io.py,sha256=E7BzbG4Kv4WKujMwOkq-GpzA5ZAD0zNh4x_Wi8pTZPw,22587
+focus_data_toolkit/io/records.py,sha256=R78VfSqYHr2JPTso2ucUHe36AB5uS9ekbu9xJLdm95Y,2831
+focus_data_toolkit/io/row_source.py,sha256=NTs2J4X945kJ2ezTP25_pWZ0QG2yuz_cAQCKOtJrhxI,4507
+focus_data_toolkit/model/__init__.py,sha256=ezdCoMnn1-Z0ucqHJWuC-G818gBgqrd_zDEn_s6BwD8,1196
+focus_data_toolkit/model/capabilities.py,sha256=zyNrXs5JNkSyf5oM8z8GNjdtVPKkbNVxGDBydLY3n7A,2576
+focus_data_toolkit/model/focus_1_4_decimal_scale.json,sha256=d06EzrRi-nfl_sq5zMOsoWb_R0Y5uHFnU3lQnfAIpL0,893
+focus_data_toolkit/model/focus_1_4_model.json,sha256=iwBuGja8lnj-9rechHZU-RTqer4lyCdHiqRO4dFbIUs,64971
+focus_data_toolkit/model/focus_1_4_servicesubcategory.json,sha256=hTJGcJO-3R4iioLwfHgtiykobeddCnm8y_iW3r2Gf2Q,3244
+focus_data_toolkit/model/focus_json_keys.py,sha256=5wBp_FmlJoyz7NpkSShbtuT87E9UKm48O28ZWNR8OEk,4164
+focus_data_toolkit/model/iso_4217_currencies.json,sha256=ARakJ2kZJZvcKaqbDUnRpD0XXnhx-39TMefa6GR7us4,1446
+focus_data_toolkit/model/json_schema_check.py,sha256=XCN7VBaHEVyPa8Wf0T6R6F8UD4hSKSiEYjMaYwkNr-w,8953
+focus_data_toolkit/model/model_provenance.json,sha256=t4rHSZqcl15BgxOPjgdpZBw2WiJMeGDzEac_MX92hFQ,3848
+focus_data_toolkit/model/validator.py,sha256=ttquxO8jJ5MYxpIGF6Wybn-WILNg5KTXevB5272NZYU,20534
+focus_data_toolkit/model/json_schemas/allocatedmethoddetailsobjectschema.json,sha256=D1UkaZVEM8BKUa0YVh5LheE3h1cM2kpMxaczo5IU2P0,3250
+focus_data_toolkit/model/json_schemas/commitmentprogrameligibilitydetailsobjectschema.json,sha256=DML8S_w6cgdUWRwBeliZ0hT3jWMBZ6FumcKHXLkd5XY,1647
+focus_data_toolkit/model/json_schemas/contractappliedobjectschema.json,sha256=Jq04D4nBK0FPNoIvtPEnhuTqz7DQCsR4GoR1JIOl1z4,4352
+focus_data_toolkit/model/json_schemas/contractcommitmentapplicabilityobjectschema.json,sha256=bDlqpawMz-0b-xxGTWaYaEUgVUrKUNyjJYMH_3eaVls,7143
+focus_data_toolkit/model/json_schemas/json_schemas_provenance.json,sha256=oS7ns_LE2jTYbMsh-7w_6Bxd_EfXdIQ8sWFILi95auY,1778
+focus_data_toolkit/schema/__init__.py,sha256=2zLyGC0WIDk1EJnBpWs3AnULjUbUUl77qumIDfeCCHk,718
+focus_data_toolkit/schema/detection.py,sha256=oEhQsRJvNNOz7KFbtrDFDY9g8p1UvYLrGSjF6vfNxOE,10715
+focus_data_toolkit/schema/registry.py,sha256=W4CHm6HavH_m8uh1Mv9UYDvhbUhh_ymYMehiSO2Ak3w,5223
+focus_data_toolkit/storage/__init__.py,sha256=fq4abXU1niE3iASKD6YX1dv313ogXkaXxtuQd6lEWjU,80
+focus_data_toolkit/storage/external_index.py,sha256=J1gSMJosSdLXuBnA0zb4yREt7v047lLHD-iAPC7ipp4,4090
+focus_data_toolkit/storage/spill.py,sha256=E5ZannDqio3T0yaxwCa7kowXsqwONsV-_u6Bx-wzyQU,5435
+focus_data_toolkit/studio/__init__.py,sha256=RA2hPijhhWjczdTyb-r0e9yFQDFeJMANlG3XoKi460E,958
+focus_data_toolkit/studio/app.py,sha256=lU1m-UT_IOIRkxzOGLgFUWuuWwEquy_XGl9WPSQ0-v4,20472
+focus_data_toolkit/studio/config.py,sha256=Adcslwc6TMWY0SGd2AvtavQMft_8__-s_XZezTgCk7Y,1654
+focus_data_toolkit/studio/jobs.py,sha256=ALf2L_WCMAXnvwplCrpDEtE6F7HjfWg4dHrMUoj-LWw,5527
+focus_data_toolkit/studio/preview.py,sha256=aNar6QMey_7oV7TPgMLDLWHvp7R9EFw9GYGk7_mL6eE,1229
+focus_data_toolkit/studio/security.py,sha256=5LUReMjwdXwyF-eP9p2SZtjxrm9Ieta905yFD-FVXng,5457
+focus_data_toolkit/studio/server.py,sha256=WkAb5FBudFg3Wh0HktDjd_eaFccA0Gm25BhBCIBaP0o,2211
+focus_data_toolkit/studio/frontend/app.js,sha256=zioD0JAY5RE30WNIPUx3sW5Yq0mA2FsLM_9fIToVjL4,9731
+focus_data_toolkit/studio/frontend/index.html,sha256=9YtZuY9RAL67hj6wlkQlx4HD-l2iMgLR3STtgI95sjg,3606
+focus_data_toolkit/studio/frontend/style.css,sha256=KLfzs335aTfs7-xkx_cT9rAr66NjaQ6hhlRIOpDYHE8,3906
+focus_data_toolkit/supplement/__init__.py,sha256=Py15wzVq6jVt2Avd88dcUvoxNpbmfr38B2tzUoZ0bNU,1845
+focus_data_toolkit/supplement/apply.py,sha256=gQPrSeC29ApSvTZrFVQdvJEjURON6kxry3TcP45cVCQ,12981
+focus_data_toolkit/supplement/gaps.py,sha256=Lov9yU5OEKgNKwPVyBvgqYQ4AVviAYZqKKGlMWpmh44,9502
+focus_data_toolkit/supplement/kinds.py,sha256=LePJW3QZmqvPYDP2fhNDFX0MDr1ymVGb_Jk0ooKDsbM,3986
+focus_data_toolkit/supplement/loader.py,sha256=og8UVY_sGL2G1dJ5R5vviFoQYkZVG4V0r1F6ZW-5_1U,16650
+focus_data_toolkit/supplement/spec.py,sha256=hEjVMSKNM_8Z7j07S2z7PsQF89jAOJY3zSZCXT65OmU,2959
+focus_data_toolkit/supplement/validate.py,sha256=XqiZ3CGShcR-KfxpVhM3OSHQELDOilbQ9gVzOKz2CX4,8977
+focus_data_toolkit/supplement/adapters/__init__.py,sha256=VGo1WZSo595WbdyBH9nDycJTf3cX54dXy2hGxe_rK7M,428
+focus_data_toolkit/supplement/adapters/adapters_provenance.json,sha256=Ddu1sYYkSqB55X2G-_DoMfwpdyG3p-mrdFhNz2KVsqk,1965
+focus_data_toolkit/supplement/adapters/aws_invoice_summary.json,sha256=9p0xkUaU6c_X-ew746_yc9I0Ck-DaChdhsMctpY-4K4,1385
+focus_data_toolkit/supplement/adapters/aws_savings_plans.json,sha256=uvCZuhd1NnLFK6UE2d7vl4wnnEoWjIvnKwVQGoeN9vs,2390
+focus_data_toolkit/supplement/adapters/azure_invoice.json,sha256=NTm-up2EyrWuf6j34sNfsy08Mz6FoB_4h3SK1qR4DoU,1852
+focus_data_toolkit/supplement/adapters/gcp_compute_commitments.json,sha256=itwVsiIRTvb5DeygWq1EgBi1aipyFMQBOp42A5uzSsM,2179
+focus_data_toolkit/supplement/adapters/registry.py,sha256=xJfewTrtsutS4ilCxn2hRC0UjzvgXfKpcL4wgscFbIQ,8240
+focus_data_toolkit/validate/__init__.py,sha256=q26nUoQXAp3ikNbyvE9wNPpsif4Mk5i5ZOMHGPec9O8,399
+focus_data_toolkit/validate/allocation.py,sha256=bMAOwo-d_iuJnCaohSV7IYmbxtXJ7_31sYp79Q6YoSM,13211
+focus_data_toolkit/validate/bundle.py,sha256=OI-MaIKUp1EHqJBS9aTcGmkn7YRZTXnBCeJMg4rfGcQ,9906
+focus_data_toolkit/validate/codes.py,sha256=EUukxWGTopAouuQBdq551qY5v8Kwxzp-ZiyUv7UiLF4,5762
+focus_data_toolkit/validate/corrections.py,sha256=klLSgorVh7ST9mSbPkLCIVmmRtRpa4qli1CzOAk-nok,10164
+focus_data_toolkit/validate/reconciliation.py,sha256=qNBEObAtQG_VLCAm0JF9AnYAZlEdI_D5fZdbM6ZvrkQ,4119
+focus_data_toolkit/validate/referential.py,sha256=dO_uXy6mIwLfxPnwsFKSE7fRiBw-K55XK_De-KzoXsc,11472
+focus_data_toolkit-0.11.0.dist-info/licenses/LICENSE,sha256=NtTLMNmquvok1ABHks0U0ap9Kvz9PWA0JLg5PtQUMi4,1088
+focus_data_toolkit-0.11.0.dist-info/licenses/NOTICE,sha256=J51RSKahdwtYmrhwgKyGK_mZjaJc04zTjguO5TJUgSg,2841
+focus_data_toolkit-0.11.0.dist-info/licenses/LICENSES/CC-BY-4.0.txt,sha256=1VdTnfaOdxzB7tzJHRP3D8qTDlCNEe7cr6SxXbSeN0Q,17023
+focus_data_toolkit-0.11.0.dist-info/METADATA,sha256=meeu3PZZ_w98Vo79s8PBBKUdfVL5oQanEFwID35NtqY,27002
+focus_data_toolkit-0.11.0.dist-info/WHEEL,sha256=wUyA8OaulRlbfwMtmQsvNngGrxQHAvkKcvRmdizlJi0,92
+focus_data_toolkit-0.11.0.dist-info/entry_points.txt,sha256=MiGCjfdjj1I73tNzREaD1X28p59PJ-nkboal8xXQ6hg,62
+focus_data_toolkit-0.11.0.dist-info/top_level.txt,sha256=tLPrj44kcwFwJfhRB-fXgP-z_V4e6sPBmCwRBtf7rEw,19
+focus_data_toolkit-0.11.0.dist-info/RECORD,,

focus_data_toolkit-0.11.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (80.10.2)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

focus_data_toolkit-0.11.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+focus-toolkit = focus_data_toolkit.cli:main