focus-data-toolkit 0.13.0__tar.gz → 0.14.0__tar.gz

{focus_data_toolkit-0.13.0 → focus_data_toolkit-0.14.0}/CHANGELOG.md

@@ -9,6 +9,174 @@ policy.
 
 ## [Unreleased]
 
+## [0.14.0] — 2026-10-06
+
+FOCUS 1.2/1.3 to 1.4 conversion follows the specification more closely, and commitment
+terms and provider-native commitment inventories can complete the 1.4 datasets.
+
+- **Breaking:**
+  - the `aws-savings-plans` adapter (v2) is keyed by `savingsPlanArn`;
+  - a unit-price column the source does not carry is omitted from the 1.4 Cost and Usage
+    output instead of being written null.
+- **New byte baselines for converted output:**
+  - conversion manifests label migrated columns `DERIVED`;
+  - `ContractCommitmentDurationType` is written in whole years (`1 Year`, `3 Years`).
+
+  Generated samples are unchanged.
+- **Stricter lint:** output that passed with 0.13.0 can now be refused; see the upgrade
+  notes under Changed.
+
+### Added
+
+- The supplement API and the documented failures are now public, re-exported from the
+  package root: `SupplementBundle`, `SupplementFileSpec`, `load_bundle_dir`,
+  `compute_gaps`, `GapReport`, `Mode`, `ConversionError`, `SupplementError` and
+  `AtomicWriteError`. Library callers no longer import internal modules, which
+  docs/versioning.md leaves free to change. Existing import paths keep working.
+- `ContractCommitmentDurationType` is a `contract_commitment` supplement column. A
+  supplied term wins over the derived value. It must follow the format v1.4 recommends, a
+  positive whole number and a listed unit (`1 Year`, `3 Years`, `36 Months`); anything
+  else is refused (`FDT-SUPP-004`). That format is a SHOULD in the specification, so the
+  lint of a 1.4 file does not fail on it.
+- `fdt gaps` reports `ContractCommitmentDurationType` as a conditional advisory: it is
+  derived only for periods spanning whole calendar months, so the other terms must be
+  supplied. Without a commitment period in the source it is a blocking gap.
+- A supplement that almost matches a provider adapter now names what it lacks, e.g.
+  `aws-savings-plans@2 requires savingsPlanArn` for an export from the v1 adapter era.
+- Azure commitment adapters `azure-reservation-orders` and `azure-savings-plan-orders`
+  (REST shape, api-version 2022-11-01). One row per order, keyed by the lowercased order
+  ARM id. Each provides the term (`1 Year`, `3 Years`, `5 Years`, or `1 Month` for a
+  monthly savings plan), the payment model and interval, the lifecycle status where the
+  provisioning state determines one, the creation time where the response carries it,
+  and the benefit category. The commitment model is
+  not emitted: the order does not carry the commitment grain. A savings plan order is
+  recognised by its `sku.name`; a new `none_of` detection clause rules out the plan- and
+  reservation-level lists, which share it. A `lower` adapter transform supports
+  case-insensitive provider identifiers.
+- A FOCUS 1.2 source can declare who issued it (`provider_role="csp"` or `"msp"`, CLI
+  `--provider-role`), with the cloud provider's own publisher names
+  (`first_party_publishers`, CLI `--first-party-publisher`).
+  - With `csp`, a row whose `PublisherName` differs from `ProviderName` and is no declared
+    first-party name is a Marketplace charge. Its `ServiceProviderName` and
+    `HostProviderName` become the seller, as FOCUS 1.4 requires (`FDT-MIG-005`).
+  - With `msp`, they stay `ProviderName`, the MSP.
+  - Without a declaration, `ProviderName` is kept as before, and those rows are reported
+    (`FDT-CTX-005`). Nothing is refused.
+  - Only official FOCUS columns are read. See
+    [conversion rules](docs/conversion-rules.md#participating-entities-of-a-12-source),
+    including why `HostProviderName` = `ServiceProviderName` is an interpretation for a 1.2
+    source.
+
+### Changed
+
+- **Breaking:** the `aws-savings-plans` adapter (now v2) is keyed by `savingsPlanArn`,
+  the fully qualified identifier FOCUS recommends and the one Contract Commitment and
+  `CommitmentDiscountId` carry. v1 keyed by the bare `savingsPlanId`, so its rows could
+  never join. Exports without `savingsPlanArn` are no longer auto-detected; re-export
+  with the ARN (as `describe-savings-plans` returns it). The adapter also maps
+  `termDurationInSeconds` (1 and 3 years) to the duration.
+- **New byte baseline:** `ContractCommitmentDurationType` is written in whole years when
+  the term is a multiple of 12 months (`1 Year`, `3 Years`), as in the specification's
+  examples, instead of `12 Months` / `36 Months`. Other terms stay in months.
+- The FOCUS 1.4 linter now enforces more of the static Cost and Usage rules of the v1.4
+  requirements model. Output that passed before can now fail the lint, and strict
+  publication refuses it:
+  - Tax and Credit `EffectiveCost` equals `BilledCost` (CAU-EffectiveCost-C-017);
+  - `ListCost` and `ContractedCost` equal unit price × `PricingQuantity` when both are
+    present, Correction rows included (C-011). The relative tolerance is the official
+    focus-validator's (1e-9 × max(|cost|, 1)), and the arithmetic is exact whatever the
+    caller's Decimal context. An operand whose exponent lies beyond ±1000 is reported as
+    not computable instead of being computed;
+  - a unit is null exactly when its quantity is (`PricingUnit`, `ConsumedUnit`,
+    `CommitmentDiscountUnit`, C-005/C-006);
+  - `ListUnitPrice` and `ContractedUnitPrice` are present when `SkuPriceId` is set
+    (C-013/C-015). This applies in every profile, as these rules have no applicability
+    criteria, except for a column the source did not carry.
+    `lint_focus_1_4_structure(source_absent_columns=...)` lets a producer name such
+    columns that it writes all null, and a column missing from the rows counts as absent;
+  - under a declared `SupportsUnitPricing` condition, the pricing and quantity columns
+    are null when `SkuPriceId` is null (C-009 to C-016).
+
+  Other static rules of the model are not checked. These include the pricing-currency
+  unit prices' C-012, whose presence is not visible on a row and whose model condition
+  for `PricingCurrencyContractedUnitPrice` contradicts its text.
+  **Upgrade note:** expect refusals for:
+  - provider costs rounded to the cent (exact identity), and 1.2 Correction rows, which
+    1.2 exempted from the identity;
+  - Credit rows whose `EffectiveCost` differs from `BilledCost` (C-017), which the
+    converter copies as is (Tax rows are migrated, `FDT-MIG-001`);
+  - rows with a `SkuPriceId` but a null `ListUnitPrice` or `ContractedUnitPrice` in a
+    column the source carries (C-013/C-015).
+- **Breaking (output layout):** a unit-price column the source does not carry
+  (`ListUnitPrice`, `ContractedUnitPrice`, `PricingCurrencyListUnitPrice`,
+  `PricingCurrencyContractedUnitPrice`) is no longer written all null into the 1.4 Cost
+  and Usage output; it is omitted, as Invoice Detail already omits its unfilled
+  conditional columns. Its presence condition is not met, and written null it would break
+  the "MUST NOT be null when SkuPriceId is not null" rules in any validator, including
+  `focus-toolkit validate` on the published file. A column counts as carried when any
+  source row carries it. Sources that carry all four columns, such as the generated
+  samples, are unchanged.
+- The client-like test fixture violated the cost identity by a factor of 1000; its
+  quantities are corrected. Its Tax row no longer carries a pricing quantity, and its
+  pricing-currency effective cost now matches its effective cost.
+- **New byte baseline for conversion manifests.** On sources that can trigger those
+  migrations, `EffectiveCost` and the `SkuPriceId`-dependent columns are now labelled
+  `DERIVED` at column level. This follows the "weakest lineage the rule can produce"
+  convention, and `lineage_summary` records the per-value mix. Converted values of the
+  generated samples are unchanged.
+- `ConversionError` and `ConversionCancelled` are defined in `focus_data_toolkit.convert.exceptions`
+  and re-exported unchanged from `focus_data_toolkit.convert`.
+
+### Fixed
+
+- `focus-toolkit convert --stream` printed each diagnostic as `note None:`; it now prints
+  its code, as the eager path does.
+- `ContractCommitmentDurationType` is no longer rounded from `days / 30.44`, which turned
+  any span into a plausible-looking term. It is derived (`DERIVED`) only when the
+  commitment period spans a whole number of calendar months, the day clamped at month
+  end. Otherwise strict mode leaves it empty (`UNAVAILABLE`): Contract Commitment is not
+  produced until the term is supplied, and the other datasets are unaffected. Synthetic
+  mode assumes the nearest whole-month value (`ASSUMED`). `FDT-CC-001` lists the
+  commitments concerned, at the same point in the eager and streaming pipelines.
+- The duration's lineage is settled per row. A partial supplement no longer blanks, or
+  counts as supplied, the terms derived for the rows it does not cover.
+- A naive commitment timestamp is read as UTC, as FOCUS requires, instead of failing to
+  compare with an offset one; months are counted in the start's frame.
+- 1.2/1.3 Cost and Usage values that break a FOCUS 1.4 rule the converter can meet without
+  inventing a fact are migrated, instead of being copied into non-conformant output. Each
+  migration is counted and reported, and each migrated value counts as `DERIVED` in
+  `lineage_summary`:
+  - Tax `EffectiveCost` equals `BilledCost` (CAU-EffectiveCost-C-017), with the exact net
+    change per billing currency (`FDT-MIG-001`). Credit rows, also covered by C-017, are not
+    rewritten;
+  - pricing and quantity columns and their units are nulled where `SkuPriceId` is null, on
+    Tax, Credit, Adjustment and Correction rows (`FDT-MIG-002`). For a 1.2 source this
+    migrates values 1.2 allowed; a 1.3 source already broke its own version's rule;
+  - on Usage/Purchase rows that are not corrections, FOCUS 1.4 requires the same columns to
+    be non-null, so their values are kept and the conflict is reported (`FDT-MIG-004`); so
+    are rows whose `ChargeCategory` is missing or not an allowed value;
+  - null source pricing-currency values are backfilled with a warning (`FDT-MIG-003`).
+- A null `PricingCurrencyEffectiveCost` is no longer backfilled from `EffectiveCost` when
+  `PricingCurrency` differs from `BillingCurrency`, or when `BillingCurrency` is null. The
+  copy labelled a billing-currency amount with another, or an unknown, currency.
+  **Upgrade note:** a multi-currency source with such nulls used to convert; it now stops
+  with `FDT-MIG-011` (CLI exit 2). A Tax restatement across currencies stops with
+  `FDT-MIG-010`. See [conversion rules](docs/conversion-rules.md).
+
+### Security
+
+- The Runner image upgrades `perl-base` to 5.36.0-7+deb12u4 (bookworm-security). The
+  pinned base digest ships 5.36.0-7+deb12u3, which the container scan flags with three
+  CRITICAL (CVE-2026-13221, CVE-2026-42496, CVE-2026-8376) and four HIGH
+  (CVE-2026-42497, CVE-2026-48962, CVE-2026-57432, CVE-2026-57433) vulnerabilities.
+- The Runner image no longer ships the `libpcre2-8-0` 10.42-1 package flagged HIGH by
+  the container scan (CVE-2026-103111, CVE-2026-86145, CVE-2026-89157, CVE-2026-89161).
+  The base image digest (`python:3.12-slim-bookworm`) is refreshed, bringing
+  10.42-1+deb12u1. The runtime stage then installs the pinned bookworm-security fix
+  10.42-1+deb12u2, which no base digest ships yet.
+- Lock `urllib3` 2.8.0 (PYSEC-2026-4175, -4176, -4177). It reaches the lock only
+  transitively, through the optional `validator` extra (`focus-validator` -> `requests`)
+  and the release tooling (`twine`); the core toolkit has no runtime dependency.
 ## [0.13.0] — 2026-09-05
 
 Prepared in source; publishing is a separate release step.
@@ -716,7 +884,8 @@ conformance defects.
 
 <!-- Reference links. 0.2.0/0.3.0 were pre-release development milestones and were never tagged
      or published, so only the first public release (0.9.0) has a tag link. -->
-[Unreleased]: https://github.com/guymano/focus-data-toolkit/compare/v0.12.0...HEAD
+[Unreleased]: https://github.com/guymano/focus-data-toolkit/compare/v0.14.0...HEAD
+[0.14.0]: https://github.com/guymano/focus-data-toolkit/compare/v0.13.0...v0.14.0
 [0.13.0]: https://github.com/guymano/focus-data-toolkit/compare/v0.12.0...v0.13.0
 [0.12.0]: https://github.com/guymano/focus-data-toolkit/compare/v0.11.0...v0.12.0
 [0.11.0]: https://github.com/guymano/focus-data-toolkit/compare/v0.11.0rc1...v0.11.0

{focus_data_toolkit-0.13.0 → focus_data_toolkit-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: focus-data-toolkit
-Version: 0.13.0
+Version: 0.14.0
 Summary: Generate provider-realistic FOCUS 1.2/1.3 sample data (AWS, Azure, GCP) and convert it to the four FOCUS 1.4 datasets — strictly from source facts, completed by client supplements including native AWS/Azure/GCP exports — with schema detection, gap analysis, a cross-dataset validation gate and atomic writes.
 Author: Guy-Hermann Adiko
 License-Expression: MIT AND CC-BY-4.0
@@ -46,7 +46,7 @@ Requires-Dist: focus-data-toolkit[parquet,release,studio]; extra == "dev"
 Requires-Dist: pytest>=8; extra == "dev"
 Requires-Dist: pytest-cov<8,>=5; extra == "dev"
 Requires-Dist: ruff>=0.6; extra == "dev"
-Requires-Dist: mypy<2.4,>=1.11; extra == "dev"
+Requires-Dist: mypy<2.5,>=1.11; extra == "dev"
 Requires-Dist: jsonschema<5,>=4.21; extra == "dev"
 Requires-Dist: httpx<1,>=0.27; extra == "dev"
 Dynamic: license-file
@@ -63,10 +63,16 @@ Generate realistic **FOCUS 1.2 / 1.3** cost and usage data (AWS, Azure, GCP), co
 
 [FOCUS](https://focus.finops.org) is the open standard for cloud cost and usage data. FOCUS 1.4 defines four datasets: Cost and Usage, Contract Commitment, Billing Period, and Invoice Detail. The toolkit is honest by design: any value it cannot derive from your source is left empty (**strict** mode) or filled with clearly labelled assumptions (**synthetic** mode).
 
-The source prepares **0.13.0** with corrected subscriptions, traceable taxes,
-stable offers and partially used fleets of 500 machines. See the
-[generator corrections and validation evidence](docs/generator-corrections.md)
-for the new byte baseline and sample assumptions. The Docker examples below
+The source prepares **0.14.0**, which converts FOCUS 1.2/1.3 sources toward 1.4 more
+faithfully:
+- values are migrated to the 1.4 rules, and a stricter 1.4 lint checks the output;
+- commitment terms can be supplied explicitly;
+- Azure reservation and savings plan orders have adapters;
+- a 1.2 source can declare who issued it, to name its Service Provider.
+
+See the [conversion rules](docs/conversion-rules.md) and the [changelog](CHANGELOG.md).
+The [generator corrections and validation evidence](docs/generator-corrections.md)
+describe the generated samples, unchanged since 0.13.0. The Docker examples below
 continue to pin the previously released 0.12.0 image.
 
 ## Three ways to use it
@@ -118,9 +124,31 @@ from focus_data_toolkit import convert_files
 convert_files("cost_and_usage.csv", "focus-1.4", mode="strict")
 ```
 
+Everything a library caller needs is importable from the package root, including
+supplements and the failures to handle:
+
+```python
+from pathlib import Path
+
+from focus_data_toolkit import (
+    AtomicWriteError, ConversionError, Mode, SupplementBundle, SupplementError,
+    SupplementFileSpec, convert_files,
+)
+
+try:
+    bundle = SupplementBundle.load([SupplementFileSpec(path=Path("invoices.json"))])
+    convert_files("cost_and_usage.csv", "focus-1.4", mode=Mode.STRICT, supplements=bundle)
+except (ConversionError, SupplementError):
+    ...  # invalid source or unusable supplement: nothing was written
+except AtomicWriteError:
+    ...  # the lint or cross-dataset gate refused publication: nothing was written
+```
+
 ## What it does, and what it doesn't
 
-- Converts FOCUS 1.2/1.3 Cost and Usage into the four FOCUS 1.4 datasets.
+- Converts FOCUS 1.2/1.3 Cost and Usage into the four FOCUS 1.4 datasets. Values a 1.4 rule
+  fixes deterministically are migrated and reported (`FDT-MIG-*`). Values it cannot fix without
+  inventing a fact are refused. See [docs/conversion-rules.md](docs/conversion-rules.md).
 - The three other datasets come only from [supplements](docs/supplements.md) or synthetic mode.
   There is no FOCUS 1.4 generator and no invalid-data generator.
 - Built for real data: bounded-memory streaming (`--stream`), atomic writes, a deterministic

{focus_data_toolkit-0.13.0 → focus_data_toolkit-0.14.0}/README.md

@@ -10,10 +10,16 @@ Generate realistic **FOCUS 1.2 / 1.3** cost and usage data (AWS, Azure, GCP), co
 
 [FOCUS](https://focus.finops.org) is the open standard for cloud cost and usage data. FOCUS 1.4 defines four datasets: Cost and Usage, Contract Commitment, Billing Period, and Invoice Detail. The toolkit is honest by design: any value it cannot derive from your source is left empty (**strict** mode) or filled with clearly labelled assumptions (**synthetic** mode).
 
-The source prepares **0.13.0** with corrected subscriptions, traceable taxes,
-stable offers and partially used fleets of 500 machines. See the
-[generator corrections and validation evidence](docs/generator-corrections.md)
-for the new byte baseline and sample assumptions. The Docker examples below
+The source prepares **0.14.0**, which converts FOCUS 1.2/1.3 sources toward 1.4 more
+faithfully:
+- values are migrated to the 1.4 rules, and a stricter 1.4 lint checks the output;
+- commitment terms can be supplied explicitly;
+- Azure reservation and savings plan orders have adapters;
+- a 1.2 source can declare who issued it, to name its Service Provider.
+
+See the [conversion rules](docs/conversion-rules.md) and the [changelog](CHANGELOG.md).
+The [generator corrections and validation evidence](docs/generator-corrections.md)
+describe the generated samples, unchanged since 0.13.0. The Docker examples below
 continue to pin the previously released 0.12.0 image.
 
 ## Three ways to use it
@@ -65,9 +71,31 @@ from focus_data_toolkit import convert_files
 convert_files("cost_and_usage.csv", "focus-1.4", mode="strict")
 ```
 
+Everything a library caller needs is importable from the package root, including
+supplements and the failures to handle:
+
+```python
+from pathlib import Path
+
+from focus_data_toolkit import (
+    AtomicWriteError, ConversionError, Mode, SupplementBundle, SupplementError,
+    SupplementFileSpec, convert_files,
+)
+
+try:
+    bundle = SupplementBundle.load([SupplementFileSpec(path=Path("invoices.json"))])
+    convert_files("cost_and_usage.csv", "focus-1.4", mode=Mode.STRICT, supplements=bundle)
+except (ConversionError, SupplementError):
+    ...  # invalid source or unusable supplement: nothing was written
+except AtomicWriteError:
+    ...  # the lint or cross-dataset gate refused publication: nothing was written
+```
+
 ## What it does, and what it doesn't
 
-- Converts FOCUS 1.2/1.3 Cost and Usage into the four FOCUS 1.4 datasets.
+- Converts FOCUS 1.2/1.3 Cost and Usage into the four FOCUS 1.4 datasets. Values a 1.4 rule
+  fixes deterministically are migrated and reported (`FDT-MIG-*`). Values it cannot fix without
+  inventing a fact are refused. See [docs/conversion-rules.md](docs/conversion-rules.md).
 - The three other datasets come only from [supplements](docs/supplements.md) or synthetic mode.
   There is no FOCUS 1.4 generator and no invalid-data generator.
 - Built for real data: bounded-memory streaming (`--stream`), atomic writes, a deterministic

focus_data_toolkit-0.14.0/docs/conversion-rules.md

@@ -0,0 +1,176 @@
+# FOCUS 1.2/1.3 to 1.4 conversion rules
+
+This page states what the converter does to each Cost and Usage value. It also lists the 1.4 rules
+it cannot meet from a 1.2 or 1.3 source. Every rule cites the FOCUS specification
+([FOCUS_Spec](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec), tags `v1.2`
+`7296e31`, `v1.3` `c67b95b`, `v1.4` `f1eeb30`) and, where one exists, the rule identifier of the
+v1.4 requirements model (`specification/requirements_model/releases/1.4`).
+
+## Copied, renamed and dropped columns
+
+- Every 1.4 column present in the source is copied verbatim. Allowed values did not change between
+  1.2 and 1.4 for any enumerated column, so enum values are never rewritten.
+- `ProviderName` and `PublisherName` (deprecated in 1.3, removed in 1.4) are dropped. For a 1.2
+  source, `ServiceProviderName` and `HostProviderName` are derived as described in
+  [Participating entities of a 1.2 source](#participating-entities-of-a-12-source).
+- Columns new in 1.3 or 1.4 that the source does not carry are emitted null, for example
+  `CommitmentProgramEligibilityDetails` and `InvoiceDetailId`. The manifest records them as
+  `UNAVAILABLE`.
+- A unit-price column the source does not carry is **omitted** instead: `ListUnitPrice`,
+  `ContractedUnitPrice`, `PricingCurrencyListUnitPrice` and
+  `PricingCurrencyContractedUnitPrice`.
+  - These conditional columns MUST NOT be null when `SkuPriceId` is set
+    (CAU-ListUnitPrice-C-013, CAU-ContractedUnitPrice-C-015, and C-012 of both pricing-currency
+    unit prices).
+  - A source that does not carry one has not met its presence condition, and an all-null
+    column would fail those rules in any validator.
+  - A column counts as carried when any source row carries it.
+  - The manifest lists no rule for an omitted column.
+
+## Participating entities of a 1.2 source
+
+FOCUS 1.2 has `ProviderName`, `PublisherName` and `InvoiceIssuerName`. FOCUS 1.3 introduced
+`ServiceProviderName` "as a replacement for ProviderName", added `HostProviderName`, and
+deprecated `ProviderName` and `PublisherName`. FOCUS 1.4 removes those two; there
+`ServiceProviderName` is Mandatory and not nullable, and `HostProviderName` is Mandatory and
+nullable. Only these official columns are read; provider-specific `x_` columns are not.
+
+**What the specification says.**
+
+- FOCUS 1.2 appendix *Origination of Cost Data*: when `PublisherName` differs from
+  `ProviderName`, the row is either a cloud marketplace purchase (Provider = the cloud provider,
+  Publisher = the seller; scenarios 3.1 to 3.3) or cloud services bought through an MSP
+  (Provider = the MSP, Publisher = the cloud provider; scenario 2.1).
+- FOCUS 1.4 `ServiceProviderName`: "In marketplace scenarios, the Service Provider represents the
+  seller rather than the marketplace operator"; a reseller of white-labeled services is the
+  Service Provider. The appendix *Participating Entity Identification* gives the Marketplace
+  Seller in scenarios 3.1.1 to 3.3.2 and the MSP in scenarios 2.1 and 2.2.
+- FOCUS 1.4 `HostProviderName` MUST reflect the host when the customer selected it or the
+  service provider exposes it, and MUST match `ServiceProviderName` in all other cases.
+
+**What the converter does.** A 1.2 row cannot tell the marketplace case from the MSP case, so the
+caller declares who issued the file (`provider_role`, CLI `--provider-role`):
+
+| Declaration | Row whose `PublisherName` differs from `ProviderName` | Code |
+|---|---|---|
+| none (default) | `ServiceProviderName` = `ProviderName`, as before; the rows are counted | `FDT-CTX-005` (warning) |
+| `csp`: a cloud provider issued the file | Marketplace charge: `ServiceProviderName` = `PublisherName` (the seller) | `FDT-MIG-005` (info) |
+| `msp`: an MSP or reseller issued the file | `ServiceProviderName` = `ProviderName` (the MSP) | none |
+
+- **First-party publishers.** A cloud provider may spell itself differently in the two columns (for
+  example `ProviderName` "Microsoft Azure" and `PublisherName` "Microsoft"). With `csp`, such names
+  are declared as first-party publishers (`first_party_publishers`, CLI
+  `--first-party-publisher`, case-insensitive), and their rows are not treated as Marketplace
+  charges. The comparison with `ProviderName` is case-insensitive as well.
+- **Nothing is refused.** Without a declaration, strict mode still produces the dataset; the
+  warning names the publishers concerned.
+- **1.3 sources.** They carry `ServiceProviderName` themselves; the declaration does not apply.
+- **Manifest.** The declaration is recorded in the `ServiceProviderName` provenance note.
+
+**`HostProviderName` is an interpretation.** A 1.2 source has no host column, so it never exposes
+the host, and `HostProviderName` takes the `ServiceProviderName` value. The specification does
+not settle this for a 1.2 source, and the true 1.4 value can differ in two cases:
+
+- a Marketplace offering that runs on the cloud provider's infrastructure, or that exposes its
+  host, has the cloud provider as host (appendix 3.1.x and 3.2.x), where the converter writes the
+  seller (3.3.x, without visibility);
+- in an MSP file, the 1.2 `PublisherName` is the cloud provider, which could be read as exposing
+  the host (appendix 2.1), where the converter writes the MSP (2.2). The publisher is never taken
+  as the host.
+
+## Values migrated to a 1.4 rule
+
+These changes are deterministic. Each one is counted and reported as a warning in the manifest.
+The columns concerned are labelled `DERIVED`, the weakest lineage the rule can produce. In the
+per-value `lineage_summary`, a migrated value counts as `DERIVED` and a value copied unchanged
+(including those kept under `FDT-MIG-004`) as `OBSERVED`.
+
+| Code | Rule | What changes |
+|---|---|---|
+| `FDT-MIG-001` | CAU-EffectiveCost-C-017: `EffectiveCost` MUST equal `BilledCost` when `ChargeCategory` is "Tax" or "Credit". In 1.3 a tax's effective cost followed the related charges' effective cost (rule C-009, removed in 1.4). | On **Tax** rows, `EffectiveCost` takes the `BilledCost` value. A non-null `PricingCurrencyEffectiveCost` follows it when `PricingCurrency` equals `BillingCurrency`. The diagnostic gives the row count and the net `EffectiveCost` change per billing currency (exact Decimal), so a reconciliation against the source can explain the difference. **Credit** rows are not rewritten. 1.2 and 1.3 tied the equivalent rule to whether a charge relates to other charges (Credit was only an example), which a row does not reveal; a credit that breaks C-017 is copied as is. |
+| `FDT-MIG-002` | The pricing and quantity columns MUST be null when `SkuPriceId` is null. The columns concerned: `ListUnitPrice` (C-012), `ContractedUnitPrice` (C-014), both pricing-currency unit prices (C-011), `PricingCategory` (C-012), `PricingQuantity` (C-011), `ConsumedQuantity` (C-009) and `CommitmentDiscountQuantity` (C-016). 1.3 already states these rules, so they migrate 1.2 sources; 1.2 only said these values "MAY be null". | When the source carries a `SkuPriceId` column and the value is null, those columns are nulled on Tax, Credit and Adjustment rows and on any Correction row. Each unit follows its quantity (`PricingUnit`, `ConsumedUnit`, `CommitmentDiscountUnit`, rules C-005/C-006). A source without a `SkuPriceId` column is left untouched: the column's absence does not mean a null SKU price. |
+| `FDT-MIG-004` | On a Usage or Purchase row that is not a correction, the same columns MUST NOT be null (for example `ListUnitPrice` C-005, `PricingQuantity` C-005, `ConsumedQuantity` C-006, `PricingCategory` C-004). With a null `SkuPriceId`, FOCUS 1.4 requires them both null and non-null. A row whose `ChargeCategory` is missing or not an allowed value cannot tell which rule applies. | The values are **kept** on every row the `FDT-MIG-002` nulling does not cover, because real consumption is never discarded. The rows are counted per charge category (`(null)` when missing). The source is at fault: a Usage or Purchase row should carry a `SkuPriceId`, and every row a valid `ChargeCategory`. |
+| `FDT-MIG-003` | `PricingCurrency` and `PricingCurrencyEffectiveCost` MUST NOT be null. 1.2 already required this; only its constraint table said otherwise, which erratum #9/#10 later corrected. | Null values in a source column that is present are backfilled from `BillingCurrency` and `EffectiveCost`. When the source omits the columns entirely, the same backfill applies without a warning, because the provider does not price in another currency. |
+
+## Values the converter refuses to invent
+
+Conversion stops with a `ConversionError` and publishes nothing:
+
+- `FDT-MIG-010`: a Tax row's `EffectiveCost` must change (`FDT-MIG-001`), but its
+  `PricingCurrency` differs from `BillingCurrency`. Restating `PricingCurrencyEffectiveCost` would
+  need an exchange rate the source does not carry.
+- `FDT-MIG-011`: `PricingCurrencyEffectiveCost` is null, and `PricingCurrency` is set but
+  differs from `BillingCurrency` or `BillingCurrency` is null. Copying `EffectiveCost`, an amount
+  in the billing currency, would label it with another or an unknown currency.
+- A `ContractApplied` value that does not fit the 1.4 object schema: an element carrying both
+  cost and quantity, or a custom key without the `x_` prefix. The pre-erratum 1.3 key casing is
+  normalized instead, with the warning `FDT-CA-001`.
+
+Every refusal is a `ConversionError`, and the CLI exits with code 2. The `FDT-MIG-010` and
+`FDT-MIG-011` messages start with the code and name the first offending row.
+
+## 1.4 rules the linter checks on the output
+
+Besides the column formats, the linter enforces these static rules of the v1.4 requirements model
+on every Cost and Usage row. Strict publication (`write_result`, `convert_files`) refuses output
+that breaks them.
+
+| Rule (v1.4 model) | Lint rule | When |
+|---|---|---|
+| CAU-EffectiveCost-C-017 | `effective_cost_differs_from_billed` | `ChargeCategory` is "Tax" or "Credit" |
+| CAU-ListCost-C-011, CAU-ContractedCost-C-011 | `cost_not_unit_price_times_quantity`, `cost_identity_not_computable` | unit price and `PricingQuantity` both present, Correction rows included |
+| CAU-PricingUnit/ConsumedUnit/CommitmentDiscountUnit-C-005/006 | `unit_without_quantity`, `quantity_without_unit` | always |
+| CAU-ListUnitPrice-C-013, CAU-ContractedUnitPrice-C-015 | `required_with_sku_price_id` | `SkuPriceId` is set and the column is present |
+| C-009 to C-016 ("MUST be null when `SkuPriceId` is null") | `must_be_null_without_sku_price_id` | the caller declares the `SupportsUnitPricing` condition |
+
+- **Cost identity.** The tolerance is the official focus-validator's:
+  `|unit price × quantity − cost| ≤ 1e-9 × max(|cost|, 1)`. The arithmetic is exact whatever the
+  caller's Decimal context. An operand whose exponent lies beyond ±1000 is reported as not
+  computable rather than computed.
+- **Unit prices.** C-013 and C-015 carry no applicability criteria in the model, so they apply
+  in every profile. A unit-price column the source lacks is omitted from the output (see above)
+  and is therefore not evaluated.
+- **Not enforced.** The two pricing-currency unit prices' C-012 is not enforced. Their presence
+  depends on conditions a row does not show. In addition, the model's condition for
+  `CAU-PricingCurrencyContractedUnitPrice-C-012-C` contradicts its own text. That defect is
+  reported for 1.3 in
+  [FOCUS_Spec#2598](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec/issues/2598) and
+  is still present in the 1.4 model.
+- **Contract Commitment.** `ContractCommitmentDurationType` follows the 1.4 recommended format
+  ("[positive integer] [unit]", e.g. `1 Year`) when the converter derives it or a supplement
+  supplies it. That format is a SHOULD (CCT-ContractCommitmentDurationType-C-004-O/C-005-O), so
+  the lint does not refuse another form in a 1.4 file it did not produce.
+
+## 1.4 rules the converter cannot verify or meet
+
+- **Meaning shifts that the row data cannot reveal.**
+  - `ChargeClass` "Correction" now refers to a previously *closed* billing period, where 1.3 said
+    *invoiced*.
+  - Rebates tied to usage thresholds may be "Adjustment" rather than "Credit" in 1.4.
+  - A Purchase covering related, postpaid charges must have `EffectiveCost` 0.
+
+  Values are copied as they are.
+- **Credit `EffectiveCost`.** CAU-EffectiveCost-C-017 requires `EffectiveCost` to equal
+  `BilledCost` on Credit rows as well as Tax rows. 1.2 and 1.3 tied the equivalent rule to
+  whether a charge relates to other charges, which a row does not reveal, so a Credit row is
+  copied as is (see `FDT-MIG-001`) and may break C-017.
+- **Virtual currencies.** 1.4 CurrencyFormat requires ISO 4217 codes, while the `PricingCurrency`
+  definition and the v1.4 example data still use virtual currencies such as "Token". The linter
+  applies CurrencyFormat, so a virtual pricing currency fails the lint. This is a conflict inside
+  the specification, not a converter choice.
+- **A source without a `SkuPriceId` column.** The output still carries the 1.4 `SkuPriceId`
+  column, emitted null. A copied price or quantity beside it breaks the "MUST be null when
+  `SkuPriceId` is null" rules (C-009 to C-016). The converter cannot tell a provider without SKU
+  prices from a missing column, so this is a known non-conformance of such outputs.
+- **Amounts with an exponent beyond ±1000.** They are not money, and exact bookkeeping on them
+  could need billions of digits. A Tax row carrying one is copied as is, like an unparseable
+  amount, instead of being migrated by `FDT-MIG-001`.
+- **Columns that cannot be derived.** `CommitmentProgramEligibilityDetails`, `InvoiceDetailId`,
+  and an `InvoiceId` the source omits are emitted null. `InvoiceDetailId` can come from an
+  `invoice_line` supplement.
+- **Contract Commitment.** The 1.4 dataset adds 13 non-null mandatory columns that a 1.3 Contract
+  Commitment dataset does not carry. In strict mode they come only from
+  [supplements](supplements.md).
+- **Dataset-level obligations.** Dataset completeness, delivery and configuration attributes
+  (`CorrectionHandling`, `DeliveryHandling`, `DatasetCompleteness`) concern how a provider
+  publishes data, not individual rows.

{focus_data_toolkit-0.13.0 → focus_data_toolkit-0.14.0}/docs/supplements.md

@@ -34,7 +34,7 @@ FOCUS 1.x columns. No supplement can fabricate those — fix the export instead.
 | `billing_period` | Billing Period | `InvoiceIssuerName`, `BillingPeriodStart`, `BillingPeriodEnd` | `BillingPeriodCreated`, `BillingPeriodLastUpdated`, `BillingPeriodStatus` |
 | `invoice` | Invoice Detail (header level) | `InvoiceIssuerName`, `InvoiceId` | `InvoiceIssueDate`, `InvoiceIssueStatus`, `PaymentTerms`, `PaymentDueDate`, `ReferenceInvoiceId`, `PurchaseOrderNumber`, `PaymentCurrency` |
 | `invoice_line` | Invoice Detail (line level) | the full business grain (`InvoiceIssuerName`, `InvoiceId`, `BillingAccountId`, `BillingCurrency`, `BillingPeriodStart`, `BillingPeriodEnd`, `ChargeCategory`) | `InvoiceDetailId`, `InvoiceDetailCreated`, `InvoiceDetailLastUpdated`, `InvoiceDetailDescription`, `InvoiceDetailGrain`, `PaymentCurrencyBilledCost`, `PaymentCurrencyInvoiceDetailId` (+ `BilledCost`, accepted only as a reconciliation check) |
-| `contract_commitment` | Contract Commitment | `ContractCommitmentId` | the 1.4-new commercial terms (`ContractCommitmentCreated/LastUpdated`, `ContractCommitmentApplicability`, `…BenefitCategory`, `…FulfillmentInterval`, `…LifecycleStatus`, `…Model`, `…OfferCategory`, `…PaymentInterval`, `…PaymentModel`, `…PaymentUpfrontPercentage`, `…DiscountPercentage`) plus optional `ServiceProviderName` / `InvoiceIssuerName` overrides |
+| `contract_commitment` | Contract Commitment | `ContractCommitmentId` | the 1.4-new commercial terms (`ContractCommitmentCreated/LastUpdated`, `ContractCommitmentApplicability`, `…BenefitCategory`, `…FulfillmentInterval`, `…LifecycleStatus`, `…Model`, `…OfferCategory`, `…PaymentInterval`, `…PaymentModel`, `…PaymentUpfrontPercentage`, `…DiscountPercentage`), the explicit term `ContractCommitmentDurationType` (`1 Year`, `3 Years`, `36 Months`: a positive whole number and a v1.4 unit; it wins over the value derived from a period spanning whole calendar months, and is required in strict mode when the period does not), plus optional `ServiceProviderName` / `InvoiceIssuerName` overrides |
 
 Rules that apply to every kind:
 
@@ -96,7 +96,7 @@ $ focus-toolkit supplements adapters
 aws-invoice-summary (v1) -> invoice
     source: AWS Invoicing API — InvoiceSummary (ListInvoiceSummaries)
     doc:    https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_invoicing_InvoiceSummary.html
-aws-savings-plans (v1) -> contract_commitment
+aws-savings-plans (v2) -> contract_commitment
 azure-invoice (v1) -> invoice
 gcp-compute-commitments (v1) -> contract_commitment
     ...
@@ -107,10 +107,32 @@ Shipped adapters:
 | Adapter | Provider export | Target kind |
 |---|---|---|
 | `aws-invoice-summary` | AWS Invoicing API `InvoiceSummary` (`aws invoicing list-invoice-summaries`) | `invoice` |
-| `aws-savings-plans` | AWS Savings Plans inventory (`aws savingsplans describe-savings-plans`) | `contract_commitment` |
+| `aws-savings-plans` | AWS Savings Plans inventory (`aws savingsplans describe-savings-plans`), keyed by `savingsPlanArn` (v2), so it joins a Contract Commitment keyed by the fully qualified ARN | `contract_commitment` |
 | `azure-invoice` | Azure Billing Invoices REST API (`az billing invoice list`) | `invoice` |
+| `azure-reservation-orders` | Azure reservation orders, REST shape (`az rest --method get --url "/providers/Microsoft.Capacity/reservationOrders?api-version=2022-11-01"`), one row per order keyed by its ARM id **lowercased** | `contract_commitment` |
+| `azure-savings-plan-orders` | Azure savings plan orders, REST shape (`az rest --method get --url "/providers/Microsoft.BillingBenefits/savingsPlanOrders?api-version=2022-11-01"`), one row per order keyed by its ARM id **lowercased** | `contract_commitment` |
 | `gcp-compute-commitments` | GCP Compute Engine commitments (`gcloud compute commitments list --format=json`) | `contract_commitment` |
 
+The Azure order adapters key each commitment by the **order**, the purchase that carries
+the term, billing plan and price; splits and merges stay inside it. The ARM id is
+lowercased because Azure ids are case-insensitive and Microsoft's APIs and exports mix
+casings. A Contract Commitment source must use the same key. They emit the term
+(`ContractCommitmentDurationType`: `1 Year`, `3 Years`, `5 Years`, or `1 Month` for a monthly
+savings plan term), the payment model and interval, the lifecycle status, the creation time
+(`createdDateTime` for a reservation order, ARM `systemData.createdAt` for a savings plan
+order when the response carries it) and the benefit category (`Discount`). The lifecycle status
+comes from `provisioningState`, without a clock: `Expired`, `Cancelled` and the creation and
+billing states map; `Succeeded` does not, because finished provisioning cannot tell a
+running term from an exhausted pre-purchase pool. They leave out what an order does not
+determine: the commitment model and fulfillment interval (the commitment grain belongs to
+the plan or the reserved resource), the offer category, `LastUpdated`, applicability, the
+discount percentage and the invoice issuer. Supply those in a `contract_commitment` file
+keyed the same way; it merges with the adapter's columns.
+
+The two list operations are paged: each response holds one page in `value` and a
+`nextLink` to the next. Export every page, then pass the pages concatenated into one
+`value` array, or each page as its own supplement file (files of one kind are merged).
+
 Then just pass the export straight to `convert` / `supplements validate`:
 
 ```console
@@ -137,6 +159,8 @@ value is auditable back to the native export. Honesty rules:
   applicability scope) are **not** emitted — the coverage report shows the residual gap and
   you supply those separately.
 - An export that matches no adapter falls back to the generic FOCUS-named path with a clear
-  message. Nothing is ever guessed.
+  message. Nothing is ever guessed. When it almost matches one (typically an export made
+  for an older adapter version), the error names the adapter and the fields it lacks, e.g.
+  `aws-savings-plans@2 requires savingsPlanArn`.
 - Account-type / format variants are distinct versioned tables; an unrecognized variant
   falls back rather than half-matching.

{focus_data_toolkit-0.13.0 → focus_data_toolkit-0.14.0}/docs/versioning.md

@@ -78,6 +78,15 @@ Affected monetary Parquet columns widen to `decimal128(38,17)` (21 integer digit
 values exceeding the precision or scale fail rather than rounding. Existing CSV
 columns remain unchanged. See [the migration evidence](generator-corrections.md).
 
+The 0.14.0 release changes converted output in documented ways:
+- values migrated to FOCUS 1.4 rules, with `DERIVED` manifest labels;
+- `ContractCommitmentDurationType` in whole years;
+- a unit-price column the source lacks is omitted, a breaking change to the output layout;
+- the `aws-savings-plans` adapter is keyed by `savingsPlanArn`, also breaking.
+
+Generated sample bytes are unchanged. See [CHANGELOG.md](../CHANGELOG.md) and the
+[conversion rules](conversion-rules.md).
+
 The toolkit's own version is independent of the FOCUS specification versions it
 supports (generate 1.2/1.3, convert toward 1.4, detect 1.2/1.3/1.4). The FOCUS
 version support matrix and the FOCUS compatibility policy live in

{focus_data_toolkit-0.13.0 → focus_data_toolkit-0.14.0}/pyproject.toml

@@ -72,7 +72,7 @@ dev = [
     "pytest>=8",
     "pytest-cov>=5,<8",
     "ruff>=0.6",
-    "mypy>=1.11,<2.4",
+    "mypy>=1.11,<2.5",
     # Validates the FOCUS model-provenance JSON Schema in CI (test_model_provenance). Dev/CI only;
     # the runtime core and the stdlib provenance verifier do not require it.
     "jsonschema>=4.21,<5",