policyengine 4.16.0__tar.gz → 4.16.2__tar.gz

{policyengine-4.16.0 → policyengine-4.16.2}/.github/copilot-instructions.md

@@ -10,5 +10,5 @@ from branches in `PolicyEngine/policyengine.py`; never create fork PRs.
 For repository-wide API, testing, documentation, release, or package-boundary
 changes, read `docs/engineering/skills/repository-guidance.md`.
 
-For certified release-bundle updates or reviews, read
-`docs/engineering/skills/release-bundles.md`.
+For data-release certification or reviews, read
+`docs/engineering/skills/data-certification.md`.

{policyengine-4.16.0 → policyengine-4.16.2}/AGENTS.md

@@ -15,8 +15,8 @@ Before making or reviewing repository-wide API, testing, documentation, release,
 or package-boundary changes, read
 `docs/engineering/skills/repository-guidance.md`.
 
-Before updating or reviewing certified release bundles, read
-`docs/engineering/skills/release-bundles.md`.
+Before certifying or reviewing a country data release, read
+`docs/engineering/skills/data-certification.md`.
 
 ## Repository Boundaries
 

{policyengine-4.16.0 → policyengine-4.16.2}/CHANGELOG.md

@@ -1,3 +1,17 @@
+## [4.16.2] - 2026-06-12
+
+### Changed
+
+- Certify country data releases directly from their Hugging Face release manifests (`scripts/certify_data_release.py`), replacing the policyengine-bundles import flow. The vendored US manifest is regenerated through the new path (byte-identical apart from the certification source strings), and the vendored bundle archive copies are no longer shipped.
+
+
+## [4.16.1] - 2026-06-12
+
+### Changed
+
+- Import PolicyEngine bundle 4.16.0: the `populace-data` release manifest (`populace_us_2024` built from primary sources) replaces `policyengine-us-data` as the certified US data release, with policyengine-us 1.723.0 and policyengine-core 3.27.1. Dataset references and region datasets now honor per-artifact repo pins so inherited datasets keep resolving from their original repos.
+
+
 ## [4.16.0] - 2026-06-11
 
 ### Added

{policyengine-4.16.0 → policyengine-4.16.2}/CLAUDE.md

@@ -20,5 +20,5 @@ Before making or reviewing repository-wide API, testing, documentation, release,
 or package-boundary changes, read
 `docs/engineering/skills/repository-guidance.md`.
 
-Before updating or reviewing certified release bundles, read
-`docs/engineering/skills/release-bundles.md`.
+Before certifying or reviewing a country data release, read
+`docs/engineering/skills/data-certification.md`.

{policyengine-4.16.0/src/policyengine.egg-info → policyengine-4.16.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: policyengine
-Version: 4.16.0
+Version: 4.16.2
 Summary: A package to conduct policy analysis using PolicyEngine tax-benefit models.
 Author-email: PolicyEngine <hello@policyengine.org>
 License:                     GNU AFFERO GENERAL PUBLIC LICENSE
@@ -692,8 +692,8 @@ Provides-Extra: uk
 Requires-Dist: policyengine_core>=3.26.1; extra == "uk"
 Requires-Dist: policyengine-uk==2.88.20; extra == "uk"
 Provides-Extra: us
-Requires-Dist: policyengine_core==3.26.1; extra == "us"
-Requires-Dist: policyengine-us==1.722.4; extra == "us"
+Requires-Dist: policyengine_core>=3.27.1; extra == "us"
+Requires-Dist: policyengine-us==1.723.0; extra == "us"
 Provides-Extra: dev
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: furo; extra == "dev"
@@ -705,9 +705,9 @@ Requires-Dist: build; extra == "dev"
 Requires-Dist: plotly>=5.0.0; extra == "dev"
 Requires-Dist: pytest-asyncio>=0.26.0; extra == "dev"
 Requires-Dist: ruff>=0.9.0; extra == "dev"
-Requires-Dist: policyengine_core==3.26.1; extra == "dev"
+Requires-Dist: policyengine_core>=3.27.1; extra == "dev"
 Requires-Dist: policyengine-uk==2.88.20; extra == "dev"
-Requires-Dist: policyengine-us==1.722.4; extra == "dev"
+Requires-Dist: policyengine-us==1.723.0; extra == "dev"
 Requires-Dist: towncrier>=24.8.0; extra == "dev"
 Requires-Dist: mypy>=1.11.0; extra == "dev"
 Requires-Dist: pytest-cov>=5.0.0; extra == "dev"

{policyengine-4.16.0 → policyengine-4.16.2}/docs/engineering/skills/README.md

@@ -13,5 +13,5 @@ Current skills:
   changelog-fragment requirements, PR head verification, and title conventions.
 - `repository-guidance.md`: policyengine.py structure, commands, package
   boundaries, test expectations, and repo-specific anti-patterns.
-- `release-bundles.md`: certified release-bundle refresh workflow, model/data
-  compatibility gates, expected files, and bundle-specific tests.
+- `data-certification.md`: certifying country data releases from their
+  release manifests, validation semantics, expected files, and legacy paths.

policyengine-4.16.2/docs/engineering/skills/data-certification.md

@@ -0,0 +1,83 @@
+# Data certification
+
+Use this skill when certifying a country data release for `policyengine.py`,
+or reviewing a certification change.
+
+## What certification is
+
+A data release is published on Hugging Face with a release manifest: per-
+artifact repo/revision/sha256 pins, build provenance, compatibility claims,
+and region dataset templates. Certification asserts that *this*
+`policyengine.py` release, with the model package pinned in
+`pyproject.toml`, serves that data release — and the assertion is only made
+good by the test suite passing on the exact pinned pair.
+
+There is no intermediate bundle repo. The vendored country manifest at
+`src/policyengine/data/release_manifests/{country}.json` is derived directly
+from the data release manifest.
+
+## Certifying a release
+
+Open the work on a fresh branch from current `main` (use a clean worktree if
+the checkout is dirty).
+
+```bash
+python scripts/certify_data_release.py --country us \
+  --manifest-uri "hf://dataset/policyengine/populace-us@<tag>/releases/<tag>/release_manifest.json"
+```
+
+The script fetches and validates the manifest (every artifact must carry a
+revision pin; the certified dataset must be reachable), writes the vendored
+country manifest, exact-pins the country model package and raises the core
+floor in `pyproject.toml`, regenerates the TRACE TRO sidecar, and writes a
+Towncrier changelog fragment.
+
+Private data (UK) requires `HUGGING_FACE_TOKEN` or `HF_TOKEN`.
+
+After running:
+
+- `uv lock` if pins moved, then `uv sync --all-extras`,
+- run the full test suite — snapshot drift from a model bump is refreshed
+  with `PE_UPDATE_SNAPSHOTS=1 pytest tests/test_household_calculator_snapshot.py`,
+- commit the manifest, TRO, `pyproject.toml`, `uv.lock`, and fragment
+  together.
+
+A certification PR should normally change only:
+
+- `src/policyengine/data/release_manifests/{country}.json` (+ `.trace.tro.jsonld`)
+- `pyproject.toml` / `uv.lock`
+- one Towncrier fragment under `changelog.d/`
+- test constants/snapshots that pin certified versions
+
+## Validation semantics
+
+Hard failures (certification refuses): missing national default dataset,
+default dataset absent from artifacts, any artifact without a revision pin,
+unreachable certified dataset, unknown country.
+
+Certification gate: the model version must either exactly match the
+build-time model (`compatibility_basis: built_with_model_package`) or be
+covered by the publisher's `compatible_model_packages` claim
+(`compatibility_basis: compatible_model_packages`, recorded with a
+warning — the publisher's claim is made good only by this repo's test
+suite passing on the pinned pair). Neither basis means certification is
+refused: a new data build or a published compatibility claim is required.
+
+Warnings (recorded, not blocking): artifacts without sha256, and the
+publisher-claim basis above.
+
+## Legacy paths
+
+Countries whose current data release predates release manifests (the UK
+enhanced FRS) are refreshed with the legacy single-country tool until their
+next release certifies through a manifest:
+
+```bash
+python scripts/refresh_release_bundle.py --country uk --model-version 2.89.0
+```
+
+Do not hand-edit vendored country manifests for normal updates.
+
+The retired `policyengine-bundles` flow (candidates → generated bundle →
+archive import) is preserved read-only in that repo's history; bundles
+4.15.x–4.16.x remain the historical record of earlier certifications.

{policyengine-4.16.0 → policyengine-4.16.2}/docs/release-bundles.md

@@ -1,5 +1,15 @@
 # Release Bundles
 
+> **Current process.** Certification now runs inside this repository:
+> `scripts/certify_data_release.py` derives the vendored country manifest
+> directly from a country's data release manifest (see the
+> [data certification](engineering/skills/data-certification.md)
+> engineering skill). The intermediate `policyengine-bundles` repository
+> flow is retired; its published bundles remain the historical record of
+> earlier certifications. The architecture below — country `*-data`
+> repos publish immutable manifests, `policyengine.py` certifies — is
+> unchanged.
+
 This document defines the intended reproducibility boundary for `policyengine.py`.
 
 The key design decision is:
@@ -59,18 +69,19 @@ It does not define the final supported runtime bundle exposed to users.
 
 It does not rebuild microdata artifacts.
 
-New multi-country bundles are generated and archived in
-`PolicyEngine/policyengine-bundles`. `policyengine.py` vendors one current
-bundle from that archive under `src/policyengine/data/bundle/`, then generates
-the legacy country release manifests that runtime code still reads. The import
+Certification runs in this repository: the vendored country release
+manifest under `src/policyengine/data/release_manifests/` is derived
+directly from the country's published data release manifest. The
 entrypoint is:
 
 ```bash
-python scripts/import_policyengine_bundle.py 4.14.0
+python scripts/certify_data_release.py --country us \
+  --manifest-uri "hf://dataset/policyengine/populace-us@<tag>/releases/<tag>/release_manifest.json"
 ```
 
-The `policyengine-bundles` publish workflow runs this importer automatically
-when it opens the consuming `.py` PR.
+Earlier releases (policyengine 4.15.x–4.16.x) were certified through the
+`PolicyEngine/policyengine-bundles` archive flow; those bundles remain the
+historical record of their certifications.
 
 ## Two manifest layers
 
@@ -396,11 +407,16 @@ Things that should usually not affect the fingerprint:
 `policyengine.py` may certify a staged data artifact for a model version only if one of the following is true:
 
 1. the model version exactly matches the `built_with_model_package.version`
-2. the model version has the same `data_build_fingerprint` as the build-time model version
-
-If neither is true, the bundle release must fail and a new data build is required.
-
-This should be a hard failure, not a warning.
+   (`compatibility_basis: built_with_model_package`)
+2. the data publisher's `compatible_model_packages` covers the model version
+   (`compatibility_basis: compatible_model_packages` — the publisher's claim,
+   recorded with a warning and made good by the test suite passing on the
+   pinned pair)
+
+If neither is true, certification fails hard and a new data build or a
+published compatibility claim is required. A `data_build_fingerprint`
+basis (same fingerprint as the build-time model) is a planned third
+basis; it requires fingerprint metadata from the installed model package.
 
 ## Artifact states
 

{policyengine-4.16.0 → policyengine-4.16.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "policyengine"
-version = "4.16.0"
+version = "4.16.2"
 description = "A package to conduct policy analysis using PolicyEngine tax-benefit models."
 readme = "README.md"
 authors = [
@@ -48,8 +48,8 @@ uk = [
     "policyengine-uk==2.88.20",
 ]
 us = [
-    "policyengine_core==3.26.1",
-    "policyengine-us==1.722.4",
+    "policyengine_core>=3.27.1",
+    "policyengine-us==1.723.0",
 ]
 dev = [
     "pytest",
@@ -62,9 +62,9 @@ dev = [
     "plotly>=5.0.0",
     "pytest-asyncio>=0.26.0",
     "ruff>=0.9.0",
-    "policyengine_core==3.26.1",
+    "policyengine_core>=3.27.1",
     "policyengine-uk==2.88.20",
-    "policyengine-us==1.722.4",
+    "policyengine-us==1.723.0",
     "towncrier>=24.8.0",
     "mypy>=1.11.0",
     "pytest-cov>=5.0.0",

policyengine-4.16.2/scripts/certify_data_release.py

@@ -0,0 +1,111 @@
+"""Certify a country data release from its HF release manifest.
+
+Replaces the policyengine-bundles import flow: fetches the data release
+manifest, validates it, writes the vendored country manifest, exact-pins
+the country model package in ``pyproject.toml``, regenerates the TRACE
+TRO sidecar, and writes a Towncrier changelog fragment.
+
+Usage::
+
+    python scripts/certify_data_release.py --country us \\
+        --manifest-uri "hf://dataset/policyengine/populace-us@<tag>/releases/<tag>/release_manifest.json"
+
+Private data (UK) requires ``HUGGING_FACE_TOKEN`` or ``HF_TOKEN`` in the
+environment. After running: commit the changed manifest / TRO /
+pyproject.toml / changelog fragment, re-lock if pins moved, and run the
+test suite — certification is only asserted once the suite passes on the
+exact pinned pair.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(REPO_ROOT / "src"))
+
+from policyengine.provenance.certification import (  # noqa: E402
+    certify_data_release,
+)
+from policyengine.provenance.pyproject_pins import (  # noqa: E402
+    update_country_pins,
+)
+
+
+def main(argv=None) -> int:
+    parser = argparse.ArgumentParser(description="Certify a country data release.")
+    parser.add_argument("--country", required=True, choices=["us", "uk"])
+    parser.add_argument(
+        "--manifest-uri",
+        required=True,
+        help="hf://dataset/<repo_id>@<revision>/<path-to-release_manifest.json>",
+    )
+    parser.add_argument(
+        "--model-version",
+        default=None,
+        help="Model package version to certify for (default: installed).",
+    )
+    parser.add_argument("--no-pyproject", action="store_true")
+    parser.add_argument("--no-tro", action="store_true")
+    parser.add_argument("--no-changelog", action="store_true")
+    parser.add_argument(
+        "--skip-artifact-check",
+        action="store_true",
+        help="Skip the reachability HEAD on the certified dataset.",
+    )
+    args = parser.parse_args(argv)
+
+    token = os.environ.get("HUGGING_FACE_TOKEN") or os.environ.get("HF_TOKEN")
+
+    result = certify_data_release(
+        country=args.country,
+        manifest_uri=args.manifest_uri,
+        model_version=args.model_version,
+        token=token,
+        check_artifacts=not args.skip_artifact_check,
+    )
+    print(result.summary())
+
+    if not args.no_pyproject:
+        from importlib.metadata import version as installed_version
+
+        update_country_pins(
+            pyproject_path=REPO_ROOT / "pyproject.toml",
+            country=args.country,
+            model_package=result.model_package,
+            model_version=result.model_version,
+            core_version=installed_version("policyengine_core"),
+        )
+        print(f"pinned {result.model_package}=={result.model_version}")
+
+    if not args.no_tro:
+        from policyengine.provenance.bundle import regenerate_trace_tro
+
+        tro_path = regenerate_trace_tro(
+            args.country, result.country_manifest_path.parent
+        )
+        print(f"trace tro: {tro_path}")
+
+    if not args.no_changelog:
+        changelog_dir = REPO_ROOT / "changelog.d"
+        changelog_dir.mkdir(exist_ok=True)
+        fragment = (
+            changelog_dir
+            / f"certify-{args.country}-{result.build_id or 'data'}.changed.md"
+        )
+        fragment.write_text(
+            f"Certify the {args.country.upper()} data release "
+            f"`{result.build_id}` ({result.default_dataset}, "
+            f"{result.model_package} {result.model_version}) directly from "
+            "its data release manifest.\n"
+        )
+        print(f"changelog: {fragment}")
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

{policyengine-4.16.0 → policyengine-4.16.2}/src/policyengine/data/release_manifests/uk.json

@@ -1,8 +1,8 @@
 {
   "schema_version": 1,
-  "bundle_id": "uk-4.16.0",
+  "bundle_id": "uk-4.16.2",
   "country_id": "uk",
-  "policyengine_version": "4.16.0",
+  "policyengine_version": "4.16.2",
   "model_package": {
     "name": "policyengine-uk",
     "version": "2.88.20",

{policyengine-4.16.0 → policyengine-4.16.2}/src/policyengine/data/release_manifests/uk.trace.tro.jsonld

@@ -18,18 +18,18 @@
         "schema:url": "https://policyengine.org"
       },
       "schema:dateCreated": "2026-05-20T20:16:50.641086Z",
-      "schema:description": "TRACE TRO for certified runtime bundle uk-4.16.0 covering the bundle manifest, the certified dataset artifact, the country model wheel, and the country data release manifest when it is available.",
+      "schema:description": "TRACE TRO for certified runtime bundle uk-4.16.2 covering the bundle manifest, the certified dataset artifact, the country model wheel, and the country data release manifest when it is available.",
       "schema:name": "policyengine uk certified bundle TRO",
       "trov:createdWith": {
         "@type": "schema:SoftwareApplication",
         "schema:name": "policyengine",
-        "schema:softwareVersion": "4.16.0"
+        "schema:softwareVersion": "4.16.2"
       },
       "trov:hasArrangement": [
         {
           "@id": "arrangement/1",
           "@type": "trov:ArtifactArrangement",
-          "rdfs:comment": "Certified arrangement for bundle uk-4.16.0.",
+          "rdfs:comment": "Certified arrangement for bundle uk-4.16.2.",
           "trov:hasArtifactLocation": [
             {
               "@id": "arrangement/1/location/bundle_manifest",
@@ -75,7 +75,7 @@
             "@type": "trov:ResearchArtifact",
             "schema:name": "policyengine.py bundle manifest for uk",
             "trov:mimeType": "application/json",
-            "trov:sha256": "0a41c311264380a355c249c6a7e4f3569bee644806b5ec0cdcf1f1163a907a89"
+            "trov:sha256": "5f4629bb55d361c9a1a9f2a46950570acccfefd10ec5fd04865b27d9017d10ea"
           },
           {
             "@id": "composition/1/artifact/data_release_manifest",
@@ -102,7 +102,7 @@
         "trov:hasFingerprint": {
           "@id": "composition/1/fingerprint",
           "@type": "trov:CompositionFingerprint",
-          "trov:sha256": "257fcf97222403305b02103fdd04539a410da51b597cb87daac326dee12e37ef"
+          "trov:sha256": "cfdc777af8b1dd4666c26bad8ceb7a47e0fe6cec0f89c569601e90055760858e"
         }
       },
       "trov:hasPerformance": {
@@ -112,8 +112,8 @@
         "pe:certifiedBy": "policyengine.py bundled manifest",
         "pe:certifiedForModelVersion": "2.88.20",
         "pe:ciGitRef": "refs/heads/main",
-        "pe:ciGitSha": "86f1de199dcc5da6e87fdab2999adcf3d3e94eec",
-        "pe:ciRunUrl": "https://github.com/PolicyEngine/policyengine.py/actions/runs/27359946095",
+        "pe:ciGitSha": "8b084c5cefdf02d38dd6f974fd846905c3167f39",
+        "pe:ciRunUrl": "https://github.com/PolicyEngine/policyengine.py/actions/runs/27407803462",
         "pe:compatibilityBasis": "exact_build_model_version",
         "pe:dataBuildFingerprint": "sha256:77f149725a36055fd89961855230401852b0712d301c6e26d6d16565c6b23809",
         "pe:dataBuildId": "policyengine-uk-data-1.55.10",