policyengine 6.3.2__tar.gz → 6.3.3__tar.gz

{policyengine-6.3.2 → policyengine-6.3.3}/CHANGELOG.md

@@ -1,3 +1,10 @@
+## [6.3.3] - 2026-10-13
+
+### Changed
+
+- `create_datasets` stamps each US year file with the register of renamed stored inputs it was cut under and a year-file format revision. `ensure_datasets` creates a year file again, and `load_datasets` refuses it, unless its stamp matches this release exactly. A year file cut before a rename was registered, which may have lost that input, is therefore not reused. `load_datasets` says why it refuses a file, naming the renames the file may lack when its register is the reason. Year files written before the stamp existed are created again once.
+
+
 ## [6.3.2] - 2026-10-12
 
 ### Changed

{policyengine-6.3.2/src/policyengine.egg-info → policyengine-6.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: policyengine
-Version: 6.3.2
+Version: 6.3.3
 Summary: A package to conduct policy analysis using PolicyEngine tax-benefit models.
 Author-email: PolicyEngine <hello@policyengine.org>
 License:                     GNU AFFERO GENERAL PUBLIC LICENSE

{policyengine-6.3.2 → policyengine-6.3.3}/docs/microsim.md

@@ -401,15 +401,49 @@ The renames applied are recorded as `{old: new}` (`{}` when none applied):
 carried the draw: data that stores neither name runs with the new input's
 default, which for WIC is full take-up.
 
-Files written before this mapping existed have no record, and are not reused:
-
-- A saved US output may have been calculated without the mapped inputs.
-  `load()` refuses it, and `ensure()` runs the simulation again and saves the new
-  output.
-- A year file that `ensure_datasets` or `create_datasets` wrote stores
-  neither name, so the draw is lost from it. `ensure_datasets` creates such
-  year files again, and `load_datasets` refuses them. A year file opened
-  directly, as `PolicyEngineUSDataset(filepath=...)`, is not checked.
+A saved US output written before this mapping existed has no record and may
+have been calculated without the mapped inputs, so `load()` refuses it, and
+`ensure()` runs the simulation again and saves the new output. Saved outputs
+are checked only for the record, not for the register they were calculated
+under.
+
+### Year files
+
+`create_datasets` stamps each year file, after writing everything else, with
+the register it was cut under and a year-file format revision
+(`legacy_inputs.YEAR_FILE_STAMP_H5_DATASET`, UTF-8 JSON
+`{"register": {old: new, ...}, "revision": 1}`). `ensure_datasets` reuses a
+year file only when its stamp equals this release's exactly and its renames
+record lists only renames in the stamped register. Otherwise it logs why and
+creates the year files again; `load_datasets` refuses the file with the same
+reason, which names the renames the file may lack when its register is the
+cause. To decide, both read only the stamp and the record, never the tables
+or the source dataset. `legacy_inputs.year_file_stamp_problem(path)` returns
+the reason, or `None` for a current year file.
+
+A year file is out of date when:
+
+- it has no stamp. Year files that policyengine.py 6.1.1 and earlier cut,
+  with policyengine-us 2.x, from a source storing the draw only as
+  `would_claim_wic` (such as the certified default) store neither WIC take-up
+  name, so the draw is lost from them. Year files written with the renames
+  record but before the stamp are cut again once, and so is a year file whose
+  write did not finish;
+- its register lacks an entry this release has. It may have lost that
+  entry's input, as the older files lost the WIC draw;
+- its register has an entry this release lacks, or maps an old name to
+  another new name. It may store a new name this release's engine does not
+  define, and a run skips any column the engine does not define;
+- it has another format revision, or its stamp or record cannot be read.
+
+So any change to `LEGACY_INPUT_RENAMES` makes every year file out of date,
+and two policyengine.py versions with different registers that share a data
+folder cut each other's year files again. `ensure_datasets` raises, rather
+than cutting the files again, when a year file cannot be opened for reasons
+unrelated to its contents: no permission, a directory at its path, or a lock
+held by another process. A year file opened directly, as
+`PolicyEngineUSDataset(filepath=...)`, is not checked, and saving it again with
+`save()` drops its stamp.
 
 ## Pinned model versions
 

{policyengine-6.3.2 → policyengine-6.3.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "policyengine"
-version = "6.3.2"
+version = "6.3.3"
 description = "A package to conduct policy analysis using PolicyEngine tax-benefit models."
 readme = "README.md"
 authors = [

{policyengine-6.3.2 → policyengine-6.3.3}/src/policyengine/data/bundle/manifest.json

@@ -1,10 +1,10 @@
 {
-  "bundle_version": "6.3.2",
+  "bundle_version": "6.3.3",
   "citation": {
     "publisher": "PolicyEngine",
-    "title": "PolicyEngine bundle 6.3.2",
+    "title": "PolicyEngine bundle 6.3.3",
     "type": "software-bundle",
-    "version": "6.3.2"
+    "version": "6.3.3"
   },
   "countries": {
     "uk": {
@@ -17,7 +17,7 @@
   "data_releases": {
     "uk": {
       "build_id": "policyengine-uk-data-1.58.0",
-      "bundle_id": "uk-6.3.2",
+      "bundle_id": "uk-6.3.3",
       "certification": {
         "built_with_model_git_sha": "4cbedbecca352f52dffe47e849553c2759a00711",
         "built_with_model_version": "2.122.2",
@@ -99,7 +99,7 @@
         "version": "2.126.12",
         "wheel_url": "https://files.pythonhosted.org/packages/4f/3f/0a611e9b25beb75db3f2f521d359eabad0d1ab4e50194b866663355591ff/policyengine_uk-2.126.12-py3-none-any.whl"
       },
-      "policyengine_version": "6.3.2",
+      "policyengine_version": "6.3.3",
       "region_datasets": {
         "national": {
           "path_template": "enhanced_frs_2024_25.h5"
@@ -112,7 +112,7 @@
     },
     "us": {
       "build_id": "populace-us-2024-spm-20260915",
-      "bundle_id": "us-6.3.2",
+      "bundle_id": "us-6.3.3",
       "certification": {
         "built_with_model_version": "2.2.1",
         "certified_by": "policyengine.py bundle certification",
@@ -217,7 +217,7 @@
         "version": "2.2.1",
         "wheel_url": "https://files.pythonhosted.org/packages/5e/09/86ac3968d2a18e28fce3ba1186fc21728e9f299b96a806d6f204f1e0aaed/policyengine_us-2.2.1-py3-none-any.whl"
       },
-      "policyengine_version": "6.3.2",
+      "policyengine_version": "6.3.3",
       "region_datasets": {
         "congressional_district": {
           "path_template": "populace_us_2024_acs_local.h5"
@@ -295,10 +295,10 @@
   "packages": {
     "policyengine": {
       "import_name": "policyengine",
-      "install_requirement": "policyengine==6.3.2",
+      "install_requirement": "policyengine==6.3.3",
       "name": "policyengine",
       "role": "bundle_carrier",
-      "version": "6.3.2"
+      "version": "6.3.3"
     },
     "policyengine-core": {
       "import_name": "policyengine_core",
@@ -334,6 +334,6 @@
       "wheel_url": "https://files.pythonhosted.org/packages/19/a0/c484f69a0ebf88a9b9fd0ac28176f8df66550f46714f1b0c5ec0b360824e/spm_calculator-1.0.0-py3-none-any.whl"
     }
   },
-  "policyengine_version": "6.3.2",
+  "policyengine_version": "6.3.3",
   "schema_version": 2
 }

{policyengine-6.3.2 → policyengine-6.3.3}/src/policyengine/data/bundle/uk.trace.tro.jsonld

@@ -18,18 +18,18 @@
         "schema:url": "https://policyengine.org"
       },
       "schema:dateCreated": "2026-10-07T17:31:53.189867Z",
-      "schema:description": "TRACE TRO for certified runtime bundle uk-6.3.2 covering the bundle manifest, the certified dataset artifact, the country model wheel, and the country data release manifest when it is available.",
+      "schema:description": "TRACE TRO for certified runtime bundle uk-6.3.3 covering the bundle manifest, the certified dataset artifact, the country model wheel, and the country data release manifest when it is available.",
       "schema:name": "policyengine uk certified bundle TRO",
       "trov:createdWith": {
         "@type": "schema:SoftwareApplication",
         "schema:name": "policyengine",
-        "schema:softwareVersion": "6.3.2"
+        "schema:softwareVersion": "6.3.3"
       },
       "trov:hasArrangement": [
         {
           "@id": "arrangement/1",
           "@type": "trov:ArtifactArrangement",
-          "rdfs:comment": "Certified arrangement for bundle uk-6.3.2.",
+          "rdfs:comment": "Certified arrangement for bundle uk-6.3.3.",
           "trov:hasArtifactLocation": [
             {
               "@id": "arrangement/1/location/bundle_manifest",
@@ -75,7 +75,7 @@
             "@type": "trov:ResearchArtifact",
             "schema:name": "policyengine.py bundle manifest for uk",
             "trov:mimeType": "application/json",
-            "trov:sha256": "323b86d959539ed3fbde416387d136161e58483d8f7b6c7b9dc47936e3e18f9b"
+            "trov:sha256": "afae4d8bd8a45f286068273af44746295077a7ff18fce58aa91889f5555467c5"
           },
           {
             "@id": "composition/1/artifact/data_release_manifest",
@@ -102,7 +102,7 @@
         "trov:hasFingerprint": {
           "@id": "composition/1/fingerprint",
           "@type": "trov:CompositionFingerprint",
-          "trov:sha256": "c00ff93e4fc59e689bb7c50b979ca5efc749c2eee6c66a3931135fe196c385a1"
+          "trov:sha256": "222478bcb094597ab206850606909fbf18e4be5c6085d304b406eb85f69fd36d"
         }
       },
       "trov:hasPerformance": {

{policyengine-6.3.2 → policyengine-6.3.3}/src/policyengine/data/bundle/us.trace.tro.jsonld

@@ -17,18 +17,18 @@
         "schema:name": "PolicyEngine",
         "schema:url": "https://policyengine.org"
       },
-      "schema:description": "TRACE TRO for certified runtime bundle us-6.3.2 covering the bundle manifest, the certified dataset artifact, the country model wheel, and the country data release manifest when it is available.",
+      "schema:description": "TRACE TRO for certified runtime bundle us-6.3.3 covering the bundle manifest, the certified dataset artifact, the country model wheel, and the country data release manifest when it is available.",
       "schema:name": "policyengine us certified bundle TRO",
       "trov:createdWith": {
         "@type": "schema:SoftwareApplication",
         "schema:name": "policyengine",
-        "schema:softwareVersion": "6.3.2"
+        "schema:softwareVersion": "6.3.3"
       },
       "trov:hasArrangement": [
         {
           "@id": "arrangement/1",
           "@type": "trov:ArtifactArrangement",
-          "rdfs:comment": "Certified arrangement for bundle us-6.3.2.",
+          "rdfs:comment": "Certified arrangement for bundle us-6.3.3.",
           "trov:hasArtifactLocation": [
             {
               "@id": "arrangement/1/location/bundle_manifest",
@@ -74,7 +74,7 @@
             "@type": "trov:ResearchArtifact",
             "schema:name": "policyengine.py bundle manifest for us",
             "trov:mimeType": "application/json",
-            "trov:sha256": "323b86d959539ed3fbde416387d136161e58483d8f7b6c7b9dc47936e3e18f9b"
+            "trov:sha256": "afae4d8bd8a45f286068273af44746295077a7ff18fce58aa91889f5555467c5"
           },
           {
             "@id": "composition/1/artifact/data_release_manifest",
@@ -101,7 +101,7 @@
         "trov:hasFingerprint": {
           "@id": "composition/1/fingerprint",
           "@type": "trov:CompositionFingerprint",
-          "trov:sha256": "d7ecd868e200c681d97acb264562877d2f86cb9820e2cee5e48b5f3426cae693"
+          "trov:sha256": "4f978b5331c8f9ad391c67d4779c69bde9c4ff3062bfe4c038d0bde462e79dec"
         }
       },
       "trov:hasPerformance": {

{policyengine-6.3.2 → policyengine-6.3.3}/src/policyengine/tax_benefit_models/us/datasets.py

@@ -1,6 +1,7 @@
 import hashlib
 import importlib.util
 import json
+import logging
 import warnings
 from importlib import metadata as importlib_metadata
 from pathlib import Path
@@ -31,9 +32,14 @@ from policyengine.tax_benefit_models.us.legacy_inputs import (
     pending_legacy_input_renames,
     read_renames_record,
     write_renames_record,
+    write_year_file_stamp,
+    year_file_stamp,
+    year_file_stamp_problem,
 )
 from policyengine.utils.hashing import sha256_file
 
+logger = logging.getLogger(__name__)
+
 
 class USYearData(YearData):
     """Entity-level data for a single year."""
@@ -371,6 +377,12 @@ def create_datasets(
 ) -> dict[str, PolicyEngineUSDataset]:
     """Create PolicyEngineUSDataset instances from logical dataset names or URLs.
 
+    Each year file is written to ``{data_folder}/{dataset_stem}_year_{year}.h5``
+    and stamped last with the register of renamed stored inputs it was cut
+    under and the year-file format revision (see
+    ``legacy_inputs.YEAR_FILE_STAMP_H5_DATASET``), which ``ensure_datasets``
+    and ``load_datasets`` check.
+
     Args:
         datasets: List of logical dataset names or HuggingFace dataset URLs
         years: List of years to extract data for
@@ -394,6 +406,8 @@ def create_datasets(
         )
         dataset_stem = source.name
         sim = Microsimulation(dataset=source.path, spm=resolve_spm_selection())
+        # The register the mapping below uses, stamped into each year file.
+        stamp = year_file_stamp()
         # Map stored inputs the engine has since renamed before extracting,
         # so each year's file stores the live input rather than dropping it.
         legacy_input_renames = apply_legacy_input_renames_to_microsimulation(sim)
@@ -555,6 +569,9 @@ def create_datasets(
                 ),
             )
             us_dataset.save()
+            # Stamp last. ``save()`` truncates the file first, so a year file
+            # whose write stopped part-way has no stamp and is cut again.
+            write_year_file_stamp(us_dataset.filepath, stamp)
 
             dataset_key = f"{dataset_stem}_{year}"
             result[dataset_key] = us_dataset
@@ -562,16 +579,6 @@ def create_datasets(
     return result
 
 
-def _year_file_records_renames(path: Path) -> bool:
-    """Return whether a year file records the renamed stored inputs mapped.
-
-    ``create_datasets`` writes the record into every year file (``{}`` when
-    nothing needed mapping). Files written before it did may have lost a
-    renamed input such as the WIC take-up draw.
-    """
-    return read_renames_record(path) is not None
-
-
 def load_datasets(
     datasets: Optional[list[str]] = None,
     years: list[int] = [2024, 2025, 2026, 2027, 2028],
@@ -579,10 +586,13 @@ def load_datasets(
 ) -> dict[str, PolicyEngineUSDataset]:
     """Load PolicyEngineUSDataset instances from saved HDF5 files.
 
-    A year file without the record of renamed stored inputs that
-    ``create_datasets`` writes was cut before those inputs were mapped, and
-    may have lost one (see ``legacy_inputs.RENAMES_H5_DATASET``), so it is
-    refused.
+    A year file is refused unless ``create_datasets`` stamped it with this
+    release's register of renamed stored inputs and year-file format revision
+    (see ``legacy_inputs.year_file_stamp_problem``). A file cut before a
+    register entry was added may have lost that input, as files that
+    policyengine.py 6.1.1 and earlier cut from the certified default lost the
+    WIC take-up draw. The error says why, and names the renames the file may
+    lack when its register is the reason.
 
     Args:
         datasets: List of HuggingFace dataset paths (used to derive file names)
@@ -599,17 +609,14 @@ def load_datasets(
         dataset_stem = dataset_logical_name(resolved_dataset)
         for year in years:
             filepath = f"{data_folder}/{dataset_stem}_year_{year}.h5"
-            if Path(filepath).exists() and not _year_file_records_renames(
-                Path(filepath)
-            ):
-                raise ValueError(
-                    f"US year file {filepath} has no record of the renamed "
-                    "stored inputs mapped when it was cut, so it was written "
-                    "before policyengine.py mapped them and may have lost "
-                    "the WIC take-up draw (every WIC-eligible person would "
-                    "then take WIC up). Regenerate it with ensure_datasets() "
-                    "or create_datasets()."
-                )
+            if Path(filepath).exists():
+                problem = year_file_stamp_problem(filepath)
+                if problem is not None:
+                    raise ValueError(
+                        f"US year file {filepath} is out of date: {problem}. "
+                        "Regenerate it with ensure_datasets() or "
+                        "create_datasets()."
+                    )
             us_dataset = PolicyEngineUSDataset(
                 name=f"{dataset_stem}-year-{year}",
                 description=f"US Dataset for year {year} based on {dataset_stem}",
@@ -1255,10 +1262,13 @@ def ensure_datasets(
 ) -> dict[str, PolicyEngineUSDataset]:
     """Ensure datasets exist, loading if available or creating if not.
 
-    Year files without the record of renamed stored inputs that
-    ``create_datasets`` writes were cut before those inputs were mapped, and
-    may have lost one such as the WIC take-up draw, so they are created
-    again rather than loaded.
+    Existing year files are reused only when ``create_datasets`` stamped them
+    with this release's register of renamed stored inputs and year-file format
+    revision (see ``legacy_inputs.year_file_stamp_problem``). Only each file's
+    stamp and renames record are read to decide, never its tables or the
+    source dataset. If any year file is missing or out of date, every
+    year file is created again from the source dataset, and each out-of-date
+    file is logged with the reason.
 
     Args:
         datasets: List of HuggingFace dataset paths
@@ -1270,22 +1280,29 @@ def ensure_datasets(
     """
     datasets = datasets or [get_release_manifest("us").default_dataset]
 
-    # Check if all dataset files exist
-    all_exist = True
+    # Check that every year file exists and is current. A year file cut
+    # under another register of renamed stored inputs may have lost one, so
+    # it is created again rather than reused.
+    all_current = True
     for dataset in datasets:
         resolved_dataset = resolve_dataset_reference("us", dataset)
         dataset_stem = dataset_logical_name(resolved_dataset)
         for year in years:
             filepath = Path(f"{data_folder}/{dataset_stem}_year_{year}.h5")
-            # A year file written before renamed stored inputs were mapped
-            # may have lost them, so it is regenerated rather than reused.
-            if not filepath.exists() or not _year_file_records_renames(filepath):
-                all_exist = False
-                break
-        if not all_exist:
-            break
-
-    if all_exist:
+            if not filepath.exists():
+                all_current = False
+                continue
+            problem = year_file_stamp_problem(filepath)
+            if problem is not None:
+                all_current = False
+                logger.warning(
+                    "US year file %s is out of date: %s. Creating the year "
+                    "files again.",
+                    filepath,
+                    problem,
+                )
+
+    if all_current:
         return load_datasets(datasets=datasets, years=years, data_folder=data_folder)
     else:
         return create_datasets(