profilefoundry 1.0.0__tar.gz

profilefoundry-1.0.0/CITATION.cff

@@ -0,0 +1,12 @@
+cff-version: 1.2.0
+message: "If you use ProfileFoundry, please cite the paper when available; until then, cite the repository."
+type: software
+title: "ProfileFoundry: An Audited Person-Object Substrate for Stateful NLP and Agent Evaluation"
+authors:
+  - family-names: Selvam
+    given-names: Sriram
+repository-code: "https://github.com/selvamsriram/ProfileFoundry"
+url: "https://github.com/selvamsriram/ProfileFoundry"
+version: "1.0.0"
+date-released: "2026-06-14"
+license: "LicenseRef-ProfileFoundry-Citation-1.0"

profilefoundry-1.0.0/LICENSE

@@ -0,0 +1,34 @@
+ProfileFoundry Citation License 1.0
+
+Copyright (c) 2026 Sriram Selvam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, subject to the following conditions:
+
+1. Attribution and citation. Any public use, redistribution, derivative work,
+   benchmark, dataset, paper, report, model card, data card, product
+   documentation, or repository that uses or includes the Software or generated
+   ProfileFoundry artifacts must provide reasonable attribution to
+   ProfileFoundry. Cite the ProfileFoundry paper when it is available. Until
+   then, cite the ProfileFoundry repository:
+
+       ProfileFoundry: An Audited Person-Object Substrate for Stateful NLP
+       and Agent Evaluation. https://github.com/selvamsriram/ProfileFoundry
+
+2. Notice preservation. The above copyright notice, this license, and any
+   citation instructions distributed with the Software must be included in all
+   copies or substantial portions of the Software.
+
+3. Reference data. Embedded reference data and generated datasets may carry
+   additional upstream attribution requirements. Those notices must be
+   preserved when redistributing derived artifacts.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT, OR OTHERWISE, ARISING FROM,
+OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

profilefoundry-1.0.0/MANIFEST.in

@@ -0,0 +1,2 @@
+include CITATION.cff
+include LICENSE

profilefoundry-1.0.0/PKG-INFO

@@ -0,0 +1,193 @@
+Metadata-Version: 2.4
+Name: profilefoundry
+Version: 1.0.0
+Summary: An open-source generator for structured, internally-consistent, linked, temporally coherent synthetic Person Objects.
+Author: Sriram Selvam
+License-Expression: LicenseRef-ProfileFoundry-Citation-1.0
+Project-URL: Homepage, https://github.com/selvamsriram/ProfileFoundry
+Project-URL: Issues, https://github.com/selvamsriram/ProfileFoundry/issues
+Keywords: synthetic-data,pii,person-object,linkage,evaluation,privacy
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.6
+Requires-Dist: faker>=24.0
+Requires-Dist: numpy>=1.26
+Requires-Dist: pandas>=2.1
+Requires-Dist: pyarrow>=15.0
+Requires-Dist: click>=8.1
+Requires-Dist: rapidfuzz>=3.6
+Requires-Dist: pybloom-live>=4.0
+Requires-Dist: requests>=2.31
+Requires-Dist: tqdm>=4.66
+Requires-Dist: python-dotenv>=1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Requires-Dist: ipython>=8.20; extra == "dev"
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.8; extra == "viz"
+Requires-Dist: seaborn>=0.13; extra == "viz"
+Requires-Dist: plotly>=5.20; extra == "viz"
+Provides-Extra: hf
+Requires-Dist: datasets>=2.18; extra == "hf"
+Requires-Dist: huggingface-hub>=0.23; extra == "hf"
+Dynamic: license-file
+
+# ProfileFoundry
+
+> A structured, internally-consistent, linked, temporally coherent synthetic
+> Person Object - and an open-source generator that produces them at scale.
+
+This repository is the implementation home for ProfileFoundry, which builds
+on the problem setting exposed by [PANORAMA](https://arxiv.org/abs/2505.12238)
+(Selvam & Ghosh, 2025). It provides a deterministic generator, schema,
+validation suite, release reports, and Hugging Face export tooling for the
+`ProfileFoundry-Core-100K` dataset.
+
+The ACL/NeurIPS-style paper draft lives in [`Paper/`](Paper/).
+The vetted 100K release package is staged for Hugging Face at
+[`srirxml/ProfileFoundry-Core-100K`](https://huggingface.co/datasets/srirxml/ProfileFoundry-Core-100K);
+run `python scripts/verify_hf_release_current.py` with an `HF_TOKEN` before
+calling the remote artifact current. The default HF viewer table is
+`person_objects.parquet`, a complete one-row-per-person view with nested
+sections encoded as JSON strings; the remaining parquet files expose the
+normalized relational schema.
+
+## Status (2026-05-24) — v1.0 release sprint
+
+| Phase | Status |
+|---|---|
+| **0 — Foundation** (schema and package scaffolding) | ✅ complete |
+| **1 — Reference data** (manifest + bootstrap + loader) | ✅ ref data shipped for 5 validation locales |
+| **2 — Generator factory** (port + joint constraints + age-gating) | ✅ deterministic across processes (set-iteration bug fixed) |
+| **3 — Linkage** (households, employers, family graph) | ✅ household orchestrator + family edges + employer pool |
+| **4 — Temporal** (event taxonomy, backfill, replay) | ✅ replay-valid non-credit timeline + typed payload export |
+| **5a — Validation** (KS gaps, consistency) | ✅ disclosed KS gaps (max 0.124) · 100% consistency |
+| **5b — Leakage audit** (Wikidata Bloom, reserved-domain email audit, self-collision) | ✅ Wikidata Bloom + self-collision + reserved-domain email syntax audit |
+| **6 — Scale + publish** (100K + HF export) | ✅ 100K release · complete viewer table · normalized parquet star schema · MANIFEST.json |
+| **7 — Reproducibility** (verify fixture, manifest hash) | ✅ normalized fixture stable across processes; verify script + fixture |
+| **8 — Paper draft** | ✅ submission draft revised |
+| **9 — HF push + tag v1.0.0** | ⏳ HF upload target documented; remote current check requires `HF_TOKEN`; tag pending |
+
+## Quick start
+
+```bash
+pip install -e ".[dev]"
+profilefoundry verify                # smoke-test all 8 locales
+profilefoundry person --locale US    # one US Person as JSON
+profilefoundry household --locale US # one linked household (multiple Persons)
+profilefoundry scale --n 1000 --locale UK --out /tmp/uk.jsonl
+profilefoundry scale-households --n 500 --locale CA --out /tmp/ca_households.jsonl
+profilefoundry validate --n 300                     # KS gaps + leakage + consistency
+profilefoundry export --out /tmp/pf_core --n-per-locale 1000 --generation-date 2026-05-24 --exported-at 2026-05-24 --skip-hibp  # normalized parquet star schema
+profilefoundry scale-smoke --sizes 1000,10000,100000           # timings
+
+# Full v1.0 release run (100K profiles in ~4-5 minutes on a laptop):
+python scripts/run_full_core.py --generation-date 2026-05-24 --skip-hibp --verbose
+# Reproducibility check:
+python scripts/verify_reproducibility.py
+# Hugging Face upload preflight / push:
+python scripts/push_to_hf.py --dry-run
+python scripts/push_to_hf.py --repo srirxml/ProfileFoundry-Core-100K
+# Confirm the public HF manifest matches the vetted local release:
+python scripts/verify_hf_release_current.py
+# Build the Wikidata bloom (multi-hour due to WDQS rate limit):
+python scripts/ingest_wikidata_persons.py --scope sample --sleep 65 --verbose
+```
+
+Or without installing:
+
+```bash
+PYTHONPATH=src python3 -m profilefoundry.cli scale --n 1000 --locale US --out /tmp/us.jsonl
+```
+
+## CLI reference
+
+All commands are available through `profilefoundry` after installation, or via
+`PYTHONPATH=src python3 -m profilefoundry.cli` from a checkout.
+
+| Command | Purpose | Common example |
+|---|---|---|
+| `profilefoundry verify` | Generate one profile per supported locale as a smoke test. | `profilefoundry verify` |
+| `profilefoundry person` | Print one deterministic Person Object as JSON. | `profilefoundry person --locale US --seed 4321 --profile-seq 1` |
+| `profilefoundry household` | Print one linked household as JSON. | `profilefoundry household --locale UK --seed 4321 --seq 7` |
+| `profilefoundry scale` | Generate flat JSONL profiles for one locale. | `profilefoundry scale --n 1000 --locale CA --out /tmp/ca.jsonl` |
+| `profilefoundry scale-households` | Generate linked household members as JSONL. | `profilefoundry scale-households --n 500 --locale AU --out /tmp/au_households.jsonl` |
+| `profilefoundry validate` | Run distributional, leakage, replay, and consistency checks. | `profilefoundry validate --n 300 --locales US,UK,IN --skip-hibp` |
+| `profilefoundry export` | Write JSONL, parquet tables, manifest, and dataset card. | `profilefoundry export --out /tmp/pf_core --n-per-locale 1000 --generation-date 2026-05-24 --exported-at 2026-05-24T00:00:00Z --skip-hibp` |
+| `profilefoundry scale-smoke` | Time the generator at several sizes. | `profilefoundry scale-smoke --sizes 1000,10000,100000` |
+
+Script-level release utilities:
+
+| Script | Purpose |
+|---|---|
+| `python scripts/run_full_core.py --generation-date 2026-05-24 --skip-hibp --verbose` | Rebuild the 100K release artifact under `data/raw/profilefoundry-core-v1/` and reports under `reports/v1_release/`. |
+| `python scripts/verify_reproducibility.py` | Confirm the pinned reproducibility fixture still matches. |
+| `python scripts/verify_package_reference_data.py` | Confirm reference data is available from an installed wheel. |
+| `python scripts/push_to_hf.py --dry-run` | Preview the Hugging Face upload. |
+| `python scripts/push_to_hf.py --repo srirxml/ProfileFoundry-Core-100K` | Upload the vetted release artifact. |
+| `python scripts/verify_hf_release_current.py` | Compare the local release manifest with the Hugging Face dataset. Requires `HF_TOKEN` for private or gated repos. |
+| `python scripts/export_schema.py --check` | Verify the checked-in JSON Schema matches the Pydantic model. |
+| `python scripts/ingest_us_acs.py` / `python scripts/ingest_uk_ons.py` | Refresh reference-data inputs from upstream sources. |
+
+## Project layout
+
+```
+.
+├── README.md                  # this file
+├── pyproject.toml             # package metadata, deps, ruff/pytest config
+├── notes/
+│   ├── BUGS.tsv               # resolved/known issue ledger
+│   └── RELEASE_PLAN.md        # v1.0 release plan
+├── schemas/
+│   ├── README.md
+│   └── person_v0_1.schema.json  # auto-exported JSON Schema
+├── src/profilefoundry/
+│   ├── schema/                # Pydantic Person Object v0.1 (source of truth)
+│   ├── data/                  # reference-data loader (bootstrap ↔ derived)
+│   ├── generate/              # factory + sampling + locale providers
+│   ├── linkage/               # (phase 3) households, employers, families
+│   ├── validate/              # (phase 5) invariants, distributions, leakage
+│   ├── io/                    # (phase 6) Hugging Face export
+│   └── cli.py                 # `profilefoundry` entry point
+├── data/reference/
+│   ├── MANIFEST.md            # every external source enumerated
+│   └── bootstrap/             # committed minimum reference marginals
+├── scripts/
+│   ├── export_schema.py       # regenerate JSON Schema
+│   └── ingest_us_acs.py       # pull richer ACS tables live (needs API key)
+├── tests/                     # pytest invariants & unit tests
+└── Paper/                     # ACL-style paper draft + figures
+```
+
+## Testing
+
+```bash
+PYTHONPATH=src python3 -m pytest tests/
+```
+
+The full pytest suite covers schema invariants (Pydantic model), reference-data
+loading, factory smoke checks, determinism, plausibility (age, employment,
+addresses, education), **adults-only enforcement** (no profiles under 18),
+linkage (households, employers, family graph), validation (KS gaps, replay,
+consistency, leakage), release-report freshness, normalized export quality, and
+reproducibility.
+
+## License
+
+- Code and SDK: **ProfileFoundry Citation License 1.0** (see `LICENSE`).
+  Public uses and redistributions must cite the ProfileFoundry paper when
+  available, or this repository until then. Machine-readable citation metadata
+  is in `CITATION.cff`.
+- Generated dataset: **CC-BY-4.0**.
+- Embedded reference data retains its upstream license; see
+  `data/reference/MANIFEST.md`.

profilefoundry-1.0.0/README.md

@@ -0,0 +1,149 @@
+# ProfileFoundry
+
+> A structured, internally-consistent, linked, temporally coherent synthetic
+> Person Object - and an open-source generator that produces them at scale.
+
+This repository is the implementation home for ProfileFoundry, which builds
+on the problem setting exposed by [PANORAMA](https://arxiv.org/abs/2505.12238)
+(Selvam & Ghosh, 2025). It provides a deterministic generator, schema,
+validation suite, release reports, and Hugging Face export tooling for the
+`ProfileFoundry-Core-100K` dataset.
+
+The ACL/NeurIPS-style paper draft lives in [`Paper/`](Paper/).
+The vetted 100K release package is staged for Hugging Face at
+[`srirxml/ProfileFoundry-Core-100K`](https://huggingface.co/datasets/srirxml/ProfileFoundry-Core-100K);
+run `python scripts/verify_hf_release_current.py` with an `HF_TOKEN` before
+calling the remote artifact current. The default HF viewer table is
+`person_objects.parquet`, a complete one-row-per-person view with nested
+sections encoded as JSON strings; the remaining parquet files expose the
+normalized relational schema.
+
+## Status (2026-05-24) — v1.0 release sprint
+
+| Phase | Status |
+|---|---|
+| **0 — Foundation** (schema and package scaffolding) | ✅ complete |
+| **1 — Reference data** (manifest + bootstrap + loader) | ✅ ref data shipped for 5 validation locales |
+| **2 — Generator factory** (port + joint constraints + age-gating) | ✅ deterministic across processes (set-iteration bug fixed) |
+| **3 — Linkage** (households, employers, family graph) | ✅ household orchestrator + family edges + employer pool |
+| **4 — Temporal** (event taxonomy, backfill, replay) | ✅ replay-valid non-credit timeline + typed payload export |
+| **5a — Validation** (KS gaps, consistency) | ✅ disclosed KS gaps (max 0.124) · 100% consistency |
+| **5b — Leakage audit** (Wikidata Bloom, reserved-domain email audit, self-collision) | ✅ Wikidata Bloom + self-collision + reserved-domain email syntax audit |
+| **6 — Scale + publish** (100K + HF export) | ✅ 100K release · complete viewer table · normalized parquet star schema · MANIFEST.json |
+| **7 — Reproducibility** (verify fixture, manifest hash) | ✅ normalized fixture stable across processes; verify script + fixture |
+| **8 — Paper draft** | ✅ submission draft revised |
+| **9 — HF push + tag v1.0.0** | ⏳ HF upload target documented; remote current check requires `HF_TOKEN`; tag pending |
+
+## Quick start
+
+```bash
+pip install -e ".[dev]"
+profilefoundry verify                # smoke-test all 8 locales
+profilefoundry person --locale US    # one US Person as JSON
+profilefoundry household --locale US # one linked household (multiple Persons)
+profilefoundry scale --n 1000 --locale UK --out /tmp/uk.jsonl
+profilefoundry scale-households --n 500 --locale CA --out /tmp/ca_households.jsonl
+profilefoundry validate --n 300                     # KS gaps + leakage + consistency
+profilefoundry export --out /tmp/pf_core --n-per-locale 1000 --generation-date 2026-05-24 --exported-at 2026-05-24 --skip-hibp  # normalized parquet star schema
+profilefoundry scale-smoke --sizes 1000,10000,100000           # timings
+
+# Full v1.0 release run (100K profiles in ~4-5 minutes on a laptop):
+python scripts/run_full_core.py --generation-date 2026-05-24 --skip-hibp --verbose
+# Reproducibility check:
+python scripts/verify_reproducibility.py
+# Hugging Face upload preflight / push:
+python scripts/push_to_hf.py --dry-run
+python scripts/push_to_hf.py --repo srirxml/ProfileFoundry-Core-100K
+# Confirm the public HF manifest matches the vetted local release:
+python scripts/verify_hf_release_current.py
+# Build the Wikidata bloom (multi-hour due to WDQS rate limit):
+python scripts/ingest_wikidata_persons.py --scope sample --sleep 65 --verbose
+```
+
+Or without installing:
+
+```bash
+PYTHONPATH=src python3 -m profilefoundry.cli scale --n 1000 --locale US --out /tmp/us.jsonl
+```
+
+## CLI reference
+
+All commands are available through `profilefoundry` after installation, or via
+`PYTHONPATH=src python3 -m profilefoundry.cli` from a checkout.
+
+| Command | Purpose | Common example |
+|---|---|---|
+| `profilefoundry verify` | Generate one profile per supported locale as a smoke test. | `profilefoundry verify` |
+| `profilefoundry person` | Print one deterministic Person Object as JSON. | `profilefoundry person --locale US --seed 4321 --profile-seq 1` |
+| `profilefoundry household` | Print one linked household as JSON. | `profilefoundry household --locale UK --seed 4321 --seq 7` |
+| `profilefoundry scale` | Generate flat JSONL profiles for one locale. | `profilefoundry scale --n 1000 --locale CA --out /tmp/ca.jsonl` |
+| `profilefoundry scale-households` | Generate linked household members as JSONL. | `profilefoundry scale-households --n 500 --locale AU --out /tmp/au_households.jsonl` |
+| `profilefoundry validate` | Run distributional, leakage, replay, and consistency checks. | `profilefoundry validate --n 300 --locales US,UK,IN --skip-hibp` |
+| `profilefoundry export` | Write JSONL, parquet tables, manifest, and dataset card. | `profilefoundry export --out /tmp/pf_core --n-per-locale 1000 --generation-date 2026-05-24 --exported-at 2026-05-24T00:00:00Z --skip-hibp` |
+| `profilefoundry scale-smoke` | Time the generator at several sizes. | `profilefoundry scale-smoke --sizes 1000,10000,100000` |
+
+Script-level release utilities:
+
+| Script | Purpose |
+|---|---|
+| `python scripts/run_full_core.py --generation-date 2026-05-24 --skip-hibp --verbose` | Rebuild the 100K release artifact under `data/raw/profilefoundry-core-v1/` and reports under `reports/v1_release/`. |
+| `python scripts/verify_reproducibility.py` | Confirm the pinned reproducibility fixture still matches. |
+| `python scripts/verify_package_reference_data.py` | Confirm reference data is available from an installed wheel. |
+| `python scripts/push_to_hf.py --dry-run` | Preview the Hugging Face upload. |
+| `python scripts/push_to_hf.py --repo srirxml/ProfileFoundry-Core-100K` | Upload the vetted release artifact. |
+| `python scripts/verify_hf_release_current.py` | Compare the local release manifest with the Hugging Face dataset. Requires `HF_TOKEN` for private or gated repos. |
+| `python scripts/export_schema.py --check` | Verify the checked-in JSON Schema matches the Pydantic model. |
+| `python scripts/ingest_us_acs.py` / `python scripts/ingest_uk_ons.py` | Refresh reference-data inputs from upstream sources. |
+
+## Project layout
+
+```
+.
+├── README.md                  # this file
+├── pyproject.toml             # package metadata, deps, ruff/pytest config
+├── notes/
+│   ├── BUGS.tsv               # resolved/known issue ledger
+│   └── RELEASE_PLAN.md        # v1.0 release plan
+├── schemas/
+│   ├── README.md
+│   └── person_v0_1.schema.json  # auto-exported JSON Schema
+├── src/profilefoundry/
+│   ├── schema/                # Pydantic Person Object v0.1 (source of truth)
+│   ├── data/                  # reference-data loader (bootstrap ↔ derived)
+│   ├── generate/              # factory + sampling + locale providers
+│   ├── linkage/               # (phase 3) households, employers, families
+│   ├── validate/              # (phase 5) invariants, distributions, leakage
+│   ├── io/                    # (phase 6) Hugging Face export
+│   └── cli.py                 # `profilefoundry` entry point
+├── data/reference/
+│   ├── MANIFEST.md            # every external source enumerated
+│   └── bootstrap/             # committed minimum reference marginals
+├── scripts/
+│   ├── export_schema.py       # regenerate JSON Schema
+│   └── ingest_us_acs.py       # pull richer ACS tables live (needs API key)
+├── tests/                     # pytest invariants & unit tests
+└── Paper/                     # ACL-style paper draft + figures
+```
+
+## Testing
+
+```bash
+PYTHONPATH=src python3 -m pytest tests/
+```
+
+The full pytest suite covers schema invariants (Pydantic model), reference-data
+loading, factory smoke checks, determinism, plausibility (age, employment,
+addresses, education), **adults-only enforcement** (no profiles under 18),
+linkage (households, employers, family graph), validation (KS gaps, replay,
+consistency, leakage), release-report freshness, normalized export quality, and
+reproducibility.
+
+## License
+
+- Code and SDK: **ProfileFoundry Citation License 1.0** (see `LICENSE`).
+  Public uses and redistributions must cite the ProfileFoundry paper when
+  available, or this repository until then. Machine-readable citation metadata
+  is in `CITATION.cff`.
+- Generated dataset: **CC-BY-4.0**.
+- Embedded reference data retains its upstream license; see
+  `data/reference/MANIFEST.md`.

profilefoundry-1.0.0/data/reference/MANIFEST.md

@@ -0,0 +1,139 @@
+# Reference-data manifest
+
+This manifest enumerates every external statistical source the ProfileFoundry
+generator and validator consume. Two tiers of data live in this tree:
+
+1. **`bootstrap/`** — hand-curated reference marginals committed to git.
+   These are small JSON/YAML files extracted from public summary tables of
+   the listed sources. They exist so the generator can run end-to-end without
+   network access and so the realism guarantees are reproducible from the
+   repo alone. Every bootstrap file links back to its source URL and the
+   retrieval date.
+
+2. **`derived/`** — Parquet tables produced by the `scripts/ingest_*.py`
+   scripts when they pull richer joint distributions live from authoritative
+   sources. These are **not** committed to git (see `.gitignore`); they are
+   regenerated locally and used by validators. The hashes are recorded in
+   `derived/HASHES.json` for reproducibility.
+
+3. **`cache/`** — raw downloaded payloads. Also not committed.
+
+When the live `derived/` tables are present, the generator and validator
+prefer them. When they are absent (a fresh checkout), bootstrap is used.
+
+---
+
+## Validation-tier locales (Q3)
+
+The five locales below get full distributional validation. Reference data for
+each must be present in `bootstrap/` and is auto-promoted to `derived/` when
+the corresponding ingestion script runs successfully.
+
+### US — United States
+
+| Quantity | Source | URL | Retrieved |
+|---|---|---|---|
+| Age × sex × race × Hispanic origin | ACS 2023 1-year, Table B01001 + B03002 | `https://api.census.gov/data/2023/acs/acs1` | TBD |
+| Education attainment by age × sex | ACS 2023, Table B15001 | (same API) | TBD |
+| Occupation distribution | ACS 2023, Table C24010 | (same API) | TBD |
+| Household income deciles | ACS 2023, Table B19001 | (same API) | TBD |
+| Household type × size | ACS 2023, Table B11016 | (same API) | TBD |
+| Marital status by age × sex | ACS 2023, Table B12002 | (same API) | TBD |
+| Given names by year × sex | SSA National Names | `https://www.ssa.gov/oact/babynames/names.zip` | TBD |
+| Surnames (frequency, race) | US Census 2010 Frequent Surnames | `https://www.census.gov/topics/population/genealogy/data/2010_surnames.html` | TBD |
+| ZCTA-level population | Census 2020 ZCTA totals | `https://www.census.gov/data/datasets/2020/dec/2020-zcta.html` | TBD |
+| Industry × employer size | County Business Patterns 2022 | `https://www.census.gov/programs-surveys/cbp/data/datasets.html` | TBD |
+
+License: US federal-government work, public domain.
+
+### UK — United Kingdom
+
+| Quantity | Source | URL | Retrieved |
+|---|---|---|---|
+| Age × sex × ethnicity | ONS 2021 Census, TS021, TS003 | `https://www.ons.gov.uk/datasets/create` | TBD |
+| Education attainment | ONS 2021 Census, TS067 | (ONS Create-a-Dataset) | TBD |
+| Occupation (SOC2020) | ONS 2021 Census, TS063 | (ONS Create-a-Dataset) | TBD |
+| Household income | ONS HBAI / Family Resources Survey, FY2022/23 | `https://www.gov.uk/government/collections/family-resources-survey` | TBD |
+| Household composition | ONS 2021 Census, TS003 | (ONS Create-a-Dataset) | TBD |
+| Marital/civil partnership status | ONS 2021 Census, TS002 | (ONS Create-a-Dataset) | TBD |
+| Given names by year (E&W) | ONS Baby Names datasets, latest year | `https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/livebirths/datasets/babynamesenglandandwalesbabynamesstatisticsgirls` | TBD |
+| Surnames | UK Office for National Statistics Surnames archive (or 2011 Census derived list, public republications) | `https://www.ons.gov.uk/` | TBD |
+
+License: Open Government Licence v3.0 (attribution required).
+
+### IN — India
+
+| Quantity | Source | URL | Retrieved |
+|---|---|---|---|
+| Age × sex by state | Census of India 2011, C-13 | `https://censusindia.gov.in/census.website/data/census-tables` | TBD |
+| Religion / caste category breakdown | Census 2011 + NSS rounds | (Census website) | TBD |
+| Education attainment | NSS 75th round (Education) | `https://www.mospi.gov.in/` | TBD |
+| Occupation (NCO-2015) | NSO PLFS 2022–23 | `https://www.mospi.gov.in/web/mospi/download-tables-data` | TBD |
+| Household monthly income (MPCE) | NSS HCES 2022–23 | (MoSPI) | TBD |
+| Household composition | Census 2011, H-series | (Census website) | TBD |
+| Marital status by age × sex | Census 2011, C-2 | (Census website) | TBD |
+| Given names | Public birth-record corpora (Bihar, Karnataka, etc. open data portals) + transliterated SSA-Indian-diaspora cross | Multiple; see ingest script | TBD |
+| Surnames | Indian community-wise surname lists (publicly compiled) | Multiple | TBD |
+
+License: most Government of India open data is published under the
+National Data Sharing and Accessibility Policy (NDSAP), open with attribution.
+
+### CA — Canada
+
+| Quantity | Source | URL | Retrieved |
+|---|---|---|---|
+| Age × sex × visible-minority status | StatCan 2021 Census, Table 98-10-0265 | `https://www12.statcan.gc.ca/census-recensement/2021/dp-pd/dt-td/` | TBD |
+| Education attainment | StatCan 2021 Census, Table 98-10-0418 | (same) | TBD |
+| Occupation (NOC 2021) | StatCan 2021 Census, Table 98-10-0434 | (same) | TBD |
+| Household income deciles | StatCan LFS / Income Survey 2022 | `https://www150.statcan.gc.ca/` | TBD |
+| Household composition | StatCan 2021 Census, Table 98-10-0124 | (same) | TBD |
+| Marital status | StatCan 2021 Census, Table 98-10-0125 | (same) | TBD |
+| Given / family names | Public open data + diaspora reference | TBD | TBD |
+
+License: Statistics Canada Open Licence (attribution required).
+
+### AU — Australia
+
+| Quantity | Source | URL | Retrieved |
+|---|---|---|---|
+| Age × sex × ancestry | ABS 2021 Census, TableBuilder | `https://www.abs.gov.au/census/find-census-data/quickstats/2021` | TBD |
+| Education | ABS 2021 Census, NEDU + HSCP | (TableBuilder) | TBD |
+| Occupation (ANZSCO 2022) | ABS 2021 Census, OCCP | (TableBuilder) | TBD |
+| Household income | ABS Survey of Income and Housing 2019–20 | `https://www.abs.gov.au/statistics/economy/finance/household-income-and-wealth-australia/latest-release` | TBD |
+| Household composition | ABS 2021 Census, HHCD | (TableBuilder) | TBD |
+| Marital status | ABS 2021 Census, MSTP | (TableBuilder) | TBD |
+| Names | TBD | TBD | TBD |
+
+License: ABS Creative Commons CC-BY 4.0 (attribution required).
+
+---
+
+## Light-validation locales (Q3)
+
+These three are generated against the same engine but only have light
+validation. Bootstrap marginals only; no Q12 KS thresholds enforced.
+
+### IE — Ireland
+- CSO Census 2022.
+- `https://www.cso.ie/en/census/`
+
+### NZ — New Zealand
+- Stats NZ Census 2023.
+- `https://www.stats.govt.nz/`
+
+### PH — Philippines
+- PSA 2020 Census.
+- `https://psa.gov.ph/statistics/census`
+
+---
+
+## Update protocol
+
+1. Modify `bootstrap/<locale>/<table>.json` (or add a new table) only if the
+   underlying source has been **re-pulled by the supervisor** with the URL +
+   date recorded above. Random hand-tuning of bootstrap values is forbidden
+   by `AGENTS.md` §6.
+2. Live ingestion: run `python scripts/ingest_<locale>.py` to populate
+   `derived/<locale>/`. The script writes a hash into `derived/HASHES.json`.
+3. Validators prefer `derived/` over `bootstrap/`; the generator does the
+   same so realism upgrades silently when you have richer tables available.

profilefoundry-1.0.0/data/reference/bootstrap/INDEX.json

@@ -0,0 +1,17 @@
+{
+  "generated_by": "hand-curated as part of Phase 1 bootstrap",
+  "generated_at": "2026-05-20",
+  "validation_locales": ["US", "UK", "IN", "CA", "AU"],
+  "light_validation_locales": ["IE", "NZ", "PH"],
+  "tables_per_locale": {
+    "US": ["age_sex", "race_ethnicity", "education", "marital", "household", "income", "regions"],
+    "UK": ["age_sex", "household", "race_ethnicity", "education", "marital"],
+    "IN": ["age_sex", "household", "education", "marital"],
+    "CA": ["age_sex", "household", "education", "marital"],
+    "AU": ["age_sex", "household", "education", "marital"],
+    "IE": ["age_sex", "household"],
+    "NZ": ["age_sex", "household"],
+    "PH": ["age_sex", "household"]
+  },
+  "fallback_strategy": "For tables absent in a locale, the generator falls back to US values, then to hard-coded defaults. The validator records every fallback usage in its consistency report so the dataset card can disclose them."
+}