starsim-data 2.0.0__tar.gz

starsim_data-2.0.0/.claude/skills/add-dataset/SKILL.md

@@ -0,0 +1,81 @@
+---
+name: add-dataset
+description: Add a new data source to starsim_data — another country's subnational data, a new contact source, populations, commuting or mobility flows, or any other dataset — by writing its download script and build step and wiring it into the registry, provenance, docs and tests. Use when asked to add, import or package data for a new location or source (e.g. "add Germany's districts", "import Canada's provinces from epydemix", "add the commuter matrix").
+---
+
+# Add a dataset to starsim_data
+
+A dataset here is three things: a **download script** that copies the upstream data into `raw_data/` as readable CSVs, a **build step** that turns those CSVs into a self-describing HDF5 store, and the **wiring** (license, docs, tests) that the repo's invariants require. The US states (`mistry2021_us`, `population_us`) are the worked example to copy: `scripts/download_epydemix.py`, and `build_population_us()` and the tables at the top of `scripts/build_hdf5.py`.
+
+Read `CLAUDE.md` first: the tier invariant, the registry, and "Keeping the docs in step" all apply.
+
+## 1. Settle these before writing code
+
+Ask Cliff about any of these you can't answer from the source itself. They decide everything after.
+
+- **License.** Find the exact license of the data (not of the repo it sits in: epydemix-data is GPL-3.0 as a whole, but its Census numbers are public domain). If it can't be redistributed, it can't be a store here; it can only be fetched from upstream at use time, as GADM is (`redistributable=False`). Record the license as it will appear in `NOTICE.md`.
+- **Tier.** Bundled (tier 0, in the wheel) only if it is small and most users need it; Cliff wants the wheel's size unchanged unless he agrees otherwise. Everything else is a fetched store (tier 1) in `fetched/`.
+- **Location keys.** A subnational unit is keyed `<Country>:<Unit>`, and the key must be the unit's `dot_name` in the node table, so that `resolve()`, `load_nodes()` and the new data agree. Check with `ssd.load_nodes(country, level=1).dot_name`. Only levels 0 and 1 are bundled. For level 2 and finer, get the names from `ssd.fetch_nodes(country, level=2)` (needs `[spatial]`), and if the source's names don't match them (e.g. `Landkreis Harburg` against `Harburg`), build a crosswalk from codes rather than fuzzy-matching names. Commit the crosswalk as a CSV in `raw_data/`.
+- **Layout.** Reuse one if it fits: `matrix` (square, by age), `population` and `mortality` (a year × sex × age array), `crude` (one value per year), `nodes`. A new shape, such as origin–destination flows, needs a new shaper in `starsim_data/loaders.py`, registered with `ssd.registry.register_layout()` and documented in `docs/08-architecture.qmd`. Return the same columns an existing dataset returns wherever the meaning is the same: `population_us` uses `population`'s `Time, Sex, AgeGrp, Value`, with `Sex='Both sexes'` because the source doesn't split by sex.
+- **Name.** Lowercase, source then scope: `mistry2021_us`, `population_us`. Don't put new locations into a store that is already published (for example, adding states to `mistry2021`): it changes the store's bytes, and its pinned download, for everyone.
+- **Upstream version.** Pin a commit, release or vintage, so that rerunning the download gives the same files.
+
+## 2. Download script: `scripts/download_<source>.py`
+
+Model it on `scripts/download_epydemix.py`:
+
+- A module docstring saying what it downloads, from where, into which files, and the command to run next.
+- The upstream version pinned in a constant (`COMMIT = ...`), and URLs built from it.
+- Output to `raw_data/`, as plain CSVs a person can read and diff: contact matrices as `raw_data/contact_matrices/<Country>__<Unit>/<source>_contacts_<kind>.csv` (headerless, square; the `__` stands for `:`, which isn't legal in a folder name everywhere), and anything else as one tidy CSV keyed by `location` (e.g. `raw_data/population_us.csv`: `location, age, population`).
+- Check every location against the node table before writing anything, and raise, listing the ones that don't match.
+- Any token goes in `scripts/.env` (git-ignored), as `download_wpp.py` does, and never in the code.
+- Read every CSV with `keep_default_na=False` or the `csv` module (Namibia's ISO-2 code is `NA`).
+- Keep downloading separate from building, so that the build and its `--check` run offline from the committed CSVs.
+
+Tell Cliff how big the download and the raw CSVs will be before running it. Raw CSVs are committed, so 30 MB of them is 30 MB in git history.
+
+## 3. Build step
+
+All fetched stores are listed in `scripts/build_hdf5.py`, which writes the manifest `starsim_data/data/fetched.json` from them. Add the new store there:
+
+- Add `name` to `FETCHED_SOURCES`, and bump `RELEASE` to a new `data-*` tag: the new store is downloaded from that GitHub release, which Cliff creates with `scripts/publish.py` once the change is on `main`.
+- `SOURCES[name]`: `source`, `source_url`, `via` (if it came through another project), `license`, `citation`, `note`. This is written into the store as its provenance.
+- `DESCRIPTIONS[name]`: one line for `list_datasets()`.
+- Contact matrices need nothing more than their CSVs and an `AGE_WIDTH` entry: sources are discovered from the file names.
+- Anything else needs a `build_<name>(check=False)` function, like `build_population_us()`. It writes one dataset per location key and a root `spec` attribute for the layout, for example `dict(layout='population', years=[...], sexes=[...], ages=[...], age_col='AgeGrp', value_col='Value')`, plus `provenance` from `stamp()`. Under `check`, it reads each value back and compares it with its CSV. Call it from `main()` when its name is passed to `--fetched`.
+- A bundled store goes into `starsim_data/data/` instead, needs an entry in `registry.DESCRIPTIONS`, and is discovered from its file. Bundling something new is Cliff's call (see Tier).
+
+Build only the new store: `python scripts/build_hdf5.py --check --fetched <name>`. Never rebuild a published store, since that changes its bytes and breaks its pinned download. Every run also rewrites the bundled `prem2021.h5` and `prem/prem2017.h5`, with a new build time and the same data, so restore them afterwards with `git restore starsim_data/data/prem2021.h5 starsim_data/data/prem/prem2017.h5` unless their data really changed.
+
+Levels are worked out from the keys (`United_States:Massachusetts` is level 1). `fetch()` reads any fetched store with no further code, and `load()` points to it from other datasets of the same layout ("it is in fetch(...)").
+
+## 4. Wiring
+
+Everything below is required: the tests check some of it, and none of it updates itself.
+
+- `NOTICE.md`: a section for a new source (or a bullet under the existing one for the same study), and the list of fetched stores in its introduction.
+- `tests/test_provenance.py::test_licenses`: the new dataset and its license.
+- Tests: one compact test of what's new, like `tests/test_load.py::test_us_states` (shape, schema, a sanity total, and the error that points to it). The offline suite reads fetched stores from `fetched/` through `tests/conftest.py`, so no network is needed. Update `tests/README.md` if a file's coverage changes.
+- Docs: `docs/03-datasets.qmd` (the schema and an example; downloading examples must be static ```` ```python ```` blocks), `docs/06-datasources.qmd` (coverage, license, limitations), `docs/02-tiers.qmd` (the tier table), `docs/10-building-the-data.qmd`, and `docs/04-locations.qmd` if naming changes. Update the comparison table in `docs/07-related-projects.qmd` if coverage changes.
+- `README.md` (the source table), `CHANGELOG.md` (one line), `raw_data/README.md`, `scripts/README.md`, and the commands in `CLAUDE.md`.
+
+## 5. Check
+
+```bash
+python scripts/build_hdf5.py --check
+pytest -m "not network"
+pytest tests/test_docs.py
+great-docs build    # if Quarto is available
+```
+
+Then try it the way a user would, including the errors: a misspelled unit, a unit the dataset lacks, and an argument that doesn't apply (it must be rejected, not ignored). Don't commit; leave everything for Cliff, and remind Cliff to create the new data release once the change is on `main`.
+
+## Worked sketch: Germany's districts (not built yet)
+
+Requested for MEmilio users: the roughly 400 districts (Kreise), with populations by age group, centroids, and the commuter matrix between them. Verify every URL, table number and license below before relying on it.
+
+- **Populations:** Destatis GENESIS-Online, table 12411-0017 (population by district and age group) or its successor. The API needs an account; put the token in `scripts/.env`. The license is probably Datenlizenz Deutschland – Namensnennung 2.0 (`dl-de/by-2-0`), which allows redistribution with attribution. Use the `population` layout with Germany's age groups, keyed `Germany:<Kreis>`.
+- **Commuter flows:** the Federal Employment Agency's commuter statistics (Pendlerstatistik), one spreadsheet per state. MEmilio's `getCommuterMobility` (in `memilio.epidata`) shows the URLs and how to parse them. Check the agency's terms of reuse. This needs a new layout, e.g. `flows`, returning `origin, destination, value` for a location (`Germany`), from a sparse array stored as three columns.
+- **Keys:** districts are identified by their AGS code (Amtlicher Gemeindeschlüssel). Get the level-2 `dot_name`s from `fetch_nodes('Germany', level=2)` and crosswalk AGS to `dot_name`, using codes where the boundary source has them and a committed crosswalk CSV where it doesn't.
+- **Places:** `load_nodes('Germany', level=2)` needs level 2 in the bundled node table, which is a `scripts/build_nodes.py` change and a size decision for Cliff. Otherwise `fetch_nodes()` already gives centroids at level 2.
+- **Size:** about 400 × 400 sparse flows and 400 × (6–21) age groups, so a fetched store of well under 1 MB.

starsim_data-2.0.0/.github/workflows/docs.yml

@@ -0,0 +1,110 @@
+name: CI Docs
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  build-docs:
+    name: "Build Docs"
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0 # Full history for accurate page timestamps
+
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.13"
+
+      - name: Install package and dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -e ".[dev,docs]"
+          python -m pip install great-docs
+
+      - name: Set up Quarto
+        uses: quarto-dev/quarto-actions/setup@v2
+
+      - name: Build docs
+        run: great-docs build
+
+      - name: Save docs artifact
+        uses: actions/upload-artifact@v7
+        with:
+          name: docs-html
+          path: great-docs/_site
+          include-hidden-files: true
+
+      - name: Upload build timings
+        uses: actions/upload-artifact@v7
+        with:
+          name: build-timings
+          path: great-docs/_site/build-timings.json
+
+  publish-docs:
+    name: "Publish Docs"
+    runs-on: ubuntu-latest
+    needs: "build-docs"
+    if: github.ref == 'refs/heads/main'
+    permissions:
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - uses: actions/download-artifact@v7
+        with:
+          name: docs-html
+          path: great-docs/_site
+
+      - name: Upload Pages artifact
+        uses: actions/upload-pages-artifact@v5
+        with:
+          path: great-docs/_site
+          include-hidden-files: true
+
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v5
+
+  preview-docs:
+    name: "Preview Docs"
+    runs-on: ubuntu-latest
+    needs: "build-docs"
+    if: github.event_name == 'pull_request'
+    permissions:
+      deployments: write
+      pull-requests: write
+    steps:
+      - uses: actions/download-artifact@v7
+        with:
+          name: docs-html
+          path: great-docs/_site
+
+      # Start deployment
+      - name: Configure pull release name
+        if: ${{ github.event_name == 'pull_request' }}
+        run: |
+          echo "RELEASE_NAME=pr-${{ github.event.number }}" >> $GITHUB_ENV
+
+      - name: Configure branch release name
+        if: ${{ github.event_name != 'pull_request' }}
+        run: |
+          # use branch name, but replace slashes. E.g. feat/a -> feat-a
+          echo "RELEASE_NAME=${GITHUB_REF_NAME//\//-}" >> $GITHUB_ENV
+
+      # Deploy
+      - name: Create Github Deployment
+        uses: bobheadxi/deployments@v1
+        id: deployment
+        if: ${{ !github.event.pull_request.head.repo.fork }}
+        with:
+          step: start
+          token: ${{ secrets.GITHUB_TOKEN }}
+          env: ${{ env.RELEASE_NAME }}
+          ref: ${{ github.head_ref }}
+          logs: "https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"

starsim_data-2.0.0/.github/workflows/publish.yml

@@ -0,0 +1,45 @@
+name: Publish
+
+# Publishes the package to PyPI when a `v*` release is published (e.g. `v2.0.0`), by
+# trusted publishing (no token; set up once on pypi.org). Make the release with
+# `python scripts/publish.py <data tag>`, which also creates the data release the
+# fetched stores are downloaded from. This refuses to publish a package whose fetched
+# stores can't be downloaded.
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  pypi:
+    name: Publish to PyPI
+    if: startsWith(github.event.release.tag_name, 'v')
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write # trusted publishing
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v7
+      - uses: actions/setup-python@master
+        with:
+          python-version: '3.14'
+      - name: Check the version matches the tag, and the fetched stores can be downloaded
+        run: |
+          python - <<'EOF'
+          import json, pathlib, re, urllib.request
+          tag = '${{ github.event.release.tag_name }}'
+          version = re.search(r"__version__ = '(.+)'", pathlib.Path('starsim_data/version.py').read_text())[1]
+          assert tag == f'v{version}', f'Tag {tag} does not match version {version}'
+          manifest = json.loads(pathlib.Path('starsim_data/data/fetched.json').read_text())
+          for name, entry in manifest.items():
+              with urllib.request.urlopen(urllib.request.Request(entry['url'], method='HEAD')) as response:
+                  assert response.status == 200, f'{name}: {entry["url"]} returned {response.status}'
+          print(f'Version {version}; {len(manifest)} fetched stores are downloadable')
+          EOF
+      - name: Build
+        run: |
+          pip install build
+          python -m build
+      - name: Publish
+        uses: pypa/gh-action-pypi-publish@release/v1

starsim_data-2.0.0/.github/workflows/tests.yml

@@ -0,0 +1,87 @@
+name: starsim_data CI
+
+on:
+  workflow_call:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  install_and_test:
+    name: Tests
+    timeout-minutes: 10
+    strategy:
+      fail-fast: false
+      max-parallel: 8
+      matrix:
+        os: ['ubuntu-latest']
+        python-version: ['3.14']
+    runs-on: ${{ matrix.os }}
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v7
+      - uses: actions/setup-python@master
+        with:
+          python-version: ${{ matrix.python-version }}
+          architecture: x64
+      - name: Install starsim_data
+        run: pip install -e .[dev]
+      - name: Run tests
+        # -m "not network" so the suite stays offline by default; the bundled data
+        # makes every core test hermetic, and that property is worth protecting.
+        run: pytest tests -v -m "not network" --durations=0 --junitxml=test-results.xml
+      - name: Publish test report
+        uses: mikepenz/action-junit-report@v6
+        if: always()
+        with:
+          report_paths: './test-results.xml'
+          check_name: 'Test Report (Python ${{ matrix.python-version }})'
+
+  check_data_is_reproducible:
+    name: Data rebuild is reproducible
+    timeout-minutes: 10
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v7
+      - uses: actions/setup-python@master
+        with:
+          python-version: '3.14'
+          architecture: x64
+      - name: Install starsim_data
+        run: pip install -e .[dev]
+      # The bundled .h5 stores are build artifacts committed to the repo, so they can
+      # drift from the raw data they claim to be built from. Rebuilding and validating
+      # here is what keeps `raw_data/` the actual source of truth.
+      - name: Rebuild contact matrices and validate against the CSVs
+        run: python scripts/build_hdf5.py --check
+      - name: Rebuild demographics and validate via the loader
+        run: python scripts/build_demographics.py --check
+      - name: Check locations.csv is up to date
+        run: python scripts/build_locations.py --check
+      - name: Tests still pass against the rebuilt data
+        run: pytest tests -q -m "not network"
+
+  # The node table is deliberately left out of the job above: rebuilding it pulls ~420 MB
+  # of boundary data from GitHub, which is too much to spend on every pull request. Run it
+  # by hand when the boundary sources change.
+  check_nodes_are_reproducible:
+    name: Node table rebuild is reproducible
+    if: github.event_name == 'workflow_dispatch'
+    timeout-minutes: 30
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v7
+      - uses: actions/setup-python@master
+        with:
+          python-version: '3.14'
+          architecture: x64
+      - name: Install starsim_data
+        run: pip install -e .[dev,spatial]
+      - name: Rebuild the node table and validate via the loader
+        run: python scripts/build_nodes.py --check
+      - name: Tests still pass against the rebuilt node table
+        run: pytest tests -q -m "not network"

starsim_data-2.0.0/.gitignore

@@ -0,0 +1,110 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# mypy / ruff / type checkers
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.ruff_cache/
+
+# IDEs
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Build artifacts (HDF5 data stores are generated from the CSVs; not committed)
+*.h5
+*.hdf5
+
+# ...except the bundled national-tier stores, which ship inside the package.
+# These are small (~MBs), built by scripts/build_hdf5.py, and committed so that
+# `pip install` is self-contained. Larger subnational tiers stay uncommitted.
+!starsim_data/data/*.h5
+!starsim_data/data/prem/*.h5
+
+# ...and the contact stores that are fetched on demand rather than bundled: committed
+# here, outside the package, and downloaded by fetch() from a pinned commit.
+!fetched/*.h5
+
+# UN WPP demographics source. The normalized CSVs (raw_data/demographics/wpp_*.csv)
+# and README are tracked as the demographic source of truth; only the bulky raw
+# downloads (xlsx/gz/zip) and logs are scratch and ignored.
+raw_data/demographics/*.xlsx
+raw_data/demographics/*.csv.gz
+raw_data/demographics/*.zip
+raw_data/demographics/*.log
+
+# great-docs build directory (ephemeral; great-docs.yml is the committed source)
+great-docs/
+
+# Quarto's execution cache. Deliberately not committed: letting CI execute the code cells
+# afresh means the docs build also checks that the documented examples still run.
+_freeze/

starsim_data-2.0.0/CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+All notable changes to `starsim_data` are recorded here. Versions follow [semantic versioning](https://semver.org); the *data* is versioned too, so a change to a bundled dataset is a release, not an amendment.
+
+## 2.0.0 (2026-10-22)
+
+- **Contact matrices for 187 locations by default.** `ssd.load('kenya')` returns the contact matrices of Prem et al. 2021 (`prem2021`), and for the ten locations that study left out (Japan, Australia, Haiti and seven others) those of Prem et al. 2017 (`prem2017`). Pass `which='prem2021'` for that study alone.
+- **A smaller package.** The other contact sources, `prem2017` and `mistry2021`, are downloaded the first time they are used, e.g. `ssd.fetch('france', which='mistry2021')`, rather than shipped with the package.
+- **Subnational populations and borders, offline.** `ssd.load_nodes('kenya', level=1)` now gives the population of every state or province (from WorldPop, 2020), and `ssd.adjacency('kenya', level=1)` says which of them share a border.
+- **Finer levels on request.** `ssd.fetch_nodes('kenya', level=2)` builds the same table (centroids, areas, populations and borders) at any level, e.g. Kenya's 290 sub-counties.
+- **US states.** `ssd.fetch('United States:Massachusetts', which='mistry2021_us')` gives a state's contact matrices (Mistry et al. 2021), and `which='population_us'` its population by single year of age (US Census Bureau, 2023). A subnational unit is named `'<country>:<unit>'`, as in the node table.
+- `fetch()` takes `kind=` for the contact matrices, and rejects arguments that do not apply.
+- The fetched stores are downloaded from a GitHub release rather than from raw.githubusercontent.com, which rate-limits heavily.
+- Requires Sciris 3.5.2 or later.
+
+## 1.0.0 (2026-08-08)
+
+Merged with the [laser-init](https://github.com/laser-base/laser-init) project with additional features for geospatial modeling, name resolution, etc.
+
+### Resolution and provenance
+
+- **Location resolution.** Locations can now be given as an ISO-2 or ISO-3 code, an official name, a translation, or a common abbreviation: `ssd.load('USA', ...)`, `ssd.load('KEN', ...)`, `ssd.load("Côte d'Ivoire", ...)` and `ssd.load('DRC', ...)` all work. Matching is exact against a table generated into `data/locations.csv`; fuzzy matching (via `sc.suggest`) is used only to suggest alternatives in the error message when a lookup fails, never to resolve one.
+- `ssd.resolve()`, `ssd.location_info()` and `ssd.list_all_locations()` for working with the location table directly.
+- `ssd.parse_level()`, accepting `'admin1'` / `'ADM2'` / `'3'` / `2`. Only level 0 has data today; this fixes the vocabulary before subnational data arrives.
+- **Provenance.** Every bundled store now records its source, URL, license, citation and build details, written into the HDF5 at build time so it cannot be separated from the data. Read it with `ssd.provenance(which)`, or from `df.attrs['provenance']` on any loaded frame. Tests assert that the licenses recorded in the data match the ones asserted in `NOTICE.md`.
+- **Revision-versioned demographics.** UN WPP data is now stored as `demographics-<revision>.h5`. `load()` uses the newest bundled revision by default; `load(..., revision=2024)` pins one, and `ssd.list_revisions()` lists them. A future WPP release becomes an additive rebuild rather than a silent change to data already in use.
+- `iso3`, `official_name` and `aliases` columns in `data/locations.csv`, generated by the new `scripts/build_locations.py` from `pycountry` (a build-time dependency only).
+- CI (`.github/workflows/tests.yml`), pytest markers (`network`, `slow`, `integration`), and this changelog. CI additionally rebuilds every store from `raw_data/` and re-runs the tests against the result, so the committed build artifacts cannot drift from their sources.
+
+### Data loading
+
+- **`fetch()` — the data that is deliberately not bundled.** `load()` is still instant, offline and frozen; `fetch()` closes the three gaps the README has always named: annual resolution instead of decadal, complete life tables (`qx`, `lx`, `ex`, ...) instead of just `mx`, and other projection variants instead of Medium only. It reads the UN's public bulk CSVs — no API token, unlike the Data Portal — and caches them. `ssd.fetch('kenya', which='deaths', years=(2000, 2025))`, `ssd.fetch('kenya', which='life_table', years=2020)`, `ssd.fetch('kenya', which='births', variant='High')`. For the four series that are also bundled it returns exactly the schema `load()` returns, verified against the bundled data; `life_table` and `indicators` are fetch-only. See `ssd.list_fetchable()`.
+- **A download cache**, with a provenance sidecar per file recording its URL, timestamp, size and SHA-256. `ssd.cache_dir()`, `ssd.cache_info()`, `ssd.cache_provenance()`, `ssd.clear_cache()`. The location defaults to a platform-appropriate directory and is overridable via `ssd.options.cache_dir` or `STARSIM_DATA_CACHE_DIR`.
+- **`ssd.options`**, following Starsim's `ss.options` conventions, with `set()`, `reset()`, `help()` and a `context()` manager. Settings are read from the environment but acted on lazily: importing the package still creates no directories, opens no files and emits no warnings.
+- **`plot()` — validation figures.** `ssd.plot('kenya')` draws a four-panel summary (age pyramid, crude birth rate, mortality curve, contact matrix); `ssd.plot('kenya', which='population')` draws one panel. The point is to catch the wrong country or the wrong units before the data reaches a simulation.
+- **A dataset registry** (`ssd.registry`) holding a descriptor per dataset — layout, tier, admin level, license, revisions, location count — with public `register_dataset()` and `register_layout()` hooks. `ssd.list_datasets(detailed=True)` returns it as a dataframe, and `list_datasets(tier=0)` selects just the bundled data.
+- **Tiers.** Datasets now declare whether they are bundled in the wheel (0), a downloaded release asset (1) or fetched from upstream (2). `load()` refuses anything above tier 0 and points at `fetch()`, so its "never touches the network" guarantee is now enforced by the code rather than by convention.
+
+### Nodes and distances
+
+- **The node table** (`ssd.load_nodes`), bundled. For every administrative unit at levels 0 and 1: `nodeid`, `location`, `iso3`, `level`, `name`, `pcode`, `dot_name`, `parent`, `lat`, `lon`, `area_km2`, `population`, `source`. 3,254 nodes across 184 countries in 1.4 MB, readable with no geospatial dependencies — which settles the question the plan left open: it bundles rather than shipping as a release asset. Built by `scripts/build_nodes.py` from UNOCHA COD-AB where it exists (the source of the p-codes, ~110 countries) and geoBoundaries CGAZ v6.0.0 elsewhere; the two disagree in places about how a country subdivides, so every row records which source it came from. `population` is national-only by design.
+- **`ssd.distances()`** — the great-circle distance matrix between node centroids, which with `load_nodes()` gives a gravity model its `P_i`, `P_j` and `D_ij`. The gravity formulation itself stays in the model.
+- **`ssd.fetch_shapes()`** — administrative boundaries with full geometry, from UNOCHA, geoBoundaries or GADM, normalized onto one schema and joining to the node table on `pcode`. `source=None` tries UNOCHA then geoBoundaries. **GADM is never reached automatically**: it is non-commercial only, so it must be named explicitly, and the constraint is enforced as data (`ShapeSource.redistributable`) rather than remembered — the node-table build refuses any source that sets it False.
+- **`ssd.fetch_raster()`, `ssd.aggregate_raster()`, `ssd.fetch_nodes()`** — WorldPop 1 km gridded population, and summing it inside each administrative unit. This is the step that turns a boundary file into a metapopulation. `ssd.rescale_to_national()` reconciles WorldPop's modeled totals with UN WPP's (they differ by about 1.2% for Kenya 2020).
+- **`ssd.choropleth()`**, and a `nodes` panel for `plot()` that draws centroids without needing any geospatial dependency.
+- **`[spatial]` extra** (`geopandas`, `pyogrio`, `rastertoolkit`), imported lazily: `import starsim_data` works without them, and only the spatial functions raise if they are missing.
+- **A documentation site**, built with [great-docs](https://posit-dev.github.io/great-docs/): ten narrative pages in `docs/` covering the tier model, the schema of every dataset, location resolution, the spatial tier, the source comparison, how this package relates to [epydemix-data](https://github.com/epistorm/epydemix-data), [laser-init](https://github.com/laser-base/laser-init) and the other projects distributing the same data, the internal architecture, provenance, and how the bundled files are generated — plus an API reference generated from the docstrings. Code examples are executed at build time, so the frames and figures shown are the package's actual output. Both the sidebar and the API reference are built by great-docs itself — from the pages' `guide-section` frontmatter and from the public API — so neither can fall out of step with the code; the only hand-written styling is three overrides great-docs has no setting for (figure scaling, navbar centering, and the Starsim footer); `.github/workflows/docs.yml` builds and deploys it. NOTICE.md gains entries for UNOCHA, geoBoundaries, WorldPop and GADM.
+
+
+## Version 0.1.0
+
+Initial release: UN WPP demographics and `prem2017` / `prem2021` / `mistry2021` contact matrices, bundled as HDF5 and read with `load()`.

starsim_data-2.0.0/CLAUDE.md

@@ -0,0 +1,108 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this is
+
+`starsim_data` distributes UN WPP demographics, contact matrices, and administrative geography for [Starsim](https://starsim.org) models. It returns dataframes and nothing else — no model code, no simulation logic. See `docs/07-related-projects.qmd` for where that line is drawn and why.
+
+## Commands
+
+```bash
+pip install -e ".[dev]"              # runtime + pytest + pycountry + great-docs
+pip install -e ".[dev,spatial]"      # also geopandas/pyogrio/rastertoolkit, needed by build_nodes.py
+
+pytest                               # the offline suite
+pytest -m "not network"              # what CI runs
+pytest -m network                    # also hits UN/WorldPop servers (a few hundred MB)
+pytest tests/test_load.py -k matrix  # one file, one pattern
+pytest tests/test_load.py::test_matrix
+
+great-docs build                     # -> great-docs/_site/ (needs Quarto)
+great-docs preview                   # live reload on localhost:3000
+```
+
+Markers are declared in `pyproject.toml`: `network`, `slow`, `integration`, `spatial` (the last skips automatically without the extra). `tests/conftest.py` redirects the cache to a temp dir so tests never touch the user's real one; set `STARSIM_DATA_CACHE_DIR` to reuse downloads across network-test runs.
+
+### Rebuilding the bundled data
+
+The `.h5` files in `starsim_data/data/` are **committed build artifacts**, generated from `raw_data/` by `scripts/`. Never hand-edit them. Every script takes `--check` (validate without writing), which is what CI runs to stop the artifacts drifting from their sources:
+
+```bash
+python scripts/build_locations.py --check     # derived columns of locations.csv, from pycountry
+python scripts/build_hdf5.py --check          # prem2021.h5, prem/prem2017.h5 and fetched.json, from raw_data/contact_matrices/ (--fetched also rebuilds fetched/*.h5)
+python scripts/build_demographics.py --check  # demographics, from raw_data/demographics/
+python scripts/build_nodes.py --check         # node table; ~1.4 GB cold, [spatial] required
+python scripts/build_nodes.py --limit 5       # a quick subset while developing
+python scripts/download_wpp.py                # refresh raw_data/demographics/ (needs a token in scripts/.env)
+python scripts/download_epydemix.py           # refresh the US states in raw_data/ (then build_hdf5.py --fetched mistry2021_us population_us)
+```
+
+`build_nodes.py` is excluded from the per-PR CI job because of its size; it has a `workflow_dispatch`-only job in `.github/workflows/tests.yml`.
+
+## Architecture
+
+### The tier invariant
+
+Every function belongs to exactly one tier, and this is the design constraint the rest of the package is arranged around:
+
+- **Tier 0** — `load()`, `load_nodes()`, `distances()`. Reads only what is inside the wheel. **Never touches the network**, so it is instant, offline and byte-identical for everyone on a given version.
+- **Tier 1** — `fetch()`, `fetch_shapes()`, `fetch_nodes()`, `fetch_raster()`. Downloads and caches, with a provenance sidecar per file.
+
+Tier is a field on the dataset descriptor and `load()` enforces it: asking `load()` for a tier-1 dataset raises and names `fetch()` instead. Do not add a network call to a tier-0 path — CI runs `-m "not network"` and `tests/test_docs.py` fails the build if an executed docs cell would fetch.
+
+### Registry-driven dispatch
+
+`registry.py` holds two tables, and nearly every extension goes through one of them rather than through an `if` branch:
+
+- `DATASETS` — name → `Dataset` descriptor (layout, tier, levels, license, revisions, location count, store). Bundled descriptors are **discovered** by opening each `.h5` and reading its root attributes, so the stores are self-describing: adding a new contact source is dropping CSVs in `raw_data/` and re-running `build_hdf5.py`. The contact sources that are fetched rather than bundled (`prem2017`, `mistry2021`, in `fetched/` outside the wheel) are described by the manifest `data/fetched.json`, which `build_hdf5.py` writes with each store's URL (a GitHub release asset), SHA-256, locations and provenance. The default contact matrices, `prem`, are `prem2021` plus the `prem2017` matrices for the ten locations it lacks (bundled in `data/prem/prem2017.h5`, a subfolder so discovery doesn't list it as its own dataset); discovery describes them from both stores, with a `loader` that reads whichever has the location.
+- `LAYOUTS` — layout name → the shaper that turns a stored array into a dataframe (`matrix`, `crude`, `mortality`, `population`, `nodes`).
+
+`load()` resolves the descriptor, picks the source for the revision, resolves the location scoped to that dataset, then hands off to the shaper with an `opts` dict carrying layout-specific arguments (`kind` for matrices, `level` for nodes). An argument a layout has no use for is **rejected, not ignored**. `register_dataset()`, `register_layout()` and `plotting.PLOTTERS` are the public hooks; a descriptor with a `loader` makes `load()` delegate entirely, which is how downstream code adds data this package doesn't ship.
+
+Descriptor discovery is cached — call `registry.refresh()` if the files change inside a running process (e.g. a test that rebuilds a store).
+
+### Import conventions
+
+Follow Starsim's convention: **`from` imports live only in `__init__.py`**. Every other module does `import starsim_data as ssd` and reaches everything as `ssd.load`, `ssd.registry.datasets()`, so modules can refer to each other freely without cycles. The import order in `__init__.py` is load-bearing: `registry` must exist before `loaders` (which registers layouts) and `fetching` (which registers tier-1 datasets).
+
+Importing the package must stay side-effect free — no directories created, no files opened, no warnings. `options` reads the environment at import but acts on it lazily; the cache dir is resolved and created only when something is written. The `[spatial]` imports are lazy for the same reason: `import starsim_data` works without geopandas, and only the spatial functions raise.
+
+### Invariants that are enforced as data, not as memory
+
+- **Provenance travels with the data.** `scripts/provenance.py` stamps source/URL/license/citation/build details into each store at build time; `load()` and `fetch()` put the same record in `df.attrs['provenance']`. `tests/test_provenance.py` asserts the licenses recorded in the stores match `NOTICE.md`.
+- **GADM is never redistributed.** Its `ShapeSource` sets `redistributable=False`, `fetch_shapes()` leaves it out of the UNOCHA→geoBoundaries fallback chain, and `build_nodes.py` refuses any source with that flag. Keep the constraint on the descriptor rather than in a comment or a conditional.
+- **Demographic stores are versioned by WPP revision**, as `demographics-<revision>.h5`. A new UN release is an additive rebuild, never an edit to data already in use.
+- **Namibia's ISO-2 code is the string `NA`.** Any CSV read must pass `keep_default_na=False` or use the `csv` module, or pandas turns Namibia into a missing value. There is a regression test for it in `tests/test_locations.py`.
+- **Location resolution is exact.** Fuzzy matching (`sc.suggest`) only builds the "did you mean" error message; it never resolves a name.
+
+### Licensing
+
+Code is MIT; each bundled dataset keeps its own upstream license (GPL-3.0 for the Prem matrices, Apache-2.0 for `mistry2021`, CC BY 3.0 IGO for WPP). `NOTICE.md` is the operative attribution text and `licenses/` holds the full upstream texts. If you add or change a data source, update `NOTICE.md` — a test checks it against what the stores record.
+
+## Documentation
+
+Narrative pages are `.qmd` files in `docs/`, built by [great-docs](https://posit-dev.github.io/great-docs/). There is no navigation or API-reference config to maintain: the sidebar comes from each page's `guide-section` frontmatter and the reference is discovered from the public API. Rules `tests/test_docs.py` enforces:
+
+- Every page needs `title` and a `guide-section` from `Getting started` / `The data` / `Internals`, and sections must stay **contiguous in filename order** — inserting a page in an earlier section means renumbering the later ones.
+- Renumbering is safe: site URLs strip the numeric prefix (`docs/09-provenance.qmd` → `user-guide/provenance.html`), so cross-page links are written as bare `provenance.qmd`.
+- Only the landing page (`01-getting-started.qmd`) prefixes links with `user-guide/`.
+- Executed `{python}` cells must not fetch anything; write downloading examples as static ` ```python ` blocks.
+
+`docs/08-architecture.qmd` is the long-form version of the Architecture section above, and stays in step with the code by executing against it.
+
+### Keeping the docs in step
+
+Docs are part of a change, not a follow-up to it. Before treating any change as done, check what it obliges you to update.
+
+**When you add or remove a docs page:**
+
+1. Give it `title` and a `guide-section`, then **renumber siblings so the sections stay contiguous** in filename order — a new `The data` page means bumping every `Internals` page. Use `git mv`; site URLs strip the numeric prefix, so no links break.
+2. **Do not add navigation config to `great-docs.yml`.** A new page needs no entry there, by design — the sidebar is built from `guide-section` and the reference from the public API, so neither can fall out of step with the code. `test_docs.py::test_config` fails if `sections` or a hand-written `reference.contents` shows up. Wanting to edit that file is usually a sign something else is wrong. The one real exception: introducing a *new* `guide-section` means adding it to `SECTIONS` in `tests/test_docs.py`, which defines both the allowed values and their order.
+3. Link it from at least one sibling page — otherwise it is reachable only from the sidebar.
+4. Update the hand-maintained places that count or list pages: the "N narrative pages" sentence in `CHANGELOG.md`, and the per-file table in `tests/README.md` when tests change.
+5. Run `pytest tests/test_docs.py`, then `great-docs build` if Quarto is available. The tests catch frontmatter, dead cross-links and accidental network use; only a real build catches a cell that raises.
+
+**When code changes:** the API reference regenerates itself, but prose does not. A new dataset, layout or source touches `docs/03-datasets.qmd` (schema), `docs/06-datasources.qmd` (coverage, licensing, limitations) and `NOTICE.md`; a new extension point touches `docs/08-architecture.qmd`. Docs cells execute against the package at build time, so a stale *example* fails docs CI — but a stale *sentence* will not, and nothing will catch it for you.
+
+README links to the built site are `https://starsimhub.github.io/starsim_data/user-guide/<name>.html`.