pynhts 0.1.1__tar.gz

pynhts-0.1.1/.github/workflows/ci.yml

@@ -0,0 +1,18 @@
+name: ci
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        python: ["3.10", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: pip install -e ".[dev]"
+      - run: ruff check src tests
+      - run: mypy
+      - run: pytest   # data/network tests are excluded by default (no microdata in CI)

pynhts-0.1.1/.github/workflows/release.yml

@@ -0,0 +1,36 @@
+name: release
+# Publishes to PyPI with Trusted Publishing (OIDC): no token or password is stored anywhere.
+# Trigger: publishing a GitHub release, or running the workflow by hand (workflow_dispatch).
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python -m pip install build twine
+      - run: python -m build
+      - run: python -m twine check dist/*
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

pynhts-0.1.1/.gitignore

@@ -0,0 +1,28 @@
+# data and caches: never commit microdata or third-party copyrighted PDFs
+local_data/
+data_raw/
+*.csv
+*.parquet
+*.zip
+*.feather
+*.arrow
+*.parquet.*
+*.sas7bdat
+*.xpt
+*.dbf
+*.RData
+*.rds
+lit/*.pdf
+.cache/
+# python
+__pycache__/
+*.py[cod]
+.venv/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+dist/
+build/
+*.egg-info/
+.coverage
+htmlcov/

pynhts-0.1.1/CHANGELOG.md

@@ -0,0 +1,24 @@
+# Changelog
+
+## 0.1.1 (2026-10-17)
+
+First version published on PyPI (0.1.0 exists only as a GitHub release). README links made absolute so
+that they work on the PyPI page; no code changes.
+
+## 0.1.0 (2026-10-17)
+
+First release. Scope: NHTS **2017** only.
+
+* `pynhts.io`: SHA-256-pinned download of the official archives, offline mode, adoption of local
+  files, deterministic and idempotent CSV to Parquet conversion, replicate weights as float32.
+* `pynhts.core`: codebook-derived typed schema (hierarchical, label-aware column ownership);
+  `NHTSData` loader with lazy columns, validated many-to-one joins and per-column masking of
+  survey codes (`missing="nan"|"keep"`, `skip="keep"|"nan"`); `decode_labels`.
+* `pynhts.stats`: stratified jackknife (JKn, 98 replicates) totals, means, proportions and ratios
+  with standard error, margin of error, confidence interval, domains and groups;
+  `ratio_across` for rates built from two weight systems; Student-t quantile without SciPy.
+* `pynhts.recipes`: replication of 24 figures of FHWA's *Summary of Travel Trends: 2017 NHTS*
+  (20 reproduced, 2 estimate-only, 2 not reproduced; gaps documented, none tuned away).
+* `pynhts.ml`: NumPy arrays with sample weights, PyTorch tensors, household graph edges.
+* Command line: `pynhts fetch | register | info | validate`.
+* Documentation: methods, data pitfalls, generated validation report. JOSS paper draft.

pynhts-0.1.1/CITATION.cff

@@ -0,0 +1,17 @@
+cff-version: 1.2.0
+message: "If you use pynhts, please cite it as below."
+title: "pynhts: survey-weight-aware access to US National Household Travel Survey microdata"
+version: 0.1.1
+date-released: 2026-10-17
+license: BSD-3-Clause
+repository-code: "https://github.com/sabernaseralavi-60/pynhts"
+authors:
+  - family-names: Naseralavi
+    given-names: Seyed Saber
+    orcid: "https://orcid.org/0000-0002-5392-910X"
+    affiliation: "Department of Civil Engineering, Shahid Bahonar University of Kerman, Kerman, Iran"
+keywords: [NHTS, travel survey, transportation, replicate weights, jackknife]
+abstract: >-
+  Python library that downloads and verifies the 2017 NHTS public-use files, converts them to
+  typed Parquet, merges the tables without silent errors, computes jackknife (JKn) standard errors
+  and margins of error, and reproduces published FHWA tables.

pynhts-0.1.1/CONTRIBUTING.md

@@ -0,0 +1,18 @@
+# Contributing
+
+Thanks for helping. Short rules:
+
+1. **Set up:** `uv venv && uv pip install -e ".[dev]"`.
+2. **Checks before a pull request:** `ruff check src tests`, `mypy`, `pytest`. The default test
+   run needs no data. Tests that need the real files carry `@pytest.mark.data`
+   (`pytest -m data`, with the files registered via `pynhts register <folder>`); the live
+   download test is `@pytest.mark.network` and is manual.
+3. **Never commit microdata** or third-party PDFs (`.gitignore` already excludes them).
+4. **Do not tune the library to match a published figure.** If a figure does not reproduce,
+   record it with a note in `src/pynhts/recipes/data/*.yaml` and set its `expected` status
+   honestly. Explanations of the documented gaps are very welcome as issues.
+5. **New waves:** add a registry file (`_registry/<wave>.yaml`), generate the schema with
+   `tools/build_schema.py` from the official codebook, and add golden tests before any
+   estimator is touched.
+6. Regenerate `docs/validation.md` with `python tools/make_validation_report.py` after any change
+   that can move an estimate.

pynhts-0.1.1/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2026, pynhts developers
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

pynhts-0.1.1/PKG-INFO

@@ -0,0 +1,130 @@
+Metadata-Version: 2.5
+Name: pynhts
+Version: 0.1.1
+Summary: Typed, survey-weight-aware access to US National Household Travel Survey microdata.
+Project-URL: Homepage, https://github.com/sabernaseralavi-60/pynhts
+Project-URL: Issues, https://github.com/sabernaseralavi-60/pynhts/issues
+Project-URL: Changelog, https://github.com/sabernaseralavi-60/pynhts/blob/main/CHANGELOG.md
+License: BSD-3-Clause
+License-File: LICENSE
+Keywords: NHTS,jackknife,replicate weights,transportation,travel survey
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24
+Requires-Dist: pandas>=2.0
+Requires-Dist: platformdirs>=3.10
+Requires-Dist: pooch>=1.8
+Requires-Dist: pyarrow>=14
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: mypy; extra == 'dev'
+Requires-Dist: openpyxl; extra == 'dev'
+Requires-Dist: pandas-stubs; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
+Provides-Extra: graph
+Requires-Dist: torch-geometric>=2.4; extra == 'graph'
+Requires-Dist: torch>=2.0; extra == 'graph'
+Provides-Extra: torch
+Requires-Dist: torch>=2.0; extra == 'torch'
+Description-Content-Type: text/markdown
+
+# pynhts
+
+[![ci](https://github.com/sabernaseralavi-60/pynhts/actions/workflows/ci.yml/badge.svg)](https://github.com/sabernaseralavi-60/pynhts/actions/workflows/ci.yml)
+![status](https://img.shields.io/badge/status-alpha-orange)
+![python](https://img.shields.io/badge/python-3.10%2B-blue)
+![license](https://img.shields.io/badge/license-BSD--3--Clause-green)
+
+Typed, survey-weight-aware Python access to US National Household Travel Survey (NHTS)
+microdata.
+
+> **Scope and status (0.1.0, alpha).** Only the **2017** wave is supported; other waves come
+> after this one is solid. The API may still change. Results are validated
+> against FHWA's published tables: 20 of 24 figures reproduce completely and 4 gaps are reported
+> openly in [docs/validation.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/validation.md).
+
+## Install
+
+```
+pip install pynhts
+pynhts fetch --replicates        # download, verify (SHA-256) and convert; about 0.4 GB of Parquet, plus 1.8 GB of raw CSV that you may delete afterwards
+```
+
+Already have the NHTS 2017 CSV files? `pynhts register <folder>` adopts them after checking
+their hashes.
+
+## What it does
+
+```python
+import pynhts
+data = pynhts.load(2017)
+
+# lazy, validated tables: columns are joined from the table that owns them
+trips = data.trips(["TRPMILES", "TRPTRANS", "HHFAMINC", "R_AGE"])
+
+# weighted statistics with jackknife (JKn) standard errors and 95 % margins of error
+s = data.survey("person", ["R_AGE_IMP", "DRIVER"])
+s.total(domain=s.rows["DRIVER"] == "01")        # 223,277 thousand drivers, MOE 827 thousand
+data.survey("trip", ["TRPMILES"]).mean("TRPMILES")   # 10.70 miles, MOE 0.40
+
+# replicate a published table and see every gap
+from pynhts import recipes
+print(recipes.to_markdown(recipes.run("stt2017", data)))
+
+# machine-learning bridge
+from pynhts import ml
+arr = ml.to_arrays(trips, ["TRPMILES", "R_AGE"], target="TRPTRANS", weight="WTTRDFIN")
+```
+
+| Module | Purpose |
+|---|---|
+| `pynhts.io` | verified download, offline mode, deterministic Parquet cache |
+| `pynhts.core` | codebook-derived schema, `NHTSData` loader, missing-code handling |
+| `pynhts.stats` | JKn totals, means, proportions, ratios; margins of error |
+| `pynhts.recipes` | replication of published tables with honest status per figure |
+| `pynhts.ml` | NumPy / PyTorch / graph export (PyTorch optional) |
+
+## Why a library
+
+Correct NHTS analysis has many silent traps (a data server that returns HTTP 200 for missing
+files, string codes with leading zeros, `-1` skips that are not missing data, 107 copied
+parent columns in the child tables, a vehicle `PERSONID` that is not the person key, ...).
+They are documented in [docs/data_notes.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/data_notes.md) and each is covered by a test.
+
+## Validation
+
+* All 3,807 coded values of the official codebook (counts and weighted totals) reproduce.
+* 24 figures from Tables 1 and 3 of FHWA's *Summary of Travel Trends: 2017 NHTS*, with margins
+  of error: 20 reproduce completely, 2 reproduce the estimate but not the margin of error,
+  2 do not reproduce. See [docs/validation.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/validation.md).
+
+## Documentation
+
+[Methods](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/methods.md) · [Data notes](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/data_notes.md) · [Validation](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/validation.md)
+· [Roadmap](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/roadmap.md) · [Contributing](https://github.com/sabernaseralavi-60/pynhts/blob/main/CONTRIBUTING.md) · [Changelog](https://github.com/sabernaseralavi-60/pynhts/blob/main/CHANGELOG.md)
+
+## Data
+
+The NHTS microdata are public data of the US Federal Highway Administration
+(<https://nhts.ornl.gov>). They are **never** bundled with or committed to this repository.
+
+## Development
+
+```
+uv venv && uv pip install -e ".[dev]"
+pytest                             # unit tests, no data needed
+pytest -m data                     # real-data tests (files registered in the pynhts cache)
+pytest -m network -o addopts=-ra   # live ORNL download, manual only
+```
+
+## Citation
+
+See [CITATION.cff](https://github.com/sabernaseralavi-60/pynhts/blob/main/CITATION.cff). License: BSD-3-Clause.

pynhts-0.1.1/README.md

@@ -0,0 +1,92 @@
+# pynhts
+
+[![ci](https://github.com/sabernaseralavi-60/pynhts/actions/workflows/ci.yml/badge.svg)](https://github.com/sabernaseralavi-60/pynhts/actions/workflows/ci.yml)
+![status](https://img.shields.io/badge/status-alpha-orange)
+![python](https://img.shields.io/badge/python-3.10%2B-blue)
+![license](https://img.shields.io/badge/license-BSD--3--Clause-green)
+
+Typed, survey-weight-aware Python access to US National Household Travel Survey (NHTS)
+microdata.
+
+> **Scope and status (0.1.0, alpha).** Only the **2017** wave is supported; other waves come
+> after this one is solid. The API may still change. Results are validated
+> against FHWA's published tables: 20 of 24 figures reproduce completely and 4 gaps are reported
+> openly in [docs/validation.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/validation.md).
+
+## Install
+
+```
+pip install pynhts
+pynhts fetch --replicates        # download, verify (SHA-256) and convert; about 0.4 GB of Parquet, plus 1.8 GB of raw CSV that you may delete afterwards
+```
+
+Already have the NHTS 2017 CSV files? `pynhts register <folder>` adopts them after checking
+their hashes.
+
+## What it does
+
+```python
+import pynhts
+data = pynhts.load(2017)
+
+# lazy, validated tables: columns are joined from the table that owns them
+trips = data.trips(["TRPMILES", "TRPTRANS", "HHFAMINC", "R_AGE"])
+
+# weighted statistics with jackknife (JKn) standard errors and 95 % margins of error
+s = data.survey("person", ["R_AGE_IMP", "DRIVER"])
+s.total(domain=s.rows["DRIVER"] == "01")        # 223,277 thousand drivers, MOE 827 thousand
+data.survey("trip", ["TRPMILES"]).mean("TRPMILES")   # 10.70 miles, MOE 0.40
+
+# replicate a published table and see every gap
+from pynhts import recipes
+print(recipes.to_markdown(recipes.run("stt2017", data)))
+
+# machine-learning bridge
+from pynhts import ml
+arr = ml.to_arrays(trips, ["TRPMILES", "R_AGE"], target="TRPTRANS", weight="WTTRDFIN")
+```
+
+| Module | Purpose |
+|---|---|
+| `pynhts.io` | verified download, offline mode, deterministic Parquet cache |
+| `pynhts.core` | codebook-derived schema, `NHTSData` loader, missing-code handling |
+| `pynhts.stats` | JKn totals, means, proportions, ratios; margins of error |
+| `pynhts.recipes` | replication of published tables with honest status per figure |
+| `pynhts.ml` | NumPy / PyTorch / graph export (PyTorch optional) |
+
+## Why a library
+
+Correct NHTS analysis has many silent traps (a data server that returns HTTP 200 for missing
+files, string codes with leading zeros, `-1` skips that are not missing data, 107 copied
+parent columns in the child tables, a vehicle `PERSONID` that is not the person key, ...).
+They are documented in [docs/data_notes.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/data_notes.md) and each is covered by a test.
+
+## Validation
+
+* All 3,807 coded values of the official codebook (counts and weighted totals) reproduce.
+* 24 figures from Tables 1 and 3 of FHWA's *Summary of Travel Trends: 2017 NHTS*, with margins
+  of error: 20 reproduce completely, 2 reproduce the estimate but not the margin of error,
+  2 do not reproduce. See [docs/validation.md](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/validation.md).
+
+## Documentation
+
+[Methods](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/methods.md) · [Data notes](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/data_notes.md) · [Validation](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/validation.md)
+· [Roadmap](https://github.com/sabernaseralavi-60/pynhts/blob/main/docs/roadmap.md) · [Contributing](https://github.com/sabernaseralavi-60/pynhts/blob/main/CONTRIBUTING.md) · [Changelog](https://github.com/sabernaseralavi-60/pynhts/blob/main/CHANGELOG.md)
+
+## Data
+
+The NHTS microdata are public data of the US Federal Highway Administration
+(<https://nhts.ornl.gov>). They are **never** bundled with or committed to this repository.
+
+## Development
+
+```
+uv venv && uv pip install -e ".[dev]"
+pytest                             # unit tests, no data needed
+pytest -m data                     # real-data tests (files registered in the pynhts cache)
+pytest -m network -o addopts=-ra   # live ORNL download, manual only
+```
+
+## Citation
+
+See [CITATION.cff](https://github.com/sabernaseralavi-60/pynhts/blob/main/CITATION.cff). License: BSD-3-Clause.

pynhts-0.1.1/docs/data_notes.md

@@ -0,0 +1,67 @@
+# NHTS 2017: pitfalls found while building pynhts
+
+Every item below was checked against the files or the official documents, and each is covered
+by a test. They are the reason the library exists.
+
+## Download and files
+
+* **The ORNL server answers HTTP 200 with an HTML page for files that do not exist.** A status
+  check proves nothing. pynhts pins SHA-256 hashes for both archives and every member
+  (`src/pynhts/_registry/2017.yaml`).
+* The files dated 2019 and the archive currently served (modified 2023) are byte-identical:
+  all four core CSV hashes match.
+* There is **no trip-level replicate file**. `perwgt.csv` holds two blocks, the person weight
+  (`WTPERFIN`, `WTPERFIN1..98`) and the travel-day weight (`WTTRDFIN`, `WTTRDFIN1..98`); trips
+  get replicates by joining on `HOUSEID` + `PERSONID`.
+
+## Codes
+
+* Coded columns are **strings with leading zeros** (`"01"`). Letting a reader infer types
+  silently turns them into integers. pynhts types every column from the codebook.
+* Four different kinds of non-answer exist: `-9` not ascertained, `-8` don't know, `-7`
+  refused, and `-1` **appropriate skip** (the question did not apply; not missing). `-88`/`-77`
+  are "don't know"/"prefer not to answer" in numeric fields where -8/-7 could be real values.
+* The codebook counts `-88` together with the `-8` rows (and `-77` with `-7`) in `YEARMILE` and
+  `OD_READ`: 149 and 144 rows hold -8/-7 although only -88/-77 are listed. Masking only the
+  listed codes would leave them in as negative numbers. pynhts masks them too.
+* The codebook lists only the codes **observed in each table**, so the same variable can
+  declare different codes in the household and trip files. pynhts masks per column from the
+  schema of the table the column is read from, and a test proves that the real data contain no
+  undeclared missing or skip code.
+
+## Tables and keys
+
+* **Child files carry copies of parent columns** (107 of them: 35 in the person file, 35 in
+  the vehicle file, 37 in the trip file). A naive merge creates `_x`/`_y` columns or picks the
+  wrong weight. The loader reads copies from the child table and joins only genuinely missing
+  columns, with a validated many-to-one join. A test proves that every copy equals its parent.
+* **The same name can be a different variable.** In the vehicle file `PERSONID` is the person
+  who mainly drives the vehicle, not the person key, and `VEHID` is the vehicle's own key.
+  Ownership is therefore decided by name *and* label, and only toward ancestors.
+* Vehicles use the **household** weight `WTHHFIN`.
+
+## Weights and variance
+
+* The 2017 weighting report builds 98 jackknife replicates (14 variance strata of 7 units), so
+  Var = 6/7 * sum of squared replicate deviations. FHWA's margin of error equals the standard
+  error times t(0.975, 98) = 1.984 (checked against the published margins of error).
+* `WTTRDFIN` equals 365 x `WTPERFIN`, in the full weights and in every replicate (to 1e-7,
+  float32 storage). All trips of one person share one weight, which lets trip estimates run on
+  264,234 person-days instead of a 923,572 x 98 matrix.
+* Replicates are stored as float32; they are cast to float64 before any squaring.
+* Rates *per household* published by FHWA are person-weighted totals divided by
+  household-weighted households. They are ratios across two weight systems, which
+  `pynhts.stats.ratio_across` supports.
+
+## Variables
+
+* `R_AGE` leaves 496 persons without an age; the report's age bands use the imputed age
+  `R_AGE_IMP`.
+* The public person file starts at age 5. The report's printed total of persons (321,419
+  thousand) exceeds the sum of its own age bands (301,599) by 19,820 thousand; the weights
+  reproduce the sum of the bands exactly. The cause is not established.
+* 656 trips have `TRPMILES = -9`. They count as trips but add no miles.
+* The 2017 survey changed how trip distance is obtained (shortest route between geocoded
+  points instead of self-report). The report therefore prints "original" and "adjusted"
+  2017 values; pynhts reproduces the **original** ones, which are what the public variable
+  `TRPMILES` contains.

pynhts-0.1.1/docs/index.md

@@ -0,0 +1,53 @@
+# pynhts
+
+Typed, survey-weight-aware Python access to US National Household Travel Survey (NHTS)
+microdata. Phase 1 covers the **2017** wave.
+
+## Install
+
+```
+pip install pynhts
+```
+
+## Five-minute tour
+
+```python
+import pynhts
+
+data = pynhts.load(2017)           # downloads, verifies (SHA-256) and converts on first use
+
+# 1. Lazy, validated tables; columns are joined from the table that owns them
+trips = data.trips(["TRPMILES", "TRPTRANS", "HHFAMINC", "R_AGE"])
+
+# 2. Correct weighted statistics with margins of error (jackknife, 98 replicate weights)
+s = data.survey("person", ["R_AGE_IMP", "DRIVER"])
+s.total(domain=s.rows["DRIVER"] == "01")        # 223,277 thousand drivers, MOE 827 thousand
+
+t = data.survey("trip", ["TRPMILES"])
+t.mean("TRPMILES")                              # 10.70 miles, MOE 0.40
+
+# 3. Replicate published tables and see every gap
+from pynhts import recipes
+print(recipes.to_markdown(recipes.run("stt2017", data)))
+
+# 4. Hand data to machine-learning code
+from pynhts import ml
+arr = ml.to_arrays(trips, ["TRPMILES", "R_AGE"], target="TRPTRANS", weight="WTTRDFIN")
+```
+
+Command line: `pynhts fetch --replicates`, `pynhts register <folder-with-your-CSVs>`,
+`pynhts info`, `pynhts validate`.
+
+## Where the data live
+
+Raw CSV and Parquet files go to `$PYNHTS_CACHE_DIR` or the user cache directory. They are
+never part of the package or of the git repository. Files you already have can be adopted
+with `pynhts register`, after their hashes are checked. After conversion the raw CSVs can be
+deleted: the Parquet files are enough.
+
+## Pages
+
+* [Validation against published tables](validation.md): what reproduces and what does not.
+* [Methods](methods.md): weights, jackknife variance, margins of error.
+* [Data notes](data_notes.md): the NHTS 2017 pitfalls found while building the library.
+* [Roadmap](roadmap.md)

pynhts-0.1.1/docs/methods.md

@@ -0,0 +1,57 @@
+# Methods
+
+## Estimators
+
+`data.survey(table, columns)` returns a `Survey`: the rows of a table bound to the 98 replicate
+weights of the unit that carries the weight.
+
+| Table | Weight | Replicate unit |
+|---|---|---|
+| household, vehicle | `WTHHFIN` | household |
+| person | `WTPERFIN` | person |
+| trip | `WTTRDFIN` | person-day |
+
+Supported estimators, each with optional `domain=` (subpopulation) and `by=` (groups):
+
+* `total(y)`: weighted total (`y=None` counts rows),
+* `mean(y)`: ratio of the weighted total of `y` to the weighted count of rows with a value,
+* `proportion(condition)`: mean of a boolean,
+* `ratio(numerator, denominator)`: ratio of two weighted totals,
+* `ratio_across(survey_a, y_a, survey_b, y_b)`: ratio between two surveys that share the
+  replicate index (person-weighted trips per household-weighted household).
+
+Every result row has `estimate`, `se`, `moe`, `ci_low`, `ci_high`, `cv` and `n` (rows
+contributing).
+
+## Variance
+
+For an estimate `theta` with replicate estimates `theta_1 ... theta_98`:
+
+    Var(theta) = (6/7) * sum_r (theta_r - theta)^2          (stratified jackknife, JKn)
+    MOE        = t(0.975, 98) * sqrt(Var)                   (95 %)
+
+The factor 6/7 = (n_h - 1)/n_h comes from the 14 variance strata of 7 variance units described
+in the 2017 NHTS weighting report. For ratios, the ratio is recomputed under every replicate
+weight, so the variance reflects the variability of numerator and denominator jointly.
+`level=` changes the confidence level; the degrees of freedom default to 98 as in FHWA's
+tables.
+
+## Rows and units
+
+All rows of one unit share one weight, so `sum_rows w_row * y_row = sum_units W_u * (sum of y
+in unit u)`. The engine aggregates row values to the unit once (`numpy.bincount`) and works
+on a matrix of at most 264,234 x 98 doubles, even for 923,572 trips.
+
+## Missing values
+
+A row whose analysis variable is NA is dropped from numerator and denominator (complete cases
+inside the domain), like `na.rm = TRUE` in R's `survey` package. Load tables with
+`missing="nan"` (default) so that survey codes become NA. This is a statement about the
+estimator, not about nonresponse bias: item nonresponse can bias an estimate and pynhts does not
+adjust for it.
+
+## What is not provided
+
+Variance for regression coefficients and other model-based estimates is not implemented yet;
+the replicate weights are available (`data.replicate_weights`) for users who refit a model once
+per replicate. Only the 2017 wave is supported.

pynhts-0.1.1/docs/roadmap.md

@@ -0,0 +1,24 @@
+# Roadmap
+
+## Done (version 0.1.0, NHTS 2017)
+
+* Hash-pinned download, offline mode, adoption of local files (`pynhts.io`).
+* Codebook-derived typed schema, deterministic idempotent Parquet cache with float32 replicate
+  weights (`pynhts.io`, `pynhts.core.schema`).
+* `NHTSData` loader with lazy columns, validated joins and per-column missing-code masking
+  (`pynhts.core`).
+* Jackknife estimators with margins of error and a cross-survey ratio (`pynhts.stats`).
+* Recipe that reproduces 24 figures of FHWA's *Summary of Travel Trends: 2017 NHTS*
+  (`pynhts.recipes`); 20 reproduce completely, 4 gaps are documented.
+* Arrays, PyTorch tensors and household graph edges (`pynhts.ml`); command line interface.
+
+## Not done yet
+
+* **Other waves** (2022 NextGen, 2009, 2001, 1995). Deliberately postponed until the 2017
+  schema, merge logic and weighting were validated. The registry and schema layers are
+  per-wave, so a new wave adds files, not code paths.
+* Variance of regression coefficients (refit once per replicate weight).
+* More recipes: other tables of the Summary of Travel Trends and seminal NHTS papers.
+* A hosted documentation site and a PyPI release.
+* Explaining the four documented gaps (see [validation](validation.md)); reports from users
+  who can are welcome.

pynhts-0.1.1/docs/validation.md

@@ -0,0 +1,48 @@
+# Validation against published NHTS 2017 tables
+
+*Generated by `tools/make_validation_report.py` on 2026-10-17 with pynhts 0.1.1. Do not edit by hand.*
+
+**Source:** McGuckin, N. and Fucci, A. (2018). Summary of Travel Trends: 2017 National Household Travel Survey. Federal Highway Administration.
+
+**Scorecard (24 published figures):** 20 reproduced, 2 estimate reproduced but margin of error differs, 2 not reproduced.
+
+Status rule: the estimate must match the printed figure to half a unit of its last printed digit; the 95 % margin of error (MOE = t(0.975, 98) x JKn standard error) must match to the same rounding or 0.1 %, whichever is larger. Figures are in the report's units (thousands for Table 1, daily rates for Table 3). The *MOE diff* column is (ours - published) / published.
+
+| Table | Quantity | Published | MOE pub. | Estimate | MOE | MOE diff | Status |
+|---|---|---:|---:|---:|---:|---:|---|
+| 1a | Households, all (thousands) | 118,208 |  | 118,208 | 0 |  | reproduced |
+| 1a | Households, 1 person (thousands) | 32,952 |  | 32,952 | 0 |  | reproduced |
+| 1a | Households, 2 persons (thousands) | 40,056 |  | 40,056 | 0 |  | reproduced |
+| 1a | Households, 3 persons (thousands) | 18,521 | 97 | 18,521 | 97 | +0.4% | reproduced |
+| 1a | Households, 4+ persons (thousands) | 26,679 | 97 | 26,679 | 97 | +0.4% | reproduced |
+| 1b | Persons, all (thousands) | 321,419 | 0 | 301,599 | 0 |  | not_reproduced |
+| 1b | Persons under 16 (thousands) | 45,498 | 756 | 45,498 | 756 | +0.0% | reproduced |
+| 1b | Persons 16-19 (thousands) | 17,755 | 945 | 17,755 | 945 | +0.0% | reproduced |
+| 1b | Persons 20-34 (thousands) | 64,339 | 954 | 64,339 | 954 | -0.0% | reproduced |
+| 1b | Persons 35-64 (thousands) | 126,350 | 985 | 126,350 | 986 | +0.1% | reproduced |
+| 1b | Persons 65+ (thousands) | 47,657 | 0 | 47,657 | 0 |  | reproduced |
+| 1b | Persons 16+ (thousands) | 256,101 | 756 | 256,101 | 756 | +0.0% | reproduced |
+| 1c | Drivers (thousands) | 223,277 | 827 | 223,277 | 827 | +0.0% | reproduced |
+| 1c | Workers (thousands) | 156,988 | 1,012 | 156,988 | 1,012 | -0.0% | reproduced |
+| 3a | Daily person trips per household | 8.60 | 0.10 | 8.60 | 0.10 | +1.9% | reproduced |
+| 3a | Daily person miles per household | 92.02 | 3.50 | 92.02 | 3.50 | -0.1% | reproduced |
+| 3a | Daily vehicle trips per household | 5.11 | 0.06 | 5.11 | 0.06 | -1.1% | reproduced |
+| 3a | Daily vehicle miles per household | 48.81 | 2.04 | 48.81 | 2.04 | +0.1% | reproduced |
+| 3b | Daily person trips per person | 3.37 | 0.04 | 3.37 | 0.04 | -0.2% | reproduced |
+| 3b | Daily person miles per person | 36.07 | 1.47 | 36.07 | 1.37 | -6.8% | estimate_only |
+| 3b | Average person trip length (miles) | 10.70 | 0.40 | 10.70 | 0.40 | -0.6% | reproduced |
+| 3b | Daily vehicle trips per driver | 2.70 | 0.03 | 2.71 | 0.03 | -1.6% | not_reproduced |
+| 3b | Daily vehicle miles per driver | 25.84 | 1.04 | 25.84 | 1.06 | +1.4% | estimate_only |
+| 3b | Average vehicle trip length (miles) | 9.55 | 0.37 | 9.55 | 0.37 | +1.1% | reproduced |
+
+## Known gaps
+
+* **Persons, all (thousands)** (`persons_all`, Table 1b, status `not_reproduced`): The printed total exceeds the sum of the report's own five age bands (301,599) by 19,820 thousand; the weights reproduce that sum exactly and the public file has no persons under age 5. Cause not established (a population-control baseline is a plausible but unverified explanation).
+
+* **Daily person miles per person** (`pmt_per_person`, Table 3b, status `estimate_only`): Estimate reproduces (36.07) but the JKn margin of error is 1.37, 6.7 percent below the printed 1.47, while the MOE of person miles per household (3.50) and of trip length (0.40) reproduce. WTTRDFIN equals 365 x WTPERFIN in the full weights and in all 98 replicates (relative difference below 1.1e-7), so the weight system is not the cause. Not established.
+
+* **Daily vehicle trips per driver** (`vtrips_per_driver`, Table 3b, status `not_reproduced`): We obtain 2.7064, which rounds to 2.71 against the printed 2.70 (0.2 percent). The MOE (0.0295) matches. Vehicle trips are defined here as trips on which the respondent drove (DRVR_FLG = 01); a slightly different definition in the report is the likely cause but is not established.
+
+* **Daily vehicle miles per driver** (`vmt_per_driver`, Table 3b, status `estimate_only`): Estimate reproduces (25.84); JKn MOE is 1.055 against the printed 1.04 (1.4 percent).
+
+These gaps are reported, not tuned away: no rule in the library was changed to make a figure match. If you can explain one, please open an issue.