data-analyst-mcp 0.3.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +31 -6
- package/dist/adapters/_census_micro.js +2 -2
- package/dist/adapters/_census_micro.js.map +1 -1
- package/dist/adapters/bjs.d.ts +13 -0
- package/dist/adapters/bjs.js +28 -0
- package/dist/adapters/bjs.js.map +1 -0
- package/dist/adapters/cbp.d.ts +5 -0
- package/dist/adapters/cbp.js +64 -0
- package/dist/adapters/cbp.js.map +1 -0
- package/dist/adapters/cdc-wonder.d.ts +5 -0
- package/dist/adapters/cdc-wonder.js +68 -0
- package/dist/adapters/cdc-wonder.js.map +1 -0
- package/dist/adapters/cpsc.d.ts +4 -0
- package/dist/adapters/cpsc.js +125 -0
- package/dist/adapters/cpsc.js.map +1 -0
- package/dist/adapters/fhwa.d.ts +9 -0
- package/dist/adapters/fhwa.js +16 -0
- package/dist/adapters/fhwa.js.map +1 -0
- package/dist/adapters/foreign-assistance.d.ts +9 -0
- package/dist/adapters/foreign-assistance.js +42 -0
- package/dist/adapters/foreign-assistance.js.map +1 -0
- package/dist/adapters/hud-ahar.d.ts +3 -0
- package/dist/adapters/hud-ahar.js +80 -0
- package/dist/adapters/hud-ahar.js.map +1 -0
- package/dist/adapters/nces.d.ts +10 -0
- package/dist/adapters/nces.js +32 -0
- package/dist/adapters/nces.js.map +1 -0
- package/dist/adapters/nsf.d.ts +11 -0
- package/dist/adapters/nsf.js +25 -0
- package/dist/adapters/nsf.js.map +1 -0
- package/dist/adapters/onrr.d.ts +9 -0
- package/dist/adapters/onrr.js +35 -0
- package/dist/adapters/onrr.js.map +1 -0
- package/dist/adapters/sba.d.ts +10 -0
- package/dist/adapters/sba.js +134 -0
- package/dist/adapters/sba.js.map +1 -0
- package/dist/adapters/treasury-tic.d.ts +30 -0
- package/dist/adapters/treasury-tic.js +149 -0
- package/dist/adapters/treasury-tic.js.map +1 -0
- package/dist/adapters/uscourts.d.ts +8 -0
- package/dist/adapters/uscourts.js +8 -0
- package/dist/adapters/uscourts.js.map +1 -0
- package/dist/analysis/cite.d.ts +9 -1
- package/dist/analysis/cite.js +9 -2
- package/dist/analysis/cite.js.map +1 -1
- package/dist/analysis/compute.d.ts +22 -10
- package/dist/analysis/compute.js +75 -16
- package/dist/analysis/compute.js.map +1 -1
- package/dist/analysis/expr.d.ts +2 -0
- package/dist/analysis/expr.js +17 -0
- package/dist/analysis/expr.js.map +1 -1
- package/dist/analysis/replicate.d.ts +15 -0
- package/dist/analysis/replicate.js +228 -0
- package/dist/analysis/replicate.js.map +1 -0
- package/dist/analysis/rerun.js +2 -1
- package/dist/analysis/rerun.js.map +1 -1
- package/dist/analysis/table.d.ts +14 -0
- package/dist/analysis/table.js +79 -28
- package/dist/analysis/table.js.map +1 -1
- package/dist/analysis/tools.js +16 -4
- package/dist/analysis/tools.js.map +1 -1
- package/dist/bin/cli.d.ts +2 -0
- package/dist/bin/cli.js +17 -0
- package/dist/bin/cli.js.map +1 -1
- package/dist/core/generic-rest.js +42 -17
- package/dist/core/generic-rest.js.map +1 -1
- package/dist/core/guard.d.ts +32 -0
- package/dist/core/guard.js +176 -0
- package/dist/core/guard.js.map +1 -0
- package/dist/core/http.d.ts +10 -0
- package/dist/core/http.js +36 -7
- package/dist/core/http.js.map +1 -1
- package/dist/core/keys.d.ts +13 -3
- package/dist/core/keys.js +67 -7
- package/dist/core/keys.js.map +1 -1
- package/dist/core/package.d.ts +12 -3
- package/dist/core/package.js +56 -12
- package/dist/core/package.js.map +1 -1
- package/dist/core/packs.d.ts +14 -5
- package/dist/core/packs.js.map +1 -1
- package/dist/{deep/_opm_parquet.d.ts → core/parquet.d.ts} +2 -1
- package/dist/{deep/_opm_parquet.js → core/parquet.js} +7 -5
- package/dist/core/parquet.js.map +1 -0
- package/dist/core/secret_patterns.d.ts +6 -0
- package/dist/core/secret_patterns.js +20 -0
- package/dist/core/secret_patterns.js.map +1 -0
- package/dist/core/tabular.d.ts +43 -1
- package/dist/core/tabular.js +149 -6
- package/dist/core/tabular.js.map +1 -1
- package/dist/deep/_opm_io.js +2 -2
- package/dist/deep/_opm_io.js.map +1 -1
- package/dist/deep/bea.js +1 -0
- package/dist/deep/bea.js.map +1 -1
- package/dist/deep/fred.js +40 -9
- package/dist/deep/fred.js.map +1 -1
- package/dist/deep/ipums.js +1 -0
- package/dist/deep/ipums.js.map +1 -1
- package/dist/deep/opm.js +1 -1
- package/dist/deep/opm.js.map +1 -1
- package/dist/envelope.js +3 -1
- package/dist/envelope.js.map +1 -1
- package/dist/index.d.ts +29 -22
- package/dist/index.js +45 -46
- package/dist/index.js.map +1 -1
- package/dist/remote/app.js +87 -31
- package/dist/remote/app.js.map +1 -1
- package/dist/remote/oauth.js +8 -1
- package/dist/remote/oauth.js.map +1 -1
- package/dist/remote/pages.d.ts +3 -7
- package/dist/remote/pages.js +49 -19
- package/dist/remote/pages.js.map +1 -1
- package/dist/remote/sign.d.ts +7 -1
- package/dist/remote/sign.js +33 -7
- package/dist/remote/sign.js.map +1 -1
- package/dist/remote/store.js +14 -5
- package/dist/remote/store.js.map +1 -1
- package/dist/report/audit_report.js +29 -7
- package/dist/report/audit_report.js.map +1 -1
- package/dist/report/build_report.js +34 -4
- package/dist/report/build_report.js.map +1 -1
- package/dist/report/method_audit.js +3 -0
- package/dist/report/method_audit.js.map +1 -1
- package/dist/report/scan_secrets.js +2 -13
- package/dist/report/scan_secrets.js.map +1 -1
- package/dist/report/tools.js +2 -1
- package/dist/report/tools.js.map +1 -1
- package/dist/server.d.ts +3 -1
- package/dist/server.js +53 -17
- package/dist/server.js.map +1 -1
- package/dist/tools.d.ts +3 -1
- package/dist/tools.js +82 -9
- package/dist/tools.js.map +1 -1
- package/guide/WORKFLOW.md +6 -5
- package/guide/assets/download_data.ps1 +2 -0
- package/guide/assets/download_data.sh +8 -2
- package/guide/references/report_patterns.md +5 -2
- package/guide/references/report_schema.md +140 -0
- package/guide/references/routing.md +30 -3
- package/package.json +1 -1
- package/packs/bjs/endpoints.json +150 -0
- package/packs/bjs/policy.json +114 -0
- package/packs/bjs/source.md +98 -0
- package/packs/cbp/endpoints.json +31 -0
- package/packs/cbp/policy.json +85 -0
- package/packs/cbp/source.md +57 -0
- package/packs/cdc/source.md +1 -1
- package/packs/cdc-wonder/endpoints.json +26 -0
- package/packs/cdc-wonder/policy.json +84 -0
- package/packs/cdc-wonder/source.md +53 -0
- package/packs/census/source.md +8 -4
- package/packs/college-scorecard/source.md +1 -1
- package/packs/cpsc/endpoints.json +91 -0
- package/packs/cpsc/policy.json +91 -0
- package/packs/cpsc/source.md +76 -0
- package/packs/dhs-ohss/source.md +1 -1
- package/packs/doj-news/source.md +1 -1
- package/packs/fbi/source.md +3 -3
- package/packs/fhwa/endpoints.json +48 -0
- package/packs/fhwa/policy.json +91 -0
- package/packs/fhwa/source.md +59 -0
- package/packs/foreign-assistance/endpoints.json +166 -0
- package/packs/foreign-assistance/policy.json +80 -0
- package/packs/foreign-assistance/source.md +76 -0
- package/packs/fred/source.md +2 -1
- package/packs/fsa/source.md +1 -1
- package/packs/hud/source.md +1 -1
- package/packs/hud-ahar/endpoints.json +92 -0
- package/packs/hud-ahar/policy.json +88 -0
- package/packs/hud-ahar/source.md +72 -0
- package/packs/ipums/docs/report_patterns.md +1 -1
- package/packs/ipums/docs/variables.md +4 -3
- package/packs/ipums/source.md +10 -8
- package/packs/naep/source.md +1 -1
- package/packs/nces/endpoints.json +101 -0
- package/packs/nces/policy.json +99 -0
- package/packs/nces/source.md +75 -0
- package/packs/nhtsa/source.md +1 -1
- package/packs/nsf/endpoints.json +33 -0
- package/packs/nsf/policy.json +83 -0
- package/packs/nsf/source.md +59 -0
- package/packs/onrr/endpoints.json +198 -0
- package/packs/onrr/policy.json +91 -0
- package/packs/onrr/source.md +55 -0
- package/packs/opm/docs/opm_api.md +2 -1
- package/packs/opm/source.md +2 -1
- package/packs/port.json +65 -0
- package/packs/sba/endpoints.json +34 -0
- package/packs/sba/policy.json +86 -0
- package/packs/sba/source.md +60 -0
- package/packs/treasury/source.md +1 -1
- package/packs/treasury-tic/endpoints.json +77 -0
- package/packs/treasury-tic/policy.json +89 -0
- package/packs/treasury-tic/source.md +80 -0
- package/packs/uscourts/endpoints.json +22 -0
- package/packs/uscourts/policy.json +87 -0
- package/packs/uscourts/source.md +56 -0
- package/packs/uspto/source.md +1 -1
- package/dist/deep/_opm_parquet.js.map +0 -1
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# CDC WONDER: mortality, natality and population query results (manual export)
|
|
2
|
+
Depth: standard. Key: none. Manual only. CDC WONDER (NCHS vital statistics) query results that the user runs on
|
|
3
|
+
the WONDER web site and exports. Deaths by underlying or multiple cause (ICD-10), births (natality), infant
|
|
4
|
+
deaths, with populations and crude and age-adjusted rates, by year, state, county, age, sex, race and Hispanic
|
|
5
|
+
origin, as grouped in the query.
|
|
6
|
+
|
|
7
|
+
## When to use
|
|
8
|
+
- Deaths and death rates for causes, years and places that the cdc pack's Socrata tables do not have: cdc's
|
|
9
|
+
leading causes end in 2017 and life expectancy in 2018, while WONDER has final data to the latest year;
|
|
10
|
+
county rates; any ICD-10 code or group; age-adjusted rates by demographic group.
|
|
11
|
+
- Births by mother's characteristics, infant mortality.
|
|
12
|
+
- Not here: provisional weekly deaths and overdose counts that cdc's Socrata tables carry (cdc); health
|
|
13
|
+
insurance (census); injuries treated in emergency departments (cpsc NEISS).
|
|
14
|
+
|
|
15
|
+
## Endpoints
|
|
16
|
+
| Endpoint | Returns | Parameters |
|
|
17
|
+
|---|---|---|
|
|
18
|
+
| `export` | the uploaded WONDER export as published: one column per Group By field (name and code columns, e.g. `Year`, `Year Code`), then `Deaths`, `Population`, `Crude Rate`, `Age Adjusted Rate` ... as chosen; `Notes` (`Total` on total rows) | `database` (required: the WONDER database name), `query` (the query in words) |
|
|
19
|
+
|
|
20
|
+
## Spec example
|
|
21
|
+
```json
|
|
22
|
+
{"description": "Age-adjusted suicide death rates by state, 2023 (WONDER export)",
|
|
23
|
+
"requests": [{"name": "suicide_2023", "endpoint": "export",
|
|
24
|
+
"params": {"database": "Underlying Cause of Death, 2018-2023, Single Race",
|
|
25
|
+
"query": "Group By State; Year 2023; ICD-10 113 Cause List: Intentional self-harm (suicide); age-adjusted rates"}}]}
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Data format and traps
|
|
29
|
+
- The export layout was not checked against a real file here. The reader takes the first line as the header
|
|
30
|
+
and stops at the "---" line, where WONDER's notes begin. Read the columns and the notes (the uploaded file is
|
|
31
|
+
kept in `raw/`) before using the table. Cite the query criteria the notes list.
|
|
32
|
+
- `Notes` = "Total" marks total rows: drop them before summing groups.
|
|
33
|
+
- Suppression and reliability: WONDER shows small counts as "Suppressed" and flags some rates "Unreliable". Keep
|
|
34
|
+
them as text and never fill them with 0. The data use restrictions forbid presenting or publishing any
|
|
35
|
+
statistic representing nine or fewer births or deaths, including rates based on them: aggregate further
|
|
36
|
+
instead.
|
|
37
|
+
- Compare places or years with age-adjusted rates, not crude ones, and say which standard population (WONDER
|
|
38
|
+
states it in the notes).
|
|
39
|
+
- Databases differ: single-race (2018 on) and bridged-race (1999-2020) files use different race categories.
|
|
40
|
+
Final and provisional data differ. Do not join years across databases without saying so. Causes are ICD-10;
|
|
41
|
+
name the codes.
|
|
42
|
+
- Values are text with WONDER's formatting; convert with `Number()` after removing thousands separators.
|
|
43
|
+
|
|
44
|
+
## Access
|
|
45
|
+
Manual only: wonder.cdc.gov answers this client with HTTP 403 (verified 2026-11-07). The WONDER API would also
|
|
46
|
+
require every query to accept the data use restrictions on the user's behalf. The user opens
|
|
47
|
+
https://wonder.cdc.gov/, chooses the database, reads and accepts the data use restrictions, sets the query,
|
|
48
|
+
clicks Send, then Export, and uploads the saved .txt; then `ingest`.
|
|
49
|
+
|
|
50
|
+
## Citation and terms
|
|
51
|
+
"Centers for Disease Control and Prevention, National Center for Health Statistics. <database name> on CDC
|
|
52
|
+
WONDER Online Database, accessed <date>." Data use restrictions: https://wonder.cdc.gov/datause.html (statistical
|
|
53
|
+
reporting and analysis only; no re-identification; no statistics on nine or fewer births or deaths).
|
package/packs/census/source.md
CHANGED
|
@@ -196,8 +196,8 @@ Find others: `request_schema` search `"median household income"` (latest ACS 5-y
|
|
|
196
196
|
Census's replicate weight file (CPS_ASEC_ASCII_REPWGT_<year> on www2.census.gov, about 70 MB, no key), after
|
|
197
197
|
two gates: the file's weight sums equal the totals its SAS layout states, and every person's PWWGT0 equals
|
|
198
198
|
MARSUPWT (2020: the 2020-based file is tried when the main one does not match). Census's replicate standard error
|
|
199
|
-
is sqrt(4/80 * sum of squared replicate differences), MOE = 1.645 * SE (
|
|
200
|
-
Analysis below). Checked 2026-11-04 with that method: Maryland's 2024 uninsured rate in the civilian
|
|
199
|
+
is sqrt(4/80 * sum of squared replicate differences), MOE = 1.645 * SE (`compute`'s `replicate_se` step with
|
|
200
|
+
`method: "sdr"` gives both; see Analysis below). Checked 2026-11-04 with that method: Maryland's 2024 uninsured rate in the civilian
|
|
201
201
|
noninstitutionalized population from PUMS was 6.26 +/- 0.32 percent, published S2701 6.3 +/- 0.3; the nation
|
|
202
202
|
8.19 vs 8.2.
|
|
203
203
|
- Universe: published insurance and labor tables use the civilian noninstitutionalized population
|
|
@@ -220,8 +220,12 @@ Find others: `request_schema` search `"median household income"` (latest ACS 5-y
|
|
|
220
220
|
in early ASECs): the pack leaves them out and says so. CPS basic monthly has no replicate weights at all; month-to-month person links need HRHHID, HRHHID2 and
|
|
221
221
|
PULINENO with care (ipums CPSIDP links them). CPS codes differ from ACS codes (PEMLR, PRCITSHP, A_HGA /
|
|
222
222
|
PEEDUCA): recode each survey from its own codebook.
|
|
223
|
-
- Analysis: build tables with `compute`.
|
|
224
|
-
|
|
223
|
+
- Analysis: build tables with `compute`. Standard errors: fetch with `replicates`, then a `replicate_se` step,
|
|
224
|
+
e.g. `{"groupby": ["ST"], "replicate_se": {"estimates": {"uninsured": "sum(PWGTP * (HICOV == 2)) / sum(PWGTP)"},
|
|
225
|
+
"weight": "PWGTP", "replicates": "PWGTP1:PWGTP80", "method": "sdr"}}` (ASEC: `MARSUPWT`, `"PWWGT1:PWWGT160"`;
|
|
226
|
+
household estimates: `WGTP`/`HSUP_WGT` with their replicates on one row per household). A difference between
|
|
227
|
+
years or groups needs its SE from the same replicates: write it as one estimate. CPS basic monthly and SIPP from
|
|
228
|
+
the API have no replicate weights: say no standard error was computed. This server has no PUMS recode helpers.
|
|
225
229
|
- Time series (`timeseries/...`): give `time` (`2026`, `2026-08`, `2026-Q2`, or `from 2024-01 to 2026-08`).
|
|
226
230
|
Economic indicators (eits) return one row per code combination: `category_code` (industry or tax type),
|
|
227
231
|
`data_type_code` (level, percent change, ...), `seasonally_adj` (yes/no), `time_slot_id`, `error_data` (yes =
|
|
@@ -6,7 +6,7 @@ cost, aid, debt, completion, earnings and admissions data for every U.S. college
|
|
|
6
6
|
- Compare colleges on tuition, net price, admission rate, completion rate, median debt, Pell share and
|
|
7
7
|
median earnings 6, 7 or 10 years after entry; list schools in a state, by ownership or by degree type.
|
|
8
8
|
- Not here: K-12 achievement (naep), enrollment and finance surveys beyond what Scorecard carries (IPEDS
|
|
9
|
-
itself
|
|
9
|
+
itself: nces), student loan portfolio totals (fred, treasury), earnings by occupation (bls).
|
|
10
10
|
- One row per institution (UNITID). Field-of-study (program) data exist but are very large per school.
|
|
11
11
|
|
|
12
12
|
## Endpoints and datasets
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
{
|
|
2
|
+
"base_url": "https://www.saferproducts.gov/RestWebServices",
|
|
3
|
+
"timeout": 60,
|
|
4
|
+
"endpoints": {
|
|
5
|
+
"recalls": {
|
|
6
|
+
"path": "/Recall",
|
|
7
|
+
"method": "GET",
|
|
8
|
+
"description": "CPSC product recalls (SaferProducts.gov Recall API): one row per recall with products, units, hazards, remedies, injuries, manufacturers, retailers and importers. Give at least one filter; text filters match part of the field.",
|
|
9
|
+
"params": {
|
|
10
|
+
"RecallDateStart": {
|
|
11
|
+
"desc": "recalls announced on or after this date, YYYY-MM-DD"
|
|
12
|
+
},
|
|
13
|
+
"RecallDateEnd": {
|
|
14
|
+
"desc": "recalls announced on or before this date, YYYY-MM-DD"
|
|
15
|
+
},
|
|
16
|
+
"LastPublishDateStart": {
|
|
17
|
+
"desc": "recalls published or updated on or after this date, YYYY-MM-DD"
|
|
18
|
+
},
|
|
19
|
+
"RecallNumber": {
|
|
20
|
+
"desc": "CPSC recall number, e.g. 27066"
|
|
21
|
+
},
|
|
22
|
+
"ProductName": {
|
|
23
|
+
"desc": "part of a product name (crib, dresser)"
|
|
24
|
+
},
|
|
25
|
+
"ProductType": {
|
|
26
|
+
"desc": "part of CPSC's product type (Slides, Cribs, Dressers)"
|
|
27
|
+
},
|
|
28
|
+
"RecallTitle": {
|
|
29
|
+
"desc": "part of the recall title"
|
|
30
|
+
},
|
|
31
|
+
"RecallDescription": {
|
|
32
|
+
"desc": "part of the recall description"
|
|
33
|
+
},
|
|
34
|
+
"Manufacturer": {
|
|
35
|
+
"desc": "part of a manufacturer name"
|
|
36
|
+
},
|
|
37
|
+
"Retailer": {
|
|
38
|
+
"desc": "part of the retailer text (Amazon, Walmart)"
|
|
39
|
+
},
|
|
40
|
+
"Importer": {
|
|
41
|
+
"desc": "part of an importer name"
|
|
42
|
+
},
|
|
43
|
+
"Injury": {
|
|
44
|
+
"desc": "part of the injuries text (death, burn)"
|
|
45
|
+
},
|
|
46
|
+
"Remedy": {
|
|
47
|
+
"desc": "part of the remedy text (Refund, Repair, Replace)"
|
|
48
|
+
},
|
|
49
|
+
"hazard": {
|
|
50
|
+
"in": "spec",
|
|
51
|
+
"desc": "keep recalls whose hazard text contains this (case-insensitive); filtered here, because the API's own Hazard filter matches nothing"
|
|
52
|
+
},
|
|
53
|
+
"country": {
|
|
54
|
+
"in": "spec",
|
|
55
|
+
"desc": "keep recalls with this manufacturer country (China, Vietnam); filtered here"
|
|
56
|
+
}
|
|
57
|
+
},
|
|
58
|
+
"defaults": {
|
|
59
|
+
"format": "json"
|
|
60
|
+
},
|
|
61
|
+
"records": "",
|
|
62
|
+
"paging": {
|
|
63
|
+
"type": "none"
|
|
64
|
+
},
|
|
65
|
+
"max_rows": 20000,
|
|
66
|
+
"flatten": false
|
|
67
|
+
},
|
|
68
|
+
"neiss": {
|
|
69
|
+
"path": "https://www.cpsc.gov/cgibin/NEISSQuery/home.aspx",
|
|
70
|
+
"method": "GET",
|
|
71
|
+
"description": "NEISS: one year of emergency-department injury cases involving consumer products (about 300,000 sampled cases a year, with national weights). Manual only: cpsc.gov refuses scripts; the user downloads the archived year as Tab Delimited Text (or Excel) and uploads it.",
|
|
72
|
+
"params": {
|
|
73
|
+
"year": {
|
|
74
|
+
"in": "spec",
|
|
75
|
+
"required": true,
|
|
76
|
+
"desc": "the data year, 2006 or later"
|
|
77
|
+
},
|
|
78
|
+
"product": {
|
|
79
|
+
"in": "spec",
|
|
80
|
+
"desc": "keep cases with this NEISS product code in Product_1, Product_2 or Product_3 (a code or a list; the codes are in the NEISS Coding Manual)"
|
|
81
|
+
}
|
|
82
|
+
},
|
|
83
|
+
"records": "$tsv",
|
|
84
|
+
"header_row": 1,
|
|
85
|
+
"paging": {
|
|
86
|
+
"type": "none"
|
|
87
|
+
},
|
|
88
|
+
"max_rows": 1000000
|
|
89
|
+
}
|
|
90
|
+
}
|
|
91
|
+
}
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
{
|
|
2
|
+
"id": "cpsc",
|
|
3
|
+
"title": "Consumer Product Safety Commission: product recalls and NEISS injury data",
|
|
4
|
+
"agency": "U.S. Consumer Product Safety Commission",
|
|
5
|
+
"depth": "standard",
|
|
6
|
+
"domains": [
|
|
7
|
+
"consumer safety",
|
|
8
|
+
"product recalls",
|
|
9
|
+
"injuries",
|
|
10
|
+
"health"
|
|
11
|
+
],
|
|
12
|
+
"keywords": [
|
|
13
|
+
"cpsc",
|
|
14
|
+
"consumer product safety commission",
|
|
15
|
+
"product recall",
|
|
16
|
+
"recalls",
|
|
17
|
+
"recalled product",
|
|
18
|
+
"saferproducts",
|
|
19
|
+
"crib recall",
|
|
20
|
+
"dresser tip-over",
|
|
21
|
+
"toy recall",
|
|
22
|
+
"neiss",
|
|
23
|
+
"national electronic injury surveillance system",
|
|
24
|
+
"emergency department injuries",
|
|
25
|
+
"product-related injuries",
|
|
26
|
+
"injury estimates",
|
|
27
|
+
"consumer product injuries"
|
|
28
|
+
],
|
|
29
|
+
"questions": [
|
|
30
|
+
"consumer product recalls announced this year",
|
|
31
|
+
"recalls of cribs, dressers or strollers",
|
|
32
|
+
"recalled products made in China",
|
|
33
|
+
"recalls that mention deaths",
|
|
34
|
+
"emergency department visits for injuries involving a product (NEISS)",
|
|
35
|
+
"national estimate of injuries by product, age or sex"
|
|
36
|
+
],
|
|
37
|
+
"homepage": "https://www.cpsc.gov/Recalls",
|
|
38
|
+
"terms_url": "https://www.saferproducts.gov/",
|
|
39
|
+
"auth": {
|
|
40
|
+
"env": null,
|
|
41
|
+
"required": "none",
|
|
42
|
+
"placement": "none",
|
|
43
|
+
"signup_url": null,
|
|
44
|
+
"echoes_key": false
|
|
45
|
+
},
|
|
46
|
+
"limits": {
|
|
47
|
+
"rps": 2,
|
|
48
|
+
"burst": 4,
|
|
49
|
+
"max_rows": 1000000
|
|
50
|
+
},
|
|
51
|
+
"routes": {
|
|
52
|
+
"api": {
|
|
53
|
+
"base": "https://www.saferproducts.gov/RestWebServices",
|
|
54
|
+
"probe": "https://www.saferproducts.gov/RestWebServices/Recall?format=json&RecallNumber=27066",
|
|
55
|
+
"hosts": [
|
|
56
|
+
"www.saferproducts.gov"
|
|
57
|
+
]
|
|
58
|
+
},
|
|
59
|
+
"manual": {
|
|
60
|
+
"how": "Recalls: open the Recall API link (https://www.saferproducts.gov/RestWebServices/Recall?format=json&RecallDateStart=YYYY-MM-DD) in a browser and save the JSON. NEISS: open https://www.cpsc.gov/cgibin/NEISSQuery/home.aspx, pick a User Affiliation, choose the year under Archived Annual NEISS Data with File Format Tab Delimited Text, click Find Data, save the file. Upload with ingest."
|
|
61
|
+
}
|
|
62
|
+
},
|
|
63
|
+
"cache": {
|
|
64
|
+
"mode": "run_only",
|
|
65
|
+
"ttl_hours": null,
|
|
66
|
+
"why": "Recalls are added and updated daily; data/ is reused within a run."
|
|
67
|
+
},
|
|
68
|
+
"redistribution": {
|
|
69
|
+
"mode": "allowed",
|
|
70
|
+
"why": "CPSC recall notices and NEISS public-use data are U.S. government works in the public domain (17 U.S.C. 105)."
|
|
71
|
+
},
|
|
72
|
+
"notices": [],
|
|
73
|
+
"retention": null,
|
|
74
|
+
"citation": "U.S. Consumer Product Safety Commission, {dataset}, saferproducts.gov and cpsc.gov, retrieved {retrieved_long}.",
|
|
75
|
+
"audit_hooks": [],
|
|
76
|
+
"clarify": [
|
|
77
|
+
"Recalls or injuries? Recalls list products CPSC and firms recalled (counts of recalls and units); NEISS estimates emergency-department visits for injuries associated with products (national estimates from a sample, not counts of incidents). Default: recalls for 'recalled products', NEISS for 'how many injuries'.",
|
|
78
|
+
"Which period? Recalls by announcement date (RecallDateStart/End); NEISS by treatment year. Default: the last full calendar year."
|
|
79
|
+
],
|
|
80
|
+
"formats": [
|
|
81
|
+
"json",
|
|
82
|
+
"tsv",
|
|
83
|
+
"xlsx"
|
|
84
|
+
],
|
|
85
|
+
"aliases": [
|
|
86
|
+
"CPSC",
|
|
87
|
+
"Consumer Product Safety Commission",
|
|
88
|
+
"NEISS",
|
|
89
|
+
"SaferProducts.gov"
|
|
90
|
+
]
|
|
91
|
+
}
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# Consumer Product Safety Commission: product recalls and NEISS injury data
|
|
2
|
+
Depth: standard. Key: none.
|
|
3
|
+
- Recalls: from the SaferProducts.gov Recall API, one row per recall with products, units, hazards, injuries,
|
|
4
|
+
remedies, firms and countries.
|
|
5
|
+
- NEISS: the National Electronic Injury Surveillance System, a probability sample of hospital emergency
|
|
6
|
+
departments with weights for national estimates of product-related injuries. NEISS is manual only: the user
|
|
7
|
+
downloads the yearly file and uploads it.
|
|
8
|
+
|
|
9
|
+
## When to use
|
|
10
|
+
- Which consumer products were recalled, when, why (hazard), how many units, the remedy, who made, imported
|
|
11
|
+
or sold them, and where they were made; recalls that mention injuries or deaths.
|
|
12
|
+
- Emergency-department-treated injuries associated with consumer products: national estimates by product,
|
|
13
|
+
age, sex, body part, diagnosis, disposition, location.
|
|
14
|
+
- Not here: vehicle, tire and car-seat recalls (nhtsa); food, drug, device and cosmetic recalls (fda);
|
|
15
|
+
injury deaths from death certificates (cdc).
|
|
16
|
+
|
|
17
|
+
## Endpoints
|
|
18
|
+
| Endpoint | Returns | Parameters |
|
|
19
|
+
|---|---|---|
|
|
20
|
+
| `recalls` | one row per recall: `recall_number`, `recall_date`, `title`, `products`, `product_types`, `number_of_units`, `hazards`, `injuries`, `remedies`, `remedy_options`, `manufacturers`, `importers`, `distributors`, `retailers`, `manufacturer_countries`, `upcs`, `description`, `url`, `last_publish_date`, `recall_id` | API filters (at least one): `RecallDateStart`, `RecallDateEnd`, `LastPublishDateStart` (YYYY-MM-DD), `RecallNumber`, `ProductName`, `ProductType`, `RecallTitle`, `RecallDescription`, `Manufacturer`, `Retailer`, `Importer`, `Injury`, `Remedy`; filtered here: `hazard`, `country` |
|
|
21
|
+
| `neiss` | the uploaded NEISS year as published: `CPSC_Case_Number`, `Treatment_Date`, `Age`, `Sex`, `Race`, `Other_Race`, `Hispanic`, `Body_Part`, `Body_Part_2`, `Diagnosis`, `Diagnosis_2`, `Other_Diagnosis`, `Other_Diagnosis_2`, `Disposition`, `Location`, `Fire_Involvement`, `Alcohol`, `Drug`, `Product_1`, `Product_2`, `Product_3`, `Narrative`, `Stratum`, `PSU`, `Weight` (2019 on; earlier years lack the `_2`, `Hispanic`, `Alcohol`, `Drug`, `Product_3` columns) | `year` (required), `product` (codes kept in any product column) |
|
|
22
|
+
|
|
23
|
+
## Spec examples
|
|
24
|
+
```json
|
|
25
|
+
{"description": "Consumer product recalls announced in 2025 with fire hazards",
|
|
26
|
+
"requests": [{"name": "fire_2025", "endpoint": "recalls",
|
|
27
|
+
"params": {"RecallDateStart": "2025-01-01", "RecallDateEnd": "2025-12-31", "hazard": "fire"}}]}
|
|
28
|
+
```
|
|
29
|
+
```json
|
|
30
|
+
{"description": "Crib recalls since 2020",
|
|
31
|
+
"requests": [{"name": "cribs", "endpoint": "recalls", "params": {"ProductName": "crib", "RecallDateStart": "2020-01-01"}}]}
|
|
32
|
+
```
|
|
33
|
+
```json
|
|
34
|
+
{"description": "NEISS 2024 cases (manual: manual_bundle, then ingest)",
|
|
35
|
+
"requests": [{"name": "neiss_2024", "endpoint": "neiss", "params": {"year": "2024"}}]}
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
## Data format and traps
|
|
39
|
+
- Recall filters (verified 2026-11-07):
|
|
40
|
+
- Text filters match part of the field. With no filter the API returns nothing, so the pack asks for one.
|
|
41
|
+
- An unknown parameter is ignored silently. Only the names above are sent; check the rows match what was
|
|
42
|
+
asked.
|
|
43
|
+
- The API's `Hazard`, `ManufacturerCountry` and `ProductModel` filters match nothing, so use `hazard` and
|
|
44
|
+
`country` here (they filter the returned rows). Narrow by date first.
|
|
45
|
+
- One recall can cover several products. The list fields are joined with ` | `. `number_of_units` is free text
|
|
46
|
+
as published ("About 10,318", "5.9 million", "About 338,800 (In addition, about 36,300 were sold in Canada)"):
|
|
47
|
+
parse it with care, never sum it blindly, and say when Canadian or Mexican units are included.
|
|
48
|
+
- `retailers` is free text ("Sold At: ... from May 2025 through September 2026 for about $20") and sometimes also
|
|
49
|
+
names the seller or importer. `recall_date` is the announcement date. A recall number's first two digits are
|
|
50
|
+
CPSC's fiscal year (27xxx = FY2027, from October 2026).
|
|
51
|
+
- Recalls are announcements, not incidents: a count of recalls measures CPSC and firm actions, not product
|
|
52
|
+
danger.
|
|
53
|
+
- NEISS is a sample: each case has a `Weight`. National estimates are the sum of weights; the number of cases is
|
|
54
|
+
not an estimate. Report the number of cases behind every estimate, and say when an estimate rests on few
|
|
55
|
+
cases. CPSC's NEISS Estimates Query Builder gives the case count, coefficient of variation and 95% confidence
|
|
56
|
+
interval for its own estimates: prefer those for a published-quality figure. Standard errors need the
|
|
57
|
+
`Stratum` and `PSU` design and are not computed here.
|
|
58
|
+
- NEISS counts injuries *associated with* a product, not caused by it: a fall in a bathtub involves the
|
|
59
|
+
bathtub. Products are codes (`Product_1` ... `Product_3`); a case can involve up to three. The codes and the
|
|
60
|
+
values of `Body_Part`, `Diagnosis`, `Disposition` and `Location` are in the NEISS Coding Manual and the data
|
|
61
|
+
dictionary (https://www.cpsc.gov/s3fs-public/DataDictionary042022.xlsx). `Age` 201-223 means 1-23 months. Treatment
|
|
62
|
+
year, not incident year.
|
|
63
|
+
- The `Narrative` is free text from the hospital record. Quote it only as examples, never as counts.
|
|
64
|
+
|
|
65
|
+
## Access
|
|
66
|
+
No key. `fetch` works for recalls (www.saferproducts.gov). NEISS: www.cpsc.gov answers scripts with HTTP 403, and
|
|
67
|
+
the archived yearly files sit behind a form, so `fetch` refuses `neiss` and points to `manual_bundle`. The user
|
|
68
|
+
opens https://www.cpsc.gov/cgibin/NEISSQuery/home.aspx in a browser, picks a User Affiliation, chooses the year
|
|
69
|
+
under "Archived Annual NEISS Data" with File Format "Tab Delimited Text" (Excel also works), clicks Find Data,
|
|
70
|
+
saves the file and uploads it; then `ingest`.
|
|
71
|
+
|
|
72
|
+
## Citation and terms
|
|
73
|
+
"U.S. Consumer Product Safety Commission, Recalls (SaferProducts.gov Recall API), retrieved <date>" or "U.S.
|
|
74
|
+
Consumer Product Safety Commission, National Electronic Injury Surveillance System (NEISS), <year> data,
|
|
75
|
+
downloaded <date>." Public domain (17 U.S.C. 105). Estimates computed from NEISS cases are the analyst's own,
|
|
76
|
+
not CPSC's: say so.
|
package/packs/dhs-ohss/source.md
CHANGED
|
@@ -13,7 +13,7 @@ from the workbooks posted on ohss.dhs.gov, plus the quarterly Legal Immigration
|
|
|
13
13
|
- Not here: the foreign-born population or immigrants' characteristics from surveys (census ACS); visa issuance
|
|
14
14
|
(State Department, not covered); immigration court cases (EOIR, not covered); monthly enforcement statistics: the
|
|
15
15
|
OHSS "Immigration Enforcement and Legal Processes Monthly Tables" stop at November 2024 and the page says the
|
|
16
|
-
report "is delayed while it is under review" (verified 2026-10-30); this pack does not read them.
|
|
16
|
+
report "is delayed while it is under review" (verified 2026-10-30); this pack does not read them. Monthly border encounters are in cbp.
|
|
17
17
|
|
|
18
18
|
## Endpoints and datasets
|
|
19
19
|
| Endpoint | Returns | Key parameters |
|
package/packs/doj-news/source.md
CHANGED
|
@@ -9,7 +9,7 @@ the 94 U.S. Attorney's Offices, FBI, DEA, ATF and other components) and Justice
|
|
|
9
9
|
- **This is text, not statistics.** A count of press releases is a count of announcements, which depend
|
|
10
10
|
on press office practice, not a measure of prosecutions, convictions or crime. Say so whenever releases
|
|
11
11
|
are counted. For case statistics use the U.S. Sentencing Commission or EOUSA caseload reports (not
|
|
12
|
-
covered here); for crime counts use the `fbi` pack; for court
|
|
12
|
+
covered here); for crime counts use the `fbi` pack; for federal court caseload statistics, `uscourts`; for victimization and prisoners, `bjs`.
|
|
13
13
|
- Not here: speeches and congressional testimony (no API endpoint; `/api/v1/speeches.json` answers 404).
|
|
14
14
|
|
|
15
15
|
## Endpoints and datasets
|
package/packs/fbi/source.md
CHANGED
|
@@ -6,9 +6,9 @@ data from the FBI Uniform Crime Reporting (UCR) Program, served by the Crime Dat
|
|
|
6
6
|
- Offenses and clearances by month (violent, property, homicide, robbery ...), national, state or agency;
|
|
7
7
|
arrests; NIBRS incident breakdowns (victim, offender, location, weapon); Supplementary Homicide Reports;
|
|
8
8
|
hate crimes by bias; law enforcement employees; law enforcement suicide data (LESDC).
|
|
9
|
-
- Not here: victimization surveys including crimes not reported to police (
|
|
10
|
-
and
|
|
11
|
-
which is text), the FBI's annual "Crime in the Nation" estimate tables (published reports).
|
|
9
|
+
- Not here: victimization surveys including crimes not reported to police (bjs: NCVS), prisons,
|
|
10
|
+
jails, probation and parole (bjs), federal court caseloads (uscourts), sentencing (U.S. Sentencing
|
|
11
|
+
Commission, not covered), federal prosecutions and DOJ announcements (doj-news, which is text), the FBI's annual "Crime in the Nation" estimate tables (published reports).
|
|
12
12
|
- Not wrapped by this pack, though the CDE API has them (verified 2026-10): NIBRS estimates with lower and
|
|
13
13
|
upper bounds (`/nibrs-estimation/national|region|state/<numeric offense ID>?year=`, years 2021 and 2022
|
|
14
14
|
only, other years HTTP 400; the offense and state IDs are numeric codes not documented here) and use of
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
{
|
|
2
|
+
"base_url": "https://www.fhwa.dot.gov/policyinformation/statistics",
|
|
3
|
+
"timeout": 90,
|
|
4
|
+
"endpoints": {
|
|
5
|
+
"table": {
|
|
6
|
+
"path": "/{year}/xls/{table}.xlsx",
|
|
7
|
+
"method": "GET",
|
|
8
|
+
"description": "One Highway Statistics table as FHWA publishes it (Excel): travel (VM-2 vehicle-miles by state and functional system), drivers (DL-22 by age and sex), vehicles (MV-1 registrations), roads (HM-20 public road length), motor fuel (MF-1 state fuel taxes) and Highway Trust Fund (FE-9) ... Default: every line numbered (row, col1...) to find the header lines; then header_lines names the columns.",
|
|
9
|
+
"params": {
|
|
10
|
+
"year": {
|
|
11
|
+
"in": "path",
|
|
12
|
+
"required": true,
|
|
13
|
+
"desc": "the Highway Statistics edition year, e.g. 2023 (the data year for most tables)"
|
|
14
|
+
},
|
|
15
|
+
"table": {
|
|
16
|
+
"in": "path",
|
|
17
|
+
"required": true,
|
|
18
|
+
"desc": "table id in lower case without the hyphen: vm2, dl22, mv1, hm20, mf1, fe9 ..."
|
|
19
|
+
},
|
|
20
|
+
"sheet": {
|
|
21
|
+
"in": "records",
|
|
22
|
+
"desc": "sheet name or number (default 1, not counting the hidden CRYSTAL_PERSIST sheet); DL-22 has MALES, FEMALES, TOTAL"
|
|
23
|
+
},
|
|
24
|
+
"header_lines": {
|
|
25
|
+
"in": "records",
|
|
26
|
+
"desc": "the header line numbers, e.g. '5-8' for VM-2"
|
|
27
|
+
},
|
|
28
|
+
"header_row": {
|
|
29
|
+
"in": "records",
|
|
30
|
+
"desc": "one header line instead; 0 (default) returns every line numbered"
|
|
31
|
+
},
|
|
32
|
+
"carry": {
|
|
33
|
+
"in": "records",
|
|
34
|
+
"desc": "which header lines span to the right over blank cells: top (default), all, none; check the names against the numbered lines"
|
|
35
|
+
}
|
|
36
|
+
},
|
|
37
|
+
"defaults": {
|
|
38
|
+
"header_row": 0
|
|
39
|
+
},
|
|
40
|
+
"records": "$xlsx",
|
|
41
|
+
"skip_sheets": "^CRYSTAL_PERSIST$",
|
|
42
|
+
"paging": {
|
|
43
|
+
"type": "none"
|
|
44
|
+
},
|
|
45
|
+
"max_rows": 20000
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
}
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
{
|
|
2
|
+
"id": "fhwa",
|
|
3
|
+
"title": "FHWA Highway Statistics: travel, drivers, vehicles, roads, motor fuel and highway finance by state",
|
|
4
|
+
"agency": "U.S. Department of Transportation, Federal Highway Administration",
|
|
5
|
+
"depth": "standard",
|
|
6
|
+
"domains": [
|
|
7
|
+
"transportation",
|
|
8
|
+
"highways",
|
|
9
|
+
"energy"
|
|
10
|
+
],
|
|
11
|
+
"keywords": [
|
|
12
|
+
"fhwa",
|
|
13
|
+
"federal highway administration",
|
|
14
|
+
"highway statistics",
|
|
15
|
+
"vehicle miles traveled",
|
|
16
|
+
"vmt",
|
|
17
|
+
"miles driven",
|
|
18
|
+
"licensed drivers",
|
|
19
|
+
"drivers by age",
|
|
20
|
+
"registered vehicles",
|
|
21
|
+
"vehicle registrations",
|
|
22
|
+
"public road mileage",
|
|
23
|
+
"road length",
|
|
24
|
+
"lane miles",
|
|
25
|
+
"functional system",
|
|
26
|
+
"gas tax",
|
|
27
|
+
"motor fuel tax",
|
|
28
|
+
"highway trust fund",
|
|
29
|
+
"highway finance"
|
|
30
|
+
],
|
|
31
|
+
"questions": [
|
|
32
|
+
"vehicle miles traveled by state",
|
|
33
|
+
"miles driven per capita by state",
|
|
34
|
+
"licensed drivers by age and sex",
|
|
35
|
+
"registered vehicles by state",
|
|
36
|
+
"public road miles by state and functional system",
|
|
37
|
+
"state motor fuel tax receipts",
|
|
38
|
+
"Highway Trust Fund receipts attributable to each state"
|
|
39
|
+
],
|
|
40
|
+
"homepage": "https://www.fhwa.dot.gov/policyinformation/statistics.cfm",
|
|
41
|
+
"terms_url": "https://www.fhwa.dot.gov/policyinformation/statistics.cfm",
|
|
42
|
+
"auth": {
|
|
43
|
+
"env": null,
|
|
44
|
+
"required": "none",
|
|
45
|
+
"placement": "none",
|
|
46
|
+
"signup_url": null,
|
|
47
|
+
"echoes_key": false
|
|
48
|
+
},
|
|
49
|
+
"limits": {
|
|
50
|
+
"rps": 1,
|
|
51
|
+
"burst": 3,
|
|
52
|
+
"max_rows": 20000
|
|
53
|
+
},
|
|
54
|
+
"routes": {
|
|
55
|
+
"api": {
|
|
56
|
+
"base": "https://www.fhwa.dot.gov/policyinformation/statistics",
|
|
57
|
+
"probe": "https://www.fhwa.dot.gov/policyinformation/statistics/2023/xls/vm2.xlsx",
|
|
58
|
+
"hosts": [
|
|
59
|
+
"www.fhwa.dot.gov"
|
|
60
|
+
]
|
|
61
|
+
},
|
|
62
|
+
"manual": {
|
|
63
|
+
"how": "Open https://www.fhwa.dot.gov/policyinformation/statistics/<year>/<table>.cfm (e.g. 2023/vm2.cfm), click the XLSX link, save the workbook and upload it with ingest."
|
|
64
|
+
}
|
|
65
|
+
},
|
|
66
|
+
"cache": {
|
|
67
|
+
"mode": "run_only",
|
|
68
|
+
"ttl_hours": null,
|
|
69
|
+
"why": "Tables are revised in place during the year; data/ is reused within a run."
|
|
70
|
+
},
|
|
71
|
+
"redistribution": {
|
|
72
|
+
"mode": "allowed",
|
|
73
|
+
"why": "FHWA statistics are U.S. government works in the public domain (17 U.S.C. 105)."
|
|
74
|
+
},
|
|
75
|
+
"notices": [],
|
|
76
|
+
"retention": null,
|
|
77
|
+
"citation": "U.S. Department of Transportation, Federal Highway Administration, Highway Statistics {dataset}, retrieved {retrieved_long}.",
|
|
78
|
+
"audit_hooks": [],
|
|
79
|
+
"clarify": [
|
|
80
|
+
"Which edition and year? Highway Statistics <year> tables mostly hold that year's data (VM-2 2023 = travel in 2023); tables are revised in later editions. Default: the latest edition that has the table.",
|
|
81
|
+
"Which table? Travel VM-2 (by functional system), drivers DL-22, vehicles MV-1, road length HM-20, motor fuel MF-1, trust fund FE-9. Default: the table whose title matches the question, named in the answer."
|
|
82
|
+
],
|
|
83
|
+
"formats": [
|
|
84
|
+
"xlsx"
|
|
85
|
+
],
|
|
86
|
+
"aliases": [
|
|
87
|
+
"FHWA",
|
|
88
|
+
"Highway Statistics",
|
|
89
|
+
"Federal Highway Administration"
|
|
90
|
+
]
|
|
91
|
+
}
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# FHWA Highway Statistics: travel, drivers, vehicles, roads, motor fuel and highway finance by state
|
|
2
|
+
Depth: standard. Key: none. The tables of FHWA's annual Highway Statistics series, as the Excel files FHWA
|
|
3
|
+
publishes, read line by line or with a named stacked header. Examples: VM-2 (vehicle-miles of travel by state
|
|
4
|
+
and functional system), DL-22 (licensed drivers by age and sex), MV-1 (registered vehicles), HM-20 (public road
|
|
5
|
+
length), MF-1 (state motor-fuel tax receipts), FE-9 (Highway Trust Fund receipts attributable to each state).
|
|
6
|
+
|
|
7
|
+
## When to use
|
|
8
|
+
- Driving and roads by state: vehicle-miles traveled, per capita (with census population), licensed drivers,
|
|
9
|
+
registered vehicles, road miles, fuel tax receipts, Highway Trust Fund flows.
|
|
10
|
+
- Not here: national monthly travel and transportation indicators (bts); fuel prices and consumption totals
|
|
11
|
+
(eia); traffic deaths and recalls (nhtsa); federal highway grants by recipient (usaspending).
|
|
12
|
+
|
|
13
|
+
## Endpoints
|
|
14
|
+
| Endpoint | Returns | Parameters |
|
|
15
|
+
|---|---|---|
|
|
16
|
+
| `table` | one Highway Statistics table: without `header_lines`, every line numbered (`row`, `col1`...); with it, named columns | `year` (edition, e.g. `2023`), `table` (`vm2`, `dl22`, `mv1`, `hm20`, `mf1`, `fe9` ...), `sheet`, `header_lines`, `header_row`, `carry` |
|
|
17
|
+
|
|
18
|
+
The table list is at https://www.fhwa.dot.gov/policyinformation/statistics/2023/ (each table page has an XLSX
|
|
19
|
+
link). Verified 2026-11-07: vm2 (2023, 2024), dl22, mv1, hm20, mf1, fe9 (2023). `mf2` is not a 2023 file.
|
|
20
|
+
|
|
21
|
+
## Spec examples
|
|
22
|
+
```json
|
|
23
|
+
{"description": "Vehicle-miles traveled by state, 2023 (VM-2)",
|
|
24
|
+
"requests": [{"name": "vm2_2023", "endpoint": "table", "params": {"year": "2023", "table": "vm2", "header_lines": "5-8"}}]}
|
|
25
|
+
```
|
|
26
|
+
```json
|
|
27
|
+
{"description": "Licensed drivers by age, 2023 (DL-22, all drivers): first look at the lines",
|
|
28
|
+
"requests": [{"name": "dl22_total", "endpoint": "table", "params": {"year": "2023", "table": "dl22", "sheet": "TOTAL"}}]}
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Data format and traps
|
|
32
|
+
- Read the raw lines first. Each table has its own layout:
|
|
33
|
+
- VM-2 and HM-20: title lines, the date line, then a four-line header (lines 5-8).
|
|
34
|
+
- MV-1 and MF-1: the header starts on line 4 or 5.
|
|
35
|
+
- DL-22: one sheet per sex (MALES, FEMALES, TOTAL).
|
|
36
|
+
- FE-9: two pages.
|
|
37
|
+
Some workbooks have a hidden CRYSTAL_PERSIST sheet first; it is left out of the sheet numbering.
|
|
38
|
+
- Stacked headers: only the top header line spans right by default (`carry: "top"`), because FHWA centers labels
|
|
39
|
+
over columns without merging them. In VM-2 2023:
|
|
40
|
+
- the RURAL and URBAN spans come out right;
|
|
41
|
+
- column 18, the grand TOTAL of rural and urban, is named `URBAN | TOTAL_col18`; rename it;
|
|
42
|
+
- the columns after it (`URBAN | State Cd`, `URBAN | RINT` ...) are a hidden copy with database codes; drop
|
|
43
|
+
them.
|
|
44
|
+
Always compare names with the numbered lines.
|
|
45
|
+
- Units are in each table's title or date line: VM-2 is in millions of vehicle-miles, MF-1 in thousands of dollars.
|
|
46
|
+
Values are stored numbers as text, often with float noise (`5798.6313300000002`): round to the published
|
|
47
|
+
precision.
|
|
48
|
+
- Rows include state rows, a Total row, and footnote or blank rows below; Puerto Rico and DC appear in some tables.
|
|
49
|
+
Keep state rows by name before ranking. Footnotes are on a separate `footnotes` sheet or below the table.
|
|
50
|
+
- Editions: "Highway Statistics 2023" holds mostly 2023 data; tables are revised in later editions and some are
|
|
51
|
+
published later than others. State the edition and table.
|
|
52
|
+
|
|
53
|
+
## Access
|
|
54
|
+
No key. `fetch` reads https://www.fhwa.dot.gov/policyinformation/statistics/<year>/xls/<table>.xlsx (verified
|
|
55
|
+
2026-11-07). Manual: open the table page and click XLSX, then `ingest`.
|
|
56
|
+
|
|
57
|
+
## Citation and terms
|
|
58
|
+
"U.S. Department of Transportation, Federal Highway Administration, Highway Statistics 2023, Table VM-2,
|
|
59
|
+
retrieved <date>." Public domain (17 U.S.C. 105).
|