data-analyst-mcp 0.7.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/dist/adapters/rma.d.ts +10 -0
- package/dist/adapters/rma.js +40 -0
- package/dist/adapters/rma.js.map +1 -0
- package/dist/analysis/combine.js +3 -2
- package/dist/analysis/combine.js.map +1 -1
- package/dist/analysis/compute.d.ts +26 -5
- package/dist/analysis/compute.js +402 -159
- package/dist/analysis/compute.js.map +1 -1
- package/dist/analysis/expr.js +2 -1
- package/dist/analysis/expr.js.map +1 -1
- package/dist/analysis/handoff.d.ts +1 -1
- package/dist/analysis/handoff.js +44 -3
- package/dist/analysis/handoff.js.map +1 -1
- package/dist/analysis/table.d.ts +7 -0
- package/dist/analysis/table.js +30 -1
- package/dist/analysis/table.js.map +1 -1
- package/dist/analysis/tools.d.ts +10 -7
- package/dist/analysis/tools.js +32 -43
- package/dist/analysis/tools.js.map +1 -1
- package/dist/core/compact.d.ts +15 -0
- package/dist/core/compact.js +98 -0
- package/dist/core/compact.js.map +1 -0
- package/dist/core/generic-rest.js +2 -1
- package/dist/core/generic-rest.js.map +1 -1
- package/dist/core/package.d.ts +8 -0
- package/dist/core/package.js +30 -1
- package/dist/core/package.js.map +1 -1
- package/dist/core/packs.d.ts +4 -2
- package/dist/core/sections.d.ts +21 -0
- package/dist/core/sections.js +53 -0
- package/dist/core/sections.js.map +1 -0
- package/dist/core/tabular.d.ts +2 -0
- package/dist/core/tabular.js +14 -3
- package/dist/core/tabular.js.map +1 -1
- package/dist/deep/opm.js +10 -2
- package/dist/deep/opm.js.map +1 -1
- package/dist/deep/treasury.js +7 -3
- package/dist/deep/treasury.js.map +1 -1
- package/dist/envelope.js +1 -1
- package/dist/envelope.js.map +1 -1
- package/dist/index.d.ts +57 -4
- package/dist/report/audit_report.js +3 -0
- package/dist/report/audit_report.js.map +1 -1
- package/dist/report/build_report.d.ts +18 -0
- package/dist/report/build_report.js +52 -7
- package/dist/report/build_report.js.map +1 -1
- package/dist/report/tools.d.ts +26 -1
- package/dist/report/tools.js +74 -17
- package/dist/report/tools.js.map +1 -1
- package/dist/server.d.ts +1 -1
- package/dist/server.js +52 -43
- package/dist/server.js.map +1 -1
- package/dist/session.d.ts +2 -0
- package/dist/session.js +2 -0
- package/dist/session.js.map +1 -1
- package/dist/tools.d.ts +98 -6
- package/dist/tools.js +154 -17
- package/dist/tools.js.map +1 -1
- package/guide/WORKFLOW.md +7 -4
- package/guide/references/compute.md +51 -0
- package/guide/references/report_quick.md +60 -0
- package/guide/references/routing.md +2 -0
- package/package.json +1 -1
- package/packs/cbp/source.md +1 -1
- package/packs/cdc-wonder/source.md +1 -1
- package/packs/census/docs/microdata.md +77 -0
- package/packs/census/docs/timeseries.md +71 -0
- package/packs/census/source.md +10 -142
- package/packs/congress/docs/nominations_crs.md +26 -0
- package/packs/congress/docs/votes.md +30 -0
- package/packs/congress/source.md +4 -48
- package/packs/govinfo/docs/collections.md +88 -0
- package/packs/govinfo/source.md +6 -86
- package/packs/irs-soi/docs/eo.md +15 -0
- package/packs/irs-soi/docs/tables.md +50 -0
- package/packs/irs-soi/source.md +6 -57
- package/packs/port.json +5 -0
- package/packs/rma/endpoints.json +44 -0
- package/packs/rma/policy.json +94 -0
- package/packs/rma/source.md +78 -0
- package/packs/sba/source.md +1 -1
- package/packs/uscourts/source.md +1 -1
package/guide/WORKFLOW.md
CHANGED
|
@@ -20,7 +20,7 @@ fetched file. **Accuracy comes before everything else.**
|
|
|
20
20
|
| `cite(url, id, text?)` | snapshot a page a reference cites (pass the text your own fetch tool returned) |
|
|
21
21
|
| `rerun` | replay every exhibit and fact from the fetched data |
|
|
22
22
|
| `audit_answer(answer, rerun)` | the release gate for a quick answer: send the answer only when it prints AUDIT PASSED |
|
|
23
|
-
| `build_report(report)`, `
|
|
23
|
+
| `build_report(report)`, `audit_report` | a report or paper, only when asked (`style_check` lints prose alone) |
|
|
24
24
|
| `handoff` | the per-source data hand-off (data-only replies) |
|
|
25
25
|
| `read_file(path)` | a small text file from the workspace |
|
|
26
26
|
|
|
@@ -120,9 +120,12 @@ FAIL, and send it only when it prints AUDIT PASSED. Data only: send the `handoff
|
|
|
120
120
|
and what could not be fetched with its route.
|
|
121
121
|
|
|
122
122
|
## 7. Report or paper (only when asked)
|
|
123
|
-
Read `guide("
|
|
124
|
-
|
|
125
|
-
|
|
123
|
+
Read `guide("report")`: what the build and the audit require, a minimal report, the chart keys and the style
|
|
124
|
+
rules, in one page. The full references come by section when needed (`guide("report_schema")`,
|
|
125
|
+
`guide("report_patterns")`, `guide("style")`). Write the report as `build_report`'s report object, with every
|
|
126
|
+
number as `{{fact_name}}` and a `sources` list of the pack ids used. `build_report` returns the style score with the
|
|
127
|
+
sentences to fix; fix with `patch` (or with no report: it rebuilds from report.json), never by sending the whole
|
|
128
|
+
object again. Then `audit_report` (`rerun: true`): it returns the failures only; rebuild until it prints
|
|
126
129
|
AUDIT PASSED. The builder applies each source's notices, redistribution rules (microdata never leaves the
|
|
127
130
|
workspace) and vintage blocks.
|
|
128
131
|
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# compute: steps, expressions and survey standard errors
|
|
2
|
+
|
|
3
|
+
`compute` runs declarative steps over full workspace tables (`data/<id>/<table>.csv` or `.parquet`, earlier
|
|
4
|
+
exhibits, `published/*.csv`) and saves `exhibits/<name>.csv` with its lineage, so the rerun gate can replay it.
|
|
5
|
+
|
|
6
|
+
## Steps
|
|
7
|
+
Each step is an object with one verb:
|
|
8
|
+
- `{filter: expr}`, `{derive: {col: expr, ...}}` (a later name may use an earlier one in the same step),
|
|
9
|
+
`{select: [cols]}`, `{rename: {old: new}}`.
|
|
10
|
+
- `{groupby: [keys], rollup: {col: aggregate expr}}`. `groupby` also goes with `derive` or `filter`: per-group
|
|
11
|
+
windows and shares.
|
|
12
|
+
- `{join: alias, on: [keys] | {left: right}, how: inner|left, suffix?: ["_x", "_y"]}`.
|
|
13
|
+
- `{pivot_wider: {names_from, values_from, id_cols?, fill?}}`, `{pivot_longer: {cols, names_to, values_to}}`.
|
|
14
|
+
- `{orderby: [col, "-col" for descending]}`, `{slice: n | [start, end]}`, `{dedupe: true | [cols]}`,
|
|
15
|
+
`{concat: [aliases]}`.
|
|
16
|
+
- `{as: name}` keeps the current table under a name; `{from: name}` switches to a table.
|
|
17
|
+
- Combine helpers:
|
|
18
|
+
- `{fiscal_year: {date, as?}}`.
|
|
19
|
+
- `{to_fiscal: {date, value, how: sum|mean|last, freq: M|Q}}` adds `periods` and `complete`; incomplete years
|
|
20
|
+
are flagged, never full.
|
|
21
|
+
- `{state_codes: col}` adds `state_usps`, `state_fips`, `state_name`.
|
|
22
|
+
- `{deflate: {value, period, index: alias, index_period?, index_value, name: "CPI-U, BLS CUUR0000SA0", base?: "2024", as?}}`.
|
|
23
|
+
- `{align_check: {tables: [aliases or "this"], on: [keys], units: {alias: "billions of dollars, nominal, fiscal years"}}}`
|
|
24
|
+
refuses repeated keys, missing units and no overlap. Run it before a join of two sources.
|
|
25
|
+
|
|
26
|
+
## Expressions
|
|
27
|
+
Column names (`col("a b")` for odd names), numbers, `"text"`, `true`/`false`/`null`, `+ - * / % **`,
|
|
28
|
+
`== != < <= > >=`, `&& || !` (and/or/not), `c ? a : b`. Nothing else runs: no property access, no other calls.
|
|
29
|
+
CSV cells that are plain numbers are numbers; `"1,234"` stays text until `number()`; an empty cell is null (a
|
|
30
|
+
gap). Parquet cells are typed the same way. The functions, aggregates and windows are listed at the end.
|
|
31
|
+
|
|
32
|
+
## Big inputs
|
|
33
|
+
Read only the columns needed: `{alias: {path, columns: [...]}}`. compute also reads only the columns a recipe
|
|
34
|
+
names when it can tell them from the steps. A file over the cell cap is refused: name its columns, or use the
|
|
35
|
+
Parquet file when the fetch wrote one.
|
|
36
|
+
|
|
37
|
+
## Survey standard errors (replicate weights)
|
|
38
|
+
`{groupby?: [keys], replicate_se: {estimates: {rate: "sum(PWGTP * (HICOV == 2)) / sum(PWGTP)"}, weight: "PWGTP",
|
|
39
|
+
replicates: "PWGTP1:PWGTP80" | [cols], method: sdr|fay|jk1|brr|custom, rho? (fay), scale? (custom),
|
|
40
|
+
also?: {n: "count()"}, confidence?: 90|95|99}}`
|
|
41
|
+
|
|
42
|
+
It recomputes each weighted estimate with every replicate weight in place of the weight, and adds `<name>_se`
|
|
43
|
+
and `<name>_moe` (90 percent by default, as Census publishes).
|
|
44
|
+
- SE = sqrt(c × Σ_r (estimate_r − estimate)²).
|
|
45
|
+
- c = 4/R for sdr (ACS PUMS 4/80, CPS ASEC 4/160); 1/(R(1−rho)²) for fay; (R−1)/R for jk1; 1/R for brr.
|
|
46
|
+
Use the method the pack's notes name.
|
|
47
|
+
- An SE of a difference or ratio needs it as one estimate, e.g.
|
|
48
|
+
`"sum(W*(YEAR==2024)*X)/sum(W*(YEAR==2024)) - sum(W*(YEAR==2023)*X)/sum(W*(YEAR==2023))"`.
|
|
49
|
+
|
|
50
|
+
`{replicate_se: {estimate: col, replicates, method, as?}}` instead reads a table that already holds replicate
|
|
51
|
+
estimates (one row per estimate) and adds `<as>_se` and `<as>_moe` to each row.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# Report in short
|
|
2
|
+
|
|
3
|
+
What `build_report` and `audit_report` need, in one page. The full references are by section:
|
|
4
|
+
`guide("report_schema")` (every report.json key), `guide("report_patterns")` (structure, benchmarks, chart forms)
|
|
5
|
+
and `guide("style")` (the house style). Ask for a section only when this page does not settle a question.
|
|
6
|
+
|
|
7
|
+
## The steps
|
|
8
|
+
1. Every number in the prose is a fact: register it with `facts`, write it as `{{name}}` (or `{{name|fmt}}`).
|
|
9
|
+
References are `[@id]`, footnotes `[^n]`. A number typed by hand fails the audit unless `audit_allow` lists it.
|
|
10
|
+
2. Every exhibit is a CSV in `exhibits/` saved by `compute` or `write_table`.
|
|
11
|
+
3. `build_report` with the report object. It writes report.json, report.html and replication_package.zip, and
|
|
12
|
+
returns the style score with the sentences to fix. Later fixes: `build_report` with `patch`
|
|
13
|
+
(`{"sections.0.body": "..."}`) or with no report (it rebuilds from report.json). Never send the whole object
|
|
14
|
+
again for a small fix.
|
|
15
|
+
4. `audit_report` (`rerun: true` replays every recipe and fact). It returns the failures only. Fix them and
|
|
16
|
+
rebuild until it prints AUDIT PASSED. A rerun that is still going answers `in_progress`: call again.
|
|
17
|
+
|
|
18
|
+
## What the audit requires (report mode)
|
|
19
|
+
- `at_a_glance.text` and 3 to 5 findings, each `{"lead": "Section name.", "text": "..."}`.
|
|
20
|
+
- `notes_page`; `introduction.text` and `introduction.subsections` (question-style headings, one per section).
|
|
21
|
+
- 2 or more `sections`, each showing an exhibit (`"exhibits": [n]`, and "see Exhibit n" in the body).
|
|
22
|
+
- `text` on every exhibit; `about`; `methods` (Appendix A: request, sources, filters, universes, weights,
|
|
23
|
+
deflator, benchmarks, vintages, limitations).
|
|
24
|
+
- When a source records a vintage, `methods` or `notes_page` states it and says the source revises its data.
|
|
25
|
+
- `sources`: the pack ids used. The builder adds each source's notices, the Data Vintage block, the assumptions
|
|
26
|
+
table and the citation; never type those by hand.
|
|
27
|
+
|
|
28
|
+
## A minimal report
|
|
29
|
+
```json
|
|
30
|
+
{
|
|
31
|
+
"mode": "report", "title": "Unemployment, 2019 to 2025", "date": "2026-11-01", "sources": ["bls"],
|
|
32
|
+
"cover": {"exhibit": 1},
|
|
33
|
+
"at_a_glance": {"text": "...", "findings": [{"lead": "Unemployment.", "text": "The rate was {{ur_last}}."}]},
|
|
34
|
+
"notes_page": ["Unless otherwise indicated, all years are calendar years."],
|
|
35
|
+
"introduction": {"text": "...", "subsections": [{"heading": "How Did Unemployment Change?", "body": "..."}]},
|
|
36
|
+
"sections": [{"heading": "Unemployment", "body": "... (see Exhibit 1).[^1]", "exhibits": [1],
|
|
37
|
+
"footnotes": ["Seasonally adjusted."]}],
|
|
38
|
+
"exhibits": [{"title": "Unemployment Rate, 2019 to 2025", "subtitle": "Percent", "text": "...",
|
|
39
|
+
"data": "exhibits/unemployment.csv",
|
|
40
|
+
"chart": {"type": "line", "x": "month", "y": "value", "date": "date", "format": "pct100", "decimals": 1},
|
|
41
|
+
"source": "Analyst calculations using data from the Bureau of Labor Statistics."}],
|
|
42
|
+
"methods": "...", "about": "..."
|
|
43
|
+
}
|
|
44
|
+
```
|
|
45
|
+
`mode: "paper"` is prose-led, with the exhibits in the running text (a `summary` list instead of `at_a_glance`).
|
|
46
|
+
|
|
47
|
+
## Charts
|
|
48
|
+
`chart`: `{"type", "x", "y", ...}`; every name is a column of the exhibit CSV. Types: `line` (default), `column`,
|
|
49
|
+
`hbar`, `stacked`, `stacked_h`, `table`. Long data: `series` (a column); wide data: `y` as a list. Others:
|
|
50
|
+
`filter`, `x_order`, `series_order` (at most 8 series), `date` (ISO dates, a calendar axis), `recessions`,
|
|
51
|
+
`format` (`number`, `pct` for fractions, `pct100`, `dollars`), `decimals`, `y_min`/`y_max`, `columns`.
|
|
52
|
+
|
|
53
|
+
## Style rules the build checks
|
|
54
|
+
- No first person ("we", "our", "I") and no contractions; the analysis *estimates*.
|
|
55
|
+
- *percent* in words; *percentage points* for differences; zero to nine in words, 10 and up in numerals; no
|
|
56
|
+
sentence starts with a numeral.
|
|
57
|
+
- Ranges *from X to Y*; ages *ages 25 to 54*; years in full.
|
|
58
|
+
- Plain words: none of "e.g.", "i.e.", "etc.", "utilize", "prior to", "regarding", "methodology", "data set".
|
|
59
|
+
- Sentences short; titles and headings in headline style; every exhibit has a units line and a source line.
|
|
60
|
+
- Define each rate, universe and abbreviation at first use. Name the dollar year and the price index.
|
|
@@ -33,6 +33,7 @@ Use `find_sources` with the question for a ranked suggestion.
|
|
|
33
33
|
| Economic forecasts and forecast benchmarks | philfed | fred, usda-outlook | SPF and Greenbook (5-year lag) forecasts; USDA WASDE and baseline for agriculture. |
|
|
34
34
|
| Household balance sheets, consumer credit, bank lending surveys | fed-board | fred, nyfed | Fed Board Z.1, G.19, SLOOS, SCF; FRED for single-series histories. |
|
|
35
35
|
| Farm income, crop supply and demand outlook | usda-outlook | usda-nass | WASDE projections, ERS farm income forecasts, USDA baseline; NASS for survey estimates. |
|
|
36
|
+
| Federal crop insurance cost: premium subsidies, indemnities, liability, companies' underwriting gains | rma | usda-nass, treasury | Crop year (reinsurance year for gains), not fiscal year; FCIC outlays by fiscal year are in treasury and omb. |
|
|
36
37
|
| Immigration: green cards, naturalizations, refugees, enforcement | dhs-ohss | census | OHSS Yearbook by fiscal year; counts are events, not people. |
|
|
37
38
|
| Veterans: population, benefits, VA spending by place | va | census, usaspending | VetPop is a projection model; GDX for expenditures by state and district. |
|
|
38
39
|
| Student loans: portfolio, delinquency, repayment plans, volume by school | fsa | college-scorecard, treasury | FSA data center (.xls); recipients are not additive across programs. |
|
|
@@ -181,6 +182,7 @@ Questions a pack does not answer, and the pack that does (each pack's policy.jso
|
|
|
181
182
|
| pbgc | standard | none | pensions, retirement, social insurance | PBGC Pension Insurance Data Tables (single-employer and multiemployer programs) |
|
|
182
183
|
| philfed | standard | none | macroeconomic forecasts, economic outlook, inflation expectations, real-time data | Philadelphia Fed Real-Time Data Research Center (Survey of Professional Forecasters, Greenbook/Tealbook, Livingston Survey, Real-Time Data Set) |
|
|
183
184
|
| regulations | standard | optional (DATA_GOV_API_KEY) | legislation, regulation, rulemaking | Regulations.gov (API v4) |
|
|
185
|
+
| rma | standard | none | agriculture, federal spending, insurance | USDA RMA federal crop insurance: Summary of Business (liability, premium, subsidy, indemnity) and reinsurance results |
|
|
184
186
|
| sba | standard | none | small business, federal lending, federal spending | SBA loan data: 7(a) and 504 loans (FOIA loan-level files), activity reports, disaster loans |
|
|
185
187
|
| sec | standard | none | finance, corporations, securities | SEC EDGAR (XBRL financial data, filings, full-text search) |
|
|
186
188
|
| senate-lobbying | standard | optional (LDA_API_KEY) | legislation, lobbying, influence | Lobbying Disclosure (LDA.gov API) |
|
package/package.json
CHANGED
package/packs/cbp/source.md
CHANGED
|
@@ -23,7 +23,7 @@ Columns:
|
|
|
23
23
|
- By state (`-state`): Fiscal Year, Month Grouping, Month (abbv), Land Border Region, State, Demographic,
|
|
24
24
|
Citizenship, Title of Authority, Encounter Count.
|
|
25
25
|
|
|
26
|
-
## Spec
|
|
26
|
+
## Spec examples
|
|
27
27
|
```json
|
|
28
28
|
{"description": "Southwest border encounters by month, FY2024 to date (file of November 2026)",
|
|
29
29
|
"requests": [{"name": "files", "endpoint": "list_files"},
|
|
@@ -17,7 +17,7 @@ origin, as grouped in the query.
|
|
|
17
17
|
|---|---|---|
|
|
18
18
|
| `export` | the uploaded WONDER export as published: one column per Group By field (name and code columns, e.g. `Year`, `Year Code`), then `Deaths`, `Population`, `Crude Rate`, `Age Adjusted Rate` ... as chosen; `Notes` (`Total` on total rows) | `database` (required: the WONDER database name), `query` (the query in words) |
|
|
19
19
|
|
|
20
|
-
## Spec
|
|
20
|
+
## Spec examples
|
|
21
21
|
```json
|
|
22
22
|
{"description": "Age-adjusted suicide death rates by state, 2023 (WONDER export)",
|
|
23
23
|
"requests": [{"name": "suicide_2023", "endpoint": "export",
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Census microdata: ACS PUMS, CPS and SIPP
|
|
2
|
+
|
|
3
|
+
From the census playbook's "Data format and traps" section: read it before a request of this kind.
|
|
4
|
+
|
|
5
|
+
- SIPP (`sipp`, `20XX/sipp`): public-use microdata, one row per person per reference month (MONTHCODE
|
|
6
|
+
1-12). The reference year is the calendar year before the file's label: `2024/sipp` was collected in
|
|
7
|
+
2024 about January-December 2023; say "2023" for its figures. Filter MONTHCODE (12 = December) for a
|
|
8
|
+
point in time, or average the months for the year. The pack always adds SSUID, PNUM (person = SSUID + PNUM), MONTHCODE and
|
|
9
|
+
WPFINWGT, the person weight: every estimate is weighted by WPFINWGT, never a row count. No geography but
|
|
10
|
+
`state`, which filters TST_INTV (state of the interview address); SIPP is designed for national estimates.
|
|
11
|
+
-9 means NA or missing in most variables: drop it before summing. Standard errors need the replicate
|
|
12
|
+
weights, which are not on the API (bulk file `rw<YYYY>_csv.zip`, REPWGT1-REPWGT240, at
|
|
13
|
+
https://www2.census.gov/programs-surveys/sipp/data/datasets/<YYYY>/; method in the year's user guide); without them
|
|
14
|
+
give the point estimate and say no standard error was computed. Variable names and codes:
|
|
15
|
+
`request_schema` search `"<YYYY>/sipp: health insurance"` or the dataset's variables.json. The bulk primary file
|
|
16
|
+
(`pu<YYYY>_csv.zip`, about 100 MB) has every variable, for a manual download.
|
|
17
|
+
- ACS PUMS and CPS microdata (`acs1_pums`, `acs5_pums`, `cps_asec`, `<YYYY>/cps/basic/<mon>`): one row per
|
|
18
|
+
person, values and codes exactly as published, with `<request>_codebook.csv` (every code's label, the
|
|
19
|
+
variable's suggested weight). The pack adds the keys and weights: ACS SERIALNO, SPORDER, PWGTP (persons), WGTP
|
|
20
|
+
(households: SPORDER 1 with RELSHIPP 20); ASEC H_SEQ, A_LINENO, PERIDNUM, MARSUPWT, HSUP_WGT; basic HRHHID,
|
|
21
|
+
HRHHID2, PULINENO, PWSSWGT (final weight), PWCMPWGT (composite: employment levels that match BLS). For hourly
|
|
22
|
+
and weekly earnings in a basic month, request PWORWGT (outgoing rotation groups, HRMIS 4 and 8). Every estimate
|
|
23
|
+
is weighted, never a row count. Filters are upper-case variable predicates (`"AGEP": "25:54"`, `"HICOV": 2`),
|
|
24
|
+
repeated on every call; geography: `state`, or `for: "public use microdata area:*"` with a state (ACS);
|
|
25
|
+
no state = the nation (ACS fetched state by state, 51 calls, about 2 minutes for the 3.4 million persons).
|
|
26
|
+
Every result has a `state` column (the FIPS code): group on it. The ACS PUMS state variable is `ST` through
|
|
27
|
+
the 2022 file and `STATE` from 2023 on (1-year, 5-year and Puerto Rico; the other name is not a variable).
|
|
28
|
+
`replicates: true` adds PWGTP1-80 for ACS (`"household"`: WGTP1-80, `"both"`); columns past 50 come in
|
|
29
|
+
further calls joined on the keys. For the ASEC (2005 on) it adds PH_SEQ and PPPOS and joins PWWGT1-160 from
|
|
30
|
+
Census's replicate weight file (CPS_ASEC_ASCII_REPWGT_<year> on www2.census.gov, about 70 MB, no key), after
|
|
31
|
+
two gates: the file's weight sums equal the totals its SAS layout states, and every person's PWWGT0 equals
|
|
32
|
+
MARSUPWT (2020: the 2020-based file is tried when the main one does not match). Census's replicate standard error
|
|
33
|
+
is sqrt(4/80 * sum of squared replicate differences), MOE = 1.645 * SE (`compute`'s `replicate_se` step with
|
|
34
|
+
`method: "sdr"` gives both; see Analysis below). Checked 2026-11-04 with that method: Maryland's 2024 uninsured rate in the civilian
|
|
35
|
+
noninstitutionalized population from PUMS was 6.26 +/- 0.32 percent, published S2701 6.3 +/- 0.3; the nation
|
|
36
|
+
8.19 vs 8.2.
|
|
37
|
+
- Universe: published insurance and labor tables use the civilian noninstitutionalized population
|
|
38
|
+
(drop RELSHIPP 37, the institutionalized group quarters population, and ESR 4-5, armed forces). Codes below zero or 0 often mean N/A (PINCP -19999 under 15, ESR 0 under 16): read the codebook.
|
|
39
|
+
- Dollars: ACS amounts need ADJINC (ADJHSG for housing costs), which the pack adds when a dollar variable asks
|
|
40
|
+
for it: value * ADJINC / 1,000,000 is in the survey year's dollars, as the published tables are. PUMS is a
|
|
41
|
+
subsample of the full ACS: a PUMS figure differs slightly from the published table (inside its MOE);
|
|
42
|
+
publish your figure as your own calculation and benchmark it against the table.
|
|
43
|
+
- CPS: ASEC income, poverty (PERLIS) and full-year insurance (COV) refer to the previous calendar year; NOW_COV
|
|
44
|
+
is coverage at the interview. Census's method for standard errors: weight MARSUPWT with replicates
|
|
45
|
+
PWWGT1-160, SE = sqrt(4/160 * sum of squared differences); household estimates on householders only, with
|
|
46
|
+
HSUP_WGT and the same replicates. Checked 2026-11-04 with that method: 2025 official poverty from the 2026 ASEC was
|
|
47
|
+
34,505 thousand +/- 909 and 10.2 +/- 0.3 percent (PERLIS 1 among PERLIS != -1), P60-290 Table 1 34,510 +/- 909
|
|
48
|
+
and 10.2 +/- 0.3. Before 2005 there is no replicate file: use the ipums pack.
|
|
49
|
+
The API's 2020 and 2021 ASEC weights are the 2010-Census-based ones (their PWWGT0 matched the main replicate
|
|
50
|
+
file, checked 2026-11-04); Census advises its 2020-based weights to compare those years with 2022 on, and
|
|
51
|
+
they are not on the API: say which weights a 2020-2021 comparison uses.
|
|
52
|
+
ASEC files before 2001 take no state geography on the API (HTTP 500): fetch the nation and filter on the
|
|
53
|
+
file's state variable, or use ipums. Older files lack some keys and weights (HRHHID2 before 2004, PERIDNUM
|
|
54
|
+
in early ASECs): the pack leaves them out and says so. CPS basic monthly has no replicate weights at all; month-to-month person links need HRHHID, HRHHID2 and
|
|
55
|
+
PULINENO with care (ipums CPSIDP links them). CPS codes differ from ACS codes (PEMLR, PRCITSHP, A_HGA /
|
|
56
|
+
PEEDUCA): recode each survey from its own codebook.
|
|
57
|
+
- Analysis: build tables with `compute`. Standard errors: fetch with `replicates`, then a `replicate_se` step,
|
|
58
|
+
e.g. `{"groupby": ["state"], "replicate_se": {"estimates": {"uninsured": "sum(PWGTP * (HICOV == 2)) / sum(PWGTP)"},
|
|
59
|
+
"weight": "PWGTP", "replicates": "PWGTP1:PWGTP80", "method": "sdr"}}` (ASEC: `MARSUPWT`, `"PWWGT1:PWWGT160"`;
|
|
60
|
+
household estimates: `WGTP`/`HSUP_WGT` with their replicates on one row per household). A difference between
|
|
61
|
+
years or groups needs its SE from the same replicates: write it as one estimate. CPS basic monthly and SIPP from
|
|
62
|
+
the API have no replicate weights: say no standard error was computed. This server has no PUMS recode helpers.
|
|
63
|
+
|
|
64
|
+
## Spec examples
|
|
65
|
+
```json
|
|
66
|
+
{"description": "Share of people without health coverage in December, SIPP 2024 (weighted by WPFINWGT)",
|
|
67
|
+
"requests": [{"name": "sipp_coverage_dec", "endpoint": "query",
|
|
68
|
+
"params": {"dataset": "2024/sipp", "get": "TAGE,RHLTHMTH,TPTOTINC", "MONTHCODE": "12"}}]}
|
|
69
|
+
```
|
|
70
|
+
```json
|
|
71
|
+
{"description": "Uninsured rate, Maryland, ACS 2024 1-year PUMS with replicate weights; CPS September 2026",
|
|
72
|
+
"requests": [{"name": "md_pums", "endpoint": "query",
|
|
73
|
+
"params": {"dataset": "2024/acs/acs1/pums", "get": "AGEP,HICOV,RELSHIPP,ESR,PINCP", "state": "MD",
|
|
74
|
+
"replicates": true}},
|
|
75
|
+
{"name": "cps_sep", "endpoint": "query",
|
|
76
|
+
"params": {"dataset": "2026/cps/basic/sep", "get": "PRTAGE,PEMLR,PRCITSHP", "PRTAGE": "25:54"}}]}
|
|
77
|
+
```
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Census time series: economic indicators, trade, government finances, Household Pulse, HVS
|
|
2
|
+
|
|
3
|
+
From the census playbook's "Data format and traps" section: read it before a request of this kind.
|
|
4
|
+
|
|
5
|
+
- Time series (`timeseries/...`): give `time` (`2026`, `2026-08`, `2026-Q2`, or `from 2024-01 to 2026-08`).
|
|
6
|
+
Economic indicators (eits) return one row per code combination: `category_code` (industry or tax type),
|
|
7
|
+
`data_type_code` (level, percent change, ...), `seasonally_adj` (yes/no), `time_slot_id`, `error_data` (yes =
|
|
8
|
+
the row is a margin or standard error, not an estimate). The codes are not in the API's metadata: run one
|
|
9
|
+
narrow query and read the codes that come back, or the program's definitions page (census.gov/econ/currentdata),
|
|
10
|
+
then filter on them; never sum across data types or adjustment flags. Monthly indicators are revised (advance,
|
|
11
|
+
preliminary, revised): say which release. Units: retail and construction $ millions, housing units in
|
|
12
|
+
thousands, QTAX `CELL_VALUE` $ millions (variables.json labels say so); government finances `AMOUNT` $ thousands
|
|
13
|
+
with a coefficient of variation `AMOUNT_CV` (percent, sample-based local figures). Government finance years are
|
|
14
|
+
each government's fiscal year, not the calendar year. International trade: values in dollars; `GEN_VAL_MO`
|
|
15
|
+
general imports vs `CON_VAL_MO` imports for consumption; `CAL_DUT_MO` is the duty calculated on imports for
|
|
16
|
+
consumption (an estimate of duty owed, not Treasury's customs receipts); `COMM_LVL` HS2/HS4/HS6/HS10 must be
|
|
17
|
+
fixed or totals double-count; `CTY_CODE` country codes (5700 China, 2010 Mexico, 1220 Canada); it takes no
|
|
18
|
+
for= geography; `SUMMARY_LVL` CGP rows are country groups, not countries (drop them before summing).
|
|
19
|
+
- Household Pulse (`pulse`, `pulse_covid`) is an experimental data product: an online survey of adults 18+ in
|
|
20
|
+
housing units, contacted by e-mail and text, with weighted response rates of 4.7 to 7.3 percent at the end
|
|
21
|
+
(7.3 percent in week 63, October 2023; 4.7 percent in Phase 4.2 cycle 09, September 2024; each period's Source
|
|
22
|
+
and Accuracy statement, census.gov/programs-surveys/household-pulse-survey/technical-documentation.html). Say so
|
|
23
|
+
with every figure. `pulse` (timeseries/hhpulse) is long: one row per `MEASURE_NAME` (13 measures: EXPENSE,
|
|
24
|
+
FOODSCARCE, ENERGYBILL, EVICTFOR, PRICEINCR, PRICEWORRY, LONELY, ...), `CYCLE` and geography; filter
|
|
25
|
+
`MEASURE_NAME` and `CYCLE`. It holds `time=2024` only (Phase 4.0 cycles 01-03, 4.1 04-07, 4.2 08-09, January 9 to
|
|
26
|
+
September 16, 2024); `time=2025` returns no rows. `pulse_covid` (timeseries/hps) is wide: `<M>_RATE`,
|
|
27
|
+
`<M>_RT_MOE`, `<M>_TOTAL`, `<M>_MOE`, `<M>_UNIV` per collection `WEEK` (01-63, April 2020 to October 2023; from
|
|
28
|
+
Phase 2 a "week" is a two-week collection period). Questions came and went, so a measure is empty in weeks
|
|
29
|
+
that did not ask it (EXPENSE is empty in weeks 01-12, first asked in week 13, August 2020). The survey ended on 2024-09-16 and continued from 2025 inside HTOPS,
|
|
30
|
+
whose tables are not on the API.
|
|
31
|
+
`RATE` is the percent of `UNIVERSE` (`TOTAL` / `UNIVERSE`; EXPENSE, 2024 cycle 09: 87,697,083 / 236,337,862 =
|
|
32
|
+
37.1), not of `UNITS_TOTAL` (all adults). Report `RATE_MOE` (`<M>_RT_MOE`) with every rate; Census uses
|
|
33
|
+
90 percent confidence intervals for Pulse, but the API does not state the level of its margins. State margins
|
|
34
|
+
are several times the national one (cycle 09 EXPENSE: Maryland 34.3 +/- 3.4, nation 37.1 +/- 0.7): test a
|
|
35
|
+
difference against both margins before calling it one. Do not splice `pulse_covid` and `pulse` into one series
|
|
36
|
+
or compare across phases without saying the question set and contact frame changed (Phase 4.2 added prepaid
|
|
37
|
+
wireless numbers, "potentially impacting comparability with previous cycles", Phase 4.2 User Notes).
|
|
38
|
+
- HVS (`eits_homeownership`, timeseries/eits/hv) is quarterly (`time=2025-Q3`), from the CPS: `category_code`
|
|
39
|
+
RATE has `data_type_code` HOR (homeownership rate, not seasonally adjusted), SAHOR (seasonally adjusted,
|
|
40
|
+
`seasonally_adj` yes), RVR (rental vacancy rate), HVR (homeowner vacancy rate), and E_HOR, E_RVR, E_HVR
|
|
41
|
+
(`error_data` yes: the 90 percent margin of error in percentage points, e.g. 0.5 for HOR); category ESTIMATE
|
|
42
|
+
holds the housing inventory in thousands (TOTAL, OCC, OWNOCC, RNTOCC, VACANT, RENT, SALE, SEASON, ...). Never
|
|
43
|
+
mix HOR and SAHOR in one series. The API has the nation only; the release's Table 4 is the benchmark (2025
|
|
44
|
+
HOR 65.1, 65.0, 65.3, 65.7 matched the API on 2026-11-03). Fourth quarter 2025 rests on November and December
|
|
45
|
+
only: October 2025 was not collected during the lapse in federal funding (release footnote); say so when it
|
|
46
|
+
is used.
|
|
47
|
+
|
|
48
|
+
## Spec examples
|
|
49
|
+
```json
|
|
50
|
+
{"description": "Retail sales, 2026 to date; customs duty on imports from China, August 2026, by HS chapter",
|
|
51
|
+
"requests": [{"name": "retail_2026", "endpoint": "query",
|
|
52
|
+
"params": {"dataset": "eits_retail", "get": "cell_value,data_type_code,category_code,seasonally_adj,time_slot_id,error_data",
|
|
53
|
+
"for": "us:*", "time": "2026"}},
|
|
54
|
+
{"name": "china_duty_2026_08", "endpoint": "query",
|
|
55
|
+
"params": {"dataset": "trade_imports_hs", "get": "CTY_NAME,I_COMMODITY,GEN_VAL_MO,CON_VAL_MO,DUT_VAL_MO,CAL_DUT_MO",
|
|
56
|
+
"CTY_CODE": "5700", "COMM_LVL": "HS2", "time": "2026-08"}}]}
|
|
57
|
+
```
|
|
58
|
+
```json
|
|
59
|
+
{"description": "State and local government finances by state, 2024",
|
|
60
|
+
"requests": [{"name": "govfin_2024", "endpoint": "query",
|
|
61
|
+
"params": {"dataset": "govs_finances", "get": "AGG_DESC,GOVTYPE,AMOUNT,AMOUNT_CV", "for": "state:*", "time": "2024"}}]}
|
|
62
|
+
```
|
|
63
|
+
```json
|
|
64
|
+
{"description": "Adults with difficulty paying usual household expenses, nation, every 2024 Pulse cycle; homeownership rate 2025",
|
|
65
|
+
"requests": [{"name": "pulse_expense_2024", "endpoint": "query",
|
|
66
|
+
"params": {"dataset": "pulse", "get": "MEASURE_NAME,MEASURE_LABEL,RATE,RATE_MOE,TOTAL,UNIVERSE,CYCLE,PHASE,COL_START_DATE,COL_END_DATE",
|
|
67
|
+
"for": "us:*", "time": "2024", "MEASURE_NAME": "EXPENSE"}},
|
|
68
|
+
{"name": "homeownership_2025", "endpoint": "query",
|
|
69
|
+
"params": {"dataset": "eits_homeownership", "get": "cell_value,data_type_code,category_code,seasonally_adj,time_slot_id,geo_level_code,error_data",
|
|
70
|
+
"for": "us:*", "time": "2025", "category_code": "RATE"}}]}
|
|
71
|
+
```
|
package/packs/census/source.md
CHANGED
|
@@ -53,7 +53,7 @@ returns the national cross-section for one date, to filter by state FIPS),
|
|
|
53
53
|
`pep_population_2021` 2021/pep/population (POP_2020, POP_2021; states and above only),
|
|
54
54
|
`pep_population_2019` (2010-2019, down to places), `cbp` 2023/cbp,
|
|
55
55
|
`ecnbasic_2022`, `sipp` latest/sipp (SIPP microdata; 2022-2025 on the API, one per calendar year).
|
|
56
|
-
Microdata (one row per person;
|
|
56
|
+
Microdata (one row per person; the traps are in doc "microdata"): `acs1_pums` latest/acs/acs1/pums (2004-2024),
|
|
57
57
|
`acs5_pums` latest/acs/acs5/pums (2009-2024), `acs1_pumspr` (Puerto Rico), `cps_asec` latest/cps/asec/mar
|
|
58
58
|
(1992-2026, survey year: income is for the year before), CPS basic monthly `<YYYY>/cps/basic/<jan..dec>`
|
|
59
59
|
(1989-2026).
|
|
@@ -95,46 +95,6 @@ Find others: `request_schema` search `"median household income"` (latest ACS 5-y
|
|
|
95
95
|
"params": {"dataset": "dec_pl_2020", "get": "NAME,P1_001N", "for": "state:*"}}]}
|
|
96
96
|
```
|
|
97
97
|
|
|
98
|
-
```json
|
|
99
|
-
{"description": "Share of people without health coverage in December, SIPP 2024 (weighted by WPFINWGT)",
|
|
100
|
-
"requests": [{"name": "sipp_coverage_dec", "endpoint": "query",
|
|
101
|
-
"params": {"dataset": "2024/sipp", "get": "TAGE,RHLTHMTH,TPTOTINC", "MONTHCODE": "12"}}]}
|
|
102
|
-
```
|
|
103
|
-
|
|
104
|
-
```json
|
|
105
|
-
{"description": "Uninsured rate, Maryland, ACS 2024 1-year PUMS with replicate weights; CPS September 2026",
|
|
106
|
-
"requests": [{"name": "md_pums", "endpoint": "query",
|
|
107
|
-
"params": {"dataset": "2024/acs/acs1/pums", "get": "AGEP,HICOV,RELSHIPP,ESR,PINCP", "state": "MD",
|
|
108
|
-
"replicates": true}},
|
|
109
|
-
{"name": "cps_sep", "endpoint": "query",
|
|
110
|
-
"params": {"dataset": "2026/cps/basic/sep", "get": "PRTAGE,PEMLR,PRCITSHP", "PRTAGE": "25:54"}}]}
|
|
111
|
-
```
|
|
112
|
-
|
|
113
|
-
```json
|
|
114
|
-
{"description": "Retail sales, 2026 to date; customs duty on imports from China, August 2026, by HS chapter",
|
|
115
|
-
"requests": [{"name": "retail_2026", "endpoint": "query",
|
|
116
|
-
"params": {"dataset": "eits_retail", "get": "cell_value,data_type_code,category_code,seasonally_adj,time_slot_id,error_data",
|
|
117
|
-
"for": "us:*", "time": "2026"}},
|
|
118
|
-
{"name": "china_duty_2026_08", "endpoint": "query",
|
|
119
|
-
"params": {"dataset": "trade_imports_hs", "get": "CTY_NAME,I_COMMODITY,GEN_VAL_MO,CON_VAL_MO,DUT_VAL_MO,CAL_DUT_MO",
|
|
120
|
-
"CTY_CODE": "5700", "COMM_LVL": "HS2", "time": "2026-08"}}]}
|
|
121
|
-
```
|
|
122
|
-
```json
|
|
123
|
-
{"description": "State and local government finances by state, 2024",
|
|
124
|
-
"requests": [{"name": "govfin_2024", "endpoint": "query",
|
|
125
|
-
"params": {"dataset": "govs_finances", "get": "AGG_DESC,GOVTYPE,AMOUNT,AMOUNT_CV", "for": "state:*", "time": "2024"}}]}
|
|
126
|
-
```
|
|
127
|
-
|
|
128
|
-
```json
|
|
129
|
-
{"description": "Adults with difficulty paying usual household expenses, nation, every 2024 Pulse cycle; homeownership rate 2025",
|
|
130
|
-
"requests": [{"name": "pulse_expense_2024", "endpoint": "query",
|
|
131
|
-
"params": {"dataset": "pulse", "get": "MEASURE_NAME,MEASURE_LABEL,RATE,RATE_MOE,TOTAL,UNIVERSE,CYCLE,PHASE,COL_START_DATE,COL_END_DATE",
|
|
132
|
-
"for": "us:*", "time": "2024", "MEASURE_NAME": "EXPENSE"}},
|
|
133
|
-
{"name": "homeownership_2025", "endpoint": "query",
|
|
134
|
-
"params": {"dataset": "eits_homeownership", "get": "cell_value,data_type_code,category_code,seasonally_adj,time_slot_id,geo_level_code,error_data",
|
|
135
|
-
"for": "us:*", "time": "2025", "category_code": "RATE"}}]}
|
|
136
|
-
```
|
|
137
|
-
|
|
138
98
|
## Data format and traps
|
|
139
99
|
- Margins of error: every ACS `...E` estimate is fetched with its `...M` margin of error (`...PE` with
|
|
140
100
|
`...PM`), each in its own column, at the 90% confidence level. They are never dropped (`moe: false`
|
|
@@ -149,7 +109,8 @@ Find others: `request_schema` search `"median household income"` (latest ACS 5-y
|
|
|
149
109
|
interval are not exact; the group() call returns the EA annotation (median+, median-).
|
|
150
110
|
- Values arrive as text, exactly as published; NAME reads "Montgomery County, Maryland". Geography
|
|
151
111
|
code columns (state, county, place, tract) are strings with leading zeros: keep them as text.
|
|
152
|
-
- 1-year vs 5-year: 1-year (normally released each September for the prior year; the 2025 release was not on
|
|
112
|
+
- 1-year vs 5-year: 1-year (normally released each September for the prior year; the 2025 release was not on
|
|
113
|
+
the API on 2026-10-20) covers only areas of 65,000+
|
|
153
114
|
and has no tracts, block groups or ZCTAs; 5-year (normally each December) covers all areas but pools
|
|
154
115
|
five years. Never compare a 1-year with a 5-year figure, and do not compare overlapping 5-year
|
|
155
116
|
periods (2019-2023 vs 2020-2024). No standard 2020 ACS 1-year release exists (experimental only).
|
|
@@ -170,106 +131,13 @@ Find others: `request_schema` search `"median household income"` (latest ACS 5-y
|
|
|
170
131
|
the last API vintage with place and county subdivision estimates. Vintages 2022 and 2024-2025 (county
|
|
171
132
|
and city/town totals for 2022-2025) are not on the API: use the Census estimates files at
|
|
172
133
|
https://www.census.gov/programs-surveys/popest/data/tables.html (manual download) and name the vintage.
|
|
173
|
-
- SIPP (`
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
weights, which are not on the API (bulk file `rw<YYYY>_csv.zip`, REPWGT1-REPWGT240, at
|
|
181
|
-
https://www2.census.gov/programs-surveys/sipp/data/datasets/<YYYY>/; method in the year's user guide); without them
|
|
182
|
-
give the point estimate and say no standard error was computed. Variable names and codes:
|
|
183
|
-
`request_schema` search `"<YYYY>/sipp: health insurance"` or the dataset's variables.json. The bulk primary file
|
|
184
|
-
(`pu<YYYY>_csv.zip`, about 100 MB) has every variable, for a manual download.
|
|
185
|
-
- ACS PUMS and CPS microdata (`acs1_pums`, `acs5_pums`, `cps_asec`, `<YYYY>/cps/basic/<mon>`): one row per
|
|
186
|
-
person, values and codes exactly as published, with `<request>_codebook.csv` (every code's label, the
|
|
187
|
-
variable's suggested weight). The pack adds the keys and weights: ACS SERIALNO, SPORDER, PWGTP (persons), WGTP
|
|
188
|
-
(households: SPORDER 1 with RELSHIPP 20); ASEC H_SEQ, A_LINENO, PERIDNUM, MARSUPWT, HSUP_WGT; basic HRHHID,
|
|
189
|
-
HRHHID2, PULINENO, PWSSWGT (final weight), PWCMPWGT (composite: employment levels that match BLS). For hourly
|
|
190
|
-
and weekly earnings in a basic month, request PWORWGT (outgoing rotation groups, HRMIS 4 and 8). Every estimate
|
|
191
|
-
is weighted, never a row count. Filters are upper-case variable predicates (`"AGEP": "25:54"`, `"HICOV": 2`),
|
|
192
|
-
repeated on every call; geography: `state`, or `for: "public use microdata area:*"` with a state (ACS);
|
|
193
|
-
no state = the nation (ACS fetched state by state, 51 calls, about 2 minutes for the 3.4 million persons).
|
|
194
|
-
Every result has a `state` column (the FIPS code): group on it. The ACS PUMS state variable is `ST` through
|
|
195
|
-
the 2022 file and `STATE` from 2023 on (1-year, 5-year and Puerto Rico; the other name is not a variable).
|
|
196
|
-
`replicates: true` adds PWGTP1-80 for ACS (`"household"`: WGTP1-80, `"both"`); columns past 50 come in
|
|
197
|
-
further calls joined on the keys. For the ASEC (2005 on) it adds PH_SEQ and PPPOS and joins PWWGT1-160 from
|
|
198
|
-
Census's replicate weight file (CPS_ASEC_ASCII_REPWGT_<year> on www2.census.gov, about 70 MB, no key), after
|
|
199
|
-
two gates: the file's weight sums equal the totals its SAS layout states, and every person's PWWGT0 equals
|
|
200
|
-
MARSUPWT (2020: the 2020-based file is tried when the main one does not match). Census's replicate standard error
|
|
201
|
-
is sqrt(4/80 * sum of squared replicate differences), MOE = 1.645 * SE (`compute`'s `replicate_se` step with
|
|
202
|
-
`method: "sdr"` gives both; see Analysis below). Checked 2026-11-04 with that method: Maryland's 2024 uninsured rate in the civilian
|
|
203
|
-
noninstitutionalized population from PUMS was 6.26 +/- 0.32 percent, published S2701 6.3 +/- 0.3; the nation
|
|
204
|
-
8.19 vs 8.2.
|
|
205
|
-
- Universe: published insurance and labor tables use the civilian noninstitutionalized population
|
|
206
|
-
(drop RELSHIPP 37, the institutionalized group quarters population, and ESR 4-5, armed forces). Codes below zero or 0 often mean N/A (PINCP -19999 under 15, ESR 0 under 16): read the codebook.
|
|
207
|
-
- Dollars: ACS amounts need ADJINC (ADJHSG for housing costs), which the pack adds when a dollar variable asks
|
|
208
|
-
for it: value * ADJINC / 1,000,000 is in the survey year's dollars, as the published tables are. PUMS is a
|
|
209
|
-
subsample of the full ACS: a PUMS figure differs slightly from the published table (inside its MOE);
|
|
210
|
-
publish your figure as your own calculation and benchmark it against the table.
|
|
211
|
-
- CPS: ASEC income, poverty (PERLIS) and full-year insurance (COV) refer to the previous calendar year; NOW_COV
|
|
212
|
-
is coverage at the interview. Census's method for standard errors: weight MARSUPWT with replicates
|
|
213
|
-
PWWGT1-160, SE = sqrt(4/160 * sum of squared differences); household estimates on householders only, with
|
|
214
|
-
HSUP_WGT and the same replicates. Checked 2026-11-04 with that method: 2025 official poverty from the 2026 ASEC was
|
|
215
|
-
34,505 thousand +/- 909 and 10.2 +/- 0.3 percent (PERLIS 1 among PERLIS != -1), P60-290 Table 1 34,510 +/- 909
|
|
216
|
-
and 10.2 +/- 0.3. Before 2005 there is no replicate file: use the ipums pack.
|
|
217
|
-
The API's 2020 and 2021 ASEC weights are the 2010-Census-based ones (their PWWGT0 matched the main replicate
|
|
218
|
-
file, checked 2026-11-04); Census advises its 2020-based weights to compare those years with 2022 on, and
|
|
219
|
-
they are not on the API: say which weights a 2020-2021 comparison uses.
|
|
220
|
-
ASEC files before 2001 take no state geography on the API (HTTP 500): fetch the nation and filter on the
|
|
221
|
-
file's state variable, or use ipums. Older files lack some keys and weights (HRHHID2 before 2004, PERIDNUM
|
|
222
|
-
in early ASECs): the pack leaves them out and says so. CPS basic monthly has no replicate weights at all; month-to-month person links need HRHHID, HRHHID2 and
|
|
223
|
-
PULINENO with care (ipums CPSIDP links them). CPS codes differ from ACS codes (PEMLR, PRCITSHP, A_HGA /
|
|
224
|
-
PEEDUCA): recode each survey from its own codebook.
|
|
225
|
-
- Analysis: build tables with `compute`. Standard errors: fetch with `replicates`, then a `replicate_se` step,
|
|
226
|
-
e.g. `{"groupby": ["state"], "replicate_se": {"estimates": {"uninsured": "sum(PWGTP * (HICOV == 2)) / sum(PWGTP)"},
|
|
227
|
-
"weight": "PWGTP", "replicates": "PWGTP1:PWGTP80", "method": "sdr"}}` (ASEC: `MARSUPWT`, `"PWWGT1:PWWGT160"`;
|
|
228
|
-
household estimates: `WGTP`/`HSUP_WGT` with their replicates on one row per household). A difference between
|
|
229
|
-
years or groups needs its SE from the same replicates: write it as one estimate. CPS basic monthly and SIPP from
|
|
230
|
-
the API have no replicate weights: say no standard error was computed. This server has no PUMS recode helpers.
|
|
231
|
-
- Time series (`timeseries/...`): give `time` (`2026`, `2026-08`, `2026-Q2`, or `from 2024-01 to 2026-08`).
|
|
232
|
-
Economic indicators (eits) return one row per code combination: `category_code` (industry or tax type),
|
|
233
|
-
`data_type_code` (level, percent change, ...), `seasonally_adj` (yes/no), `time_slot_id`, `error_data` (yes =
|
|
234
|
-
the row is a margin or standard error, not an estimate). The codes are not in the API's metadata: run one
|
|
235
|
-
narrow query and read the codes that come back, or the program's definitions page (census.gov/econ/currentdata),
|
|
236
|
-
then filter on them; never sum across data types or adjustment flags. Monthly indicators are revised (advance,
|
|
237
|
-
preliminary, revised): say which release. Units: retail and construction $ millions, housing units in
|
|
238
|
-
thousands, QTAX `CELL_VALUE` $ millions (variables.json labels say so); government finances `AMOUNT` $ thousands
|
|
239
|
-
with a coefficient of variation `AMOUNT_CV` (percent, sample-based local figures). Government finance years are
|
|
240
|
-
each government's fiscal year, not the calendar year. International trade: values in dollars; `GEN_VAL_MO`
|
|
241
|
-
general imports vs `CON_VAL_MO` imports for consumption; `CAL_DUT_MO` is the duty calculated on imports for
|
|
242
|
-
consumption (an estimate of duty owed, not Treasury's customs receipts); `COMM_LVL` HS2/HS4/HS6/HS10 must be
|
|
243
|
-
fixed or totals double-count; `CTY_CODE` country codes (5700 China, 2010 Mexico, 1220 Canada); it takes no
|
|
244
|
-
for= geography; `SUMMARY_LVL` CGP rows are country groups, not countries (drop them before summing).
|
|
245
|
-
- Household Pulse (`pulse`, `pulse_covid`) is an experimental data product: an online survey of adults 18+ in
|
|
246
|
-
housing units, contacted by e-mail and text, with weighted response rates of 4.7 to 7.3 percent at the end
|
|
247
|
-
(7.3 percent in week 63, October 2023; 4.7 percent in Phase 4.2 cycle 09, September 2024; each period's Source
|
|
248
|
-
and Accuracy statement, census.gov/programs-surveys/household-pulse-survey/technical-documentation.html). Say so
|
|
249
|
-
with every figure. `pulse` (timeseries/hhpulse) is long: one row per `MEASURE_NAME` (13 measures: EXPENSE,
|
|
250
|
-
FOODSCARCE, ENERGYBILL, EVICTFOR, PRICEINCR, PRICEWORRY, LONELY, ...), `CYCLE` and geography; filter
|
|
251
|
-
`MEASURE_NAME` and `CYCLE`. It holds `time=2024` only (Phase 4.0 cycles 01-03, 4.1 04-07, 4.2 08-09, January 9 to
|
|
252
|
-
September 16, 2024); `time=2025` returns no rows. `pulse_covid` (timeseries/hps) is wide: `<M>_RATE`,
|
|
253
|
-
`<M>_RT_MOE`, `<M>_TOTAL`, `<M>_MOE`, `<M>_UNIV` per collection `WEEK` (01-63, April 2020 to October 2023; from
|
|
254
|
-
Phase 2 a "week" is a two-week collection period). Questions came and went, so a measure is empty in weeks
|
|
255
|
-
that did not ask it (EXPENSE is empty in weeks 01-12, first asked in week 13, August 2020). The survey ended on 2024-09-16 and continued from 2025 inside HTOPS,
|
|
256
|
-
whose tables are not on the API.
|
|
257
|
-
`RATE` is the percent of `UNIVERSE` (`TOTAL` / `UNIVERSE`; EXPENSE, 2024 cycle 09: 87,697,083 / 236,337,862 =
|
|
258
|
-
37.1), not of `UNITS_TOTAL` (all adults). Report `RATE_MOE` (`<M>_RT_MOE`) with every rate; Census uses
|
|
259
|
-
90 percent confidence intervals for Pulse, but the API does not state the level of its margins. State margins
|
|
260
|
-
are several times the national one (cycle 09 EXPENSE: Maryland 34.3 +/- 3.4, nation 37.1 +/- 0.7): test a
|
|
261
|
-
difference against both margins before calling it one. Do not splice `pulse_covid` and `pulse` into one series
|
|
262
|
-
or compare across phases without saying the question set and contact frame changed (Phase 4.2 added prepaid
|
|
263
|
-
wireless numbers, "potentially impacting comparability with previous cycles", Phase 4.2 User Notes).
|
|
264
|
-
- HVS (`eits_homeownership`, timeseries/eits/hv) is quarterly (`time=2025-Q3`), from the CPS: `category_code`
|
|
265
|
-
RATE has `data_type_code` HOR (homeownership rate, not seasonally adjusted), SAHOR (seasonally adjusted,
|
|
266
|
-
`seasonally_adj` yes), RVR (rental vacancy rate), HVR (homeowner vacancy rate), and E_HOR, E_RVR, E_HVR
|
|
267
|
-
(`error_data` yes: the 90 percent margin of error in percentage points, e.g. 0.5 for HOR); category ESTIMATE
|
|
268
|
-
holds the housing inventory in thousands (TOTAL, OCC, OWNOCC, RNTOCC, VACANT, RENT, SALE, SEASON, ...). Never
|
|
269
|
-
mix HOR and SAHOR in one series. The API has the nation only; the release's Table 4 is the benchmark (2025
|
|
270
|
-
HOR 65.1, 65.0, 65.3, 65.7 matched the API on 2026-11-03). Fourth quarter 2025 rests on November and December
|
|
271
|
-
only: October 2025 was not collected during the lapse in federal funding (release footnote); say so when it
|
|
272
|
-
is used.
|
|
134
|
+
- ACS PUMS, CPS and SIPP microdata (`acs1_pums`, `acs5_pums`, `cps_asec`, `cps/basic`, `sipp`): read
|
|
135
|
+
`source_brief({source: "census", doc: "microdata"})` before the first fetch. It has the keys and weights the
|
|
136
|
+
pack adds, the `ST`/`STATE` rename, replicate weights and standard errors, universes, ADJINC, and the
|
|
137
|
+
checked benchmarks.
|
|
138
|
+
- Time series (`timeseries/...`: economic indicators, international trade, government finances, Household
|
|
139
|
+
Pulse, HVS): read `source_brief({source: "census", doc: "timeseries"})` before the first fetch. It has the
|
|
140
|
+
`time` forms, the code columns to fix, units, revisions and each survey's caveats.
|
|
273
141
|
- Key: since 2026-05-12 every data query needs a key; the API redirects keyless queries to a "Missing
|
|
274
142
|
Key" page, which the pack reports as a key error. Metadata (variables, discovery) still answer
|
|
275
143
|
without a key.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Congress nominations, CRS reports and cost estimates
|
|
2
|
+
|
|
3
|
+
From the congress playbook's "Data format and traps" section: read it before a request of this kind.
|
|
4
|
+
|
|
5
|
+
- Cost estimates: `bill_cbo_estimates` gives only what Congress.gov lists: title, publication date,
|
|
6
|
+
description and a cbo.gov link for each estimate. No dollar figures are in the API; the estimate
|
|
7
|
+
itself is a PDF or web page (committee reports in govinfo CRPT often reprint it as text).
|
|
8
|
+
- CRS: `crs_reports` lists reports Congress.gov publishes (newest first, about 14,000 in Oct 2026);
|
|
9
|
+
there is no keyword search: use a date range and `title_contains`, then `crs_report` for the summary.
|
|
10
|
+
`fromDateTime`/`toDateTime` filter on the UPDATE date (one September window returned 1,244 products
|
|
11
|
+
published 2004-2026): for "published in a month" filter the rows on `publishDate` afterwards. `publishDate`
|
|
12
|
+
is the current version's date, so a product first issued in the month and revised later drops out (a
|
|
13
|
+
floor, say so). The list's `version` is not the product version: `crs_report` gives `currentVersion`.
|
|
14
|
+
`bill_summaries` are CRS summaries of each bill version (HTML in `text`, plain in `text_plain`); some fields
|
|
15
|
+
are over 131,072 characters: when reading the CSV with your own code, raise the reader's field size limit first.
|
|
16
|
+
- Nominations (traps seen in Oct 2026 data; the Senate Résumé's "Civilian nominees" table is the benchmark):
|
|
17
|
+
military and "Other Civilian" lists (Foreign Service, PHS, NOAA officers) are list nominations without a
|
|
18
|
+
description; `nominationType` (`isCivilian`, `isMilitary`) is missing on some rows of older Congresses
|
|
19
|
+
(every row of the 119th has one); a PN split into parts
|
|
20
|
+
(PN12-41) counts once per part; en bloc votes confirm many nominees at once. `latestAction` is the last
|
|
21
|
+
step, which can be a later procedural motion ("motion to reconsider tabled") or, for a nomination held over,
|
|
22
|
+
its confirmation in the next session: count outcomes by date from `nomination_actions`, within the
|
|
23
|
+
session's dates. A split PN's actions need `nominationNumber` "1255-17": `"for_each": {"csv":
|
|
24
|
+
"data/congress/<noms>.csv", "column": "citation", "param": "nominationNumber", "strip": "PN"}` (`number` alone
|
|
25
|
+
returns nothing for a split PN). The naive count (civilian flag + `latestAction` "Confirmed") gave 60 percent for 2023
|
|
26
|
+
against the Résumé's 47.
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Congress votes: roll calls, party totals and member ids
|
|
2
|
+
|
|
3
|
+
From the congress playbook's "Data format and traps" section: read it before a request of this kind.
|
|
4
|
+
|
|
5
|
+
- Votes: a bill usually has several recorded votes (cloture, motions to proceed, amendments, passage,
|
|
6
|
+
conference). `bill_votes` lists every one with the action text. Final passage is each chamber's LAST
|
|
7
|
+
recorded vote that cleared the bill ("On Passage", "On Passage of the Bill", "On Motion to Concur ...",
|
|
8
|
+
"On Agreeing to the Conference Report"): take the latest date, not the first "On Passage". A bill used
|
|
9
|
+
as a vehicle for other text (CHIPS and Science moved as H.R. 4346, first passed as a legislative-branch
|
|
10
|
+
appropriation) has an early "On Passage" vote on the old text; the Clerk's `description` can keep the
|
|
11
|
+
old title. Voice votes and unanimous consent leave no roll call and no row.
|
|
12
|
+
- Party totals: House rows take the Clerk's totals by party; Senate rows are counted from the member
|
|
13
|
+
list (`yea_D`, `nay_R`, ...; independents as I). `not_voting` counts absent members; vacant seats are
|
|
14
|
+
not counted anywhere, so totals can be under 435 or 100. The Vice President's tie-breaking vote is in
|
|
15
|
+
`tie_breaker`, not in the member rows.
|
|
16
|
+
- Member ids differ by source: House rows and Congress.gov use bioguide ids (A000370); Senate XML uses
|
|
17
|
+
LIS ids (S354). Join Senate votes to members by name and state, or a crosswalk, not by id.
|
|
18
|
+
- `vote` keeps the published word (Yea, Aye, Nay, No, Present, Not Voting, Guilty); `position` maps it
|
|
19
|
+
to yea, nay, present, not_voting or other. House "Aye"/"No" are recorded votes, same meaning.
|
|
20
|
+
- Whole-session vote counts and party splits: no endpoint returns party totals in bulk, so a session takes
|
|
21
|
+
one `roll_call` per vote (about 1,000 a year, several minutes). For the House's questions and results
|
|
22
|
+
(`house_votes` has no question column; `house_vote` keeps only party totals), `for_each` over the
|
|
23
|
+
`house_votes` list's `rollCallNumber` with `roll_call` `"summary": "true"` and the year: one row per vote with
|
|
24
|
+
question ("On Motion to Suspend the Rules and Pass"), vote type ("2/3 YEA-AND-NAY"), result and totals. `house_votes` lists quorum calls too (vote
|
|
25
|
+
type "Quorum"): the Résumé counts them apart from yea-and-nay and recorded votes. The Speaker election's
|
|
26
|
+
member rows have position `other` (votes are names). Say how independents are counted in a party split.
|
|
27
|
+
- `house_votes`/`house_vote` (Congress.gov) start with the 115th Congress (2017; the 114th lists 0 votes,
|
|
28
|
+
verified 2026-10-22); `roll_call` reads the Clerk
|
|
29
|
+
XML for 1990 on (and falls back to the Congress.gov member list for the 115th on if the Clerk file
|
|
30
|
+
cannot be read). Senate XML covers the 101st Congress (1989) on.
|