data-analyst-mcp 0.7.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (63) hide show
  1. package/dist/analysis/combine.js +3 -2
  2. package/dist/analysis/combine.js.map +1 -1
  3. package/dist/analysis/compute.d.ts +26 -5
  4. package/dist/analysis/compute.js +402 -159
  5. package/dist/analysis/compute.js.map +1 -1
  6. package/dist/analysis/expr.js +2 -1
  7. package/dist/analysis/expr.js.map +1 -1
  8. package/dist/analysis/handoff.d.ts +1 -1
  9. package/dist/analysis/handoff.js +44 -3
  10. package/dist/analysis/handoff.js.map +1 -1
  11. package/dist/analysis/table.d.ts +7 -0
  12. package/dist/analysis/table.js +30 -1
  13. package/dist/analysis/table.js.map +1 -1
  14. package/dist/analysis/tools.d.ts +10 -7
  15. package/dist/analysis/tools.js +32 -43
  16. package/dist/analysis/tools.js.map +1 -1
  17. package/dist/core/compact.d.ts +15 -0
  18. package/dist/core/compact.js +98 -0
  19. package/dist/core/compact.js.map +1 -0
  20. package/dist/core/package.d.ts +8 -0
  21. package/dist/core/package.js +30 -1
  22. package/dist/core/package.js.map +1 -1
  23. package/dist/core/packs.d.ts +4 -2
  24. package/dist/core/sections.d.ts +21 -0
  25. package/dist/core/sections.js +53 -0
  26. package/dist/core/sections.js.map +1 -0
  27. package/dist/deep/opm.js +10 -2
  28. package/dist/deep/opm.js.map +1 -1
  29. package/dist/deep/treasury.js +7 -3
  30. package/dist/deep/treasury.js.map +1 -1
  31. package/dist/envelope.js +1 -1
  32. package/dist/envelope.js.map +1 -1
  33. package/dist/index.d.ts +57 -4
  34. package/dist/report/build_report.d.ts +18 -0
  35. package/dist/report/build_report.js +52 -7
  36. package/dist/report/build_report.js.map +1 -1
  37. package/dist/report/tools.d.ts +26 -1
  38. package/dist/report/tools.js +74 -17
  39. package/dist/report/tools.js.map +1 -1
  40. package/dist/server.d.ts +1 -1
  41. package/dist/server.js +52 -43
  42. package/dist/server.js.map +1 -1
  43. package/dist/session.d.ts +2 -0
  44. package/dist/session.js +2 -0
  45. package/dist/session.js.map +1 -1
  46. package/dist/tools.d.ts +98 -6
  47. package/dist/tools.js +154 -17
  48. package/dist/tools.js.map +1 -1
  49. package/guide/WORKFLOW.md +7 -4
  50. package/guide/references/compute.md +51 -0
  51. package/guide/references/report_quick.md +60 -0
  52. package/package.json +1 -1
  53. package/packs/census/docs/microdata.md +77 -0
  54. package/packs/census/docs/timeseries.md +71 -0
  55. package/packs/census/source.md +10 -142
  56. package/packs/congress/docs/nominations_crs.md +26 -0
  57. package/packs/congress/docs/votes.md +30 -0
  58. package/packs/congress/source.md +4 -48
  59. package/packs/govinfo/docs/collections.md +88 -0
  60. package/packs/govinfo/source.md +6 -86
  61. package/packs/irs-soi/docs/eo.md +15 -0
  62. package/packs/irs-soi/docs/tables.md +50 -0
  63. package/packs/irs-soi/source.md +6 -57
package/guide/WORKFLOW.md CHANGED
@@ -20,7 +20,7 @@ fetched file. **Accuracy comes before everything else.**
20
20
  | `cite(url, id, text?)` | snapshot a page a reference cites (pass the text your own fetch tool returned) |
21
21
  | `rerun` | replay every exhibit and fact from the fetched data |
22
22
  | `audit_answer(answer, rerun)` | the release gate for a quick answer: send the answer only when it prints AUDIT PASSED |
23
- | `build_report(report)`, `style_check`, `audit_report` | a report or paper, only when asked |
23
+ | `build_report(report)`, `audit_report` | a report or paper, only when asked (`style_check` lints prose alone) |
24
24
  | `handoff` | the per-source data hand-off (data-only replies) |
25
25
  | `read_file(path)` | a small text file from the workspace |
26
26
 
@@ -120,9 +120,12 @@ FAIL, and send it only when it prints AUDIT PASSED. Data only: send the `handoff
120
120
  and what could not be fetched with its route.
121
121
 
122
122
  ## 7. Report or paper (only when asked)
123
- Read `guide("report_patterns")`, `guide("report_schema")` (every report.json key) and `guide("style")`. Write the
124
- report as `build_report`'s report object, with every number as `{{fact_name}}` and a `sources` list of the pack
125
- ids used; run `style_check`, `build_report`, then `audit_report` (`rerun: true`), and rebuild until it prints
123
+ Read `guide("report")`: what the build and the audit require, a minimal report, the chart keys and the style
124
+ rules, in one page. The full references come by section when needed (`guide("report_schema")`,
125
+ `guide("report_patterns")`, `guide("style")`). Write the report as `build_report`'s report object, with every
126
+ number as `{{fact_name}}` and a `sources` list of the pack ids used. `build_report` returns the style score with the
127
+ sentences to fix; fix with `patch` (or with no report: it rebuilds from report.json), never by sending the whole
128
+ object again. Then `audit_report` (`rerun: true`): it returns the failures only; rebuild until it prints
126
129
  AUDIT PASSED. The builder applies each source's notices, redistribution rules (microdata never leaves the
127
130
  workspace) and vintage blocks.
128
131
 
@@ -0,0 +1,51 @@
1
+ # compute: steps, expressions and survey standard errors
2
+
3
+ `compute` runs declarative steps over full workspace tables (`data/<id>/<table>.csv` or `.parquet`, earlier
4
+ exhibits, `published/*.csv`) and saves `exhibits/<name>.csv` with its lineage, so the rerun gate can replay it.
5
+
6
+ ## Steps
7
+ Each step is an object with one verb:
8
+ - `{filter: expr}`, `{derive: {col: expr, ...}}` (a later name may use an earlier one in the same step),
9
+ `{select: [cols]}`, `{rename: {old: new}}`.
10
+ - `{groupby: [keys], rollup: {col: aggregate expr}}`. `groupby` also goes with `derive` or `filter`: per-group
11
+ windows and shares.
12
+ - `{join: alias, on: [keys] | {left: right}, how: inner|left, suffix?: ["_x", "_y"]}`.
13
+ - `{pivot_wider: {names_from, values_from, id_cols?, fill?}}`, `{pivot_longer: {cols, names_to, values_to}}`.
14
+ - `{orderby: [col, "-col" for descending]}`, `{slice: n | [start, end]}`, `{dedupe: true | [cols]}`,
15
+ `{concat: [aliases]}`.
16
+ - `{as: name}` keeps the current table under a name; `{from: name}` switches to a table.
17
+ - Combine helpers:
18
+ - `{fiscal_year: {date, as?}}`.
19
+ - `{to_fiscal: {date, value, how: sum|mean|last, freq: M|Q}}` adds `periods` and `complete`; incomplete years
20
+ are flagged, never full.
21
+ - `{state_codes: col}` adds `state_usps`, `state_fips`, `state_name`.
22
+ - `{deflate: {value, period, index: alias, index_period?, index_value, name: "CPI-U, BLS CUUR0000SA0", base?: "2024", as?}}`.
23
+ - `{align_check: {tables: [aliases or "this"], on: [keys], units: {alias: "billions of dollars, nominal, fiscal years"}}}`
24
+ refuses repeated keys, missing units and no overlap. Run it before a join of two sources.
25
+
26
+ ## Expressions
27
+ Column names (`col("a b")` for odd names), numbers, `"text"`, `true`/`false`/`null`, `+ - * / % **`,
28
+ `== != < <= > >=`, `&& || !` (and/or/not), `c ? a : b`. Nothing else runs: no property access, no other calls.
29
+ CSV cells that are plain numbers are numbers; `"1,234"` stays text until `number()`; an empty cell is null (a
30
+ gap). Parquet cells are typed the same way. The functions, aggregates and windows are listed at the end.
31
+
32
+ ## Big inputs
33
+ Read only the columns needed: `{alias: {path, columns: [...]}}`. compute also reads only the columns a recipe
34
+ names when it can tell them from the steps. A file over the cell cap is refused: name its columns, or use the
35
+ Parquet file when the fetch wrote one.
36
+
37
+ ## Survey standard errors (replicate weights)
38
+ `{groupby?: [keys], replicate_se: {estimates: {rate: "sum(PWGTP * (HICOV == 2)) / sum(PWGTP)"}, weight: "PWGTP",
39
+ replicates: "PWGTP1:PWGTP80" | [cols], method: sdr|fay|jk1|brr|custom, rho? (fay), scale? (custom),
40
+ also?: {n: "count()"}, confidence?: 90|95|99}}`
41
+
42
+ It recomputes each weighted estimate with every replicate weight in place of the weight, and adds `<name>_se`
43
+ and `<name>_moe` (90 percent by default, as Census publishes).
44
+ - SE = sqrt(c × Σ_r (estimate_r − estimate)²).
45
+ - c = 4/R for sdr (ACS PUMS 4/80, CPS ASEC 4/160); 1/(R(1−rho)²) for fay; (R−1)/R for jk1; 1/R for brr.
46
+ Use the method the pack's notes name.
47
+ - An SE of a difference or ratio needs it as one estimate, e.g.
48
+ `"sum(W*(YEAR==2024)*X)/sum(W*(YEAR==2024)) - sum(W*(YEAR==2023)*X)/sum(W*(YEAR==2023))"`.
49
+
50
+ `{replicate_se: {estimate: col, replicates, method, as?}}` instead reads a table that already holds replicate
51
+ estimates (one row per estimate) and adds `<as>_se` and `<as>_moe` to each row.
@@ -0,0 +1,60 @@
1
+ # Report in short
2
+
3
+ What `build_report` and `audit_report` need, in one page. The full references are by section:
4
+ `guide("report_schema")` (every report.json key), `guide("report_patterns")` (structure, benchmarks, chart forms)
5
+ and `guide("style")` (the house style). Ask for a section only when this page does not settle a question.
6
+
7
+ ## The steps
8
+ 1. Every number in the prose is a fact: register it with `facts`, write it as `{{name}}` (or `{{name|fmt}}`).
9
+ References are `[@id]`, footnotes `[^n]`. A number typed by hand fails the audit unless `audit_allow` lists it.
10
+ 2. Every exhibit is a CSV in `exhibits/` saved by `compute` or `write_table`.
11
+ 3. `build_report` with the report object. It writes report.json, report.html and replication_package.zip, and
12
+ returns the style score with the sentences to fix. Later fixes: `build_report` with `patch`
13
+ (`{"sections.0.body": "..."}`) or with no report (it rebuilds from report.json). Never send the whole object
14
+ again for a small fix.
15
+ 4. `audit_report` (`rerun: true` replays every recipe and fact). It returns the failures only. Fix them and
16
+ rebuild until it prints AUDIT PASSED. A rerun that is still going answers `in_progress`: call again.
17
+
18
+ ## What the audit requires (report mode)
19
+ - `at_a_glance.text` and 3 to 5 findings, each `{"lead": "Section name.", "text": "..."}`.
20
+ - `notes_page`; `introduction.text` and `introduction.subsections` (question-style headings, one per section).
21
+ - 2 or more `sections`, each showing an exhibit (`"exhibits": [n]`, and "see Exhibit n" in the body).
22
+ - `text` on every exhibit; `about`; `methods` (Appendix A: request, sources, filters, universes, weights,
23
+ deflator, benchmarks, vintages, limitations).
24
+ - When a source records a vintage, `methods` or `notes_page` states it and says the source revises its data.
25
+ - `sources`: the pack ids used. The builder adds each source's notices, the Data Vintage block, the assumptions
26
+ table and the citation; never type those by hand.
27
+
28
+ ## A minimal report
29
+ ```json
30
+ {
31
+ "mode": "report", "title": "Unemployment, 2019 to 2025", "date": "2026-11-01", "sources": ["bls"],
32
+ "cover": {"exhibit": 1},
33
+ "at_a_glance": {"text": "...", "findings": [{"lead": "Unemployment.", "text": "The rate was {{ur_last}}."}]},
34
+ "notes_page": ["Unless otherwise indicated, all years are calendar years."],
35
+ "introduction": {"text": "...", "subsections": [{"heading": "How Did Unemployment Change?", "body": "..."}]},
36
+ "sections": [{"heading": "Unemployment", "body": "... (see Exhibit 1).[^1]", "exhibits": [1],
37
+ "footnotes": ["Seasonally adjusted."]}],
38
+ "exhibits": [{"title": "Unemployment Rate, 2019 to 2025", "subtitle": "Percent", "text": "...",
39
+ "data": "exhibits/unemployment.csv",
40
+ "chart": {"type": "line", "x": "month", "y": "value", "date": "date", "format": "pct100", "decimals": 1},
41
+ "source": "Analyst calculations using data from the Bureau of Labor Statistics."}],
42
+ "methods": "...", "about": "..."
43
+ }
44
+ ```
45
+ `mode: "paper"` is prose-led, with the exhibits in the running text (a `summary` list instead of `at_a_glance`).
46
+
47
+ ## Charts
48
+ `chart`: `{"type", "x", "y", ...}`; every name is a column of the exhibit CSV. Types: `line` (default), `column`,
49
+ `hbar`, `stacked`, `stacked_h`, `table`. Long data: `series` (a column); wide data: `y` as a list. Others:
50
+ `filter`, `x_order`, `series_order` (at most 8 series), `date` (ISO dates, a calendar axis), `recessions`,
51
+ `format` (`number`, `pct` for fractions, `pct100`, `dollars`), `decimals`, `y_min`/`y_max`, `columns`.
52
+
53
+ ## Style rules the build checks
54
+ - No first person ("we", "our", "I") and no contractions; the analysis *estimates*.
55
+ - *percent* in words; *percentage points* for differences; zero to nine in words, 10 and up in numerals; no
56
+ sentence starts with a numeral.
57
+ - Ranges *from X to Y*; ages *ages 25 to 54*; years in full.
58
+ - Plain words: none of "e.g.", "i.e.", "etc.", "utilize", "prior to", "regarding", "methodology", "data set".
59
+ - Sentences short; titles and headings in headline style; every exhibit has a units line and a source line.
60
+ - Define each rate, universe and abbreviation at first use. Name the dollar year and the price index.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "data-analyst-mcp",
3
- "version": "0.7.0",
3
+ "version": "0.8.0",
4
4
  "description": "MCP server for U.S. government data: asks the critical questions first, fetches from the agency, cites every number.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -0,0 +1,77 @@
1
+ # Census microdata: ACS PUMS, CPS and SIPP
2
+
3
+ From the census playbook's "Data format and traps" section: read it before a request of this kind.
4
+
5
+ - SIPP (`sipp`, `20XX/sipp`): public-use microdata, one row per person per reference month (MONTHCODE
6
+ 1-12). The reference year is the calendar year before the file's label: `2024/sipp` was collected in
7
+ 2024 about January-December 2023; say "2023" for its figures. Filter MONTHCODE (12 = December) for a
8
+ point in time, or average the months for the year. The pack always adds SSUID, PNUM (person = SSUID + PNUM), MONTHCODE and
9
+ WPFINWGT, the person weight: every estimate is weighted by WPFINWGT, never a row count. No geography but
10
+ `state`, which filters TST_INTV (state of the interview address); SIPP is designed for national estimates.
11
+ -9 means NA or missing in most variables: drop it before summing. Standard errors need the replicate
12
+ weights, which are not on the API (bulk file `rw<YYYY>_csv.zip`, REPWGT1-REPWGT240, at
13
+ https://www2.census.gov/programs-surveys/sipp/data/datasets/<YYYY>/; method in the year's user guide); without them
14
+ give the point estimate and say no standard error was computed. Variable names and codes:
15
+ `request_schema` search `"<YYYY>/sipp: health insurance"` or the dataset's variables.json. The bulk primary file
16
+ (`pu<YYYY>_csv.zip`, about 100 MB) has every variable, for a manual download.
17
+ - ACS PUMS and CPS microdata (`acs1_pums`, `acs5_pums`, `cps_asec`, `<YYYY>/cps/basic/<mon>`): one row per
18
+ person, values and codes exactly as published, with `<request>_codebook.csv` (every code's label, the
19
+ variable's suggested weight). The pack adds the keys and weights: ACS SERIALNO, SPORDER, PWGTP (persons), WGTP
20
+ (households: SPORDER 1 with RELSHIPP 20); ASEC H_SEQ, A_LINENO, PERIDNUM, MARSUPWT, HSUP_WGT; basic HRHHID,
21
+ HRHHID2, PULINENO, PWSSWGT (final weight), PWCMPWGT (composite: employment levels that match BLS). For hourly
22
+ and weekly earnings in a basic month, request PWORWGT (outgoing rotation groups, HRMIS 4 and 8). Every estimate
23
+ is weighted, never a row count. Filters are upper-case variable predicates (`"AGEP": "25:54"`, `"HICOV": 2`),
24
+ repeated on every call; geography: `state`, or `for: "public use microdata area:*"` with a state (ACS);
25
+ no state = the nation (ACS fetched state by state, 51 calls, about 2 minutes for the 3.4 million persons).
26
+ Every result has a `state` column (the FIPS code): group on it. The ACS PUMS state variable is `ST` through
27
+ the 2022 file and `STATE` from 2023 on (1-year, 5-year and Puerto Rico; the other name is not a variable).
28
+ `replicates: true` adds PWGTP1-80 for ACS (`"household"`: WGTP1-80, `"both"`); columns past 50 come in
29
+ further calls joined on the keys. For the ASEC (2005 on) it adds PH_SEQ and PPPOS and joins PWWGT1-160 from
30
+ Census's replicate weight file (CPS_ASEC_ASCII_REPWGT_<year> on www2.census.gov, about 70 MB, no key), after
31
+ two gates: the file's weight sums equal the totals its SAS layout states, and every person's PWWGT0 equals
32
+ MARSUPWT (2020: the 2020-based file is tried when the main one does not match). Census's replicate standard error
33
+ is sqrt(4/80 * sum of squared replicate differences), MOE = 1.645 * SE (`compute`'s `replicate_se` step with
34
+ `method: "sdr"` gives both; see Analysis below). Checked 2026-11-04 with that method: Maryland's 2024 uninsured rate in the civilian
35
+ noninstitutionalized population from PUMS was 6.26 +/- 0.32 percent, published S2701 6.3 +/- 0.3; the nation
36
+ 8.19 vs 8.2.
37
+ - Universe: published insurance and labor tables use the civilian noninstitutionalized population
38
+ (drop RELSHIPP 37, the institutionalized group quarters population, and ESR 4-5, armed forces). Codes below zero or 0 often mean N/A (PINCP -19999 under 15, ESR 0 under 16): read the codebook.
39
+ - Dollars: ACS amounts need ADJINC (ADJHSG for housing costs), which the pack adds when a dollar variable asks
40
+ for it: value * ADJINC / 1,000,000 is in the survey year's dollars, as the published tables are. PUMS is a
41
+ subsample of the full ACS: a PUMS figure differs slightly from the published table (inside its MOE);
42
+ publish your figure as your own calculation and benchmark it against the table.
43
+ - CPS: ASEC income, poverty (PERLIS) and full-year insurance (COV) refer to the previous calendar year; NOW_COV
44
+ is coverage at the interview. Census's method for standard errors: weight MARSUPWT with replicates
45
+ PWWGT1-160, SE = sqrt(4/160 * sum of squared differences); household estimates on householders only, with
46
+ HSUP_WGT and the same replicates. Checked 2026-11-04 with that method: 2025 official poverty from the 2026 ASEC was
47
+ 34,505 thousand +/- 909 and 10.2 +/- 0.3 percent (PERLIS 1 among PERLIS != -1), P60-290 Table 1 34,510 +/- 909
48
+ and 10.2 +/- 0.3. Before 2005 there is no replicate file: use the ipums pack.
49
+ The API's 2020 and 2021 ASEC weights are the 2010-Census-based ones (their PWWGT0 matched the main replicate
50
+ file, checked 2026-11-04); Census advises its 2020-based weights to compare those years with 2022 on, and
51
+ they are not on the API: say which weights a 2020-2021 comparison uses.
52
+ ASEC files before 2001 take no state geography on the API (HTTP 500): fetch the nation and filter on the
53
+ file's state variable, or use ipums. Older files lack some keys and weights (HRHHID2 before 2004, PERIDNUM
54
+ in early ASECs): the pack leaves them out and says so. CPS basic monthly has no replicate weights at all; month-to-month person links need HRHHID, HRHHID2 and
55
+ PULINENO with care (ipums CPSIDP links them). CPS codes differ from ACS codes (PEMLR, PRCITSHP, A_HGA /
56
+ PEEDUCA): recode each survey from its own codebook.
57
+ - Analysis: build tables with `compute`. Standard errors: fetch with `replicates`, then a `replicate_se` step,
58
+ e.g. `{"groupby": ["state"], "replicate_se": {"estimates": {"uninsured": "sum(PWGTP * (HICOV == 2)) / sum(PWGTP)"},
59
+ "weight": "PWGTP", "replicates": "PWGTP1:PWGTP80", "method": "sdr"}}` (ASEC: `MARSUPWT`, `"PWWGT1:PWWGT160"`;
60
+ household estimates: `WGTP`/`HSUP_WGT` with their replicates on one row per household). A difference between
61
+ years or groups needs its SE from the same replicates: write it as one estimate. CPS basic monthly and SIPP from
62
+ the API have no replicate weights: say no standard error was computed. This server has no PUMS recode helpers.
63
+
64
+ ## Spec examples
65
+ ```json
66
+ {"description": "Share of people without health coverage in December, SIPP 2024 (weighted by WPFINWGT)",
67
+ "requests": [{"name": "sipp_coverage_dec", "endpoint": "query",
68
+ "params": {"dataset": "2024/sipp", "get": "TAGE,RHLTHMTH,TPTOTINC", "MONTHCODE": "12"}}]}
69
+ ```
70
+ ```json
71
+ {"description": "Uninsured rate, Maryland, ACS 2024 1-year PUMS with replicate weights; CPS September 2026",
72
+ "requests": [{"name": "md_pums", "endpoint": "query",
73
+ "params": {"dataset": "2024/acs/acs1/pums", "get": "AGEP,HICOV,RELSHIPP,ESR,PINCP", "state": "MD",
74
+ "replicates": true}},
75
+ {"name": "cps_sep", "endpoint": "query",
76
+ "params": {"dataset": "2026/cps/basic/sep", "get": "PRTAGE,PEMLR,PRCITSHP", "PRTAGE": "25:54"}}]}
77
+ ```
@@ -0,0 +1,71 @@
1
+ # Census time series: economic indicators, trade, government finances, Household Pulse, HVS
2
+
3
+ From the census playbook's "Data format and traps" section: read it before a request of this kind.
4
+
5
+ - Time series (`timeseries/...`): give `time` (`2026`, `2026-08`, `2026-Q2`, or `from 2024-01 to 2026-08`).
6
+ Economic indicators (eits) return one row per code combination: `category_code` (industry or tax type),
7
+ `data_type_code` (level, percent change, ...), `seasonally_adj` (yes/no), `time_slot_id`, `error_data` (yes =
8
+ the row is a margin or standard error, not an estimate). The codes are not in the API's metadata: run one
9
+ narrow query and read the codes that come back, or the program's definitions page (census.gov/econ/currentdata),
10
+ then filter on them; never sum across data types or adjustment flags. Monthly indicators are revised (advance,
11
+ preliminary, revised): say which release. Units: retail and construction $ millions, housing units in
12
+ thousands, QTAX `CELL_VALUE` $ millions (variables.json labels say so); government finances `AMOUNT` $ thousands
13
+ with a coefficient of variation `AMOUNT_CV` (percent, sample-based local figures). Government finance years are
14
+ each government's fiscal year, not the calendar year. International trade: values in dollars; `GEN_VAL_MO`
15
+ general imports vs `CON_VAL_MO` imports for consumption; `CAL_DUT_MO` is the duty calculated on imports for
16
+ consumption (an estimate of duty owed, not Treasury's customs receipts); `COMM_LVL` HS2/HS4/HS6/HS10 must be
17
+ fixed or totals double-count; `CTY_CODE` country codes (5700 China, 2010 Mexico, 1220 Canada); it takes no
18
+ for= geography; `SUMMARY_LVL` CGP rows are country groups, not countries (drop them before summing).
19
+ - Household Pulse (`pulse`, `pulse_covid`) is an experimental data product: an online survey of adults 18+ in
20
+ housing units, contacted by e-mail and text, with weighted response rates of 4.7 to 7.3 percent at the end
21
+ (7.3 percent in week 63, October 2023; 4.7 percent in Phase 4.2 cycle 09, September 2024; each period's Source
22
+ and Accuracy statement, census.gov/programs-surveys/household-pulse-survey/technical-documentation.html). Say so
23
+ with every figure. `pulse` (timeseries/hhpulse) is long: one row per `MEASURE_NAME` (13 measures: EXPENSE,
24
+ FOODSCARCE, ENERGYBILL, EVICTFOR, PRICEINCR, PRICEWORRY, LONELY, ...), `CYCLE` and geography; filter
25
+ `MEASURE_NAME` and `CYCLE`. It holds `time=2024` only (Phase 4.0 cycles 01-03, 4.1 04-07, 4.2 08-09, January 9 to
26
+ September 16, 2024); `time=2025` returns no rows. `pulse_covid` (timeseries/hps) is wide: `<M>_RATE`,
27
+ `<M>_RT_MOE`, `<M>_TOTAL`, `<M>_MOE`, `<M>_UNIV` per collection `WEEK` (01-63, April 2020 to October 2023; from
28
+ Phase 2 a "week" is a two-week collection period). Questions came and went, so a measure is empty in weeks
29
+ that did not ask it (EXPENSE is empty in weeks 01-12, first asked in week 13, August 2020). The survey ended on 2024-09-16 and continued from 2025 inside HTOPS,
30
+ whose tables are not on the API.
31
+ `RATE` is the percent of `UNIVERSE` (`TOTAL` / `UNIVERSE`; EXPENSE, 2024 cycle 09: 87,697,083 / 236,337,862 =
32
+ 37.1), not of `UNITS_TOTAL` (all adults). Report `RATE_MOE` (`<M>_RT_MOE`) with every rate; Census uses
33
+ 90 percent confidence intervals for Pulse, but the API does not state the level of its margins. State margins
34
+ are several times the national one (cycle 09 EXPENSE: Maryland 34.3 +/- 3.4, nation 37.1 +/- 0.7): test a
35
+ difference against both margins before calling it one. Do not splice `pulse_covid` and `pulse` into one series
36
+ or compare across phases without saying the question set and contact frame changed (Phase 4.2 added prepaid
37
+ wireless numbers, "potentially impacting comparability with previous cycles", Phase 4.2 User Notes).
38
+ - HVS (`eits_homeownership`, timeseries/eits/hv) is quarterly (`time=2025-Q3`), from the CPS: `category_code`
39
+ RATE has `data_type_code` HOR (homeownership rate, not seasonally adjusted), SAHOR (seasonally adjusted,
40
+ `seasonally_adj` yes), RVR (rental vacancy rate), HVR (homeowner vacancy rate), and E_HOR, E_RVR, E_HVR
41
+ (`error_data` yes: the 90 percent margin of error in percentage points, e.g. 0.5 for HOR); category ESTIMATE
42
+ holds the housing inventory in thousands (TOTAL, OCC, OWNOCC, RNTOCC, VACANT, RENT, SALE, SEASON, ...). Never
43
+ mix HOR and SAHOR in one series. The API has the nation only; the release's Table 4 is the benchmark (2025
44
+ HOR 65.1, 65.0, 65.3, 65.7 matched the API on 2026-11-03). Fourth quarter 2025 rests on November and December
45
+ only: October 2025 was not collected during the lapse in federal funding (release footnote); say so when it
46
+ is used.
47
+
48
+ ## Spec examples
49
+ ```json
50
+ {"description": "Retail sales, 2026 to date; customs duty on imports from China, August 2026, by HS chapter",
51
+ "requests": [{"name": "retail_2026", "endpoint": "query",
52
+ "params": {"dataset": "eits_retail", "get": "cell_value,data_type_code,category_code,seasonally_adj,time_slot_id,error_data",
53
+ "for": "us:*", "time": "2026"}},
54
+ {"name": "china_duty_2026_08", "endpoint": "query",
55
+ "params": {"dataset": "trade_imports_hs", "get": "CTY_NAME,I_COMMODITY,GEN_VAL_MO,CON_VAL_MO,DUT_VAL_MO,CAL_DUT_MO",
56
+ "CTY_CODE": "5700", "COMM_LVL": "HS2", "time": "2026-08"}}]}
57
+ ```
58
+ ```json
59
+ {"description": "State and local government finances by state, 2024",
60
+ "requests": [{"name": "govfin_2024", "endpoint": "query",
61
+ "params": {"dataset": "govs_finances", "get": "AGG_DESC,GOVTYPE,AMOUNT,AMOUNT_CV", "for": "state:*", "time": "2024"}}]}
62
+ ```
63
+ ```json
64
+ {"description": "Adults with difficulty paying usual household expenses, nation, every 2024 Pulse cycle; homeownership rate 2025",
65
+ "requests": [{"name": "pulse_expense_2024", "endpoint": "query",
66
+ "params": {"dataset": "pulse", "get": "MEASURE_NAME,MEASURE_LABEL,RATE,RATE_MOE,TOTAL,UNIVERSE,CYCLE,PHASE,COL_START_DATE,COL_END_DATE",
67
+ "for": "us:*", "time": "2024", "MEASURE_NAME": "EXPENSE"}},
68
+ {"name": "homeownership_2025", "endpoint": "query",
69
+ "params": {"dataset": "eits_homeownership", "get": "cell_value,data_type_code,category_code,seasonally_adj,time_slot_id,geo_level_code,error_data",
70
+ "for": "us:*", "time": "2025", "category_code": "RATE"}}]}
71
+ ```
@@ -53,7 +53,7 @@ returns the national cross-section for one date, to filter by state FIPS),
53
53
  `pep_population_2021` 2021/pep/population (POP_2020, POP_2021; states and above only),
54
54
  `pep_population_2019` (2010-2019, down to places), `cbp` 2023/cbp,
55
55
  `ecnbasic_2022`, `sipp` latest/sipp (SIPP microdata; 2022-2025 on the API, one per calendar year).
56
- Microdata (one row per person; see the PUMS and CPS trap below): `acs1_pums` latest/acs/acs1/pums (2004-2024),
56
+ Microdata (one row per person; the traps are in doc "microdata"): `acs1_pums` latest/acs/acs1/pums (2004-2024),
57
57
  `acs5_pums` latest/acs/acs5/pums (2009-2024), `acs1_pumspr` (Puerto Rico), `cps_asec` latest/cps/asec/mar
58
58
  (1992-2026, survey year: income is for the year before), CPS basic monthly `<YYYY>/cps/basic/<jan..dec>`
59
59
  (1989-2026).
@@ -95,46 +95,6 @@ Find others: `request_schema` search `"median household income"` (latest ACS 5-y
95
95
  "params": {"dataset": "dec_pl_2020", "get": "NAME,P1_001N", "for": "state:*"}}]}
96
96
  ```
97
97
 
98
- ```json
99
- {"description": "Share of people without health coverage in December, SIPP 2024 (weighted by WPFINWGT)",
100
- "requests": [{"name": "sipp_coverage_dec", "endpoint": "query",
101
- "params": {"dataset": "2024/sipp", "get": "TAGE,RHLTHMTH,TPTOTINC", "MONTHCODE": "12"}}]}
102
- ```
103
-
104
- ```json
105
- {"description": "Uninsured rate, Maryland, ACS 2024 1-year PUMS with replicate weights; CPS September 2026",
106
- "requests": [{"name": "md_pums", "endpoint": "query",
107
- "params": {"dataset": "2024/acs/acs1/pums", "get": "AGEP,HICOV,RELSHIPP,ESR,PINCP", "state": "MD",
108
- "replicates": true}},
109
- {"name": "cps_sep", "endpoint": "query",
110
- "params": {"dataset": "2026/cps/basic/sep", "get": "PRTAGE,PEMLR,PRCITSHP", "PRTAGE": "25:54"}}]}
111
- ```
112
-
113
- ```json
114
- {"description": "Retail sales, 2026 to date; customs duty on imports from China, August 2026, by HS chapter",
115
- "requests": [{"name": "retail_2026", "endpoint": "query",
116
- "params": {"dataset": "eits_retail", "get": "cell_value,data_type_code,category_code,seasonally_adj,time_slot_id,error_data",
117
- "for": "us:*", "time": "2026"}},
118
- {"name": "china_duty_2026_08", "endpoint": "query",
119
- "params": {"dataset": "trade_imports_hs", "get": "CTY_NAME,I_COMMODITY,GEN_VAL_MO,CON_VAL_MO,DUT_VAL_MO,CAL_DUT_MO",
120
- "CTY_CODE": "5700", "COMM_LVL": "HS2", "time": "2026-08"}}]}
121
- ```
122
- ```json
123
- {"description": "State and local government finances by state, 2024",
124
- "requests": [{"name": "govfin_2024", "endpoint": "query",
125
- "params": {"dataset": "govs_finances", "get": "AGG_DESC,GOVTYPE,AMOUNT,AMOUNT_CV", "for": "state:*", "time": "2024"}}]}
126
- ```
127
-
128
- ```json
129
- {"description": "Adults with difficulty paying usual household expenses, nation, every 2024 Pulse cycle; homeownership rate 2025",
130
- "requests": [{"name": "pulse_expense_2024", "endpoint": "query",
131
- "params": {"dataset": "pulse", "get": "MEASURE_NAME,MEASURE_LABEL,RATE,RATE_MOE,TOTAL,UNIVERSE,CYCLE,PHASE,COL_START_DATE,COL_END_DATE",
132
- "for": "us:*", "time": "2024", "MEASURE_NAME": "EXPENSE"}},
133
- {"name": "homeownership_2025", "endpoint": "query",
134
- "params": {"dataset": "eits_homeownership", "get": "cell_value,data_type_code,category_code,seasonally_adj,time_slot_id,geo_level_code,error_data",
135
- "for": "us:*", "time": "2025", "category_code": "RATE"}}]}
136
- ```
137
-
138
98
  ## Data format and traps
139
99
  - Margins of error: every ACS `...E` estimate is fetched with its `...M` margin of error (`...PE` with
140
100
  `...PM`), each in its own column, at the 90% confidence level. They are never dropped (`moe: false`
@@ -149,7 +109,8 @@ Find others: `request_schema` search `"median household income"` (latest ACS 5-y
149
109
  interval are not exact; the group() call returns the EA annotation (median+, median-).
150
110
  - Values arrive as text, exactly as published; NAME reads "Montgomery County, Maryland". Geography
151
111
  code columns (state, county, place, tract) are strings with leading zeros: keep them as text.
152
- - 1-year vs 5-year: 1-year (normally released each September for the prior year; the 2025 release was not on the API on 2026-10-20) covers only areas of 65,000+
112
+ - 1-year vs 5-year: 1-year (normally released each September for the prior year; the 2025 release was not on
113
+ the API on 2026-10-20) covers only areas of 65,000+
153
114
  and has no tracts, block groups or ZCTAs; 5-year (normally each December) covers all areas but pools
154
115
  five years. Never compare a 1-year with a 5-year figure, and do not compare overlapping 5-year
155
116
  periods (2019-2023 vs 2020-2024). No standard 2020 ACS 1-year release exists (experimental only).
@@ -170,106 +131,13 @@ Find others: `request_schema` search `"median household income"` (latest ACS 5-y
170
131
  the last API vintage with place and county subdivision estimates. Vintages 2022 and 2024-2025 (county
171
132
  and city/town totals for 2022-2025) are not on the API: use the Census estimates files at
172
133
  https://www.census.gov/programs-surveys/popest/data/tables.html (manual download) and name the vintage.
173
- - SIPP (`sipp`, `20XX/sipp`): public-use microdata, one row per person per reference month (MONTHCODE
174
- 1-12). The reference year is the calendar year before the file's label: `2024/sipp` was collected in
175
- 2024 about January-December 2023; say "2023" for its figures. Filter MONTHCODE (12 = December) for a
176
- point in time, or average the months for the year. The pack always adds SSUID, PNUM (person = SSUID + PNUM), MONTHCODE and
177
- WPFINWGT, the person weight: every estimate is weighted by WPFINWGT, never a row count. No geography but
178
- `state`, which filters TST_INTV (state of the interview address); SIPP is designed for national estimates.
179
- -9 means NA or missing in most variables: drop it before summing. Standard errors need the replicate
180
- weights, which are not on the API (bulk file `rw<YYYY>_csv.zip`, REPWGT1-REPWGT240, at
181
- https://www2.census.gov/programs-surveys/sipp/data/datasets/<YYYY>/; method in the year's user guide); without them
182
- give the point estimate and say no standard error was computed. Variable names and codes:
183
- `request_schema` search `"<YYYY>/sipp: health insurance"` or the dataset's variables.json. The bulk primary file
184
- (`pu<YYYY>_csv.zip`, about 100 MB) has every variable, for a manual download.
185
- - ACS PUMS and CPS microdata (`acs1_pums`, `acs5_pums`, `cps_asec`, `<YYYY>/cps/basic/<mon>`): one row per
186
- person, values and codes exactly as published, with `<request>_codebook.csv` (every code's label, the
187
- variable's suggested weight). The pack adds the keys and weights: ACS SERIALNO, SPORDER, PWGTP (persons), WGTP
188
- (households: SPORDER 1 with RELSHIPP 20); ASEC H_SEQ, A_LINENO, PERIDNUM, MARSUPWT, HSUP_WGT; basic HRHHID,
189
- HRHHID2, PULINENO, PWSSWGT (final weight), PWCMPWGT (composite: employment levels that match BLS). For hourly
190
- and weekly earnings in a basic month, request PWORWGT (outgoing rotation groups, HRMIS 4 and 8). Every estimate
191
- is weighted, never a row count. Filters are upper-case variable predicates (`"AGEP": "25:54"`, `"HICOV": 2`),
192
- repeated on every call; geography: `state`, or `for: "public use microdata area:*"` with a state (ACS);
193
- no state = the nation (ACS fetched state by state, 51 calls, about 2 minutes for the 3.4 million persons).
194
- Every result has a `state` column (the FIPS code): group on it. The ACS PUMS state variable is `ST` through
195
- the 2022 file and `STATE` from 2023 on (1-year, 5-year and Puerto Rico; the other name is not a variable).
196
- `replicates: true` adds PWGTP1-80 for ACS (`"household"`: WGTP1-80, `"both"`); columns past 50 come in
197
- further calls joined on the keys. For the ASEC (2005 on) it adds PH_SEQ and PPPOS and joins PWWGT1-160 from
198
- Census's replicate weight file (CPS_ASEC_ASCII_REPWGT_<year> on www2.census.gov, about 70 MB, no key), after
199
- two gates: the file's weight sums equal the totals its SAS layout states, and every person's PWWGT0 equals
200
- MARSUPWT (2020: the 2020-based file is tried when the main one does not match). Census's replicate standard error
201
- is sqrt(4/80 * sum of squared replicate differences), MOE = 1.645 * SE (`compute`'s `replicate_se` step with
202
- `method: "sdr"` gives both; see Analysis below). Checked 2026-11-04 with that method: Maryland's 2024 uninsured rate in the civilian
203
- noninstitutionalized population from PUMS was 6.26 +/- 0.32 percent, published S2701 6.3 +/- 0.3; the nation
204
- 8.19 vs 8.2.
205
- - Universe: published insurance and labor tables use the civilian noninstitutionalized population
206
- (drop RELSHIPP 37, the institutionalized group quarters population, and ESR 4-5, armed forces). Codes below zero or 0 often mean N/A (PINCP -19999 under 15, ESR 0 under 16): read the codebook.
207
- - Dollars: ACS amounts need ADJINC (ADJHSG for housing costs), which the pack adds when a dollar variable asks
208
- for it: value * ADJINC / 1,000,000 is in the survey year's dollars, as the published tables are. PUMS is a
209
- subsample of the full ACS: a PUMS figure differs slightly from the published table (inside its MOE);
210
- publish your figure as your own calculation and benchmark it against the table.
211
- - CPS: ASEC income, poverty (PERLIS) and full-year insurance (COV) refer to the previous calendar year; NOW_COV
212
- is coverage at the interview. Census's method for standard errors: weight MARSUPWT with replicates
213
- PWWGT1-160, SE = sqrt(4/160 * sum of squared differences); household estimates on householders only, with
214
- HSUP_WGT and the same replicates. Checked 2026-11-04 with that method: 2025 official poverty from the 2026 ASEC was
215
- 34,505 thousand +/- 909 and 10.2 +/- 0.3 percent (PERLIS 1 among PERLIS != -1), P60-290 Table 1 34,510 +/- 909
216
- and 10.2 +/- 0.3. Before 2005 there is no replicate file: use the ipums pack.
217
- The API's 2020 and 2021 ASEC weights are the 2010-Census-based ones (their PWWGT0 matched the main replicate
218
- file, checked 2026-11-04); Census advises its 2020-based weights to compare those years with 2022 on, and
219
- they are not on the API: say which weights a 2020-2021 comparison uses.
220
- ASEC files before 2001 take no state geography on the API (HTTP 500): fetch the nation and filter on the
221
- file's state variable, or use ipums. Older files lack some keys and weights (HRHHID2 before 2004, PERIDNUM
222
- in early ASECs): the pack leaves them out and says so. CPS basic monthly has no replicate weights at all; month-to-month person links need HRHHID, HRHHID2 and
223
- PULINENO with care (ipums CPSIDP links them). CPS codes differ from ACS codes (PEMLR, PRCITSHP, A_HGA /
224
- PEEDUCA): recode each survey from its own codebook.
225
- - Analysis: build tables with `compute`. Standard errors: fetch with `replicates`, then a `replicate_se` step,
226
- e.g. `{"groupby": ["state"], "replicate_se": {"estimates": {"uninsured": "sum(PWGTP * (HICOV == 2)) / sum(PWGTP)"},
227
- "weight": "PWGTP", "replicates": "PWGTP1:PWGTP80", "method": "sdr"}}` (ASEC: `MARSUPWT`, `"PWWGT1:PWWGT160"`;
228
- household estimates: `WGTP`/`HSUP_WGT` with their replicates on one row per household). A difference between
229
- years or groups needs its SE from the same replicates: write it as one estimate. CPS basic monthly and SIPP from
230
- the API have no replicate weights: say no standard error was computed. This server has no PUMS recode helpers.
231
- - Time series (`timeseries/...`): give `time` (`2026`, `2026-08`, `2026-Q2`, or `from 2024-01 to 2026-08`).
232
- Economic indicators (eits) return one row per code combination: `category_code` (industry or tax type),
233
- `data_type_code` (level, percent change, ...), `seasonally_adj` (yes/no), `time_slot_id`, `error_data` (yes =
234
- the row is a margin or standard error, not an estimate). The codes are not in the API's metadata: run one
235
- narrow query and read the codes that come back, or the program's definitions page (census.gov/econ/currentdata),
236
- then filter on them; never sum across data types or adjustment flags. Monthly indicators are revised (advance,
237
- preliminary, revised): say which release. Units: retail and construction $ millions, housing units in
238
- thousands, QTAX `CELL_VALUE` $ millions (variables.json labels say so); government finances `AMOUNT` $ thousands
239
- with a coefficient of variation `AMOUNT_CV` (percent, sample-based local figures). Government finance years are
240
- each government's fiscal year, not the calendar year. International trade: values in dollars; `GEN_VAL_MO`
241
- general imports vs `CON_VAL_MO` imports for consumption; `CAL_DUT_MO` is the duty calculated on imports for
242
- consumption (an estimate of duty owed, not Treasury's customs receipts); `COMM_LVL` HS2/HS4/HS6/HS10 must be
243
- fixed or totals double-count; `CTY_CODE` country codes (5700 China, 2010 Mexico, 1220 Canada); it takes no
244
- for= geography; `SUMMARY_LVL` CGP rows are country groups, not countries (drop them before summing).
245
- - Household Pulse (`pulse`, `pulse_covid`) is an experimental data product: an online survey of adults 18+ in
246
- housing units, contacted by e-mail and text, with weighted response rates of 4.7 to 7.3 percent at the end
247
- (7.3 percent in week 63, October 2023; 4.7 percent in Phase 4.2 cycle 09, September 2024; each period's Source
248
- and Accuracy statement, census.gov/programs-surveys/household-pulse-survey/technical-documentation.html). Say so
249
- with every figure. `pulse` (timeseries/hhpulse) is long: one row per `MEASURE_NAME` (13 measures: EXPENSE,
250
- FOODSCARCE, ENERGYBILL, EVICTFOR, PRICEINCR, PRICEWORRY, LONELY, ...), `CYCLE` and geography; filter
251
- `MEASURE_NAME` and `CYCLE`. It holds `time=2024` only (Phase 4.0 cycles 01-03, 4.1 04-07, 4.2 08-09, January 9 to
252
- September 16, 2024); `time=2025` returns no rows. `pulse_covid` (timeseries/hps) is wide: `<M>_RATE`,
253
- `<M>_RT_MOE`, `<M>_TOTAL`, `<M>_MOE`, `<M>_UNIV` per collection `WEEK` (01-63, April 2020 to October 2023; from
254
- Phase 2 a "week" is a two-week collection period). Questions came and went, so a measure is empty in weeks
255
- that did not ask it (EXPENSE is empty in weeks 01-12, first asked in week 13, August 2020). The survey ended on 2024-09-16 and continued from 2025 inside HTOPS,
256
- whose tables are not on the API.
257
- `RATE` is the percent of `UNIVERSE` (`TOTAL` / `UNIVERSE`; EXPENSE, 2024 cycle 09: 87,697,083 / 236,337,862 =
258
- 37.1), not of `UNITS_TOTAL` (all adults). Report `RATE_MOE` (`<M>_RT_MOE`) with every rate; Census uses
259
- 90 percent confidence intervals for Pulse, but the API does not state the level of its margins. State margins
260
- are several times the national one (cycle 09 EXPENSE: Maryland 34.3 +/- 3.4, nation 37.1 +/- 0.7): test a
261
- difference against both margins before calling it one. Do not splice `pulse_covid` and `pulse` into one series
262
- or compare across phases without saying the question set and contact frame changed (Phase 4.2 added prepaid
263
- wireless numbers, "potentially impacting comparability with previous cycles", Phase 4.2 User Notes).
264
- - HVS (`eits_homeownership`, timeseries/eits/hv) is quarterly (`time=2025-Q3`), from the CPS: `category_code`
265
- RATE has `data_type_code` HOR (homeownership rate, not seasonally adjusted), SAHOR (seasonally adjusted,
266
- `seasonally_adj` yes), RVR (rental vacancy rate), HVR (homeowner vacancy rate), and E_HOR, E_RVR, E_HVR
267
- (`error_data` yes: the 90 percent margin of error in percentage points, e.g. 0.5 for HOR); category ESTIMATE
268
- holds the housing inventory in thousands (TOTAL, OCC, OWNOCC, RNTOCC, VACANT, RENT, SALE, SEASON, ...). Never
269
- mix HOR and SAHOR in one series. The API has the nation only; the release's Table 4 is the benchmark (2025
270
- HOR 65.1, 65.0, 65.3, 65.7 matched the API on 2026-11-03). Fourth quarter 2025 rests on November and December
271
- only: October 2025 was not collected during the lapse in federal funding (release footnote); say so when it
272
- is used.
134
+ - ACS PUMS, CPS and SIPP microdata (`acs1_pums`, `acs5_pums`, `cps_asec`, `cps/basic`, `sipp`): read
135
+ `source_brief({source: "census", doc: "microdata"})` before the first fetch. It has the keys and weights the
136
+ pack adds, the `ST`/`STATE` rename, replicate weights and standard errors, universes, ADJINC, and the
137
+ checked benchmarks.
138
+ - Time series (`timeseries/...`: economic indicators, international trade, government finances, Household
139
+ Pulse, HVS): read `source_brief({source: "census", doc: "timeseries"})` before the first fetch. It has the
140
+ `time` forms, the code columns to fix, units, revisions and each survey's caveats.
273
141
  - Key: since 2026-05-12 every data query needs a key; the API redirects keyless queries to a "Missing
274
142
  Key" page, which the pack reports as a key error. Metadata (variables, discovery) still answer
275
143
  without a key.
@@ -0,0 +1,26 @@
1
+ # Congress nominations, CRS reports and cost estimates
2
+
3
+ From the congress playbook's "Data format and traps" section: read it before a request of this kind.
4
+
5
+ - Cost estimates: `bill_cbo_estimates` gives only what Congress.gov lists: title, publication date,
6
+ description and a cbo.gov link for each estimate. No dollar figures are in the API; the estimate
7
+ itself is a PDF or web page (committee reports in govinfo CRPT often reprint it as text).
8
+ - CRS: `crs_reports` lists reports Congress.gov publishes (newest first, about 14,000 in Oct 2026);
9
+ there is no keyword search: use a date range and `title_contains`, then `crs_report` for the summary.
10
+ `fromDateTime`/`toDateTime` filter on the UPDATE date (one September window returned 1,244 products
11
+ published 2004-2026): for "published in a month" filter the rows on `publishDate` afterwards. `publishDate`
12
+ is the current version's date, so a product first issued in the month and revised later drops out (a
13
+ floor, say so). The list's `version` is not the product version: `crs_report` gives `currentVersion`.
14
+ `bill_summaries` are CRS summaries of each bill version (HTML in `text`, plain in `text_plain`); some fields
15
+ are over 131,072 characters: when reading the CSV with your own code, raise the reader's field size limit first.
16
+ - Nominations (traps seen in Oct 2026 data; the Senate Résumé's "Civilian nominees" table is the benchmark):
17
+ military and "Other Civilian" lists (Foreign Service, PHS, NOAA officers) are list nominations without a
18
+ description; `nominationType` (`isCivilian`, `isMilitary`) is missing on some rows of older Congresses
19
+ (every row of the 119th has one); a PN split into parts
20
+ (PN12-41) counts once per part; en bloc votes confirm many nominees at once. `latestAction` is the last
21
+ step, which can be a later procedural motion ("motion to reconsider tabled") or, for a nomination held over,
22
+ its confirmation in the next session: count outcomes by date from `nomination_actions`, within the
23
+ session's dates. A split PN's actions need `nominationNumber` "1255-17": `"for_each": {"csv":
24
+ "data/congress/<noms>.csv", "column": "citation", "param": "nominationNumber", "strip": "PN"}` (`number` alone
25
+ returns nothing for a split PN). The naive count (civilian flag + `latestAction` "Confirmed") gave 60 percent for 2023
26
+ against the Résumé's 47.
@@ -0,0 +1,30 @@
1
+ # Congress votes: roll calls, party totals and member ids
2
+
3
+ From the congress playbook's "Data format and traps" section: read it before a request of this kind.
4
+
5
+ - Votes: a bill usually has several recorded votes (cloture, motions to proceed, amendments, passage,
6
+ conference). `bill_votes` lists every one with the action text. Final passage is each chamber's LAST
7
+ recorded vote that cleared the bill ("On Passage", "On Passage of the Bill", "On Motion to Concur ...",
8
+ "On Agreeing to the Conference Report"): take the latest date, not the first "On Passage". A bill used
9
+ as a vehicle for other text (CHIPS and Science moved as H.R. 4346, first passed as a legislative-branch
10
+ appropriation) has an early "On Passage" vote on the old text; the Clerk's `description` can keep the
11
+ old title. Voice votes and unanimous consent leave no roll call and no row.
12
+ - Party totals: House rows take the Clerk's totals by party; Senate rows are counted from the member
13
+ list (`yea_D`, `nay_R`, ...; independents as I). `not_voting` counts absent members; vacant seats are
14
+ not counted anywhere, so totals can be under 435 or 100. The Vice President's tie-breaking vote is in
15
+ `tie_breaker`, not in the member rows.
16
+ - Member ids differ by source: House rows and Congress.gov use bioguide ids (A000370); Senate XML uses
17
+ LIS ids (S354). Join Senate votes to members by name and state, or a crosswalk, not by id.
18
+ - `vote` keeps the published word (Yea, Aye, Nay, No, Present, Not Voting, Guilty); `position` maps it
19
+ to yea, nay, present, not_voting or other. House "Aye"/"No" are recorded votes, same meaning.
20
+ - Whole-session vote counts and party splits: no endpoint returns party totals in bulk, so a session takes
21
+ one `roll_call` per vote (about 1,000 a year, several minutes). For the House's questions and results
22
+ (`house_votes` has no question column; `house_vote` keeps only party totals), `for_each` over the
23
+ `house_votes` list's `rollCallNumber` with `roll_call` `"summary": "true"` and the year: one row per vote with
24
+ question ("On Motion to Suspend the Rules and Pass"), vote type ("2/3 YEA-AND-NAY"), result and totals. `house_votes` lists quorum calls too (vote
25
+ type "Quorum"): the Résumé counts them apart from yea-and-nay and recorded votes. The Speaker election's
26
+ member rows have position `other` (votes are names). Say how independents are counted in a party split.
27
+ - `house_votes`/`house_vote` (Congress.gov) start with the 115th Congress (2017; the 114th lists 0 votes,
28
+ verified 2026-10-22); `roll_call` reads the Clerk
29
+ XML for 1990 on (and falls back to the Congress.gov member list for the 115th on if the Clerk file
30
+ cannot be read). Senate XML covers the 101st Congress (1989) on.