data-analyst-mcp 0.7.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (83) hide show
  1. package/README.md +2 -2
  2. package/dist/adapters/rma.d.ts +10 -0
  3. package/dist/adapters/rma.js +40 -0
  4. package/dist/adapters/rma.js.map +1 -0
  5. package/dist/analysis/combine.js +3 -2
  6. package/dist/analysis/combine.js.map +1 -1
  7. package/dist/analysis/compute.d.ts +26 -5
  8. package/dist/analysis/compute.js +402 -159
  9. package/dist/analysis/compute.js.map +1 -1
  10. package/dist/analysis/expr.js +2 -1
  11. package/dist/analysis/expr.js.map +1 -1
  12. package/dist/analysis/handoff.d.ts +1 -1
  13. package/dist/analysis/handoff.js +44 -3
  14. package/dist/analysis/handoff.js.map +1 -1
  15. package/dist/analysis/table.d.ts +7 -0
  16. package/dist/analysis/table.js +30 -1
  17. package/dist/analysis/table.js.map +1 -1
  18. package/dist/analysis/tools.d.ts +10 -7
  19. package/dist/analysis/tools.js +32 -43
  20. package/dist/analysis/tools.js.map +1 -1
  21. package/dist/core/compact.d.ts +15 -0
  22. package/dist/core/compact.js +98 -0
  23. package/dist/core/compact.js.map +1 -0
  24. package/dist/core/generic-rest.js +2 -1
  25. package/dist/core/generic-rest.js.map +1 -1
  26. package/dist/core/package.d.ts +8 -0
  27. package/dist/core/package.js +30 -1
  28. package/dist/core/package.js.map +1 -1
  29. package/dist/core/packs.d.ts +4 -2
  30. package/dist/core/sections.d.ts +21 -0
  31. package/dist/core/sections.js +53 -0
  32. package/dist/core/sections.js.map +1 -0
  33. package/dist/core/tabular.d.ts +2 -0
  34. package/dist/core/tabular.js +14 -3
  35. package/dist/core/tabular.js.map +1 -1
  36. package/dist/deep/opm.js +10 -2
  37. package/dist/deep/opm.js.map +1 -1
  38. package/dist/deep/treasury.js +7 -3
  39. package/dist/deep/treasury.js.map +1 -1
  40. package/dist/envelope.js +1 -1
  41. package/dist/envelope.js.map +1 -1
  42. package/dist/index.d.ts +57 -4
  43. package/dist/report/audit_report.js +3 -0
  44. package/dist/report/audit_report.js.map +1 -1
  45. package/dist/report/build_report.d.ts +18 -0
  46. package/dist/report/build_report.js +52 -7
  47. package/dist/report/build_report.js.map +1 -1
  48. package/dist/report/tools.d.ts +26 -1
  49. package/dist/report/tools.js +74 -17
  50. package/dist/report/tools.js.map +1 -1
  51. package/dist/server.d.ts +1 -1
  52. package/dist/server.js +52 -43
  53. package/dist/server.js.map +1 -1
  54. package/dist/session.d.ts +2 -0
  55. package/dist/session.js +2 -0
  56. package/dist/session.js.map +1 -1
  57. package/dist/tools.d.ts +98 -6
  58. package/dist/tools.js +154 -17
  59. package/dist/tools.js.map +1 -1
  60. package/guide/WORKFLOW.md +7 -4
  61. package/guide/references/compute.md +51 -0
  62. package/guide/references/report_quick.md +60 -0
  63. package/guide/references/routing.md +2 -0
  64. package/package.json +1 -1
  65. package/packs/cbp/source.md +1 -1
  66. package/packs/cdc-wonder/source.md +1 -1
  67. package/packs/census/docs/microdata.md +77 -0
  68. package/packs/census/docs/timeseries.md +71 -0
  69. package/packs/census/source.md +10 -142
  70. package/packs/congress/docs/nominations_crs.md +26 -0
  71. package/packs/congress/docs/votes.md +30 -0
  72. package/packs/congress/source.md +4 -48
  73. package/packs/govinfo/docs/collections.md +88 -0
  74. package/packs/govinfo/source.md +6 -86
  75. package/packs/irs-soi/docs/eo.md +15 -0
  76. package/packs/irs-soi/docs/tables.md +50 -0
  77. package/packs/irs-soi/source.md +6 -57
  78. package/packs/port.json +5 -0
  79. package/packs/rma/endpoints.json +44 -0
  80. package/packs/rma/policy.json +94 -0
  81. package/packs/rma/source.md +78 -0
  82. package/packs/sba/source.md +1 -1
  83. package/packs/uscourts/source.md +1 -1
@@ -93,47 +93,14 @@ confirm each bill with `bill_titles` (`for_each` on congress, billType, billNumb
93
93
  (verified 2026-10-22): those need one `bill` call per law, a second spec with
94
94
  `"for_each": {"csv": "data/congress/<laws>.csv", "map": {"billType": "type", "billNumber": "number"},
95
95
  "lower": true}` (bill numbers repeat across types; one call per row of the laws CSV). Counting "bills passed" needs a chamber action, not `latestAction` alone.
96
- - Votes: a bill usually has several recorded votes (cloture, motions to proceed, amendments, passage,
97
- conference). `bill_votes` lists every one with the action text. Final passage is each chamber's LAST
98
- recorded vote that cleared the bill ("On Passage", "On Passage of the Bill", "On Motion to Concur ...",
99
- "On Agreeing to the Conference Report"): take the latest date, not the first "On Passage". A bill used
100
- as a vehicle for other text (CHIPS and Science moved as H.R. 4346, first passed as a legislative-branch
101
- appropriation) has an early "On Passage" vote on the old text; the Clerk's `description` can keep the
102
- old title. Voice votes and unanimous consent leave no roll call and no row.
96
+ - Votes (roll calls, party totals, member ids, whole-session counts, `house_votes`): read
97
+ `source_brief({source: "congress", doc: "votes"})` before counting votes.
103
98
  - Zero rows can be the right answer (no private laws in the 118th Congress, no votes on a bill): mark
104
99
  such requests `"allow_empty": true`, or the fetch stops and names them.
105
- - Party totals: House rows take the Clerk's totals by party; Senate rows are counted from the member
106
- list (`yea_D`, `nay_R`, ...; independents as I). `not_voting` counts absent members; vacant seats are
107
- not counted anywhere, so totals can be under 435 or 100. The Vice President's tie-breaking vote is in
108
- `tie_breaker`, not in the member rows.
109
- - Member ids differ by source: House rows and Congress.gov use bioguide ids (A000370); Senate XML uses
110
- LIS ids (S354). Join Senate votes to members by name and state, or a crosswalk, not by id.
111
- - `vote` keeps the published word (Yea, Aye, Nay, No, Present, Not Voting, Guilty); `position` maps it
112
- to yea, nay, present, not_voting or other. House "Aye"/"No" are recorded votes, same meaning.
113
100
  - Cosponsors: `sponsorshipWithdrawnDate` marks withdrawals (still listed);
114
101
  `cosponsors.countIncludingWithdrawnCosponsors` in `bill` counts them. Count by `party` for a party split.
115
- - Cost estimates: `bill_cbo_estimates` gives only what Congress.gov lists: title, publication date,
116
- description and a cbo.gov link for each estimate. No dollar figures are in the API; the estimate
117
- itself is a PDF or web page (committee reports in govinfo CRPT often reprint it as text).
118
- - CRS: `crs_reports` lists reports Congress.gov publishes (newest first, about 14,000 in Oct 2026);
119
- there is no keyword search: use a date range and `title_contains`, then `crs_report` for the summary.
120
- `fromDateTime`/`toDateTime` filter on the UPDATE date (one September window returned 1,244 products
121
- published 2004-2026): for "published in a month" filter the rows on `publishDate` afterwards. `publishDate`
122
- is the current version's date, so a product first issued in the month and revised later drops out (a
123
- floor, say so). The list's `version` is not the product version: `crs_report` gives `currentVersion`.
124
- `bill_summaries` are CRS summaries of each bill version (HTML in `text`, plain in `text_plain`); some fields
125
- are over 131,072 characters: when reading the CSV with your own code, raise the reader's field size limit first.
126
- - Nominations (traps seen in Oct 2026 data; the Senate Résumé's "Civilian nominees" table is the benchmark):
127
- military and "Other Civilian" lists (Foreign Service, PHS, NOAA officers) are list nominations without a
128
- description; `nominationType` (`isCivilian`, `isMilitary`) is missing on some rows of older Congresses
129
- (every row of the 119th has one); a PN split into parts
130
- (PN12-41) counts once per part; en bloc votes confirm many nominees at once. `latestAction` is the last
131
- step, which can be a later procedural motion ("motion to reconsider tabled") or, for a nomination held over,
132
- its confirmation in the next session: count outcomes by date from `nomination_actions`, within the
133
- session's dates. A split PN's actions need `nominationNumber` "1255-17": `"for_each": {"csv":
134
- "data/congress/<noms>.csv", "column": "citation", "param": "nominationNumber", "strip": "PN"}` (`number` alone
135
- returns nothing for a split PN). The naive count (civilian flag + `latestAction` "Confirmed") gave 60 percent for 2023
136
- against the Résumé's 47.
102
+ - Nominations, CRS reports and cost estimates: read `source_brief({source: "congress", doc:
103
+ "nominations_crs"})` before using them.
137
104
  - Paging: 250 per request (`limit`/`offset`). Lists default to the newest update first.
138
105
  - Vetoes: a vetoed bill's `latestAction` often is a later procedural step, and no endpoint filters bills by
139
106
  action. Read `bill_actions` (for_each over candidate bills that passed both chambers and are not law) for
@@ -141,17 +108,6 @@ confirm each bill with `bill_titles` (`for_each` on congress, billType, billNumb
141
108
  President to the contrary notwithstanding"). The Senate's veto page (senate.gov/legislative/vetoes, regular
142
109
  and pocket vetoes per President) is the benchmark. Reserved bill numbers (H.R. 1-10, S. 1-10 for leaders)
143
110
  have no actions: allow_empty or drop them first.
144
- - Whole-session vote counts and party splits: no endpoint returns party totals in bulk, so a session takes
145
- one `roll_call` per vote (about 1,000 a year, several minutes). For the House's questions and results
146
- (`house_votes` has no question column; `house_vote` keeps only party totals), `for_each` over the
147
- `house_votes` list's `rollCallNumber` with `roll_call` `"summary": "true"` and the year: one row per vote with
148
- question ("On Motion to Suspend the Rules and Pass"), vote type ("2/3 YEA-AND-NAY"), result and totals. `house_votes` lists quorum calls too (vote
149
- type "Quorum"): the Résumé counts them apart from yea-and-nay and recorded votes. The Speaker election's
150
- member rows have position `other` (votes are names). Say how independents are counted in a party split.
151
- - `house_votes`/`house_vote` (Congress.gov) start with the 115th Congress (2017; the 114th lists 0 votes,
152
- verified 2026-10-22); `roll_call` reads the Clerk
153
- XML for 1990 on (and falls back to the Congress.gov member list for the 115th on if the Clerk file
154
- cannot be read). Senate XML covers the 101st Congress (1989) on.
155
111
  - Dates: Congress.gov timestamps are UTC (`2025-01-22T...Z`); vote `date` is the vote's own calendar
156
112
  date (ISO), `date_published` keeps the source text.
157
113
 
@@ -0,0 +1,88 @@
1
+ # GovInfo collections: how each one is packaged and counted
2
+
3
+ From the govinfo playbook's "Data format and traps" section: read it before a request of this kind.
4
+
5
+ - Cost estimates: the budget office's own estimates are not a GovInfo collection. Two partial routes,
6
+ both text, not tables: (1) `collection: GOVPUB` with the publisher filter
7
+ (`publisher: "Congressional Budget Office"`, a report subset) returns the reports GPO has catalogued
8
+ (765 records in Oct 2026, far from the office's full list of publications);
9
+ (2) `collection: CRPT` with `phrase: "cost estimate"` finds committee reports, which usually reprint the
10
+ estimate for a reported bill in the report text. Congress.gov bill records (congress pack, endpoint
11
+ `bill_cbo_estimates`) list the titles and links of the estimates for a bill. No API here returns the
12
+ estimated dollar figures as data.
13
+ - CREC (daily Congressional Record): one package per issue date, but a two-book day can appear twice
14
+ (`CREC-2025-03-11` and `CREC-2025-03-11-i45`, same pages; `-i46` the second book): deduplicate by
15
+ `volume` + `issue` before counting issues or summing pages. Other suffixes are distinct issues, not
16
+ duplicates: `-v169`/`-v170` on January 3 (two volumes, the outgoing and incoming Congress) and a day with
17
+ `-i64`/`-i65`. Chamber sections are granules of class SENATE, HOUSE, EXTENSIONS and DAILYDIGEST (the
18
+ summary's `download.pdfSenateLink` / `pdfHouseLink` too). Early-January issues of a calendar year
19
+ belong to the outgoing Congress (volume and congress differ; `CREC-2015-01-02` is volume 160 but is
20
+ labelled congress 114), and an issue can be missing. Proceedings print under the legislative day, not the calendar day: a
21
+ Senate session that runs on (July 1, 2025, a continuation of June 30's legislative day) has no issue of its
22
+ own, so `granules` for that date returns no rows; find the issue with `search` (CREC, a phrase from the
23
+ proceedings) instead. `pages` counts PDF pages (Daily Digest and front
24
+ matter included), so it runs about 1 percent above the Résumé of Congressional Activity's numbered pages.
25
+ - "How many pages is the law": the convention is its Statutes at Large pages (first page to the last
26
+ `[[Page ... STAT. ...]]` marker of `text`, e.g. 139 Stat. 72-401 is 330 pages); the slip-law and enrolled-bill
27
+ PDFs differ by a page or so (cover, signatures). Say which measure the answer uses.
28
+ - CPD (Compilation of Presidential Documents, 1993 on; daily from 2009-01-20): one package per document,
29
+ docClass PRESDOC plus a few DIGEST, ACTSAPPROVED, CHECKLIST and NOMINATIONS rows (in a transition year the
30
+ outgoing President's supplements, `DCPD-2025DIGEST-BIDEN` and the like, are docClass PRESDOC too: count
31
+ numbered DCPD-YYYY0NNNN packages). The `published` list has no document type; each `package` summary has
32
+ `dcpdCategory` (level1 "Executive Orders", "Proclamations" ...), and `search` takes `collection:CPD AND
33
+ cpdcategory:proclamation` (signing remarks included). Some orders are missing or have no number in the
34
+ title, and `dateIssued` can be a day after signing. Count executive orders from the
35
+ federal-register pack (`executive_orders`, by signing date) and use CPD as the cross-check.
36
+ - CRPT (committee reports): one package per numbered report (its parts and errata inside it); GovInfo's
37
+ browse pages count documents (parts and errata), so they run above the report count. `published` CRPT also
38
+ lists GPO duplicate packages (GPO-J6-REPORT, GPO-CRPT-117hrpt646): keep `packageId` CRPT-*. A few
39
+ `dateIssued` values are wrong (three Dec-2022 reports dated 2021-12-31): count by Congress, not by date.
40
+ - CPRT (committee prints, "select prints": GovInfo is not a census of them): summaries carry no
41
+ `committees`; attribute a print to its committee by `suDocClassNumber` (the `Y 4.` stem, without the
42
+ `-NN` series suffix; a few prints have two classes or none). `published` with collection CPRT also returns
43
+ packages of other collections (GPO-PLUMBOOK-2024, collectionCode GPO): keep `collectionCode` CPRT.
44
+ GovInfo's browse pages count documents (errata included), not packages.
45
+ - STATUTE: one package per volume and volume = a session of Congress, not a calendar year. A law goes in
46
+ the volume of the session that passed it, not the year it was signed: laws signed in early January after
47
+ a session ends are in the closing session's volume (116-344, signed 2021-01-13, in vol. 134; 118-274,
48
+ 2025-01-06, in vol. 138), and numbering can interleave (113-74, signed 2014-01-16, closes vol. 127 while
49
+ 113-73 opens vol. 128). Assign laws to volumes from the granules, never by date. Granule `dateIssued` can
50
+ be wrong by a year (117-343's granule says 2022-01-05; signed 2023-01-05). Count laws by granules
51
+ with `granuleClass` PUBLICLAW (granules also hold proclamations and concurrent resolutions). Older
52
+ summaries (vol. 127) have no `pages`; `pages` counts front matter too. A granule id carries its start page
53
+ (`STATUTE-136-Pg6299`; a continuation adds a suffix, `STATUTE-135-Pg2531-2`). Each volume prints public
54
+ laws, then private laws, then concurrent resolutions and proclamations, with a few blank pages between the
55
+ parts. The public laws' pages run from the first law's start page to the **last law's last page**: the last
56
+ `[[Page N STAT. M]]` marker in that law's `text` (`packageId` PLAW-...). The next granule's start page
57
+ overstates it by 3 or 4 pages (6 more in vol. 136, where private laws come first); with the end from the
58
+ text, the 116th-118th Congresses give 8,432, 8,742 and 4,350 pages, the figures in Brookings' Vital
59
+ Statistics on Congress (Table 6-4).
60
+ - BUDGET: one package per volume (`BUDGET-2027-BUD` main volume, `-PER` Analytical Perspectives, `-APP`
61
+ Appendix). Summaries report collectionCode GPO (FY2017 and FY2027 alike) although `published` with
62
+ collection BUDGET lists them. Recent summaries' `pages` equals the PDF's page count; older ones (FY2017)
63
+ carry none: download `https://www.govinfo.gov/content/pkg/<packageId>/pdf/<packageId>.pdf`, count its pages
64
+ (pypdf) and record them in `published/`. A missing volume (FY2026 had no Analytical
65
+ Perspectives) is a 404: allow_empty. OMB posts supplemental chapters outside the volumes.
66
+ - USCOURTS (federal court opinions): one package per case, one granule per opinion; `search` returns one row
67
+ per opinion (`collection:USCOURTS AND courttype:appellate`, `courtname:fifth`; `ingestdate:range(...)`
68
+ filters on the date added to GovInfo, not the opinion date, and 2025 opinions keep arriving in 2026). A year of appellate opinions (about 15,000) passes the
69
+ 10,000-row search cap: split by half-year. The court code is in the packageId (`USCOURTS-ca5-...`; the
70
+ Federal Circuit is ca13). Courts choose what they send (the Ninth Circuit sends precedential opinions only),
71
+ so a ranking reflects submission practice, not caseload: say so; the AO's Judicial Business tables (B-12)
72
+ count opinions by circuit.
73
+ - CFR: one package per title, edition and volume (`CFR-2025-title40-vol17`). `dateIssued` is the
74
+ "revised as of" date (January 1 for titles 1-16, April 1 for 17-27, July 1 for 28-41, October 1 for
75
+ 42-50), not the posting date, which lags by months: a recent
76
+ edition can be incomplete, with volumes missing (404) while `volumeCount` counts them. `published`
77
+ rows carry no page count, so a title's pages take one `package` call per volume (`pages`, every PDF page
78
+ including front matter); give missing volumes `"allow_empty": true` and say they are missing, or fill
79
+ them from the prior edition as a labeled sensitivity case. Some volumes are covers only (`coveronly`).
80
+ - Bill to law: `related_collection` with `accessId` BILLS-...enr and `collection` PLAW gives the law's
81
+ package, citation and date (the plain `related` lists only the collections). An enrolled version's
82
+ `dateIssued` is GPO's print date, not the enrollment date: it can fall after the signing (BILLS-119hr1enr
83
+ is dated 2025-07-09, signed 2025-07-04).
84
+ - PLAW holds private laws too: `published` rows carry `docClass` PUBLIC or PRIVATE (packageId `publ` /
85
+ `pvtl`); 2022 had 247 public and 2 private. Count public laws on `docClass == "PUBLIC"`. For PLAW,
86
+ `dateIssued` is usually the approval date, but GovInfo has errors (PLAW-119publ76 is dated 2025-02-06; it
87
+ was approved 2026-02-06): for counts by date, check the Office of the Federal Register's list or the
88
+ congress pack's `laws`.
@@ -9,7 +9,7 @@ agency publications catalogued by GPO.
9
9
  Code sections it amends; public laws enacted in a date range; committee reports, hearings and Congressional Record issues on a topic; agency reports in GOVPUB.
10
10
  - Not here: bill status, sponsors, cosponsors, actions and votes (congress); rule metadata and counts
11
11
  (federal-register); public comments (regulations). Cost estimates as numbers are not available from
12
- any API here (see traps).
12
+ any API here (doc "collections").
13
13
 
14
14
  ## Endpoints and datasets
15
15
  | Endpoint | Returns (one row per) | Key parameters |
@@ -99,17 +99,13 @@ chamber), enr enrolled (passed both).
99
99
  `committee_meetings`. A committee's hearings are best selected by its SuDoc class
100
100
  (`query: 'collection:CHRG AND congress:118 AND sudocclass:"y 4.j 89/2:"'`, Senate Judiciary), which also
101
101
  leaves out joint hearings another committee printed.
102
- - `dateIssued` is otherwise the publication date; for GOVPUB catalog records it can be a cataloguing date that is
102
+ - `dateIssued` is otherwise the publication date; for GOVPUB catalog records it can be a cataloguing date that
103
+ is
103
104
  later than the document itself (seen in Oct 2026: a record ingested in 2024 shows dateIssued 2026-06-27).
104
105
  Check the PDF before dating a document from GOVPUB.
105
- - Cost estimates: the budget office's own estimates are not a GovInfo collection. Two partial routes,
106
- both text, not tables: (1) `collection: GOVPUB` with the publisher filter
107
- (`publisher: "Congressional Budget Office"`, a report subset) returns the reports GPO has catalogued
108
- (765 records in Oct 2026, far from the office's full list of publications);
109
- (2) `collection: CRPT` with `phrase: "cost estimate"` finds committee reports, which usually reprint the
110
- estimate for a reported bill in the report text. Congress.gov bill records (congress pack, endpoint
111
- `bill_cbo_estimates`) list the titles and links of the estimates for a bill. No API here returns the
112
- estimated dollar figures as data.
106
+ - Collection-specific traps (CREC, CPD, CRPT, CPRT, STATUTE, BUDGET, USCOURTS, CFR, PLAW, bill to law, page
107
+ counts, cost estimates): read `source_brief({source: "govinfo", doc: "collections"})` before counting or
108
+ reading a collection.
113
109
  - `text` returns the GPO plain-text rendition (`[[Page 138 STAT. 2687]]` page markers kept). Large
114
110
  packages (an omnibus law) give a text column of several MB; set `max_chars` to keep a sample. Some
115
111
  packages have only PDF: the request then fails with "no text rendition". Committee reports and hearings
@@ -145,82 +141,6 @@ chamber), enr enrolled (passed both).
145
141
  Issue 45") are in `package` summaries: fetch the `published` list,
146
142
  then one `package` request per row with `"for_each": {"csv": "data/govinfo/<list>.csv", "column":
147
143
  "packageId"}`.
148
- - CREC (daily Congressional Record): one package per issue date, but a two-book day can appear twice
149
- (`CREC-2025-03-11` and `CREC-2025-03-11-i45`, same pages; `-i46` the second book): deduplicate by
150
- `volume` + `issue` before counting issues or summing pages. Other suffixes are distinct issues, not
151
- duplicates: `-v169`/`-v170` on January 3 (two volumes, the outgoing and incoming Congress) and a day with
152
- `-i64`/`-i65`. Chamber sections are granules of class SENATE, HOUSE, EXTENSIONS and DAILYDIGEST (the
153
- summary's `download.pdfSenateLink` / `pdfHouseLink` too). Early-January issues of a calendar year
154
- belong to the outgoing Congress (volume and congress differ; `CREC-2015-01-02` is volume 160 but is
155
- labelled congress 114), and an issue can be missing. Proceedings print under the legislative day, not the calendar day: a
156
- Senate session that runs on (July 1, 2025, a continuation of June 30's legislative day) has no issue of its
157
- own, so `granules` for that date returns no rows; find the issue with `search` (CREC, a phrase from the
158
- proceedings) instead. `pages` counts PDF pages (Daily Digest and front
159
- matter included), so it runs about 1 percent above the Résumé of Congressional Activity's numbered pages.
160
- - "How many pages is the law": the convention is its Statutes at Large pages (first page to the last
161
- `[[Page ... STAT. ...]]` marker of `text`, e.g. 139 Stat. 72-401 is 330 pages); the slip-law and enrolled-bill
162
- PDFs differ by a page or so (cover, signatures). Say which measure the answer uses.
163
- - CPD (Compilation of Presidential Documents, 1993 on; daily from 2009-01-20): one package per document,
164
- docClass PRESDOC plus a few DIGEST, ACTSAPPROVED, CHECKLIST and NOMINATIONS rows (in a transition year the
165
- outgoing President's supplements, `DCPD-2025DIGEST-BIDEN` and the like, are docClass PRESDOC too: count
166
- numbered DCPD-YYYY0NNNN packages). The `published` list has no document type; each `package` summary has
167
- `dcpdCategory` (level1 "Executive Orders", "Proclamations" ...), and `search` takes `collection:CPD AND
168
- cpdcategory:proclamation` (signing remarks included). Some orders are missing or have no number in the
169
- title, and `dateIssued` can be a day after signing. Count executive orders from the
170
- federal-register pack (`executive_orders`, by signing date) and use CPD as the cross-check.
171
- - CRPT (committee reports): one package per numbered report (its parts and errata inside it); GovInfo's
172
- browse pages count documents (parts and errata), so they run above the report count. `published` CRPT also
173
- lists GPO duplicate packages (GPO-J6-REPORT, GPO-CRPT-117hrpt646): keep `packageId` CRPT-*. A few
174
- `dateIssued` values are wrong (three Dec-2022 reports dated 2021-12-31): count by Congress, not by date.
175
- - CPRT (committee prints, "select prints": GovInfo is not a census of them): summaries carry no
176
- `committees`; attribute a print to its committee by `suDocClassNumber` (the `Y 4.` stem, without the
177
- `-NN` series suffix; a few prints have two classes or none). `published` with collection CPRT also returns
178
- packages of other collections (GPO-PLUMBOOK-2024, collectionCode GPO): keep `collectionCode` CPRT.
179
- GovInfo's browse pages count documents (errata included), not packages.
180
- - STATUTE: one package per volume and volume = a session of Congress, not a calendar year. A law goes in
181
- the volume of the session that passed it, not the year it was signed: laws signed in early January after
182
- a session ends are in the closing session's volume (116-344, signed 2021-01-13, in vol. 134; 118-274,
183
- 2025-01-06, in vol. 138), and numbering can interleave (113-74, signed 2014-01-16, closes vol. 127 while
184
- 113-73 opens vol. 128). Assign laws to volumes from the granules, never by date. Granule `dateIssued` can
185
- be wrong by a year (117-343's granule says 2022-01-05; signed 2023-01-05). Count laws by granules
186
- with `granuleClass` PUBLICLAW (granules also hold proclamations and concurrent resolutions). Older
187
- summaries (vol. 127) have no `pages`; `pages` counts front matter too. A granule id carries its start page
188
- (`STATUTE-136-Pg6299`; a continuation adds a suffix, `STATUTE-135-Pg2531-2`). Each volume prints public
189
- laws, then private laws, then concurrent resolutions and proclamations, with a few blank pages between the
190
- parts. The public laws' pages run from the first law's start page to the **last law's last page**: the last
191
- `[[Page N STAT. M]]` marker in that law's `text` (`packageId` PLAW-...). The next granule's start page
192
- overstates it by 3 or 4 pages (6 more in vol. 136, where private laws come first); with the end from the
193
- text, the 116th-118th Congresses give 8,432, 8,742 and 4,350 pages, the figures in Brookings' Vital
194
- Statistics on Congress (Table 6-4).
195
- - BUDGET: one package per volume (`BUDGET-2027-BUD` main volume, `-PER` Analytical Perspectives, `-APP`
196
- Appendix). Summaries report collectionCode GPO (FY2017 and FY2027 alike) although `published` with
197
- collection BUDGET lists them. Recent summaries' `pages` equals the PDF's page count; older ones (FY2017)
198
- carry none: download `https://www.govinfo.gov/content/pkg/<packageId>/pdf/<packageId>.pdf`, count its pages
199
- (pypdf) and record them in `published/`. A missing volume (FY2026 had no Analytical
200
- Perspectives) is a 404: allow_empty. OMB posts supplemental chapters outside the volumes.
201
- - USCOURTS (federal court opinions): one package per case, one granule per opinion; `search` returns one row
202
- per opinion (`collection:USCOURTS AND courttype:appellate`, `courtname:fifth`; `ingestdate:range(...)`
203
- filters on the date added to GovInfo, not the opinion date, and 2025 opinions keep arriving in 2026). A year of appellate opinions (about 15,000) passes the
204
- 10,000-row search cap: split by half-year. The court code is in the packageId (`USCOURTS-ca5-...`; the
205
- Federal Circuit is ca13). Courts choose what they send (the Ninth Circuit sends precedential opinions only),
206
- so a ranking reflects submission practice, not caseload: say so; the AO's Judicial Business tables (B-12)
207
- count opinions by circuit.
208
- - CFR: one package per title, edition and volume (`CFR-2025-title40-vol17`). `dateIssued` is the
209
- "revised as of" date (January 1 for titles 1-16, April 1 for 17-27, July 1 for 28-41, October 1 for
210
- 42-50), not the posting date, which lags by months: a recent
211
- edition can be incomplete, with volumes missing (404) while `volumeCount` counts them. `published`
212
- rows carry no page count, so a title's pages take one `package` call per volume (`pages`, every PDF page
213
- including front matter); give missing volumes `"allow_empty": true` and say they are missing, or fill
214
- them from the prior edition as a labeled sensitivity case. Some volumes are covers only (`coveronly`).
215
- - Bill to law: `related_collection` with `accessId` BILLS-...enr and `collection` PLAW gives the law's
216
- package, citation and date (the plain `related` lists only the collections). An enrolled version's
217
- `dateIssued` is GPO's print date, not the enrollment date: it can fall after the signing (BILLS-119hr1enr
218
- is dated 2025-07-09, signed 2025-07-04).
219
- - PLAW holds private laws too: `published` rows carry `docClass` PUBLIC or PRIVATE (packageId `publ` /
220
- `pvtl`); 2022 had 247 public and 2 private. Count public laws on `docClass == "PUBLIC"`. For PLAW,
221
- `dateIssued` is usually the approval date, but GovInfo has errors (PLAW-119publ76 is dated 2025-02-06; it
222
- was approved 2026-02-06): for counts by date, check the Office of the Federal Register's list or the
223
- congress pack's `laws`.
224
144
  - `governmentAuthor` is a "; "-joined list. `collectionCode` can hold two codes (`SERIALSET;CRPT`).
225
145
  - Paging: `search` uses GovInfo's `offsetMark` cursor (100 per page); `published` and `granules` follow
226
146
  `nextPage` (1,000 per page).
@@ -0,0 +1,15 @@
1
+ # IRS exempt organizations: eo_bmf and eo_extract
2
+
3
+ From the irs-soi playbook's "Exempt organizations (`eo_bmf`, `eo_extract`)" section.
4
+
5
+ - `eo_bmf`: `https://www.irs.gov/pub/irs-soi/eo_{st}.csv` (one per state, PR, XX = international; updated monthly,
6
+ 1,975,975 organizations on 2026-10-13), 28 columns. Filters: `ntee` (prefix), `subsection` (`03` = 501(c)(3)),
7
+ `ein`, `name`, `columns`. EIN and ZIP stay text. ASSET_AMT, INCOME_AMT, REVENUE_AMT are whole dollars from the
8
+ latest return on file (TAX_PERIOD differs by organization). Codes: https://www.irs.gov/pub/foia/ig/tege/eo-info.pdf.
9
+ - `eo_extract`: the SOI annual extract of Form 990 (`form: 990`), 990-EZ (`990ez`) or 990-PF (`990pf`) data,
10
+ `{yy}eoextract990.zip` etc. (2018-2024; `{yy}eofinextract...` 2012-2017), listed on
11
+ soi-tax-stats-annual-extract-of-tax-exempt-organization-financial-data. The 2024 Form 990 zip is 57 MB holding a
12
+ 247 MB CSV of 246 columns; it is read as a stream keeping only `ein` or the organizations of `state` (their EINs
13
+ from the current EO BMF, so an organization that moved or lost exemption is missed). The 990-PF CSV drops EIN
14
+ leading zeros: they are restored. Field definitions: `{yy}eofinextractdoc.xlsx`. Years whose zip holds a
15
+ fixed-width `.dat` file (some of 2012-2017) are refused.
@@ -0,0 +1,50 @@
1
+ # IRS SOI published tables (table, table_list)
2
+
3
+ From the irs-soi playbook's "Published tables (`table`, `table_list`)" section.
4
+
5
+ SOI publishes its national statistics as .xls/.xlsx workbooks, one per table and year. `packs/irs-soi/tables.json`
6
+ catalogs 125 of them (checked 2026-11-08); `table_list` lists them (`product`, `query`; `live: true` adds the
7
+ years posted now). File names change over the years (the Data Book renumbered its tables in 2006, `97in11.xls`,
8
+ `ccr`/`nr`/`tabl` suffixes), so the adapter reads the table's index page and takes the year's link from it; a link
9
+ the page gets wrong is corrected in the catalog (`overrides`: partnership Tables 5 and 7 for 2023).
10
+
11
+ | Product (ids) | Tables | Years | Year |
12
+ |---|---|---|---|
13
+ | Individual, Publication 1304 (`in_1.1` ... `in_3.6`) | 1.1 income and tax by size of AGI, 1.2 by marital status, 1.4 sources of income, 1.4A capital gains, 1.6 age, 1.7 dependents, 2.1 itemized deductions, 2.5 EITC, 2.7 ACA, 3.1-3.6 tax computation, tax by rate (3.5: income taxed at each rate) | TY1996-2023 | tax year |
14
+ | Individual, percentiles (`shares_4.1`, `shares_4.2`, `shares_4.3`) | returns, AGI floor, AGI, income tax, average tax rate and shares, top 0.001% ... top 50% and bottom 50% | 2001-2023 in one file | edition |
15
+ | Corporations, Publication 16 (`corp_1` ... `corp_14`) | 1 minor industry, 2.1-2.4A size of total assets (2.3 C corporations, 2.4 S corporations), 3.x size of receipts, 4 size of tax, 5.x minor industry, 6-9 S corporations, 10 foreign (1120-F), 11 tax and credits, 12-14 COGS, depreciation, research credit; `corp_1-CV` coefficients of variation | TY2014-2023 (Table 1 from 1994) | tax year |
16
+ | Partnerships (`partnership_1` ... `_7`, `_10`, `_23`) | by industrial group: income and deductions, balance sheets, rental real estate, income allocated to partners, LLCs, receipts, foreign transactions | TY1993-2023 | tax year |
17
+ | Sole proprietorships (`soleprop_1` ... `_4`) | by industrial sector: receipts, deductions, payroll, net income; income statements; size of receipts; returns with Schedule C | TY1996-2023 | tax year |
18
+ | Integrated business data (`business_1`, `business_3`) | returns, receipts and net income by form of business, 1980-2015; by NAICS sector | to 2015 | edition / tax year |
19
+ | IRS Data Book (`databook_1-1` ... `_1-8`, `_3-1`, `_6-1`, `_6-2`) | collections and refunds by type of tax, returns filed, by state; gross collections by type of tax 1960 onward (1-6); audit coverage; costs; cost of collecting $100 | FY1995-2025 | fiscal year |
20
+ | Historical tables (`historical_1` ... `_24`) | individual series since 1913/1950, corporations since 1909, NIPA vs SOI income (8, 15), excise taxes by type FY1999 onward (20), returns filed (21), corporate rates 1909-2010 (24) | one file each | edition |
21
+ | Estate, gift, wealth (`estate_1` ... `_5`, `gift_1`, `wealth_1` ... `_6`) | estate returns by size of gross estate and tax status, by state, charitable bequests, beneficiaries, occupation; gift returns; top wealth holders | FY2001-2024; gift 2003-2021; wealth 2001-2019 | filing year |
22
+ | Exempt organizations (`eo_1` ... `_4`, `eo_excise`, `pf_1` ... `_4`) | Form 990 501(c)(3) by asset size, by Code section, 990-EZ, Form 4720 excise; private foundations | 1985-2024 | tax year |
23
+ | International (`cfc_1` ... `_3`, `ftc_1` ... `_3`, `cbc_1A` ... `cbc_4`) | controlled foreign corporations, corporate foreign tax credit, country-by-country reports | to 2021/2023 | tax year |
24
+
25
+ Parameters: `table` (an id, or a product word and number: "corporation 2.1", "data book 1-6"); `year` (or
26
+ `tax_year`, `fiscal_year`, `filing_year`; `start_year`/`end_year`, one workbook each); default the latest posted.
27
+ A whole-series table (`shares_4.x`, `databook_1-6`, `historical_*`, `business_1`) holds every year in one file:
28
+ the latest edition is read and the years are rows or columns (filter them with `rows`, `section` or `columns`).
29
+ `rows` keeps rows by label (an exact label, else the labels containing the text), `section` by section label,
30
+ `columns` by published column number or header text; `layout: "wide"` gives one row per line; `sheet` picks a
31
+ sheet of a multi-sheet workbook; `header_lines` overrides the header reading (rarely needed: see the notes).
32
+
33
+ How a workbook is read (no settings per table): title and headnote lines; the header lines down to the row of
34
+ column numbers (`1`, `(1)`, or `-1` shown as `(1)`); label-only lines become `SECTION` (a label running over two
35
+ lines is joined); the row level comes from the cell indent (or leading spaces) and gives `LEVEL` and `PARENT`;
36
+ footnote lines after the last data line go to the notes. A sheet printed as pages (1998 tables: columns 1-20 on
37
+ page 1, 21-40 on page 2) is read block by block; side-by-side panels (older partnership and sole proprietorship
38
+ tables) take each cell's label from the nearest label column to its left. A table without column numbers
39
+ (Publication 16 Table 2.1 for 2014, some historical tables) is read with the header down to the first line with
40
+ a label and a number, and says so in the notes, with `COL` empty.
41
+
42
+ Output (long, the default): `TABLE, YEAR, YEAR_TYPE, SHEET, SECTION, LINE, LEVEL, PARENT, ROW_LABEL, COL, COLUMN,
43
+ VALUE, FLAG, UNITS`. `COLUMN` joins the header lines with " | " ("All returns | Adjusted gross income less deficit
44
+ | Amount"); `COL` is the published column number, stable within a year (not across years: check `COLUMN`).
45
+ `VALUE` is the number as published (no thousands separators); `FLAG` holds the cell's markers: `*` estimate based
46
+ on few sample returns (high sampling variability; in newer workbooks the `*` is only in the number format and is
47
+ read from it), `**` data combined or suppressed, `d`/`[d]` deleted to avoid disclosure (included in totals), `[n]`
48
+ a footnote (e.g. `[2]` less than 0.05 percent), `n.a.` not available, `--`/`-` zero or not applicable as the
49
+ table's footnotes say. `UNITS` is the money unit of the headnote (usually thousands of dollars; gift tables whole
50
+ dollars; historical tables millions or billions); count, percent and average columns are as their header says.
@@ -49,65 +49,14 @@ the index pages (ZIP `{yy}zpdoc.docx`, county `{yy}incydocguide.docx`, state `{y
49
49
  migration `{y1y2}inpublicmigdoc.pdf`).
50
50
 
51
51
  ## Published tables (`table`, `table_list`)
52
- SOI publishes its national statistics as .xls/.xlsx workbooks, one per table and year. `packs/irs-soi/tables.json`
53
- catalogs 125 of them (checked 2026-11-08); `table_list` lists them (`product`, `query`; `live: true` adds the
54
- years posted now). File names change over the years (the Data Book renumbered its tables in 2006, `97in11.xls`,
55
- `ccr`/`nr`/`tabl` suffixes), so the adapter reads the table's index page and takes the year's link from it; a link
56
- the page gets wrong is corrected in the catalog (`overrides`: partnership Tables 5 and 7 for 2023).
57
-
58
- | Product (ids) | Tables | Years | Year |
59
- |---|---|---|---|
60
- | Individual, Publication 1304 (`in_1.1` ... `in_3.6`) | 1.1 income and tax by size of AGI, 1.2 by marital status, 1.4 sources of income, 1.4A capital gains, 1.6 age, 1.7 dependents, 2.1 itemized deductions, 2.5 EITC, 2.7 ACA, 3.1-3.6 tax computation, tax by rate (3.5: income taxed at each rate) | TY1996-2023 | tax year |
61
- | Individual, percentiles (`shares_4.1`, `shares_4.2`, `shares_4.3`) | returns, AGI floor, AGI, income tax, average tax rate and shares, top 0.001% ... top 50% and bottom 50% | 2001-2023 in one file | edition |
62
- | Corporations, Publication 16 (`corp_1` ... `corp_14`) | 1 minor industry, 2.1-2.4A size of total assets (2.3 C corporations, 2.4 S corporations), 3.x size of receipts, 4 size of tax, 5.x minor industry, 6-9 S corporations, 10 foreign (1120-F), 11 tax and credits, 12-14 COGS, depreciation, research credit; `corp_1-CV` coefficients of variation | TY2014-2023 (Table 1 from 1994) | tax year |
63
- | Partnerships (`partnership_1` ... `_7`, `_10`, `_23`) | by industrial group: income and deductions, balance sheets, rental real estate, income allocated to partners, LLCs, receipts, foreign transactions | TY1993-2023 | tax year |
64
- | Sole proprietorships (`soleprop_1` ... `_4`) | by industrial sector: receipts, deductions, payroll, net income; income statements; size of receipts; returns with Schedule C | TY1996-2023 | tax year |
65
- | Integrated business data (`business_1`, `business_3`) | returns, receipts and net income by form of business, 1980-2015; by NAICS sector | to 2015 | edition / tax year |
66
- | IRS Data Book (`databook_1-1` ... `_1-8`, `_3-1`, `_6-1`, `_6-2`) | collections and refunds by type of tax, returns filed, by state; gross collections by type of tax 1960 onward (1-6); audit coverage; costs; cost of collecting $100 | FY1995-2025 | fiscal year |
67
- | Historical tables (`historical_1` ... `_24`) | individual series since 1913/1950, corporations since 1909, NIPA vs SOI income (8, 15), excise taxes by type FY1999 onward (20), returns filed (21), corporate rates 1909-2010 (24) | one file each | edition |
68
- | Estate, gift, wealth (`estate_1` ... `_5`, `gift_1`, `wealth_1` ... `_6`) | estate returns by size of gross estate and tax status, by state, charitable bequests, beneficiaries, occupation; gift returns; top wealth holders | FY2001-2024; gift 2003-2021; wealth 2001-2019 | filing year |
69
- | Exempt organizations (`eo_1` ... `_4`, `eo_excise`, `pf_1` ... `_4`) | Form 990 501(c)(3) by asset size, by Code section, 990-EZ, Form 4720 excise; private foundations | 1985-2024 | tax year |
70
- | International (`cfc_1` ... `_3`, `ftc_1` ... `_3`, `cbc_1A` ... `cbc_4`) | controlled foreign corporations, corporate foreign tax credit, country-by-country reports | to 2021/2023 | tax year |
71
-
72
- Parameters: `table` (an id, or a product word and number: "corporation 2.1", "data book 1-6"); `year` (or
73
- `tax_year`, `fiscal_year`, `filing_year`; `start_year`/`end_year`, one workbook each); default the latest posted.
74
- A whole-series table (`shares_4.x`, `databook_1-6`, `historical_*`, `business_1`) holds every year in one file:
75
- the latest edition is read and the years are rows or columns (filter them with `rows`, `section` or `columns`).
76
- `rows` keeps rows by label (an exact label, else the labels containing the text), `section` by section label,
77
- `columns` by published column number or header text; `layout: "wide"` gives one row per line; `sheet` picks a
78
- sheet of a multi-sheet workbook; `header_lines` overrides the header reading (rarely needed: see the notes).
79
-
80
- How a workbook is read (no settings per table): title and headnote lines; the header lines down to the row of
81
- column numbers (`1`, `(1)`, or `-1` shown as `(1)`); label-only lines become `SECTION` (a label running over two
82
- lines is joined); the row level comes from the cell indent (or leading spaces) and gives `LEVEL` and `PARENT`;
83
- footnote lines after the last data line go to the notes. A sheet printed as pages (1998 tables: columns 1-20 on
84
- page 1, 21-40 on page 2) is read block by block; side-by-side panels (older partnership and sole proprietorship
85
- tables) take each cell's label from the nearest label column to its left. A table without column numbers
86
- (Publication 16 Table 2.1 for 2014, some historical tables) is read with the header down to the first line with
87
- a label and a number, and says so in the notes, with `COL` empty.
88
-
89
- Output (long, the default): `TABLE, YEAR, YEAR_TYPE, SHEET, SECTION, LINE, LEVEL, PARENT, ROW_LABEL, COL, COLUMN,
90
- VALUE, FLAG, UNITS`. `COLUMN` joins the header lines with " | " ("All returns | Adjusted gross income less deficit
91
- | Amount"); `COL` is the published column number, stable within a year (not across years: check `COLUMN`).
92
- `VALUE` is the number as published (no thousands separators); `FLAG` holds the cell's markers: `*` estimate based
93
- on few sample returns (high sampling variability; in newer workbooks the `*` is only in the number format and is
94
- read from it), `**` data combined or suppressed, `d`/`[d]` deleted to avoid disclosure (included in totals), `[n]`
95
- a footnote (e.g. `[2]` less than 0.05 percent), `n.a.` not available, `--`/`-` zero or not applicable as the
96
- table's footnotes say. `UNITS` is the money unit of the headnote (usually thousands of dollars; gift tables whole
97
- dollars; historical tables millions or billions); count, percent and average columns are as their header says.
52
+ The SOI published tables (`table`, `table_list`: Publication 1304, tax shares by percentile, Publication 16,
53
+ partnerships, sole proprietorships, the Data Book, historical, estate, gift, wealth, exempt organizations,
54
+ international) and how their layouts are read: `source_brief({source: "irs-soi", doc: "tables"})`. Read it before
55
+ the first `table` request. Estimate recipes from them: doc `estimate_patterns`.
98
56
 
99
57
  ## Exempt organizations (`eo_bmf`, `eo_extract`)
100
- - `eo_bmf`: `https://www.irs.gov/pub/irs-soi/eo_{st}.csv` (one per state, PR, XX = international; updated monthly,
101
- 1,975,975 organizations on 2026-10-13), 28 columns. Filters: `ntee` (prefix), `subsection` (`03` = 501(c)(3)),
102
- `ein`, `name`, `columns`. EIN and ZIP stay text. ASSET_AMT, INCOME_AMT, REVENUE_AMT are whole dollars from the
103
- latest return on file (TAX_PERIOD differs by organization). Codes: https://www.irs.gov/pub/foia/ig/tege/eo-info.pdf.
104
- - `eo_extract`: the SOI annual extract of Form 990 (`form: 990`), 990-EZ (`990ez`) or 990-PF (`990pf`) data,
105
- `{yy}eoextract990.zip` etc. (2018-2024; `{yy}eofinextract...` 2012-2017), listed on
106
- soi-tax-stats-annual-extract-of-tax-exempt-organization-financial-data. The 2024 Form 990 zip is 57 MB holding a
107
- 247 MB CSV of 246 columns; it is read as a stream keeping only `ein` or the organizations of `state` (their EINs
108
- from the current EO BMF, so an organization that moved or lost exemption is missed). The 990-PF CSV drops EIN
109
- leading zeros: they are restored. Field definitions: `{yy}eofinextractdoc.xlsx`. Years whose zip holds a
110
- fixed-width `.dat` file (some of 2012-2017) are refused.
58
+ The exempt-organization registry (`eo_bmf`) and Form 990 extracts (`eo_extract`): `source_brief({source:
59
+ "irs-soi", doc: "eo"})`. Read it before the first request for either.
111
60
 
112
61
  ## Spec examples
113
62
  ```json
package/packs/port.json CHANGED
@@ -264,6 +264,11 @@
264
264
  "adapter.py"
265
265
  ]
266
266
  },
267
+ "rma": {
268
+ "python": [
269
+ "adapter.py"
270
+ ]
271
+ },
267
272
  "sba": {
268
273
  "python": [
269
274
  "adapter.py"
@@ -0,0 +1,44 @@
1
+ {
2
+ "base_url": "https://pubfs-rma.fpac.usda.gov",
3
+ "timeout": 240,
4
+ "endpoints": {
5
+ "coverage": {
6
+ "path": "/pub/Web_Data_Files/Summary_of_Business/state_county_crop/sobcov_{year}.zip",
7
+ "method": "GET",
8
+ "description": "Federal crop insurance Summary of Business by commodity year, state, county, crop, insurance plan, coverage category, delivery type and coverage level: policies, units, insured quantity, liability, total premium, premium subsidy, indemnity and loss ratio, dollars (RMA, 1989 on; updated weekly for open years).",
9
+ "params": {
10
+ "year": {"in": "path", "required": true, "desc": "commodity year, 1989 on (the current and next years are partial)"},
11
+ "state": {"in": "spec", "desc": "keep these states (USPS abbreviation or 2-digit FIPS code; a list works)"},
12
+ "county": {"in": "spec", "desc": "keep these 3-digit county FIPS codes (999 is All Other Counties)"},
13
+ "commodity": {"in": "spec", "desc": "keep these commodities (name as published, e.g. Corn, Soybeans, or the 4-digit RMA code; case-insensitive)"},
14
+ "plan": {"in": "spec", "desc": "keep these insurance plans (abbreviation, e.g. RP, YP, RPHPE, APH, or the plan code)"},
15
+ "coverage_category": {"in": "spec", "desc": "keep A (buy-up), C (CAT), E (existing coverage policy) or L (limited)"},
16
+ "delivery_type": {"in": "spec", "desc": "keep RBUP, RCAT (reinsured, sold by companies), FBUP or FCAT (federal)"}
17
+ },
18
+ "records": "$zip:*.txt",
19
+ "delimiter": "|",
20
+ "columns": ["Commodity Year", "Location State Code", "Location State Abbreviation", "Location County Code",
21
+ "Location County Name", "Commodity Code", "Commodity Name", "Insurance Plan Code", "Insurance Plan Name Abbreviation",
22
+ "Coverage Category", "Delivery Type", "Coverage Level", "Policies Sold Count", "Policies Earning Premium Count",
23
+ "Policies Indemnified Count", "Units Earning Premium Count", "Units Indemnified Count", "Quantity Type",
24
+ "Net Reported Quantity", "Endorsed/Companion Acres", "Liability Amount", "Total Premium Amount", "Subsidy Amount",
25
+ "State/Private Subsidy", "Additional Subsidy", "EFA Premium Discount", "Indemnity Amount", "Loss Ratio"],
26
+ "paging": {"type": "none"},
27
+ "max_rows": 2000000
28
+ },
29
+ "reinsurance": {
30
+ "path": "/pub/Web_Data_Files/Reinsurance_Reports/standard_reinsurance_agreement_reports/{year}_Standard_Reinsurance_Report_{level}.xls",
31
+ "method": "GET",
32
+ "description": "Standard Reinsurance Agreement results by reinsurance year and fund (A assigned, C commercial), national or by state: gross liability, premium and indemnity, and the companies' retained liability, premium, indemnity and net underwriting gain or loss, dollars (RMA, 2014 on).",
33
+ "params": {
34
+ "year": {"in": "path", "required": true, "desc": "reinsurance year, 2014 on (the latest is not final)"},
35
+ "level": {"in": "path", "required": true, "desc": "National (totals by fund) or State (each state by fund)"},
36
+ "state": {"in": "spec", "desc": "keep these states (USPS abbreviation) in the State report"}
37
+ },
38
+ "records": "$xls",
39
+ "header_row": 7,
40
+ "paging": {"type": "none"},
41
+ "max_rows": 10000
42
+ }
43
+ }
44
+ }
@@ -0,0 +1,94 @@
1
+ {
2
+ "id": "rma",
3
+ "title": "USDA RMA federal crop insurance: Summary of Business (liability, premium, subsidy, indemnity) and reinsurance results",
4
+ "agency": "U.S. Department of Agriculture, Risk Management Agency",
5
+ "depth": "standard",
6
+ "domains": [
7
+ "agriculture",
8
+ "federal spending",
9
+ "insurance"
10
+ ],
11
+ "keywords": [
12
+ "rma",
13
+ "risk management agency",
14
+ "crop insurance",
15
+ "federal crop insurance",
16
+ "federal crop insurance corporation",
17
+ "fcic",
18
+ "premium subsidy",
19
+ "crop insurance subsidy",
20
+ "indemnity",
21
+ "indemnities",
22
+ "loss ratio",
23
+ "insured acres",
24
+ "liability",
25
+ "summary of business",
26
+ "revenue protection",
27
+ "yield protection",
28
+ "catastrophic coverage",
29
+ "coverage level",
30
+ "standard reinsurance agreement",
31
+ "underwriting gains",
32
+ "approved insurance providers"
33
+ ],
34
+ "questions": [
35
+ "federal crop insurance premium subsidies by state or crop",
36
+ "crop insurance indemnities and loss ratios over time",
37
+ "how much would raising the premium subsidy rate cost",
38
+ "insured acres and liability for corn in Iowa",
39
+ "crop insurance companies' underwriting gains",
40
+ "coverage levels farmers buy by crop"
41
+ ],
42
+ "homepage": "https://www.rma.usda.gov/tools-reports/summary-of-business",
43
+ "terms_url": "https://pubfs-rma.fpac.usda.gov/pub/Web_Data_Files/",
44
+ "auth": {
45
+ "env": null,
46
+ "required": "none",
47
+ "placement": "none",
48
+ "signup_url": null,
49
+ "echoes_key": false
50
+ },
51
+ "limits": {
52
+ "rps": 1,
53
+ "burst": 2,
54
+ "max_rows": 2000000
55
+ },
56
+ "routes": {
57
+ "api": {
58
+ "base": "https://pubfs-rma.fpac.usda.gov",
59
+ "probe": "https://pubfs-rma.fpac.usda.gov/pub/Web_Data_Files/Summary_of_Business/state_county_crop/",
60
+ "hosts": [
61
+ "pubfs-rma.fpac.usda.gov"
62
+ ]
63
+ },
64
+ "manual": {
65
+ "how": "Open https://pubfs-rma.fpac.usda.gov/pub/Web_Data_Files/Summary_of_Business/state_county_crop/ (sobcov_<year>.zip) or .../Reinsurance_Reports/standard_reinsurance_agreement_reports/ (<year>_Standard_Reinsurance_Report_National.xls or _State.xls), download the file and upload it with ingest."
66
+ }
67
+ },
68
+ "cache": {
69
+ "mode": "run_only",
70
+ "ttl_hours": null,
71
+ "why": "Open crop years are updated every week; data/ is reused within a run."
72
+ },
73
+ "redistribution": {
74
+ "mode": "allowed",
75
+ "why": "RMA data are U.S. government works in the public domain (17 U.S.C. 105)."
76
+ },
77
+ "notices": [],
78
+ "retention": null,
79
+ "citation": "U.S. Department of Agriculture, Risk Management Agency, {dataset}, pubfs-rma.fpac.usda.gov, retrieved {retrieved_long}.",
80
+ "audit_hooks": [],
81
+ "clarify": [
82
+ "Which money? Total premium is the full premium; the subsidy is the federal share of it (farmer-paid premium = total premium - subsidy - state/private subsidy - additional subsidy - EFA discount); indemnities are losses paid. Program cost also includes delivery expense reimbursements and companies' underwriting gains (reinsurance), which the Summary of Business does not hold. Default: say which of these the number is.",
83
+ "Crop year or fiscal year? The Summary of Business is by commodity (crop) year, and reinsurance results by reinsurance year (July-June); federal outlays are by fiscal year (budget: treasury, omb). Default: crop year, said as such, and never added to fiscal-year outlays without saying so."
84
+ ],
85
+ "formats": [
86
+ "csv"
87
+ ],
88
+ "aliases": [
89
+ "RMA",
90
+ "FCIC",
91
+ "Summary of Business",
92
+ "crop insurance"
93
+ ]
94
+ }