data-analyst-mcp 0.6.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -2
- package/dist/adapters/_census_micro.d.ts +2 -0
- package/dist/adapters/_census_micro.js +24 -1
- package/dist/adapters/_census_micro.js.map +1 -1
- package/dist/analysis/combine.js +3 -2
- package/dist/analysis/combine.js.map +1 -1
- package/dist/analysis/compute.d.ts +30 -7
- package/dist/analysis/compute.js +414 -162
- package/dist/analysis/compute.js.map +1 -1
- package/dist/analysis/expr.js +6 -2
- package/dist/analysis/expr.js.map +1 -1
- package/dist/analysis/handoff.d.ts +1 -1
- package/dist/analysis/handoff.js +44 -3
- package/dist/analysis/handoff.js.map +1 -1
- package/dist/analysis/rerun.js +2 -1
- package/dist/analysis/rerun.js.map +1 -1
- package/dist/analysis/rerun_job.d.ts +30 -0
- package/dist/analysis/rerun_job.js +173 -0
- package/dist/analysis/rerun_job.js.map +1 -0
- package/dist/analysis/rerun_worker.d.ts +1 -0
- package/dist/analysis/rerun_worker.js +13 -0
- package/dist/analysis/rerun_worker.js.map +1 -0
- package/dist/analysis/table.d.ts +7 -0
- package/dist/analysis/table.js +30 -1
- package/dist/analysis/table.js.map +1 -1
- package/dist/analysis/tools.d.ts +10 -7
- package/dist/analysis/tools.js +43 -45
- package/dist/analysis/tools.js.map +1 -1
- package/dist/core/compact.d.ts +15 -0
- package/dist/core/compact.js +98 -0
- package/dist/core/compact.js.map +1 -0
- package/dist/core/package.d.ts +8 -0
- package/dist/core/package.js +30 -1
- package/dist/core/package.js.map +1 -1
- package/dist/core/packs.d.ts +4 -2
- package/dist/core/sections.d.ts +21 -0
- package/dist/core/sections.js +53 -0
- package/dist/core/sections.js.map +1 -0
- package/dist/deep/opm.js +10 -2
- package/dist/deep/opm.js.map +1 -1
- package/dist/deep/treasury.js +7 -3
- package/dist/deep/treasury.js.map +1 -1
- package/dist/envelope.d.ts +1 -1
- package/dist/envelope.js +1 -1
- package/dist/envelope.js.map +1 -1
- package/dist/index.d.ts +57 -4
- package/dist/report/audit_answer.d.ts +1 -1
- package/dist/report/audit_answer.js +2 -2
- package/dist/report/audit_answer.js.map +1 -1
- package/dist/report/audit_report.d.ts +1 -1
- package/dist/report/audit_report.js +2 -2
- package/dist/report/audit_report.js.map +1 -1
- package/dist/report/build_report.d.ts +18 -0
- package/dist/report/build_report.js +52 -7
- package/dist/report/build_report.js.map +1 -1
- package/dist/report/method_audit.d.ts +4 -3
- package/dist/report/method_audit.js +8 -5
- package/dist/report/method_audit.js.map +1 -1
- package/dist/report/tools.d.ts +31 -1
- package/dist/report/tools.js +101 -23
- package/dist/report/tools.js.map +1 -1
- package/dist/server.d.ts +2 -2
- package/dist/server.js +54 -41
- package/dist/server.js.map +1 -1
- package/dist/session.d.ts +8 -2
- package/dist/session.js +9 -0
- package/dist/session.js.map +1 -1
- package/dist/tools.d.ts +98 -6
- package/dist/tools.js +154 -17
- package/dist/tools.js.map +1 -1
- package/dist/workspace_tool.d.ts +47 -0
- package/dist/workspace_tool.js +94 -0
- package/dist/workspace_tool.js.map +1 -0
- package/guide/WORKFLOW.md +7 -4
- package/guide/references/compute.md +51 -0
- package/guide/references/report_quick.md +60 -0
- package/package.json +1 -1
- package/packs/census/docs/microdata.md +77 -0
- package/packs/census/docs/timeseries.md +71 -0
- package/packs/census/source.md +10 -140
- package/packs/congress/docs/nominations_crs.md +26 -0
- package/packs/congress/docs/votes.md +30 -0
- package/packs/congress/source.md +4 -48
- package/packs/govinfo/docs/collections.md +88 -0
- package/packs/govinfo/source.md +6 -86
- package/packs/irs-soi/docs/eo.md +15 -0
- package/packs/irs-soi/docs/tables.md +50 -0
- package/packs/irs-soi/source.md +6 -57
package/packs/congress/source.md
CHANGED
|
@@ -93,47 +93,14 @@ confirm each bill with `bill_titles` (`for_each` on congress, billType, billNumb
|
|
|
93
93
|
(verified 2026-10-22): those need one `bill` call per law, a second spec with
|
|
94
94
|
`"for_each": {"csv": "data/congress/<laws>.csv", "map": {"billType": "type", "billNumber": "number"},
|
|
95
95
|
"lower": true}` (bill numbers repeat across types; one call per row of the laws CSV). Counting "bills passed" needs a chamber action, not `latestAction` alone.
|
|
96
|
-
- Votes
|
|
97
|
-
|
|
98
|
-
recorded vote that cleared the bill ("On Passage", "On Passage of the Bill", "On Motion to Concur ...",
|
|
99
|
-
"On Agreeing to the Conference Report"): take the latest date, not the first "On Passage". A bill used
|
|
100
|
-
as a vehicle for other text (CHIPS and Science moved as H.R. 4346, first passed as a legislative-branch
|
|
101
|
-
appropriation) has an early "On Passage" vote on the old text; the Clerk's `description` can keep the
|
|
102
|
-
old title. Voice votes and unanimous consent leave no roll call and no row.
|
|
96
|
+
- Votes (roll calls, party totals, member ids, whole-session counts, `house_votes`): read
|
|
97
|
+
`source_brief({source: "congress", doc: "votes"})` before counting votes.
|
|
103
98
|
- Zero rows can be the right answer (no private laws in the 118th Congress, no votes on a bill): mark
|
|
104
99
|
such requests `"allow_empty": true`, or the fetch stops and names them.
|
|
105
|
-
- Party totals: House rows take the Clerk's totals by party; Senate rows are counted from the member
|
|
106
|
-
list (`yea_D`, `nay_R`, ...; independents as I). `not_voting` counts absent members; vacant seats are
|
|
107
|
-
not counted anywhere, so totals can be under 435 or 100. The Vice President's tie-breaking vote is in
|
|
108
|
-
`tie_breaker`, not in the member rows.
|
|
109
|
-
- Member ids differ by source: House rows and Congress.gov use bioguide ids (A000370); Senate XML uses
|
|
110
|
-
LIS ids (S354). Join Senate votes to members by name and state, or a crosswalk, not by id.
|
|
111
|
-
- `vote` keeps the published word (Yea, Aye, Nay, No, Present, Not Voting, Guilty); `position` maps it
|
|
112
|
-
to yea, nay, present, not_voting or other. House "Aye"/"No" are recorded votes, same meaning.
|
|
113
100
|
- Cosponsors: `sponsorshipWithdrawnDate` marks withdrawals (still listed);
|
|
114
101
|
`cosponsors.countIncludingWithdrawnCosponsors` in `bill` counts them. Count by `party` for a party split.
|
|
115
|
-
-
|
|
116
|
-
|
|
117
|
-
itself is a PDF or web page (committee reports in govinfo CRPT often reprint it as text).
|
|
118
|
-
- CRS: `crs_reports` lists reports Congress.gov publishes (newest first, about 14,000 in Oct 2026);
|
|
119
|
-
there is no keyword search: use a date range and `title_contains`, then `crs_report` for the summary.
|
|
120
|
-
`fromDateTime`/`toDateTime` filter on the UPDATE date (one September window returned 1,244 products
|
|
121
|
-
published 2004-2026): for "published in a month" filter the rows on `publishDate` afterwards. `publishDate`
|
|
122
|
-
is the current version's date, so a product first issued in the month and revised later drops out (a
|
|
123
|
-
floor, say so). The list's `version` is not the product version: `crs_report` gives `currentVersion`.
|
|
124
|
-
`bill_summaries` are CRS summaries of each bill version (HTML in `text`, plain in `text_plain`); some fields
|
|
125
|
-
are over 131,072 characters: when reading the CSV with your own code, raise the reader's field size limit first.
|
|
126
|
-
- Nominations (traps seen in Oct 2026 data; the Senate Résumé's "Civilian nominees" table is the benchmark):
|
|
127
|
-
military and "Other Civilian" lists (Foreign Service, PHS, NOAA officers) are list nominations without a
|
|
128
|
-
description; `nominationType` (`isCivilian`, `isMilitary`) is missing on some rows of older Congresses
|
|
129
|
-
(every row of the 119th has one); a PN split into parts
|
|
130
|
-
(PN12-41) counts once per part; en bloc votes confirm many nominees at once. `latestAction` is the last
|
|
131
|
-
step, which can be a later procedural motion ("motion to reconsider tabled") or, for a nomination held over,
|
|
132
|
-
its confirmation in the next session: count outcomes by date from `nomination_actions`, within the
|
|
133
|
-
session's dates. A split PN's actions need `nominationNumber` "1255-17": `"for_each": {"csv":
|
|
134
|
-
"data/congress/<noms>.csv", "column": "citation", "param": "nominationNumber", "strip": "PN"}` (`number` alone
|
|
135
|
-
returns nothing for a split PN). The naive count (civilian flag + `latestAction` "Confirmed") gave 60 percent for 2023
|
|
136
|
-
against the Résumé's 47.
|
|
102
|
+
- Nominations, CRS reports and cost estimates: read `source_brief({source: "congress", doc:
|
|
103
|
+
"nominations_crs"})` before using them.
|
|
137
104
|
- Paging: 250 per request (`limit`/`offset`). Lists default to the newest update first.
|
|
138
105
|
- Vetoes: a vetoed bill's `latestAction` often is a later procedural step, and no endpoint filters bills by
|
|
139
106
|
action. Read `bill_actions` (for_each over candidate bills that passed both chambers and are not law) for
|
|
@@ -141,17 +108,6 @@ confirm each bill with `bill_titles` (`for_each` on congress, billType, billNumb
|
|
|
141
108
|
President to the contrary notwithstanding"). The Senate's veto page (senate.gov/legislative/vetoes, regular
|
|
142
109
|
and pocket vetoes per President) is the benchmark. Reserved bill numbers (H.R. 1-10, S. 1-10 for leaders)
|
|
143
110
|
have no actions: allow_empty or drop them first.
|
|
144
|
-
- Whole-session vote counts and party splits: no endpoint returns party totals in bulk, so a session takes
|
|
145
|
-
one `roll_call` per vote (about 1,000 a year, several minutes). For the House's questions and results
|
|
146
|
-
(`house_votes` has no question column; `house_vote` keeps only party totals), `for_each` over the
|
|
147
|
-
`house_votes` list's `rollCallNumber` with `roll_call` `"summary": "true"` and the year: one row per vote with
|
|
148
|
-
question ("On Motion to Suspend the Rules and Pass"), vote type ("2/3 YEA-AND-NAY"), result and totals. `house_votes` lists quorum calls too (vote
|
|
149
|
-
type "Quorum"): the Résumé counts them apart from yea-and-nay and recorded votes. The Speaker election's
|
|
150
|
-
member rows have position `other` (votes are names). Say how independents are counted in a party split.
|
|
151
|
-
- `house_votes`/`house_vote` (Congress.gov) start with the 115th Congress (2017; the 114th lists 0 votes,
|
|
152
|
-
verified 2026-10-22); `roll_call` reads the Clerk
|
|
153
|
-
XML for 1990 on (and falls back to the Congress.gov member list for the 115th on if the Clerk file
|
|
154
|
-
cannot be read). Senate XML covers the 101st Congress (1989) on.
|
|
155
111
|
- Dates: Congress.gov timestamps are UTC (`2025-01-22T...Z`); vote `date` is the vote's own calendar
|
|
156
112
|
date (ISO), `date_published` keeps the source text.
|
|
157
113
|
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
# GovInfo collections: how each one is packaged and counted
|
|
2
|
+
|
|
3
|
+
From the govinfo playbook's "Data format and traps" section: read it before a request of this kind.
|
|
4
|
+
|
|
5
|
+
- Cost estimates: the budget office's own estimates are not a GovInfo collection. Two partial routes,
|
|
6
|
+
both text, not tables: (1) `collection: GOVPUB` with the publisher filter
|
|
7
|
+
(`publisher: "Congressional Budget Office"`, a report subset) returns the reports GPO has catalogued
|
|
8
|
+
(765 records in Oct 2026, far from the office's full list of publications);
|
|
9
|
+
(2) `collection: CRPT` with `phrase: "cost estimate"` finds committee reports, which usually reprint the
|
|
10
|
+
estimate for a reported bill in the report text. Congress.gov bill records (congress pack, endpoint
|
|
11
|
+
`bill_cbo_estimates`) list the titles and links of the estimates for a bill. No API here returns the
|
|
12
|
+
estimated dollar figures as data.
|
|
13
|
+
- CREC (daily Congressional Record): one package per issue date, but a two-book day can appear twice
|
|
14
|
+
(`CREC-2025-03-11` and `CREC-2025-03-11-i45`, same pages; `-i46` the second book): deduplicate by
|
|
15
|
+
`volume` + `issue` before counting issues or summing pages. Other suffixes are distinct issues, not
|
|
16
|
+
duplicates: `-v169`/`-v170` on January 3 (two volumes, the outgoing and incoming Congress) and a day with
|
|
17
|
+
`-i64`/`-i65`. Chamber sections are granules of class SENATE, HOUSE, EXTENSIONS and DAILYDIGEST (the
|
|
18
|
+
summary's `download.pdfSenateLink` / `pdfHouseLink` too). Early-January issues of a calendar year
|
|
19
|
+
belong to the outgoing Congress (volume and congress differ; `CREC-2015-01-02` is volume 160 but is
|
|
20
|
+
labelled congress 114), and an issue can be missing. Proceedings print under the legislative day, not the calendar day: a
|
|
21
|
+
Senate session that runs on (July 1, 2025, a continuation of June 30's legislative day) has no issue of its
|
|
22
|
+
own, so `granules` for that date returns no rows; find the issue with `search` (CREC, a phrase from the
|
|
23
|
+
proceedings) instead. `pages` counts PDF pages (Daily Digest and front
|
|
24
|
+
matter included), so it runs about 1 percent above the Résumé of Congressional Activity's numbered pages.
|
|
25
|
+
- "How many pages is the law": the convention is its Statutes at Large pages (first page to the last
|
|
26
|
+
`[[Page ... STAT. ...]]` marker of `text`, e.g. 139 Stat. 72-401 is 330 pages); the slip-law and enrolled-bill
|
|
27
|
+
PDFs differ by a page or so (cover, signatures). Say which measure the answer uses.
|
|
28
|
+
- CPD (Compilation of Presidential Documents, 1993 on; daily from 2009-01-20): one package per document,
|
|
29
|
+
docClass PRESDOC plus a few DIGEST, ACTSAPPROVED, CHECKLIST and NOMINATIONS rows (in a transition year the
|
|
30
|
+
outgoing President's supplements, `DCPD-2025DIGEST-BIDEN` and the like, are docClass PRESDOC too: count
|
|
31
|
+
numbered DCPD-YYYY0NNNN packages). The `published` list has no document type; each `package` summary has
|
|
32
|
+
`dcpdCategory` (level1 "Executive Orders", "Proclamations" ...), and `search` takes `collection:CPD AND
|
|
33
|
+
cpdcategory:proclamation` (signing remarks included). Some orders are missing or have no number in the
|
|
34
|
+
title, and `dateIssued` can be a day after signing. Count executive orders from the
|
|
35
|
+
federal-register pack (`executive_orders`, by signing date) and use CPD as the cross-check.
|
|
36
|
+
- CRPT (committee reports): one package per numbered report (its parts and errata inside it); GovInfo's
|
|
37
|
+
browse pages count documents (parts and errata), so they run above the report count. `published` CRPT also
|
|
38
|
+
lists GPO duplicate packages (GPO-J6-REPORT, GPO-CRPT-117hrpt646): keep `packageId` CRPT-*. A few
|
|
39
|
+
`dateIssued` values are wrong (three Dec-2022 reports dated 2021-12-31): count by Congress, not by date.
|
|
40
|
+
- CPRT (committee prints, "select prints": GovInfo is not a census of them): summaries carry no
|
|
41
|
+
`committees`; attribute a print to its committee by `suDocClassNumber` (the `Y 4.` stem, without the
|
|
42
|
+
`-NN` series suffix; a few prints have two classes or none). `published` with collection CPRT also returns
|
|
43
|
+
packages of other collections (GPO-PLUMBOOK-2024, collectionCode GPO): keep `collectionCode` CPRT.
|
|
44
|
+
GovInfo's browse pages count documents (errata included), not packages.
|
|
45
|
+
- STATUTE: one package per volume and volume = a session of Congress, not a calendar year. A law goes in
|
|
46
|
+
the volume of the session that passed it, not the year it was signed: laws signed in early January after
|
|
47
|
+
a session ends are in the closing session's volume (116-344, signed 2021-01-13, in vol. 134; 118-274,
|
|
48
|
+
2025-01-06, in vol. 138), and numbering can interleave (113-74, signed 2014-01-16, closes vol. 127 while
|
|
49
|
+
113-73 opens vol. 128). Assign laws to volumes from the granules, never by date. Granule `dateIssued` can
|
|
50
|
+
be wrong by a year (117-343's granule says 2022-01-05; signed 2023-01-05). Count laws by granules
|
|
51
|
+
with `granuleClass` PUBLICLAW (granules also hold proclamations and concurrent resolutions). Older
|
|
52
|
+
summaries (vol. 127) have no `pages`; `pages` counts front matter too. A granule id carries its start page
|
|
53
|
+
(`STATUTE-136-Pg6299`; a continuation adds a suffix, `STATUTE-135-Pg2531-2`). Each volume prints public
|
|
54
|
+
laws, then private laws, then concurrent resolutions and proclamations, with a few blank pages between the
|
|
55
|
+
parts. The public laws' pages run from the first law's start page to the **last law's last page**: the last
|
|
56
|
+
`[[Page N STAT. M]]` marker in that law's `text` (`packageId` PLAW-...). The next granule's start page
|
|
57
|
+
overstates it by 3 or 4 pages (6 more in vol. 136, where private laws come first); with the end from the
|
|
58
|
+
text, the 116th-118th Congresses give 8,432, 8,742 and 4,350 pages, the figures in Brookings' Vital
|
|
59
|
+
Statistics on Congress (Table 6-4).
|
|
60
|
+
- BUDGET: one package per volume (`BUDGET-2027-BUD` main volume, `-PER` Analytical Perspectives, `-APP`
|
|
61
|
+
Appendix). Summaries report collectionCode GPO (FY2017 and FY2027 alike) although `published` with
|
|
62
|
+
collection BUDGET lists them. Recent summaries' `pages` equals the PDF's page count; older ones (FY2017)
|
|
63
|
+
carry none: download `https://www.govinfo.gov/content/pkg/<packageId>/pdf/<packageId>.pdf`, count its pages
|
|
64
|
+
(pypdf) and record them in `published/`. A missing volume (FY2026 had no Analytical
|
|
65
|
+
Perspectives) is a 404: allow_empty. OMB posts supplemental chapters outside the volumes.
|
|
66
|
+
- USCOURTS (federal court opinions): one package per case, one granule per opinion; `search` returns one row
|
|
67
|
+
per opinion (`collection:USCOURTS AND courttype:appellate`, `courtname:fifth`; `ingestdate:range(...)`
|
|
68
|
+
filters on the date added to GovInfo, not the opinion date, and 2025 opinions keep arriving in 2026). A year of appellate opinions (about 15,000) passes the
|
|
69
|
+
10,000-row search cap: split by half-year. The court code is in the packageId (`USCOURTS-ca5-...`; the
|
|
70
|
+
Federal Circuit is ca13). Courts choose what they send (the Ninth Circuit sends precedential opinions only),
|
|
71
|
+
so a ranking reflects submission practice, not caseload: say so; the AO's Judicial Business tables (B-12)
|
|
72
|
+
count opinions by circuit.
|
|
73
|
+
- CFR: one package per title, edition and volume (`CFR-2025-title40-vol17`). `dateIssued` is the
|
|
74
|
+
"revised as of" date (January 1 for titles 1-16, April 1 for 17-27, July 1 for 28-41, October 1 for
|
|
75
|
+
42-50), not the posting date, which lags by months: a recent
|
|
76
|
+
edition can be incomplete, with volumes missing (404) while `volumeCount` counts them. `published`
|
|
77
|
+
rows carry no page count, so a title's pages take one `package` call per volume (`pages`, every PDF page
|
|
78
|
+
including front matter); give missing volumes `"allow_empty": true` and say they are missing, or fill
|
|
79
|
+
them from the prior edition as a labeled sensitivity case. Some volumes are covers only (`coveronly`).
|
|
80
|
+
- Bill to law: `related_collection` with `accessId` BILLS-...enr and `collection` PLAW gives the law's
|
|
81
|
+
package, citation and date (the plain `related` lists only the collections). An enrolled version's
|
|
82
|
+
`dateIssued` is GPO's print date, not the enrollment date: it can fall after the signing (BILLS-119hr1enr
|
|
83
|
+
is dated 2025-07-09, signed 2025-07-04).
|
|
84
|
+
- PLAW holds private laws too: `published` rows carry `docClass` PUBLIC or PRIVATE (packageId `publ` /
|
|
85
|
+
`pvtl`); 2022 had 247 public and 2 private. Count public laws on `docClass == "PUBLIC"`. For PLAW,
|
|
86
|
+
`dateIssued` is usually the approval date, but GovInfo has errors (PLAW-119publ76 is dated 2025-02-06; it
|
|
87
|
+
was approved 2026-02-06): for counts by date, check the Office of the Federal Register's list or the
|
|
88
|
+
congress pack's `laws`.
|
package/packs/govinfo/source.md
CHANGED
|
@@ -9,7 +9,7 @@ agency publications catalogued by GPO.
|
|
|
9
9
|
Code sections it amends; public laws enacted in a date range; committee reports, hearings and Congressional Record issues on a topic; agency reports in GOVPUB.
|
|
10
10
|
- Not here: bill status, sponsors, cosponsors, actions and votes (congress); rule metadata and counts
|
|
11
11
|
(federal-register); public comments (regulations). Cost estimates as numbers are not available from
|
|
12
|
-
any API here (
|
|
12
|
+
any API here (doc "collections").
|
|
13
13
|
|
|
14
14
|
## Endpoints and datasets
|
|
15
15
|
| Endpoint | Returns (one row per) | Key parameters |
|
|
@@ -99,17 +99,13 @@ chamber), enr enrolled (passed both).
|
|
|
99
99
|
`committee_meetings`. A committee's hearings are best selected by its SuDoc class
|
|
100
100
|
(`query: 'collection:CHRG AND congress:118 AND sudocclass:"y 4.j 89/2:"'`, Senate Judiciary), which also
|
|
101
101
|
leaves out joint hearings another committee printed.
|
|
102
|
-
- `dateIssued` is otherwise the publication date; for GOVPUB catalog records it can be a cataloguing date that
|
|
102
|
+
- `dateIssued` is otherwise the publication date; for GOVPUB catalog records it can be a cataloguing date that
|
|
103
|
+
is
|
|
103
104
|
later than the document itself (seen in Oct 2026: a record ingested in 2024 shows dateIssued 2026-06-27).
|
|
104
105
|
Check the PDF before dating a document from GOVPUB.
|
|
105
|
-
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
(765 records in Oct 2026, far from the office's full list of publications);
|
|
109
|
-
(2) `collection: CRPT` with `phrase: "cost estimate"` finds committee reports, which usually reprint the
|
|
110
|
-
estimate for a reported bill in the report text. Congress.gov bill records (congress pack, endpoint
|
|
111
|
-
`bill_cbo_estimates`) list the titles and links of the estimates for a bill. No API here returns the
|
|
112
|
-
estimated dollar figures as data.
|
|
106
|
+
- Collection-specific traps (CREC, CPD, CRPT, CPRT, STATUTE, BUDGET, USCOURTS, CFR, PLAW, bill to law, page
|
|
107
|
+
counts, cost estimates): read `source_brief({source: "govinfo", doc: "collections"})` before counting or
|
|
108
|
+
reading a collection.
|
|
113
109
|
- `text` returns the GPO plain-text rendition (`[[Page 138 STAT. 2687]]` page markers kept). Large
|
|
114
110
|
packages (an omnibus law) give a text column of several MB; set `max_chars` to keep a sample. Some
|
|
115
111
|
packages have only PDF: the request then fails with "no text rendition". Committee reports and hearings
|
|
@@ -145,82 +141,6 @@ chamber), enr enrolled (passed both).
|
|
|
145
141
|
Issue 45") are in `package` summaries: fetch the `published` list,
|
|
146
142
|
then one `package` request per row with `"for_each": {"csv": "data/govinfo/<list>.csv", "column":
|
|
147
143
|
"packageId"}`.
|
|
148
|
-
- CREC (daily Congressional Record): one package per issue date, but a two-book day can appear twice
|
|
149
|
-
(`CREC-2025-03-11` and `CREC-2025-03-11-i45`, same pages; `-i46` the second book): deduplicate by
|
|
150
|
-
`volume` + `issue` before counting issues or summing pages. Other suffixes are distinct issues, not
|
|
151
|
-
duplicates: `-v169`/`-v170` on January 3 (two volumes, the outgoing and incoming Congress) and a day with
|
|
152
|
-
`-i64`/`-i65`. Chamber sections are granules of class SENATE, HOUSE, EXTENSIONS and DAILYDIGEST (the
|
|
153
|
-
summary's `download.pdfSenateLink` / `pdfHouseLink` too). Early-January issues of a calendar year
|
|
154
|
-
belong to the outgoing Congress (volume and congress differ; `CREC-2015-01-02` is volume 160 but is
|
|
155
|
-
labelled congress 114), and an issue can be missing. Proceedings print under the legislative day, not the calendar day: a
|
|
156
|
-
Senate session that runs on (July 1, 2025, a continuation of June 30's legislative day) has no issue of its
|
|
157
|
-
own, so `granules` for that date returns no rows; find the issue with `search` (CREC, a phrase from the
|
|
158
|
-
proceedings) instead. `pages` counts PDF pages (Daily Digest and front
|
|
159
|
-
matter included), so it runs about 1 percent above the Résumé of Congressional Activity's numbered pages.
|
|
160
|
-
- "How many pages is the law": the convention is its Statutes at Large pages (first page to the last
|
|
161
|
-
`[[Page ... STAT. ...]]` marker of `text`, e.g. 139 Stat. 72-401 is 330 pages); the slip-law and enrolled-bill
|
|
162
|
-
PDFs differ by a page or so (cover, signatures). Say which measure the answer uses.
|
|
163
|
-
- CPD (Compilation of Presidential Documents, 1993 on; daily from 2009-01-20): one package per document,
|
|
164
|
-
docClass PRESDOC plus a few DIGEST, ACTSAPPROVED, CHECKLIST and NOMINATIONS rows (in a transition year the
|
|
165
|
-
outgoing President's supplements, `DCPD-2025DIGEST-BIDEN` and the like, are docClass PRESDOC too: count
|
|
166
|
-
numbered DCPD-YYYY0NNNN packages). The `published` list has no document type; each `package` summary has
|
|
167
|
-
`dcpdCategory` (level1 "Executive Orders", "Proclamations" ...), and `search` takes `collection:CPD AND
|
|
168
|
-
cpdcategory:proclamation` (signing remarks included). Some orders are missing or have no number in the
|
|
169
|
-
title, and `dateIssued` can be a day after signing. Count executive orders from the
|
|
170
|
-
federal-register pack (`executive_orders`, by signing date) and use CPD as the cross-check.
|
|
171
|
-
- CRPT (committee reports): one package per numbered report (its parts and errata inside it); GovInfo's
|
|
172
|
-
browse pages count documents (parts and errata), so they run above the report count. `published` CRPT also
|
|
173
|
-
lists GPO duplicate packages (GPO-J6-REPORT, GPO-CRPT-117hrpt646): keep `packageId` CRPT-*. A few
|
|
174
|
-
`dateIssued` values are wrong (three Dec-2022 reports dated 2021-12-31): count by Congress, not by date.
|
|
175
|
-
- CPRT (committee prints, "select prints": GovInfo is not a census of them): summaries carry no
|
|
176
|
-
`committees`; attribute a print to its committee by `suDocClassNumber` (the `Y 4.` stem, without the
|
|
177
|
-
`-NN` series suffix; a few prints have two classes or none). `published` with collection CPRT also returns
|
|
178
|
-
packages of other collections (GPO-PLUMBOOK-2024, collectionCode GPO): keep `collectionCode` CPRT.
|
|
179
|
-
GovInfo's browse pages count documents (errata included), not packages.
|
|
180
|
-
- STATUTE: one package per volume and volume = a session of Congress, not a calendar year. A law goes in
|
|
181
|
-
the volume of the session that passed it, not the year it was signed: laws signed in early January after
|
|
182
|
-
a session ends are in the closing session's volume (116-344, signed 2021-01-13, in vol. 134; 118-274,
|
|
183
|
-
2025-01-06, in vol. 138), and numbering can interleave (113-74, signed 2014-01-16, closes vol. 127 while
|
|
184
|
-
113-73 opens vol. 128). Assign laws to volumes from the granules, never by date. Granule `dateIssued` can
|
|
185
|
-
be wrong by a year (117-343's granule says 2022-01-05; signed 2023-01-05). Count laws by granules
|
|
186
|
-
with `granuleClass` PUBLICLAW (granules also hold proclamations and concurrent resolutions). Older
|
|
187
|
-
summaries (vol. 127) have no `pages`; `pages` counts front matter too. A granule id carries its start page
|
|
188
|
-
(`STATUTE-136-Pg6299`; a continuation adds a suffix, `STATUTE-135-Pg2531-2`). Each volume prints public
|
|
189
|
-
laws, then private laws, then concurrent resolutions and proclamations, with a few blank pages between the
|
|
190
|
-
parts. The public laws' pages run from the first law's start page to the **last law's last page**: the last
|
|
191
|
-
`[[Page N STAT. M]]` marker in that law's `text` (`packageId` PLAW-...). The next granule's start page
|
|
192
|
-
overstates it by 3 or 4 pages (6 more in vol. 136, where private laws come first); with the end from the
|
|
193
|
-
text, the 116th-118th Congresses give 8,432, 8,742 and 4,350 pages, the figures in Brookings' Vital
|
|
194
|
-
Statistics on Congress (Table 6-4).
|
|
195
|
-
- BUDGET: one package per volume (`BUDGET-2027-BUD` main volume, `-PER` Analytical Perspectives, `-APP`
|
|
196
|
-
Appendix). Summaries report collectionCode GPO (FY2017 and FY2027 alike) although `published` with
|
|
197
|
-
collection BUDGET lists them. Recent summaries' `pages` equals the PDF's page count; older ones (FY2017)
|
|
198
|
-
carry none: download `https://www.govinfo.gov/content/pkg/<packageId>/pdf/<packageId>.pdf`, count its pages
|
|
199
|
-
(pypdf) and record them in `published/`. A missing volume (FY2026 had no Analytical
|
|
200
|
-
Perspectives) is a 404: allow_empty. OMB posts supplemental chapters outside the volumes.
|
|
201
|
-
- USCOURTS (federal court opinions): one package per case, one granule per opinion; `search` returns one row
|
|
202
|
-
per opinion (`collection:USCOURTS AND courttype:appellate`, `courtname:fifth`; `ingestdate:range(...)`
|
|
203
|
-
filters on the date added to GovInfo, not the opinion date, and 2025 opinions keep arriving in 2026). A year of appellate opinions (about 15,000) passes the
|
|
204
|
-
10,000-row search cap: split by half-year. The court code is in the packageId (`USCOURTS-ca5-...`; the
|
|
205
|
-
Federal Circuit is ca13). Courts choose what they send (the Ninth Circuit sends precedential opinions only),
|
|
206
|
-
so a ranking reflects submission practice, not caseload: say so; the AO's Judicial Business tables (B-12)
|
|
207
|
-
count opinions by circuit.
|
|
208
|
-
- CFR: one package per title, edition and volume (`CFR-2025-title40-vol17`). `dateIssued` is the
|
|
209
|
-
"revised as of" date (January 1 for titles 1-16, April 1 for 17-27, July 1 for 28-41, October 1 for
|
|
210
|
-
42-50), not the posting date, which lags by months: a recent
|
|
211
|
-
edition can be incomplete, with volumes missing (404) while `volumeCount` counts them. `published`
|
|
212
|
-
rows carry no page count, so a title's pages take one `package` call per volume (`pages`, every PDF page
|
|
213
|
-
including front matter); give missing volumes `"allow_empty": true` and say they are missing, or fill
|
|
214
|
-
them from the prior edition as a labeled sensitivity case. Some volumes are covers only (`coveronly`).
|
|
215
|
-
- Bill to law: `related_collection` with `accessId` BILLS-...enr and `collection` PLAW gives the law's
|
|
216
|
-
package, citation and date (the plain `related` lists only the collections). An enrolled version's
|
|
217
|
-
`dateIssued` is GPO's print date, not the enrollment date: it can fall after the signing (BILLS-119hr1enr
|
|
218
|
-
is dated 2025-07-09, signed 2025-07-04).
|
|
219
|
-
- PLAW holds private laws too: `published` rows carry `docClass` PUBLIC or PRIVATE (packageId `publ` /
|
|
220
|
-
`pvtl`); 2022 had 247 public and 2 private. Count public laws on `docClass == "PUBLIC"`. For PLAW,
|
|
221
|
-
`dateIssued` is usually the approval date, but GovInfo has errors (PLAW-119publ76 is dated 2025-02-06; it
|
|
222
|
-
was approved 2026-02-06): for counts by date, check the Office of the Federal Register's list or the
|
|
223
|
-
congress pack's `laws`.
|
|
224
144
|
- `governmentAuthor` is a "; "-joined list. `collectionCode` can hold two codes (`SERIALSET;CRPT`).
|
|
225
145
|
- Paging: `search` uses GovInfo's `offsetMark` cursor (100 per page); `published` and `granules` follow
|
|
226
146
|
`nextPage` (1,000 per page).
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# IRS exempt organizations: eo_bmf and eo_extract
|
|
2
|
+
|
|
3
|
+
From the irs-soi playbook's "Exempt organizations (`eo_bmf`, `eo_extract`)" section.
|
|
4
|
+
|
|
5
|
+
- `eo_bmf`: `https://www.irs.gov/pub/irs-soi/eo_{st}.csv` (one per state, PR, XX = international; updated monthly,
|
|
6
|
+
1,975,975 organizations on 2026-10-13), 28 columns. Filters: `ntee` (prefix), `subsection` (`03` = 501(c)(3)),
|
|
7
|
+
`ein`, `name`, `columns`. EIN and ZIP stay text. ASSET_AMT, INCOME_AMT, REVENUE_AMT are whole dollars from the
|
|
8
|
+
latest return on file (TAX_PERIOD differs by organization). Codes: https://www.irs.gov/pub/foia/ig/tege/eo-info.pdf.
|
|
9
|
+
- `eo_extract`: the SOI annual extract of Form 990 (`form: 990`), 990-EZ (`990ez`) or 990-PF (`990pf`) data,
|
|
10
|
+
`{yy}eoextract990.zip` etc. (2018-2024; `{yy}eofinextract...` 2012-2017), listed on
|
|
11
|
+
soi-tax-stats-annual-extract-of-tax-exempt-organization-financial-data. The 2024 Form 990 zip is 57 MB holding a
|
|
12
|
+
247 MB CSV of 246 columns; it is read as a stream keeping only `ein` or the organizations of `state` (their EINs
|
|
13
|
+
from the current EO BMF, so an organization that moved or lost exemption is missed). The 990-PF CSV drops EIN
|
|
14
|
+
leading zeros: they are restored. Field definitions: `{yy}eofinextractdoc.xlsx`. Years whose zip holds a
|
|
15
|
+
fixed-width `.dat` file (some of 2012-2017) are refused.
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# IRS SOI published tables (table, table_list)
|
|
2
|
+
|
|
3
|
+
From the irs-soi playbook's "Published tables (`table`, `table_list`)" section.
|
|
4
|
+
|
|
5
|
+
SOI publishes its national statistics as .xls/.xlsx workbooks, one per table and year. `packs/irs-soi/tables.json`
|
|
6
|
+
catalogs 125 of them (checked 2026-11-08); `table_list` lists them (`product`, `query`; `live: true` adds the
|
|
7
|
+
years posted now). File names change over the years (the Data Book renumbered its tables in 2006, `97in11.xls`,
|
|
8
|
+
`ccr`/`nr`/`tabl` suffixes), so the adapter reads the table's index page and takes the year's link from it; a link
|
|
9
|
+
the page gets wrong is corrected in the catalog (`overrides`: partnership Tables 5 and 7 for 2023).
|
|
10
|
+
|
|
11
|
+
| Product (ids) | Tables | Years | Year |
|
|
12
|
+
|---|---|---|---|
|
|
13
|
+
| Individual, Publication 1304 (`in_1.1` ... `in_3.6`) | 1.1 income and tax by size of AGI, 1.2 by marital status, 1.4 sources of income, 1.4A capital gains, 1.6 age, 1.7 dependents, 2.1 itemized deductions, 2.5 EITC, 2.7 ACA, 3.1-3.6 tax computation, tax by rate (3.5: income taxed at each rate) | TY1996-2023 | tax year |
|
|
14
|
+
| Individual, percentiles (`shares_4.1`, `shares_4.2`, `shares_4.3`) | returns, AGI floor, AGI, income tax, average tax rate and shares, top 0.001% ... top 50% and bottom 50% | 2001-2023 in one file | edition |
|
|
15
|
+
| Corporations, Publication 16 (`corp_1` ... `corp_14`) | 1 minor industry, 2.1-2.4A size of total assets (2.3 C corporations, 2.4 S corporations), 3.x size of receipts, 4 size of tax, 5.x minor industry, 6-9 S corporations, 10 foreign (1120-F), 11 tax and credits, 12-14 COGS, depreciation, research credit; `corp_1-CV` coefficients of variation | TY2014-2023 (Table 1 from 1994) | tax year |
|
|
16
|
+
| Partnerships (`partnership_1` ... `_7`, `_10`, `_23`) | by industrial group: income and deductions, balance sheets, rental real estate, income allocated to partners, LLCs, receipts, foreign transactions | TY1993-2023 | tax year |
|
|
17
|
+
| Sole proprietorships (`soleprop_1` ... `_4`) | by industrial sector: receipts, deductions, payroll, net income; income statements; size of receipts; returns with Schedule C | TY1996-2023 | tax year |
|
|
18
|
+
| Integrated business data (`business_1`, `business_3`) | returns, receipts and net income by form of business, 1980-2015; by NAICS sector | to 2015 | edition / tax year |
|
|
19
|
+
| IRS Data Book (`databook_1-1` ... `_1-8`, `_3-1`, `_6-1`, `_6-2`) | collections and refunds by type of tax, returns filed, by state; gross collections by type of tax 1960 onward (1-6); audit coverage; costs; cost of collecting $100 | FY1995-2025 | fiscal year |
|
|
20
|
+
| Historical tables (`historical_1` ... `_24`) | individual series since 1913/1950, corporations since 1909, NIPA vs SOI income (8, 15), excise taxes by type FY1999 onward (20), returns filed (21), corporate rates 1909-2010 (24) | one file each | edition |
|
|
21
|
+
| Estate, gift, wealth (`estate_1` ... `_5`, `gift_1`, `wealth_1` ... `_6`) | estate returns by size of gross estate and tax status, by state, charitable bequests, beneficiaries, occupation; gift returns; top wealth holders | FY2001-2024; gift 2003-2021; wealth 2001-2019 | filing year |
|
|
22
|
+
| Exempt organizations (`eo_1` ... `_4`, `eo_excise`, `pf_1` ... `_4`) | Form 990 501(c)(3) by asset size, by Code section, 990-EZ, Form 4720 excise; private foundations | 1985-2024 | tax year |
|
|
23
|
+
| International (`cfc_1` ... `_3`, `ftc_1` ... `_3`, `cbc_1A` ... `cbc_4`) | controlled foreign corporations, corporate foreign tax credit, country-by-country reports | to 2021/2023 | tax year |
|
|
24
|
+
|
|
25
|
+
Parameters: `table` (an id, or a product word and number: "corporation 2.1", "data book 1-6"); `year` (or
|
|
26
|
+
`tax_year`, `fiscal_year`, `filing_year`; `start_year`/`end_year`, one workbook each); default the latest posted.
|
|
27
|
+
A whole-series table (`shares_4.x`, `databook_1-6`, `historical_*`, `business_1`) holds every year in one file:
|
|
28
|
+
the latest edition is read and the years are rows or columns (filter them with `rows`, `section` or `columns`).
|
|
29
|
+
`rows` keeps rows by label (an exact label, else the labels containing the text), `section` by section label,
|
|
30
|
+
`columns` by published column number or header text; `layout: "wide"` gives one row per line; `sheet` picks a
|
|
31
|
+
sheet of a multi-sheet workbook; `header_lines` overrides the header reading (rarely needed: see the notes).
|
|
32
|
+
|
|
33
|
+
How a workbook is read (no settings per table): title and headnote lines; the header lines down to the row of
|
|
34
|
+
column numbers (`1`, `(1)`, or `-1` shown as `(1)`); label-only lines become `SECTION` (a label running over two
|
|
35
|
+
lines is joined); the row level comes from the cell indent (or leading spaces) and gives `LEVEL` and `PARENT`;
|
|
36
|
+
footnote lines after the last data line go to the notes. A sheet printed as pages (1998 tables: columns 1-20 on
|
|
37
|
+
page 1, 21-40 on page 2) is read block by block; side-by-side panels (older partnership and sole proprietorship
|
|
38
|
+
tables) take each cell's label from the nearest label column to its left. A table without column numbers
|
|
39
|
+
(Publication 16 Table 2.1 for 2014, some historical tables) is read with the header down to the first line with
|
|
40
|
+
a label and a number, and says so in the notes, with `COL` empty.
|
|
41
|
+
|
|
42
|
+
Output (long, the default): `TABLE, YEAR, YEAR_TYPE, SHEET, SECTION, LINE, LEVEL, PARENT, ROW_LABEL, COL, COLUMN,
|
|
43
|
+
VALUE, FLAG, UNITS`. `COLUMN` joins the header lines with " | " ("All returns | Adjusted gross income less deficit
|
|
44
|
+
| Amount"); `COL` is the published column number, stable within a year (not across years: check `COLUMN`).
|
|
45
|
+
`VALUE` is the number as published (no thousands separators); `FLAG` holds the cell's markers: `*` estimate based
|
|
46
|
+
on few sample returns (high sampling variability; in newer workbooks the `*` is only in the number format and is
|
|
47
|
+
read from it), `**` data combined or suppressed, `d`/`[d]` deleted to avoid disclosure (included in totals), `[n]`
|
|
48
|
+
a footnote (e.g. `[2]` less than 0.05 percent), `n.a.` not available, `--`/`-` zero or not applicable as the
|
|
49
|
+
table's footnotes say. `UNITS` is the money unit of the headnote (usually thousands of dollars; gift tables whole
|
|
50
|
+
dollars; historical tables millions or billions); count, percent and average columns are as their header says.
|
package/packs/irs-soi/source.md
CHANGED
|
@@ -49,65 +49,14 @@ the index pages (ZIP `{yy}zpdoc.docx`, county `{yy}incydocguide.docx`, state `{y
|
|
|
49
49
|
migration `{y1y2}inpublicmigdoc.pdf`).
|
|
50
50
|
|
|
51
51
|
## Published tables (`table`, `table_list`)
|
|
52
|
-
SOI
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
the page gets wrong is corrected in the catalog (`overrides`: partnership Tables 5 and 7 for 2023).
|
|
57
|
-
|
|
58
|
-
| Product (ids) | Tables | Years | Year |
|
|
59
|
-
|---|---|---|---|
|
|
60
|
-
| Individual, Publication 1304 (`in_1.1` ... `in_3.6`) | 1.1 income and tax by size of AGI, 1.2 by marital status, 1.4 sources of income, 1.4A capital gains, 1.6 age, 1.7 dependents, 2.1 itemized deductions, 2.5 EITC, 2.7 ACA, 3.1-3.6 tax computation, tax by rate (3.5: income taxed at each rate) | TY1996-2023 | tax year |
|
|
61
|
-
| Individual, percentiles (`shares_4.1`, `shares_4.2`, `shares_4.3`) | returns, AGI floor, AGI, income tax, average tax rate and shares, top 0.001% ... top 50% and bottom 50% | 2001-2023 in one file | edition |
|
|
62
|
-
| Corporations, Publication 16 (`corp_1` ... `corp_14`) | 1 minor industry, 2.1-2.4A size of total assets (2.3 C corporations, 2.4 S corporations), 3.x size of receipts, 4 size of tax, 5.x minor industry, 6-9 S corporations, 10 foreign (1120-F), 11 tax and credits, 12-14 COGS, depreciation, research credit; `corp_1-CV` coefficients of variation | TY2014-2023 (Table 1 from 1994) | tax year |
|
|
63
|
-
| Partnerships (`partnership_1` ... `_7`, `_10`, `_23`) | by industrial group: income and deductions, balance sheets, rental real estate, income allocated to partners, LLCs, receipts, foreign transactions | TY1993-2023 | tax year |
|
|
64
|
-
| Sole proprietorships (`soleprop_1` ... `_4`) | by industrial sector: receipts, deductions, payroll, net income; income statements; size of receipts; returns with Schedule C | TY1996-2023 | tax year |
|
|
65
|
-
| Integrated business data (`business_1`, `business_3`) | returns, receipts and net income by form of business, 1980-2015; by NAICS sector | to 2015 | edition / tax year |
|
|
66
|
-
| IRS Data Book (`databook_1-1` ... `_1-8`, `_3-1`, `_6-1`, `_6-2`) | collections and refunds by type of tax, returns filed, by state; gross collections by type of tax 1960 onward (1-6); audit coverage; costs; cost of collecting $100 | FY1995-2025 | fiscal year |
|
|
67
|
-
| Historical tables (`historical_1` ... `_24`) | individual series since 1913/1950, corporations since 1909, NIPA vs SOI income (8, 15), excise taxes by type FY1999 onward (20), returns filed (21), corporate rates 1909-2010 (24) | one file each | edition |
|
|
68
|
-
| Estate, gift, wealth (`estate_1` ... `_5`, `gift_1`, `wealth_1` ... `_6`) | estate returns by size of gross estate and tax status, by state, charitable bequests, beneficiaries, occupation; gift returns; top wealth holders | FY2001-2024; gift 2003-2021; wealth 2001-2019 | filing year |
|
|
69
|
-
| Exempt organizations (`eo_1` ... `_4`, `eo_excise`, `pf_1` ... `_4`) | Form 990 501(c)(3) by asset size, by Code section, 990-EZ, Form 4720 excise; private foundations | 1985-2024 | tax year |
|
|
70
|
-
| International (`cfc_1` ... `_3`, `ftc_1` ... `_3`, `cbc_1A` ... `cbc_4`) | controlled foreign corporations, corporate foreign tax credit, country-by-country reports | to 2021/2023 | tax year |
|
|
71
|
-
|
|
72
|
-
Parameters: `table` (an id, or a product word and number: "corporation 2.1", "data book 1-6"); `year` (or
|
|
73
|
-
`tax_year`, `fiscal_year`, `filing_year`; `start_year`/`end_year`, one workbook each); default the latest posted.
|
|
74
|
-
A whole-series table (`shares_4.x`, `databook_1-6`, `historical_*`, `business_1`) holds every year in one file:
|
|
75
|
-
the latest edition is read and the years are rows or columns (filter them with `rows`, `section` or `columns`).
|
|
76
|
-
`rows` keeps rows by label (an exact label, else the labels containing the text), `section` by section label,
|
|
77
|
-
`columns` by published column number or header text; `layout: "wide"` gives one row per line; `sheet` picks a
|
|
78
|
-
sheet of a multi-sheet workbook; `header_lines` overrides the header reading (rarely needed: see the notes).
|
|
79
|
-
|
|
80
|
-
How a workbook is read (no settings per table): title and headnote lines; the header lines down to the row of
|
|
81
|
-
column numbers (`1`, `(1)`, or `-1` shown as `(1)`); label-only lines become `SECTION` (a label running over two
|
|
82
|
-
lines is joined); the row level comes from the cell indent (or leading spaces) and gives `LEVEL` and `PARENT`;
|
|
83
|
-
footnote lines after the last data line go to the notes. A sheet printed as pages (1998 tables: columns 1-20 on
|
|
84
|
-
page 1, 21-40 on page 2) is read block by block; side-by-side panels (older partnership and sole proprietorship
|
|
85
|
-
tables) take each cell's label from the nearest label column to its left. A table without column numbers
|
|
86
|
-
(Publication 16 Table 2.1 for 2014, some historical tables) is read with the header down to the first line with
|
|
87
|
-
a label and a number, and says so in the notes, with `COL` empty.
|
|
88
|
-
|
|
89
|
-
Output (long, the default): `TABLE, YEAR, YEAR_TYPE, SHEET, SECTION, LINE, LEVEL, PARENT, ROW_LABEL, COL, COLUMN,
|
|
90
|
-
VALUE, FLAG, UNITS`. `COLUMN` joins the header lines with " | " ("All returns | Adjusted gross income less deficit
|
|
91
|
-
| Amount"); `COL` is the published column number, stable within a year (not across years: check `COLUMN`).
|
|
92
|
-
`VALUE` is the number as published (no thousands separators); `FLAG` holds the cell's markers: `*` estimate based
|
|
93
|
-
on few sample returns (high sampling variability; in newer workbooks the `*` is only in the number format and is
|
|
94
|
-
read from it), `**` data combined or suppressed, `d`/`[d]` deleted to avoid disclosure (included in totals), `[n]`
|
|
95
|
-
a footnote (e.g. `[2]` less than 0.05 percent), `n.a.` not available, `--`/`-` zero or not applicable as the
|
|
96
|
-
table's footnotes say. `UNITS` is the money unit of the headnote (usually thousands of dollars; gift tables whole
|
|
97
|
-
dollars; historical tables millions or billions); count, percent and average columns are as their header says.
|
|
52
|
+
The SOI published tables (`table`, `table_list`: Publication 1304, tax shares by percentile, Publication 16,
|
|
53
|
+
partnerships, sole proprietorships, the Data Book, historical, estate, gift, wealth, exempt organizations,
|
|
54
|
+
international) and how their layouts are read: `source_brief({source: "irs-soi", doc: "tables"})`. Read it before
|
|
55
|
+
the first `table` request. Estimate recipes from them: doc `estimate_patterns`.
|
|
98
56
|
|
|
99
57
|
## Exempt organizations (`eo_bmf`, `eo_extract`)
|
|
100
|
-
- `eo_bmf
|
|
101
|
-
|
|
102
|
-
`ein`, `name`, `columns`. EIN and ZIP stay text. ASSET_AMT, INCOME_AMT, REVENUE_AMT are whole dollars from the
|
|
103
|
-
latest return on file (TAX_PERIOD differs by organization). Codes: https://www.irs.gov/pub/foia/ig/tege/eo-info.pdf.
|
|
104
|
-
- `eo_extract`: the SOI annual extract of Form 990 (`form: 990`), 990-EZ (`990ez`) or 990-PF (`990pf`) data,
|
|
105
|
-
`{yy}eoextract990.zip` etc. (2018-2024; `{yy}eofinextract...` 2012-2017), listed on
|
|
106
|
-
soi-tax-stats-annual-extract-of-tax-exempt-organization-financial-data. The 2024 Form 990 zip is 57 MB holding a
|
|
107
|
-
247 MB CSV of 246 columns; it is read as a stream keeping only `ein` or the organizations of `state` (their EINs
|
|
108
|
-
from the current EO BMF, so an organization that moved or lost exemption is missed). The 990-PF CSV drops EIN
|
|
109
|
-
leading zeros: they are restored. Field definitions: `{yy}eofinextractdoc.xlsx`. Years whose zip holds a
|
|
110
|
-
fixed-width `.dat` file (some of 2012-2017) are refused.
|
|
58
|
+
The exempt-organization registry (`eo_bmf`) and Form 990 extracts (`eo_extract`): `source_brief({source:
|
|
59
|
+
"irs-soi", doc: "eo"})`. Read it before the first request for either.
|
|
111
60
|
|
|
112
61
|
## Spec examples
|
|
113
62
|
```json
|