data-analyst-mcp 0.5.0 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,95 @@
1
+ # Estimate patterns for IRS SOI tables
2
+
3
+ Use this file when a question asks what a tax change would raise or cost, who pays a tax, how large a tax base
4
+ is, or how IRS collections are made up. The published tables give the base and the tax at today's law; an
5
+ estimate built on them is **static**: it holds incomes and behavior fixed. Say so in the answer, every time.
6
+
7
+ The method each recipe follows:
8
+ 1. Fetch the table (`table`), long by default; keep only the rows and columns the estimate needs (`rows`,
9
+ `columns`, `section`).
10
+ 2. Compute with `compute` on the fetched CSV (`filter`, `derive`, `groupby` + `rollup`, `pivot_wider`, `deflate`);
11
+ every number in the answer comes from a fetched table or a step recorded in the lineage.
12
+ 3. State the year type (tax, fiscal, filing), the units (`UNITS`: usually thousands of dollars), the flags of
13
+ the cells used (`FLAG`), and that the estimate is static.
14
+
15
+ Never invent a parameter. A behavioral response (an elasticity of taxable income, a realization response for
16
+ capital gains, avoidance) is an assumption: use it only when the user gives one or a cited source does, record it
17
+ with `record_decisions`, and show the static number beside the adjusted one.
18
+
19
+ ## Individual income tax
20
+
21
+ - **Revenue from a change in a bracket rate** (`in_3.5`, tax generated by rate). Column "Tax generated at
22
+ specified rate | <r> percent | Income taxed at rate" is the income taxed at that rate. A change of d
23
+ percentage points raises about income taxed at the rate x d / 100. Check: tax generated at the rate / income
24
+ taxed at the rate = the rate. Example: TY2023, 37 percent: 1,159,707,580 thousand dollars of income, 429,091,805
25
+ thousand of tax; one more point = about $11.6 billion a year, static, before credits interact. A change in a
26
+ bracket threshold needs the income near the threshold, which the table does not give: say so.
27
+ ```json
28
+ {"name": "rate37", "endpoint": "table", "params": {"table": "in_3.5", "year": 2023, "rows": ["Total"], "columns": ["37 percent"]}}
29
+ ```
30
+ `compute` steps: `{"filter": "COLUMN == \"Tax generated at specified rate | 37 percent | Income taxed at rate\""}`,
31
+ `{"derive": {"change_thousand_dollars": "VALUE * 0.01"}}`.
32
+ - **Average and effective tax rates by income** (`in_1.1`, `in_3.3`): total income tax / AGI by size of AGI,
33
+ from the same table and year. Table 1.1 has the percentages already in "Taxable returns | Total income tax |
34
+ Percent of | Adjusted gross income less deficit".
35
+ - **Who pays: shares of the top 1 percent and others** (`shares_4.1`, every year 2001 onward in one file):
36
+ sections "Adjusted gross income share", "Total income tax share", "Average tax rate", "Adjusted gross income
37
+ floor on percentiles" (current and constant dollars). Quote the share with its year and the percentile
38
+ definition (returns excluding dependents, positive AGI).
39
+ - **Credits and deductions** (`in_2.5` EITC, `in_2.1` itemized deductions, `in_1.4` sources of income and
40
+ adjustments): the cost of a credit change is about the change per claim x the number of claims in the affected
41
+ AGI classes, static. Itemized deductions are in Table 2.1 by type (SALT, mortgage interest, charitable).
42
+ - **Capital gains** (`in_1.4A`): gains and losses by size of AGI; rate changes on gains are where a realization
43
+ response matters most: never leave it out without saying so.
44
+ - **Over time**: one workbook per year (`start_year`/`end_year`); compare `COLUMN` text across years (column
45
+ numbers move); deflate with a price index fetched from bls or bea (`deflate`), and say which.
46
+
47
+ ## Business taxes
48
+
49
+ - **Corporate tax base and effective rate** (`corp_2.1`, `corp_1`, `corp_5.1`; C corporations alone:
50
+ `corp_2.3`, `corp_3.3`, `corp_5.3`): "Net income (less deficit)", "Income subject to tax", "Total income tax
51
+ before credits", "Total income tax after credits". Average effective rate = tax after credits / net income (or
52
+ / income subject to tax: name the denominator). Static revenue of a statutory rate change on C corporations
53
+ = income subject to tax x d / 100, before credits and the minimum tax; label it an upper bound.
54
+ - **By industry or size**: `corp_1` / `corp_5.x` rows are industries (`LEVEL` and `PARENT` give the hierarchy:
55
+ sum one level only); `corp_2.x` columns are asset-size classes. Cells marked `*` have high sampling
56
+ variability (the CVs are in `corp_1-CV`); `d` cells are deleted but counted in the totals.
57
+ - **Pass-through income** (`partnership_1`, `corp_2.4` / `corp_6.1` for S corporations, `soleprop_1`): net income
58
+ by industry. A change aimed at pass-through income (the qualified business income deduction, a surtax) starts
59
+ from these totals; the share that reaches individuals in each bracket is not in these tables (only microdata).
60
+ - **Corporate profits in the national accounts** differ from SOI net income (`historical_8`, `historical_15`
61
+ reconcile them); use bea for NIPA profits.
62
+
63
+ ## Collections and administration (IRS Data Book, fiscal years)
64
+
65
+ - **Collections by type of tax** (`databook_1-6`: gross collections 1960 onward in one file; `databook_1-1`: gross,
66
+ refunds and net for the last two years) and **by state** (`databook_1-5`, `databook_1-8` refunds). Gross
67
+ collections are cash received in the fiscal year, of any tax year; state is the filer's address (a business
68
+ pays where its headquarters files), not where income was earned: say so for state comparisons.
69
+ - **Excise taxes by type** (`historical_20`, fiscal years 1999 onward).
70
+ - **Enforcement**: audit coverage and recommended additional tax (`databook_3-1`); cost of collecting $100
71
+ (`databook_6-2`).
72
+
73
+ ## Estate and gift tax, wealth
74
+
75
+ - **Exemption or rate change** (`estate_1`, by size of gross estate and tax status): the number of taxable
76
+ returns, the taxable estate, the tentative tax and the net estate tax by size class. Raising the exemption
77
+ removes the returns below it and lowers the rest; the table gives the classes, not the returns: say how a class
78
+ was split, or keep to class boundaries. Filing years mix years of death.
79
+ - **By state** (`estate_2`), **charitable bequests** (`estate_3`), **gifts** (`gift_1`, whole dollars, not
80
+ thousands), **top wealth holders** (`wealth_1` ... `_6`, estimates from estate returns, millions of dollars).
81
+
82
+ ## Nonprofits
83
+
84
+ - **Sector size** (`eo_1`, `eo_3`, `pf_1`, `historical_16`): revenue, expenses and assets by asset size or Code
85
+ section, from Form 990 samples. **One organization or a state** (`eo_bmf` registry; `eo_extract` Form 990 data
86
+ by EIN or state): the registry's amounts come from each organization's latest return, of different years.
87
+ - The charitable deduction's cost is an individual-income-tax item (`in_2.1`, charitable contributions by size
88
+ of AGI), not a nonprofit table.
89
+
90
+ ## What to write
91
+
92
+ Lead with the number and its frame: "One more percentage point on the 37 percent rate would have raised about
93
+ $11.6 billion on tax year 2023 income, before any change in behavior (IRS SOI, Publication 1304 Table 3.5)."
94
+ Then the base, the year type, the units, the flags, and what the static estimate leaves out. Ship the fetched
95
+ tables and the computed exhibit; the citation names the table and year.
@@ -83,6 +83,78 @@
83
83
  "paging": {"type": "none"},
84
84
  "max_rows": 200000,
85
85
  "flatten": false
86
+ },
87
+ "table": {
88
+ "path": "/{file}",
89
+ "method": "GET",
90
+ "description": "One SOI published table (an .xls/.xlsx workbook on irs.gov) as tidy rows: national individual tables by size of AGI and by percentile (Publication 1304, tax shares of the top 1%), corporations (Publication 16), partnerships, sole proprietorships, integrated business data, the IRS Data Book (collections, refunds and returns by type of tax and state), historical tables, estate tax, gift tax, personal wealth, exempt organizations and private foundations, and international (CFC, foreign tax credit, country-by-country). `table` names a catalog entry (table_list lists all 125, e.g. in_1.1, in_3.5, shares_4.1, corp_1, corp_2.1, partnership_1, soleprop_1, databook_1-6, estate_1, eo_1). The adapter finds the year's file on the table's index page, downloads it and reads the layout: header lines, the column-number row, sections, row levels, footnote markers. Long layout (default): one row per cell, TABLE, YEAR, YEAR_TYPE, SHEET, SECTION, LINE, LEVEL, PARENT, ROW_LABEL, COL (the published column number), COLUMN (the header lines joined with ' | '), VALUE (as published), FLAG (marker: * high sampling variability, ** or d or [d] suppressed/combined, [n] footnote, n.a.), UNITS (the headnote's money unit, usually thousands of dollars).",
91
+ "params": {
92
+ "table": {"in": "spec", "required": true, "desc": "catalog id (see table_list), or a product word and table number such as 'corporation 2.1' or 'data book 1-6'"},
93
+ "year": {"in": "spec", "desc": "tax year (fiscal year for the Data Book, filing year for estate and gift tables); default the latest posted"},
94
+ "tax_year": {"in": "spec", "desc": "same as year"},
95
+ "fiscal_year": {"in": "spec", "desc": "same as year (Data Book)"},
96
+ "filing_year": {"in": "spec", "desc": "same as year (estate and gift)"},
97
+ "start_year": {"in": "spec", "desc": "first year of a range (one workbook each)"},
98
+ "end_year": {"in": "spec", "desc": "last year of a range"},
99
+ "rows": {"in": "spec", "desc": "keep rows whose label equals one of these texts (or, for an item equal to no label, contains it), e.g. ['$1,000,000 or more']"},
100
+ "columns": {"in": "spec", "desc": "keep columns by published column number (e.g. [1, 3]) or header text (e.g. ['Taxable income'])"},
101
+ "section": {"in": "spec", "desc": "keep rows under a section label containing this text (e.g. 'Accumulated from largest', 'Average tax rate')"},
102
+ "layout": {"in": "spec", "enum": ["long", "wide"], "desc": "long (default: one row per cell) or wide (one row per line, a column per table column)"},
103
+ "sheet": {"in": "spec", "desc": "sheet name or 1-based number for multi-sheet workbooks; default every sheet with a table"},
104
+ "header_lines": {"in": "spec", "desc": "1-based header line numbers, only for a workbook whose header the reader gets wrong (see the notes)"}
105
+ },
106
+ "records": "$adapter",
107
+ "paging": {"type": "none"},
108
+ "max_rows": 200000,
109
+ "flatten": false
110
+ },
111
+ "table_list": {
112
+ "path": "/",
113
+ "method": "GET",
114
+ "description": "The catalog of SOI published tables the table endpoint reads (packs/irs-soi/tables.json): TABLE id, PRODUCT, NUMBER, TITLE, YEAR_TYPE, FIRST_YEAR, LAST_YEAR_VERIFIED, WHOLE_SERIES (one file holds every year), INDEX (the page listing the files). live: true also reads each index page and lists YEARS_POSTED now.",
115
+ "params": {
116
+ "product": {"in": "spec", "desc": "keep products containing this text: individual, percentile, corporations, partnerships, sole, integrated, Data Book, historical, estate, gift, wealth, exempt, international"},
117
+ "query": {"in": "spec", "desc": "words to find in the table titles, e.g. 'tax shares percentile' or 'excise'"},
118
+ "live": {"in": "spec", "desc": "true: read the index pages and list the years posted now (one request per page)"}
119
+ },
120
+ "records": "$adapter",
121
+ "paging": {"type": "none"},
122
+ "max_rows": 1000,
123
+ "flatten": false
124
+ },
125
+ "eo_bmf": {
126
+ "path": "/eo_{state}.csv",
127
+ "method": "GET",
128
+ "description": "Exempt Organizations Business Master File extract: every organization recognized as tax-exempt, one CSV per state (updated monthly; 2.0 million organizations nationally), 28 columns: EIN, NAME, address, SUBSECTION (501(c) type), FOUNDATION, RULING, DEDUCTIBILITY, ACTIVITY, NTEE_CD, ASSET_CD, INCOME_CD, ASSET_AMT, INCOME_AMT, REVENUE_AMT (whole dollars, latest return on file), TAX_PERIOD and more. Codes in https://www.irs.gov/pub/foia/ig/tege/eo-info.pdf.",
129
+ "params": {
130
+ "state": {"in": "spec", "required": true, "desc": "state(s): USPS code or name, PR, or XX (international); one file each, 10 at most"},
131
+ "ntee": {"in": "spec", "desc": "NTEE code prefix(es), e.g. 'E' (health) or 'B82'"},
132
+ "subsection": {"in": "spec", "desc": "501(c) subsection(s), e.g. '03' for 501(c)(3)"},
133
+ "ein": {"in": "spec", "desc": "EIN(s), 9 digits (leading zeros kept)"},
134
+ "name": {"in": "spec", "desc": "text to find in NAME or SORT_NAME"},
135
+ "columns": {"in": "spec", "desc": "columns to keep beside EIN, NAME and STATE"}
136
+ },
137
+ "records": "$adapter",
138
+ "paging": {"type": "none"},
139
+ "max_rows": 200000,
140
+ "flatten": false
141
+ },
142
+ "eo_extract": {
143
+ "path": "/{yy}eoextract{form}.zip",
144
+ "method": "GET",
145
+ "description": "SOI annual extract of Form 990, 990-EZ or 990-PF financial data (every return filed in the year; 2012-2024): one zip per form and year (57 MB for the 2024 Form 990 file, a 247 MB CSV of 246 columns), read as a stream keeping only the asked EINs, or the organizations of the asked states (EINs from the current EO BMF). Amounts in whole dollars; field names and definitions in the year's {yy}eofinextractdoc.xlsx. Some 2012-2017 files are fixed-width .dat files and are refused.",
146
+ "params": {
147
+ "year": {"in": "spec", "required": false, "desc": "filing year of the extract, 2012-2024"},
148
+ "filing_year": {"in": "spec", "desc": "same as year"},
149
+ "form": {"in": "spec", "enum": ["990", "990ez", "990pf", "990EZ", "990PF", "990-EZ", "990-PF"], "desc": "990 (default), 990ez or 990pf"},
150
+ "ein": {"in": "spec", "desc": "EIN(s) to keep"},
151
+ "state": {"in": "spec", "desc": "keep the organizations of these states (EO BMF)"},
152
+ "columns": {"in": "spec", "desc": "columns to keep beside EIN (names as in the extract, e.g. totrevenue, totfuncexpns)"}
153
+ },
154
+ "records": "$adapter",
155
+ "paging": {"type": "none"},
156
+ "max_rows": 200000,
157
+ "flatten": false
86
158
  }
87
159
  },
88
160
  "catalog": {
@@ -109,6 +181,30 @@
109
181
  "name": "SOI Migration Data (state-to-state and county-to-county)",
110
182
  "endpoint": "migration",
111
183
  "description": "taxpayer migration inflow outflow between states and counties; households, people and income moving; net migration"
184
+ },
185
+ "tables": {
186
+ "id": "table",
187
+ "name": "SOI published tables (Publication 1304, Publication 16, Data Book, ...)",
188
+ "endpoint": "table",
189
+ "description": "national tax statistics tables: income and tax by AGI size, tax shares by percentile, corporate tax base, partnerships, sole proprietorships, IRS collections by type of tax and state, estate tax, nonprofits; budget and revenue estimates"
190
+ },
191
+ "table_list": {
192
+ "id": "table_list",
193
+ "name": "Catalog of SOI published tables",
194
+ "endpoint": "table_list",
195
+ "description": "list of SOI tables, titles and years"
196
+ },
197
+ "eo_bmf": {
198
+ "id": "eo_bmf",
199
+ "name": "Exempt Organizations Business Master File",
200
+ "endpoint": "eo_bmf",
201
+ "description": "nonprofit organizations registry by state: EIN, name, 501(c) type, NTEE, assets, revenue"
202
+ },
203
+ "eo_extract": {
204
+ "id": "eo_extract",
205
+ "name": "SOI Form 990 / 990-EZ / 990-PF annual extracts",
206
+ "endpoint": "eo_extract",
207
+ "description": "nonprofit financial data from Form 990 returns: revenue, expenses, assets by organization"
112
208
  }
113
209
  }
114
210
  }
@@ -1,13 +1,17 @@
1
1
  {
2
2
  "id": "irs-soi",
3
- "title": "IRS Statistics of Income: individual income tax data by ZIP code, county, state and migration (irs.gov)",
3
+ "title": "IRS Statistics of Income: income tax data by ZIP code, county, state and migration; national tax tables (Publication 1304, Publication 16, Data Book), business, estate, nonprofit and international statistics (irs.gov)",
4
4
  "agency": "Internal Revenue Service, Statistics of Income Division",
5
5
  "depth": "standard",
6
6
  "domains": [
7
7
  "taxes",
8
8
  "income",
9
9
  "migration",
10
- "geography"
10
+ "geography",
11
+ "budget",
12
+ "business",
13
+ "nonprofits",
14
+ "wealth"
11
15
  ],
12
16
  "keywords": [
13
17
  "irs",
@@ -37,7 +41,51 @@
37
41
  "inflow",
38
42
  "outflow",
39
43
  "taxpayer migration",
40
- "income moving between states"
44
+ "income moving between states",
45
+ "publication 1304",
46
+ "individual complete report",
47
+ "publication 16",
48
+ "corporation complete report",
49
+ "irs data book",
50
+ "gross collections",
51
+ "tax collections by state",
52
+ "irs collections",
53
+ "refunds",
54
+ "tax shares",
55
+ "top 1 percent",
56
+ "average tax rate",
57
+ "effective tax rate",
58
+ "tax generated by rate",
59
+ "income taxed at each rate",
60
+ "revenue estimate",
61
+ "budget estimate",
62
+ "static score",
63
+ "tax base",
64
+ "corporate income tax",
65
+ "corporate tax base",
66
+ "income subject to tax",
67
+ "corporations by industry",
68
+ "s corporations",
69
+ "partnerships",
70
+ "pass-through income",
71
+ "sole proprietorships",
72
+ "schedule c",
73
+ "business receipts",
74
+ "estate tax",
75
+ "gift tax",
76
+ "personal wealth",
77
+ "top wealth holders",
78
+ "excise taxes",
79
+ "historical tax tables",
80
+ "tax rates history",
81
+ "exempt organizations",
82
+ "nonprofits",
83
+ "form 990",
84
+ "private foundations",
85
+ "eo bmf",
86
+ "controlled foreign corporations",
87
+ "foreign tax credit",
88
+ "country-by-country"
41
89
  ],
42
90
  "questions": [
43
91
  "average adjusted gross income by ZIP code in Maryland",
@@ -46,7 +94,16 @@
46
94
  "earned income tax credit claims by county",
47
95
  "how many taxpayers moved from New York to Florida and how much income they took",
48
96
  "net migration of tax filers into Texas counties",
49
- "state and local tax deduction by ZIP code"
97
+ "state and local tax deduction by ZIP code",
98
+ "how much would raising the top income tax rate by one point raise",
99
+ "share of federal income tax paid by the top 1 percent",
100
+ "average tax rate by income group",
101
+ "corporate income tax after credits as a share of net income by industry",
102
+ "IRS gross collections by type of tax and by state",
103
+ "how many estates pay estate tax and how much",
104
+ "partnership and S corporation net income by industry",
105
+ "revenue, expenses and assets of 501(c)(3) organizations",
106
+ "nonprofit organizations in a state by NTEE category"
50
107
  ],
51
108
  "homepage": "https://www.irs.gov/statistics/soi-tax-stats-data-by-geographic-area",
52
109
  "terms_url": "https://www.irs.gov/statistics/soi-tax-stats-about-soi",
@@ -86,7 +143,7 @@
86
143
  "cache": {
87
144
  "mode": "run_only",
88
145
  "ttl_hours": null,
89
- "why": "Each request downloads one published all-states CSV (0.1 MB for state migration up to 216 MB for ZIP code data) and keeps only the asked rows; data/ is reused within a run only."
146
+ "why": "Each request downloads one published all-states CSV (0.1 MB for state migration up to 216 MB for ZIP code data) and keeps only the asked rows; data/ is reused within a run only. The published tables (20-300 KB workbooks) and their index pages are read once per run."
90
147
  },
91
148
  "redistribution": {
92
149
  "mode": "allowed",
@@ -99,9 +156,14 @@
99
156
  "clarify": [
100
157
  "Which tax year (ZIP, county and state data) or which pair of filing years (migration: 2022-2023 compares addresses on returns filed in 2022 and 2023)? Default: the latest posted (tax year 2023; migration 2022-2023).",
101
158
  "All returns combined, or broken out by AGI size class (agi_stub)? Averages such as AGI per return need the all-classes total (ZIP 00000 / county 000 rows or the no-AGI-class file).",
102
- "Migration: people moving into the place (inflow) or out of it (outflow), between states or between counties? Net migration needs both."
159
+ "Migration: people moving into the place (inflow) or out of it (outflow), between states or between counties? Net migration needs both.",
160
+ "Published tables: which table and year (table_list lists them; default the latest posted)? Tax year (Publication 1304, 16), fiscal year (Data Book) or filing year (estate)?",
161
+ "An estimate from these tables is static (no behavioral response): is that what is wanted, or is there a cited behavioral assumption to apply on top?"
103
162
  ],
104
163
  "formats": [
105
- "csv"
164
+ "csv",
165
+ "xls",
166
+ "xlsx",
167
+ "zip"
106
168
  ]
107
169
  }
@@ -1,6 +1,9 @@
1
- # IRS Statistics of Income: individual income tax data by geography (irs.gov)
2
- Depth: standard. Key: none. Counts and amounts from all individual income tax returns (Forms 1040), by ZIP code,
3
- county and state and by AGI size class, plus taxpayer migration between states and counties.
1
+ # IRS Statistics of Income: tax statistics by geography, national tables, business, estate and nonprofits (irs.gov)
2
+ Depth: standard. Key: none. Counts and amounts from tax returns: individual returns (Forms 1040) by ZIP code,
3
+ county and state and by AGI size class, taxpayer migration between states and counties, and the SOI published
4
+ tables (national individual tables and tax shares by percentile, corporations, partnerships, sole
5
+ proprietorships, the IRS Data Book, historical tables, estate and gift tax, personal wealth, exempt organizations,
6
+ international), plus the Exempt Organizations Business Master File and the Form 990 extracts.
4
7
 
5
8
  ## When to use
6
9
  - Number of returns, adjusted gross income (AGI), wages, dividends, capital gains, business income, deductions
@@ -8,12 +11,20 @@ county and state and by AGI size class, plus taxpayer migration between states a
8
11
  or state; average AGI per return; the income mix of a place; returns with AGI of $200,000 or more (ZIP,
9
12
  county) or $1 million or more (state).
10
13
  - Households (returns), people and AGI moving into or out of a state or county between two filing years.
14
+ - Budget and economic estimates from published tax data (`table`): national income and tax by size of AGI, income
15
+ taxed at each rate (static revenue effect of a rate change), average tax rates and tax shares of the top 1%,
16
+ 0.1% ..., the corporate tax base (net income, income subject to tax, tax after credits by industry or asset
17
+ size), pass-through business income (partnerships, S corporations, sole proprietorships), IRS gross collections
18
+ and refunds by type of tax and by state, estate and gift tax by size of estate and by state, wealth of top
19
+ wealth holders, the nonprofit sector (Form 990 totals, private foundations), controlled foreign corporations,
20
+ foreign tax credits and country-by-country reports. Recipes: `docs/estimate_patterns.md`.
21
+ - One nonprofit or the nonprofits of a state (`eo_bmf`: registry; `eo_extract`: Form 990 financial data).
11
22
  - Not here: person- or household-level income, poverty and median household income (census: ACS); personal
12
- income and GDP by state or county (bea); national tables such as Publication 1304 / Table 1.1 (SOI publishes
13
- them as .xls workbooks only: https://www.irs.gov/statistics/soi-tax-stats-individual-statistical-tables-by-size-of-adjusted-gross-income;
14
- manual); business, corporate, estate and exempt-organization statistics (other SOI pages, not covered);
15
- tax-return microdata: the Individual Public-Use Microdata File is not public (it was sold under license; tax
16
- years 2012-2015 are no longer available, and SOI has announced a synthetic file instead), so no row-level data.
23
+ income, GDP and corporate profits in the national accounts (bea); federal receipts by month and the budget
24
+ totals (treasury, omb); projections (CBO, not covered); tax-return microdata: the Individual Public-Use
25
+ Microdata File is not public (it was sold under license; tax years 2012-2015 are no longer available, and SOI
26
+ has announced a synthetic file instead), so no row-level data; Form 990 e-file XML (100-500 MB zips of single
27
+ returns); tax gap estimates (PDF only); the Corporation Source Book (frozen at 2013).
17
28
 
18
29
  ## Endpoints and datasets
19
30
  One all-states CSV per year under `https://www.irs.gov/pub/irs-soi/` (verified 2026-10-30). The adapter
@@ -37,8 +48,86 @@ not posted yet ends with `not_found` naming the file. Variable lists and layouts
37
48
  the index pages (ZIP `{yy}zpdoc.docx`, county `{yy}incydocguide.docx`, state `{yy}incmdocguide.doc`,
38
49
  migration `{y1y2}inpublicmigdoc.pdf`).
39
50
 
51
+ ## Published tables (`table`, `table_list`)
52
+ SOI publishes its national statistics as .xls/.xlsx workbooks, one per table and year. `packs/irs-soi/tables.json`
53
+ catalogs 125 of them (checked 2026-11-08); `table_list` lists them (`product`, `query`; `live: true` adds the
54
+ years posted now). File names change over the years (the Data Book renumbered its tables in 2006, `97in11.xls`,
55
+ `ccr`/`nr`/`tabl` suffixes), so the adapter reads the table's index page and takes the year's link from it; a link
56
+ the page gets wrong is corrected in the catalog (`overrides`: partnership Tables 5 and 7 for 2023).
57
+
58
+ | Product (ids) | Tables | Years | Year |
59
+ |---|---|---|---|
60
+ | Individual, Publication 1304 (`in_1.1` ... `in_3.6`) | 1.1 income and tax by size of AGI, 1.2 by marital status, 1.4 sources of income, 1.4A capital gains, 1.6 age, 1.7 dependents, 2.1 itemized deductions, 2.5 EITC, 2.7 ACA, 3.1-3.6 tax computation, tax by rate (3.5: income taxed at each rate) | TY1996-2023 | tax year |
61
+ | Individual, percentiles (`shares_4.1`, `shares_4.2`, `shares_4.3`) | returns, AGI floor, AGI, income tax, average tax rate and shares, top 0.001% ... top 50% and bottom 50% | 2001-2023 in one file | edition |
62
+ | Corporations, Publication 16 (`corp_1` ... `corp_14`) | 1 minor industry, 2.1-2.4A size of total assets (2.3 C corporations, 2.4 S corporations), 3.x size of receipts, 4 size of tax, 5.x minor industry, 6-9 S corporations, 10 foreign (1120-F), 11 tax and credits, 12-14 COGS, depreciation, research credit; `corp_1-CV` coefficients of variation | TY2014-2023 (Table 1 from 1994) | tax year |
63
+ | Partnerships (`partnership_1` ... `_7`, `_10`, `_23`) | by industrial group: income and deductions, balance sheets, rental real estate, income allocated to partners, LLCs, receipts, foreign transactions | TY1993-2023 | tax year |
64
+ | Sole proprietorships (`soleprop_1` ... `_4`) | by industrial sector: receipts, deductions, payroll, net income; income statements; size of receipts; returns with Schedule C | TY1996-2023 | tax year |
65
+ | Integrated business data (`business_1`, `business_3`) | returns, receipts and net income by form of business, 1980-2015; by NAICS sector | to 2015 | edition / tax year |
66
+ | IRS Data Book (`databook_1-1` ... `_1-8`, `_3-1`, `_6-1`, `_6-2`) | collections and refunds by type of tax, returns filed, by state; gross collections by type of tax 1960 onward (1-6); audit coverage; costs; cost of collecting $100 | FY1995-2025 | fiscal year |
67
+ | Historical tables (`historical_1` ... `_24`) | individual series since 1913/1950, corporations since 1909, NIPA vs SOI income (8, 15), excise taxes by type FY1999 onward (20), returns filed (21), corporate rates 1909-2010 (24) | one file each | edition |
68
+ | Estate, gift, wealth (`estate_1` ... `_5`, `gift_1`, `wealth_1` ... `_6`) | estate returns by size of gross estate and tax status, by state, charitable bequests, beneficiaries, occupation; gift returns; top wealth holders | FY2001-2024; gift 2003-2021; wealth 2001-2019 | filing year |
69
+ | Exempt organizations (`eo_1` ... `_4`, `eo_excise`, `pf_1` ... `_4`) | Form 990 501(c)(3) by asset size, by Code section, 990-EZ, Form 4720 excise; private foundations | 1985-2024 | tax year |
70
+ | International (`cfc_1` ... `_3`, `ftc_1` ... `_3`, `cbc_1A` ... `cbc_4`) | controlled foreign corporations, corporate foreign tax credit, country-by-country reports | to 2021/2023 | tax year |
71
+
72
+ Parameters: `table` (an id, or a product word and number: "corporation 2.1", "data book 1-6"); `year` (or
73
+ `tax_year`, `fiscal_year`, `filing_year`; `start_year`/`end_year`, one workbook each); default the latest posted.
74
+ A whole-series table (`shares_4.x`, `databook_1-6`, `historical_*`, `business_1`) holds every year in one file:
75
+ the latest edition is read and the years are rows or columns (filter them with `rows`, `section` or `columns`).
76
+ `rows` keeps rows by label (an exact label, else the labels containing the text), `section` by section label,
77
+ `columns` by published column number or header text; `layout: "wide"` gives one row per line; `sheet` picks a
78
+ sheet of a multi-sheet workbook; `header_lines` overrides the header reading (rarely needed: see the notes).
79
+
80
+ How a workbook is read (no settings per table): title and headnote lines; the header lines down to the row of
81
+ column numbers (`1`, `(1)`, or `-1` shown as `(1)`); label-only lines become `SECTION` (a label running over two
82
+ lines is joined); the row level comes from the cell indent (or leading spaces) and gives `LEVEL` and `PARENT`;
83
+ footnote lines after the last data line go to the notes. A sheet printed as pages (1998 tables: columns 1-20 on
84
+ page 1, 21-40 on page 2) is read block by block; side-by-side panels (older partnership and sole proprietorship
85
+ tables) take each cell's label from the nearest label column to its left. A table without column numbers
86
+ (Publication 16 Table 2.1 for 2014, some historical tables) is read with the header down to the first line with
87
+ a label and a number, and says so in the notes, with `COL` empty.
88
+
89
+ Output (long, the default): `TABLE, YEAR, YEAR_TYPE, SHEET, SECTION, LINE, LEVEL, PARENT, ROW_LABEL, COL, COLUMN,
90
+ VALUE, FLAG, UNITS`. `COLUMN` joins the header lines with " | " ("All returns | Adjusted gross income less deficit
91
+ | Amount"); `COL` is the published column number, stable within a year (not across years: check `COLUMN`).
92
+ `VALUE` is the number as published (no thousands separators); `FLAG` holds the cell's markers: `*` estimate based
93
+ on few sample returns (high sampling variability; in newer workbooks the `*` is only in the number format and is
94
+ read from it), `**` data combined or suppressed, `d`/`[d]` deleted to avoid disclosure (included in totals), `[n]`
95
+ a footnote (e.g. `[2]` less than 0.05 percent), `n.a.` not available, `--`/`-` zero or not applicable as the
96
+ table's footnotes say. `UNITS` is the money unit of the headnote (usually thousands of dollars; gift tables whole
97
+ dollars; historical tables millions or billions); count, percent and average columns are as their header says.
98
+
99
+ ## Exempt organizations (`eo_bmf`, `eo_extract`)
100
+ - `eo_bmf`: `https://www.irs.gov/pub/irs-soi/eo_{st}.csv` (one per state, PR, XX = international; updated monthly,
101
+ 1,975,975 organizations on 2026-10-13), 28 columns. Filters: `ntee` (prefix), `subsection` (`03` = 501(c)(3)),
102
+ `ein`, `name`, `columns`. EIN and ZIP stay text. ASSET_AMT, INCOME_AMT, REVENUE_AMT are whole dollars from the
103
+ latest return on file (TAX_PERIOD differs by organization). Codes: https://www.irs.gov/pub/foia/ig/tege/eo-info.pdf.
104
+ - `eo_extract`: the SOI annual extract of Form 990 (`form: 990`), 990-EZ (`990ez`) or 990-PF (`990pf`) data,
105
+ `{yy}eoextract990.zip` etc. (2018-2024; `{yy}eofinextract...` 2012-2017), listed on
106
+ soi-tax-stats-annual-extract-of-tax-exempt-organization-financial-data. The 2024 Form 990 zip is 57 MB holding a
107
+ 247 MB CSV of 246 columns; it is read as a stream keeping only `ein` or the organizations of `state` (their EINs
108
+ from the current EO BMF, so an organization that moved or lost exemption is missed). The 990-PF CSV drops EIN
109
+ leading zeros: they are restored. Field definitions: `{yy}eofinextractdoc.xlsx`. Years whose zip holds a
110
+ fixed-width `.dat` file (some of 2012-2017) are refused.
111
+
40
112
  ## Spec examples
41
113
  ```json
114
+ {"description": "Static revenue effect of one more point on the 37 percent rate, TY2023",
115
+ "requests": [{"name": "rate37", "endpoint": "table",
116
+ "params": {"table": "in_3.5", "year": 2023, "rows": ["Total"], "columns": ["37 percent"]}}]}
117
+ ```
118
+ ```json
119
+ {"description": "Tax shares and average tax rates of the top 1 percent, 2001-2023",
120
+ "requests": [{"name": "top1", "endpoint": "table",
121
+ "params": {"table": "shares_4.1", "columns": ["Top 1 percent"],
122
+ "section": ["Total income tax share", "Average tax rate", "Adjusted gross income share"]}}]}
123
+ ```
124
+ ```json
125
+ {"description": "Corporate tax base by asset size and IRS collections by type of tax",
126
+ "requests": [{"name": "corp", "endpoint": "table", "params": {"table": "corp_2.1", "year": 2023,
127
+ "rows": ["Net income (less deficit)", "Income subject to tax", "Total income tax after credits"]}},
128
+ {"name": "coll", "endpoint": "table", "params": {"table": "databook_1-6", "rows": ["2021", "2022", "2023", "2024", "2025"]}}]}
129
+ ```
130
+ ```json
42
131
  {"description": "Maryland counties, tax year 2022: returns and AGI by AGI class",
43
132
  "requests": [{"name": "md_county", "endpoint": "county_agi",
44
133
  "params": {"tax_year": 2022, "state": "MD", "columns": ["N1", "N2", "A00100", "A00200"]}}]}
@@ -117,8 +206,26 @@ Manual route (`manual_bundle`): INSTRUCTIONS.md lists the direct link of each fi
117
206
  e.g. `22incyallagi.csv`); download it from the index page (ZIP: soi-tax-stats-individual-income-tax-statistics-zip-code-data-soi,
118
207
  county: soi-tax-stats-county-data, state: soi-tax-stats-historic-table-2, migration: soi-tax-stats-migration-data,
119
208
  all under https://www.irs.gov/statistics/), upload it unchanged, and `ingest` filters it like `fetch` does.
120
- Per-state Excel files, years before the CSV files, and the national .xls tables are manual and not read here.
209
+ Per-state Excel files and years before the CSV files are manual and not read here.
210
+ `table`: one GET of the table's index page per run, then one GET per workbook (20-300 KB). Manual route: the index
211
+ page and the file pattern; upload the workbook under its published name (`23in11si.xls`), and `ingest` reads it
212
+ like `fetch` (the year from the name). `eo_bmf`: one GET per state file (up to about 25 MB for California);
213
+ `eo_extract`: one GET of the extract page, the state's EO BMF file, and the zip (10-57 MB, held in memory and
214
+ read as a stream). Manual: upload the zip under its published name (and `eo_<state>.csv` with `state`).
215
+
216
+ Traps for estimates from the published tables (see `docs/estimate_patterns.md`):
217
+ - Estimates from samples: totals are reliable; cells marked `*` are not; Publication 16 Table 1 has its
218
+ coefficients of variation in `corp_1-CV`.
219
+ - Tax year, filing year, fiscal year: Publication 1304 and Publication 16 are tax years; the Data Book is IRS
220
+ fiscal years (October-September) of collections; estate tables are filing years (returns filed in the year, of
221
+ decedents of earlier years). Do not mix them without saying so.
222
+ - Lag: the latest tax year is 2-3 years back (TY2023 posted in 2025-2026).
223
+ - Definitions change: AGI classes, the "total income tax" item, and table numbers change between years; compare
224
+ `COLUMN` text, not only `COL`, across years.
225
+ - Business tables count returns, not firms; the corporate tables include S corporations, REITs and RICs unless the
226
+ table is "other than Forms 1120S, 1120-REIT and 1120-RIC" (`corp_2.3`, `corp_3.3`, `corp_5.3`).
121
227
 
122
228
  ## Citation and terms
123
- "Internal Revenue Service, Statistics of Income Division, <product, e.g. County Data, Tax Year 2022>,
124
- irs.gov/statistics, retrieved <date>." Public domain (U.S. government work); redistribution allowed.
229
+ "Internal Revenue Service, Statistics of Income Division, <product, e.g. County Data, Tax Year 2022, or
230
+ Publication 1304 Table 1.1, Tax Year 2023>, irs.gov/statistics, retrieved <date>." The EO BMF: "Internal Revenue
231
+ Service, Exempt Organizations Business Master File Extract, <month year>". Public domain (U.S. government work); redistribution allowed.