data-analyst-mcp 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +286 -262
- package/dist/adapters/bjs.d.ts +13 -0
- package/dist/adapters/bjs.js +28 -0
- package/dist/adapters/bjs.js.map +1 -0
- package/dist/adapters/cbp.d.ts +5 -0
- package/dist/adapters/cbp.js +64 -0
- package/dist/adapters/cbp.js.map +1 -0
- package/dist/adapters/cdc-wonder.d.ts +5 -0
- package/dist/adapters/cdc-wonder.js +68 -0
- package/dist/adapters/cdc-wonder.js.map +1 -0
- package/dist/adapters/cpsc.d.ts +4 -0
- package/dist/adapters/cpsc.js +125 -0
- package/dist/adapters/cpsc.js.map +1 -0
- package/dist/adapters/fhwa.d.ts +9 -0
- package/dist/adapters/fhwa.js +16 -0
- package/dist/adapters/fhwa.js.map +1 -0
- package/dist/adapters/foreign-assistance.d.ts +9 -0
- package/dist/adapters/foreign-assistance.js +42 -0
- package/dist/adapters/foreign-assistance.js.map +1 -0
- package/dist/adapters/hud-ahar.d.ts +3 -0
- package/dist/adapters/hud-ahar.js +80 -0
- package/dist/adapters/hud-ahar.js.map +1 -0
- package/dist/adapters/nces.d.ts +10 -0
- package/dist/adapters/nces.js +32 -0
- package/dist/adapters/nces.js.map +1 -0
- package/dist/adapters/nsf.d.ts +11 -0
- package/dist/adapters/nsf.js +25 -0
- package/dist/adapters/nsf.js.map +1 -0
- package/dist/adapters/onrr.d.ts +9 -0
- package/dist/adapters/onrr.js +35 -0
- package/dist/adapters/onrr.js.map +1 -0
- package/dist/adapters/sba.d.ts +10 -0
- package/dist/adapters/sba.js +134 -0
- package/dist/adapters/sba.js.map +1 -0
- package/dist/adapters/treasury-tic.d.ts +30 -0
- package/dist/adapters/treasury-tic.js +149 -0
- package/dist/adapters/treasury-tic.js.map +1 -0
- package/dist/adapters/uscourts.d.ts +8 -0
- package/dist/adapters/uscourts.js +8 -0
- package/dist/adapters/uscourts.js.map +1 -0
- package/dist/analysis/cite.d.ts +9 -1
- package/dist/analysis/cite.js +9 -2
- package/dist/analysis/cite.js.map +1 -1
- package/dist/bin/cli.d.ts +2 -0
- package/dist/bin/cli.js +17 -0
- package/dist/bin/cli.js.map +1 -1
- package/dist/core/generic-rest.js +42 -17
- package/dist/core/generic-rest.js.map +1 -1
- package/dist/core/guard.d.ts +32 -0
- package/dist/core/guard.js +176 -0
- package/dist/core/guard.js.map +1 -0
- package/dist/core/http.d.ts +10 -0
- package/dist/core/http.js +36 -7
- package/dist/core/http.js.map +1 -1
- package/dist/core/keys.d.ts +13 -3
- package/dist/core/keys.js +67 -7
- package/dist/core/keys.js.map +1 -1
- package/dist/core/package.d.ts +12 -3
- package/dist/core/package.js +56 -12
- package/dist/core/package.js.map +1 -1
- package/dist/core/secret_patterns.d.ts +6 -0
- package/dist/core/secret_patterns.js +20 -0
- package/dist/core/secret_patterns.js.map +1 -0
- package/dist/core/tabular.d.ts +43 -1
- package/dist/core/tabular.js +149 -6
- package/dist/core/tabular.js.map +1 -1
- package/dist/envelope.js +3 -1
- package/dist/envelope.js.map +1 -1
- package/dist/index.d.ts +29 -22
- package/dist/index.js +45 -46
- package/dist/index.js.map +1 -1
- package/dist/remote/app.js +87 -31
- package/dist/remote/app.js.map +1 -1
- package/dist/remote/oauth.js +8 -1
- package/dist/remote/oauth.js.map +1 -1
- package/dist/remote/pages.d.ts +3 -7
- package/dist/remote/pages.js +49 -19
- package/dist/remote/pages.js.map +1 -1
- package/dist/remote/sign.d.ts +7 -1
- package/dist/remote/sign.js +33 -7
- package/dist/remote/sign.js.map +1 -1
- package/dist/remote/store.js +14 -5
- package/dist/remote/store.js.map +1 -1
- package/dist/report/audit_report.js +29 -7
- package/dist/report/audit_report.js.map +1 -1
- package/dist/report/build_report.js +34 -4
- package/dist/report/build_report.js.map +1 -1
- package/dist/report/method_audit.js +10 -1
- package/dist/report/method_audit.js.map +1 -1
- package/dist/report/scan_secrets.d.ts +4 -1
- package/dist/report/scan_secrets.js +38 -18
- package/dist/report/scan_secrets.js.map +1 -1
- package/dist/server.d.ts +3 -1
- package/dist/server.js +46 -13
- package/dist/server.js.map +1 -1
- package/dist/tools.js +27 -6
- package/dist/tools.js.map +1 -1
- package/guide/WORKFLOW.md +134 -134
- package/guide/assets/disclaimer.txt +1 -1
- package/guide/assets/download_data.ps1 +164 -162
- package/guide/assets/download_data.sh +8 -2
- package/guide/assets/report_template.html +597 -597
- package/guide/references/routing.md +198 -171
- package/guide/references/style_guide.md +398 -398
- package/package.json +56 -56
- package/packs/acf/endpoints.json +114 -114
- package/packs/acf/policy.json +104 -104
- package/packs/actuaries/endpoints.json +82 -82
- package/packs/actuaries/policy.json +109 -109
- package/packs/bea/assets/download_bea.ps1 +182 -182
- package/packs/bea/docs/bea_api.md +461 -461
- package/packs/bea/docs/datasets.md +157 -157
- package/packs/bea/docs/report_patterns.md +112 -112
- package/packs/bea/policy.json +170 -170
- package/packs/bea/source.md +152 -152
- package/packs/bjs/endpoints.json +150 -0
- package/packs/bjs/policy.json +114 -0
- package/packs/bjs/source.md +98 -0
- package/packs/bls/assets/download_bls.ps1 +152 -152
- package/packs/bls/docs/analysis_patterns.md +108 -108
- package/packs/bls/docs/bls_api.md +73 -73
- package/packs/bls/docs/series_ids.md +149 -149
- package/packs/bls/policy.json +238 -238
- package/packs/bts/endpoints.json +346 -346
- package/packs/bts/policy.json +94 -94
- package/packs/cbp/endpoints.json +31 -0
- package/packs/cbp/policy.json +85 -0
- package/packs/cbp/source.md +57 -0
- package/packs/cdc/endpoints.json +197 -197
- package/packs/cdc/policy.json +83 -83
- package/packs/cdc/source.md +1 -1
- package/packs/cdc-wonder/endpoints.json +26 -0
- package/packs/cdc-wonder/policy.json +84 -0
- package/packs/cdc-wonder/source.md +53 -0
- package/packs/census/endpoints.json +99 -99
- package/packs/cfpb/endpoints.json +326 -326
- package/packs/cfpb/policy.json +119 -119
- package/packs/clinical-trials/endpoints.json +166 -166
- package/packs/clinical-trials/policy.json +84 -84
- package/packs/cms/endpoints.json +69 -69
- package/packs/cms/policy.json +96 -96
- package/packs/college-scorecard/endpoints.json +252 -252
- package/packs/college-scorecard/policy.json +106 -106
- package/packs/college-scorecard/source.md +1 -1
- package/packs/congress/endpoints.json +1701 -1701
- package/packs/congress/policy.json +117 -117
- package/packs/cpsc/endpoints.json +91 -0
- package/packs/cpsc/policy.json +91 -0
- package/packs/cpsc/source.md +76 -0
- package/packs/dhs-ohss/endpoints.json +66 -66
- package/packs/dhs-ohss/policy.json +107 -107
- package/packs/dhs-ohss/source.md +1 -1
- package/packs/doj-news/endpoints.json +59 -59
- package/packs/doj-news/policy.json +96 -96
- package/packs/doj-news/source.md +1 -1
- package/packs/dol/endpoints.json +817 -817
- package/packs/dol/policy.json +135 -135
- package/packs/eia/endpoints.json +717 -717
- package/packs/eia/policy.json +112 -112
- package/packs/epa/endpoints.json +291 -291
- package/packs/epa/policy.json +99 -99
- package/packs/epa-aqs/endpoints.json +173 -173
- package/packs/epa-aqs/policy.json +96 -96
- package/packs/fbi/endpoints.json +1841 -1841
- package/packs/fbi/policy.json +118 -118
- package/packs/fbi/source.md +3 -3
- package/packs/fda/endpoints.json +237 -237
- package/packs/fda/policy.json +121 -121
- package/packs/fdic/endpoints.json +416 -416
- package/packs/fdic/policy.json +83 -83
- package/packs/fec/endpoints.json +707 -707
- package/packs/fec/policy.json +93 -93
- package/packs/fed-board/endpoints.json +89 -89
- package/packs/fed-board/policy.json +136 -136
- package/packs/federal-register/endpoints.json +143 -143
- package/packs/federal-register/policy.json +86 -86
- package/packs/fema/endpoints.json +645 -645
- package/packs/fema/policy.json +103 -103
- package/packs/fhfa/endpoints.json +105 -105
- package/packs/fhfa/policy.json +109 -109
- package/packs/fhwa/endpoints.json +48 -0
- package/packs/fhwa/policy.json +91 -0
- package/packs/fhwa/source.md +59 -0
- package/packs/foreign-assistance/endpoints.json +166 -0
- package/packs/foreign-assistance/policy.json +80 -0
- package/packs/foreign-assistance/source.md +76 -0
- package/packs/fred/assets/download_fred.ps1 +122 -122
- package/packs/fred/docs/analysis_patterns.md +124 -124
- package/packs/fred/docs/fred_api.md +94 -94
- package/packs/fred/docs/series.md +188 -188
- package/packs/fred/policy.json +253 -253
- package/packs/fsa/endpoints.json +150 -150
- package/packs/fsa/policy.json +99 -99
- package/packs/fsa/source.md +1 -1
- package/packs/govinfo/endpoints.json +519 -519
- package/packs/gsa-calc/endpoints.json +93 -93
- package/packs/gsa-calc/policy.json +97 -97
- package/packs/hud/endpoints.json +146 -146
- package/packs/hud/policy.json +84 -84
- package/packs/hud/source.md +1 -1
- package/packs/hud-ahar/endpoints.json +92 -0
- package/packs/hud-ahar/policy.json +88 -0
- package/packs/hud-ahar/source.md +72 -0
- package/packs/ipums/docs/report_patterns.md +117 -117
- package/packs/irs-soi/endpoints.json +114 -114
- package/packs/irs-soi/policy.json +107 -107
- package/packs/medicaid/endpoints.json +412 -412
- package/packs/medicaid/policy.json +108 -108
- package/packs/naep/endpoints.json +217 -217
- package/packs/naep/policy.json +100 -100
- package/packs/naep/source.md +1 -1
- package/packs/nces/endpoints.json +101 -0
- package/packs/nces/policy.json +99 -0
- package/packs/nces/source.md +75 -0
- package/packs/nhtsa/endpoints.json +132 -132
- package/packs/nhtsa/policy.json +93 -93
- package/packs/nhtsa/source.md +1 -1
- package/packs/nih/endpoints.json +452 -452
- package/packs/nih/policy.json +86 -86
- package/packs/noaa/endpoints.json +431 -431
- package/packs/noaa/policy.json +95 -95
- package/packs/nrel/endpoints.json +327 -327
- package/packs/nrel/policy.json +96 -96
- package/packs/nsf/endpoints.json +33 -0
- package/packs/nsf/policy.json +83 -0
- package/packs/nsf/source.md +59 -0
- package/packs/nws/endpoints.json +271 -271
- package/packs/nws/policy.json +82 -82
- package/packs/nyfed/endpoints.json +80 -80
- package/packs/nyfed/policy.json +92 -92
- package/packs/omb/endpoints.json +460 -460
- package/packs/omb/policy.json +106 -106
- package/packs/onrr/endpoints.json +198 -0
- package/packs/onrr/policy.json +91 -0
- package/packs/onrr/source.md +55 -0
- package/packs/open-payments/endpoints.json +297 -297
- package/packs/open-payments/policy.json +101 -101
- package/packs/opm/assets/download_opm.ps1 +125 -125
- package/packs/opm/docs/fields.md +184 -184
- package/packs/opm/docs/opm_api.md +71 -71
- package/packs/opm/docs/report_patterns.md +83 -83
- package/packs/opm/lib/data_dictionary.csv +73 -73
- package/packs/pbgc/endpoints.json +483 -483
- package/packs/pbgc/policy.json +94 -94
- package/packs/philfed/endpoints.json +85 -85
- package/packs/philfed/policy.json +94 -94
- package/packs/port.json +65 -0
- package/packs/regulations/endpoints.json +287 -287
- package/packs/regulations/policy.json +86 -86
- package/packs/sba/endpoints.json +34 -0
- package/packs/sba/policy.json +86 -0
- package/packs/sba/source.md +60 -0
- package/packs/sec/endpoints.json +236 -236
- package/packs/sec/policy.json +86 -86
- package/packs/senate-lobbying/endpoints.json +553 -553
- package/packs/senate-lobbying/policy.json +99 -99
- package/packs/snapqc/endpoints.json +1426 -1426
- package/packs/snapqc/policy.json +92 -92
- package/packs/treasury/catalog.json +2267 -2267
- package/packs/treasury/docs/api_guide.md +245 -245
- package/packs/treasury/docs/datasets.md +56 -56
- package/packs/treasury/docs/report_patterns.md +126 -126
- package/packs/treasury/policy.json +159 -159
- package/packs/treasury/source.md +170 -165
- package/packs/treasury-tic/endpoints.json +77 -0
- package/packs/treasury-tic/policy.json +89 -0
- package/packs/treasury-tic/source.md +80 -0
- package/packs/treasury-yields/endpoints.json +226 -226
- package/packs/treasury-yields/policy.json +86 -86
- package/packs/usaspending/endpoints.json +1315 -1315
- package/packs/usaspending/policy.json +109 -109
- package/packs/uscourts/endpoints.json +22 -0
- package/packs/uscourts/policy.json +87 -0
- package/packs/uscourts/source.md +56 -0
- package/packs/usda-fna/policy.json +133 -133
- package/packs/usda-fooddata/endpoints.json +180 -180
- package/packs/usda-fooddata/policy.json +94 -94
- package/packs/usda-nass/endpoints.json +323 -323
- package/packs/usda-nass/policy.json +95 -95
- package/packs/usda-outlook/endpoints.json +86 -86
- package/packs/usda-outlook/policy.json +109 -109
- package/packs/usgs/endpoints.json +313 -313
- package/packs/usgs/policy.json +95 -95
- package/packs/usitc/endpoints.json +74 -74
- package/packs/usitc/policy.json +111 -111
- package/packs/uspto/endpoints.json +141 -141
- package/packs/uspto/policy.json +95 -95
- package/packs/uspto/source.md +1 -1
- package/packs/va/endpoints.json +414 -414
- package/packs/va/policy.json +101 -101
- package/packs/world-bank/endpoints.json +122 -122
- package/packs/world-bank/policy.json +101 -101
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# ForeignAssistance.gov: U.S. foreign aid by country, sector and agency
|
|
2
|
+
Depth: standard. Key: none. The aggregate data files of ForeignAssistance.gov, the U.S. government's foreign
|
|
3
|
+
assistance reporting site (State Department and USAID). They give foreign aid by:
|
|
4
|
+
- recipient country or region;
|
|
5
|
+
- international (OECD DAC) and U.S. government sector;
|
|
6
|
+
- managing agency;
|
|
7
|
+
- funding agency and account.
|
|
8
|
+
Each file has transaction type (obligations, disbursements, appropriated and planned, President's budget
|
|
9
|
+
requests) and fiscal year, in current and constant dollars.
|
|
10
|
+
|
|
11
|
+
## When to use
|
|
12
|
+
- How much U.S. aid a country received; the largest recipients; aid by sector (health, peace and security,
|
|
13
|
+
humanitarian, economic development, democracy); which agencies manage it; trends since the 1940s (in constant
|
|
14
|
+
dollars).
|
|
15
|
+
- Not here: award-level federal grants and contracts with recipient names (usaspending); federal budget totals
|
|
16
|
+
by function (omb, treasury); other donors' aid (world-bank has some ODA indicators).
|
|
17
|
+
|
|
18
|
+
## Endpoints
|
|
19
|
+
All take the same filters (applied to the published file): `country` (ISO3 code like `UKR`, or the name),
|
|
20
|
+
`fiscal_year`, `from`/`to`, `transaction_type` (name or id).
|
|
21
|
+
|
|
22
|
+
| Endpoint | File | Columns before the amounts |
|
|
23
|
+
|---|---|---|
|
|
24
|
+
| `by_country` | us_foreign_aid_country.csv | Country Code, Country Name, Region ID, Region Name, Income Group Acronym, Income Group Name |
|
|
25
|
+
| `by_dac_sector` | us_foreign_aid_dac_sector.csv | International Category ID/Name, International Sector Code/Name (no country) |
|
|
26
|
+
| `by_usg_sector` | us_foreign_aid_usg_sector.csv | US Category ID/Name, US Sector ID/Name, Country Code/Name |
|
|
27
|
+
| `by_managing_agency` | us_foreign_aid_implementing.csv | Managing Agency ID/Acronym/Name, Country Code/Name |
|
|
28
|
+
| `by_funding_agency` | us_foreign_aid_funding.csv | Funding Agency ID/Acronym/Name, Funding Account ID/Name, Country Code/Name |
|
|
29
|
+
|
|
30
|
+
Every file then has `Transaction Type ID`, `Transaction Type Name`, `Fiscal Year`, `current_amount` and
|
|
31
|
+
`constant_amount`.
|
|
32
|
+
|
|
33
|
+
## Spec examples
|
|
34
|
+
```json
|
|
35
|
+
{"description": "U.S. aid obligations to Ukraine, FY2020-2025",
|
|
36
|
+
"requests": [{"name": "ukraine", "endpoint": "by_country",
|
|
37
|
+
"params": {"country": "UKR", "from": "2020", "to": "2025", "transaction_type": "Obligations"}}]}
|
|
38
|
+
```
|
|
39
|
+
```json
|
|
40
|
+
{"description": "Aid to Ukraine by managing agency, FY2023",
|
|
41
|
+
"requests": [{"name": "ukraine_agency_2023", "endpoint": "by_managing_agency",
|
|
42
|
+
"params": {"country": "UKR", "fiscal_year": "2023", "transaction_type": "Obligations"}}]}
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Data format and traps
|
|
46
|
+
- Transaction types are different measures; never add them:
|
|
47
|
+
- 2, Obligations: commitments made, FY1946 on.
|
|
48
|
+
- 3, Disbursements: money paid out, FY2001 on.
|
|
49
|
+
- 1, Appropriated and Planned: FY2004-2023.
|
|
50
|
+
- 18, President's Budget Requests: FY2005-2024.
|
|
51
|
+
(Ranges in the by-country file of 2026-10-30.)
|
|
52
|
+
- Obligations and disbursements in one year are not the same money: disbursements lag obligations.
|
|
53
|
+
- Dollars are whole dollars as text. `constant_amount` is inflation-adjusted and equals `current_amount` in the
|
|
54
|
+
latest complete year (FY2025 in the files of 2026-10-30), so it is in that year's dollars. Name the base year.
|
|
55
|
+
- Fiscal years run October to September. The newest year (2026 in those files) holds partial reporting or
|
|
56
|
+
budget figures only: do not compare it with full years.
|
|
57
|
+
- The country file includes regions and worldwide programs as rows (`WLD` World, `AFR` Africa Region, `ASN` Asia
|
|
58
|
+
Region ...). Former countries have the code `NULL` ("Soviet Union (former)"). Kosovo is `CS-KM`. Some names
|
|
59
|
+
contain line breaks. Drop regional rows before ranking countries, and say that regional and worldwide programs
|
|
60
|
+
are not attributed to countries.
|
|
61
|
+
- Sector and agency files split the same money by country and category: totals from different files agree
|
|
62
|
+
only roughly (late or unattributed reporting). Do not mix files in one sum.
|
|
63
|
+
- Security and military assistance appears where agencies report it (e.g. State and Defense programs): check
|
|
64
|
+
the managing-agency file before calling a total "civilian" or "military".
|
|
65
|
+
|
|
66
|
+
## Access
|
|
67
|
+
No key. Manual only: on 2026-11-07 foreignassistance.gov answered this server's requests with HTTP 403 (a plain
|
|
68
|
+
browser download works), and the site's Data API at /data-api/ answered 404, its documentation listing
|
|
69
|
+
authentication errors. So `fetch` refuses and `manual_bundle` gives the link. The user downloads the CSV
|
|
70
|
+
from https://foreignassistance.gov/data (Data Download; files up to 22 MB, e.g.
|
|
71
|
+
https://foreignassistance.gov/static/us_foreign_aid_country.csv) and uploads it, and `ingest` applies the
|
|
72
|
+
filters.
|
|
73
|
+
|
|
74
|
+
## Citation and terms
|
|
75
|
+
"U.S. Department of State and USAID, ForeignAssistance.gov, <file> (data last updated <date shown on the site>),
|
|
76
|
+
retrieved <date>." Open Data License (as stated on the site).
|
|
@@ -1,122 +1,122 @@
|
|
|
1
|
-
<#
|
|
2
|
-
.SYNOPSIS
|
|
3
|
-
Download the FRED files for one data-analyst request into the folder that holds this script.
|
|
4
|
-
|
|
5
|
-
.DESCRIPTION
|
|
6
|
-
Files marked key, bearer or group come from the FRED API and need your own free FRED API key
|
|
7
|
-
(https://fredaccount.stlouisfed.org/apikeys). The script uses $env:FRED_API_KEY when it is set,
|
|
8
|
-
otherwise asks for it with hidden input; press Enter to skip those files. The key is never printed
|
|
9
|
-
and never written into the saved files.
|
|
10
|
-
|
|
11
|
-
.EXAMPLE
|
|
12
|
-
powershell -NoProfile -ExecutionPolicy Bypass -File "$HOME\Downloads\download_fred.ps1"
|
|
13
|
-
#>
|
|
14
|
-
[CmdletBinding()]
|
|
15
|
-
param([switch]$NoPause)
|
|
16
|
-
$ErrorActionPreference = 'Stop'
|
|
17
|
-
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
|
|
18
|
-
$here = Split-Path -Parent $MyInvocation.MyCommand.Path
|
|
19
|
-
|
|
20
|
-
# file <TAB> kind <TAB> pages <TAB> url <TAB> then_file <TAB> then_url ("-" = empty, pages 0 = unpaged)
|
|
21
|
-
# kind: plain (no key), key (key in the query), bearer (FRED API v2, key in a header, paged),
|
|
22
|
-
# group (GeoFRED series group; its fields fill the __GROUP_*__ placeholders of then_url)
|
|
23
|
-
$manifest = @'
|
|
24
|
-
@MANIFEST@
|
|
25
|
-
'@
|
|
26
|
-
$items = $manifest -split "`n" | ForEach-Object { $_.TrimEnd("`r") } | Where-Object { $_ } | ForEach-Object {
|
|
27
|
-
$f = $_ -split "`t"
|
|
28
|
-
[pscustomobject]@{ File = $f[0]; Kind = $f[1]; Pages = [int]$f[2]; Url = $f[3]; ThenFile = $f[4]; ThenUrl = $f[5] }
|
|
29
|
-
}
|
|
30
|
-
|
|
31
|
-
$key = $env:FRED_API_KEY
|
|
32
|
-
if (-not $key -and ($items | Where-Object { $_.Kind -ne 'plain' })) {
|
|
33
|
-
Write-Host 'Some files need a FRED API key (https://fredaccount.stlouisfed.org/apikeys). Press Enter to skip them.'
|
|
34
|
-
if ([Console]::IsInputRedirected) { $key = [Console]::In.ReadLine() }
|
|
35
|
-
else {
|
|
36
|
-
$secure = Read-Host -Prompt 'FRED API key (hidden)' -AsSecureString
|
|
37
|
-
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
|
38
|
-
try { $key = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
|
39
|
-
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
|
40
|
-
}
|
|
41
|
-
}
|
|
42
|
-
if ($null -eq $key) { $key = '' }
|
|
43
|
-
$key = $key -replace '\s', ''
|
|
44
|
-
|
|
45
|
-
$utf8 = New-Object System.Text.UTF8Encoding($false)
|
|
46
|
-
$saved = New-Object System.Collections.ArrayList
|
|
47
|
-
|
|
48
|
-
# Download one keyed file, wrap it with the request URL (placeholder, not the key) and save it.
|
|
49
|
-
# Returns the reply text, or $null when the request failed.
|
|
50
|
-
function Save-Keyed([string]$file, [string]$url, [bool]$bearer) {
|
|
51
|
-
try {
|
|
52
|
-
if ($bearer) {
|
|
53
|
-
$r = Invoke-WebRequest -UseBasicParsing -UserAgent 'data-analyst/1.0' -Uri $url `
|
|
54
|
-
-Headers @{ Authorization = "Bearer $key" }
|
|
55
|
-
} else {
|
|
56
|
-
$r = Invoke-WebRequest -UseBasicParsing -UserAgent 'data-analyst/1.0' -Uri ($url.Replace('__FRED_KEY__', $key))
|
|
57
|
-
}
|
|
58
|
-
} catch {
|
|
59
|
-
$msg = "$($_.Exception.Message) $($_.ErrorDetails.Message)"
|
|
60
|
-
if ($key) { $msg = $msg.Replace($key, '[KEY]') }
|
|
61
|
-
Write-Host "${file}: download failed ($msg)"
|
|
62
|
-
return $null
|
|
63
|
-
}
|
|
64
|
-
$text = $utf8.GetString($r.RawContentStream.ToArray())
|
|
65
|
-
$wrapped = '{"request": "' + $url + '", "response": ' + $text + '}'
|
|
66
|
-
$dest = Join-Path $here $file
|
|
67
|
-
[IO.File]::WriteAllText($dest, $wrapped, $utf8)
|
|
68
|
-
[void]$saved.Add($dest)
|
|
69
|
-
Write-Host "saved $file"
|
|
70
|
-
return $text
|
|
71
|
-
}
|
|
72
|
-
|
|
73
|
-
foreach ($it in $items) {
|
|
74
|
-
if ($it.Kind -eq 'plain') {
|
|
75
|
-
$dest = Join-Path $here $it.File
|
|
76
|
-
try {
|
|
77
|
-
Invoke-WebRequest -UseBasicParsing -UserAgent 'data-analyst/1.0' -Uri $it.Url -OutFile $dest
|
|
78
|
-
[void]$saved.Add($dest)
|
|
79
|
-
Write-Host "saved $($it.File)"
|
|
80
|
-
} catch { Write-Host "$($it.File): download failed ($($_.Exception.Message))" }
|
|
81
|
-
Start-Sleep -Milliseconds 600
|
|
82
|
-
continue
|
|
83
|
-
}
|
|
84
|
-
if (-not $key) { Write-Host "skipped (needs a key): $($it.File)"; continue }
|
|
85
|
-
switch ($it.Kind) {
|
|
86
|
-
'key' { [void](Save-Keyed $it.File $it.Url $false) }
|
|
87
|
-
'bearer' {
|
|
88
|
-
# FRED API v2 release/observations: follow next_cursor up to Pages pages
|
|
89
|
-
$n = 1; $u = $it.Url; $f = $it.File
|
|
90
|
-
while ($true) {
|
|
91
|
-
$body = Save-Keyed $f $u $true
|
|
92
|
-
if ($null -eq $body -or $body -notmatch '"has_more"\s*:\s*true') { break }
|
|
93
|
-
if ($body -notmatch '"next_cursor"\s*:\s*"([^"]*)"' -or $n -ge $it.Pages) { break }
|
|
94
|
-
$n++
|
|
95
|
-
$u = $it.Url + '&next_cursor=' + [Uri]::EscapeDataString($Matches[1])
|
|
96
|
-
$f = $it.File -replace '\.json$', ('__p{0:D2}.json' -f $n)
|
|
97
|
-
Start-Sleep -Milliseconds 600
|
|
98
|
-
}
|
|
99
|
-
}
|
|
100
|
-
'group' {
|
|
101
|
-
# GeoFRED: the series group first, then the cross-section it names
|
|
102
|
-
$body = Save-Keyed $it.File $it.Url $false
|
|
103
|
-
if ($null -ne $body) {
|
|
104
|
-
$g = (ConvertFrom-Json $body).series_group
|
|
105
|
-
$u = $it.ThenUrl
|
|
106
|
-
foreach ($k in 'series_group', 'region_type', 'season', 'units', 'frequency', 'max_date') {
|
|
107
|
-
$v = "$($g.$k)"
|
|
108
|
-
if ($k -eq 'frequency') { $v = $v.Substring(0, [Math]::Min(1, $v.Length)).ToLower() }
|
|
109
|
-
$u = $u.Replace('__GROUP_' + $k.ToUpper() + '__', [Uri]::EscapeDataString($v))
|
|
110
|
-
}
|
|
111
|
-
Start-Sleep -Milliseconds 600
|
|
112
|
-
[void](Save-Keyed $it.ThenFile $u $false)
|
|
113
|
-
}
|
|
114
|
-
}
|
|
115
|
-
}
|
|
116
|
-
Start-Sleep -Milliseconds 600
|
|
117
|
-
}
|
|
118
|
-
$key = $null
|
|
119
|
-
Write-Host ''
|
|
120
|
-
Write-Host 'Done. Upload these files to the chat:'
|
|
121
|
-
$saved | ForEach-Object { Write-Host " $_" }
|
|
122
|
-
if (-not $NoPause -and -not [Console]::IsInputRedirected) { Read-Host 'Press Enter to close' | Out-Null }
|
|
1
|
+
<#
|
|
2
|
+
.SYNOPSIS
|
|
3
|
+
Download the FRED files for one data-analyst request into the folder that holds this script.
|
|
4
|
+
|
|
5
|
+
.DESCRIPTION
|
|
6
|
+
Files marked key, bearer or group come from the FRED API and need your own free FRED API key
|
|
7
|
+
(https://fredaccount.stlouisfed.org/apikeys). The script uses $env:FRED_API_KEY when it is set,
|
|
8
|
+
otherwise asks for it with hidden input; press Enter to skip those files. The key is never printed
|
|
9
|
+
and never written into the saved files.
|
|
10
|
+
|
|
11
|
+
.EXAMPLE
|
|
12
|
+
powershell -NoProfile -ExecutionPolicy Bypass -File "$HOME\Downloads\download_fred.ps1"
|
|
13
|
+
#>
|
|
14
|
+
[CmdletBinding()]
|
|
15
|
+
param([switch]$NoPause)
|
|
16
|
+
$ErrorActionPreference = 'Stop'
|
|
17
|
+
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
|
|
18
|
+
$here = Split-Path -Parent $MyInvocation.MyCommand.Path
|
|
19
|
+
|
|
20
|
+
# file <TAB> kind <TAB> pages <TAB> url <TAB> then_file <TAB> then_url ("-" = empty, pages 0 = unpaged)
|
|
21
|
+
# kind: plain (no key), key (key in the query), bearer (FRED API v2, key in a header, paged),
|
|
22
|
+
# group (GeoFRED series group; its fields fill the __GROUP_*__ placeholders of then_url)
|
|
23
|
+
$manifest = @'
|
|
24
|
+
@MANIFEST@
|
|
25
|
+
'@
|
|
26
|
+
$items = $manifest -split "`n" | ForEach-Object { $_.TrimEnd("`r") } | Where-Object { $_ } | ForEach-Object {
|
|
27
|
+
$f = $_ -split "`t"
|
|
28
|
+
[pscustomobject]@{ File = $f[0]; Kind = $f[1]; Pages = [int]$f[2]; Url = $f[3]; ThenFile = $f[4]; ThenUrl = $f[5] }
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
$key = $env:FRED_API_KEY
|
|
32
|
+
if (-not $key -and ($items | Where-Object { $_.Kind -ne 'plain' })) {
|
|
33
|
+
Write-Host 'Some files need a FRED API key (https://fredaccount.stlouisfed.org/apikeys). Press Enter to skip them.'
|
|
34
|
+
if ([Console]::IsInputRedirected) { $key = [Console]::In.ReadLine() }
|
|
35
|
+
else {
|
|
36
|
+
$secure = Read-Host -Prompt 'FRED API key (hidden)' -AsSecureString
|
|
37
|
+
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
|
38
|
+
try { $key = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
|
39
|
+
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
if ($null -eq $key) { $key = '' }
|
|
43
|
+
$key = $key -replace '\s', ''
|
|
44
|
+
|
|
45
|
+
$utf8 = New-Object System.Text.UTF8Encoding($false)
|
|
46
|
+
$saved = New-Object System.Collections.ArrayList
|
|
47
|
+
|
|
48
|
+
# Download one keyed file, wrap it with the request URL (placeholder, not the key) and save it.
|
|
49
|
+
# Returns the reply text, or $null when the request failed.
|
|
50
|
+
function Save-Keyed([string]$file, [string]$url, [bool]$bearer) {
|
|
51
|
+
try {
|
|
52
|
+
if ($bearer) {
|
|
53
|
+
$r = Invoke-WebRequest -UseBasicParsing -UserAgent 'data-analyst/1.0' -Uri $url `
|
|
54
|
+
-Headers @{ Authorization = "Bearer $key" }
|
|
55
|
+
} else {
|
|
56
|
+
$r = Invoke-WebRequest -UseBasicParsing -UserAgent 'data-analyst/1.0' -Uri ($url.Replace('__FRED_KEY__', $key))
|
|
57
|
+
}
|
|
58
|
+
} catch {
|
|
59
|
+
$msg = "$($_.Exception.Message) $($_.ErrorDetails.Message)"
|
|
60
|
+
if ($key) { $msg = $msg.Replace($key, '[KEY]') }
|
|
61
|
+
Write-Host "${file}: download failed ($msg)"
|
|
62
|
+
return $null
|
|
63
|
+
}
|
|
64
|
+
$text = $utf8.GetString($r.RawContentStream.ToArray())
|
|
65
|
+
$wrapped = '{"request": "' + $url + '", "response": ' + $text + '}'
|
|
66
|
+
$dest = Join-Path $here $file
|
|
67
|
+
[IO.File]::WriteAllText($dest, $wrapped, $utf8)
|
|
68
|
+
[void]$saved.Add($dest)
|
|
69
|
+
Write-Host "saved $file"
|
|
70
|
+
return $text
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
foreach ($it in $items) {
|
|
74
|
+
if ($it.Kind -eq 'plain') {
|
|
75
|
+
$dest = Join-Path $here $it.File
|
|
76
|
+
try {
|
|
77
|
+
Invoke-WebRequest -UseBasicParsing -UserAgent 'data-analyst/1.0' -Uri $it.Url -OutFile $dest
|
|
78
|
+
[void]$saved.Add($dest)
|
|
79
|
+
Write-Host "saved $($it.File)"
|
|
80
|
+
} catch { Write-Host "$($it.File): download failed ($($_.Exception.Message))" }
|
|
81
|
+
Start-Sleep -Milliseconds 600
|
|
82
|
+
continue
|
|
83
|
+
}
|
|
84
|
+
if (-not $key) { Write-Host "skipped (needs a key): $($it.File)"; continue }
|
|
85
|
+
switch ($it.Kind) {
|
|
86
|
+
'key' { [void](Save-Keyed $it.File $it.Url $false) }
|
|
87
|
+
'bearer' {
|
|
88
|
+
# FRED API v2 release/observations: follow next_cursor up to Pages pages
|
|
89
|
+
$n = 1; $u = $it.Url; $f = $it.File
|
|
90
|
+
while ($true) {
|
|
91
|
+
$body = Save-Keyed $f $u $true
|
|
92
|
+
if ($null -eq $body -or $body -notmatch '"has_more"\s*:\s*true') { break }
|
|
93
|
+
if ($body -notmatch '"next_cursor"\s*:\s*"([^"]*)"' -or $n -ge $it.Pages) { break }
|
|
94
|
+
$n++
|
|
95
|
+
$u = $it.Url + '&next_cursor=' + [Uri]::EscapeDataString($Matches[1])
|
|
96
|
+
$f = $it.File -replace '\.json$', ('__p{0:D2}.json' -f $n)
|
|
97
|
+
Start-Sleep -Milliseconds 600
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
'group' {
|
|
101
|
+
# GeoFRED: the series group first, then the cross-section it names
|
|
102
|
+
$body = Save-Keyed $it.File $it.Url $false
|
|
103
|
+
if ($null -ne $body) {
|
|
104
|
+
$g = (ConvertFrom-Json $body).series_group
|
|
105
|
+
$u = $it.ThenUrl
|
|
106
|
+
foreach ($k in 'series_group', 'region_type', 'season', 'units', 'frequency', 'max_date') {
|
|
107
|
+
$v = "$($g.$k)"
|
|
108
|
+
if ($k -eq 'frequency') { $v = $v.Substring(0, [Math]::Min(1, $v.Length)).ToLower() }
|
|
109
|
+
$u = $u.Replace('__GROUP_' + $k.ToUpper() + '__', [Uri]::EscapeDataString($v))
|
|
110
|
+
}
|
|
111
|
+
Start-Sleep -Milliseconds 600
|
|
112
|
+
[void](Save-Keyed $it.ThenFile $u $false)
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
Start-Sleep -Milliseconds 600
|
|
117
|
+
}
|
|
118
|
+
$key = $null
|
|
119
|
+
Write-Host ''
|
|
120
|
+
Write-Host 'Done. Upload these files to the chat:'
|
|
121
|
+
$saved | ForEach-Object { Write-Host " $_" }
|
|
122
|
+
if (-not $NoPause -and -not [Console]::IsInputRedirected) { Read-Host 'Press Enter to close' | Out-Null }
|
|
@@ -1,124 +1,124 @@
|
|
|
1
|
-
# Analysis patterns for FRED data (budget and economic analysis)
|
|
2
|
-
|
|
3
|
-
Read this once at the start of any analysis. It lists the choices that change headline numbers (and so
|
|
4
|
-
become the critical questions), the conventions to follow (those of the institution the question comes from, else the source agency's), how to benchmark, and
|
|
5
|
-
how a report is laid out.
|
|
6
|
-
|
|
7
|
-
Contents: 1. Critical questions catalog · 2. Conventions · 3. Benchmarks and checks · 4. Typical questions
|
|
8
|
-
· 5. Publication structure · 6. Chart forms
|
|
9
|
-
|
|
10
|
-
## 1. Critical questions catalog
|
|
11
|
-
Ask only the ones the request leaves open **and** that would change a headline number. Each has a default
|
|
12
|
-
rule for "Don't know — decide empirically" (show the main alternative as a sensitivity test).
|
|
13
|
-
|
|
14
|
-
| Choice | Typical options | Default rule when the user does not know |
|
|
15
|
-
|---|---|---|
|
|
16
|
-
| **Which measure** | Inflation: PCE vs. CPI-U vs. CPI-W vs. C-CPI-U; headline vs. core. Output: real GDP, GDP per capita, GDI. Debt: held by the public vs. total public debt. Interest: budget net interest (FYOINT) vs. NIPA interest payments | The measure the question's institution uses (for example, the Federal Reserve and the budget agencies lead with PCE for inflation and debt **held by the public**); indexing questions use the index the law names (CPI-W for Social Security COLAs, C-CPI-U for tax brackets, CPI-U for most other programs); budget questions use budget-basis series (OMB/Treasury), not NIPA |
|
|
17
|
-
| **Calendar or fiscal year** | Calendar year; federal fiscal year (October to September) | Budget amounts (outlays, revenues, deficits, debt, interest): fiscal years, and a share of GDP uses fiscal-year GDP (the average of the four quarters October to September). Economic variables: calendar years, unless they feed a fiscal-year ratio |
|
|
18
|
-
| **Nominal or real; deflator; base year** | Current dollars; real via PCE, CPI-U, GDP deflator; base = latest full year or a chosen year | Budget amounts: nominal, plus shares of GDP (the usual convention for comparisons over time). Incomes and wages: real, deflated by PCE or CPI-U when the user's context is CPI-based; base = latest full year; say "in <year> dollars" |
|
|
19
|
-
| **Growth measure** | Quarterly annualized; year over year; fourth quarter to fourth quarter; annual average over annual average; CAGR between dates | The publishing agency's convention: BEA's annual growth of GDP is **annual average over annual average** (its headline "real GDP increased 2.8 percent in 2024"); **Q4 over Q4** when comparing with Federal Reserve or CBO projections, which use it; annual average for levels such as unemployment and interest rates; quarterly annualized for the latest quarter (BEA's headline). Name the measure every time |
|
|
20
|
-
| **Frequency conversion** | Average, end of period, sum | Rates and prices: average. Stocks (debt, balances): end of period. Flows reported per period (monthly outlays): sum. Never let FRED convert silently: convert in `compute` (a rollup by period), and drop incomplete periods |
|
|
21
|
-
| **Vintage** | Latest data; as published on a date (ALFRED); first release; full revision history | Latest, unless the question is about what was known at a time (a baseline's starting point, a forecast evaluation, "as first reported"): then the vintage dated just before that time, and say so. Revision questions compare first release with latest |
|
|
22
|
-
| **Seasonal adjustment** | SA vs. NSA | Short-run changes and charts: SA. 12-month changes of CPI and annual averages: NSA (BLS convention). Fiscal-year sums: NSA source data summed over complete years |
|
|
23
|
-
| **Period and baseline** | A business-cycle peak (2019 Q4, Feb 2020), a policy date, the last 10 or 20 years | The business-cycle peak before the event the user cares about; otherwise the latest 10 complete years (20 for budget ratios). Show one alternative baseline as a sensitivity |
|
|
24
|
-
| **Projections** | Include CBO's projected potential output or NROU beyond the last actual date, or stop at the last actual | Stop at the last actual date for every comparison with data. Use projections only when the user asks about them, labeled as CBO's projections and with the vintage of the outlook |
|
|
25
|
-
|
|
26
|
-
Ask at most four, in one round. Series IDs are never a question for the user: resolve them with
|
|
27
|
-
`docs/series.md` and `request_schema` searches (words, or `describe <IDs>`).
|
|
28
|
-
|
|
29
|
-
## 2. Conventions
|
|
30
|
-
- **Shares of GDP:** numerator and denominator on the same basis and period. Fiscal-year budget amounts over
|
|
31
|
-
fiscal-year GDP; mind the units (FYOINT in millions, GDP in billions: divide the numerator by 1,000 first).
|
|
32
|
-
Benchmark against FRED's own ratio where one exists (FYOIGDA188S, FYFSGDA188S, FYGFGDQ188S), but know how it is
|
|
33
|
-
built: FRED's annual budget ratios divide the fiscal-year amount by **calendar-year** GDP (GDPA) as published
|
|
34
|
-
when the ratio was last updated (FY2025 interest: 970,065 / about 30,762, GDPA as published in April 2026, = 3.15
|
|
35
|
-
percent; current GDPA is higher after BEA's September 2026 revision; on fiscal-year GDP it is 3.18 percent;
|
|
36
|
-
checked 2026-10-20). CBO and OMB use fiscal-year GDP. So a ratio on fiscal-year GDP differs from
|
|
37
|
-
FRED's annual ratios by a few hundredths of a point; a larger gap means a units or fiscal-year error.
|
|
38
|
-
FYGFGDQ188S is different: quarterly debt over that quarter's annualized GDP, so its Q3 value differs from
|
|
39
|
-
debt over fiscal-year GDP by about 0.5 to 1.5 points (FY2019: 77.5 against 79.0). Never present FRED's
|
|
40
|
-
ratio as a fiscal-year-GDP ratio. `FYGDP` is OMB's fiscal-year GDP (from its latest budget; it lags BEA's
|
|
41
|
-
revisions); averaging the four quarters of `GDP` gives the current-vintage equivalent.
|
|
42
|
-
- **Annualized rates:** quarterly GDP growth is reported at an annual rate: `(Q_t/Q_{t-1})^4 - 1`.
|
|
43
|
-
GDP levels are already at an annual rate (SAAR): do not multiply by four; a fiscal-year
|
|
44
|
-
GDP is the mean of four quarterly SAAR values, not their sum.
|
|
45
|
-
- **Real values:** chained-dollar components do not add up to their totals; do not sum chained-dollar series.
|
|
46
|
-
Compute shares in nominal terms.
|
|
47
|
-
- **Indexes:** an index level is not a price. Percent changes of indexes from the published values, rounded to one
|
|
48
|
-
decimal at the end.
|
|
49
|
-
- **Fiscal years:** FY2025 = October 1, 2024, to September 30, 2025. `compute`'s `fiscal_year` and
|
|
50
|
-
`to_fiscal` steps assign fiscal years (FRED dates fiscal-year dollar series on Sep 30 and some ratios on Jan 1).
|
|
51
|
-
- **Gaps:** October 2025 was not collected for the CPI, CPS and LAUS (lapse in appropriations); daily market
|
|
52
|
-
series skip holidays. Changes and averages across a gap need both endpoints or a complete period.
|
|
53
|
-
- **Levels across vintages:** a comprehensive revision changes the reference year of chained-dollar series (real
|
|
54
|
-
GDP: chained 2012 dollars in vintages before September 2023, 2017 dollars after) and can redefine nominal
|
|
55
|
-
levels. Compare growth rates, shares and ratios across vintages, and say which base year each vintage uses.
|
|
56
|
-
- **Revisions:** BEA revises GDP in each later estimate and in annual and comprehensive updates; CBO's potential
|
|
57
|
-
output is re-estimated with each outlook; BLS revises payrolls and seasonal factors. State the retrieval date
|
|
58
|
-
(the builder adds it), and use ALFRED vintages when the timing matters.
|
|
59
|
-
- **CBO projections in FRED** (GDPPOT, NGDPPOT, NROU) reflect the outlook named on the series page
|
|
60
|
-
(last_updated). After a BEA comprehensive revision, an output gap that mixes the new GDP with potential built on
|
|
61
|
-
the old GDP can shift; say so when `last_updated` of the two differs by more than a few months.
|
|
62
|
-
- **Recessions:** shade from USREC (downloaded, never from memory) and the builder adds the note.
|
|
63
|
-
- **Correlation, not cause:** an event "coincided with" a change unless a method that supports a causal claim was
|
|
64
|
-
used. No forecasts of your own; CBO's or the Federal Reserve's projections are cited as theirs.
|
|
65
|
-
|
|
66
|
-
## 3. Benchmarks and checks
|
|
67
|
-
- **Benchmark every headline number** against a published figure, and save each comparison in
|
|
68
|
-
`exhibits/benchmark.csv` (measure, period, computed, published, source, URL, accessed date):
|
|
69
|
-
- FRED's own derived series: A191RL1Q225SBEA (real GDP growth), FYOIGDA188S / FYFSGDA188S / FYGFGDQ188S
|
|
70
|
-
(budget ratios), the `pc1` transform of a price index (download it as a second variant).
|
|
71
|
-
- The source's release: BEA's GDP release, BLS's CPI release, OMB's Historical Tables, the Treasury's Monthly
|
|
72
|
-
Treasury Statement, CBO's Monthly Budget Review or The Budget and Economic Outlook (cbo.gov, read through
|
|
73
|
-
the browser or the Wayback Machine when scripts are blocked).
|
|
74
|
-
Say "matches" only for rows in that file; a source not opened is "not checked".
|
|
75
|
-
- **Sanity checks before writing:** units (millions vs. billions), dates (fiscal-year end vs. start), the last
|
|
76
|
-
date of each series (a lagging annual series silently shortens a ratio), projections cut off, no FRED-side
|
|
77
|
-
conversion across a gap.
|
|
78
|
-
- **Sampling error:** most FRED macro series are not survey estimates with published standard errors; do not
|
|
79
|
-
call differences "statistically significant". For survey-based series (unemployment), see the bls pack's
|
|
80
|
-
thresholds or say "little changed" for small moves.
|
|
81
|
-
|
|
82
|
-
## 4. Typical questions
|
|
83
|
-
- **Output gap and potential growth** (GDPC1, GDPPOT, NROU vs. UNRATE): gap by quarter, versus 2019 Q4;
|
|
84
|
-
potential growth Q4/Q4; the unemployment gap.
|
|
85
|
-
- **Inflation for indexing or deflating a cost estimate** (CPI-W, CPI-U, C-CPI-U, PCE): Q3-average CPI-W growth
|
|
86
|
-
for COLAs, calendar- or fiscal-year averages, cumulative growth between two dates.
|
|
87
|
-
- **Federal interest costs** (FYOINT, FYOIGDA188S, GS10, TB3MS, FYGFDPUN): net interest as a share of GDP by
|
|
88
|
-
fiscal year, the average rate on debt (interest over average debt held by the public), versus market yields.
|
|
89
|
-
- **Deficits and debt** (FYFSD, FYFR, FYONET, FYGFGDQ188S): shares of GDP, past averages (50-year average),
|
|
90
|
-
peaks.
|
|
91
|
-
- **Revisions and real-time data** (ALFRED): GDP growth as first reported vs. current, the data a past baseline
|
|
92
|
-
used.
|
|
93
|
-
- **State comparisons** (GeoFRED or `<ST>UR`, `<ST>PCPI`): rank states, change over a year.
|
|
94
|
-
|
|
95
|
-
## 5. Publication structure (the report builder's `"mode": "report"`)
|
|
96
|
-
1. Cover with the month and year, the title with the period, and an optional hero chart.
|
|
97
|
-
2. **At a Glance:** one framing paragraph (which measures, which period, calendar or fiscal years) and 3–5
|
|
98
|
-
findings, each with a bold lead naming its section.
|
|
99
|
-
3. Contents and List of Exhibits (built automatically).
|
|
100
|
-
4. **Notes About This Report:** calendar or fiscal years; rounding; seasonal adjustment and annual rates; dollar
|
|
101
|
-
year and deflator; data source, retrieval date and vintage; projections; definitions; recession shading.
|
|
102
|
-
5. **Introduction:** a lead paragraph with the headline numbers; question-style subsections ("How Large Is the
|
|
103
|
-
Output Gap?") and "What Data Were Used?".
|
|
104
|
-
6. **Sections** (2–5): a summary page, then one exhibit per page with 2–4 paragraphs beside the chart (level,
|
|
105
|
-
change, comparison).
|
|
106
|
-
7. **Appendix A: Data and Methods:** series IDs and sources, units and conversions, fiscal-year construction,
|
|
107
|
-
deflators, vintages, benchmark comparisons, limitations. The builder appends the data vintage, the series
|
|
108
|
-
citations and the FRED notice.
|
|
109
|
-
8. References; About This Document (optional); replication materials and audit trail.
|
|
110
|
-
|
|
111
|
-
Exhibit titles end with the period: "Output Gap, 2007 Q1 to 2026 Q2"; "Net Interest Outlays as a Percentage of
|
|
112
|
-
GDP, Fiscal Years 2000 to 2025". The units line names the basis: "Percent of GDP"; "Billions of chained 2017
|
|
113
|
-
dollars, seasonally adjusted annual rate".
|
|
114
|
-
|
|
115
|
-
## 6. Chart forms
|
|
116
|
-
- Trends: line charts with a real date axis (the exhibit CSV keeps the `date`
|
|
117
|
-
column) and recession shading (`"recessions": true`); at most 4–5 series.
|
|
118
|
-
- Fiscal-year amounts: columns for a few years, a line for long histories; label years by number and say "fiscal
|
|
119
|
-
years" in the title.
|
|
120
|
-
- Comparisons at one date (states, components): horizontal bars when labels are long.
|
|
121
|
-
- Composition (contributions to growth, outlays by category): stacked columns.
|
|
122
|
-
- Indexes compared over time: rebase to 100 at the same date (a `compute` derive step).
|
|
123
|
-
- No dual axes (put the second measure in its own exhibit), no pies, no 3-D. Leave gaps empty in the exhibit CSV
|
|
124
|
-
so the line breaks there.
|
|
1
|
+
# Analysis patterns for FRED data (budget and economic analysis)
|
|
2
|
+
|
|
3
|
+
Read this once at the start of any analysis. It lists the choices that change headline numbers (and so
|
|
4
|
+
become the critical questions), the conventions to follow (those of the institution the question comes from, else the source agency's), how to benchmark, and
|
|
5
|
+
how a report is laid out.
|
|
6
|
+
|
|
7
|
+
Contents: 1. Critical questions catalog · 2. Conventions · 3. Benchmarks and checks · 4. Typical questions
|
|
8
|
+
· 5. Publication structure · 6. Chart forms
|
|
9
|
+
|
|
10
|
+
## 1. Critical questions catalog
|
|
11
|
+
Ask only the ones the request leaves open **and** that would change a headline number. Each has a default
|
|
12
|
+
rule for "Don't know — decide empirically" (show the main alternative as a sensitivity test).
|
|
13
|
+
|
|
14
|
+
| Choice | Typical options | Default rule when the user does not know |
|
|
15
|
+
|---|---|---|
|
|
16
|
+
| **Which measure** | Inflation: PCE vs. CPI-U vs. CPI-W vs. C-CPI-U; headline vs. core. Output: real GDP, GDP per capita, GDI. Debt: held by the public vs. total public debt. Interest: budget net interest (FYOINT) vs. NIPA interest payments | The measure the question's institution uses (for example, the Federal Reserve and the budget agencies lead with PCE for inflation and debt **held by the public**); indexing questions use the index the law names (CPI-W for Social Security COLAs, C-CPI-U for tax brackets, CPI-U for most other programs); budget questions use budget-basis series (OMB/Treasury), not NIPA |
|
|
17
|
+
| **Calendar or fiscal year** | Calendar year; federal fiscal year (October to September) | Budget amounts (outlays, revenues, deficits, debt, interest): fiscal years, and a share of GDP uses fiscal-year GDP (the average of the four quarters October to September). Economic variables: calendar years, unless they feed a fiscal-year ratio |
|
|
18
|
+
| **Nominal or real; deflator; base year** | Current dollars; real via PCE, CPI-U, GDP deflator; base = latest full year or a chosen year | Budget amounts: nominal, plus shares of GDP (the usual convention for comparisons over time). Incomes and wages: real, deflated by PCE or CPI-U when the user's context is CPI-based; base = latest full year; say "in <year> dollars" |
|
|
19
|
+
| **Growth measure** | Quarterly annualized; year over year; fourth quarter to fourth quarter; annual average over annual average; CAGR between dates | The publishing agency's convention: BEA's annual growth of GDP is **annual average over annual average** (its headline "real GDP increased 2.8 percent in 2024"); **Q4 over Q4** when comparing with Federal Reserve or CBO projections, which use it; annual average for levels such as unemployment and interest rates; quarterly annualized for the latest quarter (BEA's headline). Name the measure every time |
|
|
20
|
+
| **Frequency conversion** | Average, end of period, sum | Rates and prices: average. Stocks (debt, balances): end of period. Flows reported per period (monthly outlays): sum. Never let FRED convert silently: convert in `compute` (a rollup by period), and drop incomplete periods |
|
|
21
|
+
| **Vintage** | Latest data; as published on a date (ALFRED); first release; full revision history | Latest, unless the question is about what was known at a time (a baseline's starting point, a forecast evaluation, "as first reported"): then the vintage dated just before that time, and say so. Revision questions compare first release with latest |
|
|
22
|
+
| **Seasonal adjustment** | SA vs. NSA | Short-run changes and charts: SA. 12-month changes of CPI and annual averages: NSA (BLS convention). Fiscal-year sums: NSA source data summed over complete years |
|
|
23
|
+
| **Period and baseline** | A business-cycle peak (2019 Q4, Feb 2020), a policy date, the last 10 or 20 years | The business-cycle peak before the event the user cares about; otherwise the latest 10 complete years (20 for budget ratios). Show one alternative baseline as a sensitivity |
|
|
24
|
+
| **Projections** | Include CBO's projected potential output or NROU beyond the last actual date, or stop at the last actual | Stop at the last actual date for every comparison with data. Use projections only when the user asks about them, labeled as CBO's projections and with the vintage of the outlook |
|
|
25
|
+
|
|
26
|
+
Ask at most four, in one round. Series IDs are never a question for the user: resolve them with
|
|
27
|
+
`docs/series.md` and `request_schema` searches (words, or `describe <IDs>`).
|
|
28
|
+
|
|
29
|
+
## 2. Conventions
|
|
30
|
+
- **Shares of GDP:** numerator and denominator on the same basis and period. Fiscal-year budget amounts over
|
|
31
|
+
fiscal-year GDP; mind the units (FYOINT in millions, GDP in billions: divide the numerator by 1,000 first).
|
|
32
|
+
Benchmark against FRED's own ratio where one exists (FYOIGDA188S, FYFSGDA188S, FYGFGDQ188S), but know how it is
|
|
33
|
+
built: FRED's annual budget ratios divide the fiscal-year amount by **calendar-year** GDP (GDPA) as published
|
|
34
|
+
when the ratio was last updated (FY2025 interest: 970,065 / about 30,762, GDPA as published in April 2026, = 3.15
|
|
35
|
+
percent; current GDPA is higher after BEA's September 2026 revision; on fiscal-year GDP it is 3.18 percent;
|
|
36
|
+
checked 2026-10-20). CBO and OMB use fiscal-year GDP. So a ratio on fiscal-year GDP differs from
|
|
37
|
+
FRED's annual ratios by a few hundredths of a point; a larger gap means a units or fiscal-year error.
|
|
38
|
+
FYGFGDQ188S is different: quarterly debt over that quarter's annualized GDP, so its Q3 value differs from
|
|
39
|
+
debt over fiscal-year GDP by about 0.5 to 1.5 points (FY2019: 77.5 against 79.0). Never present FRED's
|
|
40
|
+
ratio as a fiscal-year-GDP ratio. `FYGDP` is OMB's fiscal-year GDP (from its latest budget; it lags BEA's
|
|
41
|
+
revisions); averaging the four quarters of `GDP` gives the current-vintage equivalent.
|
|
42
|
+
- **Annualized rates:** quarterly GDP growth is reported at an annual rate: `(Q_t/Q_{t-1})^4 - 1`.
|
|
43
|
+
GDP levels are already at an annual rate (SAAR): do not multiply by four; a fiscal-year
|
|
44
|
+
GDP is the mean of four quarterly SAAR values, not their sum.
|
|
45
|
+
- **Real values:** chained-dollar components do not add up to their totals; do not sum chained-dollar series.
|
|
46
|
+
Compute shares in nominal terms.
|
|
47
|
+
- **Indexes:** an index level is not a price. Percent changes of indexes from the published values, rounded to one
|
|
48
|
+
decimal at the end.
|
|
49
|
+
- **Fiscal years:** FY2025 = October 1, 2024, to September 30, 2025. `compute`'s `fiscal_year` and
|
|
50
|
+
`to_fiscal` steps assign fiscal years (FRED dates fiscal-year dollar series on Sep 30 and some ratios on Jan 1).
|
|
51
|
+
- **Gaps:** October 2025 was not collected for the CPI, CPS and LAUS (lapse in appropriations); daily market
|
|
52
|
+
series skip holidays. Changes and averages across a gap need both endpoints or a complete period.
|
|
53
|
+
- **Levels across vintages:** a comprehensive revision changes the reference year of chained-dollar series (real
|
|
54
|
+
GDP: chained 2012 dollars in vintages before September 2023, 2017 dollars after) and can redefine nominal
|
|
55
|
+
levels. Compare growth rates, shares and ratios across vintages, and say which base year each vintage uses.
|
|
56
|
+
- **Revisions:** BEA revises GDP in each later estimate and in annual and comprehensive updates; CBO's potential
|
|
57
|
+
output is re-estimated with each outlook; BLS revises payrolls and seasonal factors. State the retrieval date
|
|
58
|
+
(the builder adds it), and use ALFRED vintages when the timing matters.
|
|
59
|
+
- **CBO projections in FRED** (GDPPOT, NGDPPOT, NROU) reflect the outlook named on the series page
|
|
60
|
+
(last_updated). After a BEA comprehensive revision, an output gap that mixes the new GDP with potential built on
|
|
61
|
+
the old GDP can shift; say so when `last_updated` of the two differs by more than a few months.
|
|
62
|
+
- **Recessions:** shade from USREC (downloaded, never from memory) and the builder adds the note.
|
|
63
|
+
- **Correlation, not cause:** an event "coincided with" a change unless a method that supports a causal claim was
|
|
64
|
+
used. No forecasts of your own; CBO's or the Federal Reserve's projections are cited as theirs.
|
|
65
|
+
|
|
66
|
+
## 3. Benchmarks and checks
|
|
67
|
+
- **Benchmark every headline number** against a published figure, and save each comparison in
|
|
68
|
+
`exhibits/benchmark.csv` (measure, period, computed, published, source, URL, accessed date):
|
|
69
|
+
- FRED's own derived series: A191RL1Q225SBEA (real GDP growth), FYOIGDA188S / FYFSGDA188S / FYGFGDQ188S
|
|
70
|
+
(budget ratios), the `pc1` transform of a price index (download it as a second variant).
|
|
71
|
+
- The source's release: BEA's GDP release, BLS's CPI release, OMB's Historical Tables, the Treasury's Monthly
|
|
72
|
+
Treasury Statement, CBO's Monthly Budget Review or The Budget and Economic Outlook (cbo.gov, read through
|
|
73
|
+
the browser or the Wayback Machine when scripts are blocked).
|
|
74
|
+
Say "matches" only for rows in that file; a source not opened is "not checked".
|
|
75
|
+
- **Sanity checks before writing:** units (millions vs. billions), dates (fiscal-year end vs. start), the last
|
|
76
|
+
date of each series (a lagging annual series silently shortens a ratio), projections cut off, no FRED-side
|
|
77
|
+
conversion across a gap.
|
|
78
|
+
- **Sampling error:** most FRED macro series are not survey estimates with published standard errors; do not
|
|
79
|
+
call differences "statistically significant". For survey-based series (unemployment), see the bls pack's
|
|
80
|
+
thresholds or say "little changed" for small moves.
|
|
81
|
+
|
|
82
|
+
## 4. Typical questions
|
|
83
|
+
- **Output gap and potential growth** (GDPC1, GDPPOT, NROU vs. UNRATE): gap by quarter, versus 2019 Q4;
|
|
84
|
+
potential growth Q4/Q4; the unemployment gap.
|
|
85
|
+
- **Inflation for indexing or deflating a cost estimate** (CPI-W, CPI-U, C-CPI-U, PCE): Q3-average CPI-W growth
|
|
86
|
+
for COLAs, calendar- or fiscal-year averages, cumulative growth between two dates.
|
|
87
|
+
- **Federal interest costs** (FYOINT, FYOIGDA188S, GS10, TB3MS, FYGFDPUN): net interest as a share of GDP by
|
|
88
|
+
fiscal year, the average rate on debt (interest over average debt held by the public), versus market yields.
|
|
89
|
+
- **Deficits and debt** (FYFSD, FYFR, FYONET, FYGFGDQ188S): shares of GDP, past averages (50-year average),
|
|
90
|
+
peaks.
|
|
91
|
+
- **Revisions and real-time data** (ALFRED): GDP growth as first reported vs. current, the data a past baseline
|
|
92
|
+
used.
|
|
93
|
+
- **State comparisons** (GeoFRED or `<ST>UR`, `<ST>PCPI`): rank states, change over a year.
|
|
94
|
+
|
|
95
|
+
## 5. Publication structure (the report builder's `"mode": "report"`)
|
|
96
|
+
1. Cover with the month and year, the title with the period, and an optional hero chart.
|
|
97
|
+
2. **At a Glance:** one framing paragraph (which measures, which period, calendar or fiscal years) and 3–5
|
|
98
|
+
findings, each with a bold lead naming its section.
|
|
99
|
+
3. Contents and List of Exhibits (built automatically).
|
|
100
|
+
4. **Notes About This Report:** calendar or fiscal years; rounding; seasonal adjustment and annual rates; dollar
|
|
101
|
+
year and deflator; data source, retrieval date and vintage; projections; definitions; recession shading.
|
|
102
|
+
5. **Introduction:** a lead paragraph with the headline numbers; question-style subsections ("How Large Is the
|
|
103
|
+
Output Gap?") and "What Data Were Used?".
|
|
104
|
+
6. **Sections** (2–5): a summary page, then one exhibit per page with 2–4 paragraphs beside the chart (level,
|
|
105
|
+
change, comparison).
|
|
106
|
+
7. **Appendix A: Data and Methods:** series IDs and sources, units and conversions, fiscal-year construction,
|
|
107
|
+
deflators, vintages, benchmark comparisons, limitations. The builder appends the data vintage, the series
|
|
108
|
+
citations and the FRED notice.
|
|
109
|
+
8. References; About This Document (optional); replication materials and audit trail.
|
|
110
|
+
|
|
111
|
+
Exhibit titles end with the period: "Output Gap, 2007 Q1 to 2026 Q2"; "Net Interest Outlays as a Percentage of
|
|
112
|
+
GDP, Fiscal Years 2000 to 2025". The units line names the basis: "Percent of GDP"; "Billions of chained 2017
|
|
113
|
+
dollars, seasonally adjusted annual rate".
|
|
114
|
+
|
|
115
|
+
## 6. Chart forms
|
|
116
|
+
- Trends: line charts with a real date axis (the exhibit CSV keeps the `date`
|
|
117
|
+
column) and recession shading (`"recessions": true`); at most 4–5 series.
|
|
118
|
+
- Fiscal-year amounts: columns for a few years, a line for long histories; label years by number and say "fiscal
|
|
119
|
+
years" in the title.
|
|
120
|
+
- Comparisons at one date (states, components): horizontal bars when labels are long.
|
|
121
|
+
- Composition (contributions to growth, outlays by category): stacked columns.
|
|
122
|
+
- Indexes compared over time: rebase to 100 at the same date (a `compute` derive step).
|
|
123
|
+
- No dual axes (put the second measure in its own exhibit), no pies, no 3-D. Leave gaps empty in the exhibit CSV
|
|
124
|
+
so the line breaks there.
|