data-analyst-mcp 0.0.0-stage → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +261 -2
- package/dist/adapters/_acf_tidy.d.ts +18 -0
- package/dist/adapters/_acf_tidy.js +432 -0
- package/dist/adapters/_acf_tidy.js.map +1 -0
- package/dist/adapters/_actuaries_read.d.ts +38 -0
- package/dist/adapters/_actuaries_read.js +728 -0
- package/dist/adapters/_actuaries_read.js.map +1 -0
- package/dist/adapters/_census_micro.d.ts +70 -0
- package/dist/adapters/_census_micro.js +622 -0
- package/dist/adapters/_census_micro.js.map +1 -0
- package/dist/adapters/_census_util.d.ts +40 -0
- package/dist/adapters/_census_util.js +275 -0
- package/dist/adapters/_census_util.js.map +1 -0
- package/dist/adapters/_dol_states.d.ts +6 -0
- package/dist/adapters/_dol_states.js +58 -0
- package/dist/adapters/_dol_states.js.map +1 -0
- package/dist/adapters/_govinfo_entities.d.ts +4 -0
- package/dist/adapters/_govinfo_entities.js +6 -0
- package/dist/adapters/_govinfo_entities.js.map +1 -0
- package/dist/adapters/_govinfo_json.d.ts +13 -0
- package/dist/adapters/_govinfo_json.js +107 -0
- package/dist/adapters/_govinfo_json.js.map +1 -0
- package/dist/adapters/_govinfo_py.d.ts +22 -0
- package/dist/adapters/_govinfo_py.js +272 -0
- package/dist/adapters/_govinfo_py.js.map +1 -0
- package/dist/adapters/_govinfo_xml.d.ts +22 -0
- package/dist/adapters/_govinfo_xml.js +233 -0
- package/dist/adapters/_govinfo_xml.js.map +1 -0
- package/dist/adapters/_hud_excel.d.ts +2 -0
- package/dist/adapters/_hud_excel.js +166 -0
- package/dist/adapters/_hud_excel.js.map +1 -0
- package/dist/adapters/_omb_grid.d.ts +17 -0
- package/dist/adapters/_omb_grid.js +132 -0
- package/dist/adapters/_omb_grid.js.map +1 -0
- package/dist/adapters/_omb_util.d.ts +40 -0
- package/dist/adapters/_omb_util.js +192 -0
- package/dist/adapters/_omb_util.js.map +1 -0
- package/dist/adapters/_usaspending_state.d.ts +6 -0
- package/dist/adapters/_usaspending_state.js +54 -0
- package/dist/adapters/_usaspending_state.js.map +1 -0
- package/dist/adapters/_usda-fna_book.d.ts +59 -0
- package/dist/adapters/_usda-fna_book.js +869 -0
- package/dist/adapters/_usda-fna_book.js.map +1 -0
- package/dist/adapters/_usda-fna_pdf.d.ts +20 -0
- package/dist/adapters/_usda-fna_pdf.js +361 -0
- package/dist/adapters/_usda-fna_pdf.js.map +1 -0
- package/dist/adapters/_usda-outlook_read.d.ts +70 -0
- package/dist/adapters/_usda-outlook_read.js +362 -0
- package/dist/adapters/_usda-outlook_read.js.map +1 -0
- package/dist/adapters/acf.d.ts +3 -0
- package/dist/adapters/acf.js +187 -0
- package/dist/adapters/acf.js.map +1 -0
- package/dist/adapters/actuaries.d.ts +3 -0
- package/dist/adapters/actuaries.js +293 -0
- package/dist/adapters/actuaries.js.map +1 -0
- package/dist/adapters/bts.d.ts +3 -0
- package/dist/adapters/bts.js +26 -0
- package/dist/adapters/bts.js.map +1 -0
- package/dist/adapters/census-geocoder.d.ts +10 -0
- package/dist/adapters/census-geocoder.js +47 -0
- package/dist/adapters/census-geocoder.js.map +1 -0
- package/dist/adapters/census.d.ts +17 -0
- package/dist/adapters/census.js +983 -0
- package/dist/adapters/census.js.map +1 -0
- package/dist/adapters/cfpb.d.ts +7 -0
- package/dist/adapters/cfpb.js +348 -0
- package/dist/adapters/cfpb.js.map +1 -0
- package/dist/adapters/clinical-trials.d.ts +3 -0
- package/dist/adapters/clinical-trials.js +135 -0
- package/dist/adapters/clinical-trials.js.map +1 -0
- package/dist/adapters/cms.d.ts +3 -0
- package/dist/adapters/cms.js +292 -0
- package/dist/adapters/cms.js.map +1 -0
- package/dist/adapters/college-scorecard.d.ts +3 -0
- package/dist/adapters/college-scorecard.js +122 -0
- package/dist/adapters/college-scorecard.js.map +1 -0
- package/dist/adapters/congress.d.ts +33 -0
- package/dist/adapters/congress.js +589 -0
- package/dist/adapters/congress.js.map +1 -0
- package/dist/adapters/dhs-ohss.d.ts +13 -0
- package/dist/adapters/dhs-ohss.js +598 -0
- package/dist/adapters/dhs-ohss.js.map +1 -0
- package/dist/adapters/doj-news.d.ts +7 -0
- package/dist/adapters/doj-news.js +286 -0
- package/dist/adapters/doj-news.js.map +1 -0
- package/dist/adapters/dol.d.ts +4 -0
- package/dist/adapters/dol.js +634 -0
- package/dist/adapters/dol.js.map +1 -0
- package/dist/adapters/eia.d.ts +15 -0
- package/dist/adapters/eia.js +456 -0
- package/dist/adapters/eia.js.map +1 -0
- package/dist/adapters/epa-aqs.d.ts +3 -0
- package/dist/adapters/epa-aqs.js +357 -0
- package/dist/adapters/epa-aqs.js.map +1 -0
- package/dist/adapters/epa.d.ts +3 -0
- package/dist/adapters/epa.js +298 -0
- package/dist/adapters/epa.js.map +1 -0
- package/dist/adapters/fbi.d.ts +3 -0
- package/dist/adapters/fbi.js +192 -0
- package/dist/adapters/fbi.js.map +1 -0
- package/dist/adapters/fda.d.ts +5 -0
- package/dist/adapters/fda.js +184 -0
- package/dist/adapters/fda.js.map +1 -0
- package/dist/adapters/fdic.d.ts +4 -0
- package/dist/adapters/fdic.js +95 -0
- package/dist/adapters/fdic.js.map +1 -0
- package/dist/adapters/fec.d.ts +3 -0
- package/dist/adapters/fec.js +142 -0
- package/dist/adapters/fec.js.map +1 -0
- package/dist/adapters/fed-board.d.ts +12 -0
- package/dist/adapters/fed-board.js +581 -0
- package/dist/adapters/fed-board.js.map +1 -0
- package/dist/adapters/federal-register.d.ts +11 -0
- package/dist/adapters/federal-register.js +43 -0
- package/dist/adapters/federal-register.js.map +1 -0
- package/dist/adapters/fhfa.d.ts +3 -0
- package/dist/adapters/fhfa.js +253 -0
- package/dist/adapters/fhfa.js.map +1 -0
- package/dist/adapters/fsa.d.ts +9 -0
- package/dist/adapters/fsa.js +335 -0
- package/dist/adapters/fsa.js.map +1 -0
- package/dist/adapters/govinfo.d.ts +30 -0
- package/dist/adapters/govinfo.js +636 -0
- package/dist/adapters/govinfo.js.map +1 -0
- package/dist/adapters/gsa-calc.d.ts +10 -0
- package/dist/adapters/gsa-calc.js +318 -0
- package/dist/adapters/gsa-calc.js.map +1 -0
- package/dist/adapters/hud.d.ts +3 -0
- package/dist/adapters/hud.js +200 -0
- package/dist/adapters/hud.js.map +1 -0
- package/dist/adapters/index.d.ts +3 -0
- package/dist/adapters/index.js +25 -0
- package/dist/adapters/index.js.map +1 -0
- package/dist/adapters/irs-soi.d.ts +6 -0
- package/dist/adapters/irs-soi.js +576 -0
- package/dist/adapters/irs-soi.js.map +1 -0
- package/dist/adapters/naep.d.ts +15 -0
- package/dist/adapters/naep.js +341 -0
- package/dist/adapters/naep.js.map +1 -0
- package/dist/adapters/nhtsa.d.ts +10 -0
- package/dist/adapters/nhtsa.js +239 -0
- package/dist/adapters/nhtsa.js.map +1 -0
- package/dist/adapters/nih.d.ts +5 -0
- package/dist/adapters/nih.js +196 -0
- package/dist/adapters/nih.js.map +1 -0
- package/dist/adapters/noaa.d.ts +3 -0
- package/dist/adapters/noaa.js +148 -0
- package/dist/adapters/noaa.js.map +1 -0
- package/dist/adapters/nrel.d.ts +3 -0
- package/dist/adapters/nrel.js +186 -0
- package/dist/adapters/nrel.js.map +1 -0
- package/dist/adapters/nws.d.ts +3 -0
- package/dist/adapters/nws.js +119 -0
- package/dist/adapters/nws.js.map +1 -0
- package/dist/adapters/omb.d.ts +15 -0
- package/dist/adapters/omb.js +517 -0
- package/dist/adapters/omb.js.map +1 -0
- package/dist/adapters/open-payments.d.ts +5 -0
- package/dist/adapters/open-payments.js +149 -0
- package/dist/adapters/open-payments.js.map +1 -0
- package/dist/adapters/pbgc.d.ts +13 -0
- package/dist/adapters/pbgc.js +283 -0
- package/dist/adapters/pbgc.js.map +1 -0
- package/dist/adapters/philfed.d.ts +12 -0
- package/dist/adapters/philfed.js +59 -0
- package/dist/adapters/philfed.js.map +1 -0
- package/dist/adapters/regulations.d.ts +6 -0
- package/dist/adapters/regulations.js +58 -0
- package/dist/adapters/regulations.js.map +1 -0
- package/dist/adapters/sec.d.ts +13 -0
- package/dist/adapters/sec.js +504 -0
- package/dist/adapters/sec.js.map +1 -0
- package/dist/adapters/senate-lobbying.d.ts +3 -0
- package/dist/adapters/senate-lobbying.js +181 -0
- package/dist/adapters/senate-lobbying.js.map +1 -0
- package/dist/adapters/snapqc.d.ts +32 -0
- package/dist/adapters/snapqc.js +701 -0
- package/dist/adapters/snapqc.js.map +1 -0
- package/dist/adapters/treasury-yields.d.ts +3 -0
- package/dist/adapters/treasury-yields.js +170 -0
- package/dist/adapters/treasury-yields.js.map +1 -0
- package/dist/adapters/usaspending.d.ts +3 -0
- package/dist/adapters/usaspending.js +340 -0
- package/dist/adapters/usaspending.js.map +1 -0
- package/dist/adapters/usda-fna.d.ts +7 -0
- package/dist/adapters/usda-fna.js +707 -0
- package/dist/adapters/usda-fna.js.map +1 -0
- package/dist/adapters/usda-fooddata.d.ts +3 -0
- package/dist/adapters/usda-fooddata.js +101 -0
- package/dist/adapters/usda-fooddata.js.map +1 -0
- package/dist/adapters/usda-nass.d.ts +3 -0
- package/dist/adapters/usda-nass.js +108 -0
- package/dist/adapters/usda-nass.js.map +1 -0
- package/dist/adapters/usda-outlook.d.ts +12 -0
- package/dist/adapters/usda-outlook.js +696 -0
- package/dist/adapters/usda-outlook.js.map +1 -0
- package/dist/adapters/usgs.d.ts +9 -0
- package/dist/adapters/usgs.js +265 -0
- package/dist/adapters/usgs.js.map +1 -0
- package/dist/adapters/usitc.d.ts +18 -0
- package/dist/adapters/usitc.js +170 -0
- package/dist/adapters/usitc.js.map +1 -0
- package/dist/adapters/uspto.d.ts +3 -0
- package/dist/adapters/uspto.js +47 -0
- package/dist/adapters/uspto.js.map +1 -0
- package/dist/adapters/world-bank.d.ts +3 -0
- package/dist/adapters/world-bank.js +163 -0
- package/dist/adapters/world-bank.js.map +1 -0
- package/dist/analysis/cite.d.ts +31 -0
- package/dist/analysis/cite.js +229 -0
- package/dist/analysis/cite.js.map +1 -0
- package/dist/analysis/combine.d.ts +27 -0
- package/dist/analysis/combine.js +198 -0
- package/dist/analysis/combine.js.map +1 -0
- package/dist/analysis/compute.d.ts +62 -0
- package/dist/analysis/compute.js +583 -0
- package/dist/analysis/compute.js.map +1 -0
- package/dist/analysis/expr.d.ts +69 -0
- package/dist/analysis/expr.js +780 -0
- package/dist/analysis/expr.js.map +1 -0
- package/dist/analysis/factfmt.d.ts +16 -0
- package/dist/analysis/factfmt.js +187 -0
- package/dist/analysis/factfmt.js.map +1 -0
- package/dist/analysis/facts.d.ts +63 -0
- package/dist/analysis/facts.js +355 -0
- package/dist/analysis/facts.js.map +1 -0
- package/dist/analysis/handoff.d.ts +58 -0
- package/dist/analysis/handoff.js +208 -0
- package/dist/analysis/handoff.js.map +1 -0
- package/dist/analysis/pyfmt.d.ts +16 -0
- package/dist/analysis/pyfmt.js +156 -0
- package/dist/analysis/pyfmt.js.map +1 -0
- package/dist/analysis/rerun.d.ts +13 -0
- package/dist/analysis/rerun.js +125 -0
- package/dist/analysis/rerun.js.map +1 -0
- package/dist/analysis/table.d.ts +18 -0
- package/dist/analysis/table.js +109 -0
- package/dist/analysis/table.js.map +1 -0
- package/dist/analysis/tools.d.ts +67 -0
- package/dist/analysis/tools.js +217 -0
- package/dist/analysis/tools.js.map +1 -0
- package/dist/analysis/write_table.d.ts +13 -0
- package/dist/analysis/write_table.js +59 -0
- package/dist/analysis/write_table.js.map +1 -0
- package/dist/bin/cli.d.ts +10 -0
- package/dist/bin/cli.js +177 -0
- package/dist/bin/cli.js.map +1 -0
- package/dist/core/cache.d.ts +14 -0
- package/dist/core/cache.js +90 -0
- package/dist/core/cache.js.map +1 -0
- package/dist/core/context.d.ts +12 -0
- package/dist/core/context.js +15 -0
- package/dist/core/context.js.map +1 -0
- package/dist/core/env.d.ts +14 -0
- package/dist/core/env.js +52 -0
- package/dist/core/env.js.map +1 -0
- package/dist/core/generic-rest.d.ts +61 -0
- package/dist/core/generic-rest.js +639 -0
- package/dist/core/generic-rest.js.map +1 -0
- package/dist/core/http.d.ts +120 -0
- package/dist/core/http.js +514 -0
- package/dist/core/http.js.map +1 -0
- package/dist/core/keys.d.ts +43 -0
- package/dist/core/keys.js +174 -0
- package/dist/core/keys.js.map +1 -0
- package/dist/core/manual.d.ts +41 -0
- package/dist/core/manual.js +218 -0
- package/dist/core/manual.js.map +1 -0
- package/dist/core/package.d.ts +58 -0
- package/dist/core/package.js +228 -0
- package/dist/core/package.js.map +1 -0
- package/dist/core/packs.d.ts +139 -0
- package/dist/core/packs.js +173 -0
- package/dist/core/packs.js.map +1 -0
- package/dist/core/policy.d.ts +69 -0
- package/dist/core/policy.js +189 -0
- package/dist/core/policy.js.map +1 -0
- package/dist/core/py.d.ts +40 -0
- package/dist/core/py.js +171 -0
- package/dist/core/py.js.map +1 -0
- package/dist/core/routing.d.ts +10 -0
- package/dist/core/routing.js +39 -0
- package/dist/core/routing.js.map +1 -0
- package/dist/core/tabular.d.ts +21 -0
- package/dist/core/tabular.js +229 -0
- package/dist/core/tabular.js.map +1 -0
- package/dist/deep/_bea_api.d.ts +102 -0
- package/dist/deep/_bea_api.js +457 -0
- package/dist/deep/_bea_api.js.map +1 -0
- package/dist/deep/_bea_fetch.d.ts +151 -0
- package/dist/deep/_bea_fetch.js +1274 -0
- package/dist/deep/_bea_fetch.js.map +1 -0
- package/dist/deep/_bea_util.d.ts +14 -0
- package/dist/deep/_bea_util.js +156 -0
- package/dist/deep/_bea_util.js.map +1 -0
- package/dist/deep/_bls_api.d.ts +84 -0
- package/dist/deep/_bls_api.js +231 -0
- package/dist/deep/_bls_api.js.map +1 -0
- package/dist/deep/_bls_ids.d.ts +43 -0
- package/dist/deep/_bls_ids.js +188 -0
- package/dist/deep/_bls_ids.js.map +1 -0
- package/dist/deep/_bls_lib.d.ts +61 -0
- package/dist/deep/_bls_lib.js +442 -0
- package/dist/deep/_bls_lib.js.map +1 -0
- package/dist/deep/_fred_api.d.ts +148 -0
- package/dist/deep/_fred_api.js +545 -0
- package/dist/deep/_fred_api.js.map +1 -0
- package/dist/deep/_fred_pack.d.ts +162 -0
- package/dist/deep/_fred_pack.js +1027 -0
- package/dist/deep/_fred_pack.js.map +1 -0
- package/dist/deep/_fred_py.d.ts +40 -0
- package/dist/deep/_fred_py.js +199 -0
- package/dist/deep/_fred_py.js.map +1 -0
- package/dist/deep/_ipums_bundle.d.ts +29 -0
- package/dist/deep/_ipums_bundle.js +194 -0
- package/dist/deep/_ipums_bundle.js.map +1 -0
- package/dist/deep/_ipums_ddi.d.ts +23 -0
- package/dist/deep/_ipums_ddi.js +162 -0
- package/dist/deep/_ipums_ddi.js.map +1 -0
- package/dist/deep/_opm_io.d.ts +26 -0
- package/dist/deep/_opm_io.js +383 -0
- package/dist/deep/_opm_io.js.map +1 -0
- package/dist/deep/_opm_parquet.d.ts +31 -0
- package/dist/deep/_opm_parquet.js +744 -0
- package/dist/deep/_opm_parquet.js.map +1 -0
- package/dist/deep/_opm_py.d.ts +6 -0
- package/dist/deep/_opm_py.js +74 -0
- package/dist/deep/_opm_py.js.map +1 -0
- package/dist/deep/_treasury_api.d.ts +48 -0
- package/dist/deep/_treasury_api.js +221 -0
- package/dist/deep/_treasury_api.js.map +1 -0
- package/dist/deep/_treasury_context.d.ts +34 -0
- package/dist/deep/_treasury_context.js +142 -0
- package/dist/deep/_treasury_context.js.map +1 -0
- package/dist/deep/_treasury_lib.d.ts +109 -0
- package/dist/deep/_treasury_lib.js +686 -0
- package/dist/deep/_treasury_lib.js.map +1 -0
- package/dist/deep/bea.d.ts +3 -0
- package/dist/deep/bea.js +248 -0
- package/dist/deep/bea.js.map +1 -0
- package/dist/deep/bls.d.ts +3 -0
- package/dist/deep/bls.js +220 -0
- package/dist/deep/bls.js.map +1 -0
- package/dist/deep/fred.d.ts +3 -0
- package/dist/deep/fred.js +325 -0
- package/dist/deep/fred.js.map +1 -0
- package/dist/deep/ipums.d.ts +59 -0
- package/dist/deep/ipums.js +744 -0
- package/dist/deep/ipums.js.map +1 -0
- package/dist/deep/opm.d.ts +56 -0
- package/dist/deep/opm.js +1033 -0
- package/dist/deep/opm.js.map +1 -0
- package/dist/deep/treasury.d.ts +3 -0
- package/dist/deep/treasury.js +218 -0
- package/dist/deep/treasury.js.map +1 -0
- package/dist/envelope.d.ts +47 -0
- package/dist/envelope.js +88 -0
- package/dist/envelope.js.map +1 -0
- package/dist/index.d.ts +277 -0
- package/dist/index.js +106 -0
- package/dist/index.js.map +1 -0
- package/dist/remote/app.d.ts +21 -0
- package/dist/remote/app.js +218 -0
- package/dist/remote/app.js.map +1 -0
- package/dist/remote/oauth.d.ts +14 -0
- package/dist/remote/oauth.js +90 -0
- package/dist/remote/oauth.js.map +1 -0
- package/dist/remote/pages.d.ts +8 -0
- package/dist/remote/pages.js +56 -0
- package/dist/remote/pages.js.map +1 -0
- package/dist/remote/sign.d.ts +4 -0
- package/dist/remote/sign.js +40 -0
- package/dist/remote/sign.js.map +1 -0
- package/dist/remote/store.d.ts +31 -0
- package/dist/remote/store.js +129 -0
- package/dist/remote/store.js.map +1 -0
- package/dist/report/audit_answer.d.ts +94 -0
- package/dist/report/audit_answer.js +1349 -0
- package/dist/report/audit_answer.js.map +1 -0
- package/dist/report/audit_report.d.ts +23 -0
- package/dist/report/audit_report.js +637 -0
- package/dist/report/audit_report.js.map +1 -0
- package/dist/report/build_report.d.ts +27 -0
- package/dist/report/build_report.js +1168 -0
- package/dist/report/build_report.js.map +1 -0
- package/dist/report/method_audit.d.ts +47 -0
- package/dist/report/method_audit.js +716 -0
- package/dist/report/method_audit.js.map +1 -0
- package/dist/report/policyset.d.ts +50 -0
- package/dist/report/policyset.js +421 -0
- package/dist/report/policyset.js.map +1 -0
- package/dist/report/py.d.ts +58 -0
- package/dist/report/py.js +311 -0
- package/dist/report/py.js.map +1 -0
- package/dist/report/scan_secrets.d.ts +24 -0
- package/dist/report/scan_secrets.js +265 -0
- package/dist/report/scan_secrets.js.map +1 -0
- package/dist/report/style_check.d.ts +29 -0
- package/dist/report/style_check.js +173 -0
- package/dist/report/style_check.js.map +1 -0
- package/dist/report/tools.d.ts +87 -0
- package/dist/report/tools.js +167 -0
- package/dist/report/tools.js.map +1 -0
- package/dist/server.d.ts +5 -0
- package/dist/server.js +247 -0
- package/dist/server.js.map +1 -0
- package/dist/session.d.ts +47 -0
- package/dist/session.js +107 -0
- package/dist/session.js.map +1 -0
- package/dist/toolkit.d.ts +21 -0
- package/dist/toolkit.js +2 -0
- package/dist/toolkit.js.map +1 -0
- package/dist/tools.d.ts +299 -0
- package/dist/tools.js +491 -0
- package/dist/tools.js.map +1 -0
- package/guide/WORKFLOW.md +134 -0
- package/guide/assets/disclaimer.txt +1 -0
- package/guide/assets/download_data.ps1 +162 -0
- package/guide/assets/download_data.sh +143 -0
- package/guide/assets/report_template.html +597 -0
- package/guide/references/report_patterns.md +199 -0
- package/guide/references/routing.md +171 -0
- package/guide/references/style_guide.md +398 -0
- package/package.json +54 -4
- package/packs/acf/endpoints.json +114 -0
- package/packs/acf/policy.json +104 -0
- package/packs/acf/source.md +138 -0
- package/packs/actuaries/endpoints.json +82 -0
- package/packs/actuaries/policy.json +109 -0
- package/packs/actuaries/source.md +137 -0
- package/packs/bea/assets/download_bea.ps1 +182 -0
- package/packs/bea/assets/download_bea.sh +144 -0
- package/packs/bea/docs/bea_api.md +461 -0
- package/packs/bea/docs/datasets.md +157 -0
- package/packs/bea/docs/report_patterns.md +112 -0
- package/packs/bea/policy.json +170 -0
- package/packs/bea/source.md +152 -0
- package/packs/bls/assets/download_bls.ps1 +152 -0
- package/packs/bls/assets/download_bls.sh +167 -0
- package/packs/bls/docs/analysis_patterns.md +108 -0
- package/packs/bls/docs/bls_api.md +73 -0
- package/packs/bls/docs/series_ids.md +149 -0
- package/packs/bls/policy.json +238 -0
- package/packs/bls/series_ids.json +1375 -0
- package/packs/bls/source.md +156 -0
- package/packs/bts/endpoints.json +347 -0
- package/packs/bts/policy.json +94 -0
- package/packs/bts/source.md +97 -0
- package/packs/cdc/endpoints.json +197 -0
- package/packs/cdc/policy.json +83 -0
- package/packs/cdc/source.md +74 -0
- package/packs/census/endpoints.json +99 -0
- package/packs/census/policy.json +238 -0
- package/packs/census/source.md +304 -0
- package/packs/census-geocoder/endpoints.json +70 -0
- package/packs/census-geocoder/policy.json +80 -0
- package/packs/census-geocoder/source.md +53 -0
- package/packs/cfpb/endpoints.json +327 -0
- package/packs/cfpb/policy.json +119 -0
- package/packs/cfpb/source.md +96 -0
- package/packs/clinical-trials/endpoints.json +167 -0
- package/packs/clinical-trials/policy.json +84 -0
- package/packs/clinical-trials/source.md +82 -0
- package/packs/cms/endpoints.json +69 -0
- package/packs/cms/policy.json +96 -0
- package/packs/cms/source.md +112 -0
- package/packs/college-scorecard/endpoints.json +253 -0
- package/packs/college-scorecard/policy.json +106 -0
- package/packs/college-scorecard/source.md +92 -0
- package/packs/congress/endpoints.json +1701 -0
- package/packs/congress/policy.json +117 -0
- package/packs/congress/source.md +190 -0
- package/packs/dhs-ohss/endpoints.json +66 -0
- package/packs/dhs-ohss/policy.json +107 -0
- package/packs/dhs-ohss/source.md +120 -0
- package/packs/doj-news/endpoints.json +59 -0
- package/packs/doj-news/policy.json +96 -0
- package/packs/doj-news/source.md +99 -0
- package/packs/dol/endpoints.json +818 -0
- package/packs/dol/policy.json +135 -0
- package/packs/dol/source.md +181 -0
- package/packs/eia/endpoints.json +718 -0
- package/packs/eia/policy.json +112 -0
- package/packs/eia/source.md +112 -0
- package/packs/epa/endpoints.json +292 -0
- package/packs/epa/policy.json +99 -0
- package/packs/epa/source.md +99 -0
- package/packs/epa-aqs/endpoints.json +174 -0
- package/packs/epa-aqs/policy.json +96 -0
- package/packs/epa-aqs/source.md +97 -0
- package/packs/fbi/endpoints.json +1842 -0
- package/packs/fbi/policy.json +118 -0
- package/packs/fbi/source.md +108 -0
- package/packs/fda/endpoints.json +238 -0
- package/packs/fda/policy.json +121 -0
- package/packs/fda/source.md +117 -0
- package/packs/fdic/endpoints.json +417 -0
- package/packs/fdic/policy.json +83 -0
- package/packs/fdic/source.md +92 -0
- package/packs/fec/endpoints.json +708 -0
- package/packs/fec/policy.json +93 -0
- package/packs/fec/source.md +90 -0
- package/packs/fed-board/endpoints.json +89 -0
- package/packs/fed-board/policy.json +136 -0
- package/packs/fed-board/source.md +118 -0
- package/packs/federal-register/endpoints.json +143 -0
- package/packs/federal-register/policy.json +86 -0
- package/packs/federal-register/source.md +98 -0
- package/packs/fema/endpoints.json +646 -0
- package/packs/fema/policy.json +103 -0
- package/packs/fema/source.md +103 -0
- package/packs/fhfa/endpoints.json +105 -0
- package/packs/fhfa/policy.json +109 -0
- package/packs/fhfa/source.md +108 -0
- package/packs/fred/assets/download_fred.ps1 +122 -0
- package/packs/fred/assets/download_fred.sh +123 -0
- package/packs/fred/docs/analysis_patterns.md +124 -0
- package/packs/fred/docs/fred_api.md +94 -0
- package/packs/fred/docs/series.md +188 -0
- package/packs/fred/policy.json +253 -0
- package/packs/fred/source.md +142 -0
- package/packs/fsa/endpoints.json +150 -0
- package/packs/fsa/policy.json +99 -0
- package/packs/fsa/source.md +102 -0
- package/packs/govinfo/endpoints.json +520 -0
- package/packs/govinfo/policy.json +116 -0
- package/packs/govinfo/source.md +241 -0
- package/packs/gsa-calc/endpoints.json +93 -0
- package/packs/gsa-calc/policy.json +97 -0
- package/packs/gsa-calc/source.md +108 -0
- package/packs/hud/endpoints.json +147 -0
- package/packs/hud/policy.json +84 -0
- package/packs/hud/source.md +97 -0
- package/packs/ipums/assets/download_ipums.ps1 +215 -0
- package/packs/ipums/assets/download_ipums.sh +192 -0
- package/packs/ipums/docs/ipums_api.md +67 -0
- package/packs/ipums/docs/report_patterns.md +117 -0
- package/packs/ipums/docs/variables.md +284 -0
- package/packs/ipums/lib/fetch_text.json +112 -0
- package/packs/ipums/policy.json +215 -0
- package/packs/ipums/source.md +169 -0
- package/packs/irs-soi/endpoints.json +114 -0
- package/packs/irs-soi/policy.json +107 -0
- package/packs/irs-soi/source.md +124 -0
- package/packs/medicaid/endpoints.json +412 -0
- package/packs/medicaid/policy.json +108 -0
- package/packs/medicaid/source.md +136 -0
- package/packs/naep/endpoints.json +217 -0
- package/packs/naep/policy.json +100 -0
- package/packs/naep/source.md +120 -0
- package/packs/nhtsa/endpoints.json +132 -0
- package/packs/nhtsa/policy.json +93 -0
- package/packs/nhtsa/source.md +109 -0
- package/packs/nih/endpoints.json +453 -0
- package/packs/nih/policy.json +86 -0
- package/packs/nih/source.md +108 -0
- package/packs/noaa/endpoints.json +432 -0
- package/packs/noaa/policy.json +95 -0
- package/packs/noaa/source.md +88 -0
- package/packs/nrel/endpoints.json +328 -0
- package/packs/nrel/policy.json +96 -0
- package/packs/nrel/source.md +91 -0
- package/packs/nws/endpoints.json +272 -0
- package/packs/nws/policy.json +82 -0
- package/packs/nws/source.md +81 -0
- package/packs/nyfed/endpoints.json +80 -0
- package/packs/nyfed/policy.json +92 -0
- package/packs/nyfed/source.md +89 -0
- package/packs/omb/endpoints.json +460 -0
- package/packs/omb/policy.json +106 -0
- package/packs/omb/source.md +130 -0
- package/packs/open-payments/endpoints.json +298 -0
- package/packs/open-payments/policy.json +101 -0
- package/packs/open-payments/source.md +99 -0
- package/packs/opm/assets/download_opm.ps1 +125 -0
- package/packs/opm/assets/download_opm.sh +85 -0
- package/packs/opm/docs/fields.md +184 -0
- package/packs/opm/docs/opm_api.md +71 -0
- package/packs/opm/docs/report_patterns.md +83 -0
- package/packs/opm/lib/data_dictionary.csv +73 -0
- package/packs/opm/policy.json +142 -0
- package/packs/opm/source.md +137 -0
- package/packs/pbgc/endpoints.json +483 -0
- package/packs/pbgc/policy.json +94 -0
- package/packs/pbgc/source.md +95 -0
- package/packs/philfed/endpoints.json +85 -0
- package/packs/philfed/policy.json +94 -0
- package/packs/philfed/source.md +125 -0
- package/packs/port.json +287 -0
- package/packs/regulations/endpoints.json +288 -0
- package/packs/regulations/policy.json +86 -0
- package/packs/regulations/source.md +84 -0
- package/packs/sec/endpoints.json +237 -0
- package/packs/sec/policy.json +86 -0
- package/packs/sec/source.md +101 -0
- package/packs/senate-lobbying/endpoints.json +554 -0
- package/packs/senate-lobbying/policy.json +99 -0
- package/packs/senate-lobbying/source.md +100 -0
- package/packs/snapqc/endpoints.json +1426 -0
- package/packs/snapqc/policy.json +92 -0
- package/packs/snapqc/source.md +96 -0
- package/packs/treasury/catalog.json +2267 -0
- package/packs/treasury/docs/api_guide.md +245 -0
- package/packs/treasury/docs/datasets.md +56 -0
- package/packs/treasury/docs/report_patterns.md +126 -0
- package/packs/treasury/policy.json +159 -0
- package/packs/treasury/source.md +170 -0
- package/packs/treasury-yields/endpoints.json +226 -0
- package/packs/treasury-yields/policy.json +86 -0
- package/packs/treasury-yields/source.md +80 -0
- package/packs/usaspending/endpoints.json +1316 -0
- package/packs/usaspending/policy.json +109 -0
- package/packs/usaspending/source.md +103 -0
- package/packs/usda-fna/endpoints.json +427 -0
- package/packs/usda-fna/policy.json +133 -0
- package/packs/usda-fna/source.md +217 -0
- package/packs/usda-fooddata/endpoints.json +181 -0
- package/packs/usda-fooddata/policy.json +94 -0
- package/packs/usda-fooddata/source.md +105 -0
- package/packs/usda-nass/endpoints.json +324 -0
- package/packs/usda-nass/policy.json +95 -0
- package/packs/usda-nass/source.md +85 -0
- package/packs/usda-outlook/endpoints.json +86 -0
- package/packs/usda-outlook/policy.json +109 -0
- package/packs/usda-outlook/source.md +144 -0
- package/packs/usgs/endpoints.json +313 -0
- package/packs/usgs/policy.json +95 -0
- package/packs/usgs/source.md +96 -0
- package/packs/usitc/endpoints.json +74 -0
- package/packs/usitc/policy.json +111 -0
- package/packs/usitc/source.md +90 -0
- package/packs/uspto/endpoints.json +141 -0
- package/packs/uspto/policy.json +95 -0
- package/packs/uspto/source.md +101 -0
- package/packs/va/endpoints.json +414 -0
- package/packs/va/policy.json +101 -0
- package/packs/va/source.md +129 -0
- package/packs/world-bank/endpoints.json +122 -0
- package/packs/world-bank/policy.json +101 -0
- package/packs/world-bank/source.md +93 -0
|
@@ -0,0 +1,125 @@
|
|
|
1
|
+
<#
|
|
2
|
+
.SYNOPSIS
|
|
3
|
+
Download the OPM Federal Workforce Data files for this analysis into the folder that holds this script.
|
|
4
|
+
(Windows PowerShell counterpart of download_opm.sh.)
|
|
5
|
+
|
|
6
|
+
.DESCRIPTION
|
|
7
|
+
- The request is built into this script (written by data-analyst-mcp; do not edit it).
|
|
8
|
+
- It lists the files OPM publishes now (data.opm.gov, no account or key), picks the months the request
|
|
9
|
+
asks for, and saves each one as <dataset>_<YYYYMM>_<version>.parquet.
|
|
10
|
+
- Employment files are 26-75 MB each; accession and separation files are under 4 MB.
|
|
11
|
+
- Needs nothing beyond Windows PowerShell 5.1 or PowerShell 7. Run it again after an interruption:
|
|
12
|
+
files already saved are kept.
|
|
13
|
+
|
|
14
|
+
.PARAMETER NoPause
|
|
15
|
+
Do not wait for Enter before closing the window.
|
|
16
|
+
|
|
17
|
+
.EXAMPLE
|
|
18
|
+
powershell -NoProfile -ExecutionPolicy Bypass -File .\download_opm.ps1
|
|
19
|
+
#>
|
|
20
|
+
[CmdletBinding()]
|
|
21
|
+
param([switch]$NoPause)
|
|
22
|
+
|
|
23
|
+
$ErrorActionPreference = 'Stop'
|
|
24
|
+
$ProgressPreference = 'SilentlyContinue' # Invoke-WebRequest is very slow with the progress bar
|
|
25
|
+
try {
|
|
26
|
+
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
|
27
|
+
} catch { }
|
|
28
|
+
|
|
29
|
+
$Api = 'https://data.opm.gov/api/v1/files'
|
|
30
|
+
if ($env:OPM_API_BASE) { $Api = $env:OPM_API_BASE.TrimEnd('/') }
|
|
31
|
+
|
|
32
|
+
# The request. Columns, tab-separated: dataset, then either
|
|
33
|
+
# periods YYYYMM,... (or "latest") the listed months
|
|
34
|
+
# range FROM|first TO|latest MONTHS every month from FROM to TO in MONTHS (1-12)
|
|
35
|
+
# pin YYYYMM VERSION use this version of one month instead of the current one
|
|
36
|
+
$Rules = @'
|
|
37
|
+
@RULES@
|
|
38
|
+
'@
|
|
39
|
+
|
|
40
|
+
$Here = $PSScriptRoot
|
|
41
|
+
if (-not $Here) { $Here = (Get-Location).Path }
|
|
42
|
+
|
|
43
|
+
function Finish([int]$Code) {
|
|
44
|
+
if (-not $NoPause -and -not [Console]::IsInputRedirected) {
|
|
45
|
+
Write-Host ''
|
|
46
|
+
[void](Read-Host -Prompt 'Press Enter to close')
|
|
47
|
+
}
|
|
48
|
+
exit $Code
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
function Fail([string]$Message) {
|
|
52
|
+
Write-Host "ERROR: $Message" -ForegroundColor Red
|
|
53
|
+
Finish 1
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
$rows = @(foreach ($line in ($Rules -split "`r?`n")) {
|
|
57
|
+
if (-not $line.Trim()) { continue }
|
|
58
|
+
$c = $line -split "`t"
|
|
59
|
+
[pscustomobject]@{ Ds = $c[0]; Kind = $c[1]; A = $c[2]; B = $c[3]; C = $c[4] }
|
|
60
|
+
})
|
|
61
|
+
$saved = @(); $failed = 0; $missing = @()
|
|
62
|
+
foreach ($ds in @($rows | ForEach-Object { $_.Ds } | Sort-Object -Unique)) {
|
|
63
|
+
Write-Host "Listing the $ds files..."
|
|
64
|
+
try {
|
|
65
|
+
$reply = Invoke-RestMethod -Uri "$Api/$($ds)?current=true" -UseBasicParsing -UserAgent 'data-analyst/1.0'
|
|
66
|
+
if ($reply -is [string]) { $reply = $reply | ConvertFrom-Json } # served without a JSON content type
|
|
67
|
+
$list = @($reply | ForEach-Object { $_ })
|
|
68
|
+
} catch {
|
|
69
|
+
Fail "Could not list the $ds files at $Api ($($_.Exception.Message)). Check the connection, VPN and proxy, then run again."
|
|
70
|
+
}
|
|
71
|
+
$avail = @{}
|
|
72
|
+
foreach ($f in $list) {
|
|
73
|
+
if ([string]$f.filename -match "^$($ds)_(\d{6})_(\d+)$") { $avail[$Matches[1]] = $Matches[2] }
|
|
74
|
+
}
|
|
75
|
+
if ($avail.Count -eq 0) { Fail "OPM listed no $ds files. Run again later." }
|
|
76
|
+
$periods = @($avail.Keys | Sort-Object)
|
|
77
|
+
$first = $periods[0]; $latest = $periods[-1]
|
|
78
|
+
$want = @()
|
|
79
|
+
foreach ($r in @($rows | Where-Object { $_.Ds -eq $ds })) {
|
|
80
|
+
if ($r.Kind -eq 'periods') {
|
|
81
|
+
foreach ($p in $r.A.Split(',')) {
|
|
82
|
+
if ($p -eq 'latest') { $p = $latest }
|
|
83
|
+
if ($avail.ContainsKey($p)) { $want += $p } else { $missing += "$ds $p" }
|
|
84
|
+
}
|
|
85
|
+
} elseif ($r.Kind -eq 'range') {
|
|
86
|
+
$lo = $r.A; $hi = $r.B
|
|
87
|
+
if ($lo -eq 'first') { $lo = $first }
|
|
88
|
+
if ($hi -eq 'latest') { $hi = $latest }
|
|
89
|
+
$months = @($r.C.Split(',') | ForEach-Object { [int]$_ })
|
|
90
|
+
foreach ($p in $periods) {
|
|
91
|
+
if ($p -ge $lo -and $p -le $hi -and $months -contains [int]$p.Substring(4, 2)) { $want += $p }
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
foreach ($p in @($want | Sort-Object -Unique)) {
|
|
96
|
+
$v = $avail[$p]
|
|
97
|
+
$pin = @($rows | Where-Object { $_.Ds -eq $ds -and $_.Kind -eq 'pin' -and $_.A -eq $p })
|
|
98
|
+
if ($pin.Count -gt 0) { $v = $pin[0].B }
|
|
99
|
+
$name = "$($ds)_$($p)_$($v).parquet"
|
|
100
|
+
$dest = Join-Path $Here $name
|
|
101
|
+
if ((Test-Path -LiteralPath $dest) -and (Get-Item -LiteralPath $dest).Length -gt 0) {
|
|
102
|
+
Write-Host "already saved: $name"; $saved += $dest; continue
|
|
103
|
+
}
|
|
104
|
+
Write-Host "Downloading $name ..."
|
|
105
|
+
try {
|
|
106
|
+
Invoke-WebRequest -Uri "$Api/$ds/$($p.Substring(0, 4))/$($p.Substring(4, 2))/$v/download" -OutFile "$dest.part" `
|
|
107
|
+
-UseBasicParsing -UserAgent 'data-analyst/1.0'
|
|
108
|
+
Move-Item -LiteralPath "$dest.part" -Destination $dest -Force
|
|
109
|
+
$saved += $dest
|
|
110
|
+
} catch {
|
|
111
|
+
if (Test-Path -LiteralPath "$dest.part") { Remove-Item -LiteralPath "$dest.part" -Force }
|
|
112
|
+
Write-Host "FAILED $name ($($_.Exception.Message))" -ForegroundColor Red
|
|
113
|
+
$failed++
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
Write-Host ''
|
|
119
|
+
if ($missing.Count -gt 0) { Write-Host "Not published by OPM (skipped): $($missing -join ', ')" }
|
|
120
|
+
if ($saved.Count -gt 0) {
|
|
121
|
+
Write-Host 'Done. Upload these files to the chat (zip them first if they are large; do not rename them):' -ForegroundColor Green
|
|
122
|
+
foreach ($f in $saved) { Write-Host " $f" }
|
|
123
|
+
}
|
|
124
|
+
if ($failed -gt 0) { Write-Host "$failed file(s) failed: run this script again." -ForegroundColor Red; Finish 1 }
|
|
125
|
+
Finish 0
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# Download the OPM Federal Workforce Data files for this analysis into the folder that holds this script.
|
|
3
|
+
# (macOS, Linux, Git Bash, WSL; on Windows PowerShell use download_opm.ps1.)
|
|
4
|
+
#
|
|
5
|
+
# Usage: bash download_opm.sh
|
|
6
|
+
#
|
|
7
|
+
# - The request is built into this script (written by data-analyst-mcp; do not edit it).
|
|
8
|
+
# - It lists the files OPM publishes now (data.opm.gov, no account or key), picks the months the
|
|
9
|
+
# request asks for, and saves each one as <dataset>_<YYYYMM>_<version>.parquet.
|
|
10
|
+
# - Employment files are 26-75 MB each; accession and separation files are under 4 MB.
|
|
11
|
+
# - Needs only curl. Run it again after an interruption: files already saved are kept.
|
|
12
|
+
|
|
13
|
+
API="${OPM_API_BASE:-https://data.opm.gov/api/v1/files}"
|
|
14
|
+
API="${API%/}"
|
|
15
|
+
die() { echo "ERROR: $*" >&2; exit 1; }
|
|
16
|
+
command -v curl >/dev/null 2>&1 || die "curl is not installed. Install it (for example: sudo apt install curl) and run this script again."
|
|
17
|
+
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
18
|
+
|
|
19
|
+
# The request. Columns, tab-separated: dataset, then either
|
|
20
|
+
# periods YYYYMM,... (or "latest") the listed months
|
|
21
|
+
# range FROM|first TO|latest MONTHS every month from FROM to TO in MONTHS (1-12)
|
|
22
|
+
# pin YYYYMM VERSION use this version of one month instead of the current one
|
|
23
|
+
rules() {
|
|
24
|
+
cat <<'DATA_ANALYST_RULES'
|
|
25
|
+
@RULES@
|
|
26
|
+
DATA_ANALYST_RULES
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
# available DATASET: "YYYYMM VERSION" lines of the current files, oldest first
|
|
30
|
+
available() {
|
|
31
|
+
curl -fsS -A "data-analyst/1.0" "$API/$1?current=true" | tr -d '\r\n' \
|
|
32
|
+
| grep -o "\"filename\" *: *\"$1_[0-9]\{6\}_[0-9]*\"" \
|
|
33
|
+
| sed 's/.*_\([0-9]\{6\}\)_\([0-9]*\)"$/\1 \2/' | sort -u
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
SAVED=(); FAILED=0; MISSING=()
|
|
37
|
+
for ds in $(rules | cut -f1 | sort -u); do
|
|
38
|
+
[ -n "$ds" ] || continue
|
|
39
|
+
echo "Listing the $ds files..."
|
|
40
|
+
avail="$(available "$ds")" || die "Could not list the $ds files at $API. Check the connection, VPN and proxy, then run again."
|
|
41
|
+
[ -n "$avail" ] || die "OPM listed no $ds files. Run again later."
|
|
42
|
+
first="$(printf '%s\n' "$avail" | head -n 1 | cut -d' ' -f1)"
|
|
43
|
+
latest="$(printf '%s\n' "$avail" | tail -n 1 | cut -d' ' -f1)"
|
|
44
|
+
want=""
|
|
45
|
+
while IFS=$'\t' read -r rds kind a b c; do
|
|
46
|
+
[ "$rds" = "$ds" ] || continue
|
|
47
|
+
case "$kind" in
|
|
48
|
+
periods)
|
|
49
|
+
for p in ${a//,/ }; do
|
|
50
|
+
[ "$p" = latest ] && p="$latest"
|
|
51
|
+
if printf '%s\n' "$avail" | grep -q "^$p "; then want="$want $p"; else MISSING+=("$ds $p"); fi
|
|
52
|
+
done ;;
|
|
53
|
+
range)
|
|
54
|
+
lo="$a"; hi="$b"
|
|
55
|
+
[ "$lo" = first ] && lo="$first"
|
|
56
|
+
[ "$hi" = latest ] && hi="$latest"
|
|
57
|
+
while read -r p v; do
|
|
58
|
+
m=$((10#${p:4:2}))
|
|
59
|
+
[ "$p" -ge "$lo" ] && [ "$p" -le "$hi" ] && case ",$c," in *",$m,"*) want="$want $p" ;; esac
|
|
60
|
+
done <<< "$avail" ;;
|
|
61
|
+
esac
|
|
62
|
+
done < <(rules)
|
|
63
|
+
for p in $(printf '%s\n' $want | sort -u); do
|
|
64
|
+
v="$(rules | awk -F'\t' -v d="$ds" -v p="$p" '$1 == d && $2 == "pin" && $3 == p { print $4 }' | head -n 1)"
|
|
65
|
+
[ -n "$v" ] || v="$(printf '%s\n' "$avail" | grep "^$p " | head -n 1 | cut -d' ' -f2)"
|
|
66
|
+
name="${ds}_${p}_${v}.parquet"
|
|
67
|
+
if [ -s "$HERE/$name" ]; then echo "already saved: $name"; SAVED+=("$HERE/$name"); continue; fi
|
|
68
|
+
echo "Downloading $name ..."
|
|
69
|
+
progress=-sS; [ -t 2 ] && progress=-#
|
|
70
|
+
if curl "$progress" -fL -A "data-analyst/1.0" -o "$HERE/$name.part" "$API/$ds/${p:0:4}/${p:4:2}/$v/download"; then
|
|
71
|
+
mv -f "$HERE/$name.part" "$HERE/$name"; SAVED+=("$HERE/$name")
|
|
72
|
+
else
|
|
73
|
+
rm -f "$HERE/$name.part"; echo "FAILED $name"; FAILED=$((FAILED + 1))
|
|
74
|
+
fi
|
|
75
|
+
done
|
|
76
|
+
done
|
|
77
|
+
|
|
78
|
+
echo
|
|
79
|
+
[ ${#MISSING[@]} -gt 0 ] && echo "Not published by OPM (skipped): ${MISSING[*]}"
|
|
80
|
+
if [ ${#SAVED[@]} -gt 0 ]; then
|
|
81
|
+
echo "Done. Upload these files to the chat (zip them first if they are large; do not rename them):"
|
|
82
|
+
for f in "${SAVED[@]}"; do echo " $f"; done
|
|
83
|
+
fi
|
|
84
|
+
[ $FAILED -gt 0 ] && { echo "$FAILED file(s) failed: run this script again." >&2; exit 1; }
|
|
85
|
+
exit 0
|
|
@@ -0,0 +1,184 @@
|
|
|
1
|
+
# OPM workforce data: fields, codes and traps
|
|
2
|
+
|
|
3
|
+
Read §1 and §2 before choosing data for any analysis; they decide whether a question can be answered
|
|
4
|
+
at all. `data_dictionary.csv` (shipped with the pack, and copied into every `data/opm/` folder) has the
|
|
5
|
+
official definition of all 72 fields and which datasets carry each one.
|
|
6
|
+
|
|
7
|
+
## Contents
|
|
8
|
+
1. What the data are (and are not)
|
|
9
|
+
2. Traps that silently produce wrong numbers
|
|
10
|
+
3. Key fields and codes
|
|
11
|
+
4. Standard definitions
|
|
12
|
+
5. Columns to request for common questions
|
|
13
|
+
|
|
14
|
+
## 1. What the data are (and are not)
|
|
15
|
+
- **Source:** OPM's Enterprise Human Resources Integration (EHRI) system, fed by agencies' core HR
|
|
16
|
+
systems. **Administrative records for the whole covered workforce, not a survey**: no weights, no
|
|
17
|
+
sampling error. A difference of 50 employees is a real difference in the records; whether it is
|
|
18
|
+
meaningful depends on reporting quality, not on a confidence interval.
|
|
19
|
+
- **One row = one employee** (employment: on board on the last day of the month) or **one personnel
|
|
20
|
+
action** (accessions, separations). `count` is always 1. There is **no person identifier**, so you
|
|
21
|
+
cannot follow individuals across months, link a separation to the employee's earlier record, or
|
|
22
|
+
distinguish one person's two actions.
|
|
23
|
+
- **Coverage:** civilian employees of the executive branch, **except** USPS, the Postal Regulatory
|
|
24
|
+
Commission, CIA, NSA, DIA, NGA, ODNI, the White House Office, the Office and Residence of the Vice
|
|
25
|
+
President, Census Bureau temporary workers, Foreign Service personnel at State, TVA, the Federal
|
|
26
|
+
Reserve Board of Governors (CFPB *is* included, under the Federal Reserve), the PHS Commissioned
|
|
27
|
+
Corps, nonappropriated-fund employees, foreign nationals overseas, and most of the legislative and
|
|
28
|
+
judicial branches (a few bodies do report: the Government Publishing Office `LP` and some judicial-branch
|
|
29
|
+
records `JL` appear; drop them by `agency_code` for an executive-branch count and say so). Uniformed
|
|
30
|
+
military are not civilians and are not in the data. Totals are therefore below
|
|
31
|
+
"all federal employees" figures that include USPS (about 600,000) or the intelligence community.
|
|
32
|
+
- **Not in the public files:** sex, race, ethnicity, disability, citizenship, exact age (only 5-year
|
|
33
|
+
`age_bracket`), names, performance ratings, awards, telework, leave, bonuses, total compensation,
|
|
34
|
+
benefits, retirement eligibility dates, adverse-action detail (removals, probationary terminations
|
|
35
|
+
and suspensions are withheld by OPM policy). Say so plainly when a question needs them, and offer
|
|
36
|
+
what the data can show instead (for example age bracket and length of service as a proxy for
|
|
37
|
+
retirement eligibility).
|
|
38
|
+
|
|
39
|
+
## 2. Traps that silently produce wrong numbers
|
|
40
|
+
1. **Redacted pay and location (the big one).** `annualized_adjusted_basic_pay`, all `duty_station_*`
|
|
41
|
+
fields, CBSA/CSA, `locality_pay_area*`, `bargaining_unit*` and `personnel_office_identifier_code`
|
|
42
|
+
are the string `REDACTED` for:
|
|
43
|
+
- **every Department of Defense employee** (Army, Navy, Air Force, and defense agencies: about
|
|
44
|
+
680,000 people, a third of the workforce), in every year;
|
|
45
|
+
- security/sensitive agencies: FBI, DEA, ATF, TTB, Secret Service, CBP, ICE, Bureau of Prisons, U.S.
|
|
46
|
+
Attorneys' offices, DHS intelligence (HSOS04) — outside the Washington area;
|
|
47
|
+
- sensitive occupations in any agency: series 0007, 0082, 0083, 0084, 0132, 0134, 0401 (DHS), 0436,
|
|
48
|
+
0512, 0840, 0930, 1169, 1171 (IRS), 1801, 1802, 1805, 1810, 1811, 1812, 1816, 1854, 1881, 1884,
|
|
49
|
+
1890, 1895, 1896.
|
|
50
|
+
Result: about 45 to 48 percent of employment records lack pay and location (DHS about 76 percent,
|
|
51
|
+
Justice about 66 percent, Treasury and Energy about 17 percent, most others under 5 percent).
|
|
52
|
+
**Consequences:** a "governmentwide average salary" from these files is the average of non-DoD,
|
|
53
|
+
non-security employees, and state counts undercount every state with a big DoD presence. Always count
|
|
54
|
+
redacted records by the groups reported (`aggregate` gives `pay_redacted`), save that coverage table as an
|
|
55
|
+
exhibit, report the coverage next to every pay or location
|
|
56
|
+
figure, and restrict the stated universe ("employees whose pay is published, which excludes DoD …").
|
|
57
|
+
OPM's own "Average salary" (Table Builder, dashboards) is the mean over every record of the group whose
|
|
58
|
+
pay is published, on any work schedule; the default universe here (full-time, per annum) differs by up
|
|
59
|
+
to $10,000 a department: match OPM's definition when benchmarking against it, and show the other as a
|
|
60
|
+
sensitivity.
|
|
61
|
+
For full state counts (DoD included), OPM's Federal Workforce Data location dashboard
|
|
62
|
+
(data.opm.gov/explore-data/analytics/location) publishes them; FedScope's old state tables also included
|
|
63
|
+
DoD. Read them into `published/` and keep the file-based count as the covered-part check.
|
|
64
|
+
OPM's Table Builder and dashboards count separations and accessions by effective date: count the same
|
|
65
|
+
way to match them. Processing month differs a lot for FY2026, because about 11,000 Deferred Resignation
|
|
66
|
+
separations effective before October 2025 were processed in FY2026 files (it flips the sign of a
|
|
67
|
+
year-over-year comparison).
|
|
68
|
+
Read the map's state values: the dashboard's state table puts the D.C. commuting area in its own "D.C.
|
|
69
|
+
AREA" row, so its MD, VA and WV rows leave out their D.C.-area employees (Aug 2026: MD 54,097 in the
|
|
70
|
+
table, 121,612 on the map). The map has a "table view" (states 25 to a page) that lists the map's values;
|
|
71
|
+
the dashboard is a live app with no JSON to fetch, so read that view in a browser. The location map counts employees in pay status, 1 to 2 percent below the
|
|
72
|
+
files' on-board count; the Demographics dashboard (age, education, veteran status by department) and
|
|
73
|
+
Table Builder match the files exactly (Aug 2026), so they benchmark shares and headcounts by department.
|
|
74
|
+
Headcounts by department, occupation, grade, age, education and so on are complete — only pay and
|
|
75
|
+
location are withheld.
|
|
76
|
+
2. **Counting accessions and separations.** A file holds actions *processed* that month; 5 to 10
|
|
77
|
+
percent are effective in earlier months (some a year or more back). Two defensible bases:
|
|
78
|
+
- **effective month** (`personnel_action_effective_date_yyyymm`): what happened when; needs the files for later
|
|
79
|
+
months too, and the most recent two or three months are incomplete;
|
|
80
|
+
- **processing month** (`file_yyyymm`): what was recorded each month; complete as soon as the file
|
|
81
|
+
is out.
|
|
82
|
+
Pick one, state it, and show the other in a sensitivity check when they differ; comparing the two shows
|
|
83
|
+
how much arrives late.
|
|
84
|
+
3. **Versions and revisions.** Use `current` versions; numbers in an earlier report can differ because
|
|
85
|
+
OPM has since reissued the file. Pin versions (the spec's `pin`) to reproduce exactly.
|
|
86
|
+
4. **Pay plans change too.** Wildland firefighters moved from GS to their own pay plan GW in 2025 (about
|
|
87
|
+
16,000 to 17,000 employees at Agriculture and Interior): a GS-only series breaks there; count GS+GW for
|
|
88
|
+
a like-for-like trend. OPM's "pay plan and grade" dashboard table counts permanent full-time employees only
|
|
89
|
+
(`nsftp_indicator` = Y): filter the same way to match it.
|
|
90
|
+
5. **Agency names change; codes are stable.** For example, "DEPARTMENT OF DEFENSE" also appears as
|
|
91
|
+
"DEPARTMENT OF WAR" on OPM pages from 2025. Group by `department_code`, `agency_code`,
|
|
92
|
+
`agency_subelement_code`, `occupational_series_code`, and display the latest label
|
|
93
|
+
of each code. Reorganizations (new subelements, transfers between agencies) show up
|
|
94
|
+
as jumps: check before calling them hiring or losses. Occupational series move too: wildland firefighters
|
|
95
|
+
were reclassified from 0462 (Forestry Technician) into 0456 (Wildland Fire Management, created 2022), and
|
|
96
|
+
the 0x99 student-trainee series swing with internship cycles; check the receiving series before calling
|
|
97
|
+
a drop a loss.
|
|
98
|
+
6. **Snapshot frequency.** Employment files are quarterly before 2011. A fiscal-year average headcount
|
|
99
|
+
uses 12 monthly snapshots from FY2012 on and four before: flag fiscal years with fewer snapshots.
|
|
100
|
+
7. **Missing agency submissions.** An agency that misses a month shows a dip and a rebound. Check the
|
|
101
|
+
FWD Data Quality page ("Missing EHRI files") before reporting a one-month change.
|
|
102
|
+
8. **Sentinel values.** `NO DATA REPORTED`, `INVALID`, `UNSPECIFIED`, `*` appear in coded fields;
|
|
103
|
+
the data keep them as text: treat them as missing. Report how many records are missing for the fields you use.
|
|
104
|
+
A department code reported as missing on records with a valid agency code belongs to that agency's department
|
|
105
|
+
(each agency code belongs to one department): fill it from the agency, so department totals add up to the
|
|
106
|
+
governmentwide total, and say so.
|
|
107
|
+
9. **Pay oddities.** Pay is annualized basic pay plus locality or special-rate supplement, after the
|
|
108
|
+
pay cap — not total compensation (no overtime, awards, benefits). Records with pay 0 (without
|
|
109
|
+
compensation), hourly or per-diem pay bases, intermittent and seasonal schedules distort means.
|
|
110
|
+
Default pay universe: full-time (`work_schedule_code` F), per annum (`pay_basis_code` PA), pay > 0.
|
|
111
|
+
Physicians (VM/VN pay plans), some financial regulators and SES earn above
|
|
112
|
+
the GS cap legitimately; screen pay for implausible values but do not drop them without a reason.
|
|
113
|
+
10. **Strings everywhere.** Codes keep leading zeros (`occupational_series_code` "0346", `grade` "08").
|
|
114
|
+
Do not convert codes to integers.
|
|
115
|
+
11. **Separations on the last day of a month show up in the next snapshot.** An employee whose
|
|
116
|
+
separation takes effect on the last day of a month is still on board in that month's snapshot. A
|
|
117
|
+
wave of end-of-month separations (retirements on September 30 or December 31, or the 2025 Deferred
|
|
118
|
+
Resignation Program, effective September 30) therefore shows up in the following month's
|
|
119
|
+
headcount. In 2025: counted across the files available in October 2026, 132,949 separations were
|
|
120
|
+
effective in September (106,883 of them deferred resignations; earlier files showed fewer because
|
|
121
|
+
actions keep arriving late, see §2.2), but headcount fell 28,342 from August to September
|
|
122
|
+
and 92,685 from September to October. A September-to-September change misses such a wave. When separations bunch at the end of
|
|
123
|
+
the period's last month, also show the change to the next month's snapshot, or use December
|
|
124
|
+
snapshots, and say which you used.
|
|
125
|
+
|
|
126
|
+
## 3. Key fields and codes
|
|
127
|
+
| Field | Datasets | Notes |
|
|
128
|
+
|---|---|---|
|
|
129
|
+
| `snapshot_yyyymm` | E | month of the status snapshot |
|
|
130
|
+
| `personnel_action_effective_date_yyyymm` | A, S | effective month of the action |
|
|
131
|
+
| `department_code` / `department` | all | the 15 executive departments: AG Agriculture, CM Commerce, DOD Defense, ED Education, DN Energy, HE HHS, HS Homeland Security, HU HUD, DJ Justice, DL Labor, ST State, IN Interior, TR Treasury, TD Transportation, VA Veterans Affairs; independent agencies have their own codes (SZ SSA, EP EPA, ...) |
|
|
132
|
+
| `agency_code` / `agency` | all | e.g. AR Army, NV Navy, AF Air Force, VA, HS |
|
|
133
|
+
| `agency_subelement_code` | all | bureau or component, e.g. IN05 Bureau of Land Management, TR93 IRS |
|
|
134
|
+
| `cfo_act_agency_indicator` | all | CFO ACT AGENCY (24 large agencies) / OTHER AGENCY |
|
|
135
|
+
| `separation_category_code` | S | SA transfer out (individual), SB transfer out (mass), SC quit, SD retirement voluntary, SE retirement early out, SG retirement other, SH reduction in force (RIF), SJ termination (expired appointment/other), SL other separation |
|
|
136
|
+
| `drp_indicator` | S | Y = separated under the 2025 Deferred Resignation Program (about 96,000 processed in the September 2025 file alone; about 107,000 effective in September 2025 across all files as of October 2026; coded as quit or retirement) |
|
|
137
|
+
| `accession_category_code` | A | AA transfer in (individual), AB transfer in (mass), AC new hire competitive service, AD new hire excepted service, AE new hire SES |
|
|
138
|
+
| `work_schedule_code` | all | F full-time, P part-time, I intermittent, G full-time seasonal, Q part-time seasonal, J intermittent seasonal; the files also hold B, R, S, T and blank: read their labels with `--values work_schedule` and state the grouping (part-time = P, Q, S, T, R; intermittent = I, J) |
|
|
139
|
+
| `pay_plan_code` | all | GS General Schedule (~63 percent), WG/WL/WS wage grade, VN/VM VA medical, NH/ND DoD demo, AD administratively determined, SV TSA, ES Senior Executive Service |
|
|
140
|
+
| `grade`, `step_or_rate_type_code` | all | strings; GS grades "01" to "15" |
|
|
141
|
+
| `pay_basis_code` | all | PA per annum, PH per hour, SY school year, PD per diem, WC without compensation |
|
|
142
|
+
| `annualized_adjusted_basic_pay` | all | dollars as a string, or REDACTED |
|
|
143
|
+
| `occupational_series_code` | all | 4-digit OPM series ("2210" IT management); `occupational_group_code` 2-digit family; `occupational_category_code` P/A/T/C/O/B (professional, administrative, technical, clerical, other white collar, blue collar) |
|
|
144
|
+
| `stem_occupation_type` | all | science, technology, engineering, mathematics, health, all other |
|
|
145
|
+
| `supervisory_status_code` | all | 2 supervisor or manager, 4 supervisor (CSRA), 5 management official, 6 leader, 7 team leader, 8 all other |
|
|
146
|
+
| `appointment_type_code`, `tenure_code`, `position_occupied_code` | all | permanence and service: 10 career, 15 career-conditional, 30/32/35/38 excepted permanent, 50/55 SES; tenure 1 career, 2 conditional (probationary), 3 term; position 1 competitive, 2 excepted, 3/4 SES |
|
|
147
|
+
| `nsftp_indicator` | all | Y = nonseasonal full-time permanent (OPM's standard "permanent workforce" universe) |
|
|
148
|
+
| `age_bracket` | all | LESS THAN 20, 20-24, …, 60-64, 65 OR MORE |
|
|
149
|
+
| `education_level_bracket` | all | HIGH SCHOOL OR LESS, SOME COLLEGE OR ASSOCIATES DEGREE, BACHELORS DEGREE, MASTERS OR PROFESSIONAL DEGREE, DOCTORATE DEGREE |
|
|
150
|
+
| `length_of_service_years` | all | federal civilian plus creditable military service, years with one decimal. **Definition break in March 2026**: from the 2026-03 file it equals years since the leave service computation date for every record; before, for only 72-79 percent (a quarter of records ran about 4 years shorter). A change across March 2026 (+1.6 years Sep 2024 to Aug 2026 as published) is mostly this break: compare on one basis (years since `service_computation_date_leave`) and say so. OPM's Table Builder metric "Average length of service" follows the published field |
|
|
151
|
+
| `veteran_indicator` | all | Y / N, derived by OPM from several veteran fields |
|
|
152
|
+
| `bargaining_unit_status` | all | ELIGIBLE_IN_BU, ELIGIBLE_NOT_IN_BU, NOT_ELIGIBLE (or REDACTED) |
|
|
153
|
+
| `duty_station_state_abbreviation` | all | state of the worksite (REDACTED as in §2) |
|
|
154
|
+
|
|
155
|
+
## 4. Standard definitions
|
|
156
|
+
- **Headcount (on-board employment):** employees in the status file for the month. Annual series use
|
|
157
|
+
the **September** snapshot (end of the fiscal year; FedScope and OPM convention) unless the user
|
|
158
|
+
wants calendar-year (December) or another month. State the month. End-of-month separations leave
|
|
159
|
+
the headcount only in the next month's snapshot (§2.10).
|
|
160
|
+
- **Permanent workforce:** `nsftp_indicator` = Y (nonseasonal, full-time, permanent). Common in OPM
|
|
161
|
+
and CRS work when temporary and seasonal swings would mislead.
|
|
162
|
+
- **Separation rate:** separations in the period ÷ average headcount in the period (mean of the monthly
|
|
163
|
+
snapshots). Report quit, retirement, RIF and transfer rates separately when they matter; transfers
|
|
164
|
+
between agencies are separations for the losing agency but not losses to the government.
|
|
165
|
+
- **Accession (hiring) rate:** accessions ÷ average headcount. New hires = AC, AD, AE; transfers in are
|
|
166
|
+
not new to government.
|
|
167
|
+
- **Net change:** headcount at the end minus headcount at the start (from status files). Accessions
|
|
168
|
+
minus separations will not match it exactly (conversions, late actions, coverage changes); say so if
|
|
169
|
+
both are shown.
|
|
170
|
+
- **Average and median pay:** of full-time, per-annum employees whose pay is published (§2.1).
|
|
171
|
+
Prefer the median for comparisons across groups; quote coverage.
|
|
172
|
+
- **Retirement-eligible proxy:** no birth date or retirement plan is public. Age 60 or older with 20 or
|
|
173
|
+
more years of service, or age 62+ with 5+ years, approximates FERS eligibility; label it a proxy.
|
|
174
|
+
- **Fiscal year:** October through September (in `compute`, make `yyyymm` a `YYYY-MM` date with `substr`, then `fiscal_year`; FY2025 = October 2024 to
|
|
175
|
+
September 2025). Say "fiscal year 2025" in prose.
|
|
176
|
+
|
|
177
|
+
## 5. Columns to request for common questions
|
|
178
|
+
Keep the spec lean: ten September snapshots with all 64 columns are ~20 million rows.
|
|
179
|
+
- Headcount trends by agency: `department_code, department, agency_code, agency, nsftp_indicator, work_schedule_code`
|
|
180
|
+
- Composition (age, tenure, education): add `age_bracket, length_of_service_years, education_level_bracket, supervisory_status_code, veteran_indicator`
|
|
181
|
+
- Occupations: `occupational_series_code, occupational_series, occupational_category_code, stem_occupation_type, grade, pay_plan_code`
|
|
182
|
+
- Pay: `annualized_adjusted_basic_pay, pay_plan_code, grade, pay_basis_code, work_schedule_code, occupational_series_code, duty_station_state_abbreviation` (+ the department/agency codes to measure redaction)
|
|
183
|
+
- Turnover: separations `separation_category_code, separation_category, drp_indicator, agency_code, department_code, age_bracket, length_of_service_years`; accessions `accession_category_code, …`; and the employment snapshots of the same months for the denominator
|
|
184
|
+
- Location: `duty_station_state_abbreviation, duty_station_state, core_based_statistical_area` (redaction!)
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# OPM Federal Workforce Data API cheat sheet
|
|
2
|
+
|
|
3
|
+
This server's `fetch` for opm implements all of this. Read this file when debugging or
|
|
4
|
+
when you need something the spec does not expose. Source: https://data.opm.gov/get-data/api-access
|
|
5
|
+
(checked October 2026; the page is a JavaScript app, so read it in a browser, not with curl).
|
|
6
|
+
|
|
7
|
+
## Basics
|
|
8
|
+
- Base URL `https://data.opm.gov/api/v1/files`. **No authentication, no key, no account.**
|
|
9
|
+
- Data are public (OPM releases them under its Public Data Release Policy), so the filtered data may ship
|
|
10
|
+
with a report and its replication package.
|
|
11
|
+
- OPM releases each month's data about five weeks after the month ends (for example, August data on
|
|
12
|
+
October 7). The FWD Data Sources page shows the next release date.
|
|
13
|
+
|
|
14
|
+
## Endpoints
|
|
15
|
+
| What | Call | Returns |
|
|
16
|
+
|---|---|---|
|
|
17
|
+
| List files | `GET /api/v1/files/{dataset}` | JSON list: `{filename, publishDate, version, current, month, year}` (~75 KiB) |
|
|
18
|
+
| Download | `GET /api/v1/files/{dataset}/{year}/{month}/{version}/download` | one Parquet file |
|
|
19
|
+
|
|
20
|
+
`dataset` is `employment`, `accessions` or `separations`. `month` is two digits (`08`).
|
|
21
|
+
|
|
22
|
+
List filters (query string): `current=true|false`, `version`, `month` (`09`, `Sep`, `September`),
|
|
23
|
+
`year` (YYYY), `dataCapturedThrough` (MM/DD/YYYY), `publishedBefore` / `publishedAfter` (MM/DD/YYYY, inclusive).
|
|
24
|
+
|
|
25
|
+
Response codes: 200 ok; 204 no content; 400 malformed or over the 25 MB request limit; 404 file not
|
|
26
|
+
found (a wrong version returns "The specified blob does not exist."); 429 rate limit; 500 server error.
|
|
27
|
+
The client retries 429 and 5xx with backoff.
|
|
28
|
+
|
|
29
|
+
## Files
|
|
30
|
+
| Dataset | Content | Rows per file | Size | Periods |
|
|
31
|
+
|---|---|---|---|---|
|
|
32
|
+
| employment | status snapshot: every employee on board on the last day of the month | ~2.0-2.3 million | 26-75 MB | quarterly (Mar/Jun/Sep/Dec) 2005 to March 2011, monthly from June 2011 (no Jan, Feb, Apr, May 2011), latest about two months back |
|
|
33
|
+
| accessions | personnel actions that brought people in, **processed** in the month | 10k-60k | 0.06-1 MB | monthly from 2005-01 |
|
|
34
|
+
| separations | personnel actions that took people out, processed in the month | 10k-120k | 0.1-4 MB | monthly from 2005-01 |
|
|
35
|
+
|
|
36
|
+
- **Versions:** OPM reissues files when late or corrected agency submissions arrive (versions 1 to 5
|
|
37
|
+
seen). Exactly one version per month has `current: true`; use it unless reproducing earlier work, in
|
|
38
|
+
which case pin the versions from `data/opm/manifest.json` (the spec's `pin`, e.g. `{"employment_202509": 4}`).
|
|
39
|
+
- **Schema changes:** current versions have 64 (employment), 68 (accessions) and 69 (separations)
|
|
40
|
+
columns. Some older, superseded versions had only 31 columns, so always use current versions, and
|
|
41
|
+
check `missing_columns` in the manifest.
|
|
42
|
+
- Every column is a string in Parquet, including numbers (`count`, `annualized_adjusted_basic_pay`,
|
|
43
|
+
`length_of_service_years`). The CSV keeps them as published; in `compute`, a column that mixes numbers and
|
|
44
|
+
text (pay with `REDACTED`) is read as text: convert it with `number()`.
|
|
45
|
+
- Download speed seen: a 57 MB employment file in about 4 seconds. Ten years of September snapshots
|
|
46
|
+
(11 files, ~750 MB raw) take one to three minutes plus processing. Raw files are cached in
|
|
47
|
+
`<cache root>/opm-fwd` (`~/.cache/data-analyst/opm-fwd`; `$OPM_CACHE_DIR` overrides).
|
|
48
|
+
|
|
49
|
+
## Formats
|
|
50
|
+
- **API:** Parquet only. Query parameters such as `?format=txt` are ignored (checked October 2026).
|
|
51
|
+
Parquet is columnar and compressed: the August 2026 employment month is 57 MB for 2.0 million rows ×
|
|
52
|
+
64 columns.
|
|
53
|
+
- **Data Downloads web page** (https://data.opm.gov/get-data/data-downloads): each file's **Download**
|
|
54
|
+
menu offers **.TXT** and **.JSON** only. The download runs through the page's live server connection
|
|
55
|
+
(no plain URL to script), so these suit users clicking in a browser. Expect them to be several
|
|
56
|
+
times larger than Parquet. OPM warns that employment files exceed Excel's row limit. The exact TXT
|
|
57
|
+
delimiter was not confirmed; `ingest` sniffs it (comma, pipe, tab, semicolon, caret) and expects
|
|
58
|
+
field names in the first row.
|
|
59
|
+
- **Data dictionary:** an Excel file (`FWD Data Dictionary.xlsx`); this pack ships it as
|
|
60
|
+
`data_dictionary.csv` (copied into every `data/opm/` folder; `request_schema` lists its columns), so it
|
|
61
|
+
never needs downloading.
|
|
62
|
+
- **Reading Parquet:** `fetch` and `ingest` read the Parquet files themselves; no other software is needed.
|
|
63
|
+
|
|
64
|
+
## Other OPM tools (no API)
|
|
65
|
+
- **Table Builder** (https://data.opm.gov/explore-data/data/table-builder): aggregate tables in a browser;
|
|
66
|
+
the fallback when a user cannot download files at all.
|
|
67
|
+
- **Dashboards** ("Workforce Insights" pages): published headline figures to benchmark against.
|
|
68
|
+
- **FedScope** (fedscope.opm.gov) was the predecessor; FWD replaced it. Older reports cite FedScope's
|
|
69
|
+
September snapshots, which is why September is the conventional annual snapshot.
|
|
70
|
+
- **Data Quality page** (https://data.opm.gov/resources/data-quality): validation errors by agency and a
|
|
71
|
+
"Missing EHRI files" report. Check it when an agency's count jumps or drops.
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# Report patterns for federal workforce data products
|
|
2
|
+
|
|
3
|
+
Use this file to shape both the analysis and the report. The aim is a product a careful reviewer
|
|
4
|
+
(OPM, CRS, GAO, the Partnership for Public Service) would recognize: a precisely stated universe and
|
|
5
|
+
counting basis, numbered exhibits with notes and source lines, data shipped with the report, and the
|
|
6
|
+
limits of the public files stated plainly.
|
|
7
|
+
|
|
8
|
+
## Typical questions and how they are framed
|
|
9
|
+
These recur in published work, so there is usually a precedent to follow and a number to benchmark.
|
|
10
|
+
- **Size of the federal workforce** (status, September snapshots): total and by department or agency,
|
|
11
|
+
over 5 to 20 years; permanent (`nsftp_indicator`) vs. all; full-time vs. other schedules. Headline
|
|
12
|
+
style: "In September 2025, 2.2 million civilian employees worked for executive branch agencies covered
|
|
13
|
+
by OPM's data, about X percent fewer than a year earlier." Always say what EHRI excludes (USPS, the
|
|
14
|
+
intelligence community) because readers compare with "3 million federal workers" figures.
|
|
15
|
+
- **Workforce changes, 2025 onward** (separations and accessions): separations by type (quits,
|
|
16
|
+
retirements, RIFs, Deferred Resignation Program), hiring, net change by agency; rates per 100
|
|
17
|
+
employees. Separate the deferred resignation wave (`drp_indicator`; most effective in September 2025) from
|
|
18
|
+
normal attrition, and transfers between agencies from exits from government.
|
|
19
|
+
- **Retirement wave / aging:** age-bracket composition, share 55+ or 60+, length of service, the
|
|
20
|
+
retirement-eligible proxy (fields.md §4), by agency or occupation; retirement separation rates.
|
|
21
|
+
- **Occupations and skills:** IT (2210), cybersecurity proxies, STEM and health shares, mission-critical
|
|
22
|
+
series (e.g. 0602 physicians, 0610 nurses, 1811 criminal investigators — redacted pay!), supervisory
|
|
23
|
+
ratios (supervisors per employee).
|
|
24
|
+
- **Pay:** median and distribution by grade, pay plan, occupation or agency; GS grade structure; growth
|
|
25
|
+
in nominal and real terms (deflate with CPI-U or the ECI and cite it). Always quote redaction coverage
|
|
26
|
+
and avoid governmentwide pay averages.
|
|
27
|
+
- **Location:** employees by state or metro area, Washington-area share. Redaction removes DoD and
|
|
28
|
+
security staff, so present location only for the covered part, or by department excluding DoD, and say so.
|
|
29
|
+
- **Veterans, education, probationary employees** (tenure group 2), **bargaining unit coverage**.
|
|
30
|
+
|
|
31
|
+
## Benchmarks (compute the same definition and compare)
|
|
32
|
+
- FWD dashboards ("Workforce Size & Composition", "Workforce Changes", "Compensation") for the same month.
|
|
33
|
+
- OPM's earlier FedScope tables and "Sizing Up the Executive Branch" reports (September snapshots).
|
|
34
|
+
- Congressional Research Service reports on the size of the federal workforce; GAO reports on agency
|
|
35
|
+
workforce planning, retirements and the 2025 workforce reductions; Partnership for Public Service "Fed
|
|
36
|
+
Figures". BLS Current Employment Statistics "federal government" employment includes USPS, so expect
|
|
37
|
+
a gap of roughly 600,000 and explain it.
|
|
38
|
+
- OPM's subject reports (veterans employment, ...) can count only employees in pay status, fewer than the
|
|
39
|
+
snapshot's on-board total: the FY2024 veterans report gives 27.4 percent in Table 1 (in pay status) and
|
|
40
|
+
27.8 percent in Table 6 (on board, as the FWD files give it). Match the table to the universe you use.
|
|
41
|
+
Search with WebSearch, read the page, and cite it (`references` in report.json). Never cite from memory.
|
|
42
|
+
|
|
43
|
+
## Conventions to follow
|
|
44
|
+
**Population and definitions**
|
|
45
|
+
- State the universe: executive-branch civilian employees covered by EHRI; which snapshot month; all
|
|
46
|
+
vs. permanent (NSFTP) vs. full-time; any agency exclusions you made.
|
|
47
|
+
- State the counting basis for actions (effective month or processing month) and the period (fiscal or
|
|
48
|
+
calendar year).
|
|
49
|
+
- Rates use average headcount over the period as the denominator; say "per 100 employees" or "percent".
|
|
50
|
+
- Group by codes; show the latest name.
|
|
51
|
+
|
|
52
|
+
**Precision and suppression**
|
|
53
|
+
- Counts: round to the nearest hundred or thousand in prose ("about 447,200"), exact in tables. A
|
|
54
|
+
rounded number is its own fact: register it with `facts` and a rounding format (`num:-2` hundreds,
|
|
55
|
+
`num:-3` thousands) so the answer gate finds it.
|
|
56
|
+
- Flag rates and pay statistics for groups with fewer than 30 records and avoid publishing
|
|
57
|
+
pay for very small groups (individuals could be identified).
|
|
58
|
+
- No sampling error: do not write "statistically significant". Describe differences as differences in
|
|
59
|
+
the records, and discuss data-quality limits (redaction, late reports, missing submissions) instead.
|
|
60
|
+
|
|
61
|
+
**Publication structure (mirrored in `build_report`, `"mode": "report"`)**
|
|
62
|
+
The default layout, the **exhibit chartbook**, runs in this order:
|
|
63
|
+
1. **Cover:** "Month Year", the title (with the period for trend reports), and a hero chart.
|
|
64
|
+
2. **At a Glance** (one page, about 300-420 words): a framing paragraph (what was examined, the
|
|
65
|
+
universe, the period) and 3-5 bullets with a **bold lead phrase.** naming a section and the key numbers.
|
|
66
|
+
3. **Contents** and **List of Exhibits** (built automatically).
|
|
67
|
+
4. **Notes About This Report:** calendar vs. fiscal years; rounding; data source and universe (with the
|
|
68
|
+
EHRI exclusions); the snapshot month; the counting basis for actions; redaction of pay and location;
|
|
69
|
+
key definitions; a pointer to Appendix A.
|
|
70
|
+
5. **Introduction:** a lead paragraph with the headline numbers; question-style subheadings, one per
|
|
71
|
+
section; "What Data Were Used?"; "How Do These Figures Compare With Other Counts?" when relevant.
|
|
72
|
+
6. **Thematic sections** (2-5): a summary page, then one exhibit per page with 2-4 paragraphs beside it.
|
|
73
|
+
7. **Appendix A: Data and Methods:** files and versions (the manifest), universe, definitions, counting
|
|
74
|
+
basis, redaction coverage table, late-reporting check, benchmark comparison, limitations (§1 of
|
|
75
|
+
fields.md: what the public files cannot show).
|
|
76
|
+
8. **About This Document** (optional).
|
|
77
|
+
|
|
78
|
+
**Chart forms**
|
|
79
|
+
- Trends: line charts, at most 4-5 series (facet or pick the largest agencies plus "all other").
|
|
80
|
+
- Agencies or occupations at one point in time: horizontal bars, sorted.
|
|
81
|
+
- Composition (age brackets, separation types): stacked bars.
|
|
82
|
+
- Monthly actions around an event (e.g. September 2025): columns by month, with the event noted.
|
|
83
|
+
- No dual axes, pie charts or 3-D.
|