data-analyst-mcp 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +286 -262
- package/dist/adapters/bjs.d.ts +13 -0
- package/dist/adapters/bjs.js +28 -0
- package/dist/adapters/bjs.js.map +1 -0
- package/dist/adapters/cbp.d.ts +5 -0
- package/dist/adapters/cbp.js +64 -0
- package/dist/adapters/cbp.js.map +1 -0
- package/dist/adapters/cdc-wonder.d.ts +5 -0
- package/dist/adapters/cdc-wonder.js +68 -0
- package/dist/adapters/cdc-wonder.js.map +1 -0
- package/dist/adapters/cpsc.d.ts +4 -0
- package/dist/adapters/cpsc.js +125 -0
- package/dist/adapters/cpsc.js.map +1 -0
- package/dist/adapters/fhwa.d.ts +9 -0
- package/dist/adapters/fhwa.js +16 -0
- package/dist/adapters/fhwa.js.map +1 -0
- package/dist/adapters/foreign-assistance.d.ts +9 -0
- package/dist/adapters/foreign-assistance.js +42 -0
- package/dist/adapters/foreign-assistance.js.map +1 -0
- package/dist/adapters/hud-ahar.d.ts +3 -0
- package/dist/adapters/hud-ahar.js +80 -0
- package/dist/adapters/hud-ahar.js.map +1 -0
- package/dist/adapters/nces.d.ts +10 -0
- package/dist/adapters/nces.js +32 -0
- package/dist/adapters/nces.js.map +1 -0
- package/dist/adapters/nsf.d.ts +11 -0
- package/dist/adapters/nsf.js +25 -0
- package/dist/adapters/nsf.js.map +1 -0
- package/dist/adapters/onrr.d.ts +9 -0
- package/dist/adapters/onrr.js +35 -0
- package/dist/adapters/onrr.js.map +1 -0
- package/dist/adapters/sba.d.ts +10 -0
- package/dist/adapters/sba.js +134 -0
- package/dist/adapters/sba.js.map +1 -0
- package/dist/adapters/treasury-tic.d.ts +30 -0
- package/dist/adapters/treasury-tic.js +149 -0
- package/dist/adapters/treasury-tic.js.map +1 -0
- package/dist/adapters/uscourts.d.ts +8 -0
- package/dist/adapters/uscourts.js +8 -0
- package/dist/adapters/uscourts.js.map +1 -0
- package/dist/analysis/cite.d.ts +9 -1
- package/dist/analysis/cite.js +9 -2
- package/dist/analysis/cite.js.map +1 -1
- package/dist/bin/cli.d.ts +2 -0
- package/dist/bin/cli.js +17 -0
- package/dist/bin/cli.js.map +1 -1
- package/dist/core/generic-rest.js +42 -17
- package/dist/core/generic-rest.js.map +1 -1
- package/dist/core/guard.d.ts +32 -0
- package/dist/core/guard.js +176 -0
- package/dist/core/guard.js.map +1 -0
- package/dist/core/http.d.ts +10 -0
- package/dist/core/http.js +36 -7
- package/dist/core/http.js.map +1 -1
- package/dist/core/keys.d.ts +13 -3
- package/dist/core/keys.js +67 -7
- package/dist/core/keys.js.map +1 -1
- package/dist/core/package.d.ts +12 -3
- package/dist/core/package.js +56 -12
- package/dist/core/package.js.map +1 -1
- package/dist/core/secret_patterns.d.ts +6 -0
- package/dist/core/secret_patterns.js +20 -0
- package/dist/core/secret_patterns.js.map +1 -0
- package/dist/core/tabular.d.ts +43 -1
- package/dist/core/tabular.js +149 -6
- package/dist/core/tabular.js.map +1 -1
- package/dist/envelope.js +3 -1
- package/dist/envelope.js.map +1 -1
- package/dist/index.d.ts +29 -22
- package/dist/index.js +45 -46
- package/dist/index.js.map +1 -1
- package/dist/remote/app.js +87 -31
- package/dist/remote/app.js.map +1 -1
- package/dist/remote/oauth.js +8 -1
- package/dist/remote/oauth.js.map +1 -1
- package/dist/remote/pages.d.ts +3 -7
- package/dist/remote/pages.js +49 -19
- package/dist/remote/pages.js.map +1 -1
- package/dist/remote/sign.d.ts +7 -1
- package/dist/remote/sign.js +33 -7
- package/dist/remote/sign.js.map +1 -1
- package/dist/remote/store.js +14 -5
- package/dist/remote/store.js.map +1 -1
- package/dist/report/audit_report.js +29 -7
- package/dist/report/audit_report.js.map +1 -1
- package/dist/report/build_report.js +34 -4
- package/dist/report/build_report.js.map +1 -1
- package/dist/report/method_audit.js +10 -1
- package/dist/report/method_audit.js.map +1 -1
- package/dist/report/scan_secrets.d.ts +4 -1
- package/dist/report/scan_secrets.js +38 -18
- package/dist/report/scan_secrets.js.map +1 -1
- package/dist/server.d.ts +3 -1
- package/dist/server.js +46 -13
- package/dist/server.js.map +1 -1
- package/dist/tools.js +27 -6
- package/dist/tools.js.map +1 -1
- package/guide/WORKFLOW.md +134 -134
- package/guide/assets/disclaimer.txt +1 -1
- package/guide/assets/download_data.ps1 +164 -162
- package/guide/assets/download_data.sh +8 -2
- package/guide/assets/report_template.html +597 -597
- package/guide/references/routing.md +198 -171
- package/guide/references/style_guide.md +398 -398
- package/package.json +56 -56
- package/packs/acf/endpoints.json +114 -114
- package/packs/acf/policy.json +104 -104
- package/packs/actuaries/endpoints.json +82 -82
- package/packs/actuaries/policy.json +109 -109
- package/packs/bea/assets/download_bea.ps1 +182 -182
- package/packs/bea/docs/bea_api.md +461 -461
- package/packs/bea/docs/datasets.md +157 -157
- package/packs/bea/docs/report_patterns.md +112 -112
- package/packs/bea/policy.json +170 -170
- package/packs/bea/source.md +152 -152
- package/packs/bjs/endpoints.json +150 -0
- package/packs/bjs/policy.json +114 -0
- package/packs/bjs/source.md +98 -0
- package/packs/bls/assets/download_bls.ps1 +152 -152
- package/packs/bls/docs/analysis_patterns.md +108 -108
- package/packs/bls/docs/bls_api.md +73 -73
- package/packs/bls/docs/series_ids.md +149 -149
- package/packs/bls/policy.json +238 -238
- package/packs/bts/endpoints.json +346 -346
- package/packs/bts/policy.json +94 -94
- package/packs/cbp/endpoints.json +31 -0
- package/packs/cbp/policy.json +85 -0
- package/packs/cbp/source.md +57 -0
- package/packs/cdc/endpoints.json +197 -197
- package/packs/cdc/policy.json +83 -83
- package/packs/cdc/source.md +1 -1
- package/packs/cdc-wonder/endpoints.json +26 -0
- package/packs/cdc-wonder/policy.json +84 -0
- package/packs/cdc-wonder/source.md +53 -0
- package/packs/census/endpoints.json +99 -99
- package/packs/cfpb/endpoints.json +326 -326
- package/packs/cfpb/policy.json +119 -119
- package/packs/clinical-trials/endpoints.json +166 -166
- package/packs/clinical-trials/policy.json +84 -84
- package/packs/cms/endpoints.json +69 -69
- package/packs/cms/policy.json +96 -96
- package/packs/college-scorecard/endpoints.json +252 -252
- package/packs/college-scorecard/policy.json +106 -106
- package/packs/college-scorecard/source.md +1 -1
- package/packs/congress/endpoints.json +1701 -1701
- package/packs/congress/policy.json +117 -117
- package/packs/cpsc/endpoints.json +91 -0
- package/packs/cpsc/policy.json +91 -0
- package/packs/cpsc/source.md +76 -0
- package/packs/dhs-ohss/endpoints.json +66 -66
- package/packs/dhs-ohss/policy.json +107 -107
- package/packs/dhs-ohss/source.md +1 -1
- package/packs/doj-news/endpoints.json +59 -59
- package/packs/doj-news/policy.json +96 -96
- package/packs/doj-news/source.md +1 -1
- package/packs/dol/endpoints.json +817 -817
- package/packs/dol/policy.json +135 -135
- package/packs/eia/endpoints.json +717 -717
- package/packs/eia/policy.json +112 -112
- package/packs/epa/endpoints.json +291 -291
- package/packs/epa/policy.json +99 -99
- package/packs/epa-aqs/endpoints.json +173 -173
- package/packs/epa-aqs/policy.json +96 -96
- package/packs/fbi/endpoints.json +1841 -1841
- package/packs/fbi/policy.json +118 -118
- package/packs/fbi/source.md +3 -3
- package/packs/fda/endpoints.json +237 -237
- package/packs/fda/policy.json +121 -121
- package/packs/fdic/endpoints.json +416 -416
- package/packs/fdic/policy.json +83 -83
- package/packs/fec/endpoints.json +707 -707
- package/packs/fec/policy.json +93 -93
- package/packs/fed-board/endpoints.json +89 -89
- package/packs/fed-board/policy.json +136 -136
- package/packs/federal-register/endpoints.json +143 -143
- package/packs/federal-register/policy.json +86 -86
- package/packs/fema/endpoints.json +645 -645
- package/packs/fema/policy.json +103 -103
- package/packs/fhfa/endpoints.json +105 -105
- package/packs/fhfa/policy.json +109 -109
- package/packs/fhwa/endpoints.json +48 -0
- package/packs/fhwa/policy.json +91 -0
- package/packs/fhwa/source.md +59 -0
- package/packs/foreign-assistance/endpoints.json +166 -0
- package/packs/foreign-assistance/policy.json +80 -0
- package/packs/foreign-assistance/source.md +76 -0
- package/packs/fred/assets/download_fred.ps1 +122 -122
- package/packs/fred/docs/analysis_patterns.md +124 -124
- package/packs/fred/docs/fred_api.md +94 -94
- package/packs/fred/docs/series.md +188 -188
- package/packs/fred/policy.json +253 -253
- package/packs/fsa/endpoints.json +150 -150
- package/packs/fsa/policy.json +99 -99
- package/packs/fsa/source.md +1 -1
- package/packs/govinfo/endpoints.json +519 -519
- package/packs/gsa-calc/endpoints.json +93 -93
- package/packs/gsa-calc/policy.json +97 -97
- package/packs/hud/endpoints.json +146 -146
- package/packs/hud/policy.json +84 -84
- package/packs/hud/source.md +1 -1
- package/packs/hud-ahar/endpoints.json +92 -0
- package/packs/hud-ahar/policy.json +88 -0
- package/packs/hud-ahar/source.md +72 -0
- package/packs/ipums/docs/report_patterns.md +117 -117
- package/packs/irs-soi/endpoints.json +114 -114
- package/packs/irs-soi/policy.json +107 -107
- package/packs/medicaid/endpoints.json +412 -412
- package/packs/medicaid/policy.json +108 -108
- package/packs/naep/endpoints.json +217 -217
- package/packs/naep/policy.json +100 -100
- package/packs/naep/source.md +1 -1
- package/packs/nces/endpoints.json +101 -0
- package/packs/nces/policy.json +99 -0
- package/packs/nces/source.md +75 -0
- package/packs/nhtsa/endpoints.json +132 -132
- package/packs/nhtsa/policy.json +93 -93
- package/packs/nhtsa/source.md +1 -1
- package/packs/nih/endpoints.json +452 -452
- package/packs/nih/policy.json +86 -86
- package/packs/noaa/endpoints.json +431 -431
- package/packs/noaa/policy.json +95 -95
- package/packs/nrel/endpoints.json +327 -327
- package/packs/nrel/policy.json +96 -96
- package/packs/nsf/endpoints.json +33 -0
- package/packs/nsf/policy.json +83 -0
- package/packs/nsf/source.md +59 -0
- package/packs/nws/endpoints.json +271 -271
- package/packs/nws/policy.json +82 -82
- package/packs/nyfed/endpoints.json +80 -80
- package/packs/nyfed/policy.json +92 -92
- package/packs/omb/endpoints.json +460 -460
- package/packs/omb/policy.json +106 -106
- package/packs/onrr/endpoints.json +198 -0
- package/packs/onrr/policy.json +91 -0
- package/packs/onrr/source.md +55 -0
- package/packs/open-payments/endpoints.json +297 -297
- package/packs/open-payments/policy.json +101 -101
- package/packs/opm/assets/download_opm.ps1 +125 -125
- package/packs/opm/docs/fields.md +184 -184
- package/packs/opm/docs/opm_api.md +71 -71
- package/packs/opm/docs/report_patterns.md +83 -83
- package/packs/opm/lib/data_dictionary.csv +73 -73
- package/packs/pbgc/endpoints.json +483 -483
- package/packs/pbgc/policy.json +94 -94
- package/packs/philfed/endpoints.json +85 -85
- package/packs/philfed/policy.json +94 -94
- package/packs/port.json +65 -0
- package/packs/regulations/endpoints.json +287 -287
- package/packs/regulations/policy.json +86 -86
- package/packs/sba/endpoints.json +34 -0
- package/packs/sba/policy.json +86 -0
- package/packs/sba/source.md +60 -0
- package/packs/sec/endpoints.json +236 -236
- package/packs/sec/policy.json +86 -86
- package/packs/senate-lobbying/endpoints.json +553 -553
- package/packs/senate-lobbying/policy.json +99 -99
- package/packs/snapqc/endpoints.json +1426 -1426
- package/packs/snapqc/policy.json +92 -92
- package/packs/treasury/catalog.json +2267 -2267
- package/packs/treasury/docs/api_guide.md +245 -245
- package/packs/treasury/docs/datasets.md +56 -56
- package/packs/treasury/docs/report_patterns.md +126 -126
- package/packs/treasury/policy.json +159 -159
- package/packs/treasury/source.md +170 -165
- package/packs/treasury-tic/endpoints.json +77 -0
- package/packs/treasury-tic/policy.json +89 -0
- package/packs/treasury-tic/source.md +80 -0
- package/packs/treasury-yields/endpoints.json +226 -226
- package/packs/treasury-yields/policy.json +86 -86
- package/packs/usaspending/endpoints.json +1315 -1315
- package/packs/usaspending/policy.json +109 -109
- package/packs/uscourts/endpoints.json +22 -0
- package/packs/uscourts/policy.json +87 -0
- package/packs/uscourts/source.md +56 -0
- package/packs/usda-fna/policy.json +133 -133
- package/packs/usda-fooddata/endpoints.json +180 -180
- package/packs/usda-fooddata/policy.json +94 -94
- package/packs/usda-nass/endpoints.json +323 -323
- package/packs/usda-nass/policy.json +95 -95
- package/packs/usda-outlook/endpoints.json +86 -86
- package/packs/usda-outlook/policy.json +109 -109
- package/packs/usgs/endpoints.json +313 -313
- package/packs/usgs/policy.json +95 -95
- package/packs/usitc/endpoints.json +74 -74
- package/packs/usitc/policy.json +111 -111
- package/packs/uspto/endpoints.json +141 -141
- package/packs/uspto/policy.json +95 -95
- package/packs/uspto/source.md +1 -1
- package/packs/va/endpoints.json +414 -414
- package/packs/va/policy.json +101 -101
- package/packs/world-bank/endpoints.json +122 -122
- package/packs/world-bank/policy.json +101 -101
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# Bureau of Justice Statistics: National Crime Victimization Survey and corrections statistics
|
|
2
|
+
Depth: standard. Key: none. Two kinds of data:
|
|
3
|
+
- NCVS Select records from BJS's API at api.ojp.gov: the person and household files of the National Crime
|
|
4
|
+
Victimization Survey, 1993-2024 (collection years), one row per victimization, person or household, with
|
|
5
|
+
weights.
|
|
6
|
+
- The CSV tables of BJS reports from bjs.ojp.gov: Criminal Victimization (official NCVS numbers, rates and
|
|
7
|
+
standard errors), Prisoners (National Prisoner Statistics), Jail Inmates, Probation and Parole, Capital
|
|
8
|
+
Punishment.
|
|
9
|
+
|
|
10
|
+
## When to use
|
|
11
|
+
- Crime as victims report it, including crimes never reported to police: violent and property victimization
|
|
12
|
+
numbers and rates; the share reported to police; victim characteristics (age, sex, race, income); weapon,
|
|
13
|
+
injury and victim-offender relationship.
|
|
14
|
+
- People in prison or jail, on probation or parole, under sentence of death; imprisonment rates by state.
|
|
15
|
+
- Not here: crimes known to police, arrests and hate crimes (fbi); federal prosecutions and DOJ press releases
|
|
16
|
+
(doj-news); homicide (NCVS covers nonfatal crimes only; deaths by assault are in cdc); state court cases.
|
|
17
|
+
|
|
18
|
+
## Endpoints
|
|
19
|
+
| Endpoint | Returns | Parameters |
|
|
20
|
+
|---|---|---|
|
|
21
|
+
| `query` | NCVS Select records (JSON rows, every value text) | `dataset` (catalog alias or 4x4 id), `$select` (plain columns), `$where`, `$order` |
|
|
22
|
+
| `statistical_table` | one CSV table of a report package | `report` (code or alias), `member` (file, e.g. `cv24t01.csv`), `header_lines` or `header_row` |
|
|
23
|
+
|
|
24
|
+
NCVS Select datasets: `personal_victimization` (gcuy-rt5g), `personal_population` (r4j4-fdwx),
|
|
25
|
+
`household_victimization` (gkck-euys), `household_population` (ya4e-n9zp). Report packages (verified
|
|
26
|
+
2026-11-07): `cv24` and `cv23` (Criminal Victimization, 2024 and 2023), `p23st` (Prisoners in 2023), `ji24st`
|
|
27
|
+
and `ji23st` (Jail Inmates), `ppus23` (Probation and Parole, 2023), `cp22st` (Capital Punishment, 2022). In a
|
|
28
|
+
package, `<code>t01.csv` is table 1, `at01` appendix table 1 (estimates with standard errors and confidence
|
|
29
|
+
intervals), `f01` figure 1. A `member` that matches nothing fails with the list of files in the zip.
|
|
30
|
+
|
|
31
|
+
## Spec examples
|
|
32
|
+
```json
|
|
33
|
+
{"description": "Violent victimization numbers and rates, 2020-2024, as BJS published them",
|
|
34
|
+
"requests": [{"name": "cv24_t01", "endpoint": "statistical_table",
|
|
35
|
+
"params": {"report": "cv24", "member": "cv24t01.csv", "header_lines": "11-12"}}]}
|
|
36
|
+
```
|
|
37
|
+
```json
|
|
38
|
+
{"description": "2024 violent victimizations by type and reporting to police, from NCVS Select records",
|
|
39
|
+
"requests": [{"name": "violent_2024", "endpoint": "query", "params": {"dataset": "personal_victimization",
|
|
40
|
+
"$select": "year, newoff, notify, newwgt", "$where": "year = '2024' AND newcrime = '1'"}},
|
|
41
|
+
{"name": "persons_2024", "endpoint": "query", "params": {"dataset": "personal_population",
|
|
42
|
+
"$select": "year, wgtpercy", "$where": "year = '2024'"}}]}
|
|
43
|
+
```
|
|
44
|
+
Then `compute`: the sum of `newwgt` by `newoff` gives victimizations. Divide by the sum of `wgtpercy` for the
|
|
45
|
+
year and multiply by 1,000 for the rate per 1,000 persons age 12 or older.
|
|
46
|
+
|
|
47
|
+
## Data format and traps
|
|
48
|
+
- Report tables (`statistical_table`): the first fetch, without `header_lines`, gives every line numbered (`row`,
|
|
49
|
+
`col1`...). Lines 1-8 are the BJS banner, file name, title, report, data source, authors and version date;
|
|
50
|
+
the title repeats above the header. Read the header line numbers, then fetch again with `header_lines` (e.g.
|
|
51
|
+
`11-12`). Column names join the stacked header (`2024* | Number`, `2024* | Rate per 1,000/a`). A column of
|
|
52
|
+
footnote marks next to a value is named `<parent>_colN`: `†` or `‡` mean the difference from the comparison year
|
|
53
|
+
(marked `*`, e.g. `2024*`) is significant at the 95% or 90% level (table 1's notes).
|
|
54
|
+
Values are text with thousands separators (`6,671,640`). Row labels carry footnote letters (`Total violent
|
|
55
|
+
crime/b`), and the notes below the table explain them. Sub-rows are indented into the next column (`col2`).
|
|
56
|
+
Keep the footnote and significance notes with any number you quote.
|
|
57
|
+
- NCVS codes, from BJS's NCVS Select codebooks (person and household level):
|
|
58
|
+
- `newcrime` (person) is 1 violent crime, 2 personal theft/larceny. `newoff` (person) is 1 rape/sexual
|
|
59
|
+
assault, 2 robbery, 3 aggravated assault, 4 simple assault, 5 personal theft/larceny. `seriousviolent` 1
|
|
60
|
+
is violent crime excluding simple assault.
|
|
61
|
+
- Household: `newcrime` 3 is property crime; `newoff` is 6 burglary/trespassing, 7 motor-vehicle theft,
|
|
62
|
+
8 theft.
|
|
63
|
+
- Demographic codes (`ager` 1-6, `sex`, `race_ethnicity` 1-6, `hincome1`, ...), the `notify` codes (reported to
|
|
64
|
+
police) and residue codes (98, -1 "invalid until ...") are in the codebooks:
|
|
65
|
+
https://bjs.ojp.gov/document/NCVS_Select_person_level_codebook.pdf and
|
|
66
|
+
https://bjs.ojp.gov/document/NCVS_Select_household_level_codebook.pdf. Read the code before filtering on it.
|
|
67
|
+
- Weights:
|
|
68
|
+
- `newwgt` is the series-adjusted victimization weight; use it for victimization counts (`wgtviccy` is the
|
|
69
|
+
unadjusted annual weight).
|
|
70
|
+
- `wgtpercy` (persons) and `wgthhcy` (households) are the population weights, the rate denominators.
|
|
71
|
+
- For prevalence (unique victims), use `wgt_ovam_per` and the `victim` flag.
|
|
72
|
+
- Never count rows: each row stands for thousands of victimizations.
|
|
73
|
+
- Reproducing BJS: the sum of `newwgt` over 2024 violent records is 6,671,636, and BJS publishes 6,671,640
|
|
74
|
+
(rounded to tens). The types match the same way (rape/sexual assault 560,890; robbery 642,150). Standard errors
|
|
75
|
+
and significance tests need the design variables (`pseudostratum`, `secucode`) and BJS's generalized variance
|
|
76
|
+
method; they are not computed here. Take them from the report's appendix tables, or say that none were computed.
|
|
77
|
+
- 2024 is the legacy-instrument half of a split sample (109,341 persons). Weights are larger and estimates less
|
|
78
|
+
precise (the codebook). Collection year: crimes are counted in the interview year, not the
|
|
79
|
+
year they happened. Victimizations outside the U.S. are excluded.
|
|
80
|
+
- Every column is text: quote values in `$where` (`year = '2024'`). Aggregate SoQL (`sum`, `count`, `$group`)
|
|
81
|
+
is refused before the request is sent, because the site answers it with a bot check. Fetch plain columns and
|
|
82
|
+
aggregate with `compute`. A year of `personal_population` has well over 100,000 rows, so select only the
|
|
83
|
+
columns you need.
|
|
84
|
+
- Small cells: a weighted sum over a handful of records is not a reliable estimate. When a tabulation splits a
|
|
85
|
+
year finely (by place size, race and crime type), say how many records are behind each cell, and prefer the
|
|
86
|
+
report tables, whose notes flag unreliable estimates. The person populations behind the rates are in
|
|
87
|
+
Criminal Victimization appendix tables (2024: appendix table 19).
|
|
88
|
+
|
|
89
|
+
## Access
|
|
90
|
+
No key. `fetch` uses api.ojp.gov (records) and bjs.ojp.gov (report zips). Keep requests few: the site has a bot
|
|
91
|
+
check. Manual: `https://api.ojp.gov/bjsdataset/v1/<id>.csv?$where=year='2024'&$limit=200000` in a browser, or
|
|
92
|
+
the "Data tables" zip on the report's page (https://bjs.ojp.gov/library/publications/...), then `ingest`. API
|
|
93
|
+
documentation: https://bjs.ojp.gov/national-crime-victimization-survey-ncvs-api.
|
|
94
|
+
|
|
95
|
+
## Citation and terms
|
|
96
|
+
"U.S. Department of Justice, Bureau of Justice Statistics, National Crime Victimization Survey, NCVS Select
|
|
97
|
+
<dataset> (or <report title and NCJ number>, table <n>), bjs.ojp.gov, retrieved <date>." Public domain (17 U.S.C.
|
|
98
|
+
105). A tabulation of NCVS Select records is the analyst's own estimate, not a BJS estimate: say so.
|
|
@@ -1,152 +1,152 @@
|
|
|
1
|
-
<#
|
|
2
|
-
.SYNOPSIS
|
|
3
|
-
Download BLS time series with the BLS Public Data API: send the requests built into this script and
|
|
4
|
-
save the replies as response_*.json next to it. (Windows PowerShell counterpart of download_bls.sh.)
|
|
5
|
-
|
|
6
|
-
.DESCRIPTION
|
|
7
|
-
- Needs no other file: the requests are inside this script.
|
|
8
|
-
- When the requests need a BLS registration key (API v2), uses $env:BLS_API_KEY when it is set and
|
|
9
|
-
otherwise asks for the key (hidden input). The key is only sent to api.bls.gov: it is never shown,
|
|
10
|
-
saved, or put on a command line. With a key set, keyless requests also go through API v2.
|
|
11
|
-
- Saves response_01.json, response_02.json, ... next to this script, each with the request it answers
|
|
12
|
-
(without the key), and prints their full paths at the end.
|
|
13
|
-
- Needs nothing beyond Windows PowerShell 5.1 (built into Windows 10 and 11) or PowerShell 7.
|
|
14
|
-
|
|
15
|
-
Get a free key at https://data.bls.gov/registrationEngine/ (BLS emails it within minutes).
|
|
16
|
-
|
|
17
|
-
.PARAMETER NoPause
|
|
18
|
-
Do not wait for Enter before closing the window.
|
|
19
|
-
|
|
20
|
-
.EXAMPLE
|
|
21
|
-
powershell -NoProfile -ExecutionPolicy Bypass -File "$HOME\Downloads\download_bls.ps1"
|
|
22
|
-
#>
|
|
23
|
-
[CmdletBinding()]
|
|
24
|
-
param(
|
|
25
|
-
[switch]$NoPause
|
|
26
|
-
)
|
|
27
|
-
|
|
28
|
-
$ErrorActionPreference = 'Stop'
|
|
29
|
-
$ProgressPreference = 'SilentlyContinue' # Invoke-WebRequest is very slow with the progress bar
|
|
30
|
-
try {
|
|
31
|
-
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
|
32
|
-
} catch { }
|
|
33
|
-
|
|
34
|
-
$Here = $PSScriptRoot
|
|
35
|
-
if (-not $Here) { $Here = (Get-Location).Path }
|
|
36
|
-
$Utf8 = New-Object Text.UTF8Encoding($false)
|
|
37
|
-
$Key = $null
|
|
38
|
-
$Req = $null
|
|
39
|
-
|
|
40
|
-
# Keep the window open when the script was started with "Run with PowerShell".
|
|
41
|
-
function Finish([int]$Code) {
|
|
42
|
-
$script:Key = $null
|
|
43
|
-
if ($script:Req) { Remove-Item -LiteralPath $script:Req -Recurse -Force -ErrorAction SilentlyContinue }
|
|
44
|
-
if (-not $NoPause -and -not [Console]::IsInputRedirected) {
|
|
45
|
-
Write-Host ''
|
|
46
|
-
[void](Read-Host -Prompt 'Press Enter to close')
|
|
47
|
-
}
|
|
48
|
-
exit $Code
|
|
49
|
-
}
|
|
50
|
-
|
|
51
|
-
function Fail([string]$Message) {
|
|
52
|
-
Write-Host ''
|
|
53
|
-
Write-Host "ERROR: $(Hide-Key $Message)" -ForegroundColor Red
|
|
54
|
-
Finish 1
|
|
55
|
-
}
|
|
56
|
-
|
|
57
|
-
# BLS repeats a rejected key in its error message: never let it reach the screen or a file.
|
|
58
|
-
function Hide-Key([string]$Text) {
|
|
59
|
-
if ($script:Key -and $Text) { return $Text.Replace($script:Key, '[REDACTED]') }
|
|
60
|
-
return $Text
|
|
61
|
-
}
|
|
62
|
-
|
|
63
|
-
# ---------------------------------------------------------------------------- the requests
|
|
64
|
-
# manifest.json and query_NN.json are built into this script; they go to a temporary folder that is
|
|
65
|
-
# removed when the script ends.
|
|
66
|
-
$Embedded = [ordered]@{}
|
|
67
|
-
# @QUERIES@
|
|
68
|
-
$Req = Join-Path ([IO.Path]::GetTempPath()) ('download_bls.' + [IO.Path]::GetRandomFileName())
|
|
69
|
-
[void](New-Item -ItemType Directory -Path $Req)
|
|
70
|
-
foreach ($name in $Embedded.Keys) { [IO.File]::WriteAllText((Join-Path $Req $name), $Embedded[$name], $Utf8) }
|
|
71
|
-
$manifestFile = Join-Path $Req 'manifest.json'
|
|
72
|
-
if (-not (Test-Path -LiteralPath $manifestFile)) {
|
|
73
|
-
Fail "This script has no requests built in. Use the download_bls.ps1 from the analyst's download files."
|
|
74
|
-
}
|
|
75
|
-
$Manifest = [IO.File]::ReadAllText($manifestFile, $Utf8) | ConvertFrom-Json
|
|
76
|
-
$Queries = @(Get-ChildItem -LiteralPath $Req -Filter 'query_*.json' | Sort-Object Name)
|
|
77
|
-
if ($Queries.Count -eq 0) { Fail "This script has no requests built in. Use the download_bls.ps1 from the analyst's download files." }
|
|
78
|
-
$Bodies = @()
|
|
79
|
-
foreach ($q in $Queries) { $Bodies += ,([IO.File]::ReadAllText($q.FullName, $Utf8) | ConvertFrom-Json) }
|
|
80
|
-
|
|
81
|
-
$needKey = ($Manifest.api_version -eq 'v2')
|
|
82
|
-
foreach ($b in $Bodies) { if ($b.PSObject.Properties['registrationkey']) { $needKey = $true } }
|
|
83
|
-
|
|
84
|
-
# ---------------------------------------------------------------------------- API key
|
|
85
|
-
$Key = $env:BLS_API_KEY
|
|
86
|
-
if ($Key) {
|
|
87
|
-
Write-Host 'Using the BLS key in $env:BLS_API_KEY.'
|
|
88
|
-
} elseif ($needKey) {
|
|
89
|
-
Write-Host 'These requests need a free BLS registration key (https://data.bls.gov/registrationEngine/).'
|
|
90
|
-
if ([Console]::IsInputRedirected) {
|
|
91
|
-
$Key = [Console]::In.ReadLine()
|
|
92
|
-
} else {
|
|
93
|
-
$secure = Read-Host -Prompt 'Paste your BLS key (input is hidden), then press Enter' -AsSecureString
|
|
94
|
-
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
|
95
|
-
try { $Key = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
|
96
|
-
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
|
97
|
-
Remove-Variable secure
|
|
98
|
-
}
|
|
99
|
-
}
|
|
100
|
-
if ($null -eq $Key) { $Key = '' }
|
|
101
|
-
$Key = $Key -replace '\s', ''
|
|
102
|
-
if ($needKey -and -not $Key) { Fail 'No key entered. Get one at https://data.bls.gov/registrationEngine/' }
|
|
103
|
-
|
|
104
|
-
$Version = 'v1'
|
|
105
|
-
if ($Key -and $Manifest.urls.v2) { $Version = 'v2' }
|
|
106
|
-
$Url = [string]$Manifest.urls.$Version
|
|
107
|
-
Write-Host "Sending $($Queries.Count) request(s) to BLS API $Version ..."
|
|
108
|
-
|
|
109
|
-
# ---------------------------------------------------------------------------- send each request
|
|
110
|
-
$Saved = @()
|
|
111
|
-
for ($i = 0; $i -lt $Queries.Count; $i++) {
|
|
112
|
-
$q = $Queries[$i]
|
|
113
|
-
$body = $Bodies[$i]
|
|
114
|
-
$body.PSObject.Properties.Remove('registrationkey')
|
|
115
|
-
if ($Version -eq 'v2' -and $Manifest.v2_options) {
|
|
116
|
-
foreach ($p in $Manifest.v2_options.PSObject.Properties) {
|
|
117
|
-
$body | Add-Member -NotePropertyName $p.Name -NotePropertyValue $p.Value -Force
|
|
118
|
-
}
|
|
119
|
-
}
|
|
120
|
-
$requestJson = $body | ConvertTo-Json -Depth 10 -Compress # the request without the key
|
|
121
|
-
$body | Add-Member -NotePropertyName 'registrationkey' -NotePropertyValue $Key -Force
|
|
122
|
-
if ($Version -ne 'v2') { $body.PSObject.Properties.Remove('registrationkey') }
|
|
123
|
-
$sendJson = $body | ConvertTo-Json -Depth 10 -Compress # in memory only
|
|
124
|
-
$body.PSObject.Properties.Remove('registrationkey')
|
|
125
|
-
|
|
126
|
-
try {
|
|
127
|
-
$resp = Invoke-WebRequest -Uri $Url -Method Post -Body $Utf8.GetBytes($sendJson) `
|
|
128
|
-
-ContentType 'application/json' -UserAgent 'data-analyst/1.0' -UseBasicParsing -TimeoutSec 120
|
|
129
|
-
} catch {
|
|
130
|
-
Fail "$($q.Name): the request to api.bls.gov failed ($($_.Exception.Message)). Check the internet connection, VPN and proxy, then run this script again."
|
|
131
|
-
}
|
|
132
|
-
$sendJson = $null
|
|
133
|
-
$text = $Utf8.GetString($resp.RawContentStream.ToArray()).Trim()
|
|
134
|
-
try { $reply = $text | ConvertFrom-Json } catch { $reply = $null }
|
|
135
|
-
if ($null -eq $reply -or -not $text.EndsWith('}') -or $reply.PSObject.Properties.Count -eq 0) {
|
|
136
|
-
Fail "$($q.Name): BLS did not answer with JSON. The reply began: $($text.Substring(0, [Math]::Min(200, $text.Length)))"
|
|
137
|
-
}
|
|
138
|
-
# Keep the reply exactly as BLS sent it and add the request it answers, like download_bls.sh.
|
|
139
|
-
$text = $text.Substring(0, $text.LastIndexOf('}')).TrimEnd() + ",`n `"_request`": " + $requestJson + "`n}"
|
|
140
|
-
$dest = Join-Path $Here ($q.Name -replace '^query_', 'response_')
|
|
141
|
-
[IO.File]::WriteAllText($dest, (Hide-Key $text) + "`n", $Utf8)
|
|
142
|
-
$Saved += $dest
|
|
143
|
-
$msg = Hide-Key ((@($reply.message) | Where-Object { $_ }) -join '; ')
|
|
144
|
-
if ($msg.Length -gt 200) { $msg = $msg.Substring(0, 200) }
|
|
145
|
-
Write-Host "$(Split-Path -Leaf $dest) $($reply.status) $msg"
|
|
146
|
-
if ($i -lt $Queries.Count - 1) { Start-Sleep -Milliseconds 500 }
|
|
147
|
-
}
|
|
148
|
-
|
|
149
|
-
Write-Host ''
|
|
150
|
-
Write-Host 'Done. Upload these files to the chat (do not edit them):' -ForegroundColor Green
|
|
151
|
-
foreach ($p in $Saved) { Write-Host " $p" }
|
|
152
|
-
Finish 0
|
|
1
|
+
<#
|
|
2
|
+
.SYNOPSIS
|
|
3
|
+
Download BLS time series with the BLS Public Data API: send the requests built into this script and
|
|
4
|
+
save the replies as response_*.json next to it. (Windows PowerShell counterpart of download_bls.sh.)
|
|
5
|
+
|
|
6
|
+
.DESCRIPTION
|
|
7
|
+
- Needs no other file: the requests are inside this script.
|
|
8
|
+
- When the requests need a BLS registration key (API v2), uses $env:BLS_API_KEY when it is set and
|
|
9
|
+
otherwise asks for the key (hidden input). The key is only sent to api.bls.gov: it is never shown,
|
|
10
|
+
saved, or put on a command line. With a key set, keyless requests also go through API v2.
|
|
11
|
+
- Saves response_01.json, response_02.json, ... next to this script, each with the request it answers
|
|
12
|
+
(without the key), and prints their full paths at the end.
|
|
13
|
+
- Needs nothing beyond Windows PowerShell 5.1 (built into Windows 10 and 11) or PowerShell 7.
|
|
14
|
+
|
|
15
|
+
Get a free key at https://data.bls.gov/registrationEngine/ (BLS emails it within minutes).
|
|
16
|
+
|
|
17
|
+
.PARAMETER NoPause
|
|
18
|
+
Do not wait for Enter before closing the window.
|
|
19
|
+
|
|
20
|
+
.EXAMPLE
|
|
21
|
+
powershell -NoProfile -ExecutionPolicy Bypass -File "$HOME\Downloads\download_bls.ps1"
|
|
22
|
+
#>
|
|
23
|
+
[CmdletBinding()]
|
|
24
|
+
param(
|
|
25
|
+
[switch]$NoPause
|
|
26
|
+
)
|
|
27
|
+
|
|
28
|
+
$ErrorActionPreference = 'Stop'
|
|
29
|
+
$ProgressPreference = 'SilentlyContinue' # Invoke-WebRequest is very slow with the progress bar
|
|
30
|
+
try {
|
|
31
|
+
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
|
32
|
+
} catch { }
|
|
33
|
+
|
|
34
|
+
$Here = $PSScriptRoot
|
|
35
|
+
if (-not $Here) { $Here = (Get-Location).Path }
|
|
36
|
+
$Utf8 = New-Object Text.UTF8Encoding($false)
|
|
37
|
+
$Key = $null
|
|
38
|
+
$Req = $null
|
|
39
|
+
|
|
40
|
+
# Keep the window open when the script was started with "Run with PowerShell".
|
|
41
|
+
function Finish([int]$Code) {
|
|
42
|
+
$script:Key = $null
|
|
43
|
+
if ($script:Req) { Remove-Item -LiteralPath $script:Req -Recurse -Force -ErrorAction SilentlyContinue }
|
|
44
|
+
if (-not $NoPause -and -not [Console]::IsInputRedirected) {
|
|
45
|
+
Write-Host ''
|
|
46
|
+
[void](Read-Host -Prompt 'Press Enter to close')
|
|
47
|
+
}
|
|
48
|
+
exit $Code
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
function Fail([string]$Message) {
|
|
52
|
+
Write-Host ''
|
|
53
|
+
Write-Host "ERROR: $(Hide-Key $Message)" -ForegroundColor Red
|
|
54
|
+
Finish 1
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
# BLS repeats a rejected key in its error message: never let it reach the screen or a file.
|
|
58
|
+
function Hide-Key([string]$Text) {
|
|
59
|
+
if ($script:Key -and $Text) { return $Text.Replace($script:Key, '[REDACTED]') }
|
|
60
|
+
return $Text
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
# ---------------------------------------------------------------------------- the requests
|
|
64
|
+
# manifest.json and query_NN.json are built into this script; they go to a temporary folder that is
|
|
65
|
+
# removed when the script ends.
|
|
66
|
+
$Embedded = [ordered]@{}
|
|
67
|
+
# @QUERIES@
|
|
68
|
+
$Req = Join-Path ([IO.Path]::GetTempPath()) ('download_bls.' + [IO.Path]::GetRandomFileName())
|
|
69
|
+
[void](New-Item -ItemType Directory -Path $Req)
|
|
70
|
+
foreach ($name in $Embedded.Keys) { [IO.File]::WriteAllText((Join-Path $Req $name), $Embedded[$name], $Utf8) }
|
|
71
|
+
$manifestFile = Join-Path $Req 'manifest.json'
|
|
72
|
+
if (-not (Test-Path -LiteralPath $manifestFile)) {
|
|
73
|
+
Fail "This script has no requests built in. Use the download_bls.ps1 from the analyst's download files."
|
|
74
|
+
}
|
|
75
|
+
$Manifest = [IO.File]::ReadAllText($manifestFile, $Utf8) | ConvertFrom-Json
|
|
76
|
+
$Queries = @(Get-ChildItem -LiteralPath $Req -Filter 'query_*.json' | Sort-Object Name)
|
|
77
|
+
if ($Queries.Count -eq 0) { Fail "This script has no requests built in. Use the download_bls.ps1 from the analyst's download files." }
|
|
78
|
+
$Bodies = @()
|
|
79
|
+
foreach ($q in $Queries) { $Bodies += ,([IO.File]::ReadAllText($q.FullName, $Utf8) | ConvertFrom-Json) }
|
|
80
|
+
|
|
81
|
+
$needKey = ($Manifest.api_version -eq 'v2')
|
|
82
|
+
foreach ($b in $Bodies) { if ($b.PSObject.Properties['registrationkey']) { $needKey = $true } }
|
|
83
|
+
|
|
84
|
+
# ---------------------------------------------------------------------------- API key
|
|
85
|
+
$Key = $env:BLS_API_KEY
|
|
86
|
+
if ($Key) {
|
|
87
|
+
Write-Host 'Using the BLS key in $env:BLS_API_KEY.'
|
|
88
|
+
} elseif ($needKey) {
|
|
89
|
+
Write-Host 'These requests need a free BLS registration key (https://data.bls.gov/registrationEngine/).'
|
|
90
|
+
if ([Console]::IsInputRedirected) {
|
|
91
|
+
$Key = [Console]::In.ReadLine()
|
|
92
|
+
} else {
|
|
93
|
+
$secure = Read-Host -Prompt 'Paste your BLS key (input is hidden), then press Enter' -AsSecureString
|
|
94
|
+
$bstr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secure)
|
|
95
|
+
try { $Key = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($bstr) }
|
|
96
|
+
finally { [Runtime.InteropServices.Marshal]::ZeroFreeBSTR($bstr) }
|
|
97
|
+
Remove-Variable secure
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
if ($null -eq $Key) { $Key = '' }
|
|
101
|
+
$Key = $Key -replace '\s', ''
|
|
102
|
+
if ($needKey -and -not $Key) { Fail 'No key entered. Get one at https://data.bls.gov/registrationEngine/' }
|
|
103
|
+
|
|
104
|
+
$Version = 'v1'
|
|
105
|
+
if ($Key -and $Manifest.urls.v2) { $Version = 'v2' }
|
|
106
|
+
$Url = [string]$Manifest.urls.$Version
|
|
107
|
+
Write-Host "Sending $($Queries.Count) request(s) to BLS API $Version ..."
|
|
108
|
+
|
|
109
|
+
# ---------------------------------------------------------------------------- send each request
|
|
110
|
+
$Saved = @()
|
|
111
|
+
for ($i = 0; $i -lt $Queries.Count; $i++) {
|
|
112
|
+
$q = $Queries[$i]
|
|
113
|
+
$body = $Bodies[$i]
|
|
114
|
+
$body.PSObject.Properties.Remove('registrationkey')
|
|
115
|
+
if ($Version -eq 'v2' -and $Manifest.v2_options) {
|
|
116
|
+
foreach ($p in $Manifest.v2_options.PSObject.Properties) {
|
|
117
|
+
$body | Add-Member -NotePropertyName $p.Name -NotePropertyValue $p.Value -Force
|
|
118
|
+
}
|
|
119
|
+
}
|
|
120
|
+
$requestJson = $body | ConvertTo-Json -Depth 10 -Compress # the request without the key
|
|
121
|
+
$body | Add-Member -NotePropertyName 'registrationkey' -NotePropertyValue $Key -Force
|
|
122
|
+
if ($Version -ne 'v2') { $body.PSObject.Properties.Remove('registrationkey') }
|
|
123
|
+
$sendJson = $body | ConvertTo-Json -Depth 10 -Compress # in memory only
|
|
124
|
+
$body.PSObject.Properties.Remove('registrationkey')
|
|
125
|
+
|
|
126
|
+
try {
|
|
127
|
+
$resp = Invoke-WebRequest -Uri $Url -Method Post -Body $Utf8.GetBytes($sendJson) `
|
|
128
|
+
-ContentType 'application/json' -UserAgent 'data-analyst/1.0' -UseBasicParsing -TimeoutSec 120
|
|
129
|
+
} catch {
|
|
130
|
+
Fail "$($q.Name): the request to api.bls.gov failed ($($_.Exception.Message)). Check the internet connection, VPN and proxy, then run this script again."
|
|
131
|
+
}
|
|
132
|
+
$sendJson = $null
|
|
133
|
+
$text = $Utf8.GetString($resp.RawContentStream.ToArray()).Trim()
|
|
134
|
+
try { $reply = $text | ConvertFrom-Json } catch { $reply = $null }
|
|
135
|
+
if ($null -eq $reply -or -not $text.EndsWith('}') -or $reply.PSObject.Properties.Count -eq 0) {
|
|
136
|
+
Fail "$($q.Name): BLS did not answer with JSON. The reply began: $($text.Substring(0, [Math]::Min(200, $text.Length)))"
|
|
137
|
+
}
|
|
138
|
+
# Keep the reply exactly as BLS sent it and add the request it answers, like download_bls.sh.
|
|
139
|
+
$text = $text.Substring(0, $text.LastIndexOf('}')).TrimEnd() + ",`n `"_request`": " + $requestJson + "`n}"
|
|
140
|
+
$dest = Join-Path $Here ($q.Name -replace '^query_', 'response_')
|
|
141
|
+
[IO.File]::WriteAllText($dest, (Hide-Key $text) + "`n", $Utf8)
|
|
142
|
+
$Saved += $dest
|
|
143
|
+
$msg = Hide-Key ((@($reply.message) | Where-Object { $_ }) -join '; ')
|
|
144
|
+
if ($msg.Length -gt 200) { $msg = $msg.Substring(0, 200) }
|
|
145
|
+
Write-Host "$(Split-Path -Leaf $dest) $($reply.status) $msg"
|
|
146
|
+
if ($i -lt $Queries.Count - 1) { Start-Sleep -Milliseconds 500 }
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
Write-Host ''
|
|
150
|
+
Write-Host 'Done. Upload these files to the chat (do not edit them):' -ForegroundColor Green
|
|
151
|
+
foreach ($p in $Saved) { Write-Host " $p" }
|
|
152
|
+
Finish 0
|
|
@@ -1,108 +1,108 @@
|
|
|
1
|
-
# Analysis patterns for BLS time series
|
|
2
|
-
|
|
3
|
-
Read this once at the start of any analysis. It lists the choices that change headline numbers (and
|
|
4
|
-
so become the critical questions), BLS's own conventions to follow, and how a BLS data product is
|
|
5
|
-
laid out.
|
|
6
|
-
|
|
7
|
-
Contents: 1. Critical questions catalog · 2. Conventions · 3. Benchmarks and reliability ·
|
|
8
|
-
4. Typical questions · 5. Publication structure · 6. Chart forms
|
|
9
|
-
|
|
10
|
-
## 1. Critical questions catalog
|
|
11
|
-
Ask only the ones the request leaves open **and** that would change a headline number. Each has a
|
|
12
|
-
default rule for "Don't know — decide empirically".
|
|
13
|
-
|
|
14
|
-
| Choice | Typical options | Default rule when the user does not know |
|
|
15
|
-
|---|---|---|
|
|
16
|
-
| **Which measure** | Unemployment: U-3 (official) vs. U-6. Wages: average hourly earnings (CES, all employees or production and nonsupervisory), median weekly earnings (CPS, full-time), ECI (fixed job mix). Inflation: CPI-U, CPI-W, C-CPI-U; headline vs. core. Jobs: payroll (CES) vs. household (CPS) employment | Use the measure BLS leads with in its news release (U-3, CES payrolls, CPI-U headline). For wages, show AHE and say that composition shifts move it (2020); add ECI when the period includes large composition changes |
|
|
17
|
-
| **Seasonal adjustment** | SA vs. NSA | Month-to-month or short-run changes, levels in charts: SA. 12-month changes of the CPI: NSA (BLS's headline 12-month CPI change uses NSA indexes). Import/export price indexes have no SA version: NSA throughout. Annual averages: NSA (BLS computes them from NSA data) |
|
|
18
|
-
| **Frequency** | Monthly, quarterly, annual averages | Monthly for spans up to about 10 years; annual averages for long histories. Use BLS-published annual averages (v2 key) over computed ones |
|
|
19
|
-
| **Period and baseline** | Since a date (Feb 2020 pre-pandemic peak, Jan 2021, 10 years), calendar years, a recession | The business-cycle peak before the event the user cares about (Feb 2020 for the pandemic); otherwise the latest 10 years. Show a sensitivity with one alternative baseline (2019 average) |
|
|
20
|
-
| **Nominal vs. real** | Current dollars vs. constant dollars; deflator CPI-U, CPI-W, C-CPI-U (PCE is BEA, not BLS) | Real, deflated as BLS does it: CPI-W for production and nonsupervisory employees, CPI-U for all employees and other groups (all items, NSA for annual, SA for monthly), base = latest full year; say "in <year> dollars". CPI-U-RS is not in the API |
|
|
21
|
-
| **Change measure** | Net change, percent change, 12-month change, annualized 3- or 6-month rate, CAGR | Follow the series' BLS convention: payrolls as net change (thousands); rates as percentage-point change; prices and wages as 12-month percent change, plus annualized 3-month for momentum |
|
|
22
|
-
| **Geography** | U.S., state (LAUS, CES state), metro, county | As asked. State unemployment: LAUS SA. Note that LAUS and state CES are model-based and revised each March |
|
|
23
|
-
| **Gaps and preliminary data** | Interpolate a missing month or leave it; include preliminary months | Leave gaps empty; compute changes only between published values (say so); include preliminary values but label them |
|
|
24
|
-
| **Comparison groups** | Demographic groups (CPS), industries (CES), areas | As asked; flag thin CPS groups (small groups have wide confidence intervals; use 12-month averages or annual averages) |
|
|
25
|
-
|
|
26
|
-
Ask at most four, in one round. A question about which series ID to use is never put to the user:
|
|
27
|
-
resolve it yourself with the `request_schema` search `describe <ID>` and the tables in `series_ids.md`.
|
|
28
|
-
|
|
29
|
-
## 2. Conventions
|
|
30
|
-
- **Definitions:** unemployment rate = unemployed ÷ labor force; participation rate = labor force ÷
|
|
31
|
-
civilian noninstitutional population; employment-population ratio = employed ÷ population. CES counts
|
|
32
|
-
jobs (a person with two jobs counts twice); CPS counts people.
|
|
33
|
-
- **Units:** CES and CPS levels are in thousands. JOLTS levels are in thousands; rates are percentages.
|
|
34
|
-
CPI is an index (1982-84=100); never report an index level as a price.
|
|
35
|
-
- **Percent changes of indexes:** compute from the published index values (rounded to three decimals) and
|
|
36
|
-
round the result to one decimal; that reproduces BLS's published CPI changes.
|
|
37
|
-
- **Social Security COLA** (SSA's rule, not a BLS series): the CPI-W, not seasonally adjusted (`CWUR0000SA0`),
|
|
38
|
-
averaged over July-September and rounded to three decimals, compared with the same average of the last
|
|
39
|
-
year a COLA was paid; the percent increase rounded to one decimal is the COLA payable from the next
|
|
40
|
-
January. SSA announces it with the September CPI (mid-October); benchmark against SSA's announcement.
|
|
41
|
-
- **12-month change across a gap:** October 2026 vs. October 2025 has no value when October 2025 was not
|
|
42
|
-
collected. Say so; do not substitute September or an interpolated value.
|
|
43
|
-
- **Revisions:** CES revises the two prior months each release and benchmarks every February (back to the
|
|
44
|
-
previous March); seasonal factors for CPI are revised each February for five years; LAUS is revised each
|
|
45
|
-
March; C-CPI-U is final 10 to 12 months later. State the retrieval date (the builder adds it).
|
|
46
|
-
- **January population controls (CPS):** levels shift in January; compare levels across Januaries only with
|
|
47
|
-
a note, or use rates.
|
|
48
|
-
- **Annual averages:** BLS's M13 values are preferred. A computed average needs all 12 months; a year with a
|
|
49
|
-
gap has no computed average (leave it empty).
|
|
50
|
-
- **Recessions (NBER):** March–November 2001, December 2007–June 2009, February–April 2020. Use the note
|
|
51
|
-
"Recessions begin just after a peak in economic activity and run through the subsequent trough."
|
|
52
|
-
- **Correlation, not cause:** a policy or event "coincided with" a change unless a method that supports a
|
|
53
|
-
causal claim was used.
|
|
54
|
-
|
|
55
|
-
## 3. Benchmarks and reliability
|
|
56
|
-
- **Benchmark every headline number** against a BLS publication: the latest news release (Employment
|
|
57
|
-
Situation, CPI, JOLTS, Employment Cost Index, Productivity and Costs, State Employment and Unemployment),
|
|
58
|
-
or the API's own `calculations` (fetch with a key and `"calculations": true`; they arrive as
|
|
59
|
-
`bls_pct_12m`, `bls_net_1m` ... columns). Record the comparison in Data and Methods.
|
|
60
|
-
- **Statistical significance:** BLS tests at the 90-percent level. Published thresholds (read the current
|
|
61
|
-
technical note before using, they are updated; values below are from 2026-10-19):
|
|
62
|
-
- CES total nonfarm monthly change: about ±122,000 (Employment Situation technical note).
|
|
63
|
-
- CPS monthly change in unemployment: about ±425,000; in the unemployment rate: about ±0.3 percentage point.
|
|
64
|
-
- JOLTS: median standard errors in the "significant change" tables on the JOLTS page.
|
|
65
|
-
- CPI: standard errors of 1-, 2-, 6- and 12-month changes on the CPI variance estimates page.
|
|
66
|
-
Compare each change with its threshold before describing it. Call a change "statistically significant"
|
|
67
|
-
only when it exceeds the threshold; otherwise "little changed" (BLS's own phrase) or "about the same".
|
|
68
|
-
- Sampling error does not apply to QCEW (a census of UI records), but revisions do.
|
|
69
|
-
|
|
70
|
-
## 4. Typical questions
|
|
71
|
-
- **Labor market since the pandemic:** unemployment (U-3, U-6), participation and prime-age employment rate,
|
|
72
|
-
payrolls vs. February 2020, job openings and quits. Benchmark: the latest Employment Situation and JOLTS.
|
|
73
|
-
- **Inflation:** 12-month CPI-U change (NSA), core vs. headline, contributions of food, energy, shelter;
|
|
74
|
-
3-month annualized core for momentum. Benchmark: CPI news release table A.
|
|
75
|
-
- **Real wages:** BLS's own real AHE: `CES0500000013` all employees (CPI-U), `CES0500000032` production and
|
|
76
|
-
nonsupervisory employees (CPI-W, `CWSR0000SA0`), as the Real Earnings release reports them; real median weekly
|
|
77
|
-
earnings, `LES1252881600` seasonally adjusted, as the Usual Weekly Earnings release headlines it); ECI vs. CPI over 12 months.
|
|
78
|
-
- **States and metros:** LAUS unemployment rates ranked, change over a year; state payrolls (SM).
|
|
79
|
-
- **Industries:** CES employment by supersector, net change since a baseline; JOLTS by industry.
|
|
80
|
-
- **Productivity and labor costs:** quarterly annualized rates; long-run averages by business cycle.
|
|
81
|
-
|
|
82
|
-
## 5. Publication structure (the report builder's `"mode": "report"`)
|
|
83
|
-
Same exhibit chartbook as the style guide describes:
|
|
84
|
-
1. Cover with the month and year, the title with the period, and an optional hero chart.
|
|
85
|
-
2. **At a Glance:** one framing paragraph (which measures, which period) and 3–5 findings, each with a bold
|
|
86
|
-
lead naming its section.
|
|
87
|
-
3. Contents and List of Exhibits (built automatically).
|
|
88
|
-
4. **Notes About This Report:** calendar years; rounding; seasonal adjustment; dollar year and deflator;
|
|
89
|
-
data source and retrieval date; definitions; preliminary values and gaps; recession shading.
|
|
90
|
-
5. **Introduction:** a lead paragraph with the headline numbers; question-style subsections ("How Has
|
|
91
|
-
Unemployment Changed Since 2019?") and "What Data Were Used?".
|
|
92
|
-
6. **Sections** (2–5): a summary page, then one exhibit per page with 2–4 paragraphs beside the chart
|
|
93
|
-
(level, change, comparison).
|
|
94
|
-
7. **Appendix A: Data and Methods:** series IDs and titles, seasonal adjustment, definitions, computations
|
|
95
|
-
(change measures, deflation, annual averages), benchmark comparisons, significance thresholds,
|
|
96
|
-
limitations. The builder appends the data vintage and the BLS terms statement.
|
|
97
|
-
8. References; About This Document (optional); replication materials and audit trail.
|
|
98
|
-
|
|
99
|
-
Exhibit titles end with the period: "Unemployment Rate, January 2019 to September 2026". The units line
|
|
100
|
-
names the adjustment: "Percent, seasonally adjusted". Source line: "Analyst calculations using data from the
|
|
101
|
-
Bureau of Labor Statistics, Current Population Survey (series LNS14000000)."
|
|
102
|
-
|
|
103
|
-
## 6. Chart forms
|
|
104
|
-
- Trends: line charts, at most 4–5 series; facet rather than crowd. Monthly x-axis labels "Sep 2026".
|
|
105
|
-
- Changes by group at one date: horizontal bars when labels are long (industries, states), columns otherwise.
|
|
106
|
-
- Composition (contributions to CPI change, payroll gains by industry): stacked columns.
|
|
107
|
-
- Index comparisons (wages vs. prices): rebase both to 100 at the same date (a `compute` derive step).
|
|
108
|
-
- No dual axes, no pies, no 3-D. Do not draw a line through a gap: leave the cell empty in the exhibit CSV.
|
|
1
|
+
# Analysis patterns for BLS time series
|
|
2
|
+
|
|
3
|
+
Read this once at the start of any analysis. It lists the choices that change headline numbers (and
|
|
4
|
+
so become the critical questions), BLS's own conventions to follow, and how a BLS data product is
|
|
5
|
+
laid out.
|
|
6
|
+
|
|
7
|
+
Contents: 1. Critical questions catalog · 2. Conventions · 3. Benchmarks and reliability ·
|
|
8
|
+
4. Typical questions · 5. Publication structure · 6. Chart forms
|
|
9
|
+
|
|
10
|
+
## 1. Critical questions catalog
|
|
11
|
+
Ask only the ones the request leaves open **and** that would change a headline number. Each has a
|
|
12
|
+
default rule for "Don't know — decide empirically".
|
|
13
|
+
|
|
14
|
+
| Choice | Typical options | Default rule when the user does not know |
|
|
15
|
+
|---|---|---|
|
|
16
|
+
| **Which measure** | Unemployment: U-3 (official) vs. U-6. Wages: average hourly earnings (CES, all employees or production and nonsupervisory), median weekly earnings (CPS, full-time), ECI (fixed job mix). Inflation: CPI-U, CPI-W, C-CPI-U; headline vs. core. Jobs: payroll (CES) vs. household (CPS) employment | Use the measure BLS leads with in its news release (U-3, CES payrolls, CPI-U headline). For wages, show AHE and say that composition shifts move it (2020); add ECI when the period includes large composition changes |
|
|
17
|
+
| **Seasonal adjustment** | SA vs. NSA | Month-to-month or short-run changes, levels in charts: SA. 12-month changes of the CPI: NSA (BLS's headline 12-month CPI change uses NSA indexes). Import/export price indexes have no SA version: NSA throughout. Annual averages: NSA (BLS computes them from NSA data) |
|
|
18
|
+
| **Frequency** | Monthly, quarterly, annual averages | Monthly for spans up to about 10 years; annual averages for long histories. Use BLS-published annual averages (v2 key) over computed ones |
|
|
19
|
+
| **Period and baseline** | Since a date (Feb 2020 pre-pandemic peak, Jan 2021, 10 years), calendar years, a recession | The business-cycle peak before the event the user cares about (Feb 2020 for the pandemic); otherwise the latest 10 years. Show a sensitivity with one alternative baseline (2019 average) |
|
|
20
|
+
| **Nominal vs. real** | Current dollars vs. constant dollars; deflator CPI-U, CPI-W, C-CPI-U (PCE is BEA, not BLS) | Real, deflated as BLS does it: CPI-W for production and nonsupervisory employees, CPI-U for all employees and other groups (all items, NSA for annual, SA for monthly), base = latest full year; say "in <year> dollars". CPI-U-RS is not in the API |
|
|
21
|
+
| **Change measure** | Net change, percent change, 12-month change, annualized 3- or 6-month rate, CAGR | Follow the series' BLS convention: payrolls as net change (thousands); rates as percentage-point change; prices and wages as 12-month percent change, plus annualized 3-month for momentum |
|
|
22
|
+
| **Geography** | U.S., state (LAUS, CES state), metro, county | As asked. State unemployment: LAUS SA. Note that LAUS and state CES are model-based and revised each March |
|
|
23
|
+
| **Gaps and preliminary data** | Interpolate a missing month or leave it; include preliminary months | Leave gaps empty; compute changes only between published values (say so); include preliminary values but label them |
|
|
24
|
+
| **Comparison groups** | Demographic groups (CPS), industries (CES), areas | As asked; flag thin CPS groups (small groups have wide confidence intervals; use 12-month averages or annual averages) |
|
|
25
|
+
|
|
26
|
+
Ask at most four, in one round. A question about which series ID to use is never put to the user:
|
|
27
|
+
resolve it yourself with the `request_schema` search `describe <ID>` and the tables in `series_ids.md`.
|
|
28
|
+
|
|
29
|
+
## 2. Conventions
|
|
30
|
+
- **Definitions:** unemployment rate = unemployed ÷ labor force; participation rate = labor force ÷
|
|
31
|
+
civilian noninstitutional population; employment-population ratio = employed ÷ population. CES counts
|
|
32
|
+
jobs (a person with two jobs counts twice); CPS counts people.
|
|
33
|
+
- **Units:** CES and CPS levels are in thousands. JOLTS levels are in thousands; rates are percentages.
|
|
34
|
+
CPI is an index (1982-84=100); never report an index level as a price.
|
|
35
|
+
- **Percent changes of indexes:** compute from the published index values (rounded to three decimals) and
|
|
36
|
+
round the result to one decimal; that reproduces BLS's published CPI changes.
|
|
37
|
+
- **Social Security COLA** (SSA's rule, not a BLS series): the CPI-W, not seasonally adjusted (`CWUR0000SA0`),
|
|
38
|
+
averaged over July-September and rounded to three decimals, compared with the same average of the last
|
|
39
|
+
year a COLA was paid; the percent increase rounded to one decimal is the COLA payable from the next
|
|
40
|
+
January. SSA announces it with the September CPI (mid-October); benchmark against SSA's announcement.
|
|
41
|
+
- **12-month change across a gap:** October 2026 vs. October 2025 has no value when October 2025 was not
|
|
42
|
+
collected. Say so; do not substitute September or an interpolated value.
|
|
43
|
+
- **Revisions:** CES revises the two prior months each release and benchmarks every February (back to the
|
|
44
|
+
previous March); seasonal factors for CPI are revised each February for five years; LAUS is revised each
|
|
45
|
+
March; C-CPI-U is final 10 to 12 months later. State the retrieval date (the builder adds it).
|
|
46
|
+
- **January population controls (CPS):** levels shift in January; compare levels across Januaries only with
|
|
47
|
+
a note, or use rates.
|
|
48
|
+
- **Annual averages:** BLS's M13 values are preferred. A computed average needs all 12 months; a year with a
|
|
49
|
+
gap has no computed average (leave it empty).
|
|
50
|
+
- **Recessions (NBER):** March–November 2001, December 2007–June 2009, February–April 2020. Use the note
|
|
51
|
+
"Recessions begin just after a peak in economic activity and run through the subsequent trough."
|
|
52
|
+
- **Correlation, not cause:** a policy or event "coincided with" a change unless a method that supports a
|
|
53
|
+
causal claim was used.
|
|
54
|
+
|
|
55
|
+
## 3. Benchmarks and reliability
|
|
56
|
+
- **Benchmark every headline number** against a BLS publication: the latest news release (Employment
|
|
57
|
+
Situation, CPI, JOLTS, Employment Cost Index, Productivity and Costs, State Employment and Unemployment),
|
|
58
|
+
or the API's own `calculations` (fetch with a key and `"calculations": true`; they arrive as
|
|
59
|
+
`bls_pct_12m`, `bls_net_1m` ... columns). Record the comparison in Data and Methods.
|
|
60
|
+
- **Statistical significance:** BLS tests at the 90-percent level. Published thresholds (read the current
|
|
61
|
+
technical note before using, they are updated; values below are from 2026-10-19):
|
|
62
|
+
- CES total nonfarm monthly change: about ±122,000 (Employment Situation technical note).
|
|
63
|
+
- CPS monthly change in unemployment: about ±425,000; in the unemployment rate: about ±0.3 percentage point.
|
|
64
|
+
- JOLTS: median standard errors in the "significant change" tables on the JOLTS page.
|
|
65
|
+
- CPI: standard errors of 1-, 2-, 6- and 12-month changes on the CPI variance estimates page.
|
|
66
|
+
Compare each change with its threshold before describing it. Call a change "statistically significant"
|
|
67
|
+
only when it exceeds the threshold; otherwise "little changed" (BLS's own phrase) or "about the same".
|
|
68
|
+
- Sampling error does not apply to QCEW (a census of UI records), but revisions do.
|
|
69
|
+
|
|
70
|
+
## 4. Typical questions
|
|
71
|
+
- **Labor market since the pandemic:** unemployment (U-3, U-6), participation and prime-age employment rate,
|
|
72
|
+
payrolls vs. February 2020, job openings and quits. Benchmark: the latest Employment Situation and JOLTS.
|
|
73
|
+
- **Inflation:** 12-month CPI-U change (NSA), core vs. headline, contributions of food, energy, shelter;
|
|
74
|
+
3-month annualized core for momentum. Benchmark: CPI news release table A.
|
|
75
|
+
- **Real wages:** BLS's own real AHE: `CES0500000013` all employees (CPI-U), `CES0500000032` production and
|
|
76
|
+
nonsupervisory employees (CPI-W, `CWSR0000SA0`), as the Real Earnings release reports them; real median weekly
|
|
77
|
+
earnings, `LES1252881600` seasonally adjusted, as the Usual Weekly Earnings release headlines it); ECI vs. CPI over 12 months.
|
|
78
|
+
- **States and metros:** LAUS unemployment rates ranked, change over a year; state payrolls (SM).
|
|
79
|
+
- **Industries:** CES employment by supersector, net change since a baseline; JOLTS by industry.
|
|
80
|
+
- **Productivity and labor costs:** quarterly annualized rates; long-run averages by business cycle.
|
|
81
|
+
|
|
82
|
+
## 5. Publication structure (the report builder's `"mode": "report"`)
|
|
83
|
+
Same exhibit chartbook as the style guide describes:
|
|
84
|
+
1. Cover with the month and year, the title with the period, and an optional hero chart.
|
|
85
|
+
2. **At a Glance:** one framing paragraph (which measures, which period) and 3–5 findings, each with a bold
|
|
86
|
+
lead naming its section.
|
|
87
|
+
3. Contents and List of Exhibits (built automatically).
|
|
88
|
+
4. **Notes About This Report:** calendar years; rounding; seasonal adjustment; dollar year and deflator;
|
|
89
|
+
data source and retrieval date; definitions; preliminary values and gaps; recession shading.
|
|
90
|
+
5. **Introduction:** a lead paragraph with the headline numbers; question-style subsections ("How Has
|
|
91
|
+
Unemployment Changed Since 2019?") and "What Data Were Used?".
|
|
92
|
+
6. **Sections** (2–5): a summary page, then one exhibit per page with 2–4 paragraphs beside the chart
|
|
93
|
+
(level, change, comparison).
|
|
94
|
+
7. **Appendix A: Data and Methods:** series IDs and titles, seasonal adjustment, definitions, computations
|
|
95
|
+
(change measures, deflation, annual averages), benchmark comparisons, significance thresholds,
|
|
96
|
+
limitations. The builder appends the data vintage and the BLS terms statement.
|
|
97
|
+
8. References; About This Document (optional); replication materials and audit trail.
|
|
98
|
+
|
|
99
|
+
Exhibit titles end with the period: "Unemployment Rate, January 2019 to September 2026". The units line
|
|
100
|
+
names the adjustment: "Percent, seasonally adjusted". Source line: "Analyst calculations using data from the
|
|
101
|
+
Bureau of Labor Statistics, Current Population Survey (series LNS14000000)."
|
|
102
|
+
|
|
103
|
+
## 6. Chart forms
|
|
104
|
+
- Trends: line charts, at most 4–5 series; facet rather than crowd. Monthly x-axis labels "Sep 2026".
|
|
105
|
+
- Changes by group at one date: horizontal bars when labels are long (industries, states), columns otherwise.
|
|
106
|
+
- Composition (contributions to CPI change, payroll gains by industry): stacked columns.
|
|
107
|
+
- Index comparisons (wages vs. prices): rebase both to 100 at the same date (a `compute` derive step).
|
|
108
|
+
- No dual axes, no pies, no 3-D. Do not draw a line through a gap: leave the cell empty in the exhibit CSV.
|