neissai 1.4.0__tar.gz

neissai-1.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Subash Kumar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

neissai-1.4.0/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: neissai
+Version: 1.4.0
+Summary: AI-assisted national injury estimates from NEISS emergency department narratives: consistent e-bike and e-scooter counts across product-code changes, with survey-correct confidence intervals.
+Author: Subash Kumar
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/subash28kumar/neiss-ai
+Project-URL: Dashboard, https://subash28kumar.github.io/neiss-ai/
+Project-URL: Documentation, https://github.com/subash28kumar/neiss-ai#readme
+Project-URL: Changelog, https://github.com/subash28kumar/neiss-ai/blob/main/CHANGELOG.md
+Keywords: NEISS,injury surveillance,e-bike,e-scooter,micromobility,text classification,survey statistics
+Classifier: Programming Language :: Python :: 3
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=1.5
+Requires-Dist: numpy>=1.22
+Provides-Extra: train
+Requires-Dist: scikit-learn>=1.1; extra == "train"
+Provides-Extra: excel
+Requires-Dist: openpyxl; extra == "excel"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Dynamic: license-file
+
+# NEISS-AI
+
+**AI-assisted national injury estimates from US emergency department narratives.**
+
+NEISS-AI turns the US Consumer Product Safety Commission's [NEISS](https://www.cpsc.gov/Research--Statistics/NEISS-Injury-Data) data into consistent national estimates for e-bikes, e-scooters, bicycles and other micromobility devices, with survey-correct 95% confidence intervals.
+
+**Live web app:** https://subash28kumar.github.io/neiss-ai/
+
+The web app has four parts, all free and running in the browser:
+
+- **Overview:** headline numbers and trends for e-bikes, e-scooters and kick scooters.
+- **Ask a question:** type a question in plain English (for example, *"How many teenagers broke a bone on an e-scooter in 2025?"*) and get a national estimate with a 95% confidence interval, a trend chart and an "Understood as" line showing how the question was read.
+- **Explore the data:** filter by device, years, age, sex, injury, body part, outcome and circumstance; download the results as CSV; share a link to any view.
+- **Search injury stories:** search the hospital narratives (for example *battery*, *hit by car*) to see national estimates by year and real example stories.
+
+The site refreshes itself: a GitHub Actions workflow (`.github/workflows/refresh.yml`) checks monthly for a new year of NEISS data, reruns the tool and republishes the site. It never publishes fewer years than the site already shows.
+
+## Why it exists
+
+In January 2024, CPSC gave e-bikes their own product code (5045). Before then, e-bikes shared code 3215 ("mopeds or power-assisted cycles") with gas mopeds. Counts based on codes alone therefore jump at 2024 and cannot show a true trend.
+
+NEISS-AI reads the short injury story in each record:
+
+1. **Rules.** Product codes and whole-word keywords label each injury (e-bike, e-scooter, bicycle, kick scooter, hoverboard, skateboard or moped). Whole-word matching matters: "the bike" is not an "e bike".
+2. **Text classifier.** For code-3215 records whose story is unclear, a classifier trained on 2024-2025 records (where 5045 and 5046 are separate) decides e-bike vs moped. It is TF-IDF + logistic regression, stored as plain JSON and run with numpy only. Five-fold cross-validated AUC: **0.995**.
+3. **National estimates.** NEISS is a stratified sample of hospitals. Estimates use the survey weights and Taylor-linearized standard errors (hospitals as clusters within strata), and are flagged "unstable" by CPSC's rule (fewer than 20 records, an estimate below 1,200, or a coefficient of variation above 33%).
+
+**Validation:** e-bike estimates for 2022-2024 are within 4% of CPSC's published figures (27,800 vs 26,800; 37,600 vs 37,600; 59,000 vs 59,200).
+
+## Install
+
+```
+pip install git+https://github.com/subash28kumar/neiss-ai
+```
+
+Requires Python 3.9+, pandas and numpy. (scikit-learn is needed only to retrain the classifier: `pip install "neissai[train] @ git+https://github.com/subash28kumar/neiss-ai"`.)
+
+## Use from the command line
+
+Download the annual NEISS files (for example `neiss2019.tsv` ... `neiss2025.tsv`) from the [CPSC NEISS page](https://www.cpsc.gov/Research--Statistics/NEISS-Injury-Data) into a folder called `data`, or try `neissai download --years 2019-2025 --data-dir data`. Then:
+
+```
+neissai micromobility --years 2019-2025 --data-dir data --out-dir results
+```
+
+This prints a table of national estimates and saves:
+
+| File | Contents |
+|---|---|
+| `estimates_by_year_device.csv` | national ED visits by year and device, 95% CI, CV, records, unstable flag |
+| `estimates_by_year_device_age.csv` | the same by age group (0-17, 18-64, 65+) |
+| `shares_by_device.csv` | weighted shares hospitalized, head/face injury, motor vehicle involved, no helmet stated, alcohol |
+| `micromobility_records.csv` | every labelled record, with device, label source and classifier probability |
+
+Check a single story:
+
+```
+neissai classify "14YOM FELL OFF HIS ELECTRIC BIKE GOING 20MPH"
+```
+
+## Use from Python
+
+```python
+import neissai
+
+df = neissai.read_neiss("data", years="2019-2025")
+df = neissai.add_outcomes(neissai.label_devices(df))
+mm = df["device"].isin(neissai.DEVICES)
+
+# Pass the full table plus a filter, so the variance uses the whole sample design
+by_year = neissai.national_estimates(df, by=["year", "device"], where=mm)
+kids = neissai.national_estimates(df, by=["year"], where=(df["device"] == "E-bike") & (df["age_group"] == "0-17"))
+hosp = neissai.shares(df, "hospitalized", by=["device", "age_group"], where=mm)
+```
+
+## Rebuild the website data
+
+```
+python scripts/update_site.py --data-dir data --years 2019-2025
+```
+
+## Retrain the classifier
+
+```
+python scripts/train_classifier.py --data-dir data --years 2024-2025
+```
+
+## Limits
+
+NEISS counts emergency department visits, not riders or miles, so results are counts, not risk rates. Injuries treated elsewhere and deaths at the scene are not included. Helmet use is only what the story mentions. Early e-bike estimates (2019-2021) rest on small samples. The powered-scooter code may include some seated, moped-style scooters.
+
+## Cite
+
+If you use NEISS-AI, please cite the software (see `CITATION.cff` or "Cite this repository" on GitHub):
+
+> Kumar S. NEISS-AI: AI-assisted national injury estimates from NEISS emergency department narratives. Version 1.4.0 [software]. Zenodo; 2026. https://doi.org/10.5281/zenodo.23550272
+>
+> To cite all versions, use https://doi.org/10.5281/zenodo.23306744 (always resolves to the latest version).
+
+and the method paper:
+
+> Kumar S. E-bike injuries among US children surged in 2025: national estimates from emergency department reports, 2019-2025, using machine learning to read injury narratives [preprint]. Zenodo; 2026. https://doi.org/10.5281/zenodo.23354066
+
+and the labeled dataset:
+
+> Kumar S. NEISS-AI device labels for US micromobility injuries, 2019–2025 [dataset]. Version 1.1. Zenodo; 2026. https://doi.org/10.5281/zenodo.23550382
+
+Data come from the US Consumer Product Safety Commission. CPSC does not endorse this tool.
+
+## Licence
+
+MIT. Built by Subash Kumar (ORCID [0009-0003-6661-3424](https://orcid.org/0009-0003-6661-3424)).

neissai-1.4.0/README.md

@@ -0,0 +1,112 @@
+# NEISS-AI
+
+**AI-assisted national injury estimates from US emergency department narratives.**
+
+NEISS-AI turns the US Consumer Product Safety Commission's [NEISS](https://www.cpsc.gov/Research--Statistics/NEISS-Injury-Data) data into consistent national estimates for e-bikes, e-scooters, bicycles and other micromobility devices, with survey-correct 95% confidence intervals.
+
+**Live web app:** https://subash28kumar.github.io/neiss-ai/
+
+The web app has four parts, all free and running in the browser:
+
+- **Overview:** headline numbers and trends for e-bikes, e-scooters and kick scooters.
+- **Ask a question:** type a question in plain English (for example, *"How many teenagers broke a bone on an e-scooter in 2025?"*) and get a national estimate with a 95% confidence interval, a trend chart and an "Understood as" line showing how the question was read.
+- **Explore the data:** filter by device, years, age, sex, injury, body part, outcome and circumstance; download the results as CSV; share a link to any view.
+- **Search injury stories:** search the hospital narratives (for example *battery*, *hit by car*) to see national estimates by year and real example stories.
+
+The site refreshes itself: a GitHub Actions workflow (`.github/workflows/refresh.yml`) checks monthly for a new year of NEISS data, reruns the tool and republishes the site. It never publishes fewer years than the site already shows.
+
+## Why it exists
+
+In January 2024, CPSC gave e-bikes their own product code (5045). Before then, e-bikes shared code 3215 ("mopeds or power-assisted cycles") with gas mopeds. Counts based on codes alone therefore jump at 2024 and cannot show a true trend.
+
+NEISS-AI reads the short injury story in each record:
+
+1. **Rules.** Product codes and whole-word keywords label each injury (e-bike, e-scooter, bicycle, kick scooter, hoverboard, skateboard or moped). Whole-word matching matters: "the bike" is not an "e bike".
+2. **Text classifier.** For code-3215 records whose story is unclear, a classifier trained on 2024-2025 records (where 5045 and 5046 are separate) decides e-bike vs moped. It is TF-IDF + logistic regression, stored as plain JSON and run with numpy only. Five-fold cross-validated AUC: **0.995**.
+3. **National estimates.** NEISS is a stratified sample of hospitals. Estimates use the survey weights and Taylor-linearized standard errors (hospitals as clusters within strata), and are flagged "unstable" by CPSC's rule (fewer than 20 records, an estimate below 1,200, or a coefficient of variation above 33%).
+
+**Validation:** e-bike estimates for 2022-2024 are within 4% of CPSC's published figures (27,800 vs 26,800; 37,600 vs 37,600; 59,000 vs 59,200).
+
+## Install
+
+```
+pip install git+https://github.com/subash28kumar/neiss-ai
+```
+
+Requires Python 3.9+, pandas and numpy. (scikit-learn is needed only to retrain the classifier: `pip install "neissai[train] @ git+https://github.com/subash28kumar/neiss-ai"`.)
+
+## Use from the command line
+
+Download the annual NEISS files (for example `neiss2019.tsv` ... `neiss2025.tsv`) from the [CPSC NEISS page](https://www.cpsc.gov/Research--Statistics/NEISS-Injury-Data) into a folder called `data`, or try `neissai download --years 2019-2025 --data-dir data`. Then:
+
+```
+neissai micromobility --years 2019-2025 --data-dir data --out-dir results
+```
+
+This prints a table of national estimates and saves:
+
+| File | Contents |
+|---|---|
+| `estimates_by_year_device.csv` | national ED visits by year and device, 95% CI, CV, records, unstable flag |
+| `estimates_by_year_device_age.csv` | the same by age group (0-17, 18-64, 65+) |
+| `shares_by_device.csv` | weighted shares hospitalized, head/face injury, motor vehicle involved, no helmet stated, alcohol |
+| `micromobility_records.csv` | every labelled record, with device, label source and classifier probability |
+
+Check a single story:
+
+```
+neissai classify "14YOM FELL OFF HIS ELECTRIC BIKE GOING 20MPH"
+```
+
+## Use from Python
+
+```python
+import neissai
+
+df = neissai.read_neiss("data", years="2019-2025")
+df = neissai.add_outcomes(neissai.label_devices(df))
+mm = df["device"].isin(neissai.DEVICES)
+
+# Pass the full table plus a filter, so the variance uses the whole sample design
+by_year = neissai.national_estimates(df, by=["year", "device"], where=mm)
+kids = neissai.national_estimates(df, by=["year"], where=(df["device"] == "E-bike") & (df["age_group"] == "0-17"))
+hosp = neissai.shares(df, "hospitalized", by=["device", "age_group"], where=mm)
+```
+
+## Rebuild the website data
+
+```
+python scripts/update_site.py --data-dir data --years 2019-2025
+```
+
+## Retrain the classifier
+
+```
+python scripts/train_classifier.py --data-dir data --years 2024-2025
+```
+
+## Limits
+
+NEISS counts emergency department visits, not riders or miles, so results are counts, not risk rates. Injuries treated elsewhere and deaths at the scene are not included. Helmet use is only what the story mentions. Early e-bike estimates (2019-2021) rest on small samples. The powered-scooter code may include some seated, moped-style scooters.
+
+## Cite
+
+If you use NEISS-AI, please cite the software (see `CITATION.cff` or "Cite this repository" on GitHub):
+
+> Kumar S. NEISS-AI: AI-assisted national injury estimates from NEISS emergency department narratives. Version 1.4.0 [software]. Zenodo; 2026. https://doi.org/10.5281/zenodo.23550272
+>
+> To cite all versions, use https://doi.org/10.5281/zenodo.23306744 (always resolves to the latest version).
+
+and the method paper:
+
+> Kumar S. E-bike injuries among US children surged in 2025: national estimates from emergency department reports, 2019-2025, using machine learning to read injury narratives [preprint]. Zenodo; 2026. https://doi.org/10.5281/zenodo.23354066
+
+and the labeled dataset:
+
+> Kumar S. NEISS-AI device labels for US micromobility injuries, 2019–2025 [dataset]. Version 1.1. Zenodo; 2026. https://doi.org/10.5281/zenodo.23550382
+
+Data come from the US Consumer Product Safety Commission. CPSC does not endorse this tool.
+
+## Licence
+
+MIT. Built by Subash Kumar (ORCID [0009-0003-6661-3424](https://orcid.org/0009-0003-6661-3424)).

neissai-1.4.0/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "neissai"
+version = "1.4.0"
+description = "AI-assisted national injury estimates from NEISS emergency department narratives: consistent e-bike and e-scooter counts across product-code changes, with survey-correct confidence intervals."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{name = "Subash Kumar"}]
+keywords = ["NEISS", "injury surveillance", "e-bike", "e-scooter", "micromobility", "text classification", "survey statistics"]
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Intended Audience :: Science/Research",
+  "Topic :: Scientific/Engineering :: Medical Science Apps.",
+]
+dependencies = ["pandas>=1.5", "numpy>=1.22"]
+
+[project.optional-dependencies]
+train = ["scikit-learn>=1.1"]
+excel = ["openpyxl"]
+test = ["pytest"]
+
+[project.urls]
+Homepage = "https://github.com/subash28kumar/neiss-ai"
+Dashboard = "https://subash28kumar.github.io/neiss-ai/"
+Documentation = "https://github.com/subash28kumar/neiss-ai#readme"
+Changelog = "https://github.com/subash28kumar/neiss-ai/blob/main/CHANGELOG.md"
+
+[project.scripts]
+neissai = "neissai.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+neissai = ["model/*.json"]

neissai-1.4.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

neissai-1.4.0/src/neissai/__init__.py

@@ -0,0 +1,22 @@
+"""neissai: AI-assisted national injury estimates from NEISS emergency department narratives.
+
+Quick start::
+
+    import neissai
+    df = neissai.read_neiss("data", years="2019-2025")      # CPSC annual files
+    labeled = neissai.label_devices(df)                       # rules + text classifier
+    labeled = neissai.add_outcomes(labeled)
+    est = neissai.national_estimates(labeled, by=["year", "device"],
+                                     where=labeled["device"].isin(neissai.DEVICES))
+"""
+from .classifier import EbikeMopedClassifier
+from . import circumstances
+from .devices import DEVICES, add_outcomes, label_devices, micromobility
+from .estimate import national_estimates, proportion, shares, total
+from .io import download, parse_years, read_neiss, read_neiss_file, standardize
+
+__version__ = "1.4.0"
+
+__all__ = ["DEVICES", "circumstances", "EbikeMopedClassifier", "add_outcomes", "download", "label_devices",
+           "micromobility", "national_estimates", "parse_years", "proportion", "read_neiss",
+           "read_neiss_file", "shares", "standardize", "total", "__version__"]

neissai-1.4.0/src/neissai/circumstances.py

@@ -0,0 +1,73 @@
+"""InjuryCoder: circumstance flags read from NEISS injury narratives (added in v1.4).
+
+NEISS does not code whether a motor vehicle was involved or whether the rider wore a helmet,
+but the narrative often says so. These rules read the narrative text:
+
+* ``motor_vehicle(text)`` -> True when the patient was hit by, crashed into, or struck part of
+  a motor vehicle (moving or parked, including a car door), or the narrative calls it an MVC/MVA.
+  Swerving to avoid a car without contact is not counted.
+* ``helmet(text)`` -> "yes", "no" or "not mentioned". "Not mentioned" includes unclear wording
+  such as "HELMET NS" and is never treated as "no".
+
+Validation (held-out test of 300 random narratives, 2019-2025; see VALIDATION.md): motor vehicle
+accuracy 98.3%, sensitivity 93.6%, precision 95.7%; helmet 300/300 correct.
+"""
+from __future__ import annotations
+
+import re
+MOD = r"(?:(?:A|AN|THE|HIS|HER|MOVING|PARKED|ONCOMING|PASSING|GARBAGE|DUMP|BOX|PICKUP|PICK-UP|DELIVERY|POLICE|SCHOOL|CITY|SEMI|TOW|FIRE|ANOTHER|SMALL|LARGE|BIG|WHITE|BLACK|RED|SLOW|SLOW-MOVING|REVERSING|BACKING|TURNING|STATIONARY|[A-Z]+'S|STOPPED|SPEEDING|DADS|MOMS|DAD'S|MOM'S|NEIGHBORS|NEIGHBOR'S|FRIENDS|FRIEND'S|PARKED)\s+){0,3}"
+VEH = r"(?:MVS?|CARS?|TRUCKS?|VEHICLES?|SUVS?|VANS?|MINIVAN|BUS|BUSES|AUTOS?|AUTOMOBILES?|PICKUPS?|SEMIS?|TAXI|CAB|JEEP|SEDAN|MOTOR\s+VEHICLES?|MOTORCYCLES?|TRACTOR[- ]TRAILER|18[- ]WHEELER|UBER|LYFT)"
+MV = re.compile("|".join([
+    rf"\b(?:HIT|STRUCK|STRIKE[SD]?|STRIKING|HITTING|GRAZED|CLIPPED|SIDE\s*SWIPED|SIDESWIPED|BUMPED|KNOCKED\s+(?:DOWN|OVER|OFF)|RUN\s+OVER|RAN\s+OVER|REAR[- ]ENDED|T-?BONED|COLLID\w*\s+WITH|CRASH\w*\s+INTO)\s+(?:BY\s+)?(?:(?:THE|A)\s+)?(?:(?:SIDE|BACK|FRONT|REAR|HOOD|DOOR)\s+OF\s+)?{MOD}{VEH}\b",
+    rf"(?:\bINTO|\bWITH|\bAGAINST|\bONTO|\bOVER|\bAT|\bW/)\s+(?:THE\s+|A\s+)?(?:BACK|SIDE|FRONT|HOOD|DOOR|REAR)?\s*(?:OF\s+)?{MOD}{VEH}\b(?!\s+(?:SEAT|PARK|LOT|WASH))",
+    rf"\b{VEH}\s*(?:VS\.?|V\.?|VERSUS)\s",
+    rf"\s(?:VS\.?|V\.?|VERSUS)\s+{MOD}{VEH}\b",
+    rf"\b{VEH}\b[^.;]{{0,40}}\b(?:HIT|HITS|STRUCK|STRIKES|RAN\s+INTO|CRASHED\s+INTO|COLLIDED|PULLED\s+OUT|CUT\s+(?:HIM|HER|THEM)\s+OFF|RAN\s+(?:HIM|HER)\s+OVER|BACKED\s+INTO|TURNED\s+INTO|DOOR\s+OPENED|OPENED\s+(?:THE|A|ITS|HIS|HER)?\s*DOOR|SIDE\s*SWIPED|SIDESWIPED|CLIPP\w+|SWERVED\s+INTO|KNOCKED|RAN\s+OVER|REAR[- ]ENDED|T-?BONED|BACKED\s+(?:UP\s+)?INTO)",
+    rf"\b{VEH}\b[^.;]{{0,60}}\b(?:CRASHED|RAN|SLAMMED|RODE|WENT|HIT)\s+(?:HEAD[- ]ON\s+)?(?:INTO\s+)?(?:IT|HIM|HER|THEM)\b",
+    rf"(?<!CUT OFF )(?<!CUT HIM OFF )(?<!CUT HER OFF )(?<!AVOID )\bBY\s+{MOD}{VEH}\b",
+    rf"\b(?:MIRROR|DOOR|TIRE|WHEEL|HOOD|BUMPER|PEDAL|FENDER|WINDSHIELD|TAILGATE|TRAILER)\s+OF\s+{MOD}{VEH}\b",
+    rf"\b(?:HIT|STRUCK|BUMPED)\s+(?:HIS\s+|HER\s+)?(?:HEAD|FACE|ARM|LEG|KNEE|SHOULDER|HAND)\s+(?:ON|AGAINST)\s+{MOD}{VEH}\b",
+    r"\b(?:MVC|MVA|MV\s*C|AUTO\s*(?:VS|PED)|AUTO-PED|PED(?:ESTRIAN)?\s+STRUCK(?!\s+BY\s+(?:A\s+|AN\s+)?(?:BICYCL\w*|BIKE\w*|E-?BIKE|SCOOTER|E-?SCOOTER|SKATEBOARD|MOPED|CYCLIST|BICYCLIST))|DOORED|CAR\s+DOOR|MOTOR\s+VEHICLE\s+(?:COLLISION|ACCIDENT|CRASH)|BIKE\s+VS\.?\s+(?:CAR|AUTO|TRUCK)|BICYCLE\s+VS\.?\s+(?:CAR|AUTO|TRUCK))\b",
+]))
+EARLIER = re.compile(r"\b(?:\d+\s+(?:DAYS?|WEEKS?|MONTHS?|YEARS?|YRS?|WKS?)\s+AGO|LAST\s+(?:WEEK|MONTH|YEAR))\b[^.]{0,60}\b(?:MVC|MVA|CAR\s+ACCIDENT)")
+
+LOADING = re.compile(rf"\b(?:LIFT\w*|LOAD\w*|PUT\w*|PLAC\w*|PUSH\w*|PULL\w*|CARRY\w*|GETTING|GET)\b[^.;]{{0,30}}\b(?:INTO|ONTO|IN|OUT\s+OF|OFF)\s+(?:THE\s+|A\s+|HIS\s+|HER\s+)?{VEH}\b")
+
+def _clean(text):
+    t = (text or "").upper()
+    return re.sub(r"(?<=[A-Z])DX\b", " DX", t)   # 'VEHICLEDX' -> 'VEHICLE DX'
+
+def motor_vehicle(text):
+    t = _clean(text)
+    t = LOADING.sub(" ", t)   # loading a bike into a car is not a crash
+    return bool(MV.search(t))
+
+NO = re.compile("|".join([
+    r"\bNO\s+(?:BIKE\s+|BICYCLE\s+)?HELMETS?(?=\b|DX)", r"\bUN-?HELM\w*", r"\bNO\s+WEARING\b", r"\bNON-?HELMETT?ED\b", r"\bU[A-Z]{1,2}HELMETT?ED\b", r"\bNOT\s+HELMETT?ED\b", r"\bREFUSED\s+TO\s+WEAR\s+(?:A\s+|HIS\s+|HER\s+)?HELMET", r"\bW/\s?O\s+(?:A\s+)?HELMET", r"\bNO\s+(?:HELMET\s+)?(?:AND|OR|/)\s*(?:NO\s+)?LOC\b",
+    r"\bW/O\s+(?:A\s+|HIS\s+|HER\s+)?HELMET", r"\bWITHOUT\s+(?:A\s+|HIS\s+|HER\s+|WEARING\s+(?:A\s+)?)?HELMET",
+    r"\bNOT\s+(?:WEARING|WEAR|USING|HAVE|HAVING)\s+(?:ON\s+)?(?:A\s+|HIS\s+|HER\s+|ANY\s+)?HELMET",
+    r"\b(?:WAS|WERE)\s*N'?O?T\s+(?:WEARING|HELMETT?ED)", r"\bWASN'?T\s+(?:WEARING|HELMETT?ED)",
+    r"\bDIDN'?T\s+(?:HAVE|WEAR)\s+(?:ON\s+)?(?:A\s+)?HELMET", r"\bDID\s+NOT\s+(?:HAVE|WEAR)\s+(?:ON\s+)?(?:A\s+)?HELMET",
+    r"\bDENIES\s+(?:WEARING\s+)?(?:A\s+)?HELMET", r"(?<![A-Z0-9])-\s?HELMET", r"\bHELMET\s*[-:]\s*(?:NO|NEG|NONE|-)(?!\w)",
+    r"\bHELMET\s*(?:-)(?!\w)", r"\bHELMET\s+(?:NOT|NEVER)\s+(?:WORN|USED|ON)", r"\bNEG(?:ATIVE)?\s+(?:FOR\s+)?HELMET",
+    r"\bNO\s+(?:LOC\s*/\s*|LOC\s*,\s*|LOC\s+OR\s+|LOC\s+AND\s+)HELMET", r"\bNOR\s+HELMET",
+]))
+YES = re.compile("|".join([
+    r"\+\s?HELMET", r"\bHELMETT?ED\b", r"\bHELMET\s*\+", r"\b(?:WEARING|WEARIN|WERARING|WORE|WEARS|WITH|HAD\s+ON|HAD|W/)\s*(?:ON\s+)?(?:A\s+|HIS\s+|HER\s+|THEIR\s+|FULL[- ]FACE\s+|FULL\s+|HALF\s+|BIKE\s+|BICYCLE\s+|SCOOTER\s+|PROPER\s+|DIRT\s*BIKE\s+)*HELMETS?(?=\b|DX)",
+    r"\bHELMETS?\s+(?:ON|USE|USED|WORN|IN\s+PLACE|PRESENT|WAS\s+ON|INTACT|DAMAGED|BROKE|BROKEN|CRACKED|SPLIT)\b",
+    r"\b(?:UNDER|THROUGH|CRACKED|CRACKING|BROKE|BREAKING|BROKEN|DAMAGED|DENTED|SPLIT|HIT|HITTING|STRIKING|STRUCK)\s+(?:HIS\s+|HER\s+|THE\s+|THEIR\s+)?HELMET",
+    r"\bHELMET\s*[:=]\s*(?:YES|Y|POS|\+)(?!\w)", r"\+\s?[A-Z ]{0,25}(?:AND|&)\s+HELMET", r"\bHELMETT?(?:ED)?\s+(?:DRIVER|RIDER|BICYCLIST|CYCLIST|OPERATOR|BIKER)", r"\bHELMETS?\s+(?:WAS|IS|GOT|WERE)\s+(?:CRACKED|BROKEN|DAMAGED|SPLIT|DENTED|ON|WORN|SCRATCHED)", r"\bPOS(?:ITIVE)?\s+(?:FOR\s+)?HELMET",
+]))
+UNK = re.compile(r"(?:\bHELMET\s*(?:USE\s+)?(?:NS|N/S|UNK|UNKNOWN|NOT\s+(?:STATED|SPECIFIED|DOCUMENTED|REPORTED|NOTED|KNOWN))|UNKNOWN\s+(?:IF\s+|WHETHER\s+)?(?:\w+\s+)?HELMET|UNK\s+(?:IF\s+)?HELMET|\?\s*HELMET|HELMET\s*\?|HELMET\s+USE\s+UNK)")
+
+def helmet(text):
+    t = _clean(text)
+    if "HELMET" not in t:
+        return "not mentioned"
+    if UNK.search(t):
+        return "not mentioned"
+    no = bool(NO.search(t))
+    yes = bool(YES.search(NO.sub(" ", t)))
+    if no and not yes: return "no"
+    if yes and not no: return "yes"
+    return "not mentioned"

neissai-1.4.0/src/neissai/classifier.py

@@ -0,0 +1,102 @@
+"""A small, dependency-light text classifier: e-bike vs moped, from NEISS injury narratives.
+
+The model is TF-IDF (words and two-word phrases) + logistic regression, trained on NEISS
+2024-2025 records where CPSC's codes separate e-bikes (5045) from other power-assisted
+cycles (5046). It is stored as plain JSON (vocabulary, IDF weights, coefficients), so it
+runs with numpy only and does not depend on a scikit-learn version. Retrain with
+``scripts/train_classifier.py``.
+"""
+from __future__ import annotations
+
+import json
+import math
+import re
+from collections import Counter
+from dataclasses import dataclass, field
+from importlib import resources
+from pathlib import Path
+from typing import Iterable, Union
+
+import numpy as np
+
+_TOKEN = re.compile(r"(?u)\b\w\w+\b")  # scikit-learn's default token pattern
+
+
+def _features(text: str, ngram_max: int = 2) -> Counter:
+    tokens = _TOKEN.findall(text.lower())
+    feats: Counter = Counter(tokens)
+    for n in range(2, ngram_max + 1):
+        feats.update(" ".join(tokens[i:i + n]) for i in range(len(tokens) - n + 1))
+    return feats
+
+
+@dataclass
+class EbikeMopedClassifier:
+    idf: dict[str, float]
+    coef: dict[str, float]
+    intercept: float
+    ngram_max: int = 2
+    sublinear_tf: bool = True
+    meta: dict = field(default_factory=dict)
+
+    # ---------- prediction ----------
+    def decision(self, text: str) -> float:
+        feats = _features(text, self.ngram_max)
+        weights = {}
+        for term, tf in feats.items():
+            idf = self.idf.get(term)
+            if idf is None:
+                continue
+            weights[term] = (1.0 + math.log(tf) if self.sublinear_tf else float(tf)) * idf
+        norm = math.sqrt(sum(w * w for w in weights.values())) or 1.0
+        return self.intercept + sum(w / norm * self.coef.get(t, 0.0) for t, w in weights.items())
+
+    def predict_proba(self, texts: Iterable[str]) -> np.ndarray:
+        """Probability that each narrative describes an e-bike (vs a moped or similar)."""
+        z = np.array([self.decision(t) for t in texts], dtype=float)
+        return 1.0 / (1.0 + np.exp(-z))
+
+    # ---------- storage ----------
+    def to_json(self, path: Union[str, Path]) -> None:
+        terms = sorted(self.idf)
+        payload = {"meta": self.meta, "ngram_max": self.ngram_max, "sublinear_tf": self.sublinear_tf,
+                   "intercept": round(self.intercept, 8),
+                   "terms": terms,
+                   "idf": [round(self.idf[t], 6) for t in terms],
+                   "coef": [round(self.coef.get(t, 0.0), 6) for t in terms]}
+        Path(path).write_text(json.dumps(payload, separators=(",", ":")))
+
+    @classmethod
+    def from_json(cls, path_or_text: Union[str, Path]) -> "EbikeMopedClassifier":
+        s = str(path_or_text)
+        text = s if s.lstrip().startswith("{") else Path(s).read_text()
+        p = json.loads(text)
+        idf = dict(zip(p["terms"], p["idf"]))
+        coef = {t: c for t, c in zip(p["terms"], p["coef"]) if c != 0.0}
+        return cls(idf=idf, coef=coef, intercept=p["intercept"], ngram_max=p.get("ngram_max", 2),
+                   sublinear_tf=p.get("sublinear_tf", True), meta=p.get("meta", {}))
+
+    @classmethod
+    def load_default(cls) -> "EbikeMopedClassifier":
+        text = resources.files("neissai").joinpath("model/ebike_moped_v1.json").read_text()
+        return cls.from_json(text)
+
+    # ---------- training (needs scikit-learn) ----------
+    @classmethod
+    def fit(cls, texts: list[str], is_ebike: list[int], min_df: int = 3, C: float = 4.0,
+            cv_folds: int = 5) -> "EbikeMopedClassifier":
+        from sklearn.feature_extraction.text import TfidfVectorizer
+        from sklearn.linear_model import LogisticRegression
+        from sklearn.model_selection import cross_val_score
+
+        vec = TfidfVectorizer(ngram_range=(1, 2), min_df=min_df, sublinear_tf=True)
+        X = vec.fit_transform(texts)
+        lr = LogisticRegression(max_iter=2000, C=C, class_weight="balanced")
+        auc = float(cross_val_score(lr, X, is_ebike, cv=cv_folds, scoring="roc_auc").mean())
+        lr.fit(X, is_ebike)
+        vocab = vec.get_feature_names_out()
+        idf = dict(zip(vocab, vec.idf_.tolist()))
+        coef = dict(zip(vocab, lr.coef_[0].tolist()))
+        meta = {"n_train": len(texts), "n_ebike": int(sum(is_ebike)), "cv_auc": round(auc, 4),
+                "model": "TF-IDF (1-2 grams, sublinear) + logistic regression (balanced)"}
+        return cls(idf=idf, coef=coef, intercept=float(lr.intercept_[0]), meta=meta)

neissai-1.4.0/src/neissai/cli.py

@@ -0,0 +1,98 @@
+"""Command line interface.
+
+    neissai download     --years 2019-2025 --data-dir data
+    neissai micromobility --years 2019-2025 --data-dir data --out-dir results
+    neissai classify "14YOM RIDING ELECTRIC BIKE FELL OFF"
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+import pandas as pd
+
+from . import __version__
+from .classifier import EbikeMopedClassifier
+from .devices import DEVICES, add_outcomes, label_devices
+from .estimate import national_estimates, shares
+from .io import download, read_neiss
+
+
+def _fmt(x: float) -> str:
+    return f"{x/1000:,.1f}k"
+
+
+def cmd_download(a: argparse.Namespace) -> int:
+    for p in download(a.years, a.data_dir):
+        print(f"saved {p}")
+    return 0
+
+
+def cmd_micromobility(a: argparse.Namespace) -> int:
+    out = Path(a.out_dir)
+    out.mkdir(parents=True, exist_ok=True)
+    print(f"Reading NEISS files for {a.years} from {a.data_dir} ...")
+    df = read_neiss(a.data_dir, a.years)
+    print(f"  {len(df):,} injury records")
+    print("Labelling devices (codes + narratives + text classifier) ...")
+    df = add_outcomes(label_devices(df, threshold=a.threshold))
+    mm = df["device"].isin(DEVICES)
+    print(f"  {int(mm.sum()):,} micromobility records; "
+          f"{int((df['label_source'] == 'classifier').sum()):,} labelled by the classifier")
+
+    # Estimates use the full sample design (all records), counting only the selected ones.
+    by_year = national_estimates(df, by=["year", "device"], where=mm)
+    by_age = national_estimates(df, by=["year", "device", "age_group"], where=mm & df["age_group"].notna())
+    sev = pd.concat([shares(df, o, by=["device"], where=mm)
+                     for o in ("hospitalized", "head_injury", "motor_vehicle", "helmet_no", "alcohol")])
+    df.loc[mm].to_csv(out / "micromobility_records.csv", index=False)
+    by_year.to_csv(out / "estimates_by_year_device.csv", index=False)
+    by_age.to_csv(out / "estimates_by_year_device_age.csv", index=False)
+    sev.to_csv(out / "shares_by_device.csv", index=False)
+
+    tab = by_year.pivot(index="device", columns="year", values="estimate").reindex(DEVICES).dropna(how="all")
+    print("\nEstimated US emergency department visits (thousands)")
+    print(tab.map(lambda v: "" if pd.isna(v) else _fmt(v)).to_string())
+    unstable = by_year[by_year["unstable"]]
+    if len(unstable):
+        print(f"\n{len(unstable)} estimate(s) are flagged unstable (CPSC rule); see the 'unstable' column.")
+    print(f"\nSaved results to {out.resolve()}")
+    return 0
+
+
+def cmd_classify(a: argparse.Namespace) -> int:
+    clf = EbikeMopedClassifier.load_default()
+    for text in a.text:
+        p = float(clf.predict_proba([text])[0])
+        print(f"{p:.3f}  {'E-bike' if p >= 0.5 else 'Moped / other power-assisted cycle'}  |  {text}")
+    return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(prog="neissai", description="AI-assisted national injury estimates from NEISS narratives.")
+    ap.add_argument("--version", action="version", version=f"neissai {__version__}")
+    sub = ap.add_subparsers(dest="command", required=True)
+
+    d = sub.add_parser("download", help="download CPSC NEISS annual files (best effort)")
+    d.add_argument("--years", default="2019-2025")
+    d.add_argument("--data-dir", default="data")
+    d.set_defaults(func=cmd_download)
+
+    m = sub.add_parser("micromobility", help="e-bike, e-scooter, bicycle and related injury estimates")
+    m.add_argument("--years", default="2019-2025")
+    m.add_argument("--data-dir", default="data")
+    m.add_argument("--out-dir", default="results")
+    m.add_argument("--threshold", type=float, default=0.5, help="classifier cut-off for 'E-bike' (default 0.5)")
+    m.set_defaults(func=cmd_micromobility)
+
+    c = sub.add_parser("classify", help="e-bike vs moped probability for narrative text")
+    c.add_argument("text", nargs="+")
+    c.set_defaults(func=cmd_classify)
+
+    a = ap.parse_args(argv)
+    return a.func(a)
+
+
+if __name__ == "__main__":  # pragma: no cover
+    sys.exit(main())