pymetamaplite 0.1.0__tar.gz

pymetamaplite-0.1.0/.gitignore

@@ -0,0 +1,46 @@
+.venv/
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+ivf/
+# Java MetaMapLite output: contains UMLS-derived strings, do not commit
+tests/parity/fixtures/
+*.RRF
+.pytest_cache/
+.ruff_cache/
+# examples/: the scripts, their README and their tests are source and ship. Everything they
+# *produce* is UMLS-derived and does not -- annotation exports, the loaded databases and the
+# filtered MRCONSO/MRREL extracts. Listed by extension rather than by name so a new export
+# format is excluded by default: only .py, .md and the synthetic .txt notes are tracked.
+examples/*.csv
+examples/*.jsonl
+examples/*.parquet
+examples/*.db
+examples/*.sqlite
+examples/*.json
+
+# annotation run output (examples/parse_to_*.py): verbatim note text and UMLS-derived
+# strings, one directory per run. Never commit it.
+out/
+# corpora of notes to annotate: real ones are PHI, and even synthetic ones are not source
+free_texts/
+
+# local agent/tool state (a shared .claude/settings.json can still be committed)
+# and per-user editor/notes vaults
+.obsidian/
+.claude/*.lock
+
+# local credentials
+.env
+.env.*
+
+# internal working document, not distributed
+docs/DEVELOPMENT_PLAN.md
+
+# internal working documents, not distributed
+docs/OPERATIONS_MANUAL.md
+docs/USER_GUIDE.md
+docs/Negation detection.md
+docs/Deploying to another machine.md

pymetamaplite-0.1.0/CHANGELOG.md

@@ -0,0 +1,189 @@
+# Changelog
+
+All notable changes to pymetamaplite are recorded here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Rationale for the parity-driven decisions behind these changes is recorded in the project's
+development plan, an internal working document that is not distributed.
+
+## [0.1.0] — 2026-09-19
+
+First release. A Python port of MetaMapLite 3.6.2rc8 — the whole `EntityLookup4` path — as a
+library, a CLI and a REST API, with no JVM.
+
+Measured against Java MetaMapLite 3.6.2rc8 on UMLS 2026AA: the index tables are identical row
+for row, tokenization is exact, and concepts found `(CUI, positions)` agree at P 0.975 / R 0.963
+/ F1 0.969. Most of the residual gap is the two NLP models MetaMapLite uses (OpenNLP's sentence
+detector and POS tagger), which have no Python equivalent. See
+[tests/parity/README.md](tests/parity/README.md) for the method.
+
+### Annotation
+
+- **Index building** from UMLS RRF (`MRCONSO`, `MRSTY`, optionally `MRSAT` for MeSH tree codes),
+  producing the `cuisourceinfo`, `cuiconcept`, `cuist` and `meshtcrelaxed` tables in SQLite.
+  `--sources` restricts the build; `--include-suppressed` keeps `SUPPRESS=O/E/Y` rows.
+- **The lookup pipeline**: normalization, sentence segmentation, tokenization, POS gating,
+  longest-match candidate generation, semantic-type and source restriction (abbreviations or
+  TUIs, any case), subsumption removal, Schwartz–Hearst abbreviation propagation, user-defined
+  acronyms, custom concept lists and excluded terms.
+- **Negation**, two detectors: NegEx (the default, as in Java) and ConText (`--usecontext`),
+  which also reports temporality and experiencer.
+- **Greek-letter spans.** A span may start or end with a Greek letter, as Java's `CharUtils`
+  allows, so `β-blocker` matches.
+- **Non-ASCII dictionary keys** resolve reliably. Java's binary search compares UTF-8 as signed
+  bytes against a file sorted in `String` order and finds only about 44% of them, missing terms
+  such as `Sjögren` and `Ménière disease`; this is a deliberate divergence, pinned by name in the
+  test suite rather than averaged into the parity figures.
+- **Precision filter** (`Settings.precision_filter`, default off, not in Java): drops a concept
+  whose only dictionary strings for the matched text are acronyms when the text is not in
+  capitals ("Plan" loses OMIM's `PLAN`), and one-word function-word mentions ("for", "with")
+  unless capitalised. Over the parity corpus it removed 267 of 2,907 matches, 261 of them (97.8%)
+  judged wrong or useless by hand.
+
+### Input and output
+
+- **Every input format MetaMapLite registers**: `freetext`, `sli`, `sldi`, `sldiwi`, `chemdner`,
+  `chemdnersldi`, `ncbicorpus`, `pubtator`, `pubmed`, `medline`, `bioc`. `pubtator` is a
+  correction rather than a port — Java's loader closes its reader inside the read loop and throws
+  on every input. A document model and loader registry (`pymetamaplite.documents`) takes a new
+  format without changes elsewhere.
+- **Output formats**: `mmi` (the default), `json`, `brat`, `cuilist`, `full`, and `bc` with
+  Java's aliases `bc-evaluate`, `cdi` and `bioc`.
+- **Segmentation**: `SENTENCES`, `BLANKLINES`, `LINES`.
+
+### Interfaces
+
+- **CLI** (`pymetamaplite`): `annotate`, `lookup`, `normalize`, `tokenize`, `build-index`,
+  `index-stats`, `serve`. `annotate` mirrors `metamaplite.sh`.
+- **Python API**: `MetaMapLite(index_directory=...)`, `process_text`, `process_file`, `format`.
+  Configuration comes from a `Settings` object, MetaMapLite's own `metamaplite.properties` keys,
+  or `PYMETAMAPLITE_<NAME>` environment variables.
+- **REST API** (`serve`): `POST /annotate`, `/annotate/batch`, `/annotate/text`,
+  `/annotate/formatted`, `GET /lookup`, `/formats`, `/semantic-types`, `/health`, with OpenAPI
+  docs at `/docs`. Requests are serialised behind a lock — annotation is CPU-bound and the
+  underlying objects are not thread-safe — so scale out with processes.
+- **Opt-in REST authentication and CORS.** `PYMETAMAPLITE_API_TOKEN` makes every endpoint except
+  `/health` require `Authorization: Bearer <token>` (constant-time comparison);
+  `PYMETAMAPLITE_CORS_ORIGINS` adds `CORSMiddleware`. Both off unless set.
+- **Request limits**: `PYMETAMAPLITE_MAX_TEXT_CHARS` (50,000, per document) and
+  `PYMETAMAPLITE_MAX_FORMATTED_CHARS` (1,000,000, the raw `/annotate/formatted` body). Both are
+  checked right after parsing, before anything is annotated.
+
+### Configuration and extension
+
+- **`strict_parity`** (default `true`). `true` reproduces MetaMapLite 3.6.2rc8 exactly, bugs
+  included. `false` fixes the upstream bugs that give wrong answers, all in ConText:
+  - Only the mentions inside the sentence being read are analysed, each within its own bullet
+    item. Java applies every sentence to every mention in the document, so a later "Family
+    history of hypertension in mother" turned the patient's own earlier "hypertension" into
+    `Historical` / `Other`. This also makes ConText close to linear in document length rather
+    than quadratic.
+  - The longest trigger at a position wins, instead of whichever bucket is substituted first.
+    "Pneumonia was ruled out." is negated by the *post* trigger "was ruled out"; Java tagged it
+    with the shorter *pre* trigger "ruled out", which lands at the end of the sentence with
+    nothing after it to negate. "may be ruled out for X" is `Possible`, not a negation.
+  - Triggers stored with a trailing space (`denies `, `no `) fire anywhere, not only right before
+    the concept, so a whole "Denies A, B, C" list is negated rather than its first item — and
+    "no evidence of X" / "without evidence of X" negate at all.
+  - Time patterns match ("3 months of", "since last march"); they were character classes where
+    groups were meant. "previous" marks the next word historical. The first pseudo-trigger keeps
+    its first letter. "Rule out X" / "r/o X" gives `Possible`.
+  - Experiencer and temporality triggers inside a concept's own name count; negation triggers
+    there deliberately do not ("No known allergies" is itself the finding).
+  - `mmi`, `json` and `brat` list semantic types and sources alphabetically and documents in
+    input order, instead of Java `HashSet` order.
+
+  NegEx output is identical in both modes. `GET /health` reports `strict_parity`, and a server
+  running strict-parity ConText logs a warning once.
+- **Extra ConText triggers** (`context_triggers.EXTRA_TRIGGERS`, fixed mode only). `cannot` is a
+  negation trigger on its own, so "cannot exclude pneumonia" came out *negated* — the opposite of
+  what it says. Adds the exclusion family (`cannot exclude`, `cannot be excluded`,
+  `cannot rule out`, `unable to exclude`, …), the hedges `concerning for`, `suspicious for`,
+  `suspected`, `differential includes`, `low suspicion for`, `possible`, `probable`, and
+  `absent` / `is absent` / `are absent`. `ConText(extra_triggers=...)` takes your own list.
+- **`Entity.assertion`** carries ConText's verdict — `Affirmed`, `Negated` or `Possible`. Java
+  keeps only a boolean and discards the verdict, so a hedged mention read exactly like an
+  affirmed one. Shown as `[possible]` in `full` output and returned by the REST API; `negated` is
+  unchanged, so `mmi`, `json` and `brat` are untouched. `None` under NegEx.
+- **Pluggable assertion.** `AssertionClassifier` adapts any per-sentence assertion model
+  (medspaCy's ConText, a transformer classifier, site rules) to the pipeline.
+  `register_negation_detector(name, factory, aliases)` makes it selectable wherever `negex` /
+  `context` are, and packages can declare one under the `pymetamaplite.negation_detectors`
+  entry-point group. `--negation-detector NAME` on `annotate` and `serve`.
+- **Selectable POS tagger.** `Settings.postag_model` names the spaCy pipeline whose tagger gates
+  candidate spans; the default is `en_core_web_sm`. scispaCy's `en_core_sci_sm` agrees with Java
+  better (F1 0.977 vs 0.969) and is opt-in because its models need spaCy 3.7 / Python ≤ 3.12 —
+  `requirements-scispacy-py312.txt` pins a tested environment for it.
+- **POS verb rescue** (`Settings.postag_verb_rescue`, default `0.03`) re-tags a token the model
+  calls a finite verb with its most likely noun or adjective reading. `en_core_web_sm` calls rare
+  clinical words verbs far more readily than OpenNLP, and the POS gate then never looks them up.
+  Raises parity F1 from 0.969 to 0.974 while *reducing* extras.
+
+### Performance
+
+- Annotation runs at ~500 sentences/s on one core; resident memory ~120 MB. An index build over
+  UMLS 2026AA (all English sources) takes ~2.5 minutes and produces a 2.2 GB SQLite database:
+  9.1 M terms, 7.3 M distinct keys, 3.5 M CUIs.
+- Abbreviation propagation tokenizes each passage once rather than once per abbreviation-bearing
+  entity (48 KB: 7.0 s → 2.1 s, byte-identical output), and NegEx receives only the entities
+  starting inside each sentence.
+- The four per-concept caches are LRU-capped at 50,000 entries each (~125 MB together) instead of
+  growing for the life of the process.
+- `--preload` / `PYMETAMAPLITE_PRELOAD_INDEX` is off by default and documented as rarely worth
+  turning on: measured A/B with the index on local disk it is no faster, and costs ~4 s of
+  start-up and ~1.65 GB RSS per process.
+
+### Not implemented, deliberately
+
+- **MetaMap-style scoring** (`EntityLookup5`, Java's `--enable_scoring`), off by default in Java.
+  Measured rather than assumed: with NLM's own index the score takes one of two values, carrying
+  ~0.04 bits per evidence. See the development plan, §10.1.
+- **Word sense disambiguation, derivational variants, disjoint entities** — absent from
+  MetaMapLite itself.
+
+### Examples
+
+Runnable scripts in [examples/](examples/) — a starting point to adapt, not a library. See
+[examples/README.md](examples/README.md).
+
+- **Annotation** — `analyze_file.py`, `analyze_folder.py` (one loaded index reused across a
+  folder, `--workers` for a process pool) and `json_to_sqlite.py`, which loads `--outputformat
+  json` files into a normalized `entities`/`concepts`/`evidence` database.
+- **Corpus exports** — `parse_to_csv.py`, `parse_to_jsonl.py`, `parse_to_jsonl_batch.py`,
+  `parse_to_parquet.py` and `parse_to_sqlite.py` write one row per (span, concept) in five
+  formats from a single row shape (`_annotations.py`), so exports of one corpus agree and join
+  on the same key. `compare_exports.py` checks that rather than asserting it, and exits non-zero
+  on disagreement. `load_to_sqlite.py` completes the two-step route, sharing its schema with
+  `parse_to_sqlite.py` so both end at the same database.
+  - An attribute the run never assessed gets **no column**, and where one is kept it is
+    empty/NULL rather than 0: NegEx assesses `negated` only, `--usecontext` adds `assertion`,
+    `temporality` and `experiencer`, `--no-negation` assesses none. NULL means "not assessed"
+    and 0 means "assessed, and absent".
+  - Each run saves itself into a fresh `out/runs/<run-id>/` directory beside a `manifest.json`
+    recording the settings that produced it (`_runs.py`), so no run overwrites another.
+- **Phenotyping** — `notes_of_interest.py` finds the notes that *affirm* a concept, counting a
+  document only when the mention is not negated, and matching by CUI so synonym and
+  morphological variants collapse.
+- **Terminology crosswalks** — `filter_mrconso.py` / `filter_mrrel.py` and the `add_*_codes.py`
+  and `add_*_hierarchy.py` scripts map matched concepts to SNOMED CT, ICD-10-CM, RxNorm and
+  LOINC, with `create_combined_codes_view.py` over the result.
+
+### Documentation
+
+- [docs/GETTING_STARTED.md](docs/GETTING_STARTED.md) — install to first results.
+- [docs/UMLS identifiers.md](<docs/UMLS identifiers.md>) — CUI, LUI, SUI, AUI and TUI explained.
+
+### Licensing
+
+Released under the same licence as MetaMapLite — NLM's open-source BSD licence — with the
+upstream Informational Notice retained in [NOTICE-MetaMapLite.md](NOTICE-MetaMapLite.md).
+pymetamaplite is an independent port and is **not** developed, funded or endorsed by NLM, NIH or
+the U.S. Government. The UMLS data it indexes is licensed separately.
+
+### Quality
+
+706 tests, `ruff` clean, `mypy --strict` clean over `src/pymetamaplite`, CI on Python 3.11–3.13.
+Every `examples/` script has its own test file under `tests/examples/`, run against small
+synthetic MRCONSO/MRSTY fixtures and an index built from them, so the suite needs no UMLS
+download. The parity tests skip wherever licensed fixtures are absent.

pymetamaplite-0.1.0/LICENSE

@@ -0,0 +1,75 @@
+# Terms and Conditions for Use of pymetamaplite
+
+pymetamaplite is a Python port of MetaMapLite. It is released under the same terms as
+MetaMapLite itself: the open-source BSD license published by the U.S. National Library of
+Medicine, reproduced in full below.
+
+Scope, so the provenance is unambiguous:
+
+* MetaMapLite was developed and funded by the National Library of Medicine. The Informational
+  Notice and LICENSE below are NLM's, and are reproduced verbatim from the MetaMapLite
+  distribution (see NOTICE-MetaMapLite.md). They apply to MetaMapLite and to the portions of
+  this software derived from it — algorithms, data tables, field layouts and output formats.
+* pymetamaplite itself was not developed, funded, endorsed or reviewed by the National Library
+  of Medicine, the National Institutes of Health, or the U.S. Government. Copyright in the
+  Python code written for this port is held by its contributors, who release it under these
+  same terms.
+* The UMLS Metathesaurus data this software indexes is licensed separately by NLM and is not
+  covered by this license. See the note in the Informational Notice below.
+
+---
+
+# Informational Notice:
+
+This software, “MetaMapLite” was developed and funded by the National
+Library of Medicine, part of the National Institutes of Health, and
+agency of the United States Department of Health and Human Services,
+which is making the software available to the public for any
+commercial or non-commercial purpose under the following open-source
+BSD license.
+
+NOTE: Users of the data distributed with MetaMapLite are
+responsible for compliance with the UMLS Metathesaurus License
+Agreement which requires you to respect the copyrights of the
+constituent vocabularies and to file a brief annual report on your use
+of the UMLS. You also must have activated a UMLS Terminology Services
+(UTS) account.
+
+# LICENSE:
+
+Government Usage Rights Notice: The U.S. Government retains unlimited,
+royalty-free usage rights to this software, but not ownership, as
+provided by Federal law.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+* Redistributions of source code must retain this Informational Notice.
+* Redistributions in binary form must reproduce this Informational
+  Notice, this list of conditions and the following disclaimer in the
+  documentation and/or other materials provided with the distribution.
+* Neither the names of the National Library of Medicine, the National
+  Institutes of Health, nor the names of any of the software
+  developers may be used to endorse or promote products derived from
+  this software without specific prior written permission.  The
+  U.S. Government retains an unlimited, royalty-free right to use,
+  distribute or modify the software.
+* Please acknowledge NLM as the source of the MetaMapLite software by
+  including the phrase “Courtesy of the U.S. National Library of
+  Medicine” or “Source: U.S. National Library of Medicine.”
+
+
+THIS SOFTWARE IS PROVIDED BY THE U.S. GOVERNMENT AND CONTRIBUTORS "AS
+IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITEDTO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+U.S. GOVERNMENT
+
+OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

pymetamaplite-0.1.0/NOTICE-MetaMapLite.md

@@ -0,0 +1,56 @@
+# Terms and Conditions for Use of MetaMapLite
+
+# Informational Notice:
+
+This software, “MetaMapLite” was developed and funded by the National
+Library of Medicine, part of the National Institutes of Health, and
+agency of the United States Department of Health and Human Services,
+which is making the software available to the public for any
+commercial or non-commercial purpose under the following open-source
+BSD license.
+
+NOTE: Users of the data distributed with MetaMapLite are
+responsible for compliance with the UMLS Metathesaurus License
+Agreement which requires you to respect the copyrights of the
+constituent vocabularies and to file a brief annual report on your use
+of the UMLS. You also must have activated a UMLS Terminology Services
+(UTS) account.
+
+# LICENSE:
+
+Government Usage Rights Notice: The U.S. Government retains unlimited,
+royalty-free usage rights to this software, but not ownership, as
+provided by Federal law.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+* Redistributions of source code must retain this Informational Notice. 
+* Redistributions in binary form must reproduce this Informational
+  Notice, this list of conditions and the following disclaimer in the
+  documentation and/or other materials provided with the distribution.
+* Neither the names of the National Library of Medicine, the National
+  Institutes of Health, nor the names of any of the software
+  developers may be used to endorse or promote products derived from
+  this software without specific prior written permission.  The
+  U.S. Government retains an unlimited, royalty-free right to use,
+  distribute or modify the software.
+* Please acknowledge NLM as the source of the MetaMapLite software by
+  including the phrase “Courtesy of the U.S. National Library of
+  Medicine” or “Source: U.S. National Library of Medicine.”
+
+
+THIS SOFTWARE IS PROVIDED BY THE U.S. GOVERNMENT AND CONTRIBUTORS "AS
+IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITEDTO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+U.S. GOVERNMENT
+
+OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

pymetamaplite-0.1.0/PKG-INFO

@@ -0,0 +1,267 @@
+Metadata-Version: 2.5
+Name: pymetamaplite
+Version: 0.1.0
+Summary: Python port of NLM MetaMapLite: near-real-time UMLS concept recognition
+Project-URL: MetaMapLite (upstream), https://lhncbc.nlm.nih.gov/LHC-research/LHC-projects/NLP/MetaMapLite.html
+Project-URL: MetaMapLite source (upstream), https://github.com/lhncbc/metamaplite
+Project-URL: UMLS licensing, https://uts.nlm.nih.gov/uts/
+License-Expression: LicenseRef-MetaMapLite-BSD
+License-File: LICENSE
+License-File: NOTICE-MetaMapLite.md
+Keywords: biomedical,concept-recognition,metamap,metamaplite,ner,nlp,umls
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Healthcare Industry
+Classifier: Intended Audience :: Science/Research
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: typer>=0.12
+Provides-Extra: all
+Requires-Dist: fastapi>=0.110; extra == 'all'
+Requires-Dist: spacy>=3.7; extra == 'all'
+Requires-Dist: uvicorn>=0.29; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: httpx>=0.27; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Provides-Extra: nlp
+Requires-Dist: spacy>=3.7; extra == 'nlp'
+Provides-Extra: server
+Requires-Dist: fastapi>=0.110; extra == 'server'
+Requires-Dist: uvicorn>=0.29; extra == 'server'
+Description-Content-Type: text/markdown
+
+# pymetamaplite
+
+A Python port of [MetaMapLite](https://lhncbc.nlm.nih.gov/LHC-research/LHC-projects/NLP/MetaMapLite.html),
+the U.S. National Library of Medicine's near-real-time UMLS concept recognizer — as a library,
+a CLI, and a REST API, with no JVM.
+
+*Courtesy of the U.S. National Library of Medicine.*
+
+This README is a quick overview. **New here? Start with
+[docs/GETTING_STARTED.md](docs/GETTING_STARTED.md)** — install, build the index and get your
+first results in under an hour. New to UMLS identifiers (CUI, LUI, SUI, AUI, TUI)? See
+[docs/UMLS identifiers.md](<docs/UMLS identifiers.md>). Release notes are in
+[CHANGELOG.md](CHANGELOG.md). Every CLI flag has `--help`, and the REST API documents itself at
+`/docs`.
+
+## Verified against the original
+
+Measured against **Java MetaMapLite 3.6.2rc8** on UMLS 2026AA (see
+[tests/parity/README.md](tests/parity/README.md) for the method and how to reproduce):
+
+| | |
+|---|---|
+| Index tables (`cuisourceinfo`, `cuiconcept`, `meshtcrelaxed`) | **identical, row for row** |
+| Tokenization | **exact**, token for token |
+| Concepts found, `(CUI, positions)` | **P 0.975 / R 0.963 / F1 0.969** |
+| MMI records byte-identical | 430/565 |
+| `json` output, spans found by both | 285/296 agree on every field Java emits |
+
+Index contents, normalization, candidate generation, restriction, subsumption, negation,
+abbreviations, MMI scoring and field layout are reproduced deterministically. Most of the
+residual gap is the two NLP models MetaMapLite uses — OpenNLP's sentence detector (F1 0.923 vs
+ours) and its POS tagger (0.850 tag agreement, 0.960 on the decision that affects results) —
+which have no Python equivalent. One known difference is deliberate: Java's index lookup is
+**unreliable for dictionary keys containing non-ASCII characters** — its binary search compares
+UTF-8 as signed bytes against a file sorted in `String` order, so it finds some and misses others
+depending on where the key falls. Measured over a 600-key sample it resolves about 44% of them,
+missing terms such as `Sjögren` and `Ménière disease` that this port finds. That accounts for five
+of the 35 concept-level disagreements, each pinned by name in the test suite rather than averaged
+into the figures above. See the development plan, §16 and §21.
+
+## Install
+
+```powershell
+uv venv
+uv pip install --python .venv -e ".[dev,nlp,server]"
+uv pip install --python .venv en_core_web_sm@https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
+```
+
+Extras: `nlp` (spaCy — POS tagging and spaCy sentence detection), `server` (FastAPI + uvicorn),
+`all` for both. The core package needs only `typer`; without `nlp`, pass `--no-postag`.
+
+The POS tagger is a spaCy model chosen by name (`--postag-model`, `PYMETAMAPLITE_POSTAG_MODEL`).
+On Python ≤ 3.12, scispaCy's `en_core_sci_sm` measures **closer to Java than the default**
+(F1 0.977 vs 0.969) and finds 27/30 drugs in a medication list where `en_core_web_sm` finds 20 —
+see `--postag-model` in `pymetamaplite annotate --help`.
+
+## Build an index
+
+You supply the UMLS — a licensed download, never redistributed with this package. `MRCONSO.RRF`
+and `MRSTY.RRF` are required, `MRSAT.RRF` adds MeSH tree codes (used by MMI scoring):
+
+```powershell
+pymetamaplite -v build-index --mrconso ...\MRCONSO.RRF --mrsty ...\MRSTY.RRF --mrsat ...\MRSAT.RRF --out ivf
+```
+
+UMLS 2026AA (all English sources) takes ~2.5 min and produces a 2.2 GB SQLite database:
+9.1 M terms / 7.3 M distinct keys / 3.5 M CUIs. `--sources MSH,SNOMEDCT_US` restricts it;
+`--include-suppressed` keeps `SUPPRESS=O/E/Y` rows.
+
+## Command line
+
+```powershell
+pymetamaplite annotate "No history of myocardial infarction." --restrict-to-sts dsyn,sosy
+pymetamaplite annotate --input note.txt --outputformat json     # mmi (default) | json | brat | cuilist | full | bc
+pymetamaplite annotate --pipe --no-negation < note.txt
+pymetamaplite lookup "type 2 diabetes mellitus"                 # dictionary spot-check
+pymetamaplite normalize "Alzheimer's Disease"                   # show the lookup keys for a span
+pymetamaplite tokenize "Dr. Smith has type 2 diabetes."         # sentences + tokens + POS
+pymetamaplite index-stats
+```
+
+`annotate` mirrors `metamaplite.sh`: `--restrict-to-sts`, `--restrict-to-sources`,
+`--segmentation SENTENCES|BLANKLINES|LINES`, `--inputformat` (`freetext`, `sli`, `sldi`,
+`sldiwi`, `chemdner`, `chemdnersldi`, `ncbicorpus`, `pubtator`, `pubmed`, `medline`, `bioc`),
+`--excluded-terms`, `--uda`,
+`--cui-term-list`, `--usecontext`, `--keep-subsumed`, `--preload`. Restrictions take semantic type abbreviations or TUIs and UMLS
+SABs, in any case (`dsyn`, `DSYN`, `T047` and `t047` all select the same concepts).
+
+## Python API
+
+```python
+from pymetamaplite import MetaMapLite
+
+mml = MetaMapLite(index_directory="ivf")
+entities = mml.process_text(
+    "No history of type 2 diabetes mellitus (T2DM). T2DM is treated with metformin.",
+    restrict_to_sts={"dsyn", "phsu"},
+)
+for e in entities:
+    print(e.start, e.length, e.text, e.negated, [(ev.cui, ev.concept.preferred_name) for ev in e.evs])
+print(mml.format(entities, "mmi"))
+```
+
+```
+14 24 type 2 diabetes mellitus True  [('C0011860', 'Diabetes Mellitus, Non-Insulin-Dependent')]
+40  4 T2DM                     False [('C0011860', 'Diabetes Mellitus, Non-Insulin-Dependent')]
+47  4 T2DM                     False [('C0011860', 'Diabetes Mellitus, Non-Insulin-Dependent')]
+68  9 metformin                False [('C0025598', 'metformin')]
+```
+
+`mml.process_file("note.txt")` annotates a file directly — offsets index the file verbatim
+(line endings are never translated), so `brat` output lines up with a CRLF `.txt`.
+
+`MetaMapLite(...)` takes a `Settings` object or `properties_file=` accepting MetaMapLite's own
+`metamaplite.properties` keys (`metamaplite.index.directory`, `metamaplite.semanticgroup`,
+`metamaplite.enable.postagging`, `metamaplite.detect.negations`, `metamaplite.postaglist`,
+`metamaplite.uda.filename`, …); every key also reads from `PYMETAMAPLITE_<NAME>` env vars.
+
+## REST API
+
+```powershell
+pymetamaplite serve --indexdir ivf --port 8000        # OpenAPI docs at /docs
+```
+
+| Endpoint | Purpose |
+|---|---|
+| `POST /annotate` | one document → structured JSON entities |
+| `POST /annotate/batch` | several documents in one request |
+| `POST /annotate/text?format=mmi` | MetaMapLite's own output formats as `text/plain` |
+| `POST /annotate/formatted` | a blob in any `--inputformat` (ChemDNER, PubMed XML, BioC, …) → one result per document |
+| `GET /lookup?term=…` | dictionary lookup for a single term |
+| `GET /formats` | every accepted input/output format, segmentation method and negation detector |
+| `GET /semantic-types` | the UMLS semantic types, for building `restrict_to_sts` |
+| `GET /health` | status, index metadata, uptime |
+
+Any annotate request may pass `"detector": "context"` to use ConText for that request instead of
+NegEx, which adds `temporality` and `experiencer` to each entity.
+
+```bash
+curl -X POST http://127.0.0.1:8000/annotate -H "Content-Type: application/json" \
+  -d '{"text":"No chest pain.","restrict_to_sts":["sosy"]}'
+```
+
+The index and spaCy model load once at start-up (~10 s; `--preload` adds a 4 s term-map load and
+~1.6 GB of RAM, and measured no faster, so leave it off). Requests take ~10–20 ms for a short clinical note. Annotation is
+CPU-bound and the underlying objects are not thread-safe, so requests are serialised behind a
+lock — **scale out with processes** (`uvicorn pymetamaplite.server:app --workers N`, each holding
+its own copy of the index), not threads.
+
+## Performance
+
+UMLS 2026AA, all English sources, on a laptop: index build 2.5 min; annotation ~500 sentences/s
+(the absolute figure varies a lot with machine state);
+resident memory ~120 MB, or ~1.8 GB with `preload_index=True` (a 419 MB pickled term map, 4 s to
+load). Preloading measured no faster than the default SQLite lookups with the index on local disk,
+so it is off by default and rarely worth turning on.
+
+## What is and isn't implemented
+
+Implemented: the whole `EntityLookup4` path — index building, normalization, segmentation,
+tokenization, POS gating, longest-match candidate generation, semantic-type/source restriction,
+subsumption removal, NegEx negation, Schwartz–Hearst abbreviations, user-defined acronyms,
+custom concept lists, excluded terms, both negation detectors (NegEx by default, ConText via
+`--usecontext`, which adds temporality and experiencer), every `--inputformat` MetaMapLite registers (`freetext`,
+`sli`/`sldi`, `sldiwi`, `chemdner`, `chemdnersldi`, `ncbicorpus`, `pubtator`, `pubmed`, `medline`,
+`bioc` — `pubtator` as a correction, since Java's own loader throws on every input), and the
+MMI / JSON / BRAT / CuiList output formats.
+
+Not implemented, deliberately:
+
+- **MetaMap-style scoring** (`EntityLookup5`, Java's `--enable_scoring`) — off by default in Java,
+  and measured rather than assumed: with NLM's own index the `vars` table is a single placeholder
+  row, so the variation term is identically zero and the coverage and cohesiveness terms reduce to
+  1.0 by construction. The score takes one of two values — 666.67, or 833.33 in the rare case that
+  a defective head test fires (749 / 4 over the parity corpus). `EntityLookup5`'s substantive
+  difference from `EntityLookup4` is its candidate generation, not its scores. See
+  the development plan, §10.1, for the closed form and how to reproduce it.
+- **Word sense disambiguation, derivational variants, disjoint entities** — absent from
+  MetaMapLite itself.
+
+## Development
+
+```powershell
+.venv\Scripts\python -m pytest -q        # 706 tests; the parity tests skip without Java fixtures
+.venv\Scripts\ruff check src tests scripts examples
+.venv\Scripts\ruff format --check src tests scripts examples
+.venv\Scripts\mypy                       # strict, over src/pymetamaplite (see pyproject.toml)
+python scripts/compare_with_java.py --java-dir <public_mm_lite>      # parity vs the real thing
+```
+
+`ruff format` is the formatter; lint rules and the 100-character line length are configured in
+`pyproject.toml`. [CI](.github/workflows/ci.yml) runs the same lint, `mypy --strict` and the tests
+on Python 3.11–3.13 on every push; the parity tests skip there, since their fixtures are
+licensed. Anything that changes annotation output must keep the parity harness green — see
+[tests/parity/README.md](tests/parity/README.md).
+
+## Releasing
+
+[`release.yml`](.github/workflows/release.yml) publishes via PyPI Trusted Publishing, so no API
+token is stored anywhere — PyPI verifies that this workflow, in this repository, produced the
+upload. Running the workflow by hand publishes to **TestPyPI**; pushing a `v*` tag publishes to
+**PyPI**. Both build, run `twine check`, and install the wheel into a clean environment before
+uploading, and a tagged run additionally refuses to publish if the tag disagrees with
+`__version__`. A version number, once on PyPI, can never be reused — so bump
+`src/pymetamaplite/__init__.py` and tag it, rather than re-cutting one.
+
+The project's development plan, an internal working document, records what each Java class does
+and why each behaviour was reproduced — including deliberate quirks (a span must start and end with a
+character Java's `CharUtils` calls alphanumeric, which is ASCII *and Greek*, so `β-blocker`
+matches but `-blocker` cannot; sources and semantic types print in Java `HashSet` order).
+
+## Licensing
+
+This package is released under the **same licence as MetaMapLite** — NLM's open-source BSD
+licence — reproduced in [LICENSE](LICENSE), with the upstream notice retained verbatim in
+[NOTICE-MetaMapLite.md](NOTICE-MetaMapLite.md). That licence requires you to keep the
+Informational Notice in redistributions and to acknowledge NLM as the source; hence the
+"Courtesy of the U.S. National Library of Medicine" line at the top of this README.
+
+pymetamaplite is an independent port and is **not** developed, funded or endorsed by NLM, NIH or
+the U.S. Government. The UMLS data it indexes is licensed separately: you need a UMLS
+Metathesaurus License and a UTS account, and the index you build from it is yours alone to keep —
+don't redistribute it.