oddb2xml 3.0.35 → 3.0.36

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 99321f1703f8ab1645b7cdb4049433c355330f95db7e45d8510c31cd15cf5bc7
-  data.tar.gz: 1c5c50119b700ac8591dc087b452cdcf06be4a35382e1b5e2280a7913cd6f5d6
+  metadata.gz: a85937ce6661b7094f1ca21340b46073ef281dcaf13aa5fe9f319412a0dd2bb1
+  data.tar.gz: 9cf9304ad319e03fc8b9d5077a6a90805aa253aff7297740480b09324a9551c7
 SHA512:
-  metadata.gz: 98e7a7a8445bdad56f3d4d722c2243a068167d8242e874d37c4d0e4ad10c2ac14588a84c031e1c279948898602d416ba91be6e59c532e9637de80818a3bc4b06
-  data.tar.gz: 993e517ee499b12d448efb16a36f3dd72b4b1d758c7685a33b7cab6ae3b4f72bb0fd1f1ac9eb0aa2f58b7a3493432118e1221734a7b00b2976a3cc6be9ba2d1f
+  metadata.gz: de43402f34d886b009654fc0cd614e999f11c7323757441f5c99442236c727689fd1d0d3c1345f6895fb1eb33bd57e9acb6b587235af1547be7f9bcfb4196ee6
+  data.tar.gz: f9e124c449ca9bb9ec23ce90697bf70e05d48681c80fba8408752e746f62d0428b0107356c9bd0e2275128e707abcde3d7666ae7c159b02db85b675f759e018d

data/CLAUDE.md

@@ -24,7 +24,7 @@ The system follows a **download → extract → build → compress** pipeline:
 
 7. **FHIR support** (`lib/oddb2xml/fhir_support.rb`) — Self-contained module providing `FhirDownloader` and FHIR NDJSON parsing. Activated via `--fhir` (or `--fhir-url=<URL>`). Downloads per-language NDJSON files (`foph-sl-publication-latest-{de,fr,it}.ndjson`) from `epl.bag.admin.ch/static/sl/publication/fhir` (**3.0.32 onwards**; BAG moved the export there on 24.08.2026, announced it the next evening, and oddb2xml followed in 3.0.32; the old `/static/fhir/foph-sl-export-*` answers 404 for `-latest-` while the old dated snapshots remain, so the failure looks like an import that does nothing) to populate French and Italian product names/descriptions. Maps legal status codes `756005022007` and `756005022008` to Swissmedic category D. Reads the BAG **Indikationscode** (`XXXXX.NN`) from the explicit `indicationCode` extension on each `RegulatedAuthorization.indication[].extension[regulatedAuthorization-limitation]` (BAG SL FHIR export >= v2.0.5; handled from 3.0.10). The BAG changelog states the limitation code (`ClinicalUseDefinition.id`) and the indication code are **independent** fields, so the older derivation — combining each indication CUD's `.NN` id-suffix with the reimbursement RA's `FOPHDossierNumber` — is kept only as a fallback for feeds lacking the extension. Exposed as `item[:indication_codes]` and per-package `:indication_codes` (each entry a `{code:, cud_id:, text:}` hash, where `cud_id` is the `limitationIndication` CUD reference used to resolve the text). From 3.0.7 onwards, `Builder#build_product` emits one `<INDICATION_CODE code="XXXXX.NN" cud_id="DRUG.NN">limitation text</INDICATION_CODE>` child per indication on every `<PRD>` in `oddb_product.xml`; live feed numbers: 539 products / 1,293 codes / 100 % with non-empty indication text. Mandatory on prescriptions/invoices for SL price-model drugs from 2026-07-01 — see issue [#113](https://github.com/zdavatz/oddb2xml/issues/113). **Limitation texts** (3.0.8 onwards): the `regulatedAuthorization-limitation` extension has no inline `limitationText` in the live BAG feed — it carries a `limitationIndication` reference to a `ClinicalUseDefinition` whose `indication.diseaseSymptomProcedure.concept.text` is the actual text. The parser stores the ref as `cud_ref` on each Limitation, `Bundle#cud_text_by_id` resolves DE, and `merge_language` propagates FR/IT from the per-language NDJSON files via the same CUD id. Coverage on the live feed jumped from 0 / 9'108 to 9'108 / 9'108 (issue [#116](https://github.com/zdavatz/oddb2xml/issues/116)). **Limitation code / LIMNAMEBAG** (3.0.12 onwards): FHIR has no native BAG limitation code (LIMCD), so `create_limitations_for_package` sets `LimitationCode = cud_ref` (the `limitationIndication` CUD id) instead of `""`. Without this, every FHIR limitation shared an empty `:code`; `Builder#build_artikelstamm` groups its `<LIMITATIONS>` section by code, so all of them collapsed into a single `<LIMITATION>` with an empty `<LIMNAMEBAG>` and only one text survived. Using the CUD id as the key makes each distinct limitation emit and be referenced from its `<PRODUCT>`. The downstream `bin/check_artikelstamm` (`semantic_check.rb`) also crashed on the lone-element output because Ox `:hash_no_attrs` collapses a one-child section into a Hash (and an empty one into nil) — `SemanticCheckXML#get_items` now normalises every section to an Array. **v6 Artikelstamm / per-article INDC (3.0.26 onwards):** `--artikelstamm` now emits the **Elexis Artikelstamm v6** format (namespace `http://elexis.ch/Elexis_Artikelstamm_v6`, file `artikelstamm_DDMMYYYY_v6.xml`/`.csv`, validated against the bundled `Elexis_Artikelstamm_v6.xsd`) — replacing v5. The new piece is a per-`<ITEM>` `<ARTSL>` block carrying the BAG Indikationscodes (issue [#113](https://github.com/zdavatz/oddb2xml/issues/113)): `<PM>true</PM>` plus one `<ARTLIM>` per limitation with `<LIMCD>` (= `cud_ref`, the BAG limitation code), `<INDCD>` (the `XXXXX.NN` indication code from the `indicationCode` extension), and `<VDAT>`/`<VTDAT>` (period start/end). To feed it, `create_limitations_for_package` now also carries `IndicationCode` (→ per-package `:indcd`) and `ValidThruDate` (→ `:vtdate`) on each limitation; `Builder#append_artsl`/`elexis_datetime` emit one `<ARTLIM>` per limitation that has a non-empty `:indcd` (so non-price-model items get no `<ARTSL>`). `PM` is always `true` here because the indication code is required only for SL price-model drugs, which is exactly the set of items that reach this block. The bundled `Elexis_Artikelstamm_v6.xsd` is the canonical MEDEVIT schema extended with oddb2xml's historical Italian elements (`DSCRI` on PRODUCT/LIMITATION/ITEM, `DOSAGE_FORMI` on ITEM) so the output still validates. The legacy `--no-fhir` path emits no `<ARTSL>` (no FHIR limitations). **Per-article INDC in the `-e`/`-b` feeds (3.0.27 onwards):** the same indication codes are also emitted per `<ART>` in `oddb_article.xml` via `Builder#append_indication_codes` — one `<INDICATION_CODE code="XXXXX.NN" cud_id="DRUG.NN" limcd="DRUG.NN" vdat=… vtdat=…>limitation text</INDICATION_CODE>` per limitation that carries an `:indcd` (sourced from the article's `pac[:limitations]`). This reuses the flat `<INDICATION_CODE>` element already emitted on `<PRD>` in `oddb_product.xml` (since 3.0.7), enriched with the limitation code and validity dates so it carries the full `<ARTSL>` payload. The bundled `oddb2xml.xsd` now defines `<INDICATION_CODE>` (simpleContent + `code`/`cud_id`/`limcd`/`vdat`/`vtdat` attributes) and references it from both `<ART>` and `<PRD>` — the `<PRD>` reference had been missing since the 3.0.7 addition, so the FHIR product feed only validated once this was added. **Legacy v5 opt-in (3.0.28 onwards):** `--artikelstamm-v5` additionally emits the older Artikelstamm v5 file alongside the v6 one (for consumers not yet migrated). It implies `--artikelstamm`; `Options.parse` sets `@opts[:artikelstamm] = true`. `build_artikelstamm` takes a `version:` keyword (default `6`) that drives the namespace (`Elexis_Artikelstamm_v#{version}`) and CSV/XML filename suffix; `build_artikelstamm_v5` calls it with `version: 5`. The only content difference is that the v5 path skips `append_artsl` (the v5 schema has no `<ARTSL>` element). The CLI `files` map adds an `:artikelstamm_v5` entry (→ `build_artikelstamm_v5` via the `to_xml` subject dispatch) and validates it against the bundled `Elexis_Artikelstamm_v5.xsd`. Both files come from one build — the `@prepared` guard means the shared download/prepare phase runs once. **Selbstbehalt / costShare (3.0.31 onwards):** the `reimbursementSL` extension's `costShare` valueInteger is the BAG Selbstbehalt percentage; it maps to the legacy `FlagSB` semantics of the old BAG `Preparations.xml` — `40` → `item[:deductible] = "Y"` (raised Selbstbehalt: `<SLOPLUS>1</SLOPLUS>` in `oddb_article.xml`, `<DEDUCTIBLE>40</DEDUCTIBLE>` in the Artikelstamm), `10` → `"N"` (ordinary Selbstbehalt: `SLOPLUS` 2 / `DEDUCTIBLE` 10), `20` → `deductible20 = "Y"` (transitional rate, issue #81). Until 3.0.31 the mapping was inverted (`10` → `"Y"`), so **every** SL article claimed a 40 % Selbstbehalt. On the live feed only 385 of 10'388 packages (222 preparations) carry `costShare` 40; the value never mixes within one preparation, so keeping the flag at sequence level is safe.
 
-8. **Refdata cleanup** (`lib/oddb2xml/refdata_cleanup.rb`) — Compensates for known data-quality issues in upstream Refdata.Articles.xml before they reach the output. Each fix is guarded by a Swissmedic-side heuristic (e.g. comma in `substance_swissmedic` to distinguish mono products from real combinations). Currently fixes (a) the doubled-dose template bug (`X mg / X mg / Stk`, `fix_double_dose`, guarded by `single_substance?`); (b) the spelled-out German galenic form `Retardtabletten` → house-style abbreviation `Ret Tabl` (`normalize_galenic_form` / `GALENIC_NORMALISATIONS`, issue #112 case #13, e.g. RINVOQ — a narrow word-boundary substitution that leaves legitimate brand suffixes like `TRAMAL retard` and Mepha's `Lactab` untouched); and (c) dose info Refdata dropped from `<FullName>`, sourced from the Swissmedic composition string `pack[:composition_swissmedic]` — `fix_missing_combo_dose` (#6, appends a combination's 2nd component strength), `fix_missing_dose` (#4, inserts a mono product's missing strength before the pack count), `fix_missing_volume` (#7, appends an injectable's per-pen volume); and (d) 50-char-truncation repairs — `fix_truncated_metoject` (#1, rebuilds METOJECT Autoinjektor names from the intact `<brand> Autoinjektor <dose>/<vol>` prefix + Swissmedic `size`, localised DE/FR/IT) and `fix_truncated_volume_unit` (#3, restores the cut `ml` of the VERACTIV Vitamin D3 drops). The (c) and (d) fixes are scoped to explicit IKSNR allow-lists (`COMBO_DOSE_IKSNR`/`MISSING_DOSE_IKSNR`/`MISSING_VOLUME_IKSNR`/`METOJECT_IKSNR`/`VERACTIV_VITD3_IKSNR`): a dry run proved a blanket heuristic mis-fires on hundreds of legitimate names (sodium counter-ion doses, strength-less phyto/powder products, concentration names like `CIMZIA 200 mg/ml`), so only catalogued registrations are touched — add an IKSNR to grow coverage. Called from `Builder#apply_refdata_description_cleanups!` at the start of `prepare_articles`. See GitHub issue #112 for the catalogue.
+8. **Refdata cleanup** (`lib/oddb2xml/refdata_cleanup.rb`) — Compensates for known data-quality issues in upstream Refdata.Articles.xml before they reach the output. Each fix is guarded by a Swissmedic-side heuristic (e.g. comma in `substance_swissmedic` to distinguish mono products from real combinations). Currently fixes (a) the doubled-dose template bug (`X mg / X mg / Stk`, `fix_double_dose`, guarded by `single_substance?`); (b) the spelled-out German galenic form `Retardtabletten` → house-style abbreviation `Ret Tabl` (`normalize_galenic_form` / `GALENIC_NORMALISATIONS`, issue #112 case #13, e.g. RINVOQ — a narrow word-boundary substitution that leaves legitimate brand suffixes like `TRAMAL retard` and Mepha's `Lactab` untouched); and (c) dose info Refdata dropped from `<FullName>`, sourced from the Swissmedic composition string `pack[:composition_swissmedic]` — `fix_missing_combo_dose` (#6, appends a combination's 2nd component strength), `fix_missing_dose` (#4, inserts a mono product's missing strength before the pack count), `fix_missing_volume` (#7, appends an injectable's per-pen volume); and (d) 50-char-truncation repairs — `fix_truncated_metoject` (#1, rebuilds METOJECT Autoinjektor names from the intact `<brand> Autoinjektor <dose>/<vol>` prefix + Swissmedic `size`, localised DE/FR/IT) and `fix_truncated_volume_unit` (#3, restores the cut `ml` of the VERACTIV Vitamin D3 drops). The (c) and (d) fixes are scoped to explicit IKSNR allow-lists (`COMBO_DOSE_IKSNR`/`MISSING_DOSE_IKSNR`/`MISSING_VOLUME_IKSNR`/`METOJECT_IKSNR`/`VERACTIV_VITD3_IKSNR`): a dry run proved a blanket heuristic mis-fires on hundreds of legitimate names (sodium counter-ion doses, strength-less phyto/powder products, concentration names like `CIMZIA 200 mg/ml`), so only catalogued registrations are touched — add an IKSNR to grow coverage. (e) **German-only wrong pack volume** (`fix_german_volume_from_fr_it`, 28.09.2026, reported by zollsoft): Refdata's DE `<FullName>` for PRIVIGEN 400 ml (7680583140053) reads `… 200 ml Dfl` while FR/IT say 400 ml, and FERINJECT/VIYANA/FERYXA 1000 mg say `10 ml` in DE against `20 ml` in FR/IT. This one is deliberately **not** IKSNR-scoped: it is a cross-language check inside Refdata's own record (FR and IT must agree on exactly one standalone `<n> ml`, DE must carry exactly one and differ; concentrations like `mg/ml` or `1000 mg/20ml` are excluded by the look-behind), and a dry run over all 16'967 articles of the live file changed exactly those four, each confirmed by ZurRose `transfer.dat` and, for PRIVIGEN, the BAG FHIR name. Runs per item after the per-language pipeline, logged as `German pack volume disagreeing with FR/IT`. Called from `Builder#apply_refdata_description_cleanups!` at the start of `prepare_articles`. See GitHub issue #112 for the catalogue.
 
 9. **Chapter-70 hack** (`lib/oddb2xml/chapter_70_hack.rb`) — Legacy scraper for the SL "Komplementärarzneimittel" products (homeopathic/anthroposophic/phytotherapeutic), called only from `Builder#build_artikelstamm`. **Deprecated / non-FHIR only (3.0.11 onwards):** the source page `varia_De.htm` was rebuilt as a JavaScript SPA with no static data table, so the scraper now returns nothing there. These products + limitations now come through the FHIR feed (SL classification `20. KOMPLEMENTÄRARZNEIMITTEL`, 221 products on the live DE feed with real GTINs and limitation texts), so `build_artikelstamm` **skips the scraper entirely when `@options[:fhir]`** (the default for `--artikelstamm` since 3.0.9). In `--no-fhir` mode the scraper degrades gracefully (skips non-row/`<script>` nodes and empty tables, warns, returns `[]`) instead of raising `NoMethodError`. See GitHub issue #118.
 
@@ -50,7 +50,7 @@ These scripts run the public download server at `https://mediupdatexml.oddb.org`
 - **`generate_index_html.sh DOCROOT [FIRSTBASE_CSV]`** — single source of truth for the landing page. Writes `index.html` + a self-contained `logo.svg` **atomically** (temp + `mv`, so either owner — root from setup, `zdavatz` from cron — can refresh it). Computes live counts: PHARMA = `<SMNO>` count in `default/oddb_article.xml`, NONPHARMA = firstbase CSV rows − 1, total ART = `<ART ` count. Also runs **`visitor_stats.py`** and embeds its graph. Re-run standalone any time (it only reads already-built files); a separate cron line refreshes it **hourly** (`5 * * * * zdavatz`) so counts + graph stay current between nightly builds.
 - **`visitor_stats.py LOG_GLOB CACHE_DIR [DAYS]`** — emits the visitors/sessions/region graph as an inline-SVG HTML **fragment** (last `DAYS`, default 14): Besucher = distinct IPs/day, Sitzungen = 30-min-inactivity sessions per `(IP, User-Agent)`, plus a top-6 country breakdown by IP. Bots are filtered by User-Agent. Region lookup is **fully self-contained** — pure Python stdlib + the free **DB-IP country-lite CSV** (CC-BY, no licence key) cached in the build `downloads/` dir and refreshed monthly; **no apt package, no gem, no system GeoIP DB**. Prints nothing (page degrades to omitting the section) when the Apache log is unreadable or empty. Reading `/var/log/apache2` requires the cron user to be in the **`adm`** group (`sudo usermod -aG adm zdavatz`).
 - **`swissmedic_watch.sh`** — outage/block auto-recovery (cron: `*/30 * * * * zdavatz`). Since the Swissmedic platform migration (~2026-06-23, now a Swisscom-operated gateway), `www.swissmedic.ch` intermittently resets this host's automated connections **after the TLS handshake** (TCP RST), which aborts `run_oddb2xml.sh` under `set -e` and leaves the feeds stale (the block is host/IP- and client-fingerprint-sensitive: a real browser works, `curl`/`wget`/Ruby get reset, while other admin.ch hosts answer fine — so it is a WAF/bot rule, not an outage). The watcher polls Swissmedic with **oddb2xml's own client** (a Ruby `open-uri` canary on `listen_neu.html`); while blocked it is a silent no-op, and the moment it gets HTTP 200 it launches **one** build and emails. It fires **at most once per day** (stamp in `$STATE_DIR`, default `<OUT_DIR>-watch`, kept **outside** the wiped `$BUILD_DIR`), and skips when a build is already running or today's `default/oddb_article.xml` is already fresh. **PATH assembly (2026-07-28 onwards):** cron hands the watcher a minimal PATH, so it builds one explicitly — `/usr/local/bin` (where a root `gem install` under Debian's system Ruby puts the binstub) + the user gem bin dir (`$(ruby -e 'print Gem.user_dir')/bin`, where an *unprivileged* `gem install` silently lands instead, because `/var/lib/gems` is not writable) + rbenv shims **only when they exist** (the pre-2026-07 box; `RBENV_VERSION=3.4.5` is exported only in that case). The previous hardcoded rbenv-only PATH carried no `/usr/local/bin`, so the first watcher-triggered build on the rebuilt server died `oddb2xml: Kommando nicht gefunden` (exit 127) on all three retries before aborting. `run_oddb2xml.sh` step 1 extends PATH the same way right after its `gem install`, for the same reason.
-- **`transfer.sh`** — optional hand-off (scp) of `$OUT_DIR` to the HIN host; `SCP_DEST` is required-but-unset until the HIN host is known. **Obsolete, not merely unwired (issue [#127](https://github.com/zdavatz/oddb2xml/issues/127)):** it is not in `/etc/cron.d/mediupdatexml` and `SCP_DEST` still has no default — and it should stay that way, because **HIN no longer hosts a copy at all**. Their page `www.hin.ch/de/services/mediupdate-xml.cfm#section_2` links *this* site's URLs directly (`default/`, `45/`, `50/`, `55/`, `aips2sqlite/`) and mentions `download.hin.ch` exactly **zero** times, so there is nothing left to scp. The trap is that the unfed legacy path `https://download.hin.ch/download/oddb2xml/` **still answers HTTP 200**, serving the build of **22.06.2026** (`GENERATED_BY="oddb2xml 3.0.25"`, `Last-Modified: Sun, 21 Jun 2026 23:01:34 GMT`) — the whole directory carries that one date. It looks authoritative, it 404s nothing, and that is how issue #127 was filed against us for data that was current here all along. When someone reports stale data, **check which host they fetched from before touching the build**: `mediupdatexml.oddb.org` is the source of truth. HIN has been asked to 301 or retire the old path. `README.md` and `QA.md` now point at `mediupdatexml.oddb.org` throughout. For twenty days one link on HIN's page was genuinely broken and it was **ours**: `/aips2sqlite/oddb2xml_swissmedic_sequences.csv` answered 404 — **not** because the server rebuild lost it, but because aips2sqlite hit **the same Refdata barcode bug as issue #122**: `ean_code.length()` on the missing `<DataCarrierIdentifier>` threw a `NullPointerException` out of four copied loops, and `generate_aips_fi` deleted the published CSV *before* regenerating it, so under `set -e` each crash left nothing rather than yesterday's file. Fixed 02.09.2026 in aips2sqlite `ad2fc58` (all four loops skip barcode-less articles; the CSV is now published by `mv` only after a run that passed a row-count check). **When Refdata breaks one of the two projects, check the other** — they parse the same file with the same assumption.
+- **`transfer.sh`** — optional hand-off (scp) of `$OUT_DIR` to the HIN host; `SCP_DEST` is required-but-unset until the HIN host is known. **Obsolete, not merely unwired (issue [#127](https://github.com/zdavatz/oddb2xml/issues/127)):** it is not in `/etc/cron.d/mediupdatexml` and `SCP_DEST` still has no default — and it should stay that way, because **HIN no longer hosts a copy at all**. Their page `www.hin.ch/de/services/mediupdate-xml.cfm#section_2` links *this* site's URLs directly (`default/`, `45/`, `50/`, `55/`, `aips2sqlite/`) and mentions `download.hin.ch` exactly **zero** times, so there is nothing left to scp. The trap is that the unfed legacy path `https://download.hin.ch/download/oddb2xml/` **still answers HTTP 200**, serving the build of **22.06.2026** (`GENERATED_BY="oddb2xml 3.0.25"`, `Last-Modified: Sun, 21 Jun 2026 23:01:34 GMT`) — the whole directory carries that one date. It looks authoritative, it 404s nothing, and that is how issue #127 was filed against us for data that was current here all along. When someone reports stale data, **check which host they fetched from before touching the build**: `mediupdatexml.oddb.org` is the source of truth. HIN has been asked to 301 or retire the old path. `README.md` and `QA.md` now point at `mediupdatexml.oddb.org` throughout. For twenty days one link on HIN's page was genuinely broken and it was **ours**: `/aips2sqlite/oddb2xml_swissmedic_sequences.csv` answered 404 — **not** because the server rebuild lost it, but because aips2sqlite hit **the same Refdata barcode bug as issue #122**: `ean_code.length()` on the missing `<DataCarrierIdentifier>` threw a `NullPointerException` out of four copied loops, and `generate_aips_fi` deleted the published CSV *before* regenerating it, so under `set -e` each crash left nothing rather than yesterday's file. Fixed 02.09.2026 in aips2sqlite `ad2fc58` (all four loops skip barcode-less articles; the CSV is now published by `mv` only after a run that passed a row-count check). **When Refdata breaks one of the two projects, check the other** — they parse the same file with the same assumption. Second aips2sqlite outage of the same shape, 16.–22.09.2026: swissdrg.org retired the file id behind the DE *Fallpauschalenkatalog* download and served an HTML page *"Ungültige Datei"* with **HTTP 200** for it; `FileUtils.copyURLToFile` wrote the page over `swiss_drg_de_xlsx.xlsx`, `XSSFWorkbook` threw `POIXMLException: Package should contain a content type part` (a RuntimeException past the `IOException`-only catch), and the `--lang=de --xml` step died nightly for a week — `amiko_db_full_idx_de.db` in `jars/output/` shrank to 16 kB while the FR DB and the sequences CSV (built by the later `--nodown` steps, which never touch that file) stayed at 15.09. A downstream user (Swiss Medical Record) reported the identical trace from their own jar. Fixed 22.09.2026 in aips2sqlite `8cf81a6`: new file id `1444/1029` (same 6.0/2017 catalogue), zip-magic check with last-good keep on download, and `extractSwissDRGInfo` skips with a `SKIPPED` warning instead of aborting — the Swiss DRG Zusatzentgelte are optional enrichment, **a missing side file must never abort the run**. Nothing on the host tells you when the 04:30 job dies (no MTA), so the `jars/output/` timestamps are the monitor.
 - **`setup_new_server.sh`** (2026-07-28 onwards) — one-command rebuild of a bare Debian host into this download site, written after **HIN deleted the previous server**: everything that had accumulated by hand on that box, so a rebuild is not a day of archaeology. Root, idempotent. Installs the packages (`apache2` + `certbot`; **`ruby-full`** — Debian 13 ships Ruby 3.3, matching the repo `.ruby-version`, so the new host has **no rbenv**; `cron`, absent on a minimal image; `unzip`/`zip` for the ZurRose transfer and the build zip; `poppler-utils` for `pdftotext` in `tools/generate_bag_sl_group_prices.rb`; nokogiri/ffi build headers), adds `$RUN_USER` to **`adm`** (so `visitor_stats.py` may read `/var/log/apache2`), sets the home dir to **711** (so `www-data` can traverse to `$OUT_DIR` without listing it), `gem install oddb2xml`, creates the five directories (`$OUT_DIR`, `-build`, `-state`, `-watch` — the last two deliberately outside the nightly-wiped `-build` — and `$GET_TRANSFER_DIR`, symlinking the tracked `get_transfer.sh` into it), writes **`/etc/cron.d/mediupdatexml`** (the whole schedule in one re-creatable file rather than the old box's per-user crontab: `get_transfer` 00:30, nightly build 01:00, rust2xml Artikelstamm 03:00, aips2sqlite Fachinfos 04:30, landing page hourly at `:05`, Swissmedic watcher every 30 min) plus a logrotate rule for the cron logs, then hands over to `setup_aips2sqlite.sh` and `setup_mediupdatexml_web.sh`. Env: `RUN_USER`, `OUT_DIR`, `SKIP_WEB=1`, `SKIP_GEM=1`, `SKIP_AIPS=1`. **Deliberately out of scope** (gone with the old server, restored separately): an MTA for the cron `MAILTO`.
 - **`setup_mediupdatexml_web.sh`** — one-time root setup of the Apache vhost + initial page. Skips the Let's Encrypt step with a hint when the domain does not yet resolve in **public** DNS (checked via Cloudflare DoH, not the local resolver, which may hold a stale or negative cache).
 - **`setup_aips2sqlite.sh`** (2026-08-01 onwards) — provisions the **aips2sqlite** half of the site, restored after the server rebuild. The landing page links a whole `/aips2sqlite/` section (Fachinformationen as XML, the AmiKo `.db` files, `oddb2xml_swissmedic_sequences.csv`, `atc_codes_used_set.txt`); `setup_mediupdatexml_web.sh` writes the Apache alias `/aips2sqlite` → `<checkout>/jars/output`, but on the rebuilt host neither the JRE nor that directory existed, so **every link answered 403** — Apache falls back to the default `<Directory /> Require all denied` when the aliased path does not exist, so a *missing* target reads as *forbidden*, not *not found*. Root, idempotent: installs **`openjdk-21-jre-headless`** (the repo ships a prebuilt fat jar `jars/aips2sqlite.jar`, so a JRE suffices — no JDK/Gradle unless the jar is rebuilt; the only AWT user, `BarCode.java`/barcode4j, is off the Fachinfo path), clones the checkout if missing, creates `jars/output` + `jars/downloads` owned by `$RUN_USER`, and appends the 04:30 cron entry to `/etc/cron.d/mediupdatexml` if not already there (`SKIP_CRON=1` when called from `setup_new_server.sh`, which writes that line itself). Generating the data is a separate build-user step: `aips2sqlite/scripts/generate_aips_fi` (~1 h; `--lang=de --xml`, then `--lang=fr --xml --nodown`, then `--nodown --smsequence`, renaming `swiss_medic_sequences.csv` → `oddb2xml_swissmedic_sequences.csv`). `REFDATA_API_KEY` (developer.refdata.ch) is **optional** for this path: the Refdata Partner SOAP download it gates is consumed only by the Takeda partner export, and its failure is caught and logged, so the FI/sequences output is complete without it.
@@ -65,4 +65,4 @@ Only the scripts are git-tracked; the generated `index.html`/`logo.svg` and the
 - `spec/spec_helper.rb` defines test constants (GTINs) and configures VCR to avoid real HTTP calls during tests
 - CI runs on Ruby 3.2 and 3.3 (`.github/workflows/ruby.yml`); 3.0/3.1 cannot resolve the lockfile since `public_suffix` 7 (Feb 2026) and are EOL
 - **Never call `Kernel#exit` inside `lib/`.** Until 02.09.2026 `Builder#prepare_substances` did (`exit 2` when an `-e`/`--artikelstamm` build had no substances). Under rspec that killed the *process* mid-run: the summary still printed, reading e.g. `1193 examples, 0 failures`, but the count was a random fraction of the ~1640 examples and the exit status was 2 — CI had been red on every run for over a year while local runs looked green to anyone reading the summary line instead of `$?`. The trigger was the June-2026 FHIR default: `-e` implies `--fhir`, `epl.bag.admin.ch` was never stubbed, `FhirExtractor added 0 items`, empty substances, exit. Builder now raises `Oddb2xml::NoSubstancesError` (`util.rb`) and only `bin/oddb2xml` turns that into exit status 2, so production keeps its contract and rspec sees an ordinary failure. `mock_downloads` stubs the three NDJSON URLs with `spec/data/fhir/cyramza.ndjson` (one bundle — enough to stay non-empty, not enough for price assertions), and the four CLI specs that assert on SL prices run with `--no-fhir` against the stubbed `Preparations.xml`, which is what they were written for. **When judging a run, read the exit status and the example count, not the failure count.**
-- **Dependabot security jobs (fixed 02.09.2026, `b9e1f09`):** the "bundler in /. for savon/json" runs had failed for months. Two lessons. (a) Dependabot resolves the lockfile against the **lowest** Ruby the gemspec allows — with `required_ruby_version >= 2.5.0` it could not pick json ≥ 2.8 (needs 2.7), so a stale floor silently blocks every security update; the floor is `>= 3.2.0` now. (b) A gem that is only `require`d but never called still carries its advisories: `savon` (two HIGH, GHSA Savon::Model evaluates WSDL operation names) had been dead since the SwissReg migration `4c65a9c` (March 2026), pinned `~> 2.12.0` so the fix 2.17.2 was unreachable. Removing it also removed akami/gyoku/nori/wasabi/httpi/socksify — and exposed that `HTTPI.log = false` in `Downloader#initialize` was loaded only through savon (305 NameErrors), so that line and `httpi` went too. Before bumping a pinned gem, `grep -rn "Savon::"`-style for actual use first. **The published 3.0.34 gem still declares `savon` and `httpi`**, so every `gem install oddb2xml` — including the nightly's step 1 — pulls the vulnerable savon 2.12.1 until 3.0.35 is released. `rubyntlm`, `multi_json` and the two `net/ntlm` requires in `downloader.rb` are the same kind of SOAP-era leftover (zero references), not yet removed.
+- **Dependabot security jobs (fixed 02.09.2026, `b9e1f09`):** the "bundler in /. for savon/json" runs had failed for months. Two lessons. (a) Dependabot resolves the lockfile against the **lowest** Ruby the gemspec allows — with `required_ruby_version >= 2.5.0` it could not pick json ≥ 2.8 (needs 2.7), so a stale floor silently blocks every security update; the floor is `>= 3.2.0` now. (b) A gem that is only `require`d but never called still carries its advisories: `savon` (two HIGH, GHSA Savon::Model evaluates WSDL operation names) had been dead since the SwissReg migration `4c65a9c` (March 2026), pinned `~> 2.12.0` so the fix 2.17.2 was unreachable. Removing it also removed akami/gyoku/nori/wasabi/httpi/socksify — and exposed that `HTTPI.log = false` in `Downloader#initialize` was loaded only through savon (305 NameErrors), so that line and `httpi` went too. Before bumping a pinned gem, `grep -rn "Savon::"`-style for actual use first. **The published 3.0.34 gem still declares `savon` and `httpi`**, so every `gem install oddb2xml` — including the nightly's step 1 — pulls the vulnerable savon 2.12.1 until 3.0.35 is released. The same sweep later removed `multi_json` and the direct `rubyntlm` pin plus the two `net/ntlm` requires in `downloader.rb` (zero references, SOAP-era leftovers); `rubyntlm` itself stays in the lockfile as a transitive dependency of `mechanize`, which carries the same `>= 0.6.3` CVE floor.

data/Gemfile.lock

@@ -6,7 +6,6 @@ PATH
       htmlentities
       mechanize (>= 2.8.5)
       minitar
-      multi_json
       nokogiri (>= 1.18.10)
       optimist
       ox
@@ -14,7 +13,6 @@ PATH
       rack (>= 3.1.20)
       rexml (>= 3.3.9)
       rubyXL (~> 3.4.0)
-      rubyntlm (>= 0.6.3)
       rubyzip (~> 3.0.1)
       sax-machine
       spreadsheet
@@ -70,7 +68,6 @@ GEM
     mime-types-data (3.2026.0203)
     mini_portile2 (2.8.9)
     minitar (1.1.0)
-    multi_json (1.19.1)
     mutex_m (0.3.0)
     net-http-digest_auth (1.4.1)
     net-http-persistent (4.0.8)

data/History.txt

@@ -1,3 +1,8 @@
+=== 3.0.36 / 28.09.2026
+* Bugfix (Refdata cleanup): correct the German pack volume when Refdata's German name disagrees with its own French and Italian names. Reported by zollsoft for PRIVIGEN 10% 400 ml (GTIN 7680583140053), whose German description in oddb_product.xml and oddb_article.xml read "200 ml" -- the German name of the 200 ml pack -- while French, Italian, ZurRose and the BAG FHIR feed all say 400 ml. The same Refdata defect affects FERINJECT, VIYANA and FERYXA 1000 mg (German "10 ml", correct 20 ml). The fix changes exactly these four articles on the September 2026 Refdata file; concentrations such as mg/ml are never touched. Suite: 1646 examples, 0 failures.
+* Builder creates WORK_DIR before writing missing_in_refdata.txt and the Artikelstamm CSV (order-dependent spec failure, seed 17155).
+* Dependencies: multi_json and the direct rubyntlm pin removed (SOAP-era leftovers, no references); rubyntlm stays as a dependency of mechanize.
+
 === 3.0.35 / 02.09.2026
 * Security: drop the savon dependency (two HIGH advisories, GHSA "Savon::Model evaluates WSDL operation names as Ruby source"). It had not been used since the SwissReg migration in 3.0.19 (commit 4c65a9c, March 2026) -- only a require remained -- but the gemspec pin ~> 2.12.0 kept every installer on the vulnerable 2.12.1 and blocked Dependabot's fix to 2.17.2. Removing it also drops akami, gyoku, nori, wasabi, httpi and socksify from the dependency tree; the one leftover reference, HTTPI.log = false in Downloader#initialize, had only ever been loaded through savon and is gone too. json is raised to 2.21.2 (LOW advisory: generator heap buffer overflow when streaming to an IO). 3.0.34 and earlier still declare savon and httpi, so `gem install oddb2xml` on those versions pulls savon 2.12.1 -- upgrade.
 * required_ruby_version is now >= 3.2 (was >= 2.5.0). Ruby 3.0 and 3.1 are end-of-life and have not been able to resolve the lockfile since public_suffix 7 (February 2026); the stale floor also made Dependabot resolve against Ruby 2.5, which is why its json update could never succeed. CI runs on 3.2 and 3.3.

data/README.md

@@ -457,6 +457,14 @@ Currently active fixes (`lib/oddb2xml/refdata_cleanup.rb`):
   duplicate token is collapsed to a single occurrence. Real combination
   products (e.g. PHESGO 600 mg / 600 mg / 10 ml — pertuzumab + trastuzumab)
   are detected via the comma in `substance_swissmedic` and left untouched.
+* **German pack volume disagreeing with French and Italian** — Refdata
+  sometimes carries a wrong volume in the German name only, e.g.
+  `PRIVIGEN 10% Inf Lös 200 ml Dfl` for the 400 ml pack (GTIN 7680583140053)
+  while the French and Italian names say 400 ml. When the French and Italian
+  names agree on a single standalone `<n> ml` and the German one differs, the
+  German number is replaced. Concentrations such as `mg/ml` are ignored. On
+  the September 2026 file this changes exactly four articles (PRIVIGEN 400 ml,
+  FERINJECT, VIYANA and FERYXA 1000 mg 20 ml), all confirmed by ZurRose.
 
 The cleanup runs at the start of `prepare_articles` in `Builder` and is
 idempotent. Each rule is guarded by a Swissmedic-side heuristic so genuine
@@ -570,6 +578,14 @@ they are the reference setup for running oddb2xml unattended.
   cron entry that regenerates the Fachinformationen. Run as root; idempotent.
   Without it every `/aips2sqlite/` link answers 403, because Apache denies a
   path that does not exist.
+  The job's output is only as robust as its side files: when one of the
+  `/aips2sqlite/` files goes stale or tiny, look at the timestamps and sizes in
+  `jars/output/` first (a `amiko_db_full_idx_de.db` of a few kB means the
+  `--lang=de` step died) and then at `generate_aips_fi.log` -- twice now an
+  upstream file changed shape (Refdata articles without barcode, 08/2026;
+  SwissDRG serving an HTML page for a retired xlsx id, 09/2026) and the Java
+  side crashed instead of skipping. Both are fixed in aips2sqlite; the rule
+  there is that a missing side file costs its data, never the run.
 * `transfer.sh` — optional scp hand-off of the output tree to the HIN
   download server. **Obsolete, and deliberately not wired up**: `SCP_DEST` has
   no default and the script is not in `/etc/cron.d/mediupdatexml`. HIN links

data/lib/oddb2xml/builder.rb

@@ -131,11 +131,17 @@ module Oddb2xml
           end
           item[key] = desc if desc != original
         end
+        fixed_de = RefdataCleanup.fix_german_volume_from_fr_it(item[:desc_de], item[:desc_fr], item[:desc_it])
+        if fixed_de != item[:desc_de]
+          item[:desc_de] = fixed_de
+          counts[:de_volume] += 1
+        end
       end
       labels = {metoject: "truncated METOJECT name", veractiv_vol: "truncated volume unit",
                 double_dose: "double-dose pattern", galenic: "galenic form",
                 combo_dose: "missing combo 2nd dose", missing_dose: "missing strength",
-                volume: "missing injection volume"}
+                volume: "missing injection volume",
+                de_volume: "German pack volume disagreeing with FR/IT"}
       counts.each do |rule, n|
         Oddb2xml.log("Refdata cleanup: fixed #{labels[rule]} in #{n} description(s)") if n > 0
       end
@@ -587,6 +593,12 @@ module Oddb2xml
         ean13_to_product[ean13] = obj
         obj[:pharmacode] ||= @refdata[ean13][:pharmacode] if @refdata[ean13]
       }
+      # The CLI creates WORK_DIR before building, but Builder is also driven
+      # directly (fhir_spec), and then this was the first write into it --
+      # Errno::ENOENT whenever the random spec order got here before any CLI
+      # example had created the directory. ZurroseExtractor guards its own
+      # WORK_DIR write the same way.
+      FileUtils.makedirs(WORK_DIR)
       ausgabe = File.open(File.join(WORK_DIR, "missing_in_refdata.txt"), "w+")
       size_old = ean13_to_product.size
       @missing = []
@@ -1656,6 +1668,7 @@ module Oddb2xml
     def build_artikelstamm(version: 6)
       @@emitted_v6_gtins = []
       @artikelstamm_version = version
+      FileUtils.makedirs(WORK_DIR) # same guard as add_missing_products_from_swissmedic
       @csv_file = CSV.open(File.join(WORK_DIR, "artikelstamm_#{Date.today.strftime("%d%m%Y")}_v#{version}.csv"), "w+")
       @csv_file << ["gtin", "name", "pkg_size", "galenic_form", "price_ex_factory", "price_public", "prodno", "atc_code", "active_substance", "original", "it-code", "sl-liste"]
       @csv_file.sync = true

data/lib/oddb2xml/downloader.rb

@@ -1,5 +1,3 @@
-require "net/ntlm/version" # needed to avoid error: uninitialized constant Net::NTLM::VERSION
-require "rubyntlm"
 require "mechanize"
 require "zip"
 require "open-uri"

data/lib/oddb2xml/refdata_cleanup.rb

@@ -194,5 +194,30 @@ module Oddb2xml
       return desc unless VERACTIV_VITD3_IKSNR.include?(iksnr_of(no8))
       desc.sub(/(\d)\s*m\z/, '\1ml')
     end
+
+    # Refdata sometimes carries a wrong pack volume in the German name only,
+    # while the French and Italian names of the same article agree with each
+    # other and with ZurRose/BAG. Seen September 2026 (reported by zollsoft):
+    #   7680583140053 DE "PRIVIGEN 10% Inf Lös 200 ml Dfl"   FR/IT "400 ml"
+    #   7680578510052 DE "FERINJECT … 1000 mg 10 ml …"       FR/IT "20 ml"
+    #   7680693600058 VIYANA, 7680694120050 FERYXA           same as FERINJECT
+    # A scan of all 16'971 Refdata articles found exactly these four, all
+    # confirmed wrong by ZurRose transfer.dat. The guard is strict: each name
+    # must carry exactly one standalone "<n> ml" (a concentration like
+    # "mg/ml" or "1000 mg/20ml" is not standalone), FR and IT must agree, and
+    # only the number in DE is replaced.
+    ML_TOKEN = %r{(?<![/\d.,])(\d+(?:[.,]\d+)?)(\s*ml)\b}i
+
+    def self.fix_german_volume_from_fr_it(de, fr, it)
+      return de if [de, fr, it].any? { |d| d.nil? || d.empty? }
+      de_ml = de.scan(ML_TOKEN)
+      fr_ml = fr.scan(ML_TOKEN)
+      it_ml = it.scan(ML_TOKEN)
+      return de unless de_ml.size == 1 && fr_ml.size == 1 && it_ml.size == 1
+      want = fr_ml.first.first.tr(",", ".")
+      return de unless it_ml.first.first.tr(",", ".") == want
+      return de if de_ml.first.first.tr(",", ".") == want
+      de.sub(ML_TOKEN) { "#{fr_ml.first.first}#{$2}" }
+    end
   end
 end

data/lib/oddb2xml/version.rb

@@ -1,3 +1,3 @@
 module Oddb2xml
-  VERSION = "3.0.35"
+  VERSION = "3.0.36"
 end

data/oddb2xml.gemspec

@@ -32,8 +32,6 @@ Gem::Specification.new do |spec|
   spec.add_dependency "rubyXL", "~> 3.4.0"
   spec.add_dependency "sax-machine" # ,  '~> 0.1.0'
   spec.add_dependency "parslet" # , '~> 1.7.0'
-  spec.add_dependency "rubyntlm", ">= 0.6.3"
-  spec.add_dependency "multi_json" # , '>= 0.3.2'
   spec.add_dependency "optimist"
   spec.add_dependency "xml-simple"
   spec.add_dependency "ox"

data/spec/refdata_cleanup_spec.rb

@@ -384,4 +384,47 @@ describe Oddb2xml::Builder do
       expect(builder.refdata["7680576900046"][:desc_de]).to eq "VERACTIV Vitamin D3 Wild Huile Trp 20'000 U.I. 10ml"
     end
   end
+
+  describe ".fix_german_volume_from_fr_it" do
+    let(:fix) { ->(de, fr, it) { Oddb2xml::RefdataCleanup.fix_german_volume_from_fr_it(de, fr, it) } }
+
+    it "takes the volume from FR/IT when DE disagrees (PRIVIGEN 400 ml, zollsoft 09/2026)" do
+      expect(fix.call("PRIVIGEN 10% Inf Lös 200 ml Dfl", "PRIVIGEN 10% sol perf 400 ml flac",
+        "PRIVIGEN 10% sol inf 400 ml flac")).to eq "PRIVIGEN 10% Inf Lös 400 ml Dfl"
+    end
+
+    it "fixes FERINJECT 1000 mg 10 ml -> 20 ml" do
+      expect(fix.call("FERINJECT Disp Inj/Inf 1000 mg 10 ml 1 Flasche", "FERINJECT disp inj/pf 1000 mg 20 ml 1 flacon",
+        "FERINJECT disp inj/inf 1000 mg 20 ml 1 flacone")).to eq "FERINJECT Disp Inj/Inf 1000 mg 20 ml 1 Flasche"
+    end
+
+    it "leaves DE alone when FR and IT disagree with each other" do
+      de = "X Inf Lös 200 ml Dfl"
+      expect(fix.call(de, "X sol perf 400 ml flac", "X sol inf 300 ml flac")).to eq de
+    end
+
+    it "ignores concentrations such as mg/ml or 1000 mg/20ml" do
+      de = "CIMZIA Inj Lös 200 mg/ml Fertspr 2 Stk"
+      expect(fix.call(de, "CIMZIA sol inj 200 mg/2ml ser pré 2 pce", "CIMZIA sol inj 200 mg/2ml sir pre 2 pz")).to eq de
+    end
+
+    it "does nothing with more than one volume in a name" do
+      de = "Y 5 ml + 10 ml"
+      expect(fix.call(de, "Y 5 ml + 20 ml", "Y 5 ml + 20 ml")).to eq de
+    end
+
+    it "does nothing when a language is missing" do
+      expect(fix.call("Z 10 ml", "", "Z 20 ml")).to eq "Z 10 ml"
+    end
+
+    it "is applied by Builder#apply_refdata_description_cleanups!" do
+      builder = Oddb2xml::Builder.new
+      builder.packs = {"58314053" => {}}
+      builder.refdata = {"7680583140053" => {ean13: "7680583140053", no8: "58314053",
+        desc_de: "PRIVIGEN 10% Inf Lös 200 ml Dfl", desc_fr: "PRIVIGEN 10% sol perf 400 ml flac",
+        desc_it: "PRIVIGEN 10% sol inf 400 ml flac"}}
+      builder.apply_refdata_description_cleanups!
+      expect(builder.refdata["7680583140053"][:desc_de]).to eq "PRIVIGEN 10% Inf Lös 400 ml Dfl"
+    end
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: oddb2xml
 version: !ruby/object:Gem::Version
-  version: 3.0.35
+  version: 3.0.36
 platform: ruby
 authors:
 - Yasuhiro Asaka, Zeno R.R. Davatz, Niklaus Giger
@@ -121,34 +121,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: rubyntlm
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 0.6.3
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 0.6.3
-- !ruby/object:Gem::Dependency
-  name: multi_json
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   name: optimist
   requirement: !ruby/object:Gem::Requirement