oddb2xml 3.0.27 → 3.0.29

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4aa1328e4844d47543cea04ab6ab5832900bd1bdc1611b1a96d969a71632ff18
-  data.tar.gz: e592e5253f0ff09a18e7a515b4b4689484ddc61ba2969a5221bdd3864ebafa74
+  metadata.gz: 4602599ed50439225ff6ce76000f8441a9b4963b02538adf3f46d6728beff6cc
+  data.tar.gz: 55f34b946cc7a0e75435459aa90169e23719ac44c4bbfdf1ba3d63c3162f894d
 SHA512:
-  metadata.gz: 30aa032317390a02cd39b1b281897618bd59c51fffb7f793a30dd9caf97798866b8b2deb585e0e877fe0704a2b329c1b3526aaf3ef9e323adfa54e8dfdc05945
-  data.tar.gz: 1aeb220381cfd66043a5ca4e695a5c24a349642ca28c44794908c73ec164d5c3b18c3fce92acae6227694c5e68b00fb58e5c292b08aaf1c68f4d81af6697b7f0
+  metadata.gz: cf98f9144a53cd62be8599168423c4c6d560b8474dc015d5281eccb84b2f80de6d46e3765b7ab96cc6271822361ea0a4a7b1f00bae1908e16f94f8fb77006908
+  data.tar.gz: 7ed225d5f4cd6f3c3293913cb40e01430839968d34e15551a2192a1f9069343dd98f7448179a0e433bc6485d8810510904add62632e6e5ddf6d03d50b8f41dc8

data/CLAUDE.md

@@ -47,7 +47,7 @@ The system follows a **download → extract → build → compress** pipeline:
 
 6. **Compressor** (`lib/oddb2xml/compressor.rb`) — Optional ZIP/TAR.GZ output compression.
 
-7. **FHIR support** (`lib/oddb2xml/fhir_support.rb`) — Self-contained module providing `FhirDownloader` and FHIR NDJSON parsing. Activated via `--fhir` (or `--fhir-url=<URL>`). Downloads per-language NDJSON files (`foph-sl-export-latest-{de,fr,it}.ndjson`) from `epl.bag.admin.ch` to populate French and Italian product names/descriptions. Maps legal status codes `756005022007` and `756005022008` to Swissmedic category D. Reads the BAG **Indikationscode** (`XXXXX.NN`) from the explicit `indicationCode` extension on each `RegulatedAuthorization.indication[].extension[regulatedAuthorization-limitation]` (BAG SL FHIR export >= v2.0.5; handled from 3.0.10). The BAG changelog states the limitation code (`ClinicalUseDefinition.id`) and the indication code are **independent** fields, so the older derivation — combining each indication CUD's `.NN` id-suffix with the reimbursement RA's `FOPHDossierNumber` — is kept only as a fallback for feeds lacking the extension. Exposed as `item[:indication_codes]` and per-package `:indication_codes` (each entry a `{code:, cud_id:, text:}` hash, where `cud_id` is the `limitationIndication` CUD reference used to resolve the text). From 3.0.7 onwards, `Builder#build_product` emits one `<INDICATION_CODE code="XXXXX.NN" cud_id="DRUG.NN">limitation text</INDICATION_CODE>` child per indication on every `<PRD>` in `oddb_product.xml`; live feed numbers: 539 products / 1,293 codes / 100 % with non-empty indication text. Mandatory on prescriptions/invoices for SL price-model drugs from 2026-07-01 — see issue [#113](https://github.com/zdavatz/oddb2xml/issues/113). **Limitation texts** (3.0.8 onwards): the `regulatedAuthorization-limitation` extension has no inline `limitationText` in the live BAG feed — it carries a `limitationIndication` reference to a `ClinicalUseDefinition` whose `indication.diseaseSymptomProcedure.concept.text` is the actual text. The parser stores the ref as `cud_ref` on each Limitation, `Bundle#cud_text_by_id` resolves DE, and `merge_language` propagates FR/IT from the per-language NDJSON files via the same CUD id. Coverage on the live feed jumped from 0 / 9'108 to 9'108 / 9'108 (issue [#116](https://github.com/zdavatz/oddb2xml/issues/116)). **Limitation code / LIMNAMEBAG** (3.0.12 onwards): FHIR has no native BAG limitation code (LIMCD), so `create_limitations_for_package` sets `LimitationCode = cud_ref` (the `limitationIndication` CUD id) instead of `""`. Without this, every FHIR limitation shared an empty `:code`; `Builder#build_artikelstamm` groups its `<LIMITATIONS>` section by code, so all of them collapsed into a single `<LIMITATION>` with an empty `<LIMNAMEBAG>` and only one text survived. Using the CUD id as the key makes each distinct limitation emit and be referenced from its `<PRODUCT>`. The downstream `bin/check_artikelstamm` (`semantic_check.rb`) also crashed on the lone-element output because Ox `:hash_no_attrs` collapses a one-child section into a Hash (and an empty one into nil) — `SemanticCheckXML#get_items` now normalises every section to an Array. **v6 Artikelstamm / per-article INDC (3.0.26 onwards):** `--artikelstamm` now emits the **Elexis Artikelstamm v6** format (namespace `http://elexis.ch/Elexis_Artikelstamm_v6`, file `artikelstamm_DDMMYYYY_v6.xml`/`.csv`, validated against the bundled `Elexis_Artikelstamm_v6.xsd`) — replacing v5. The new piece is a per-`<ITEM>` `<ARTSL>` block carrying the BAG Indikationscodes (issue [#113](https://github.com/zdavatz/oddb2xml/issues/113)): `<PM>true</PM>` plus one `<ARTLIM>` per limitation with `<LIMCD>` (= `cud_ref`, the BAG limitation code), `<INDCD>` (the `XXXXX.NN` indication code from the `indicationCode` extension), and `<VDAT>`/`<VTDAT>` (period start/end). To feed it, `create_limitations_for_package` now also carries `IndicationCode` (→ per-package `:indcd`) and `ValidThruDate` (→ `:vtdate`) on each limitation; `Builder#append_artsl`/`elexis_datetime` emit one `<ARTLIM>` per limitation that has a non-empty `:indcd` (so non-price-model items get no `<ARTSL>`). `PM` is always `true` here because the indication code is required only for SL price-model drugs, which is exactly the set of items that reach this block. The bundled `Elexis_Artikelstamm_v6.xsd` is the canonical MEDEVIT schema extended with oddb2xml's historical Italian elements (`DSCRI` on PRODUCT/LIMITATION/ITEM, `DOSAGE_FORMI` on ITEM) so the output still validates. The legacy `--no-fhir` path emits no `<ARTSL>` (no FHIR limitations). **Per-article INDC in the `-e`/`-b` feeds (3.0.27 onwards):** the same indication codes are also emitted per `<ART>` in `oddb_article.xml` via `Builder#append_indication_codes` — one `<INDICATION_CODE code="XXXXX.NN" cud_id="DRUG.NN" limcd="DRUG.NN" vdat=… vtdat=…>limitation text</INDICATION_CODE>` per limitation that carries an `:indcd` (sourced from the article's `pac[:limitations]`). This reuses the flat `<INDICATION_CODE>` element already emitted on `<PRD>` in `oddb_product.xml` (since 3.0.7), enriched with the limitation code and validity dates so it carries the full `<ARTSL>` payload. The bundled `oddb2xml.xsd` now defines `<INDICATION_CODE>` (simpleContent + `code`/`cud_id`/`limcd`/`vdat`/`vtdat` attributes) and references it from both `<ART>` and `<PRD>` — the `<PRD>` reference had been missing since the 3.0.7 addition, so the FHIR product feed only validated once this was added.
+7. **FHIR support** (`lib/oddb2xml/fhir_support.rb`) — Self-contained module providing `FhirDownloader` and FHIR NDJSON parsing. Activated via `--fhir` (or `--fhir-url=<URL>`). Downloads per-language NDJSON files (`foph-sl-export-latest-{de,fr,it}.ndjson`) from `epl.bag.admin.ch` to populate French and Italian product names/descriptions. Maps legal status codes `756005022007` and `756005022008` to Swissmedic category D. Reads the BAG **Indikationscode** (`XXXXX.NN`) from the explicit `indicationCode` extension on each `RegulatedAuthorization.indication[].extension[regulatedAuthorization-limitation]` (BAG SL FHIR export >= v2.0.5; handled from 3.0.10). The BAG changelog states the limitation code (`ClinicalUseDefinition.id`) and the indication code are **independent** fields, so the older derivation — combining each indication CUD's `.NN` id-suffix with the reimbursement RA's `FOPHDossierNumber` — is kept only as a fallback for feeds lacking the extension. Exposed as `item[:indication_codes]` and per-package `:indication_codes` (each entry a `{code:, cud_id:, text:}` hash, where `cud_id` is the `limitationIndication` CUD reference used to resolve the text). From 3.0.7 onwards, `Builder#build_product` emits one `<INDICATION_CODE code="XXXXX.NN" cud_id="DRUG.NN">limitation text</INDICATION_CODE>` child per indication on every `<PRD>` in `oddb_product.xml`; live feed numbers: 539 products / 1,293 codes / 100 % with non-empty indication text. Mandatory on prescriptions/invoices for SL price-model drugs from 2026-07-01 — see issue [#113](https://github.com/zdavatz/oddb2xml/issues/113). **Limitation texts** (3.0.8 onwards): the `regulatedAuthorization-limitation` extension has no inline `limitationText` in the live BAG feed — it carries a `limitationIndication` reference to a `ClinicalUseDefinition` whose `indication.diseaseSymptomProcedure.concept.text` is the actual text. The parser stores the ref as `cud_ref` on each Limitation, `Bundle#cud_text_by_id` resolves DE, and `merge_language` propagates FR/IT from the per-language NDJSON files via the same CUD id. Coverage on the live feed jumped from 0 / 9'108 to 9'108 / 9'108 (issue [#116](https://github.com/zdavatz/oddb2xml/issues/116)). **Limitation code / LIMNAMEBAG** (3.0.12 onwards): FHIR has no native BAG limitation code (LIMCD), so `create_limitations_for_package` sets `LimitationCode = cud_ref` (the `limitationIndication` CUD id) instead of `""`. Without this, every FHIR limitation shared an empty `:code`; `Builder#build_artikelstamm` groups its `<LIMITATIONS>` section by code, so all of them collapsed into a single `<LIMITATION>` with an empty `<LIMNAMEBAG>` and only one text survived. Using the CUD id as the key makes each distinct limitation emit and be referenced from its `<PRODUCT>`. The downstream `bin/check_artikelstamm` (`semantic_check.rb`) also crashed on the lone-element output because Ox `:hash_no_attrs` collapses a one-child section into a Hash (and an empty one into nil) — `SemanticCheckXML#get_items` now normalises every section to an Array. **v6 Artikelstamm / per-article INDC (3.0.26 onwards):** `--artikelstamm` now emits the **Elexis Artikelstamm v6** format (namespace `http://elexis.ch/Elexis_Artikelstamm_v6`, file `artikelstamm_DDMMYYYY_v6.xml`/`.csv`, validated against the bundled `Elexis_Artikelstamm_v6.xsd`) — replacing v5. The new piece is a per-`<ITEM>` `<ARTSL>` block carrying the BAG Indikationscodes (issue [#113](https://github.com/zdavatz/oddb2xml/issues/113)): `<PM>true</PM>` plus one `<ARTLIM>` per limitation with `<LIMCD>` (= `cud_ref`, the BAG limitation code), `<INDCD>` (the `XXXXX.NN` indication code from the `indicationCode` extension), and `<VDAT>`/`<VTDAT>` (period start/end). To feed it, `create_limitations_for_package` now also carries `IndicationCode` (→ per-package `:indcd`) and `ValidThruDate` (→ `:vtdate`) on each limitation; `Builder#append_artsl`/`elexis_datetime` emit one `<ARTLIM>` per limitation that has a non-empty `:indcd` (so non-price-model items get no `<ARTSL>`). `PM` is always `true` here because the indication code is required only for SL price-model drugs, which is exactly the set of items that reach this block. The bundled `Elexis_Artikelstamm_v6.xsd` is the canonical MEDEVIT schema extended with oddb2xml's historical Italian elements (`DSCRI` on PRODUCT/LIMITATION/ITEM, `DOSAGE_FORMI` on ITEM) so the output still validates. The legacy `--no-fhir` path emits no `<ARTSL>` (no FHIR limitations). **Per-article INDC in the `-e`/`-b` feeds (3.0.27 onwards):** the same indication codes are also emitted per `<ART>` in `oddb_article.xml` via `Builder#append_indication_codes` — one `<INDICATION_CODE code="XXXXX.NN" cud_id="DRUG.NN" limcd="DRUG.NN" vdat=… vtdat=…>limitation text</INDICATION_CODE>` per limitation that carries an `:indcd` (sourced from the article's `pac[:limitations]`). This reuses the flat `<INDICATION_CODE>` element already emitted on `<PRD>` in `oddb_product.xml` (since 3.0.7), enriched with the limitation code and validity dates so it carries the full `<ARTSL>` payload. The bundled `oddb2xml.xsd` now defines `<INDICATION_CODE>` (simpleContent + `code`/`cud_id`/`limcd`/`vdat`/`vtdat` attributes) and references it from both `<ART>` and `<PRD>` — the `<PRD>` reference had been missing since the 3.0.7 addition, so the FHIR product feed only validated once this was added. **Legacy v5 opt-in (3.0.28 onwards):** `--artikelstamm-v5` additionally emits the older Artikelstamm v5 file alongside the v6 one (for consumers not yet migrated). It implies `--artikelstamm`; `Options.parse` sets `@opts[:artikelstamm] = true`. `build_artikelstamm` takes a `version:` keyword (default `6`) that drives the namespace (`Elexis_Artikelstamm_v#{version}`) and CSV/XML filename suffix; `build_artikelstamm_v5` calls it with `version: 5`. The only content difference is that the v5 path skips `append_artsl` (the v5 schema has no `<ARTSL>` element). The CLI `files` map adds an `:artikelstamm_v5` entry (→ `build_artikelstamm_v5` via the `to_xml` subject dispatch) and validates it against the bundled `Elexis_Artikelstamm_v5.xsd`. Both files come from one build — the `@prepared` guard means the shared download/prepare phase runs once.
 
 8. **Refdata cleanup** (`lib/oddb2xml/refdata_cleanup.rb`) — Compensates for known data-quality issues in upstream Refdata.Articles.xml before they reach the output. Each fix is guarded by a Swissmedic-side heuristic (e.g. comma in `substance_swissmedic` to distinguish mono products from real combinations). Currently fixes (a) the doubled-dose template bug (`X mg / X mg / Stk`, `fix_double_dose`, guarded by `single_substance?`); (b) the spelled-out German galenic form `Retardtabletten` → house-style abbreviation `Ret Tabl` (`normalize_galenic_form` / `GALENIC_NORMALISATIONS`, issue #112 case #13, e.g. RINVOQ — a narrow word-boundary substitution that leaves legitimate brand suffixes like `TRAMAL retard` and Mepha's `Lactab` untouched); and (c) dose info Refdata dropped from `<FullName>`, sourced from the Swissmedic composition string `pack[:composition_swissmedic]` — `fix_missing_combo_dose` (#6, appends a combination's 2nd component strength), `fix_missing_dose` (#4, inserts a mono product's missing strength before the pack count), `fix_missing_volume` (#7, appends an injectable's per-pen volume); and (d) 50-char-truncation repairs — `fix_truncated_metoject` (#1, rebuilds METOJECT Autoinjektor names from the intact `<brand> Autoinjektor <dose>/<vol>` prefix + Swissmedic `size`, localised DE/FR/IT) and `fix_truncated_volume_unit` (#3, restores the cut `ml` of the VERACTIV Vitamin D3 drops). The (c) and (d) fixes are scoped to explicit IKSNR allow-lists (`COMBO_DOSE_IKSNR`/`MISSING_DOSE_IKSNR`/`MISSING_VOLUME_IKSNR`/`METOJECT_IKSNR`/`VERACTIV_VITD3_IKSNR`): a dry run proved a blanket heuristic mis-fires on hundreds of legitimate names (sodium counter-ion doses, strength-less phyto/powder products, concentration names like `CIMZIA 200 mg/ml`), so only catalogued registrations are touched — add an IKSNR to grow coverage. Called from `Builder#apply_refdata_description_cleanups!` at the start of `prepare_articles`. See GitHub issue #112 for the catalogue.
 
@@ -68,7 +68,7 @@ YAML files in `data/` provide manual overrides and mappings: `article_overrides.
 
 These scripts run the public download server at `https://mediupdatexml.oddb.org` (Apache on this host) and are **not** part of the gem itself.
 
-- **`run_oddb2xml.sh`** — nightly build driver (cron: `0 1 * * * zdavatz`). Downloads the upstream sources **once**, then builds the `-b`/firstbase feed at price increments `45/50/55` plus `default` (no increment) into `$OUT_DIR` (`/home/zdavatz/oddb2xml`, one subdir each). The shared `downloads/` cache and transient zip live in `$BUILD_DIR` (`<OUT_DIR>-build`), **outside** `$OUT_DIR` so the transfer never uploads the multi-hundred-MB cache. Final step ("2b") regenerates the landing page. Each `oddb2xml` invocation is wrapped in `run_with_retry` (default **3 attempts, 120 s apart**, tunable via `ODDB2XML_RETRIES`/`ODDB2XML_RETRY_DELAY`): a transient upstream download failure (e.g. Swissmedic resetting the connection, `Errno::ECONNRESET`) previously aborted the whole `set -e` run 14 s in and left the feeds a day stale, so it now retries before giving up; a genuine repeated failure still stops the run.
+- **`run_oddb2xml.sh`** — nightly build driver (cron: `0 1 * * * zdavatz`). Downloads the upstream sources **once**, then builds the `-b`/firstbase feed at price increments `45/50/55` plus `default` (no increment) into `$OUT_DIR` (`/home/zdavatz/oddb2xml`, one subdir each). The shared `downloads/` cache and transient zip live in `$BUILD_DIR` (`<OUT_DIR>-build`), **outside** `$OUT_DIR` so the transfer never uploads the multi-hundred-MB cache. Final step ("2b") regenerates the landing page. Each `oddb2xml` invocation is wrapped in `run_with_retry` (default **3 attempts, 120 s apart**, tunable via `ODDB2XML_RETRIES`/`ODDB2XML_RETRY_DELAY`): a transient upstream download failure (e.g. Swissmedic resetting the connection, `Errno::ECONNRESET`) previously aborted the whole `set -e` run 14 s in and left the feeds a day stale, so it now retries before giving up; a genuine repeated failure still stops the run. **Firstbase (GS1 NONPHARMA) last-good fallback (3.0.29 onwards):** the GS1 `GetFirstbaseHealthcare` export (`id.gs1.ch/01/07612345000961` → `apitools.gs1.ch`) has been answering `403 - Forbidden`, which blanked `firstbase.csv` and dropped **every** NONPHARMA article from the `-b` feed (landing page then showed `NONPHARMA = 0 − 1 = −1`). The script keeps the last successful `firstbase.csv` in a persistent cache `$FIRSTBASE_CACHE` (default `<OUT_DIR>-state/firstbase.csv`) **outside** `$BUILD_DIR` so it survives the nightly `rm -rf`, seeds it into `downloads/` before the build, and refreshes it after a successful download. The gem side (`FirstbaseDownloader#download`, rewritten in 3.0.29) makes the seed usable: it still attempts the live GS1 fetch on the first (downloading) build, but only overwrites `firstbase.csv` when the response is a real non-empty CSV (`firstbase_csv?` rejects HTML/`403 - Forbidden`/empty bodies and open-uri exceptions), otherwise it **keeps the existing seeded file** instead of the old `"w+"` truncate-to-zero. A recovered GS1 therefore refreshes the data automatically; while GS1 is down the feed serves yesterday's (last-good) NONPHARMA rather than nothing. `generate_index_html.sh` also guards the NONPHARMA count so an empty CSV renders `—` (not `−1`).
 - **`generate_index_html.sh DOCROOT [FIRSTBASE_CSV]`** — single source of truth for the landing page. Writes `index.html` + a self-contained `logo.svg` **atomically** (temp + `mv`, so either owner — root from setup, `zdavatz` from cron — can refresh it). Computes live counts: PHARMA = `<SMNO>` count in `default/oddb_article.xml`, NONPHARMA = firstbase CSV rows − 1, total ART = `<ART ` count. Also runs **`visitor_stats.py`** and embeds its graph. Re-run standalone any time (it only reads already-built files); a separate cron line refreshes it **hourly** (`5 * * * * zdavatz`) so counts + graph stay current between nightly builds.
 - **`visitor_stats.py LOG_GLOB CACHE_DIR [DAYS]`** — emits the visitors/sessions/region graph as an inline-SVG HTML **fragment** (last `DAYS`, default 14): Besucher = distinct IPs/day, Sitzungen = 30-min-inactivity sessions per `(IP, User-Agent)`, plus a top-6 country breakdown by IP. Bots are filtered by User-Agent. Region lookup is **fully self-contained** — pure Python stdlib + the free **DB-IP country-lite CSV** (CC-BY, no licence key) cached in the build `downloads/` dir and refreshed monthly; **no apt package, no gem, no system GeoIP DB**. Prints nothing (page degrades to omitting the section) when the Apache log is unreadable or empty. Reading `/var/log/apache2` requires the cron user to be in the **`adm`** group (`sudo usermod -aG adm zdavatz`).
 - **`swissmedic_watch.sh`** — outage/block auto-recovery (cron: `*/30 * * * * zdavatz`). Since the Swissmedic platform migration (~2026-06-23, now a Swisscom-operated gateway), `www.swissmedic.ch` intermittently resets this host's automated connections **after the TLS handshake** (TCP RST), which aborts `run_oddb2xml.sh` under `set -e` and leaves the feeds stale (the block is host/IP- and client-fingerprint-sensitive: a real browser works, `curl`/`wget`/Ruby get reset, while other admin.ch hosts answer fine — so it is a WAF/bot rule, not an outage). The watcher polls Swissmedic with **oddb2xml's own client** (a Ruby `open-uri` canary on `listen_neu.html`); while blocked it is a silent no-op, and the moment it gets HTTP 200 it launches **one** build and emails. It fires **at most once per day** (stamp in `$STATE_DIR`, default `<OUT_DIR>-watch`, kept **outside** the wiped `$BUILD_DIR`), and skips when a build is already running or today's `default/oddb_article.xml` is already fresh. It exports `RBENV_VERSION=3.4.5` + the rbenv-shims PATH to match the nightly cron (the repo `.ruby-version` pins an uninstalled Ruby).

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    oddb2xml (3.0.27)
+    oddb2xml (3.0.29)
       csv
       htmlentities
       httpi

data/History.txt

@@ -1,3 +1,6 @@
+=== 3.0.28 / 30.06.2026
+* New (--artikelstamm-v5): optionally emit the legacy Elexis Artikelstamm Version 5 file in addition to the v6 file. --artikelstamm now always produces the v6 output (artikelstamm_DDMMYYYY_v6.xml/.csv, with the per-ITEM <ARTSL> BAG indication codes since 3.0.26); passing --artikelstamm-v5 implies --artikelstamm and additionally writes artikelstamm_DDMMYYYY_v5.xml/.csv in the older v5 format (namespace http://elexis.ch/Elexis_Artikelstamm_v5, no <ARTSL> block), validated against the bundled Elexis_Artikelstamm_v5.xsd. The shared download/prepare phase runs once, so both files come from a single build. For consumers that have not yet migrated to v6.
+
 === 3.0.27 / 29.06.2026
 * New (-e/--extended, -b/--firstbase): carry the BAG Indikationscode (INDC) per article in oddb_article.xml as well. Each SL price-model <ART> now gets one <INDICATION_CODE code="XXXXX.NN" cud_id="DRUG.NN" limcd="DRUG.NN" vdat="..." vtdat="...">limitation text</INDICATION_CODE> child per indication -- the same data as the v6 Artikelstamm <ARTSL> block (3.0.26), but in the flat <INDICATION_CODE> shape already emitted on <PRD> in oddb_product.xml, enriched with the limitation code (limcd) and validity dates (vdat/vtdat). The bundled oddb2xml.xsd now defines <INDICATION_CODE> and references it from both <ART> and <PRD> (the <PRD> reference was missing since the 3.0.7 product-feed addition). Only emitted in --fhir mode (the default), where limitations carry an indication code. See issue #113.
 

data/README.md

@@ -57,6 +57,9 @@ see `--help`.
       produced files are found under data
     -a, --append              Additional target nonpharma
     -r, --artikelstamm        Create Artikelstamm Version 6 for Elexis >= 3.1
+    --artikelstamm-v5         Additionally create the legacy Artikelstamm Version 5
+                                                          (without the <ARTSL> BAG indication codes).
+                                                          Implies --artikelstamm.
     -c, --compress-ext=<s>    format F. {tar.gz|zip}
     -e, --extended            pharma, non-pharma plus prices and non-pharma from zurrose.
                                                           Products without EAN-Code will also be listed.
@@ -371,6 +374,14 @@ The output bumps from Artikelstamm v5 to **v6** (namespace
 `artikelstamm_DDMMYYYY_v6.xml`, validated against the bundled
 `Elexis_Artikelstamm_v6.xsd`); consumers must switch to the v6 file.
 
+Consumers that have not yet migrated can keep receiving the **legacy v5
+file** by adding `--artikelstamm-v5` (implies `--artikelstamm`). This writes
+`artikelstamm_DDMMYYYY_v5.xml`/`.csv` *in addition to* the v6 files, in the
+older v5 format (namespace `http://elexis.ch/Elexis_Artikelstamm_v5`, no
+`<ARTSL>` block, validated against the bundled `Elexis_Artikelstamm_v5.xsd`).
+The download and data-preparation phase runs only once, so both versions are
+produced from a single build.
+
 Since 3.0.27 the same indication codes are also carried per article in
 the `-e`/`--extended` and `-b`/`--firstbase` feeds: each SL price-model
 `<ART>` in `oddb_article.xml` gets one `<INDICATION_CODE>` child per

data/lib/oddb2xml/builder.rb

@@ -1622,9 +1622,17 @@ module Oddb2xml
       end
     end
 
-    def build_artikelstamm
+    # Legacy entry point: emit the Artikelstamm in the older Version 5 format
+    # (Elexis_Artikelstamm_v5, no per-ITEM <ARTSL> indication-code block).
+    # Reached via subject :artikelstamm_v5 when --artikelstamm-v5 is set.
+    def build_artikelstamm_v5
+      build_artikelstamm(version: 5)
+    end
+
+    def build_artikelstamm(version: 6)
       @@emitted_v6_gtins = []
-      @csv_file = CSV.open(File.join(WORK_DIR, "artikelstamm_#{Date.today.strftime("%d%m%Y")}_v6.csv"), "w+")
+      @artikelstamm_version = version
+      @csv_file = CSV.open(File.join(WORK_DIR, "artikelstamm_#{Date.today.strftime("%d%m%Y")}_v#{version}.csv"), "w+")
       @csv_file << ["gtin", "name", "pkg_size", "galenic_form", "price_ex_factory", "price_public", "prodno", "atc_code", "active_substance", "original", "it-code", "sl-liste"]
       @csv_file.sync = true
       variant = "build_artikelstamm"
@@ -1832,7 +1840,8 @@ module Oddb2xml
                   end
                   xml.PRODNO prodno if prodno
                   # v6 <ARTSL>: per-article BAG Indikationscodes (issue #113).
-                  append_artsl(xml, package[:limitations])
+                  # The legacy v5 schema has no <ARTSL> element, so skip it there.
+                  append_artsl(xml, package[:limitations]) if @artikelstamm_version != 5
                   @csv_file << [pkg_gtin, name, package[:unit], measure,
                     pexf || "",
                     ppub || "",
@@ -1958,7 +1967,7 @@ module Oddb2xml
         elexis_strftime_format = "%FT%T\.%L%:z"
         @@cumul_ver = (Date.today.year - 2013) * 12 + Date.today.month
         options_xml = {
-          "xmlns" => "http://elexis.ch/Elexis_Artikelstamm_v6",
+          "xmlns" => "http://elexis.ch/Elexis_Artikelstamm_v#{@artikelstamm_version}",
           "CREATION_DATETIME" => Time.new.strftime(elexis_strftime_format),
           "BUILD_DATETIME" => Time.new.strftime(elexis_strftime_format),
           "DATA_SOURCE" => "oddb2xml"

data/lib/oddb2xml/cli.rb

@@ -103,14 +103,21 @@ module Oddb2xml
       end
       build
       if @options[:artikelstamm] && system("which xmllint")
-        elexis_v6_xsd = File.expand_path(File.join(__FILE__, "..", "..", "..", "Elexis_Artikelstamm_v6.xsd"))
-        cmd = "xmllint --noout --schema #{elexis_v6_xsd} #{@the_files[:artikelstamm]}"
-        if system(cmd)
-          puts "Validatied #{@the_files[:artikelstamm]}"
-        else
-          puts "Validating failed using #{cmd}"
-          require'pry'; binding.pry
-          raise "Validating failed using #{cmd}"
+        to_validate = {"Elexis_Artikelstamm_v6.xsd" => @the_files[:artikelstamm]}
+        if @options[:artikelstamm_v5] && @the_files[:artikelstamm_v5]
+          to_validate["Elexis_Artikelstamm_v5.xsd"] = @the_files[:artikelstamm_v5]
+        end
+        to_validate.each_pair do |xsd_name, xml_file|
+          xsd = File.expand_path(File.join(__FILE__, "..", "..", "..", xsd_name))
+          cmd = "xmllint --noout --schema #{xsd} #{xml_file}"
+          if system(cmd)
+            puts "Validatied #{xml_file}"
+          else
+            puts "Validating failed using #{cmd}"
+            require "pry"
+            binding.pry
+            raise "Validating failed using #{cmd}"
+          end
         end
       end
       compress if @options[:compress_ext]
@@ -400,6 +407,9 @@ module Oddb2xml
         end
         if @options[:artikelstamm]
           @the_files[:artikelstamm] = "artikelstamm_#{Date.today.strftime("%d%m%Y")}_v6.xml"
+          if @options[:artikelstamm_v5]
+            @the_files[:artikelstamm_v5] = "artikelstamm_#{Date.today.strftime("%d%m%Y")}_v5.xml"
+          end
         elsif @options[:address]
           @the_files[:company] = "#{prefix}_betrieb.xml"
           @the_files[:person] = "#{prefix}_medizinalperson.xml"

data/lib/oddb2xml/downloader.rb

@@ -396,14 +396,53 @@ module Oddb2xml
       @url = BASE_URL
     end
 
+    # A valid firstbase export is a non-empty CSV. When GS1 is unavailable it
+    # answers with an HTML error page (the GetFirstbaseHealthcare endpoint has
+    # been returning "403 - Forbidden: Access is denied") or open-uri raises.
+    # The old "w+" download truncated firstbase.csv to zero bytes on any such
+    # failure, silently dropping every NONPHARMA article. Reject non-CSV bodies
+    # so the caller can keep the previous good firstbase.csv instead.
+    def firstbase_csv?(text)
+      return false if text.nil?
+      head = text[0, 512].to_s.strip.downcase
+      return false if head.empty?
+      return false if head.start_with?("<!doctype", "<html", "<?xml")
+      return false if head.include?("403 - forbidden") || head.include?("access is denied")
+      true
+    end
+
     def download
       @file2save = File.join(DOWNLOADS, "firstbase.csv")
       report_download(@url, @file2save)
-      begin
-        download_as(@file2save, "w+")
+      # Price-increment / Artikelstamm runs (--skip-download) reuse the cached
+      # firstbase.csv. Do NOT skip merely because the file exists: the nightly
+      # deploy seeds a last-good copy so a GS1 outage does not blank the feed,
+      # and we still want a fresh download attempt on the first (downloading)
+      # build so a recovered GS1 refreshes the data.
+      if Oddb2xml.skip_download? && File.size?(@file2save)
+        Oddb2xml.log "FirstbaseDownloader: --skip-download, reusing cached #{@file2save} (#{File.size(@file2save)} bytes)"
         return File.expand_path(@file2save)
+      end
+      begin
+        data = Oddb2xml.uri_open(@url).read
+        if firstbase_csv?(data)
+          File.write(@file2save, data)
+          Oddb2xml.log "FirstbaseDownloader: fetched fresh firstbase.csv (#{data.bytesize} bytes)"
+        elsif File.size?(@file2save)
+          Oddb2xml.log "FirstbaseDownloader: GS1 returned no CSV (#{data.to_s.bytesize} bytes); keeping existing #{@file2save} (#{File.size(@file2save)} bytes)"
+        else
+          Oddb2xml.log "FirstbaseDownloader: GS1 returned no CSV and there is no cached firstbase.csv to fall back to"
+        end
       rescue Timeout::Error, Errno::ETIMEDOUT
         retrievable? ? retry : raise
+      rescue => error
+        # 403 / blocked / unreachable: keep any existing firstbase.csv (e.g. the
+        # last-good copy the deploy script seeds) rather than truncating it.
+        if File.size?(@file2save)
+          Oddb2xml.log "FirstbaseDownloader: download failed (#{error.class}: #{error}); keeping existing #{@file2save} (#{File.size(@file2save)} bytes)"
+        else
+          Oddb2xml.log "FirstbaseDownloader: download failed (#{error.class}: #{error}) and no cached firstbase.csv to fall back to"
+        end
       ensure
         Oddb2xml.download_finished(@file2save, false)
       end

data/lib/oddb2xml/options.rb

@@ -18,6 +18,7 @@ module Oddb2xml
         EOS
         opt :append, "Additional target nonpharma", default: false
         opt :artikelstamm, "Create Artikelstamm Version 6 for Elexis >= 3.1"
+        opt :artikelstamm_v5, "Additionally create the legacy Artikelstamm Version 5 (without the <ARTSL> BAG indication codes). Implies --artikelstamm.", short: :none, default: false
         opt :compress_ext, "format F. {tar.gz|zip}", type: :string, default: nil, short: "c"
         opt :extended, "pharma, non-pharma plus prices and non-pharma from zurrose.
                             Products without EAN-Code will also be listed.
@@ -68,6 +69,11 @@ module Oddb2xml
         @opts[:price] = :zurrose
         @opts[:calc] = true
       end
+      # --artikelstamm-v5 additionally emits the legacy v5 file; it requires the
+      # full Artikelstamm build, so it implies --artikelstamm (which yields v6).
+      if @opts[:artikelstamm_v5]
+        @opts[:artikelstamm] = true
+      end
       if @opts[:artikelstamm]
         @opts[:extended] = true
         @opts[:price] = :zurrose

data/lib/oddb2xml/version.rb

@@ -1,3 +1,3 @@
 module Oddb2xml
-  VERSION = "3.0.27"
+  VERSION = "3.0.29"
 end

data/scripts/generate_index_html.sh

@@ -35,7 +35,13 @@ pharma="—"
 [[ -f "$ARTICLE_XML" ]] && pharma=$(grep -c '<SMNO>' "$ARTICLE_XML" || true)
 
 nonpharma="—"
-[[ -f "$FIRSTBASE_CSV" ]] && nonpharma=$(( $(wc -l < "$FIRSTBASE_CSV") - 1 ))
+# Only count when the CSV actually has data rows. An empty firstbase.csv (the
+# GS1 firstbase dump upstream returns 403/empty from time to time) would make a
+# naive `rows - 1` render "-1"; keep the "—" fallback instead.
+if [[ -s "$FIRSTBASE_CSV" ]]; then
+  fb_rows=$(wc -l < "$FIRSTBASE_CSV")
+  (( fb_rows > 1 )) && nonpharma=$(( fb_rows - 1 ))
+fi
 
 stand=$(date '+%d.%m.%Y %H:%M')
 
@@ -148,6 +154,12 @@ cat > "$tmp" <<HTML
   <a href="default/oddb_code.xml">oddb_code.xml</a> sowie
   <a href="default/oddb2xml.zip">oddb2xml.zip</a> (alle Dateien gepackt).</p>
 
+  <h2>Elexis Artikelstamm</h2>
+  <ul>
+    <li><a href="artikelstamm/">artikelstamm/</a> <span class="desc">— Elexis Artikelstamm v6 (mit BAG-Indikationscodes) und Legacy-v5, je als XML und CSV, täglich aktualisiert (erzeugt mit oddb2xml)</span></li>
+    <li><a href="artikelstamm/rust2xml/">artikelstamm/rust2xml/</a> <span class="desc">— derselbe Artikelstamm v6 + v5, erzeugt mit <a href="https://github.com/zdavatz/rust2xml">rust2xml</a> (Rust-Port), täglich um 03:00 aus denselben Live-Quellen</span></li>
+  </ul>
+
   <h2>aips2sqlite — Fachinformationen &amp; AmiKo-Datenbanken</h2>
   <ul>
     <li><a href="/aips2sqlite/fis/">fis/</a> <span class="desc">— Fachinformationen als XML/HTML (DE/FR/IT)</span></li>

data/scripts/run_oddb2xml.sh

@@ -11,10 +11,12 @@
 # silently re-downloaded everything each time.
 #
 # Output layout (under $OUT_DIR, default /home/zdavatz/oddb2xml):
-#   <OUT_DIR>/45/        oddb_*.xml built with +45 %
-#   <OUT_DIR>/50/        oddb_*.xml built with +50 %
-#   <OUT_DIR>/55/        oddb_*.xml built with +55 %
-#   <OUT_DIR>/default/   oddb_*.xml built with no increment
+#   <OUT_DIR>/45/           oddb_*.xml built with +45 %
+#   <OUT_DIR>/50/           oddb_*.xml built with +50 %
+#   <OUT_DIR>/55/           oddb_*.xml built with +55 %
+#   <OUT_DIR>/default/      oddb_*.xml built with no increment
+#   <OUT_DIR>/artikelstamm/ Elexis Artikelstamm v6 + legacy v5 (xml + csv),
+#                           served at https://mediupdatexml.oddb.org/artikelstamm
 # Each destination dir also keeps the source archive as oddb2xml.zip.
 # The working dir ($BUILD_DIR, default <OUT_DIR>-build) holds the shared
 # downloads/ cache and the transient zip; it lives OUTSIDE $OUT_DIR so the
@@ -79,6 +81,41 @@ rm -rf "$BUILD_DIR"
 mkdir -p "$BUILD_DIR"
 cd "$BUILD_DIR"
 
+# 3. Seed the ZurRose transfer.zip from the local get_transfer mirror.
+# get_transfer.sh (crontab 00:30) downloads transfer.dat straight from
+# zurrose.ch on THIS host and uploads the zip to pillbox.oddb.org — so the
+# pillbox HTTP fetch is a needless detour back to our own file. Placing the
+# zip in downloads/ makes oddb2xml's skip_download reuse it and the build no
+# longer depends on pillbox being up (2026-07-02: pillbox refused connections
+# during the 01:00 run and all three retries died, killing the whole nightly
+# build). If the seed file is missing, oddb2xml falls back to the normal
+# pillbox download as before.
+GET_TRANSFER_ZIP="${GET_TRANSFER_ZIP:-/home/zdavatz/software/get_transfer/TRANSFER.ZIP}"
+if [[ -s "$GET_TRANSFER_ZIP" ]]; then
+  mkdir -p "$BUILD_DIR/downloads"
+  cp -p "$GET_TRANSFER_ZIP" "$BUILD_DIR/downloads/transfer.zip"
+  log "Seeded ZurRose transfer.zip from $GET_TRANSFER_ZIP ($(date -r "$GET_TRANSFER_ZIP" '+%Y-%m-%d %H:%M'), no pillbox fetch needed)"
+else
+  log "WARNING: $GET_TRANSFER_ZIP missing - falling back to pillbox.oddb.org download"
+fi
+
+# 3b. Firstbase (GS1 NONPHARMA) fallback. The GS1 GetFirstbaseHealthcare
+# endpoint has been answering "403 - Forbidden", which blanked firstbase.csv and
+# dropped every NONPHARMA article from the -b feed. Keep the last successful
+# firstbase.csv in a persistent cache OUTSIDE $BUILD_DIR (it survives the nightly
+# `rm -rf`) and seed it into downloads/ so the gem's FirstbaseDownloader falls
+# back to yesterday's file when today's download fails. A recovered GS1 still
+# refreshes the data: the first (downloading) build always retries the live
+# fetch and only keeps the seed when that fetch yields no CSV.
+FIRSTBASE_CACHE="${FIRSTBASE_CACHE:-${OUT_DIR%/}-state/firstbase.csv}"
+if [[ -s "$FIRSTBASE_CACHE" ]]; then
+  mkdir -p "$BUILD_DIR/downloads"
+  cp -p "$FIRSTBASE_CACHE" "$BUILD_DIR/downloads/firstbase.csv"
+  log "Seeded firstbase.csv from last-good cache $FIRSTBASE_CACHE ($(($(wc -l < "$FIRSTBASE_CACHE") - 1)) rows, $(date -r "$FIRSTBASE_CACHE" '+%Y-%m-%d %H:%M'))"
+else
+  log "No firstbase last-good cache at $FIRSTBASE_CACHE yet - relying on live GS1 download"
+fi
+
 first=1
 
 # build_one <increment-percent|""> <destination-subdir>
@@ -113,10 +150,60 @@ build_one() {
   log "Staged $dest"
 }
 
+# build_artikelstamm — build the Elexis Artikelstamm (v6 + legacy v5) and stage
+# it at $OUT_DIR/artikelstamm, served at
+# https://mediupdatexml.oddb.org/artikelstamm. Re-uses the shared downloads/
+# cache (--skip-download), so it adds no extra upstream fetch beyond the few
+# sources the firstbase builds don't pull (e.g. the ZurRose transfer.dat).
+build_artikelstamm() {
+  local dest="$OUT_DIR/artikelstamm"
+  log "Building Artikelstamm (v6 + v5) -> $dest"
+  rm -f artikelstamm_*.xml artikelstamm_*.csv
+  run_with_retry "oddb2xml artikelstamm" -- \
+    "$ODDB2XML_BIN" --skip-download --artikelstamm --artikelstamm-v5
+
+  shopt -s nullglob
+  local out=(artikelstamm_*.xml artikelstamm_*.csv)
+  shopt -u nullglob
+  [[ ${#out[@]} -ge 1 ]] || { log "ERROR: no artikelstamm output produced"; exit 1; }
+
+  mkdir -p "$dest"
+  # Remove only oddb2xml's own top-level files; keep sub-directories such as
+  # rust2xml/ (published independently by rust2xml's own cron at 03:00) intact.
+  # A plain `rm -rf "$dest"` used to wipe that sibling output every night.
+  rm -f "$dest"/artikelstamm_*.xml "$dest"/artikelstamm_*.csv
+  # Publish under date-less, stable names so the download URLs never change:
+  # artikelstamm_01072026_v6.xml -> artikelstamm_v6.xml (same for _v5 / .csv).
+  local f base
+  for f in "${out[@]}"; do
+    base="$(basename "$f" | sed -E 's/_[0-9]{8}_/_/')"
+    cp -p "$f" "$dest/$base"
+  done
+  log "Staged ${#out[@]} file(s) to $dest"
+}
+
+# Build order: default first (it downloads the shared sources), then the
+# Artikelstamm right after so it is published early, then the price increments.
+build_one "" "default"           # first run: downloads sources, no increment
+
+# Refresh the last-good firstbase.csv cache after the downloading build. When
+# GS1 answered, downloads/firstbase.csv now holds fresh data; when it 403'd, the
+# gem kept the seeded copy - either way a non-empty file is worth caching so the
+# next run can fall back to it. An empty file means both today's download AND the
+# seed were missing, so leave the previous cache untouched.
+FIRSTBASE_LIVE="$BUILD_DIR/downloads/firstbase.csv"
+if [[ -s "$FIRSTBASE_LIVE" ]]; then
+  mkdir -p "$(dirname "$FIRSTBASE_CACHE")"
+  cp -p "$FIRSTBASE_LIVE" "$FIRSTBASE_CACHE"
+  log "Cached firstbase.csv as last-good ($(($(wc -l < "$FIRSTBASE_LIVE") - 1)) rows) -> $FIRSTBASE_CACHE"
+else
+  log "WARNING: firstbase.csv is empty after the build (GS1 403 and no cache) - NONPHARMA missing this run"
+fi
+
+build_artikelstamm               # Elexis Artikelstamm (v6 + legacy v5)
 for inc in $INCREMENTS; do
-  build_one "$inc" "$inc"
+  build_one "$inc" "$inc"        # price increments re-use the cached downloads/
 done
-build_one "" "default"           # final run with no increment
 
 # 2b. Refresh the download landing page with the live PHARMA/NONPHARMA counts
 #     (PHARMA from default/oddb_article.xml, NONPHARMA from the GS1 firstbase CSV).

data/spec/artikelstamm_spec.rb

@@ -512,6 +512,36 @@ Der behandelnde Arzt ist verpflichtet, die erforderlichen Daten laufend im vorge
       end
     end
   end
+  context "when artikelstamm-v5 option is given" do
+    before(:all) do
+      common_run_init
+      options = Oddb2xml::Options.parse(["--artikelstamm-v5", "--no-fhir"])
+      Oddb2xml::Cli.new(options).run
+      @v6_name = File.join(Oddb2xml::WORK_DIR, "artikelstamm_#{Date.today.strftime("%d%m%Y")}_v6.xml")
+      @v5_name = File.join(Oddb2xml::WORK_DIR, "artikelstamm_#{Date.today.strftime("%d%m%Y")}_v5.xml")
+    end
+
+    it "emits both the v6 and the legacy v5 file" do
+      expect(File.exist?(@v6_name)).to eq true
+      expect(File.exist?(@v5_name)).to eq true
+    end
+
+    it "uses the v5 namespace in the v5 file and the v6 namespace in the v6 file" do
+      expect(IO.read(@v5_name)).to include("http://elexis.ch/Elexis_Artikelstamm_v5")
+      expect(IO.read(@v6_name)).to include("http://elexis.ch/Elexis_Artikelstamm_v6")
+    end
+
+    it "validates the v5 file against Elexis_Artikelstamm_v5.xsd" do
+      xsd = File.expand_path(File.join(__FILE__, "..", "..", "Elexis_Artikelstamm_v5.xsd"))
+      schema = Nokogiri::XML::Schema(File.read(xsd))
+      doc = Nokogiri::XML(File.read(@v5_name))
+      expect(schema.validate(doc)).to eq []
+    end
+
+    it "never emits an <ARTSL> block in the v5 file" do
+      expect(IO.read(@v5_name)).not_to include("<ARTSL>")
+    end
+  end
   context "chapter 70 hack" do
     before(:all) do
       mock_downloads

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: oddb2xml
 version: !ruby/object:Gem::Version
-  version: 3.0.27
+  version: 3.0.29
 platform: ruby
 authors:
 - Yasuhiro Asaka, Zeno R.R. Davatz, Niklaus Giger