oddb2xml 3.0.33 → 3.0.34

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f81e1fe93a63834d7fda23c353903f34b6b7136146bf79c1e8b465653ed5585c
-  data.tar.gz: 516d5916ab9dd00a48973db7f923f367b8e5a2281cc48e1d0a50020153e71d0e
+  metadata.gz: aba1912f4878121d6d5aa1424b2c1f33f72b1eaceefd21d7b973753209b97c6e
+  data.tar.gz: 89ba2fffeefae9dffb569cb94be5308b67a7e847813e6de9771d5fee5a626996
 SHA512:
-  metadata.gz: a19787a1570d2078c25e7525d397af16fac509425de63ded2383f2781219479485f8fa720a888618af879d08a9e206b974d457bfc5d45a57dacf70deeb263062
-  data.tar.gz: e8592299737e7ee1bb7e86331ff0579c62d5e550ff075264bdb118a23f85aaf5f5ebb35310d92048e4fb71870e817f2aae9ca184ce51dd192c03ea9a9cc1801b
+  metadata.gz: b9c54d7a0f8cf8189c5703e36aa309c40302b8880a65e552122dc8728d9827f7b15f1ae1c976abab6a342efb6547d31dc44e957ffb8b5b93ef2adb6be7cced99
+  data.tar.gz: d185bc2f3f9eadd52e618038fbae1b9050aa65ed4e36e0116cba42a68b9948d19a6794e300a131328c4db5a3075f1f1b250c7a43ef7a481129a4aaa3efd59fd4

data/CLAUDE.md

@@ -28,7 +28,7 @@ The system follows a **download → extract → build → compress** pipeline:
 
 9. **Chapter-70 hack** (`lib/oddb2xml/chapter_70_hack.rb`) — Legacy scraper for the SL "Komplementärarzneimittel" products (homeopathic/anthroposophic/phytotherapeutic), called only from `Builder#build_artikelstamm`. **Deprecated / non-FHIR only (3.0.11 onwards):** the source page `varia_De.htm` was rebuilt as a JavaScript SPA with no static data table, so the scraper now returns nothing there. These products + limitations now come through the FHIR feed (SL classification `20. KOMPLEMENTÄRARZNEIMITTEL`, 221 products on the live DE feed with real GTINs and limitation texts), so `build_artikelstamm` **skips the scraper entirely when `@options[:fhir]`** (the default for `--artikelstamm` since 3.0.9). In `--no-fhir` mode the scraper degrades gracefully (skips non-row/`<script>` nodes and empty tables, warns, returns `[]`) instead of raising `NoMethodError`. See GitHub issue #118.
 
-10. **Weleda / Kapitel-70 SL recovery** (`lib/oddb2xml/weleda_sl.rb`, 3.0.21 onwards) — Recovers the SL flag and public price for chapter-70 complementary medicines that are **missing from the FHIR feed** (the partial-replacement gap left by the dead chapter_70_hack, issue #118/#121). Many are magistral Weleda preparations with a `7611916…` trade GTIN that arrive only via ZurRose — with no SL flag and a blanked Publikumspreis (issue #117). `WeledaSL.load` joins two CSVs (downloaded at runtime from `github.com/zdavatz/oddb2xml_files` via `WeledaDownloader` / `BagSlGroupPricesDownloader`, bundled fallback copies under `data/`): `weleda_arzneimittel.csv` (GTIN → `abgabekategorie` SL flag + `csl` = **Pharma-Gruppen-Code**) and `bag_sl_group_prices.csv` (Pharma-Gruppen-Code → public price). `weleda_arzneimittel.csv` is **regenerated** by the Rust tool `weleda_scraper/` in the `oddb2xml_files` repo (`scraper --update weleda`, prompts for the medical.weleda.ch `PHPSESSID` cookie — never stored): it walks the paginated Arzneimittel-Verzeichnis listing + per-product detail pages and rewrites the CSV with exactly the currently-listed products (delisted rows dropped), preserving the on-disk format (column order, UTF-8, CRLF, quote-when-necessary, sorted by `id`) so oddb2xml reads it unchanged. The price table is extracted **offline** from the BAG SL definition PDF *"Homoeopathica, Anthroposophica, Allergene"* via `tools/generate_bag_sl_group_prices.rb` (uses system `pdftotext`; **no runtime PDF gem** — `pdf-reader`'s `afm` dep now needs Ruby ≥ 3.2, which would break the gem's Ruby floor). The join is **GTIN → csl → price**, honouring an `N x <code>` package multiplier (price = N × group price). Produces `gtin => {sl:, price:, csl:, abgabe:}` (SL rows only; ~515 priced on the live feed). **WALA products (3.0.22 onwards):** a third runtime CSV `wala_arzneimittel.csv` (GTIN prefix `7640187…`, `WalaDownloader`, bundled fallback) is merged into the same map via `WeledaSL.build_wala_map`. Its layout differs: `;`-separated with a BOM, no `/ SL` column (a row is SL when it carries a `CSL-Code` = Kapitel-70.01 group code), and the public **package** price is given inline in the `CSL 70.01.` column — **already multiplied for the pack size** (the multiplier appears only in the galenic-form text, e.g. `Solutio ad inj. 10 x 1 ml`), so it is taken **verbatim** rather than re-joined against `bag_sl_group_prices.csv` (which holds the per-unit price and would yield 1/10 of the package price for ~120 multi-unit packs). 320 WALA SL products on the live file; Weleda wins on the (unlikely) GTIN collision. `Builder#build_artikelstamm` consumes it (CLI sets `builder.weleda_sl` only for `--artikelstamm`): for any GTIN **absent from the FHIR NDJSON** it emits `<SL_ENTRY>true</SL_ENTRY>` and `<PPUB>` from the BAG group price, mirroring the old chapter-70 behaviour (`PHARMATYPE "P"`). **The FHIR/ZurRose price always wins** — the group price only fills a gap; a zeroed ZurRose `"0.00"` pub price is treated as absent so the gap-fill can apply. Match is **by GTIN only** (no pharmacode); the Swissmedic dispensing category is untouched (still from `Swissmedic_Packungen.xlsx`). The Artikelstamm output gets `<SL_ENTRY>` + `<PPUB>`; for the `-e`/`--extended` and `-b`/`--firstbase` product feeds the BAG public price is also added to `oddb_article.xml` as an `<ARTPRI><PTYP>BAGPUB</PTYP>` entry (the raw, often-blanked `ZURROSEPUB` is preserved alongside it) — `build_article`, gated by the CLI loading `weleda_sl` when `extended || firstbase || artikelstamm`. See GitHub issue #121.
+10. **Weleda / Kapitel-70 SL recovery** (`lib/oddb2xml/weleda_sl.rb`, 3.0.21 onwards) — Recovers the SL flag and public price for chapter-70 complementary medicines that are **missing from the FHIR feed** (the partial-replacement gap left by the dead chapter_70_hack, issue #118/#121). Many are magistral Weleda preparations with a `7611916…` trade GTIN that arrive only via ZurRose — with no SL flag and a blanked Publikumspreis (issue #117). `WeledaSL.load` joins two CSVs (downloaded at runtime from `github.com/zdavatz/oddb2xml_files` via `WeledaDownloader` / `BagSlGroupPricesDownloader`, bundled fallback copies under `data/`): `weleda_arzneimittel.csv` (GTIN → `abgabekategorie` SL flag + `csl` = **Pharma-Gruppen-Code**) and `bag_sl_group_prices.csv` (Pharma-Gruppen-Code → public price). `weleda_arzneimittel.csv` is **regenerated** by the Rust tool `weleda_scraper/` in the `oddb2xml_files` repo (`scraper --update weleda`, prompts for the medical.weleda.ch `PHPSESSID` cookie — never stored): it walks the paginated Arzneimittel-Verzeichnis listing + per-product detail pages and rewrites the CSV with exactly the currently-listed products (delisted rows dropped), preserving the on-disk format (column order, UTF-8, CRLF, quote-when-necessary, sorted by `id`) so oddb2xml reads it unchanged. The price table is extracted **offline** from the BAG SL definition PDF *"Homoeopathica, Anthroposophica, Allergene"* via `tools/generate_bag_sl_group_prices.rb` (uses system `pdftotext`; **no runtime PDF gem** — `pdf-reader`'s `afm` dep now needs Ruby ≥ 3.2, which would break the gem's Ruby floor). The join is **GTIN → csl → price**, honouring an `N x <code>` package multiplier (price = N × group price). Produces `gtin => {sl:, price:, csl:, abgabe:}` (SL rows only; ~515 priced on the live feed). **WALA products (3.0.22 onwards):** a third runtime CSV `wala_arzneimittel.csv` (GTIN prefix `7640187…`, `WalaDownloader`, bundled fallback) is merged into the same map via `WeledaSL.build_wala_map`. Its layout differs: `;`-separated with a BOM, no `/ SL` column (a row is SL when it carries a `CSL-Code` = Kapitel-70.01 group code), and the public **package** price is given inline in the `CSL 70.01.` column — **already multiplied for the pack size** (the multiplier appears only in the galenic-form text, e.g. `Solutio ad inj. 10 x 1 ml`), so it is taken **verbatim** rather than re-joined against `bag_sl_group_prices.csv` (which holds the per-unit price and would yield 1/10 of the package price for ~120 multi-unit packs). 320 WALA SL products on the live file; Weleda wins on the (unlikely) GTIN collision. `Builder#build_artikelstamm` consumes it (CLI sets `builder.weleda_sl` only for `--artikelstamm`): for any GTIN **absent from the FHIR NDJSON** it emits `<SL_ENTRY>true</SL_ENTRY>` and `<PPUB>` from the BAG group price, mirroring the old chapter-70 behaviour (`PHARMATYPE "P"`). **The FHIR/ZurRose price always wins** — the group price only fills a gap; a zeroed ZurRose `"0.00"` pub price is treated as absent so the gap-fill can apply. Match is **by GTIN only** (no pharmacode); the Swissmedic dispensing category is untouched (still from `Swissmedic_Packungen.xlsx`). The Artikelstamm output gets `<SL_ENTRY>` + `<PPUB>`; for the `-e`/`--extended` and `-b`/`--firstbase` product feeds the BAG public price is also added to `oddb_article.xml` as an `<ARTPRI><PTYP>BAGPUB</PTYP>` entry (the raw, often-blanked `ZURROSEPUB` is preserved alongside it) — `build_article`, gated by the CLI loading `weleda_sl` when `extended || firstbase || artikelstamm`. **Source encoding pinned (3.0.34):** `WeledaSL.source` forces the three CSVs to UTF-8. It used to pass the downloaded body on as-is, so an ASCII-8BIT response (webmock, or any body served without a charset) made the first comparison against a UTF-8 literal raise `Encoding::CompatibilityError` — and `load` rescues **every** error into `{}`, so the entire recovery vanished with no message: no `<SL_ENTRY>`, no group price, build still "successful". Same silent-total-loss shape as issues #122 and #127; if Weleda items lose their SL flag, check the `WeledaSL:` log lines with `--log` before suspecting the data. See GitHub issue #121.
 
 11. **Rogger name preferences** (`lib/oddb2xml/rogger_names.rb`, `-r`/`--rogger`, 3.0.30 onwards) — Replaces the German article description (`desc_de`) with the preferred name from the **"Rogger Mediliste"** for every GTIN on that list. The list collects the name conflicts Frau Rogger (Vitabyte/Zur Rose, task #OX-5985-1594) reports between the Spirig/Galexis names and what Vitabyte shows from raw Refdata; most entries are the issue-#112 Refdata bugs with their corrected names (METOJECT truncation, CETIRIZIN missing strength, ATOVAQUON missing 2nd combo dose, MOUNJARO missing pen volume, RINVOQ `Retardtabletten`), plus two not (yet) catalogued in `refdata_cleanup.rb`: GABAPENTIN Spirig HC Kaps (IKSNR 66594, strength doubled as a glued brand token `Spirig HC 100mg Kaps 100 mg`) and MELATONIN Spirig HC 100 Stk (redundant `retard` before `Ret Tabl`). Source of truth is the shared Google Sheet "Rogger Mediliste" (link-shared read-only); `RoggerDownloader` fetches its CSV export (`GTIN,Mediname`) **directly** (`docs.google.com/spreadsheets/d/<id>/export?format=csv&gid=0`), so sheet edits reach the feeds without any release step, with a bundled fallback `data/rogger_liste.csv` (refresh at release time). `RoggerNames.load` returns `gtin => name` (never raises; UTF-8-forced because the list carries `µg`; `rogger_csv?` rejects non-CSV responses such as a Google sign-in page if the sheet ever loses its link-sharing, engaging the fallback). The override runs as the **last step of `Builder#apply_refdata_description_cleanups!`** (`apply_rogger_name_overrides!`), so it sees and wins over the issue-#112 cleanups; the list is German-only, FR/IT descriptions are untouched. CLI wires `builder.rogger_names` only when `--rogger` is set (default off, all feeds unchanged without it).
 
@@ -50,7 +50,7 @@ These scripts run the public download server at `https://mediupdatexml.oddb.org`
 - **`generate_index_html.sh DOCROOT [FIRSTBASE_CSV]`** — single source of truth for the landing page. Writes `index.html` + a self-contained `logo.svg` **atomically** (temp + `mv`, so either owner — root from setup, `zdavatz` from cron — can refresh it). Computes live counts: PHARMA = `<SMNO>` count in `default/oddb_article.xml`, NONPHARMA = firstbase CSV rows − 1, total ART = `<ART ` count. Also runs **`visitor_stats.py`** and embeds its graph. Re-run standalone any time (it only reads already-built files); a separate cron line refreshes it **hourly** (`5 * * * * zdavatz`) so counts + graph stay current between nightly builds.
 - **`visitor_stats.py LOG_GLOB CACHE_DIR [DAYS]`** — emits the visitors/sessions/region graph as an inline-SVG HTML **fragment** (last `DAYS`, default 14): Besucher = distinct IPs/day, Sitzungen = 30-min-inactivity sessions per `(IP, User-Agent)`, plus a top-6 country breakdown by IP. Bots are filtered by User-Agent. Region lookup is **fully self-contained** — pure Python stdlib + the free **DB-IP country-lite CSV** (CC-BY, no licence key) cached in the build `downloads/` dir and refreshed monthly; **no apt package, no gem, no system GeoIP DB**. Prints nothing (page degrades to omitting the section) when the Apache log is unreadable or empty. Reading `/var/log/apache2` requires the cron user to be in the **`adm`** group (`sudo usermod -aG adm zdavatz`).
 - **`swissmedic_watch.sh`** — outage/block auto-recovery (cron: `*/30 * * * * zdavatz`). Since the Swissmedic platform migration (~2026-06-23, now a Swisscom-operated gateway), `www.swissmedic.ch` intermittently resets this host's automated connections **after the TLS handshake** (TCP RST), which aborts `run_oddb2xml.sh` under `set -e` and leaves the feeds stale (the block is host/IP- and client-fingerprint-sensitive: a real browser works, `curl`/`wget`/Ruby get reset, while other admin.ch hosts answer fine — so it is a WAF/bot rule, not an outage). The watcher polls Swissmedic with **oddb2xml's own client** (a Ruby `open-uri` canary on `listen_neu.html`); while blocked it is a silent no-op, and the moment it gets HTTP 200 it launches **one** build and emails. It fires **at most once per day** (stamp in `$STATE_DIR`, default `<OUT_DIR>-watch`, kept **outside** the wiped `$BUILD_DIR`), and skips when a build is already running or today's `default/oddb_article.xml` is already fresh. **PATH assembly (2026-07-28 onwards):** cron hands the watcher a minimal PATH, so it builds one explicitly — `/usr/local/bin` (where a root `gem install` under Debian's system Ruby puts the binstub) + the user gem bin dir (`$(ruby -e 'print Gem.user_dir')/bin`, where an *unprivileged* `gem install` silently lands instead, because `/var/lib/gems` is not writable) + rbenv shims **only when they exist** (the pre-2026-07 box; `RBENV_VERSION=3.4.5` is exported only in that case). The previous hardcoded rbenv-only PATH carried no `/usr/local/bin`, so the first watcher-triggered build on the rebuilt server died `oddb2xml: Kommando nicht gefunden` (exit 127) on all three retries before aborting. `run_oddb2xml.sh` step 1 extends PATH the same way right after its `gem install`, for the same reason.
-- **`transfer.sh`** — optional hand-off (scp) of `$OUT_DIR` to the HIN host; `SCP_DEST` is required-but-unset until the HIN host is known.
+- **`transfer.sh`** — optional hand-off (scp) of `$OUT_DIR` to the HIN host; `SCP_DEST` is required-but-unset until the HIN host is known. **Obsolete, not merely unwired (issue [#127](https://github.com/zdavatz/oddb2xml/issues/127)):** it is not in `/etc/cron.d/mediupdatexml` and `SCP_DEST` still has no default — and it should stay that way, because **HIN no longer hosts a copy at all**. Their page `www.hin.ch/de/services/mediupdate-xml.cfm#section_2` links *this* site's URLs directly (`default/`, `45/`, `50/`, `55/`, `aips2sqlite/`) and mentions `download.hin.ch` exactly **zero** times, so there is nothing left to scp. The trap is that the unfed legacy path `https://download.hin.ch/download/oddb2xml/` **still answers HTTP 200**, serving the build of **22.06.2026** (`GENERATED_BY="oddb2xml 3.0.25"`, `Last-Modified: Sun, 21 Jun 2026 23:01:34 GMT`) — the whole directory carries that one date. It looks authoritative, it 404s nothing, and that is how issue #127 was filed against us for data that was current here all along. When someone reports stale data, **check which host they fetched from before touching the build**: `mediupdatexml.oddb.org` is the source of truth. HIN has been asked to 301 or retire the old path. `README.md` and `QA.md` now point at `mediupdatexml.oddb.org` throughout. For twenty days one link on HIN's page was genuinely broken and it was **ours**: `/aips2sqlite/oddb2xml_swissmedic_sequences.csv` answered 404 — **not** because the server rebuild lost it, but because aips2sqlite hit **the same Refdata barcode bug as issue #122**: `ean_code.length()` on the missing `<DataCarrierIdentifier>` threw a `NullPointerException` out of four copied loops, and `generate_aips_fi` deleted the published CSV *before* regenerating it, so under `set -e` each crash left nothing rather than yesterday's file. Fixed 02.09.2026 in aips2sqlite `ad2fc58` (all four loops skip barcode-less articles; the CSV is now published by `mv` only after a run that passed a row-count check). **When Refdata breaks one of the two projects, check the other** — they parse the same file with the same assumption.
 - **`setup_new_server.sh`** (2026-07-28 onwards) — one-command rebuild of a bare Debian host into this download site, written after **HIN deleted the previous server**: everything that had accumulated by hand on that box, so a rebuild is not a day of archaeology. Root, idempotent. Installs the packages (`apache2` + `certbot`; **`ruby-full`** — Debian 13 ships Ruby 3.3, matching the repo `.ruby-version`, so the new host has **no rbenv**; `cron`, absent on a minimal image; `unzip`/`zip` for the ZurRose transfer and the build zip; `poppler-utils` for `pdftotext` in `tools/generate_bag_sl_group_prices.rb`; nokogiri/ffi build headers), adds `$RUN_USER` to **`adm`** (so `visitor_stats.py` may read `/var/log/apache2`), sets the home dir to **711** (so `www-data` can traverse to `$OUT_DIR` without listing it), `gem install oddb2xml`, creates the five directories (`$OUT_DIR`, `-build`, `-state`, `-watch` — the last two deliberately outside the nightly-wiped `-build` — and `$GET_TRANSFER_DIR`, symlinking the tracked `get_transfer.sh` into it), writes **`/etc/cron.d/mediupdatexml`** (the whole schedule in one re-creatable file rather than the old box's per-user crontab: `get_transfer` 00:30, nightly build 01:00, rust2xml Artikelstamm 03:00, aips2sqlite Fachinfos 04:30, landing page hourly at `:05`, Swissmedic watcher every 30 min) plus a logrotate rule for the cron logs, then hands over to `setup_aips2sqlite.sh` and `setup_mediupdatexml_web.sh`. Env: `RUN_USER`, `OUT_DIR`, `SKIP_WEB=1`, `SKIP_GEM=1`, `SKIP_AIPS=1`. **Deliberately out of scope** (gone with the old server, restored separately): an MTA for the cron `MAILTO`.
 - **`setup_mediupdatexml_web.sh`** — one-time root setup of the Apache vhost + initial page. Skips the Let's Encrypt step with a hint when the domain does not yet resolve in **public** DNS (checked via Cloudflare DoH, not the local resolver, which may hold a stale or negative cache).
 - **`setup_aips2sqlite.sh`** (2026-08-01 onwards) — provisions the **aips2sqlite** half of the site, restored after the server rebuild. The landing page links a whole `/aips2sqlite/` section (Fachinformationen as XML, the AmiKo `.db` files, `oddb2xml_swissmedic_sequences.csv`, `atc_codes_used_set.txt`); `setup_mediupdatexml_web.sh` writes the Apache alias `/aips2sqlite` → `<checkout>/jars/output`, but on the rebuilt host neither the JRE nor that directory existed, so **every link answered 403** — Apache falls back to the default `<Directory /> Require all denied` when the aliased path does not exist, so a *missing* target reads as *forbidden*, not *not found*. Root, idempotent: installs **`openjdk-21-jre-headless`** (the repo ships a prebuilt fat jar `jars/aips2sqlite.jar`, so a JRE suffices — no JDK/Gradle unless the jar is rebuilt; the only AWT user, `BarCode.java`/barcode4j, is off the Fachinfo path), clones the checkout if missing, creates `jars/output` + `jars/downloads` owned by `$RUN_USER`, and appends the 04:30 cron entry to `/etc/cron.d/mediupdatexml` if not already there (`SKIP_CRON=1` when called from `setup_new_server.sh`, which writes that line itself). Generating the data is a separate build-user step: `aips2sqlite/scripts/generate_aips_fi` (~1 h; `--lang=de --xml`, then `--lang=fr --xml --nodown`, then `--nodown --smsequence`, renaming `swiss_medic_sequences.csv` → `oddb2xml_swissmedic_sequences.csv`). `REFDATA_API_KEY` (developer.refdata.ch) is **optional** for this path: the Refdata Partner SOAP download it gates is consumed only by the Takeda partner export, and its failure is caught and logged, so the FI/sequences output is complete without it.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    oddb2xml (3.0.33)
+    oddb2xml (3.0.34)
       csv
       htmlentities
       httpi

data/History.txt

@@ -1,3 +1,7 @@
+=== 3.0.34 / 02.09.2026
+* Bugfix (WeledaSL): pin the encoding of the three runtime CSVs so the Kapitel-70 SL recovery cannot disappear without a trace. WeledaSL.source passed the downloaded content on in whatever encoding the downloader produced; when that is ASCII-8BIT -- any response served without a charset -- the first comparison against a UTF-8 literal raises Encoding::CompatibilityError, and WeledaSL.load rescues every error into an empty Hash. The whole Weleda/WALA recovery (~835 SL products, issue #121) then vanished silently: no <SL_ENTRY>, no BAG group price, and a build that still reported success. All three lists carry non-ASCII (umlauts in the product names, the "-" in the group-price table), so the live path was one charset-less response away from this; it has not happened yet only because raw.githubusercontent.com serves UTF-8. The encoding is now forced in source, as RoggerNames#parse already did, and the bundled fallback is read with an explicit encoding.
+* This was also the last failing spec: artikelstamm_spec.rb:86, the issue #121 regression test, failed because @weleda_sl was empty under webmock and the item was emitted as PHARMATYPE="N" without SL_ENTRY/PPUB. `rspec` is green again -- 1193 examples, 0 failures, 16 pending. Closes GitHub issue #104.
+
 === 3.0.33 / 26.08.2026
 * Bugfix (Refdata): skip PHARMA articles that carry no barcode instead of losing every single one of them. Refdata started publishing a handful of PHARMA articles without a <DataCarrierIdentifier> in August 2026 -- the Swiss Red Cross blood products under the collective registration 99999 (Erythrozytenkonzentrat & co), which have no GTIN at all. RefdataExtractor#to_hash called .size on that nil, and Cli#download_as rescues the resulting NoMethodError into an Oddb2xml.log call that prints nothing unless --log is given, so all 16'093 Refdata PHARMA articles silently vanished from the feeds while the build still reported success. On mediupdatexml.oddb.org the <SMNO> count in oddb_article.xml fell from ~15'300 to exactly 10'210 and stayed frozen there for twelve days -- the remainder came from the BAG FHIR SL feed -- and then to 50 once the BAG url move (3.0.32) took the FHIR feed away too. Refdata NONPHARMA was never affected, because the product-type check runs before the barcode is read. See GitHub issue #122.
 * The two rescued Refdata failures (download and extraction) now also print a warning to stderr. Issue #102 asked the build to continue past a Refdata error, and it still does, but a total loss of every Swissmedic-registered medicine must not look like a successful run in the nightly log.

data/QA.md

@@ -5,7 +5,8 @@
 * SL Preise werden am Anfang des Monats publiziert, jeweils immer am 1. Es kann auch vorkommen, dass das BAG die SL-Preise/Limitationen während dem Monat anpasst.
 * Fachinfos werden täglich publiziert.
 * Swissmedic-Codes erscheinen einmal pro Monat, normalerweise in der ersten Woche.
-* Die Daten unter [MEDIupdate XML](https://www.hin.ch/services/mediupdate-xml/) werden täglich generiert.
+* Die Daten werden täglich generiert und auf <https://mediupdatexml.oddb.org> publiziert.
+* Der Spiegel unter [MEDIupdate XML](https://www.hin.ch/de/services/mediupdate-xml.cfm) / `download.hin.ch` wird seit Juni 2026 nicht mehr nachgeführt, siehe [Issue #127](https://github.com/zdavatz/oddb2xml/issues/127).
 
 #### 2. Gibt es eine Spezifikation der XML Files? 
 * Ja, siehe [oddb2xml.xsd](https://github.com/zdavatz/oddb2xml/blob/master/oddb2xml.xsd)
@@ -21,8 +22,8 @@
 * Damit man _Registrations-_ und _Sequenznummer_ besser verstehen kann, muss man einmal das File [excel-version_zugelasseneverpackungen.xlsx](https://www.swissmedic.ch/dam/swissmedic/de/dokumente/listen/excel-version_zugelasseneverpackungen.xlsx.download.xlsx/excel-version_zugelasseneverpackungen.xlsx) öffnen und die ersten paar Spalten anschauen.
 
 #### 5. Was ist der Unterschied zwischen oddb_article.xml und oddb_product.xml
-* [oddb_article.xml](http://download.hin.ch/download/oddb2xml/oddb_article.xml) enhält alle Artikel. 
-* [oddb_product.xml](http://download.hin.ch/download/oddb2xml/oddb_product.xml) enthält nur die Produkte von der Swissmedic, also die Medikamente.
+* [oddb_article.xml](https://mediupdatexml.oddb.org/default/oddb_article.xml) enhält alle Artikel.
+* [oddb_product.xml](https://mediupdatexml.oddb.org/default/oddb_product.xml) enthält nur die Produkte von der Swissmedic, also die Medikamente.
 
 #### 6. Warum hat nicht jedes Produkt im oddb_article.xml einen GTIN?
 * Nicht alle Produkte haben zur Zeit einen GTIN. Dieser wird jedoch laufend ergänzt. Ab 1.1.2019 sollte der Pharmacode komplett verschwinden. Dies wurde von der Stiftung [Refdata](http://www.refdata.ch) auch so bestätigt.
@@ -36,7 +37,7 @@
 #### 8. Ich möchte gerne ein XML-File welches alle Produkte (Pharma und Non-Pharma) und die dazugehörigen Sequenznamen enthält. Gibt es das?
 * Ja. _oddb2xml_ mit der Option _-r_ laufen lassen, siehe [usage](https://github.com/zdavatz/oddb2xml#usage) - Option "_--artikelstamm_".
 * Dieses File wird zur Zeit nicht via [MEDIupdate XML](https://www.hin.ch/services/mediupdate-xml/) zum Download zur Verfügung gestellt. Es muss selber generiert werden mittels _oddb2xml -r_ 
-* [CSV](http://pillbox.oddb.org/artikelstamm_26042018_v5.csv) oder [XML](http://pillbox.oddb.org/artikelstamm_26042018_v5.xml) Beispiel-Download vom 26.4.2018.
+* Der täglich gebaute Artikelstamm steht unter <https://mediupdatexml.oddb.org/artikelstamm/> bereit: [v6 XML](https://mediupdatexml.oddb.org/artikelstamm/artikelstamm_v6.xml), [v6 CSV](https://mediupdatexml.oddb.org/artikelstamm/artikelstamm_v6.csv) und die [Legacy-Version v5](https://mediupdatexml.oddb.org/artikelstamm/artikelstamm_v5.xml).
 
 #### 9. Wie installiere ich _oddb2xml_?
 * Neuste, stabile Version von [Ruby](http://www.ruby-lang.org/de/) installieren.

data/README.md

@@ -46,7 +46,21 @@ Generating files for Elexis Artikelstamm is discussed in the [Readme for the Art
 
 ## usage
 
-HIN (http://hin.ch) creates daily the actual file. They can be downloaded from `https://download.hin.ch/download/oddb2xml`, e.g. using  `wget https://download.hin.ch/download/oddb2xml/oddb_article.xml`
+Ready-made files are built nightly and published on
+<https://mediupdatexml.oddb.org>, e.g. using
+`wget https://mediupdatexml.oddb.org/default/oddb_article.xml`. See
+[Deployment](#deployment-the-mediupdatexmloddborg-download-site) for what the
+site publishes and how it is generated.
+
+HIN's [MEDIupdate XML](https://www.hin.ch/de/services/mediupdate-xml.cfm#section_2)
+page links straight to those URLs — it does not host a copy of its own.
+
+The old mirror at `https://download.hin.ch/download/oddb2xml` is **not linked
+from anywhere any more** and has not been updated since June 2026 (it still
+serves the build of 22.06.2026, `GENERATED_BY="oddb2xml 3.0.25"`) — the job
+feeding it stopped with the old server. It still answers `200` with
+plausible-looking data, which is exactly what makes it dangerous. Don't use it,
+see [issue #127](https://github.com/zdavatz/oddb2xml/issues/127).
 
 see `--help`.
 
@@ -555,7 +569,12 @@ they are the reference setup for running oddb2xml unattended.
   cron entry that regenerates the Fachinformationen. Run as root; idempotent.
   Without it every `/aips2sqlite/` link answers 403, because Apache denies a
   path that does not exist.
-* `transfer.sh` — optional scp hand-off of the output tree.
+* `transfer.sh` — optional scp hand-off of the output tree to the HIN
+  download server. **Obsolete, and deliberately not wired up**: `SCP_DEST` has
+  no default and the script is not in `/etc/cron.d/mediupdatexml`. HIN links
+  this site's URLs directly, so there is nothing left to copy; the unfed mirror
+  it used to push to is
+  [issue #127](https://github.com/zdavatz/oddb2xml/issues/127).
 
 One Debian-specific pitfall worth knowing when running oddb2xml from cron: with
 the system Ruby, `gem install oddb2xml` as an unprivileged user cannot write
@@ -567,6 +586,15 @@ themselves.
 
 ## Testing
 
+On a host where the system gem directory is not writable (the mediupdatexml.oddb.org
+box, for instance), point bundler somewhere it may write, otherwise `bundle install`
+aborts while installing `rake`:
+
+```bash
+BUNDLE_PATH=/tmp/oddb2xml-bundle bundle install
+BUNDLE_PATH=/tmp/oddb2xml-bundle bundle exec rspec
+```
+
 * Calling rake spec runs spec tests.
 * Calling rake test installs the gems and runs oddb2xml with the most commonly used combinations. All output is placed under under ausgabe/<timestamp>. These files should be manually compared to the ones generated by the last release to check for possible problems.
 * we use the gem VCR to record real HTTP responses.

data/lib/oddb2xml/version.rb

@@ -1,3 +1,3 @@
 module Oddb2xml
-  VERSION = "3.0.33"
+  VERSION = "3.0.34"
 end

data/lib/oddb2xml/weleda_sl.rb

@@ -78,9 +78,17 @@ module Oddb2xml
         bundled = File.join(DATA_DIR, basename)
         if File.exist?(bundled)
           Oddb2xml.log "WeledaSL: using bundled #{basename}"
-          content = File.read(bundled)
+          content = File.read(bundled, encoding: "UTF-8")
         end
       end
+      # All three lists carry non-ASCII (umlauts in the product names, the "−"
+      # in the group-price table). A downloader that hands back ASCII-8BIT --
+      # webmock does, and so does any body served without a charset -- makes
+      # the first comparison against a UTF-8 literal raise
+      # Encoding::CompatibilityError, which load's rescue turns into an empty
+      # map: the whole Weleda/WALA SL recovery then vanishes silently. Pin the
+      # encoding here, as RoggerNames#parse does.
+      content = content.dup.force_encoding(Encoding::UTF_8) if content && content.encoding != Encoding::UTF_8
       content
     end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: oddb2xml
 version: !ruby/object:Gem::Version
-  version: 3.0.33
+  version: 3.0.34
 platform: ruby
 authors:
 - Yasuhiro Asaka, Zeno R.R. Davatz, Niklaus Giger