oddb2xml 3.0.29 → 3.0.30

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4602599ed50439225ff6ce76000f8441a9b4963b02538adf3f46d6728beff6cc
-  data.tar.gz: 55f34b946cc7a0e75435459aa90169e23719ac44c4bbfdf1ba3d63c3162f894d
+  metadata.gz: dd5aba5cd43be1930a5499b102b5a1921454d422de9aa13de227e92ad1b5cae6
+  data.tar.gz: 1861fc5dba9e46179aaa4f69e64fa6e10288aeafff677ec591156926a54e8633
 SHA512:
-  metadata.gz: cf98f9144a53cd62be8599168423c4c6d560b8474dc015d5281eccb84b2f80de6d46e3765b7ab96cc6271822361ea0a4a7b1f00bae1908e16f94f8fb77006908
-  data.tar.gz: 7ed225d5f4cd6f3c3293913cb40e01430839968d34e15551a2192a1f9069343dd98f7448179a0e433bc6485d8810510904add62632e6e5ddf6d03d50b8f41dc8
+  metadata.gz: eb67bef331b97a7f3b347d342c9c2c8128fe9d53154363b89d3868d97d61d2d6469568aa78fb597129f33687af68c2ef38ecb227a6e67d807b8e06df220e4694
+  data.tar.gz: b4835b7045e30b1e24a84afd96c499db2b09fe0ec0175ea8b8e82adfe9a7b4a096237e6be6879cb9636f9608a0161346dda37b1606abf38d62762d858375b056

data/CLAUDE.md

@@ -55,6 +55,8 @@ The system follows a **download → extract → build → compress** pipeline:
 
 10. **Weleda / Kapitel-70 SL recovery** (`lib/oddb2xml/weleda_sl.rb`, 3.0.21 onwards) — Recovers the SL flag and public price for chapter-70 complementary medicines that are **missing from the FHIR feed** (the partial-replacement gap left by the dead chapter_70_hack, issue #118/#121). Many are magistral Weleda preparations with a `7611916…` trade GTIN that arrive only via ZurRose — with no SL flag and a blanked Publikumspreis (issue #117). `WeledaSL.load` joins two CSVs (downloaded at runtime from `github.com/zdavatz/oddb2xml_files` via `WeledaDownloader` / `BagSlGroupPricesDownloader`, bundled fallback copies under `data/`): `weleda_arzneimittel.csv` (GTIN → `abgabekategorie` SL flag + `csl` = **Pharma-Gruppen-Code**) and `bag_sl_group_prices.csv` (Pharma-Gruppen-Code → public price). `weleda_arzneimittel.csv` is **regenerated** by the Rust tool `weleda_scraper/` in the `oddb2xml_files` repo (`scraper --update weleda`, prompts for the medical.weleda.ch `PHPSESSID` cookie — never stored): it walks the paginated Arzneimittel-Verzeichnis listing + per-product detail pages and rewrites the CSV with exactly the currently-listed products (delisted rows dropped), preserving the on-disk format (column order, UTF-8, CRLF, quote-when-necessary, sorted by `id`) so oddb2xml reads it unchanged. The price table is extracted **offline** from the BAG SL definition PDF *"Homoeopathica, Anthroposophica, Allergene"* via `tools/generate_bag_sl_group_prices.rb` (uses system `pdftotext`; **no runtime PDF gem** — `pdf-reader`'s `afm` dep now needs Ruby ≥ 3.2, which would break the gem's Ruby floor). The join is **GTIN → csl → price**, honouring an `N x <code>` package multiplier (price = N × group price). Produces `gtin => {sl:, price:, csl:, abgabe:}` (SL rows only; ~515 priced on the live feed). **WALA products (3.0.22 onwards):** a third runtime CSV `wala_arzneimittel.csv` (GTIN prefix `7640187…`, `WalaDownloader`, bundled fallback) is merged into the same map via `WeledaSL.build_wala_map`. Its layout differs: `;`-separated with a BOM, no `/ SL` column (a row is SL when it carries a `CSL-Code` = Kapitel-70.01 group code), and the public **package** price is given inline in the `CSL 70.01.` column — **already multiplied for the pack size** (the multiplier appears only in the galenic-form text, e.g. `Solutio ad inj. 10 x 1 ml`), so it is taken **verbatim** rather than re-joined against `bag_sl_group_prices.csv` (which holds the per-unit price and would yield 1/10 of the package price for ~120 multi-unit packs). 320 WALA SL products on the live file; Weleda wins on the (unlikely) GTIN collision. `Builder#build_artikelstamm` consumes it (CLI sets `builder.weleda_sl` only for `--artikelstamm`): for any GTIN **absent from the FHIR NDJSON** it emits `<SL_ENTRY>true</SL_ENTRY>` and `<PPUB>` from the BAG group price, mirroring the old chapter-70 behaviour (`PHARMATYPE "P"`). **The FHIR/ZurRose price always wins** — the group price only fills a gap; a zeroed ZurRose `"0.00"` pub price is treated as absent so the gap-fill can apply. Match is **by GTIN only** (no pharmacode); the Swissmedic dispensing category is untouched (still from `Swissmedic_Packungen.xlsx`). The Artikelstamm output gets `<SL_ENTRY>` + `<PPUB>`; for the `-e`/`--extended` and `-b`/`--firstbase` product feeds the BAG public price is also added to `oddb_article.xml` as an `<ARTPRI><PTYP>BAGPUB</PTYP>` entry (the raw, often-blanked `ZURROSEPUB` is preserved alongside it) — `build_article`, gated by the CLI loading `weleda_sl` when `extended || firstbase || artikelstamm`. See GitHub issue #121.
 
+11. **Rogger name preferences** (`lib/oddb2xml/rogger_names.rb`, `-r`/`--rogger`, 3.0.30 onwards) — Replaces the German article description (`desc_de`) with the preferred name from the **"Rogger Mediliste"** for every GTIN on that list. The list collects the name conflicts Frau Rogger (Vitabyte/Zur Rose, task #OX-5985-1594) reports between the Spirig/Galexis names and what Vitabyte shows from raw Refdata; most entries are the issue-#112 Refdata bugs with their corrected names (METOJECT truncation, CETIRIZIN missing strength, ATOVAQUON missing 2nd combo dose, MOUNJARO missing pen volume, RINVOQ `Retardtabletten`), plus two not (yet) catalogued in `refdata_cleanup.rb`: GABAPENTIN Spirig HC Kaps (IKSNR 66594, strength doubled as a glued brand token `Spirig HC 100mg Kaps 100 mg`) and MELATONIN Spirig HC 100 Stk (redundant `retard` before `Ret Tabl`). Source of truth is the shared Google Sheet "Rogger Mediliste" (link-shared read-only); `RoggerDownloader` fetches its CSV export (`GTIN,Mediname`) **directly** (`docs.google.com/spreadsheets/d/<id>/export?format=csv&gid=0`), so sheet edits reach the feeds without any release step, with a bundled fallback `data/rogger_liste.csv` (refresh at release time). `RoggerNames.load` returns `gtin => name` (never raises; UTF-8-forced because the list carries `µg`; `rogger_csv?` rejects non-CSV responses such as a Google sign-in page if the sheet ever loses its link-sharing, engaging the fallback). The override runs as the **last step of `Builder#apply_refdata_description_cleanups!`** (`apply_rogger_name_overrides!`), so it sees and wins over the issue-#112 cleanups; the list is German-only, FR/IT descriptions are untouched. CLI wires `builder.rogger_names` only when `--rogger` is set (default off, all feeds unchanged without it).
+
 ### Key data identifiers
 - **GTIN/EAN13**: Primary article identifier (13-digit barcode)
 - **Pharmacode**: Swiss pharmacy code
@@ -68,7 +70,7 @@ YAML files in `data/` provide manual overrides and mappings: `article_overrides.
 
 These scripts run the public download server at `https://mediupdatexml.oddb.org` (Apache on this host) and are **not** part of the gem itself.
 
-- **`run_oddb2xml.sh`** — nightly build driver (cron: `0 1 * * * zdavatz`). Downloads the upstream sources **once**, then builds the `-b`/firstbase feed at price increments `45/50/55` plus `default` (no increment) into `$OUT_DIR` (`/home/zdavatz/oddb2xml`, one subdir each). The shared `downloads/` cache and transient zip live in `$BUILD_DIR` (`<OUT_DIR>-build`), **outside** `$OUT_DIR` so the transfer never uploads the multi-hundred-MB cache. Final step ("2b") regenerates the landing page. Each `oddb2xml` invocation is wrapped in `run_with_retry` (default **3 attempts, 120 s apart**, tunable via `ODDB2XML_RETRIES`/`ODDB2XML_RETRY_DELAY`): a transient upstream download failure (e.g. Swissmedic resetting the connection, `Errno::ECONNRESET`) previously aborted the whole `set -e` run 14 s in and left the feeds a day stale, so it now retries before giving up; a genuine repeated failure still stops the run. **Firstbase (GS1 NONPHARMA) last-good fallback (3.0.29 onwards):** the GS1 `GetFirstbaseHealthcare` export (`id.gs1.ch/01/07612345000961` → `apitools.gs1.ch`) has been answering `403 - Forbidden`, which blanked `firstbase.csv` and dropped **every** NONPHARMA article from the `-b` feed (landing page then showed `NONPHARMA = 0 − 1 = −1`). The script keeps the last successful `firstbase.csv` in a persistent cache `$FIRSTBASE_CACHE` (default `<OUT_DIR>-state/firstbase.csv`) **outside** `$BUILD_DIR` so it survives the nightly `rm -rf`, seeds it into `downloads/` before the build, and refreshes it after a successful download. The gem side (`FirstbaseDownloader#download`, rewritten in 3.0.29) makes the seed usable: it still attempts the live GS1 fetch on the first (downloading) build, but only overwrites `firstbase.csv` when the response is a real non-empty CSV (`firstbase_csv?` rejects HTML/`403 - Forbidden`/empty bodies and open-uri exceptions), otherwise it **keeps the existing seeded file** instead of the old `"w+"` truncate-to-zero. A recovered GS1 therefore refreshes the data automatically; while GS1 is down the feed serves yesterday's (last-good) NONPHARMA rather than nothing. `generate_index_html.sh` also guards the NONPHARMA count so an empty CSV renders `—` (not `−1`).
+- **`run_oddb2xml.sh`** — nightly build driver (cron: `0 1 * * * zdavatz`). Downloads the upstream sources **once**, then builds the `-b`/firstbase feed at price increments `45/50/55` plus `default` (no increment) into `$OUT_DIR` (`/home/zdavatz/oddb2xml`, one subdir each). The shared `downloads/` cache and transient zip live in `$BUILD_DIR` (`<OUT_DIR>-build`), **outside** `$OUT_DIR` so the transfer never uploads the multi-hundred-MB cache. Final step ("2b") regenerates the landing page. Each `oddb2xml` invocation is wrapped in `run_with_retry` (default **3 attempts, 120 s apart**, tunable via `ODDB2XML_RETRIES`/`ODDB2XML_RETRY_DELAY`): a transient upstream download failure (e.g. Swissmedic resetting the connection, `Errno::ECONNRESET`) previously aborted the whole `set -e` run 14 s in and left the feeds a day stale, so it now retries before giving up; a genuine repeated failure still stops the run. **Firstbase (GS1 NONPHARMA) last-good fallback (3.0.29 onwards):** the GS1 `GetFirstbaseHealthcare` export (`id.gs1.ch/01/07612345000961` → `apitools.gs1.ch`) has been answering `403 - Forbidden`, which blanked `firstbase.csv` and dropped **every** NONPHARMA article from the `-b` feed (landing page then showed `NONPHARMA = 0 − 1 = −1`). The script keeps the last successful `firstbase.csv` in a persistent cache `$FIRSTBASE_CACHE` (default `<OUT_DIR>-state/firstbase.csv`) **outside** `$BUILD_DIR` so it survives the nightly `rm -rf`, seeds it into `downloads/` before the build, and refreshes it after a successful download. The gem side (`FirstbaseDownloader#download`, rewritten in 3.0.29) makes the seed usable: it still attempts the live GS1 fetch on the first (downloading) build, but only overwrites `firstbase.csv` when the response is a real non-empty CSV (`firstbase_csv?` rejects HTML/`403 - Forbidden`/empty bodies and open-uri exceptions), otherwise it **keeps the existing seeded file** instead of the old `"w+"` truncate-to-zero. A recovered GS1 therefore refreshes the data automatically; while GS1 is down the feed serves yesterday's (last-good) NONPHARMA rather than nothing. `generate_index_html.sh` also guards the NONPHARMA count so an empty CSV renders `—` (not `−1`). **ZurRose/pillbox last-good seed + first-build `--skip-download` (2026-07-07):** the actual reason the nightly feed went stale after 2026-07-03 was **not** firstbase (a firstbase failure is non-fatal) but **ZurRose**: `pillbox.oddb.org:80` started refusing connections around 01:00 every night from 2026-07-04 on, so `ZurroseDownloader` wrote a 0-byte `transfer.zip`, `unzip` produced no `transfer.dat`, and `downloader.rb:221` raised `Errno::ENOENT`, aborting the whole `set -e` run (all three retries hit the same dead pillbox). Fix is **script-only, no gem release**: `get_transfer.sh` (cron 00:30) already mirrors the ZurRose zip locally at `$GET_TRANSFER_ZIP` (`/home/zdavatz/software/get_transfer/TRANSFER.ZIP`), so the **first** `build_one` now runs a `first_build_attempt` helper that (a) `seed_downloads` — resets `downloads/` to contain only that seeded `transfer.zip` — then (b) runs `oddb2xml --skip-download -b -c zip`. The `--skip-download` flag makes `Cli#run` **skip the `FileUtils.rm_f(Dir.glob(DOWNLOADS/*))` wipe (cli.rb:50)** that used to delete the seed, so `Oddb2xml.skip_download` reuses the seeded `transfer.zip` for ZurRose (no pillbox fetch) while `download_as` still fetches **every other source fresh** (Swissmedic, Refdata, BAG, FHIR NDJSON, Weleda/WALA, and firstbase — proven: `--skip-download` only reuses files already present in `downloads/`, it does not suppress downloading absent ones). Seeding happens **per attempt** inside `run_with_retry`, so a retry restarts from a clean cache (clears partials) while keeping the seed. **Consequence for firstbase:** it is now downloaded **once** by the first build and reused by the `45/50/55`/artikelstamm increments via `--skip-download` — and it is **no longer pre-seeded into `downloads/`** from `$FIRSTBASE_CACHE` (under `--skip-download` a *present* `firstbase.csv` would be reused verbatim and never refreshed, freezing NONPHARMA). GS1's `id.gs1.ch` route currently returns a real 151 MB CSV, so firstbase fetches fresh; `$FIRSTBASE_CACHE` is still refreshed from the fresh file after the build for reference/recovery. The 3.0.29 gem-side `firstbase_csv?`/keep-existing logic is retained but only engages on the non-`--skip-download` path.
 - **`generate_index_html.sh DOCROOT [FIRSTBASE_CSV]`** — single source of truth for the landing page. Writes `index.html` + a self-contained `logo.svg` **atomically** (temp + `mv`, so either owner — root from setup, `zdavatz` from cron — can refresh it). Computes live counts: PHARMA = `<SMNO>` count in `default/oddb_article.xml`, NONPHARMA = firstbase CSV rows − 1, total ART = `<ART ` count. Also runs **`visitor_stats.py`** and embeds its graph. Re-run standalone any time (it only reads already-built files); a separate cron line refreshes it **hourly** (`5 * * * * zdavatz`) so counts + graph stay current between nightly builds.
 - **`visitor_stats.py LOG_GLOB CACHE_DIR [DAYS]`** — emits the visitors/sessions/region graph as an inline-SVG HTML **fragment** (last `DAYS`, default 14): Besucher = distinct IPs/day, Sitzungen = 30-min-inactivity sessions per `(IP, User-Agent)`, plus a top-6 country breakdown by IP. Bots are filtered by User-Agent. Region lookup is **fully self-contained** — pure Python stdlib + the free **DB-IP country-lite CSV** (CC-BY, no licence key) cached in the build `downloads/` dir and refreshed monthly; **no apt package, no gem, no system GeoIP DB**. Prints nothing (page degrades to omitting the section) when the Apache log is unreadable or empty. Reading `/var/log/apache2` requires the cron user to be in the **`adm`** group (`sudo usermod -aG adm zdavatz`).
 - **`swissmedic_watch.sh`** — outage/block auto-recovery (cron: `*/30 * * * * zdavatz`). Since the Swissmedic platform migration (~2026-06-23, now a Swisscom-operated gateway), `www.swissmedic.ch` intermittently resets this host's automated connections **after the TLS handshake** (TCP RST), which aborts `run_oddb2xml.sh` under `set -e` and leaves the feeds stale (the block is host/IP- and client-fingerprint-sensitive: a real browser works, `curl`/`wget`/Ruby get reset, while other admin.ch hosts answer fine — so it is a WAF/bot rule, not an outage). The watcher polls Swissmedic with **oddb2xml's own client** (a Ruby `open-uri` canary on `listen_neu.html`); while blocked it is a silent no-op, and the moment it gets HTTP 200 it launches **one** build and emails. It fires **at most once per day** (stamp in `$STATE_DIR`, default `<OUT_DIR>-watch`, kept **outside** the wiped `$BUILD_DIR`), and skips when a build is already running or today's `default/oddb_article.xml` is already fresh. It exports `RBENV_VERSION=3.4.5` + the rbenv-shims PATH to match the nightly cron (the repo `.ruby-version` pins an uninstalled Ruby).

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    oddb2xml (3.0.29)
+    oddb2xml (3.0.30)
       csv
       htmlentities
       httpi

data/History.txt

@@ -1,3 +1,9 @@
+=== 3.0.30 / 11.07.2026
+* New (-r/--rogger): prefer the German article names from the "Rogger Mediliste" (the name-conflict corrections Frau Rogger of Vitabyte/Zur Rose reports, task #OX-5985-1594). RoggerNames.load fetches the CSV export (GTIN,Mediname) of the shared Google Sheet directly at runtime -- sheet edits reach the feeds without any release step -- with a bundled fallback under data/ (a non-CSV response such as a Google sign-in page is rejected and falls back too). Builder#apply_rogger_name_overrides! replaces desc_de for every listed GTIN as the last step of the Refdata description cleanups, so the list wins over the issue-#112 fixes. German-only; FR/IT descriptions and all feeds without -r are unchanged. Initial list: 56 packs covering 13 conflict cases (METOJECT, ELTROXIN LF, CETIRIZIN/GABAPENTIN/ESOMEPRAZOL/EZETIMIB Spirig HC, ATOVAQUON PLUS, MOUNJARO KwikPen, MELATONIN, MONTELUKAST Mepha, FAMPYRA, RINVOQ).
+
+=== 3.0.29 / 07.07.2026
+* Bugfix (-b/--firstbase): keep the last good firstbase.csv when the GS1 GetFirstbaseHealthcare export answers 403 - Forbidden. FirstbaseDownloader only overwrites the cached CSV when the response is a real non-empty CSV (rejects HTML error pages and open-uri exceptions), so a seeded/last-good file survives a GS1 outage instead of being truncated to zero -- previously every NONPHARMA article vanished from the -b feed. See scripts/run_oddb2xml.sh for the server-side last-good cache.
+
 === 3.0.28 / 30.06.2026
 * New (--artikelstamm-v5): optionally emit the legacy Elexis Artikelstamm Version 5 file in addition to the v6 file. --artikelstamm now always produces the v6 output (artikelstamm_DDMMYYYY_v6.xml/.csv, with the per-ITEM <ARTSL> BAG indication codes since 3.0.26); passing --artikelstamm-v5 implies --artikelstamm and additionally writes artikelstamm_DDMMYYYY_v5.xml/.csv in the older v5 format (namespace http://elexis.ch/Elexis_Artikelstamm_v5, no <ARTSL> block), validated against the bundled Elexis_Artikelstamm_v5.xsd. The shared download/prepare phase runs once, so both files come from a single build. For consumers that have not yet migrated to v6.
 

data/README.md

@@ -81,6 +81,9 @@ see `--help`.
                                                           Forces -f dat -p zurrose.
     -o, --fi                  Optional fachinfo output.
     -p, --price               Price source (transfer.dat) from ZurRose
+    -r, --rogger              Prefer the German article names from the Rogger list
+                                                          (Vitabyte/Zur Rose name-conflict corrections, fetched
+                                                          directly from the shared 'Rogger Mediliste' Google Sheet)
     -t, --tag-suffix=<s>      XML tag suffix S. Default is none. [A-z0-9]
                                                           If S is given, it is also used as prefix of filename.
     -x, --context=<s>         {product|address}. product is default. (Default: product)
@@ -104,6 +107,7 @@ $ oddb2xml -f dat                       # => oddb.dat
 $ oddb2xml -f dat -a nonpharma          # => oddb_with_migel.dat
 $ oddb2xml -e                           # => oddb_article.xml (FHIR source by default since June 2026)
 $ oddb2xml -e --no-fhir                  # => oddb_article.xml using the old BAG-XML Spezialitätenliste
+$ oddb2xml -e -r                        # => oddb_article.xml with the Rogger-list article names
 ```
 
 output.

data/lib/oddb2xml/builder.rb

@@ -43,7 +43,7 @@ module Oddb2xml
     attr_accessor :subject, :refdata, :items, :flags, :lppvs,
       :actions, :migel, :orphan,
       :infos, :packs, :infos_zur_rose, :firstbase,
-      :weleda_sl,
+      :weleda_sl, :rogger_names,
       :ean14, :tag_suffix,
       :companies, :people,
       :xsd
@@ -59,6 +59,7 @@ module Oddb2xml
       @migel = {}
       @infos_zur_rose ||= {}
       @weleda_sl ||= {}
+      @rogger_names ||= {}
       @firstbase ||= {}
       @actions = []
       @orphan = []
@@ -138,6 +139,24 @@ module Oddb2xml
       counts.each do |rule, n|
         Oddb2xml.log("Refdata cleanup: fixed #{labels[rule]} in #{n} description(s)") if n > 0
       end
+      apply_rogger_name_overrides!
+    end
+
+    # -r/--rogger: replace the German description with the preferred name from
+    # the Rogger list (see Oddb2xml::RoggerNames) for every listed GTIN. Runs
+    # after the issue-#112 cleanups so the list always wins. The list is
+    # German-only, FR/IT descriptions are left untouched.
+    def apply_rogger_name_overrides!
+      return if @rogger_names.nil? || @rogger_names.empty?
+      count = 0
+      @refdata.each_value do |item|
+        next unless item.is_a?(Hash)
+        name = @rogger_names[item[:ean13].to_s.rjust(13, "0")]
+        next if name.nil? || name.empty? || item[:desc_de] == name
+        item[:desc_de] = name
+        count += 1
+      end
+      Oddb2xml.log("RoggerNames: overrode #{count} German description(s)") if count > 0
     end
 
     private_class_method

data/lib/oddb2xml/cli.rb

@@ -5,6 +5,7 @@ require "oddb2xml/compressor"
 require "oddb2xml/options"
 require "oddb2xml/proxy_check"
 require "oddb2xml/weleda_sl"
+require "oddb2xml/rogger_names"
 require "oddb2xml/util"
 require "rubyXL"
 require "date" # for today
@@ -168,6 +169,10 @@ module Oddb2xml
           else
             {}
           end
+          # -r/--rogger: preferred German article names from the Rogger list
+          # (Vitabyte/Zur Rose name-conflict corrections). Applied at the end
+          # of Builder#apply_refdata_description_cleanups!.
+          builder.rogger_names = @options[:rogger] ? Oddb2xml::RoggerNames.load(@options) : {}
         end
         builder.tag_suffix = @options[:tag_suffix]
       end

data/lib/oddb2xml/downloader.rb

@@ -203,6 +203,20 @@ module Oddb2xml
     end
   end
 
+  # "Rogger Mediliste": GTIN -> preferred German article name (Vitabyte/Zur
+  # Rose name-conflict corrections). Fetched directly as the CSV export of the
+  # shared Google Sheet (the source of truth Frau Rogger maintains), so edits
+  # there reach the feeds without any release step. The sheet must be shared
+  # as "anyone with the link can view". See Oddb2xml::RoggerNames.
+  class RoggerDownloader < Downloader
+    include DownloadMethod
+    ROGGER_SHEET_ID = "1NXJZ8KYzVsX0OQU767tl_AwCyvFVieHWnTWXqhflwdc"
+    def download
+      @url ||= "https://docs.google.com/spreadsheets/d/#{ROGGER_SHEET_ID}/export?format=csv&gid=0"
+      download_as("rogger_liste.csv", "w+")
+    end
+  end
+
   class ZurroseDownloader < Downloader
     include DownloadMethod
     def download

data/lib/oddb2xml/options.rb

@@ -36,6 +36,9 @@ module Oddb2xml
                             Forces -f dat -p zurrose.", type: :int, default: nil, short: "I"
         opt :fi, "Optional fachinfo output.", short: "o"
         opt :price, "Price source (transfer.dat) from ZurRose", default: nil
+        opt :rogger, "Prefer the German article names from the Rogger list
+                            (Vitabyte/Zur Rose name-conflict corrections, fetched
+                            directly from the shared 'Rogger Mediliste' Google Sheet)", short: "r", default: false
         opt :tag_suffix, "XML tag suffix S. Default is none. [A-z0-9]
                             If S is given, it is also used as prefix of filename.", type: :string, short: "t"
         opt :context, "{product|address}. product is default.", default: "product", type: :string, short: "x"

data/lib/oddb2xml/version.rb

@@ -1,3 +1,3 @@
 module Oddb2xml
-  VERSION = "3.0.29"
+  VERSION = "3.0.30"
 end

data/scripts/run_oddb2xml.sh

@@ -81,62 +81,70 @@ rm -rf "$BUILD_DIR"
 mkdir -p "$BUILD_DIR"
 cd "$BUILD_DIR"
 
-# 3. Seed the ZurRose transfer.zip from the local get_transfer mirror.
-# get_transfer.sh (crontab 00:30) downloads transfer.dat straight from
-# zurrose.ch on THIS host and uploads the zip to pillbox.oddb.org — so the
-# pillbox HTTP fetch is a needless detour back to our own file. Placing the
-# zip in downloads/ makes oddb2xml's skip_download reuse it and the build no
-# longer depends on pillbox being up (2026-07-02: pillbox refused connections
-# during the 01:00 run and all three retries died, killing the whole nightly
-# build). If the seed file is missing, oddb2xml falls back to the normal
-# pillbox download as before.
+# 3. ZurRose transfer.zip source. get_transfer.sh (crontab 00:30) downloads
+# transfer.dat straight from zurrose.ch on THIS host and mirrors the zip locally
+# (it also uploads it to http://pillbox.oddb.org/TRANSFER.ZIP). The build seeds
+# this local copy into a fresh downloads/ (see seed_downloads below) so the first
+# build reuses it via --skip-download instead of fetching pillbox — which refused
+# connections around 01:00 every night from 2026-07-04 on and aborted the whole
+# run at the ZurRose step (downloader.rb:221, transfer.dat ENOENT). Everything
+# else is still fetched fresh: --skip-download only reuses files already present
+# in downloads/ and downloads the rest.
 GET_TRANSFER_ZIP="${GET_TRANSFER_ZIP:-/home/zdavatz/software/get_transfer/TRANSFER.ZIP}"
-if [[ -s "$GET_TRANSFER_ZIP" ]]; then
-  mkdir -p "$BUILD_DIR/downloads"
-  cp -p "$GET_TRANSFER_ZIP" "$BUILD_DIR/downloads/transfer.zip"
-  log "Seeded ZurRose transfer.zip from $GET_TRANSFER_ZIP ($(date -r "$GET_TRANSFER_ZIP" '+%Y-%m-%d %H:%M'), no pillbox fetch needed)"
-else
-  log "WARNING: $GET_TRANSFER_ZIP missing - falling back to pillbox.oddb.org download"
-fi
-
-# 3b. Firstbase (GS1 NONPHARMA) fallback. The GS1 GetFirstbaseHealthcare
-# endpoint has been answering "403 - Forbidden", which blanked firstbase.csv and
-# dropped every NONPHARMA article from the -b feed. Keep the last successful
-# firstbase.csv in a persistent cache OUTSIDE $BUILD_DIR (it survives the nightly
-# `rm -rf`) and seed it into downloads/ so the gem's FirstbaseDownloader falls
-# back to yesterday's file when today's download fails. A recovered GS1 still
-# refreshes the data: the first (downloading) build always retries the live
-# fetch and only keeps the seed when that fetch yields no CSV.
+[[ -s "$GET_TRANSFER_ZIP" ]] || log "WARNING: $GET_TRANSFER_ZIP missing - ZurRose will fall back to pillbox.oddb.org"
+
+# 3b. Firstbase (GS1 NONPHARMA) last-good cache. Because the first build now runs
+# with --skip-download (to keep the ZurRose seed, see above), firstbase.csv must
+# NOT be pre-seeded into downloads/: under --skip-download a present firstbase.csv
+# would be reused verbatim and never refreshed from GS1. Instead firstbase is
+# fetched fresh every run (GS1's id.gs1.ch route works) and the fresh file is
+# archived to this persistent cache after the build for reference / recovery.
 FIRSTBASE_CACHE="${FIRSTBASE_CACHE:-${OUT_DIR%/}-state/firstbase.csv}"
-if [[ -s "$FIRSTBASE_CACHE" ]]; then
+
+# seed_downloads — reset downloads/ so it contains only the ZurRose transfer.zip
+# seed. A following --skip-download build reuses that zip (no pillbox fetch) and
+# downloads every other source fresh. Called before each attempt of the first
+# build, so a retry restarts from a clean cache (clearing any partial download)
+# while preserving the seed.
+seed_downloads() {
+  rm -rf "$BUILD_DIR/downloads"
   mkdir -p "$BUILD_DIR/downloads"
-  cp -p "$FIRSTBASE_CACHE" "$BUILD_DIR/downloads/firstbase.csv"
-  log "Seeded firstbase.csv from last-good cache $FIRSTBASE_CACHE ($(($(wc -l < "$FIRSTBASE_CACHE") - 1)) rows, $(date -r "$FIRSTBASE_CACHE" '+%Y-%m-%d %H:%M'))"
-else
-  log "No firstbase last-good cache at $FIRSTBASE_CACHE yet - relying on live GS1 download"
-fi
+  if [[ -s "$GET_TRANSFER_ZIP" ]]; then
+    cp -p "$GET_TRANSFER_ZIP" "$BUILD_DIR/downloads/transfer.zip"
+    log "Seeded ZurRose transfer.zip from $GET_TRANSFER_ZIP ($(date -r "$GET_TRANSFER_ZIP" '+%Y-%m-%d %H:%M'))"
+  fi
+}
+
+# first_build_attempt — seed downloads/, then run the first (downloading) build.
+# Wrapped in run_with_retry so each retry re-seeds and re-downloads cleanly.
+first_build_attempt() {
+  seed_downloads
+  "$ODDB2XML_BIN" --skip-download -b "$@" -c zip
+}
 
 first=1
 
 # build_one <increment-percent|""> <destination-subdir>
 build_one() {
   local inc="$1" name="$2" dest="$OUT_DIR/$2"
-  local inc_opt=() dl_opt=()
+  local inc_opt=()
   [[ -n "$inc" ]] && inc_opt=(-I "$inc")
 
+  log "Building increment '${inc:-none}' -> $dest"
+  rm -f oddb*.zip
   if [[ $first -eq 1 ]]; then
-    first=0                      # first build downloads the sources
+    first=0
+    # First build: seed the ZurRose zip into a clean downloads/, then fetch every
+    # other source fresh. Runs with --skip-download so cli.rb does not wipe
+    # downloads/ and the seeded transfer.zip survives (no pillbox fetch).
+    run_with_retry "oddb2xml build '${inc:-none}'" -- first_build_attempt "${inc_opt[@]}"
   else
-    dl_opt=(--skip-download)     # the rest re-use the cached downloads/
+    # Subsequent increments re-use the fully-populated downloads/ cache (firstbase
+    # and everything else were downloaded once by the first build).
+    run_with_retry "oddb2xml build '${inc:-none}'" -- \
+      "$ODDB2XML_BIN" --skip-download -b "${inc_opt[@]}" -c zip
   fi
 
-  log "Building increment '${inc:-none}' -> $dest"
-  rm -f oddb*.zip
-  # On a retry the first build re-downloads from scratch (dl_opt empty), which
-  # also clears any partial download left by the failed attempt.
-  run_with_retry "oddb2xml build '${inc:-none}'" -- \
-    "$ODDB2XML_BIN" "${dl_opt[@]}" -b "${inc_opt[@]}" -c zip
-
   shopt -s nullglob
   local zips=(oddb*.zip)
   shopt -u nullglob

data/spec/options_spec.rb

@@ -5,6 +5,8 @@ Oddb2xml::DEFAULT_OPTS = {
   fi: false,
   address: false,
   artikelstamm: false,
+  artikelstamm_v5: false,
+  rogger: false,
   nonpharma: false,
   extended: false,
   compress_ext: nil,
@@ -55,6 +57,18 @@ describe Oddb2xml::Options do
     specify { expect(test_opts).to eq expected }
   end
 
+  context "when -r is given" do
+    test_opts = Oddb2xml::Options.parse("-r")
+    expected = Oddb2xml::DEFAULT_OPTS.clone
+    expected[:rogger] = true
+    specify { expect(test_opts).to eq expected }
+  end
+
+  context "when --rogger is given" do
+    test_opts = Oddb2xml::Options.parse("--rogger")
+    specify { expect(test_opts[:rogger]).to eq true }
+  end
+
   context "when -a is given" do
     test_opts = Oddb2xml::Options.parse("-a")
     expected = Oddb2xml::DEFAULT_OPTS.clone

metadata

@@ -1,13 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: oddb2xml
 version: !ruby/object:Gem::Version
-  version: 3.0.29
+  version: 3.0.30
 platform: ruby
 authors:
 - Yasuhiro Asaka, Zeno R.R. Davatz, Niklaus Giger
+autorequire:
 bindir: bin
 cert_chain: []
-date: 1980-01-02 00:00:00.000000000 Z
+date: 2026-07-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rubyzip
@@ -590,6 +591,7 @@ homepage: https://github.com/zdavatz/oddb2xml
 licenses:
 - GPL-3.0-only
 metadata: {}
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -604,7 +606,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.9
+rubygems_version: 3.5.22
+signing_key:
 specification_version: 4
 summary: oddb2xml creates xml files.
 test_files: