oddb2xml 3.0.28 → 3.0.29

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 62af20b2609637da38197880075560b49d0f810142ca162623980490abab4f6e
-  data.tar.gz: a5ac57d7d4864331ff34fa0f60f37ccf497c42b186f5d1a0a7ed0ee3322b3178
+  metadata.gz: 4602599ed50439225ff6ce76000f8441a9b4963b02538adf3f46d6728beff6cc
+  data.tar.gz: 55f34b946cc7a0e75435459aa90169e23719ac44c4bbfdf1ba3d63c3162f894d
 SHA512:
-  metadata.gz: 5bc3e6e4bb7bf221deb8feee038a59e723b491af3ebfa40a91d4c26d4a6fc9fc5c9417801ddac922b5a3be981310225b4b84e5f0c98a1daa06e2983251c80302
-  data.tar.gz: 434a73b313996af5a2f0ee11a7d404a9cd86bcd0c853abf913867fc0a0d7b85b1bf12dcdd5075f9574229d8c6f0d7b5610d2aa2c3062adde35a7f5bc303e086c
+  metadata.gz: cf98f9144a53cd62be8599168423c4c6d560b8474dc015d5281eccb84b2f80de6d46e3765b7ab96cc6271822361ea0a4a7b1f00bae1908e16f94f8fb77006908
+  data.tar.gz: 7ed225d5f4cd6f3c3293913cb40e01430839968d34e15551a2192a1f9069343dd98f7448179a0e433bc6485d8810510904add62632e6e5ddf6d03d50b8f41dc8

data/CLAUDE.md

@@ -68,7 +68,7 @@ YAML files in `data/` provide manual overrides and mappings: `article_overrides.
 
 These scripts run the public download server at `https://mediupdatexml.oddb.org` (Apache on this host) and are **not** part of the gem itself.
 
-- **`run_oddb2xml.sh`** — nightly build driver (cron: `0 1 * * * zdavatz`). Downloads the upstream sources **once**, then builds the `-b`/firstbase feed at price increments `45/50/55` plus `default` (no increment) into `$OUT_DIR` (`/home/zdavatz/oddb2xml`, one subdir each). The shared `downloads/` cache and transient zip live in `$BUILD_DIR` (`<OUT_DIR>-build`), **outside** `$OUT_DIR` so the transfer never uploads the multi-hundred-MB cache. Final step ("2b") regenerates the landing page. Each `oddb2xml` invocation is wrapped in `run_with_retry` (default **3 attempts, 120 s apart**, tunable via `ODDB2XML_RETRIES`/`ODDB2XML_RETRY_DELAY`): a transient upstream download failure (e.g. Swissmedic resetting the connection, `Errno::ECONNRESET`) previously aborted the whole `set -e` run 14 s in and left the feeds a day stale, so it now retries before giving up; a genuine repeated failure still stops the run.
+- **`run_oddb2xml.sh`** — nightly build driver (cron: `0 1 * * * zdavatz`). Downloads the upstream sources **once**, then builds the `-b`/firstbase feed at price increments `45/50/55` plus `default` (no increment) into `$OUT_DIR` (`/home/zdavatz/oddb2xml`, one subdir each). The shared `downloads/` cache and transient zip live in `$BUILD_DIR` (`<OUT_DIR>-build`), **outside** `$OUT_DIR` so the transfer never uploads the multi-hundred-MB cache. Final step ("2b") regenerates the landing page. Each `oddb2xml` invocation is wrapped in `run_with_retry` (default **3 attempts, 120 s apart**, tunable via `ODDB2XML_RETRIES`/`ODDB2XML_RETRY_DELAY`): a transient upstream download failure (e.g. Swissmedic resetting the connection, `Errno::ECONNRESET`) previously aborted the whole `set -e` run 14 s in and left the feeds a day stale, so it now retries before giving up; a genuine repeated failure still stops the run. **Firstbase (GS1 NONPHARMA) last-good fallback (3.0.29 onwards):** the GS1 `GetFirstbaseHealthcare` export (`id.gs1.ch/01/07612345000961` → `apitools.gs1.ch`) has been answering `403 - Forbidden`, which blanked `firstbase.csv` and dropped **every** NONPHARMA article from the `-b` feed (landing page then showed `NONPHARMA = 0 − 1 = −1`). The script keeps the last successful `firstbase.csv` in a persistent cache `$FIRSTBASE_CACHE` (default `<OUT_DIR>-state/firstbase.csv`) **outside** `$BUILD_DIR` so it survives the nightly `rm -rf`, seeds it into `downloads/` before the build, and refreshes it after a successful download. The gem side (`FirstbaseDownloader#download`, rewritten in 3.0.29) makes the seed usable: it still attempts the live GS1 fetch on the first (downloading) build, but only overwrites `firstbase.csv` when the response is a real non-empty CSV (`firstbase_csv?` rejects HTML/`403 - Forbidden`/empty bodies and open-uri exceptions), otherwise it **keeps the existing seeded file** instead of the old `"w+"` truncate-to-zero. A recovered GS1 therefore refreshes the data automatically; while GS1 is down the feed serves yesterday's (last-good) NONPHARMA rather than nothing. `generate_index_html.sh` also guards the NONPHARMA count so an empty CSV renders `—` (not `−1`).
 - **`generate_index_html.sh DOCROOT [FIRSTBASE_CSV]`** — single source of truth for the landing page. Writes `index.html` + a self-contained `logo.svg` **atomically** (temp + `mv`, so either owner — root from setup, `zdavatz` from cron — can refresh it). Computes live counts: PHARMA = `<SMNO>` count in `default/oddb_article.xml`, NONPHARMA = firstbase CSV rows − 1, total ART = `<ART ` count. Also runs **`visitor_stats.py`** and embeds its graph. Re-run standalone any time (it only reads already-built files); a separate cron line refreshes it **hourly** (`5 * * * * zdavatz`) so counts + graph stay current between nightly builds.
 - **`visitor_stats.py LOG_GLOB CACHE_DIR [DAYS]`** — emits the visitors/sessions/region graph as an inline-SVG HTML **fragment** (last `DAYS`, default 14): Besucher = distinct IPs/day, Sitzungen = 30-min-inactivity sessions per `(IP, User-Agent)`, plus a top-6 country breakdown by IP. Bots are filtered by User-Agent. Region lookup is **fully self-contained** — pure Python stdlib + the free **DB-IP country-lite CSV** (CC-BY, no licence key) cached in the build `downloads/` dir and refreshed monthly; **no apt package, no gem, no system GeoIP DB**. Prints nothing (page degrades to omitting the section) when the Apache log is unreadable or empty. Reading `/var/log/apache2` requires the cron user to be in the **`adm`** group (`sudo usermod -aG adm zdavatz`).
 - **`swissmedic_watch.sh`** — outage/block auto-recovery (cron: `*/30 * * * * zdavatz`). Since the Swissmedic platform migration (~2026-06-23, now a Swisscom-operated gateway), `www.swissmedic.ch` intermittently resets this host's automated connections **after the TLS handshake** (TCP RST), which aborts `run_oddb2xml.sh` under `set -e` and leaves the feeds stale (the block is host/IP- and client-fingerprint-sensitive: a real browser works, `curl`/`wget`/Ruby get reset, while other admin.ch hosts answer fine — so it is a WAF/bot rule, not an outage). The watcher polls Swissmedic with **oddb2xml's own client** (a Ruby `open-uri` canary on `listen_neu.html`); while blocked it is a silent no-op, and the moment it gets HTTP 200 it launches **one** build and emails. It fires **at most once per day** (stamp in `$STATE_DIR`, default `<OUT_DIR>-watch`, kept **outside** the wiped `$BUILD_DIR`), and skips when a build is already running or today's `default/oddb_article.xml` is already fresh. It exports `RBENV_VERSION=3.4.5` + the rbenv-shims PATH to match the nightly cron (the repo `.ruby-version` pins an uninstalled Ruby).

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    oddb2xml (3.0.28)
+    oddb2xml (3.0.29)
       csv
       htmlentities
       httpi

data/lib/oddb2xml/downloader.rb

@@ -396,14 +396,53 @@ module Oddb2xml
       @url = BASE_URL
     end
 
+    # A valid firstbase export is a non-empty CSV. When GS1 is unavailable it
+    # answers with an HTML error page (the GetFirstbaseHealthcare endpoint has
+    # been returning "403 - Forbidden: Access is denied") or open-uri raises.
+    # The old "w+" download truncated firstbase.csv to zero bytes on any such
+    # failure, silently dropping every NONPHARMA article. Reject non-CSV bodies
+    # so the caller can keep the previous good firstbase.csv instead.
+    def firstbase_csv?(text)
+      return false if text.nil?
+      head = text[0, 512].to_s.strip.downcase
+      return false if head.empty?
+      return false if head.start_with?("<!doctype", "<html", "<?xml")
+      return false if head.include?("403 - forbidden") || head.include?("access is denied")
+      true
+    end
+
     def download
       @file2save = File.join(DOWNLOADS, "firstbase.csv")
       report_download(@url, @file2save)
-      begin
-        download_as(@file2save, "w+")
+      # Price-increment / Artikelstamm runs (--skip-download) reuse the cached
+      # firstbase.csv. Do NOT skip merely because the file exists: the nightly
+      # deploy seeds a last-good copy so a GS1 outage does not blank the feed,
+      # and we still want a fresh download attempt on the first (downloading)
+      # build so a recovered GS1 refreshes the data.
+      if Oddb2xml.skip_download? && File.size?(@file2save)
+        Oddb2xml.log "FirstbaseDownloader: --skip-download, reusing cached #{@file2save} (#{File.size(@file2save)} bytes)"
         return File.expand_path(@file2save)
+      end
+      begin
+        data = Oddb2xml.uri_open(@url).read
+        if firstbase_csv?(data)
+          File.write(@file2save, data)
+          Oddb2xml.log "FirstbaseDownloader: fetched fresh firstbase.csv (#{data.bytesize} bytes)"
+        elsif File.size?(@file2save)
+          Oddb2xml.log "FirstbaseDownloader: GS1 returned no CSV (#{data.to_s.bytesize} bytes); keeping existing #{@file2save} (#{File.size(@file2save)} bytes)"
+        else
+          Oddb2xml.log "FirstbaseDownloader: GS1 returned no CSV and there is no cached firstbase.csv to fall back to"
+        end
       rescue Timeout::Error, Errno::ETIMEDOUT
         retrievable? ? retry : raise
+      rescue => error
+        # 403 / blocked / unreachable: keep any existing firstbase.csv (e.g. the
+        # last-good copy the deploy script seeds) rather than truncating it.
+        if File.size?(@file2save)
+          Oddb2xml.log "FirstbaseDownloader: download failed (#{error.class}: #{error}); keeping existing #{@file2save} (#{File.size(@file2save)} bytes)"
+        else
+          Oddb2xml.log "FirstbaseDownloader: download failed (#{error.class}: #{error}) and no cached firstbase.csv to fall back to"
+        end
       ensure
         Oddb2xml.download_finished(@file2save, false)
       end

data/lib/oddb2xml/version.rb

@@ -1,3 +1,3 @@
 module Oddb2xml
-  VERSION = "3.0.28"
+  VERSION = "3.0.29"
 end

data/scripts/generate_index_html.sh

@@ -35,7 +35,13 @@ pharma="—"
 [[ -f "$ARTICLE_XML" ]] && pharma=$(grep -c '<SMNO>' "$ARTICLE_XML" || true)
 
 nonpharma="—"
-[[ -f "$FIRSTBASE_CSV" ]] && nonpharma=$(( $(wc -l < "$FIRSTBASE_CSV") - 1 ))
+# Only count when the CSV actually has data rows. An empty firstbase.csv (the
+# GS1 firstbase dump upstream returns 403/empty from time to time) would make a
+# naive `rows - 1` render "-1"; keep the "—" fallback instead.
+if [[ -s "$FIRSTBASE_CSV" ]]; then
+  fb_rows=$(wc -l < "$FIRSTBASE_CSV")
+  (( fb_rows > 1 )) && nonpharma=$(( fb_rows - 1 ))
+fi
 
 stand=$(date '+%d.%m.%Y %H:%M')
 
@@ -150,7 +156,8 @@ cat > "$tmp" <<HTML
 
   <h2>Elexis Artikelstamm</h2>
   <ul>
-    <li><a href="artikelstamm/">artikelstamm/</a> <span class="desc">— Elexis Artikelstamm v6 (mit BAG-Indikationscodes) und Legacy-v5, je als XML und CSV, täglich aktualisiert</span></li>
+    <li><a href="artikelstamm/">artikelstamm/</a> <span class="desc">— Elexis Artikelstamm v6 (mit BAG-Indikationscodes) und Legacy-v5, je als XML und CSV, täglich aktualisiert (erzeugt mit oddb2xml)</span></li>
+    <li><a href="artikelstamm/rust2xml/">artikelstamm/rust2xml/</a> <span class="desc">— derselbe Artikelstamm v6 + v5, erzeugt mit <a href="https://github.com/zdavatz/rust2xml">rust2xml</a> (Rust-Port), täglich um 03:00 aus denselben Live-Quellen</span></li>
   </ul>
 
   <h2>aips2sqlite — Fachinformationen &amp; AmiKo-Datenbanken</h2>

data/scripts/run_oddb2xml.sh

@@ -81,6 +81,41 @@ rm -rf "$BUILD_DIR"
 mkdir -p "$BUILD_DIR"
 cd "$BUILD_DIR"
 
+# 3. Seed the ZurRose transfer.zip from the local get_transfer mirror.
+# get_transfer.sh (crontab 00:30) downloads transfer.dat straight from
+# zurrose.ch on THIS host and uploads the zip to pillbox.oddb.org — so the
+# pillbox HTTP fetch is a needless detour back to our own file. Placing the
+# zip in downloads/ makes oddb2xml's skip_download reuse it and the build no
+# longer depends on pillbox being up (2026-07-02: pillbox refused connections
+# during the 01:00 run and all three retries died, killing the whole nightly
+# build). If the seed file is missing, oddb2xml falls back to the normal
+# pillbox download as before.
+GET_TRANSFER_ZIP="${GET_TRANSFER_ZIP:-/home/zdavatz/software/get_transfer/TRANSFER.ZIP}"
+if [[ -s "$GET_TRANSFER_ZIP" ]]; then
+  mkdir -p "$BUILD_DIR/downloads"
+  cp -p "$GET_TRANSFER_ZIP" "$BUILD_DIR/downloads/transfer.zip"
+  log "Seeded ZurRose transfer.zip from $GET_TRANSFER_ZIP ($(date -r "$GET_TRANSFER_ZIP" '+%Y-%m-%d %H:%M'), no pillbox fetch needed)"
+else
+  log "WARNING: $GET_TRANSFER_ZIP missing - falling back to pillbox.oddb.org download"
+fi
+
+# 3b. Firstbase (GS1 NONPHARMA) fallback. The GS1 GetFirstbaseHealthcare
+# endpoint has been answering "403 - Forbidden", which blanked firstbase.csv and
+# dropped every NONPHARMA article from the -b feed. Keep the last successful
+# firstbase.csv in a persistent cache OUTSIDE $BUILD_DIR (it survives the nightly
+# `rm -rf`) and seed it into downloads/ so the gem's FirstbaseDownloader falls
+# back to yesterday's file when today's download fails. A recovered GS1 still
+# refreshes the data: the first (downloading) build always retries the live
+# fetch and only keeps the seed when that fetch yields no CSV.
+FIRSTBASE_CACHE="${FIRSTBASE_CACHE:-${OUT_DIR%/}-state/firstbase.csv}"
+if [[ -s "$FIRSTBASE_CACHE" ]]; then
+  mkdir -p "$BUILD_DIR/downloads"
+  cp -p "$FIRSTBASE_CACHE" "$BUILD_DIR/downloads/firstbase.csv"
+  log "Seeded firstbase.csv from last-good cache $FIRSTBASE_CACHE ($(($(wc -l < "$FIRSTBASE_CACHE") - 1)) rows, $(date -r "$FIRSTBASE_CACHE" '+%Y-%m-%d %H:%M'))"
+else
+  log "No firstbase last-good cache at $FIRSTBASE_CACHE yet - relying on live GS1 download"
+fi
+
 first=1
 
 # build_one <increment-percent|""> <destination-subdir>
@@ -132,17 +167,43 @@ build_artikelstamm() {
   shopt -u nullglob
   [[ ${#out[@]} -ge 1 ]] || { log "ERROR: no artikelstamm output produced"; exit 1; }
 
-  rm -rf "$dest"
   mkdir -p "$dest"
-  cp -p "${out[@]}" "$dest/"
+  # Remove only oddb2xml's own top-level files; keep sub-directories such as
+  # rust2xml/ (published independently by rust2xml's own cron at 03:00) intact.
+  # A plain `rm -rf "$dest"` used to wipe that sibling output every night.
+  rm -f "$dest"/artikelstamm_*.xml "$dest"/artikelstamm_*.csv
+  # Publish under date-less, stable names so the download URLs never change:
+  # artikelstamm_01072026_v6.xml -> artikelstamm_v6.xml (same for _v5 / .csv).
+  local f base
+  for f in "${out[@]}"; do
+    base="$(basename "$f" | sed -E 's/_[0-9]{8}_/_/')"
+    cp -p "$f" "$dest/$base"
+  done
   log "Staged ${#out[@]} file(s) to $dest"
 }
 
+# Build order: default first (it downloads the shared sources), then the
+# Artikelstamm right after so it is published early, then the price increments.
+build_one "" "default"           # first run: downloads sources, no increment
+
+# Refresh the last-good firstbase.csv cache after the downloading build. When
+# GS1 answered, downloads/firstbase.csv now holds fresh data; when it 403'd, the
+# gem kept the seeded copy - either way a non-empty file is worth caching so the
+# next run can fall back to it. An empty file means both today's download AND the
+# seed were missing, so leave the previous cache untouched.
+FIRSTBASE_LIVE="$BUILD_DIR/downloads/firstbase.csv"
+if [[ -s "$FIRSTBASE_LIVE" ]]; then
+  mkdir -p "$(dirname "$FIRSTBASE_CACHE")"
+  cp -p "$FIRSTBASE_LIVE" "$FIRSTBASE_CACHE"
+  log "Cached firstbase.csv as last-good ($(($(wc -l < "$FIRSTBASE_LIVE") - 1)) rows) -> $FIRSTBASE_CACHE"
+else
+  log "WARNING: firstbase.csv is empty after the build (GS1 403 and no cache) - NONPHARMA missing this run"
+fi
+
+build_artikelstamm               # Elexis Artikelstamm (v6 + legacy v5)
 for inc in $INCREMENTS; do
-  build_one "$inc" "$inc"
+  build_one "$inc" "$inc"        # price increments re-use the cached downloads/
 done
-build_one "" "default"           # final run with no increment
-build_artikelstamm               # Elexis Artikelstamm (v6 + legacy v5)
 
 # 2b. Refresh the download landing page with the live PHARMA/NONPHARMA counts
 #     (PHARMA from default/oddb_article.xml, NONPHARMA from the GS1 firstbase CSV).

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: oddb2xml
 version: !ruby/object:Gem::Version
-  version: 3.0.28
+  version: 3.0.29
 platform: ruby
 authors:
 - Yasuhiro Asaka, Zeno R.R. Davatz, Niklaus Giger