oddb2xml 3.0.10 → 3.0.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 371c2ba25fb99bf0255bc0d336b0ae8a8a0cee64e9cd869e249e7fe7cd1d9da0
-  data.tar.gz: b85bd319a9179098af8c97bd6c1d8671a1e1cf656c0e83fdc7a0ede20ccf60fc
+  metadata.gz: 36ede269e855765b80eda67f9926c44dbc78c3eb4130b2e75ee780f635d0dbe3
+  data.tar.gz: 81ecf15485597e9a9eea503e63a7cc7aaf40dc3417f34ef2c3ca768092db9262
 SHA512:
-  metadata.gz: bab282fd20949a78607099dba3d64288cc63809b9e25f3e95015cd7f826a274a4010f9654117710976391561524c8f57b2c0727ed8b9ef7e8511f5fe9cd31c72
-  data.tar.gz: b4eef9e9aa441a567ac050c0ff27170ef8f9d64f8f9993b7c014ee9b7e79241ba6d5f917b956649cb194fcb59e8b06694db4d8e4500f113adb3f1212236d8430
+  metadata.gz: a305b713ab53201fa6c52d18db0446aa853df4f3986dbd53bcdb26cada8ad46770765800b7db5135bec1d0d8e9c58b4d793ca6f864dd203e5187912014a04103
+  data.tar.gz: 77abfc938a47bafc4579a429d4ff65dcace489566673754e4dedb74834fd0332f902c3aa80155b8a5aaf8140abc191fd6d4f7892092ab36ea541bf5ddff1bdcc

data/CLAUDE.md

@@ -51,6 +51,8 @@ The system follows a **download → extract → build → compress** pipeline:
 
 8. **Refdata cleanup** (`lib/oddb2xml/refdata_cleanup.rb`) — Compensates for known data-quality issues in upstream Refdata.Articles.xml before they reach the output. Each fix is guarded by a Swissmedic-side heuristic (e.g. comma in `substance_swissmedic` to distinguish mono products from real combinations). Currently fixes the doubled-dose template bug (`X mg / X mg / Stk`). Called from `Builder#apply_refdata_description_cleanups!` at the start of `prepare_articles`. See GitHub issue #112 for the catalogue.
 
+9. **Chapter-70 hack** (`lib/oddb2xml/chapter_70_hack.rb`) — Legacy scraper for the SL "Komplementärarzneimittel" products (homeopathic/anthroposophic/phytotherapeutic), called only from `Builder#build_artikelstamm`. **Deprecated / non-FHIR only (3.0.11 onwards):** the source page `varia_De.htm` was rebuilt as a JavaScript SPA with no static data table, so the scraper now returns nothing there. These products + limitations now come through the FHIR feed (SL classification `20. KOMPLEMENTÄRARZNEIMITTEL`, 221 products on the live DE feed with real GTINs and limitation texts), so `build_artikelstamm` **skips the scraper entirely when `@options[:fhir]`** (the default for `--artikelstamm` since 3.0.9). In `--no-fhir` mode the scraper degrades gracefully (skips non-row/`<script>` nodes and empty tables, warns, returns `[]`) instead of raising `NoMethodError`. See GitHub issue #118.
+
 ### Key data identifiers
 - **GTIN/EAN13**: Primary article identifier (13-digit barcode)
 - **Pharmacode**: Swiss pharmacy code

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    oddb2xml (3.0.10)
+    oddb2xml (3.0.11)
       csv
       htmlentities
       httpi

data/History.txt

@@ -1,3 +1,6 @@
+=== 3.0.11 / 02.06.2026
+* Artikelstamm: fix crash in the chapter-70 hack. The upstream source page http://www.spezialitaetenliste.ch/varia_De.htm has been rebuilt as a JavaScript single-page app and no longer ships the chapter-70 data table as static HTML, so Chapter70xtractor.parse raised NoMethodError. The chapter-70 (Komplementärarzneimittel) products and their limitations now arrive through the FHIR feed (SL classification "20. KOMPLEMENTÄRARZNEIMITTEL"), so the legacy scraper is now skipped entirely in FHIR mode (the default for --artikelstamm since 3.0.9). For --no-fhir runs the scraper degrades gracefully: it skips non-row nodes and an empty/missing table, logs a warning and returns no items instead of crashing (issue #118).
+
 === 3.0.10 / 01.06.2026
 * FHIR: read the BAG Indikationscode (XXXXX.NN) from the explicit `indicationCode` extension now carried on each limitation (BAG SL FHIR export >= v2.0.5) instead of reconstructing it from FOPHDossierNumber + the ClinicalUseDefinition id suffix. The changelog states the limitation code (CUD id) and the indication code are independent fields, so the old derivation was no longer guaranteed correct. Falls back to the previous derivation for older feeds that lack the extension. Output is identical on the current live feed. Other changelog items (limitation text moved to ClinicalUseDefinition, sanitized CUD ids, un-truncated `<` in texts, deduplicated/ordered Ingredients) were already handled or are transparent upstream fixes.
 

data/README.md

@@ -51,7 +51,7 @@ HIN (http://hin.ch) creates daily the actual file. They can be downloaded from `
 see `--help`.
 
 ```
-    /opt/src/oddb2xml/bin/oddb2xml version 3.0.10
+    /opt/src/oddb2xml/bin/oddb2xml version 3.0.11
     Usage:
     oddb2xml [option]
       produced files are found under data

data/lib/oddb2xml/builder.rb

@@ -1742,7 +1742,7 @@ module Oddb2xml
               # Set the pharmatype to 'Y' for outdated products, which are no longer found
               # in refdata/packungen
               chap70 = nil
-              if @chapter70items.values.find { |x| x[:pharmacode]&.eql?(obj[:pharmacode]) }
+              if @chapter70items&.values&.find { |x| x[:pharmacode]&.eql?(obj[:pharmacode]) }
                 Oddb2xml.log "found chapter #{obj[:pharmacode]}" if $VERBOSE
                 chap70 = true
               end
@@ -1804,9 +1804,17 @@ module Oddb2xml
         end
       end
       unless @prepared
-        require "oddb2xml/chapter_70_hack"
-        Oddb2xml::Chapter70xtractor.parse
-        @chapter70items = Oddb2xml::Chapter70xtractor.items
+        # The chapter-70 ("Komplementärarzneimittel") products and their
+        # limitations now arrive through the FHIR feed (SL classification
+        # "20. KOMPLEMENTÄRARZNEIMITTEL"), so the legacy varia_De.htm scraper
+        # is redundant in FHIR mode. The varia page itself has been rebuilt as
+        # a JavaScript SPA that no longer ships the data table, so the scraper
+        # can only fail there now. See GitHub issue #118.
+        unless @options[:fhir]
+          require "oddb2xml/chapter_70_hack"
+          Oddb2xml::Chapter70xtractor.parse
+          @chapter70items = Oddb2xml::Chapter70xtractor.items
+        end
         prepare_limitations
         prepare_articles
         prepare_products

data/lib/oddb2xml/chapter_70_hack.rb

@@ -39,12 +39,31 @@ module Oddb2xml
         effort: :tolerant,
         smart: true
       }
-      res = Ox.load(Oddb2xml.uri_open(html_file).read, mode: :hash_no_attrs).values.first["body"]
+      parsed = Ox.load(Oddb2xml.uri_open(html_file).read, mode: :hash_no_attrs)
+      res = parsed.values.first["body"] if parsed.respond_to?(:values) && parsed.values.first.is_a?(Hash)
       result = []
       idx = 0
       @@items = {}
-      res.values.last.each do |item|
-        item.values.first.each do |sub_elem|
+      unless res.respond_to?(:values)
+        warn "Chapter70: varia page has no <body> to parse (got #{res.class}); skipping"
+        return []
+      end
+      # The varia page used to expose the chapter-70 table as static HTML. It
+      # is now a JavaScript single-page app whose <body> only contains an empty
+      # <sl-root> shell, so there is no data table to walk. Each entry yielded
+      # by iterating a Hash is a [tag, content] pair; a real row carries a Hash
+      # of cells, while stray nodes (e.g. <script>) carry an Array. Skip the
+      # latter so a redesigned/empty page degrades to "no items" instead of
+      # raising NoMethodError. See GitHub issue #118.
+      rows = res.values.last
+      unless rows.respond_to?(:each)
+        warn "Chapter70: varia page has no parseable rows (got #{rows.class}); skipping"
+        return []
+      end
+      rows.each do |item|
+        cells = item.is_a?(Hash) ? item.values.first : nil
+        next unless cells.respond_to?(:each)
+        cells.each do |sub_elem|
           what = Chapter70xtractor.parse_td(sub_elem)
           idx += 1
           puts "#{idx}: xx #{what}" if $VERBOSE
@@ -52,6 +71,7 @@ module Oddb2xml
         end
       end
       result2 = result.find_all { |x| (x.is_a?(Array) && x.first.is_a?(String)) && x.first.to_i > 100 }
+      warn "Chapter70: varia page yielded no chapter-70 products; skipping" if result2.empty?
       result2.each do |entry|
         data = {}
         pharma_code = entry.first

data/lib/oddb2xml/version.rb

@@ -1,3 +1,3 @@
 module Oddb2xml
-  VERSION = "3.0.10"
+  VERSION = "3.0.11"
 end

data/spec/artikelstamm_spec.rb

@@ -38,7 +38,11 @@ describe Oddb2xml::Builder do
   context "when artikelstamm option is given" do
     before(:all) do
       common_run_init
-      options = Oddb2xml::Options.parse(["--artikelstamm"]) # , '--log'])
+      # --no-fhir: these fixtures (chapter-70 fake GTINs, BAG-XML preparations)
+      # exercise the legacy pipeline. Since 3.0.9 --artikelstamm defaults to the
+      # FHIR feed, which has no VCR cassette here and skips the chapter-70 hack
+      # (issue #118), so the suite must opt out to keep testing legacy output.
+      options = Oddb2xml::Options.parse(["--artikelstamm", "--no-fhir"]) # , '--log'])
       # @res = buildr_capture(:stdout){ Oddb2xml::Cli.new(options).run }
       Oddb2xml::Cli.new(options).run # to debug
       @artikelstamm_name = File.join(Oddb2xml::WORK_DIR, "artikelstamm_#{Date.today.strftime("%d%m%Y")}_v5.xml")
@@ -503,10 +507,24 @@ Der behandelnde Arzt ist verpflichtet, die erforderlichen Daten laufend im vorge
     end
     it "parsing" do
       require "oddb2xml/chapter_70_hack"
+      # Stub explicitly so this test is independent of example ordering
+      # (the SPA test below re-stubs the same URL).
+      url = "http://www.spezialitaetenliste.ch/varia_De.htm"
+      stub_request(:any, url).to_return(body: File.read(File.join(Oddb2xml::SpecData, "varia_De.htm")))
       result = Oddb2xml::Chapter70xtractor.parse
       expect(result.class).to eq Array
       expect(result.first).to eq ["2069562", "70.01.10", "Urtinktur", "1--10 g/ml", "13.40", ""]
       expect(result.last).to eq ["6516727", "70.02", "Allergenorum extractum varium / Inj. Susp. \n\tFortsetzungsbehandlung", "1 Durchstfl 1.5 ml", "311.85", "L"]
     end
+    it "degrades gracefully when varia page is a JavaScript SPA (issue #118)" do
+      require "oddb2xml/chapter_70_hack"
+      spa = File.read(File.join(Oddb2xml::SpecData, "varia_De_spa.htm"))
+      url = "http://www.spezialitaetenliste.ch/varia_De.htm"
+      stub_request(:any, url).to_return(body: spa)
+      result = nil
+      expect { result = Oddb2xml::Chapter70xtractor.parse }.not_to raise_error
+      expect(result).to eq []
+      expect(Oddb2xml::Chapter70xtractor.items).to eq({})
+    end
   end
 end

data/spec/data/varia_De_spa.htm

@@ -0,0 +1,16 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <title>BAG SL</title>
+  <base href="/">
+</head>
+<body>
+  <noscript><div><section><strong>JavaScript ist erforderlich!</strong></section></div></noscript>
+  <div><div><section><strong>Nicht unterstützter Browser!</strong></section></div></div>
+  <sl-root></sl-root>
+  <script src="polyfills-HE33VVSG.js" type="module"></script>
+  <script src="scripts-ALGZMJC4.js" defer></script>
+  <script src="main-DRAKRIKO.js" type="module"></script>
+</body>
+</html>

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: oddb2xml
 version: !ruby/object:Gem::Version
-  version: 3.0.10
+  version: 3.0.11
 platform: ruby
 authors:
 - Yasuhiro Asaka, Zeno R.R. Davatz, Niklaus Giger
@@ -547,6 +547,7 @@ files:
 - spec/data/v5_first.xml
 - spec/data/v5_second.xml
 - spec/data/varia_De.htm
+- spec/data/varia_De_spa.htm
 - spec/data/vcr/transfer.dat
 - spec/data/vcr/transfer.zip
 - spec/data/wsdl_nonpharma.xml
@@ -644,6 +645,7 @@ test_files:
 - spec/data/v5_first.xml
 - spec/data/v5_second.xml
 - spec/data/varia_De.htm
+- spec/data/varia_De_spa.htm
 - spec/data/vcr/transfer.dat
 - spec/data/vcr/transfer.zip
 - spec/data/wsdl_nonpharma.xml