oddb2xml 3.0.9 → 3.0.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 61b967bac848240cad4212eff14bd39587fc65cd1b7400b21a6a856df07dad99
-  data.tar.gz: a9e92c36306228a110b3c3cb28a8e6c66c59e6437273cc73b073a22572428b73
+  metadata.gz: 36ede269e855765b80eda67f9926c44dbc78c3eb4130b2e75ee780f635d0dbe3
+  data.tar.gz: 81ecf15485597e9a9eea503e63a7cc7aaf40dc3417f34ef2c3ca768092db9262
 SHA512:
-  metadata.gz: a109be8090b25dd4b6e150e50768f520dc38725c6486f126c6b6af2bad1e946c027894f251a2faf8620dc9e4c9e671a5a77b1171d713a16bcd2fefbf06cbb9b0
-  data.tar.gz: '006490b389a9f9ddce0e7fa90a8e11840068bfbd7671b685ea3f3d2e1c57f3413eb6e6fe1788c50cf8fbc685dcd18842de8470161f523446d9845ae4d273f899'
+  metadata.gz: a305b713ab53201fa6c52d18db0446aa853df4f3986dbd53bcdb26cada8ad46770765800b7db5135bec1d0d8e9c58b4d793ca6f864dd203e5187912014a04103
+  data.tar.gz: 77abfc938a47bafc4579a429d4ff65dcace489566673754e4dedb74834fd0332f902c3aa80155b8a5aaf8140abc191fd6d4f7892092ab36ea541bf5ddff1bdcc

data/CLAUDE.md

@@ -47,10 +47,12 @@ The system follows a **download → extract → build → compress** pipeline:
 
 6. **Compressor** (`lib/oddb2xml/compressor.rb`) — Optional ZIP/TAR.GZ output compression.
 
-7. **FHIR support** (`lib/oddb2xml/fhir_support.rb`) — Self-contained module providing `FhirDownloader` and FHIR NDJSON parsing. Activated via `--fhir` (or `--fhir-url=<URL>`). Downloads per-language NDJSON files (`foph-sl-export-latest-{de,fr,it}.ndjson`) from `epl.bag.admin.ch` to populate French and Italian product names/descriptions. Maps legal status codes `756005022007` and `756005022008` to Swissmedic category D. Parses `ClinicalUseDefinition` resources and combines each indication's `.NN` id-suffix with the reimbursement RA's `FOPHDossierNumber` to construct the BAG **Indikationscode** (`XXXXX.NN`); exposed as `item[:indication_codes]` and per-package `:indication_codes` (each entry a `{code:, cud_id:, text:}` hash). From 3.0.7 onwards, `Builder#build_product` emits one `<INDICATION_CODE code="XXXXX.NN" cud_id="DRUG.NN">limitation text</INDICATION_CODE>` child per indication on every `<PRD>` in `oddb_product.xml`; live feed numbers: 539 products / 1,293 codes / 100 % with non-empty indication text. Mandatory on prescriptions/invoices for SL price-model drugs from 2026-07-01 — see issue [#113](https://github.com/zdavatz/oddb2xml/issues/113). **Limitation texts** (3.0.8 onwards): the `regulatedAuthorization-limitation` extension has no inline `limitationText` in the live BAG feed — it carries a `limitationIndication` reference to a `ClinicalUseDefinition` whose `indication.diseaseSymptomProcedure.concept.text` is the actual text. The parser stores the ref as `cud_ref` on each Limitation, `Bundle#cud_text_by_id` resolves DE, and `merge_language` propagates FR/IT from the per-language NDJSON files via the same CUD id. Coverage on the live feed jumped from 0 / 9'108 to 9'108 / 9'108 (issue [#116](https://github.com/zdavatz/oddb2xml/issues/116)).
+7. **FHIR support** (`lib/oddb2xml/fhir_support.rb`) — Self-contained module providing `FhirDownloader` and FHIR NDJSON parsing. Activated via `--fhir` (or `--fhir-url=<URL>`). Downloads per-language NDJSON files (`foph-sl-export-latest-{de,fr,it}.ndjson`) from `epl.bag.admin.ch` to populate French and Italian product names/descriptions. Maps legal status codes `756005022007` and `756005022008` to Swissmedic category D. Reads the BAG **Indikationscode** (`XXXXX.NN`) from the explicit `indicationCode` extension on each `RegulatedAuthorization.indication[].extension[regulatedAuthorization-limitation]` (BAG SL FHIR export >= v2.0.5; handled from 3.0.10). The BAG changelog states the limitation code (`ClinicalUseDefinition.id`) and the indication code are **independent** fields, so the older derivation — combining each indication CUD's `.NN` id-suffix with the reimbursement RA's `FOPHDossierNumber` — is kept only as a fallback for feeds lacking the extension. Exposed as `item[:indication_codes]` and per-package `:indication_codes` (each entry a `{code:, cud_id:, text:}` hash, where `cud_id` is the `limitationIndication` CUD reference used to resolve the text). From 3.0.7 onwards, `Builder#build_product` emits one `<INDICATION_CODE code="XXXXX.NN" cud_id="DRUG.NN">limitation text</INDICATION_CODE>` child per indication on every `<PRD>` in `oddb_product.xml`; live feed numbers: 539 products / 1,293 codes / 100 % with non-empty indication text. Mandatory on prescriptions/invoices for SL price-model drugs from 2026-07-01 — see issue [#113](https://github.com/zdavatz/oddb2xml/issues/113). **Limitation texts** (3.0.8 onwards): the `regulatedAuthorization-limitation` extension has no inline `limitationText` in the live BAG feed — it carries a `limitationIndication` reference to a `ClinicalUseDefinition` whose `indication.diseaseSymptomProcedure.concept.text` is the actual text. The parser stores the ref as `cud_ref` on each Limitation, `Bundle#cud_text_by_id` resolves DE, and `merge_language` propagates FR/IT from the per-language NDJSON files via the same CUD id. Coverage on the live feed jumped from 0 / 9'108 to 9'108 / 9'108 (issue [#116](https://github.com/zdavatz/oddb2xml/issues/116)).
 
 8. **Refdata cleanup** (`lib/oddb2xml/refdata_cleanup.rb`) — Compensates for known data-quality issues in upstream Refdata.Articles.xml before they reach the output. Each fix is guarded by a Swissmedic-side heuristic (e.g. comma in `substance_swissmedic` to distinguish mono products from real combinations). Currently fixes the doubled-dose template bug (`X mg / X mg / Stk`). Called from `Builder#apply_refdata_description_cleanups!` at the start of `prepare_articles`. See GitHub issue #112 for the catalogue.
 
+9. **Chapter-70 hack** (`lib/oddb2xml/chapter_70_hack.rb`) — Legacy scraper for the SL "Komplementärarzneimittel" products (homeopathic/anthroposophic/phytotherapeutic), called only from `Builder#build_artikelstamm`. **Deprecated / non-FHIR only (3.0.11 onwards):** the source page `varia_De.htm` was rebuilt as a JavaScript SPA with no static data table, so the scraper now returns nothing there. These products + limitations now come through the FHIR feed (SL classification `20. KOMPLEMENTÄRARZNEIMITTEL`, 221 products on the live DE feed with real GTINs and limitation texts), so `build_artikelstamm` **skips the scraper entirely when `@options[:fhir]`** (the default for `--artikelstamm` since 3.0.9). In `--no-fhir` mode the scraper degrades gracefully (skips non-row/`<script>` nodes and empty tables, warns, returns `[]`) instead of raising `NoMethodError`. See GitHub issue #118.
+
 ### Key data identifiers
 - **GTIN/EAN13**: Primary article identifier (13-digit barcode)
 - **Pharmacode**: Swiss pharmacy code

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    oddb2xml (3.0.9)
+    oddb2xml (3.0.11)
       csv
       htmlentities
       httpi

data/History.txt

@@ -1,3 +1,9 @@
+=== 3.0.11 / 02.06.2026
+* Artikelstamm: fix crash in the chapter-70 hack. The upstream source page http://www.spezialitaetenliste.ch/varia_De.htm has been rebuilt as a JavaScript single-page app and no longer ships the chapter-70 data table as static HTML, so Chapter70xtractor.parse raised NoMethodError. The chapter-70 (Komplementärarzneimittel) products and their limitations now arrive through the FHIR feed (SL classification "20. KOMPLEMENTÄRARZNEIMITTEL"), so the legacy scraper is now skipped entirely in FHIR mode (the default for --artikelstamm since 3.0.9). For --no-fhir runs the scraper degrades gracefully: it skips non-row nodes and an empty/missing table, logs a warning and returns no items instead of crashing (issue #118).
+
+=== 3.0.10 / 01.06.2026
+* FHIR: read the BAG Indikationscode (XXXXX.NN) from the explicit `indicationCode` extension now carried on each limitation (BAG SL FHIR export >= v2.0.5) instead of reconstructing it from FOPHDossierNumber + the ClinicalUseDefinition id suffix. The changelog states the limitation code (CUD id) and the indication code are independent fields, so the old derivation was no longer guaranteed correct. Falls back to the previous derivation for older feeds that lack the extension. Output is identical on the current live feed. Other changelog items (limitation text moved to ClinicalUseDefinition, sanitized CUD ids, un-truncated `<` in texts, deduplicated/ordered Ingredients) were already handled or are transparent upstream fixes.
+
 === 3.0.9 / 01.06.2026
 * FHIR is now the default source for -e/--extended and -b/--firstbase. As of June 2026, running oddb2xml -e or oddb2xml -b downloads the FOPH/BAG FHIR NDJSON feed instead of the BAG-XML Spezialitätenliste. Plain runs are unchanged. Opt out with --no-fhir; explicit --fhir/--fhir-url still work.
 

data/README.md

@@ -51,7 +51,7 @@ HIN (http://hin.ch) creates daily the actual file. They can be downloaded from `
 see `--help`.
 
 ```
-    /opt/src/oddb2xml/bin/oddb2xml version 3.0.9
+    /opt/src/oddb2xml/bin/oddb2xml version 3.0.11
     Usage:
     oddb2xml [option]
       produced files are found under data
@@ -311,13 +311,18 @@ indication on each `<PRD>` in `oddb_product.xml`:
 </PRD>
 ```
 
-The code is built per FHIR bundle by combining the
-`FOPHDossierNumber` from the reimbursement `RegulatedAuthorization`
-(`XXXXX`) with the `.NN` suffix of each sibling
-`ClinicalUseDefinition` whose `type == "indication"` (e.g.
-`CYRAMZA.01` / `CYRAMZA.02` → `20403.01` / `20403.02`).  The element
-body carries the human-readable indication text from
-`indication.diseaseSymptomProcedure.concept.text`.
+Since 3.0.10 the code is read directly from the explicit
+`indicationCode` extension carried on each limitation
+(`RegulatedAuthorization.indication[].extension[regulatedAuthorization-limitation]`),
+introduced in the BAG SL FHIR export ≥ v2.0.5 (e.g. `20403.01` /
+`20403.02`). The BAG changelog states that the limitation code
+(`ClinicalUseDefinition.id`) and the indication code are **independent**
+fields, so the previous reconstruction from `FOPHDossierNumber` + the
+`.NN` CUD-id suffix is no longer used (it is kept only as a fallback for
+older feeds without the extension). `cud_id` still carries the
+`limitationIndication` reference, and the element body carries the
+human-readable indication text from the referenced
+`ClinicalUseDefinition`'s `indication.diseaseSymptomProcedure.concept.text`.
 
 The same data is also exposed in-memory on each item and package as
 `item[:indication_codes]` (an array of `{code:, cud_id:, text:}`

data/lib/oddb2xml/builder.rb

@@ -1742,7 +1742,7 @@ module Oddb2xml
               # Set the pharmatype to 'Y' for outdated products, which are no longer found
               # in refdata/packungen
               chap70 = nil
-              if @chapter70items.values.find { |x| x[:pharmacode]&.eql?(obj[:pharmacode]) }
+              if @chapter70items&.values&.find { |x| x[:pharmacode]&.eql?(obj[:pharmacode]) }
                 Oddb2xml.log "found chapter #{obj[:pharmacode]}" if $VERBOSE
                 chap70 = true
               end
@@ -1804,9 +1804,17 @@ module Oddb2xml
         end
       end
       unless @prepared
-        require "oddb2xml/chapter_70_hack"
-        Oddb2xml::Chapter70xtractor.parse
-        @chapter70items = Oddb2xml::Chapter70xtractor.items
+        # The chapter-70 ("Komplementärarzneimittel") products and their
+        # limitations now arrive through the FHIR feed (SL classification
+        # "20. KOMPLEMENTÄRARZNEIMITTEL"), so the legacy varia_De.htm scraper
+        # is redundant in FHIR mode. The varia page itself has been rebuilt as
+        # a JavaScript SPA that no longer ships the data table, so the scraper
+        # can only fail there now. See GitHub issue #118.
+        unless @options[:fhir]
+          require "oddb2xml/chapter_70_hack"
+          Oddb2xml::Chapter70xtractor.parse
+          @chapter70items = Oddb2xml::Chapter70xtractor.items
+        end
         prepare_limitations
         prepare_articles
         prepare_products

data/lib/oddb2xml/chapter_70_hack.rb

@@ -39,12 +39,31 @@ module Oddb2xml
         effort: :tolerant,
         smart: true
       }
-      res = Ox.load(Oddb2xml.uri_open(html_file).read, mode: :hash_no_attrs).values.first["body"]
+      parsed = Ox.load(Oddb2xml.uri_open(html_file).read, mode: :hash_no_attrs)
+      res = parsed.values.first["body"] if parsed.respond_to?(:values) && parsed.values.first.is_a?(Hash)
       result = []
       idx = 0
       @@items = {}
-      res.values.last.each do |item|
-        item.values.first.each do |sub_elem|
+      unless res.respond_to?(:values)
+        warn "Chapter70: varia page has no <body> to parse (got #{res.class}); skipping"
+        return []
+      end
+      # The varia page used to expose the chapter-70 table as static HTML. It
+      # is now a JavaScript single-page app whose <body> only contains an empty
+      # <sl-root> shell, so there is no data table to walk. Each entry yielded
+      # by iterating a Hash is a [tag, content] pair; a real row carries a Hash
+      # of cells, while stray nodes (e.g. <script>) carry an Array. Skip the
+      # latter so a redesigned/empty page degrades to "no items" instead of
+      # raising NoMethodError. See GitHub issue #118.
+      rows = res.values.last
+      unless rows.respond_to?(:each)
+        warn "Chapter70: varia page has no parseable rows (got #{rows.class}); skipping"
+        return []
+      end
+      rows.each do |item|
+        cells = item.is_a?(Hash) ? item.values.first : nil
+        next unless cells.respond_to?(:each)
+        cells.each do |sub_elem|
           what = Chapter70xtractor.parse_td(sub_elem)
           idx += 1
           puts "#{idx}: xx #{what}" if $VERBOSE
@@ -52,6 +71,7 @@ module Oddb2xml
         end
       end
       result2 = result.find_all { |x| (x.is_a?(Array) && x.first.is_a?(String)) && x.first.to_i > 100 }
+      warn "Chapter70: varia page yielded no chapter-70 products; skipping" if result2.empty?
       result2.each do |entry|
         data = {}
         pharma_code = entry.first

data/lib/oddb2xml/fhir_support.rb

@@ -395,6 +395,11 @@ module Oddb2xml
           when "limitationText"
             # Not present in the live FHIR feed — kept for forward-compat.
             limitation[:text] = sub_ext["valueString"]
+          when "indicationCode"
+            # Authoritative BAG Indikationscode XXXXX.NN (feed >= v2.0.5).
+            # Independent of the CUD id, so it must be read here rather than
+            # reconstructed from FOPHDossierNumber + CUD suffix.
+            limitation[:indication_code] = sub_ext["valueString"]
           when "limitationIndication"
             ref = sub_ext.dig("valueReference", "reference")
             limitation[:cud_ref] = ref&.sub(%r{\A.*ClinicalUseDefinition/}, "")
@@ -591,7 +596,30 @@ module Oddb2xml
 
       def build_indication_codes(bundle)
         reimbursement = bundle.authorizations.find(&:reimbursement_sl?)
-        dossier = reimbursement&.foph_dossier_no
+        return [] unless reimbursement
+
+        cud_texts = bundle.cud_text_by_id
+
+        # Preferred (BAG feed >= v2.0.5): read the explicit `indicationCode`
+        # carried on each limitation. The changelog warns that the limitation
+        # code (CUD id) and the indication code are independent, so we must NOT
+        # reconstruct XXXXX.NN from the CUD id suffix. Text is resolved via the
+        # limitationIndication reference (cud_ref) into the CUD's text.
+        from_ext = reimbursement.limitations.each_with_object([]) do |lim, acc|
+          code = lim[:indication_code]
+          next unless code && !code.empty?
+          cud_ref = lim[:cud_ref]
+          acc << OpenStruct.new(
+            code: code,
+            cud_id: cud_ref,
+            text: (cud_ref && cud_texts[cud_ref]) || lim[:text]
+          )
+        end
+        return from_ext unless from_ext.empty?
+
+        # Fallback for older feeds without the indicationCode extension:
+        # derive XXXXX.NN from FOPHDossierNumber + each indication CUD's suffix.
+        dossier = reimbursement.foph_dossier_no
         return [] unless dossier && !bundle.clinical_use_definitions.empty?
 
         bundle.clinical_use_definitions.each_with_object([]) do |cud, acc|

data/lib/oddb2xml/version.rb

@@ -1,3 +1,3 @@
 module Oddb2xml
-  VERSION = "3.0.9"
+  VERSION = "3.0.11"
 end

data/spec/artikelstamm_spec.rb

@@ -38,7 +38,11 @@ describe Oddb2xml::Builder do
   context "when artikelstamm option is given" do
     before(:all) do
       common_run_init
-      options = Oddb2xml::Options.parse(["--artikelstamm"]) # , '--log'])
+      # --no-fhir: these fixtures (chapter-70 fake GTINs, BAG-XML preparations)
+      # exercise the legacy pipeline. Since 3.0.9 --artikelstamm defaults to the
+      # FHIR feed, which has no VCR cassette here and skips the chapter-70 hack
+      # (issue #118), so the suite must opt out to keep testing legacy output.
+      options = Oddb2xml::Options.parse(["--artikelstamm", "--no-fhir"]) # , '--log'])
       # @res = buildr_capture(:stdout){ Oddb2xml::Cli.new(options).run }
       Oddb2xml::Cli.new(options).run # to debug
       @artikelstamm_name = File.join(Oddb2xml::WORK_DIR, "artikelstamm_#{Date.today.strftime("%d%m%Y")}_v5.xml")
@@ -503,10 +507,24 @@ Der behandelnde Arzt ist verpflichtet, die erforderlichen Daten laufend im vorge
     end
     it "parsing" do
       require "oddb2xml/chapter_70_hack"
+      # Stub explicitly so this test is independent of example ordering
+      # (the SPA test below re-stubs the same URL).
+      url = "http://www.spezialitaetenliste.ch/varia_De.htm"
+      stub_request(:any, url).to_return(body: File.read(File.join(Oddb2xml::SpecData, "varia_De.htm")))
       result = Oddb2xml::Chapter70xtractor.parse
       expect(result.class).to eq Array
       expect(result.first).to eq ["2069562", "70.01.10", "Urtinktur", "1--10 g/ml", "13.40", ""]
       expect(result.last).to eq ["6516727", "70.02", "Allergenorum extractum varium / Inj. Susp. \n\tFortsetzungsbehandlung", "1 Durchstfl 1.5 ml", "311.85", "L"]
     end
+    it "degrades gracefully when varia page is a JavaScript SPA (issue #118)" do
+      require "oddb2xml/chapter_70_hack"
+      spa = File.read(File.join(Oddb2xml::SpecData, "varia_De_spa.htm"))
+      url = "http://www.spezialitaetenliste.ch/varia_De.htm"
+      stub_request(:any, url).to_return(body: spa)
+      result = nil
+      expect { result = Oddb2xml::Chapter70xtractor.parse }.not_to raise_error
+      expect(result).to eq []
+      expect(Oddb2xml::Chapter70xtractor.items).to eq({})
+    end
   end
 end

data/spec/data/varia_De_spa.htm

@@ -0,0 +1,16 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <title>BAG SL</title>
+  <base href="/">
+</head>
+<body>
+  <noscript><div><section><strong>JavaScript ist erforderlich!</strong></section></div></noscript>
+  <div><div><section><strong>Nicht unterstützter Browser!</strong></section></div></div>
+  <sl-root></sl-root>
+  <script src="polyfills-HE33VVSG.js" type="module"></script>
+  <script src="scripts-ALGZMJC4.js" defer></script>
+  <script src="main-DRAKRIKO.js" type="module"></script>
+</body>
+</html>

data/spec/fhir_spec.rb

@@ -32,11 +32,46 @@ describe "FHIR Indikationscode support" do
   end
 
   describe Oddb2xml::FHIR::PreparationsParser do
-    it "constructs XXXXX.NN indication codes from FOPHDossierNumber + CUD suffix" do
+    it "reads the explicit indicationCode (XXXXX.NN) from each limitation" do
       parser = described_class.new(cyramza_fixture)
       prep = parser.preparations.first
       codes = prep.IndicationCodes.map(&:code)
       expect(codes).to include("20403.01", "20403.02")
+      # cud_id still carries the CUD reference so the text can be resolved.
+      expect(prep.IndicationCodes.map(&:cud_id)).to include("CYRAMZA.01", "CYRAMZA.02")
+    end
+
+    it "uses the explicit indicationCode field, not a dossier+CUD-suffix derivation" do
+      # The BAG changelog (>= v2.0.5) states the limitation code (CUD id) and
+      # the indication code are independent. Rewrite the explicit indicationCode
+      # values so they no longer correspond to FOPHDossierNumber + CUD suffix,
+      # and confirm the parser surfaces the explicit values verbatim.
+      bundle = JSON.parse(File.read(cyramza_fixture))
+      bundle["entry"].each do |entry|
+        res = entry["resource"]
+        next unless res["resourceType"] == "RegulatedAuthorization"
+        Array(res["indication"]).each do |ind|
+          Array(ind["extension"]).each do |ext|
+            next unless ext["url"].to_s.include?("regulatedAuthorization-limitation")
+            Array(ext["extension"]).each do |sub|
+              sub["valueString"] = "99999.77" if sub["url"] == "indicationCode"
+            end
+          end
+        end
+      end
+
+      file = Tempfile.new(["cyramza-indc", ".ndjson"])
+      begin
+        file.write(JSON.generate(bundle))
+        file.flush
+        parser = described_class.new(file.path)
+        codes = parser.preparations.first.IndicationCodes.map(&:code)
+        expect(codes).to all(eq("99999.77"))
+        expect(codes).not_to include("20403.01", "20403.02")
+      ensure
+        file.close
+        file.unlink
+      end
     end
   end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: oddb2xml
 version: !ruby/object:Gem::Version
-  version: 3.0.9
+  version: 3.0.11
 platform: ruby
 authors:
 - Yasuhiro Asaka, Zeno R.R. Davatz, Niklaus Giger
@@ -547,6 +547,7 @@ files:
 - spec/data/v5_first.xml
 - spec/data/v5_second.xml
 - spec/data/varia_De.htm
+- spec/data/varia_De_spa.htm
 - spec/data/vcr/transfer.dat
 - spec/data/vcr/transfer.zip
 - spec/data/wsdl_nonpharma.xml
@@ -644,6 +645,7 @@ test_files:
 - spec/data/v5_first.xml
 - spec/data/v5_second.xml
 - spec/data/varia_De.htm
+- spec/data/varia_De_spa.htm
 - spec/data/vcr/transfer.dat
 - spec/data/vcr/transfer.zip
 - spec/data/wsdl_nonpharma.xml