oddb2xml 3.0.33 → 3.0.35
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/workflows/ruby.yml +2 -2
- data/CLAUDE.md +5 -3
- data/Gemfile.lock +2 -30
- data/History.txt +9 -0
- data/QA.md +5 -4
- data/README.md +33 -4
- data/bin/oddb2xml +3 -0
- data/lib/oddb2xml/builder.rb +7 -2
- data/lib/oddb2xml/downloader.rb +0 -2
- data/lib/oddb2xml/util.rb +8 -0
- data/lib/oddb2xml/version.rb +1 -1
- data/lib/oddb2xml/weleda_sl.rb +9 -1
- data/oddb2xml.gemspec +1 -3
- data/spec/builder_spec.rb +13 -5
- data/spec/cli_spec.rb +3 -1
- data/spec/data/transfer.dat +1 -1
- data/spec/spec_helper.rb +9 -0
- metadata +2 -30
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 99321f1703f8ab1645b7cdb4049433c355330f95db7e45d8510c31cd15cf5bc7
|
|
4
|
+
data.tar.gz: 1c5c50119b700ac8591dc087b452cdcf06be4a35382e1b5e2280a7913cd6f5d6
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 98e7a7a8445bdad56f3d4d722c2243a068167d8242e874d37c4d0e4ad10c2ac14588a84c031e1c279948898602d416ba91be6e59c532e9637de80818a3bc4b06
|
|
7
|
+
data.tar.gz: 993e517ee499b12d448efb16a36f3dd72b4b1d758c7685a33b7cab6ae3b4f72bb0fd1f1ac9eb0aa2f58b7a3493432118e1221734a7b00b2976a3cc6be9ba2d1f
|
data/.github/workflows/ruby.yml
CHANGED
|
@@ -22,12 +22,12 @@ jobs:
|
|
|
22
22
|
fail-fast: false
|
|
23
23
|
matrix:
|
|
24
24
|
os: [ubuntu]
|
|
25
|
-
ruby: ["3.
|
|
25
|
+
ruby: ["3.2", "3.3"]
|
|
26
26
|
env:
|
|
27
27
|
BUNDLE_PATH: ".bundle/gems"
|
|
28
28
|
BUNDLE_WITHOUT: "debugger"
|
|
29
29
|
steps:
|
|
30
|
-
- uses: actions/checkout@
|
|
30
|
+
- uses: actions/checkout@v4
|
|
31
31
|
- uses: ruby/setup-ruby@v1
|
|
32
32
|
with:
|
|
33
33
|
ruby-version: ${{ matrix.ruby }}
|
data/CLAUDE.md
CHANGED
|
@@ -28,7 +28,7 @@ The system follows a **download → extract → build → compress** pipeline:
|
|
|
28
28
|
|
|
29
29
|
9. **Chapter-70 hack** (`lib/oddb2xml/chapter_70_hack.rb`) — Legacy scraper for the SL "Komplementärarzneimittel" products (homeopathic/anthroposophic/phytotherapeutic), called only from `Builder#build_artikelstamm`. **Deprecated / non-FHIR only (3.0.11 onwards):** the source page `varia_De.htm` was rebuilt as a JavaScript SPA with no static data table, so the scraper now returns nothing there. These products + limitations now come through the FHIR feed (SL classification `20. KOMPLEMENTÄRARZNEIMITTEL`, 221 products on the live DE feed with real GTINs and limitation texts), so `build_artikelstamm` **skips the scraper entirely when `@options[:fhir]`** (the default for `--artikelstamm` since 3.0.9). In `--no-fhir` mode the scraper degrades gracefully (skips non-row/`<script>` nodes and empty tables, warns, returns `[]`) instead of raising `NoMethodError`. See GitHub issue #118.
|
|
30
30
|
|
|
31
|
-
10. **Weleda / Kapitel-70 SL recovery** (`lib/oddb2xml/weleda_sl.rb`, 3.0.21 onwards) — Recovers the SL flag and public price for chapter-70 complementary medicines that are **missing from the FHIR feed** (the partial-replacement gap left by the dead chapter_70_hack, issue #118/#121). Many are magistral Weleda preparations with a `7611916…` trade GTIN that arrive only via ZurRose — with no SL flag and a blanked Publikumspreis (issue #117). `WeledaSL.load` joins two CSVs (downloaded at runtime from `github.com/zdavatz/oddb2xml_files` via `WeledaDownloader` / `BagSlGroupPricesDownloader`, bundled fallback copies under `data/`): `weleda_arzneimittel.csv` (GTIN → `abgabekategorie` SL flag + `csl` = **Pharma-Gruppen-Code**) and `bag_sl_group_prices.csv` (Pharma-Gruppen-Code → public price). `weleda_arzneimittel.csv` is **regenerated** by the Rust tool `weleda_scraper/` in the `oddb2xml_files` repo (`scraper --update weleda`, prompts for the medical.weleda.ch `PHPSESSID` cookie — never stored): it walks the paginated Arzneimittel-Verzeichnis listing + per-product detail pages and rewrites the CSV with exactly the currently-listed products (delisted rows dropped), preserving the on-disk format (column order, UTF-8, CRLF, quote-when-necessary, sorted by `id`) so oddb2xml reads it unchanged. The price table is extracted **offline** from the BAG SL definition PDF *"Homoeopathica, Anthroposophica, Allergene"* via `tools/generate_bag_sl_group_prices.rb` (uses system `pdftotext`; **no runtime PDF gem** — `pdf-reader`'s `afm` dep now needs Ruby ≥ 3.2, which would break the gem's Ruby floor). The join is **GTIN → csl → price**, honouring an `N x <code>` package multiplier (price = N × group price). Produces `gtin => {sl:, price:, csl:, abgabe:}` (SL rows only; ~515 priced on the live feed). **WALA products (3.0.22 onwards):** a third runtime CSV `wala_arzneimittel.csv` (GTIN prefix `7640187…`, `WalaDownloader`, bundled fallback) is merged into the same map via `WeledaSL.build_wala_map`. Its layout differs: `;`-separated with a BOM, no `/ SL` column (a row is SL when it carries a `CSL-Code` = Kapitel-70.01 group code), and the public **package** price is given inline in the `CSL 70.01.` column — **already multiplied for the pack size** (the multiplier appears only in the galenic-form text, e.g. `Solutio ad inj. 10 x 1 ml`), so it is taken **verbatim** rather than re-joined against `bag_sl_group_prices.csv` (which holds the per-unit price and would yield 1/10 of the package price for ~120 multi-unit packs). 320 WALA SL products on the live file; Weleda wins on the (unlikely) GTIN collision. `Builder#build_artikelstamm` consumes it (CLI sets `builder.weleda_sl` only for `--artikelstamm`): for any GTIN **absent from the FHIR NDJSON** it emits `<SL_ENTRY>true</SL_ENTRY>` and `<PPUB>` from the BAG group price, mirroring the old chapter-70 behaviour (`PHARMATYPE "P"`). **The FHIR/ZurRose price always wins** — the group price only fills a gap; a zeroed ZurRose `"0.00"` pub price is treated as absent so the gap-fill can apply. Match is **by GTIN only** (no pharmacode); the Swissmedic dispensing category is untouched (still from `Swissmedic_Packungen.xlsx`). The Artikelstamm output gets `<SL_ENTRY>` + `<PPUB>`; for the `-e`/`--extended` and `-b`/`--firstbase` product feeds the BAG public price is also added to `oddb_article.xml` as an `<ARTPRI><PTYP>BAGPUB</PTYP>` entry (the raw, often-blanked `ZURROSEPUB` is preserved alongside it) — `build_article`, gated by the CLI loading `weleda_sl` when `extended || firstbase || artikelstamm`. See GitHub issue #121.
|
|
31
|
+
10. **Weleda / Kapitel-70 SL recovery** (`lib/oddb2xml/weleda_sl.rb`, 3.0.21 onwards) — Recovers the SL flag and public price for chapter-70 complementary medicines that are **missing from the FHIR feed** (the partial-replacement gap left by the dead chapter_70_hack, issue #118/#121). Many are magistral Weleda preparations with a `7611916…` trade GTIN that arrive only via ZurRose — with no SL flag and a blanked Publikumspreis (issue #117). `WeledaSL.load` joins two CSVs (downloaded at runtime from `github.com/zdavatz/oddb2xml_files` via `WeledaDownloader` / `BagSlGroupPricesDownloader`, bundled fallback copies under `data/`): `weleda_arzneimittel.csv` (GTIN → `abgabekategorie` SL flag + `csl` = **Pharma-Gruppen-Code**) and `bag_sl_group_prices.csv` (Pharma-Gruppen-Code → public price). `weleda_arzneimittel.csv` is **regenerated** by the Rust tool `weleda_scraper/` in the `oddb2xml_files` repo (`scraper --update weleda`, prompts for the medical.weleda.ch `PHPSESSID` cookie — never stored): it walks the paginated Arzneimittel-Verzeichnis listing + per-product detail pages and rewrites the CSV with exactly the currently-listed products (delisted rows dropped), preserving the on-disk format (column order, UTF-8, CRLF, quote-when-necessary, sorted by `id`) so oddb2xml reads it unchanged. The price table is extracted **offline** from the BAG SL definition PDF *"Homoeopathica, Anthroposophica, Allergene"* via `tools/generate_bag_sl_group_prices.rb` (uses system `pdftotext`; **no runtime PDF gem** — `pdf-reader`'s `afm` dep now needs Ruby ≥ 3.2, which would break the gem's Ruby floor). The join is **GTIN → csl → price**, honouring an `N x <code>` package multiplier (price = N × group price). Produces `gtin => {sl:, price:, csl:, abgabe:}` (SL rows only; ~515 priced on the live feed). **WALA products (3.0.22 onwards):** a third runtime CSV `wala_arzneimittel.csv` (GTIN prefix `7640187…`, `WalaDownloader`, bundled fallback) is merged into the same map via `WeledaSL.build_wala_map`. Its layout differs: `;`-separated with a BOM, no `/ SL` column (a row is SL when it carries a `CSL-Code` = Kapitel-70.01 group code), and the public **package** price is given inline in the `CSL 70.01.` column — **already multiplied for the pack size** (the multiplier appears only in the galenic-form text, e.g. `Solutio ad inj. 10 x 1 ml`), so it is taken **verbatim** rather than re-joined against `bag_sl_group_prices.csv` (which holds the per-unit price and would yield 1/10 of the package price for ~120 multi-unit packs). 320 WALA SL products on the live file; Weleda wins on the (unlikely) GTIN collision. `Builder#build_artikelstamm` consumes it (CLI sets `builder.weleda_sl` only for `--artikelstamm`): for any GTIN **absent from the FHIR NDJSON** it emits `<SL_ENTRY>true</SL_ENTRY>` and `<PPUB>` from the BAG group price, mirroring the old chapter-70 behaviour (`PHARMATYPE "P"`). **The FHIR/ZurRose price always wins** — the group price only fills a gap; a zeroed ZurRose `"0.00"` pub price is treated as absent so the gap-fill can apply. Match is **by GTIN only** (no pharmacode); the Swissmedic dispensing category is untouched (still from `Swissmedic_Packungen.xlsx`). The Artikelstamm output gets `<SL_ENTRY>` + `<PPUB>`; for the `-e`/`--extended` and `-b`/`--firstbase` product feeds the BAG public price is also added to `oddb_article.xml` as an `<ARTPRI><PTYP>BAGPUB</PTYP>` entry (the raw, often-blanked `ZURROSEPUB` is preserved alongside it) — `build_article`, gated by the CLI loading `weleda_sl` when `extended || firstbase || artikelstamm`. **Source encoding pinned (3.0.34):** `WeledaSL.source` forces the three CSVs to UTF-8. It used to pass the downloaded body on as-is, so an ASCII-8BIT response (webmock, or any body served without a charset) made the first comparison against a UTF-8 literal raise `Encoding::CompatibilityError` — and `load` rescues **every** error into `{}`, so the entire recovery vanished with no message: no `<SL_ENTRY>`, no group price, build still "successful". Same silent-total-loss shape as issues #122 and #127; if Weleda items lose their SL flag, check the `WeledaSL:` log lines with `--log` before suspecting the data. See GitHub issue #121.
|
|
32
32
|
|
|
33
33
|
11. **Rogger name preferences** (`lib/oddb2xml/rogger_names.rb`, `-r`/`--rogger`, 3.0.30 onwards) — Replaces the German article description (`desc_de`) with the preferred name from the **"Rogger Mediliste"** for every GTIN on that list. The list collects the name conflicts Frau Rogger (Vitabyte/Zur Rose, task #OX-5985-1594) reports between the Spirig/Galexis names and what Vitabyte shows from raw Refdata; most entries are the issue-#112 Refdata bugs with their corrected names (METOJECT truncation, CETIRIZIN missing strength, ATOVAQUON missing 2nd combo dose, MOUNJARO missing pen volume, RINVOQ `Retardtabletten`), plus two not (yet) catalogued in `refdata_cleanup.rb`: GABAPENTIN Spirig HC Kaps (IKSNR 66594, strength doubled as a glued brand token `Spirig HC 100mg Kaps 100 mg`) and MELATONIN Spirig HC 100 Stk (redundant `retard` before `Ret Tabl`). Source of truth is the shared Google Sheet "Rogger Mediliste" (link-shared read-only); `RoggerDownloader` fetches its CSV export (`GTIN,Mediname`) **directly** (`docs.google.com/spreadsheets/d/<id>/export?format=csv&gid=0`), so sheet edits reach the feeds without any release step, with a bundled fallback `data/rogger_liste.csv` (refresh at release time). `RoggerNames.load` returns `gtin => name` (never raises; UTF-8-forced because the list carries `µg`; `rogger_csv?` rejects non-CSV responses such as a Google sign-in page if the sheet ever loses its link-sharing, engaging the fallback). The override runs as the **last step of `Builder#apply_refdata_description_cleanups!`** (`apply_rogger_name_overrides!`), so it sees and wins over the issue-#112 cleanups; the list is German-only, FR/IT descriptions are untouched. CLI wires `builder.rogger_names` only when `--rogger` is set (default off, all feeds unchanged without it).
|
|
34
34
|
|
|
@@ -50,7 +50,7 @@ These scripts run the public download server at `https://mediupdatexml.oddb.org`
|
|
|
50
50
|
- **`generate_index_html.sh DOCROOT [FIRSTBASE_CSV]`** — single source of truth for the landing page. Writes `index.html` + a self-contained `logo.svg` **atomically** (temp + `mv`, so either owner — root from setup, `zdavatz` from cron — can refresh it). Computes live counts: PHARMA = `<SMNO>` count in `default/oddb_article.xml`, NONPHARMA = firstbase CSV rows − 1, total ART = `<ART ` count. Also runs **`visitor_stats.py`** and embeds its graph. Re-run standalone any time (it only reads already-built files); a separate cron line refreshes it **hourly** (`5 * * * * zdavatz`) so counts + graph stay current between nightly builds.
|
|
51
51
|
- **`visitor_stats.py LOG_GLOB CACHE_DIR [DAYS]`** — emits the visitors/sessions/region graph as an inline-SVG HTML **fragment** (last `DAYS`, default 14): Besucher = distinct IPs/day, Sitzungen = 30-min-inactivity sessions per `(IP, User-Agent)`, plus a top-6 country breakdown by IP. Bots are filtered by User-Agent. Region lookup is **fully self-contained** — pure Python stdlib + the free **DB-IP country-lite CSV** (CC-BY, no licence key) cached in the build `downloads/` dir and refreshed monthly; **no apt package, no gem, no system GeoIP DB**. Prints nothing (page degrades to omitting the section) when the Apache log is unreadable or empty. Reading `/var/log/apache2` requires the cron user to be in the **`adm`** group (`sudo usermod -aG adm zdavatz`).
|
|
52
52
|
- **`swissmedic_watch.sh`** — outage/block auto-recovery (cron: `*/30 * * * * zdavatz`). Since the Swissmedic platform migration (~2026-06-23, now a Swisscom-operated gateway), `www.swissmedic.ch` intermittently resets this host's automated connections **after the TLS handshake** (TCP RST), which aborts `run_oddb2xml.sh` under `set -e` and leaves the feeds stale (the block is host/IP- and client-fingerprint-sensitive: a real browser works, `curl`/`wget`/Ruby get reset, while other admin.ch hosts answer fine — so it is a WAF/bot rule, not an outage). The watcher polls Swissmedic with **oddb2xml's own client** (a Ruby `open-uri` canary on `listen_neu.html`); while blocked it is a silent no-op, and the moment it gets HTTP 200 it launches **one** build and emails. It fires **at most once per day** (stamp in `$STATE_DIR`, default `<OUT_DIR>-watch`, kept **outside** the wiped `$BUILD_DIR`), and skips when a build is already running or today's `default/oddb_article.xml` is already fresh. **PATH assembly (2026-07-28 onwards):** cron hands the watcher a minimal PATH, so it builds one explicitly — `/usr/local/bin` (where a root `gem install` under Debian's system Ruby puts the binstub) + the user gem bin dir (`$(ruby -e 'print Gem.user_dir')/bin`, where an *unprivileged* `gem install` silently lands instead, because `/var/lib/gems` is not writable) + rbenv shims **only when they exist** (the pre-2026-07 box; `RBENV_VERSION=3.4.5` is exported only in that case). The previous hardcoded rbenv-only PATH carried no `/usr/local/bin`, so the first watcher-triggered build on the rebuilt server died `oddb2xml: Kommando nicht gefunden` (exit 127) on all three retries before aborting. `run_oddb2xml.sh` step 1 extends PATH the same way right after its `gem install`, for the same reason.
|
|
53
|
-
- **`transfer.sh`** — optional hand-off (scp) of `$OUT_DIR` to the HIN host; `SCP_DEST` is required-but-unset until the HIN host is known.
|
|
53
|
+
- **`transfer.sh`** — optional hand-off (scp) of `$OUT_DIR` to the HIN host; `SCP_DEST` is required-but-unset until the HIN host is known. **Obsolete, not merely unwired (issue [#127](https://github.com/zdavatz/oddb2xml/issues/127)):** it is not in `/etc/cron.d/mediupdatexml` and `SCP_DEST` still has no default — and it should stay that way, because **HIN no longer hosts a copy at all**. Their page `www.hin.ch/de/services/mediupdate-xml.cfm#section_2` links *this* site's URLs directly (`default/`, `45/`, `50/`, `55/`, `aips2sqlite/`) and mentions `download.hin.ch` exactly **zero** times, so there is nothing left to scp. The trap is that the unfed legacy path `https://download.hin.ch/download/oddb2xml/` **still answers HTTP 200**, serving the build of **22.06.2026** (`GENERATED_BY="oddb2xml 3.0.25"`, `Last-Modified: Sun, 21 Jun 2026 23:01:34 GMT`) — the whole directory carries that one date. It looks authoritative, it 404s nothing, and that is how issue #127 was filed against us for data that was current here all along. When someone reports stale data, **check which host they fetched from before touching the build**: `mediupdatexml.oddb.org` is the source of truth. HIN has been asked to 301 or retire the old path. `README.md` and `QA.md` now point at `mediupdatexml.oddb.org` throughout. For twenty days one link on HIN's page was genuinely broken and it was **ours**: `/aips2sqlite/oddb2xml_swissmedic_sequences.csv` answered 404 — **not** because the server rebuild lost it, but because aips2sqlite hit **the same Refdata barcode bug as issue #122**: `ean_code.length()` on the missing `<DataCarrierIdentifier>` threw a `NullPointerException` out of four copied loops, and `generate_aips_fi` deleted the published CSV *before* regenerating it, so under `set -e` each crash left nothing rather than yesterday's file. Fixed 02.09.2026 in aips2sqlite `ad2fc58` (all four loops skip barcode-less articles; the CSV is now published by `mv` only after a run that passed a row-count check). **When Refdata breaks one of the two projects, check the other** — they parse the same file with the same assumption.
|
|
54
54
|
- **`setup_new_server.sh`** (2026-07-28 onwards) — one-command rebuild of a bare Debian host into this download site, written after **HIN deleted the previous server**: everything that had accumulated by hand on that box, so a rebuild is not a day of archaeology. Root, idempotent. Installs the packages (`apache2` + `certbot`; **`ruby-full`** — Debian 13 ships Ruby 3.3, matching the repo `.ruby-version`, so the new host has **no rbenv**; `cron`, absent on a minimal image; `unzip`/`zip` for the ZurRose transfer and the build zip; `poppler-utils` for `pdftotext` in `tools/generate_bag_sl_group_prices.rb`; nokogiri/ffi build headers), adds `$RUN_USER` to **`adm`** (so `visitor_stats.py` may read `/var/log/apache2`), sets the home dir to **711** (so `www-data` can traverse to `$OUT_DIR` without listing it), `gem install oddb2xml`, creates the five directories (`$OUT_DIR`, `-build`, `-state`, `-watch` — the last two deliberately outside the nightly-wiped `-build` — and `$GET_TRANSFER_DIR`, symlinking the tracked `get_transfer.sh` into it), writes **`/etc/cron.d/mediupdatexml`** (the whole schedule in one re-creatable file rather than the old box's per-user crontab: `get_transfer` 00:30, nightly build 01:00, rust2xml Artikelstamm 03:00, aips2sqlite Fachinfos 04:30, landing page hourly at `:05`, Swissmedic watcher every 30 min) plus a logrotate rule for the cron logs, then hands over to `setup_aips2sqlite.sh` and `setup_mediupdatexml_web.sh`. Env: `RUN_USER`, `OUT_DIR`, `SKIP_WEB=1`, `SKIP_GEM=1`, `SKIP_AIPS=1`. **Deliberately out of scope** (gone with the old server, restored separately): an MTA for the cron `MAILTO`.
|
|
55
55
|
- **`setup_mediupdatexml_web.sh`** — one-time root setup of the Apache vhost + initial page. Skips the Let's Encrypt step with a hint when the domain does not yet resolve in **public** DNS (checked via Cloudflare DoH, not the local resolver, which may hold a stale or negative cache).
|
|
56
56
|
- **`setup_aips2sqlite.sh`** (2026-08-01 onwards) — provisions the **aips2sqlite** half of the site, restored after the server rebuild. The landing page links a whole `/aips2sqlite/` section (Fachinformationen as XML, the AmiKo `.db` files, `oddb2xml_swissmedic_sequences.csv`, `atc_codes_used_set.txt`); `setup_mediupdatexml_web.sh` writes the Apache alias `/aips2sqlite` → `<checkout>/jars/output`, but on the rebuilt host neither the JRE nor that directory existed, so **every link answered 403** — Apache falls back to the default `<Directory /> Require all denied` when the aliased path does not exist, so a *missing* target reads as *forbidden*, not *not found*. Root, idempotent: installs **`openjdk-21-jre-headless`** (the repo ships a prebuilt fat jar `jars/aips2sqlite.jar`, so a JRE suffices — no JDK/Gradle unless the jar is rebuilt; the only AWT user, `BarCode.java`/barcode4j, is off the Fachinfo path), clones the checkout if missing, creates `jars/output` + `jars/downloads` owned by `$RUN_USER`, and appends the 04:30 cron entry to `/etc/cron.d/mediupdatexml` if not already there (`SKIP_CRON=1` when called from `setup_new_server.sh`, which writes that line itself). Generating the data is a separate build-user step: `aips2sqlite/scripts/generate_aips_fi` (~1 h; `--lang=de --xml`, then `--lang=fr --xml --nodown`, then `--nodown --smsequence`, renaming `swiss_medic_sequences.csv` → `oddb2xml_swissmedic_sequences.csv`). `REFDATA_API_KEY` (developer.refdata.ch) is **optional** for this path: the Refdata Partner SOAP download it gates is consumed only by the Takeda partner export, and its failure is caught and logged, so the FI/sequences output is complete without it.
|
|
@@ -63,4 +63,6 @@ Only the scripts are git-tracked; the generated `index.html`/`logo.svg` and the
|
|
|
63
63
|
- Framework: RSpec with flexmock (mocking), webmock + VCR (HTTP recording/playback)
|
|
64
64
|
- Test fixtures: `spec/data/` (sample files), `spec/fixtures/vcr_cassettes/` (recorded HTTP responses)
|
|
65
65
|
- `spec/spec_helper.rb` defines test constants (GTINs) and configures VCR to avoid real HTTP calls during tests
|
|
66
|
-
- CI runs on Ruby 3.
|
|
66
|
+
- CI runs on Ruby 3.2 and 3.3 (`.github/workflows/ruby.yml`); 3.0/3.1 cannot resolve the lockfile since `public_suffix` 7 (Feb 2026) and are EOL
|
|
67
|
+
- **Never call `Kernel#exit` inside `lib/`.** Until 02.09.2026 `Builder#prepare_substances` did (`exit 2` when an `-e`/`--artikelstamm` build had no substances). Under rspec that killed the *process* mid-run: the summary still printed, reading e.g. `1193 examples, 0 failures`, but the count was a random fraction of the ~1640 examples and the exit status was 2 — CI had been red on every run for over a year while local runs looked green to anyone reading the summary line instead of `$?`. The trigger was the June-2026 FHIR default: `-e` implies `--fhir`, `epl.bag.admin.ch` was never stubbed, `FhirExtractor added 0 items`, empty substances, exit. Builder now raises `Oddb2xml::NoSubstancesError` (`util.rb`) and only `bin/oddb2xml` turns that into exit status 2, so production keeps its contract and rspec sees an ordinary failure. `mock_downloads` stubs the three NDJSON URLs with `spec/data/fhir/cyramza.ndjson` (one bundle — enough to stay non-empty, not enough for price assertions), and the four CLI specs that assert on SL prices run with `--no-fhir` against the stubbed `Preparations.xml`, which is what they were written for. **When judging a run, read the exit status and the example count, not the failure count.**
|
|
68
|
+
- **Dependabot security jobs (fixed 02.09.2026, `b9e1f09`):** the "bundler in /. for savon/json" runs had failed for months. Two lessons. (a) Dependabot resolves the lockfile against the **lowest** Ruby the gemspec allows — with `required_ruby_version >= 2.5.0` it could not pick json ≥ 2.8 (needs 2.7), so a stale floor silently blocks every security update; the floor is `>= 3.2.0` now. (b) A gem that is only `require`d but never called still carries its advisories: `savon` (two HIGH, GHSA Savon::Model evaluates WSDL operation names) had been dead since the SwissReg migration `4c65a9c` (March 2026), pinned `~> 2.12.0` so the fix 2.17.2 was unreachable. Removing it also removed akami/gyoku/nori/wasabi/httpi/socksify — and exposed that `HTTPI.log = false` in `Downloader#initialize` was loaded only through savon (305 NameErrors), so that line and `httpi` went too. Before bumping a pinned gem, `grep -rn "Savon::"`-style for actual use first. **The published 3.0.34 gem still declares `savon` and `httpi`**, so every `gem install oddb2xml` — including the nightly's step 1 — pulls the vulnerable savon 2.12.1 until 3.0.35 is released. `rubyntlm`, `multi_json` and the two `net/ntlm` requires in `downloader.rb` are the same kind of SOAP-era leftover (zero references), not yet removed.
|
data/Gemfile.lock
CHANGED
|
@@ -1,10 +1,9 @@
|
|
|
1
1
|
PATH
|
|
2
2
|
remote: .
|
|
3
3
|
specs:
|
|
4
|
-
oddb2xml (3.0.
|
|
4
|
+
oddb2xml (3.0.35)
|
|
5
5
|
csv
|
|
6
6
|
htmlentities
|
|
7
|
-
httpi
|
|
8
7
|
mechanize (>= 2.8.5)
|
|
9
8
|
minitar
|
|
10
9
|
multi_json
|
|
@@ -17,7 +16,6 @@ PATH
|
|
|
17
16
|
rubyXL (~> 3.4.0)
|
|
18
17
|
rubyntlm (>= 0.6.3)
|
|
19
18
|
rubyzip (~> 3.0.1)
|
|
20
|
-
savon (~> 2.12.0)
|
|
21
19
|
sax-machine
|
|
22
20
|
spreadsheet
|
|
23
21
|
standardrb
|
|
@@ -29,14 +27,9 @@ GEM
|
|
|
29
27
|
specs:
|
|
30
28
|
addressable (2.9.0)
|
|
31
29
|
public_suffix (>= 2.0.2, < 8.0)
|
|
32
|
-
akami (1.3.3)
|
|
33
|
-
base64
|
|
34
|
-
gyoku (>= 0.4.0)
|
|
35
|
-
nokogiri
|
|
36
30
|
ast (2.4.3)
|
|
37
31
|
base64 (0.3.0)
|
|
38
32
|
bigdecimal (4.0.1)
|
|
39
|
-
builder (3.3.0)
|
|
40
33
|
byebug (13.0.0)
|
|
41
34
|
reline (>= 0.6.0)
|
|
42
35
|
coderay (1.1.3)
|
|
@@ -48,18 +41,12 @@ GEM
|
|
|
48
41
|
diff-lcs (1.6.2)
|
|
49
42
|
domain_name (0.6.20240107)
|
|
50
43
|
flexmock (3.0.2)
|
|
51
|
-
gyoku (1.4.0)
|
|
52
|
-
builder (>= 2.1.2)
|
|
53
|
-
rexml (~> 3.0)
|
|
54
44
|
hashdiff (1.2.1)
|
|
55
45
|
htmlentities (4.4.2)
|
|
56
46
|
http-cookie (1.1.0)
|
|
57
47
|
domain_name (~> 0.5)
|
|
58
|
-
httpi (2.5.0)
|
|
59
|
-
rack
|
|
60
|
-
socksify
|
|
61
48
|
io-console (0.8.2)
|
|
62
|
-
json (2.
|
|
49
|
+
json (2.21.2)
|
|
63
50
|
language_server-protocol (3.17.0.5)
|
|
64
51
|
lint_roller (1.1.0)
|
|
65
52
|
logger (1.7.0)
|
|
@@ -94,8 +81,6 @@ GEM
|
|
|
94
81
|
racc (~> 1.4)
|
|
95
82
|
nokogiri (1.19.4-arm64-darwin)
|
|
96
83
|
racc (~> 1.4)
|
|
97
|
-
nori (2.7.1)
|
|
98
|
-
bigdecimal
|
|
99
84
|
optimist (3.2.1)
|
|
100
85
|
ox (2.14.23)
|
|
101
86
|
bigdecimal (>= 3.0)
|
|
@@ -165,16 +150,7 @@ GEM
|
|
|
165
150
|
rubyntlm (0.6.5)
|
|
166
151
|
base64
|
|
167
152
|
rubyzip (3.0.2)
|
|
168
|
-
savon (2.12.1)
|
|
169
|
-
akami (~> 1.2)
|
|
170
|
-
builder (>= 2.1.2)
|
|
171
|
-
gyoku (~> 1.2)
|
|
172
|
-
httpi (~> 2.3)
|
|
173
|
-
nokogiri (>= 1.8.1)
|
|
174
|
-
nori (~> 2.4)
|
|
175
|
-
wasabi (~> 3.4)
|
|
176
153
|
sax-machine (1.3.2)
|
|
177
|
-
socksify (1.8.1)
|
|
178
154
|
spreadsheet (1.3.4)
|
|
179
155
|
bigdecimal
|
|
180
156
|
logger
|
|
@@ -198,10 +174,6 @@ GEM
|
|
|
198
174
|
unicode-emoji (~> 4.1)
|
|
199
175
|
unicode-emoji (4.2.0)
|
|
200
176
|
vcr (6.4.0)
|
|
201
|
-
wasabi (3.7.0)
|
|
202
|
-
addressable
|
|
203
|
-
httpi (~> 2.0)
|
|
204
|
-
nokogiri (>= 1.4.2)
|
|
205
177
|
webmock (3.26.1)
|
|
206
178
|
addressable (>= 2.8.0)
|
|
207
179
|
crack (>= 0.3.2)
|
data/History.txt
CHANGED
|
@@ -1,3 +1,12 @@
|
|
|
1
|
+
=== 3.0.35 / 02.09.2026
|
|
2
|
+
* Security: drop the savon dependency (two HIGH advisories, GHSA "Savon::Model evaluates WSDL operation names as Ruby source"). It had not been used since the SwissReg migration in 3.0.19 (commit 4c65a9c, March 2026) -- only a require remained -- but the gemspec pin ~> 2.12.0 kept every installer on the vulnerable 2.12.1 and blocked Dependabot's fix to 2.17.2. Removing it also drops akami, gyoku, nori, wasabi, httpi and socksify from the dependency tree; the one leftover reference, HTTPI.log = false in Downloader#initialize, had only ever been loaded through savon and is gone too. json is raised to 2.21.2 (LOW advisory: generator heap buffer overflow when streaming to an IO). 3.0.34 and earlier still declare savon and httpi, so `gem install oddb2xml` on those versions pulls savon 2.12.1 -- upgrade.
|
|
3
|
+
* required_ruby_version is now >= 3.2 (was >= 2.5.0). Ruby 3.0 and 3.1 are end-of-life and have not been able to resolve the lockfile since public_suffix 7 (February 2026); the stale floor also made Dependabot resolve against Ruby 2.5, which is why its json update could never succeed. CI runs on 3.2 and 3.3.
|
|
4
|
+
* Builder no longer calls Kernel#exit when an -e/--extended or --artikelstamm build ends up with no substances; it raises Oddb2xml::NoSubstancesError and bin/oddb2xml turns that into exit status 2 with a message on stderr. The observable behaviour of the command line is unchanged. Inside the spec suite the exit used to kill the rspec process at a random point with a summary line reading "0 failures", which is why the 3.0.34 note claimed a green suite at 1193 examples: that run had been cut short. The suite now runs to completion -- 1639 examples, 0 failures, exit 0 -- on both CI Rubies. The FHIR NDJSON URLs are stubbed in the specs, the four CLI specs asserting on SL prices run with --no-fhir against the BAG Preparations.xml fixture, and the Weleda Kapitel-70 record in spec/data/transfer.dat carries a valid VAT code (2, not 4).
|
|
5
|
+
|
|
6
|
+
=== 3.0.34 / 02.09.2026
|
|
7
|
+
* Bugfix (WeledaSL): pin the encoding of the three runtime CSVs so the Kapitel-70 SL recovery cannot disappear without a trace. WeledaSL.source passed the downloaded content on in whatever encoding the downloader produced; when that is ASCII-8BIT -- any response served without a charset -- the first comparison against a UTF-8 literal raises Encoding::CompatibilityError, and WeledaSL.load rescues every error into an empty Hash. The whole Weleda/WALA recovery (~835 SL products, issue #121) then vanished silently: no <SL_ENTRY>, no BAG group price, and a build that still reported success. All three lists carry non-ASCII (umlauts in the product names, the "-" in the group-price table), so the live path was one charset-less response away from this; it has not happened yet only because raw.githubusercontent.com serves UTF-8. The encoding is now forced in source, as RoggerNames#parse already did, and the bundled fallback is read with an explicit encoding.
|
|
8
|
+
* This was also the last failing spec: artikelstamm_spec.rb:86, the issue #121 regression test, failed because @weleda_sl was empty under webmock and the item was emitted as PHARMATYPE="N" without SL_ENTRY/PPUB. That example passes now. (This entry originally claimed the whole suite was green at "1193 examples, 0 failures" -- it was not: the count was a truncated run, killed by a Kernel#exit in Builder with exit status 2, see 3.0.35. GitHub issue #104 was closed on that basis and corrected.)
|
|
9
|
+
|
|
1
10
|
=== 3.0.33 / 26.08.2026
|
|
2
11
|
* Bugfix (Refdata): skip PHARMA articles that carry no barcode instead of losing every single one of them. Refdata started publishing a handful of PHARMA articles without a <DataCarrierIdentifier> in August 2026 -- the Swiss Red Cross blood products under the collective registration 99999 (Erythrozytenkonzentrat & co), which have no GTIN at all. RefdataExtractor#to_hash called .size on that nil, and Cli#download_as rescues the resulting NoMethodError into an Oddb2xml.log call that prints nothing unless --log is given, so all 16'093 Refdata PHARMA articles silently vanished from the feeds while the build still reported success. On mediupdatexml.oddb.org the <SMNO> count in oddb_article.xml fell from ~15'300 to exactly 10'210 and stayed frozen there for twelve days -- the remainder came from the BAG FHIR SL feed -- and then to 50 once the BAG url move (3.0.32) took the FHIR feed away too. Refdata NONPHARMA was never affected, because the product-type check runs before the barcode is read. See GitHub issue #122.
|
|
3
12
|
* The two rescued Refdata failures (download and extraction) now also print a warning to stderr. Issue #102 asked the build to continue past a Refdata error, and it still does, but a total loss of every Swissmedic-registered medicine must not look like a successful run in the nightly log.
|
data/QA.md
CHANGED
|
@@ -5,7 +5,8 @@
|
|
|
5
5
|
* SL Preise werden am Anfang des Monats publiziert, jeweils immer am 1. Es kann auch vorkommen, dass das BAG die SL-Preise/Limitationen während dem Monat anpasst.
|
|
6
6
|
* Fachinfos werden täglich publiziert.
|
|
7
7
|
* Swissmedic-Codes erscheinen einmal pro Monat, normalerweise in der ersten Woche.
|
|
8
|
-
* Die Daten
|
|
8
|
+
* Die Daten werden täglich generiert und auf <https://mediupdatexml.oddb.org> publiziert.
|
|
9
|
+
* Der Spiegel unter [MEDIupdate XML](https://www.hin.ch/de/services/mediupdate-xml.cfm) / `download.hin.ch` wird seit Juni 2026 nicht mehr nachgeführt, siehe [Issue #127](https://github.com/zdavatz/oddb2xml/issues/127).
|
|
9
10
|
|
|
10
11
|
#### 2. Gibt es eine Spezifikation der XML Files?
|
|
11
12
|
* Ja, siehe [oddb2xml.xsd](https://github.com/zdavatz/oddb2xml/blob/master/oddb2xml.xsd)
|
|
@@ -21,8 +22,8 @@
|
|
|
21
22
|
* Damit man _Registrations-_ und _Sequenznummer_ besser verstehen kann, muss man einmal das File [excel-version_zugelasseneverpackungen.xlsx](https://www.swissmedic.ch/dam/swissmedic/de/dokumente/listen/excel-version_zugelasseneverpackungen.xlsx.download.xlsx/excel-version_zugelasseneverpackungen.xlsx) öffnen und die ersten paar Spalten anschauen.
|
|
22
23
|
|
|
23
24
|
#### 5. Was ist der Unterschied zwischen oddb_article.xml und oddb_product.xml
|
|
24
|
-
* [oddb_article.xml](
|
|
25
|
-
* [oddb_product.xml](
|
|
25
|
+
* [oddb_article.xml](https://mediupdatexml.oddb.org/default/oddb_article.xml) enhält alle Artikel.
|
|
26
|
+
* [oddb_product.xml](https://mediupdatexml.oddb.org/default/oddb_product.xml) enthält nur die Produkte von der Swissmedic, also die Medikamente.
|
|
26
27
|
|
|
27
28
|
#### 6. Warum hat nicht jedes Produkt im oddb_article.xml einen GTIN?
|
|
28
29
|
* Nicht alle Produkte haben zur Zeit einen GTIN. Dieser wird jedoch laufend ergänzt. Ab 1.1.2019 sollte der Pharmacode komplett verschwinden. Dies wurde von der Stiftung [Refdata](http://www.refdata.ch) auch so bestätigt.
|
|
@@ -36,7 +37,7 @@
|
|
|
36
37
|
#### 8. Ich möchte gerne ein XML-File welches alle Produkte (Pharma und Non-Pharma) und die dazugehörigen Sequenznamen enthält. Gibt es das?
|
|
37
38
|
* Ja. _oddb2xml_ mit der Option _-r_ laufen lassen, siehe [usage](https://github.com/zdavatz/oddb2xml#usage) - Option "_--artikelstamm_".
|
|
38
39
|
* Dieses File wird zur Zeit nicht via [MEDIupdate XML](https://www.hin.ch/services/mediupdate-xml/) zum Download zur Verfügung gestellt. Es muss selber generiert werden mittels _oddb2xml -r_
|
|
39
|
-
* [
|
|
40
|
+
* Der täglich gebaute Artikelstamm steht unter <https://mediupdatexml.oddb.org/artikelstamm/> bereit: [v6 XML](https://mediupdatexml.oddb.org/artikelstamm/artikelstamm_v6.xml), [v6 CSV](https://mediupdatexml.oddb.org/artikelstamm/artikelstamm_v6.csv) und die [Legacy-Version v5](https://mediupdatexml.oddb.org/artikelstamm/artikelstamm_v5.xml).
|
|
40
41
|
|
|
41
42
|
#### 9. Wie installiere ich _oddb2xml_?
|
|
42
43
|
* Neuste, stabile Version von [Ruby](http://www.ruby-lang.org/de/) installieren.
|
data/README.md
CHANGED
|
@@ -46,7 +46,21 @@ Generating files for Elexis Artikelstamm is discussed in the [Readme for the Art
|
|
|
46
46
|
|
|
47
47
|
## usage
|
|
48
48
|
|
|
49
|
-
|
|
49
|
+
Ready-made files are built nightly and published on
|
|
50
|
+
<https://mediupdatexml.oddb.org>, e.g. using
|
|
51
|
+
`wget https://mediupdatexml.oddb.org/default/oddb_article.xml`. See
|
|
52
|
+
[Deployment](#deployment-the-mediupdatexmloddborg-download-site) for what the
|
|
53
|
+
site publishes and how it is generated.
|
|
54
|
+
|
|
55
|
+
HIN's [MEDIupdate XML](https://www.hin.ch/de/services/mediupdate-xml.cfm#section_2)
|
|
56
|
+
page links straight to those URLs — it does not host a copy of its own.
|
|
57
|
+
|
|
58
|
+
The old mirror at `https://download.hin.ch/download/oddb2xml` is **not linked
|
|
59
|
+
from anywhere any more** and has not been updated since June 2026 (it still
|
|
60
|
+
serves the build of 22.06.2026, `GENERATED_BY="oddb2xml 3.0.25"`) — the job
|
|
61
|
+
feeding it stopped with the old server. It still answers `200` with
|
|
62
|
+
plausible-looking data, which is exactly what makes it dangerous. Don't use it,
|
|
63
|
+
see [issue #127](https://github.com/zdavatz/oddb2xml/issues/127).
|
|
50
64
|
|
|
51
65
|
see `--help`.
|
|
52
66
|
|
|
@@ -122,8 +136,9 @@ FR
|
|
|
122
136
|
|
|
123
137
|
## Supported ruby version
|
|
124
138
|
|
|
125
|
-
You will need
|
|
126
|
-
CI runs on Ruby 3.
|
|
139
|
+
You will need Ruby >= 3.2. Current development happens on Ruby 3.3 (`.ruby-version`).
|
|
140
|
+
CI runs on Ruby 3.2 and 3.3 via GitHub Actions — see the badge above for the latest spec results.
|
|
141
|
+
Ruby 3.0 and 3.1 dropped out with the dependency update of February 2026 (`public_suffix` 7 needs 3.2), and both are end-of-life.
|
|
127
142
|
|
|
128
143
|
|
|
129
144
|
## XSD files
|
|
@@ -555,7 +570,12 @@ they are the reference setup for running oddb2xml unattended.
|
|
|
555
570
|
cron entry that regenerates the Fachinformationen. Run as root; idempotent.
|
|
556
571
|
Without it every `/aips2sqlite/` link answers 403, because Apache denies a
|
|
557
572
|
path that does not exist.
|
|
558
|
-
* `transfer.sh` — optional scp hand-off of the output tree
|
|
573
|
+
* `transfer.sh` — optional scp hand-off of the output tree to the HIN
|
|
574
|
+
download server. **Obsolete, and deliberately not wired up**: `SCP_DEST` has
|
|
575
|
+
no default and the script is not in `/etc/cron.d/mediupdatexml`. HIN links
|
|
576
|
+
this site's URLs directly, so there is nothing left to copy; the unfed mirror
|
|
577
|
+
it used to push to is
|
|
578
|
+
[issue #127](https://github.com/zdavatz/oddb2xml/issues/127).
|
|
559
579
|
|
|
560
580
|
One Debian-specific pitfall worth knowing when running oddb2xml from cron: with
|
|
561
581
|
the system Ruby, `gem install oddb2xml` as an unprivileged user cannot write
|
|
@@ -567,6 +587,15 @@ themselves.
|
|
|
567
587
|
|
|
568
588
|
## Testing
|
|
569
589
|
|
|
590
|
+
On a host where the system gem directory is not writable (the mediupdatexml.oddb.org
|
|
591
|
+
box, for instance), point bundler somewhere it may write, otherwise `bundle install`
|
|
592
|
+
aborts while installing `rake`:
|
|
593
|
+
|
|
594
|
+
```bash
|
|
595
|
+
BUNDLE_PATH=/tmp/oddb2xml-bundle bundle install
|
|
596
|
+
BUNDLE_PATH=/tmp/oddb2xml-bundle bundle exec rspec
|
|
597
|
+
```
|
|
598
|
+
|
|
570
599
|
* Calling rake spec runs spec tests.
|
|
571
600
|
* Calling rake test installs the gems and runs oddb2xml with the most commonly used combinations. All output is placed under under ausgabe/<timestamp>. These files should be manually compared to the ones generated by the last release to check for possible problems.
|
|
572
601
|
* we use the gem VCR to record real HTTP responses.
|
data/bin/oddb2xml
CHANGED
data/lib/oddb2xml/builder.rb
CHANGED
|
@@ -277,7 +277,10 @@ module Oddb2xml
|
|
|
277
277
|
@substances.uniq!
|
|
278
278
|
@substances.sort!
|
|
279
279
|
Oddb2xml.log("prepare_substances done. Total #{@substances.size} from #{@items.size} items")
|
|
280
|
-
|
|
280
|
+
if (@options[:extended] || @options[:artikelstamm]) && @substances.empty?
|
|
281
|
+
raise NoSubstancesError, "prepare_substances: no substances from #{@items.size} items -- " \
|
|
282
|
+
"the SL source (FHIR NDJSON or BAG Preparations.xml) yielded nothing, refusing to write an empty feed"
|
|
283
|
+
end
|
|
281
284
|
end
|
|
282
285
|
end
|
|
283
286
|
|
|
@@ -417,7 +420,9 @@ module Oddb2xml
|
|
|
417
420
|
XML_OPTIONS
|
|
418
421
|
) {
|
|
419
422
|
Oddb2xml.log "build_substance #{@substances.size} substances"
|
|
420
|
-
|
|
423
|
+
if (@options[:extended] || @options[:artikelstamm]) && @substances.empty?
|
|
424
|
+
raise NoSubstancesError, "build_substance: no substances to write"
|
|
425
|
+
end
|
|
421
426
|
nbr_records = 0
|
|
422
427
|
@substances.each_with_index do |sub_name, i|
|
|
423
428
|
xml.SB("DT" => "") do
|
data/lib/oddb2xml/downloader.rb
CHANGED
|
@@ -2,7 +2,6 @@ require "net/ntlm/version" # needed to avoid error: uninitialized constant Net::
|
|
|
2
2
|
require "rubyntlm"
|
|
3
3
|
require "mechanize"
|
|
4
4
|
require "zip"
|
|
5
|
-
require "savon"
|
|
6
5
|
require "open-uri"
|
|
7
6
|
|
|
8
7
|
SKIP_MIGEL_DOWNLOADER = true # https://github.com/zdavatz/oddb2xml_files/raw/master/NON-Pharma.xls
|
|
@@ -49,7 +48,6 @@ module Oddb2xml
|
|
|
49
48
|
@options = options
|
|
50
49
|
@url = url
|
|
51
50
|
@retry_times = 3
|
|
52
|
-
HTTPI.log = false # disable httpi warning
|
|
53
51
|
Oddb2xml.log "Downloader from #{@url} for #{self.class}"
|
|
54
52
|
init
|
|
55
53
|
end
|
data/lib/oddb2xml/util.rb
CHANGED
|
@@ -7,6 +7,14 @@ module Oddb2xml
|
|
|
7
7
|
# Raised when a downloaded archive (zip/xlsx) is empty or truncated, so the
|
|
8
8
|
# caller can retry the download instead of crashing later in the parser.
|
|
9
9
|
class IncompleteDownloadError < StandardError; end
|
|
10
|
+
# Raised by Builder when an -e/--extended or --artikelstamm build ends up with
|
|
11
|
+
# no substances at all: the SL source (FHIR NDJSON or BAG Preparations.xml)
|
|
12
|
+
# came back empty, so the feed would be useless. bin/oddb2xml turns it into
|
|
13
|
+
# exit status 2; under RSpec it is an ordinary exception. Builder used to call
|
|
14
|
+
# Kernel#exit here, which killed the host process -- inside rspec that ended
|
|
15
|
+
# the run after a random number of examples with a summary line reading
|
|
16
|
+
# "0 failures", so the suite looked green for months while exiting 2.
|
|
17
|
+
class NoSubstancesError < StandardError; end
|
|
10
18
|
|
|
11
19
|
def self.gen_prodno(iksnr, seqnr)
|
|
12
20
|
sprintf("%05d", iksnr) + sprintf("%02d", seqnr)
|
data/lib/oddb2xml/version.rb
CHANGED
data/lib/oddb2xml/weleda_sl.rb
CHANGED
|
@@ -78,9 +78,17 @@ module Oddb2xml
|
|
|
78
78
|
bundled = File.join(DATA_DIR, basename)
|
|
79
79
|
if File.exist?(bundled)
|
|
80
80
|
Oddb2xml.log "WeledaSL: using bundled #{basename}"
|
|
81
|
-
content = File.read(bundled)
|
|
81
|
+
content = File.read(bundled, encoding: "UTF-8")
|
|
82
82
|
end
|
|
83
83
|
end
|
|
84
|
+
# All three lists carry non-ASCII (umlauts in the product names, the "−"
|
|
85
|
+
# in the group-price table). A downloader that hands back ASCII-8BIT --
|
|
86
|
+
# webmock does, and so does any body served without a charset -- makes
|
|
87
|
+
# the first comparison against a UTF-8 literal raise
|
|
88
|
+
# Encoding::CompatibilityError, which load's rescue turns into an empty
|
|
89
|
+
# map: the whole Weleda/WALA SL recovery then vanishes silently. Pin the
|
|
90
|
+
# encoding here, as RoggerNames#parse does.
|
|
91
|
+
content = content.dup.force_encoding(Encoding::UTF_8) if content && content.encoding != Encoding::UTF_8
|
|
84
92
|
content
|
|
85
93
|
end
|
|
86
94
|
|
data/oddb2xml.gemspec
CHANGED
|
@@ -15,7 +15,7 @@ Gem::Specification.new do |spec|
|
|
|
15
15
|
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
|
16
16
|
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
|
|
17
17
|
spec.require_paths = ["lib"]
|
|
18
|
-
spec.required_ruby_version = ">= 2.
|
|
18
|
+
spec.required_ruby_version = ">= 3.2.0"
|
|
19
19
|
|
|
20
20
|
# We fix the version of the spec to newer versions only in the third position
|
|
21
21
|
# hoping that these version fix only security/severe bugs
|
|
@@ -28,14 +28,12 @@ Gem::Specification.new do |spec|
|
|
|
28
28
|
# while Ruby 3.1 (EOL but still in production) can install 1.18.10, the newest 3.1-compatible
|
|
29
29
|
# release. oddb2xml uses no nokogiri 1.19-only API and no CSS/XSLT/C14N on untrusted input.
|
|
30
30
|
spec.add_dependency "nokogiri", ">= 1.18.10"
|
|
31
|
-
spec.add_dependency "savon" , '~> 2.12.0'
|
|
32
31
|
spec.add_dependency "spreadsheet" # , '~> 1.0.0'
|
|
33
32
|
spec.add_dependency "rubyXL", "~> 3.4.0"
|
|
34
33
|
spec.add_dependency "sax-machine" # , '~> 0.1.0'
|
|
35
34
|
spec.add_dependency "parslet" # , '~> 1.7.0'
|
|
36
35
|
spec.add_dependency "rubyntlm", ">= 0.6.3"
|
|
37
36
|
spec.add_dependency "multi_json" # , '>= 0.3.2'
|
|
38
|
-
spec.add_dependency "httpi" # , '>= 2.4.1'
|
|
39
37
|
spec.add_dependency "optimist"
|
|
40
38
|
spec.add_dependency "xml-simple"
|
|
41
39
|
spec.add_dependency "ox"
|
data/spec/builder_spec.rb
CHANGED
|
@@ -451,7 +451,9 @@ def checkProductXml(nbr_record = -1)
|
|
|
451
451
|
expect(hirudoid.elements["ATC"].text).to eq("C05BA01") # modified by atc.csv!
|
|
452
452
|
end
|
|
453
453
|
|
|
454
|
-
|
|
454
|
+
# 81 since 3.0.21: spec/data/transfer.dat gained the Weleda Kapitel-70 record
|
|
455
|
+
# 7611916162404 (issue #121), which -e emits as a ZurRose-only article.
|
|
456
|
+
NR_EXTENDED_ARTICLES = 81
|
|
455
457
|
NR_SUBSTANCES = 28
|
|
456
458
|
NR_LIMITATIONS = 15
|
|
457
459
|
|
|
@@ -685,7 +687,9 @@ describe Oddb2xml::Builder do
|
|
|
685
687
|
context "when --append -I 80 -e is given" do
|
|
686
688
|
before(:all) do
|
|
687
689
|
common_run_init
|
|
688
|
-
|
|
690
|
+
# --no-fhir: the assertions below come from the stubbed BAG Preparations.xml;
|
|
691
|
+
# the FHIR stub in mock_downloads is a single bundle (see spec_helper).
|
|
692
|
+
options = Oddb2xml::Options.parse("--append -I 80 -e --no-fhir")
|
|
689
693
|
Oddb2xml::Cli.new(options).run
|
|
690
694
|
# @res = buildr_capture(:stdout){ Oddb2xml::Cli.new(options).run }
|
|
691
695
|
end
|
|
@@ -734,7 +738,9 @@ describe Oddb2xml::Builder do
|
|
|
734
738
|
context "when option -e is given" do
|
|
735
739
|
before(:all) do
|
|
736
740
|
common_run_init
|
|
737
|
-
|
|
741
|
+
# --no-fhir: the assertions below come from the stubbed BAG Preparations.xml;
|
|
742
|
+
# the FHIR stub in mock_downloads is a single bundle (see spec_helper).
|
|
743
|
+
options = Oddb2xml::Options.parse("-e --no-fhir")
|
|
738
744
|
puts options
|
|
739
745
|
@cli = Oddb2xml::Cli.new(options)
|
|
740
746
|
if RUN_ALL
|
|
@@ -853,7 +859,7 @@ describe Oddb2xml::Builder do
|
|
|
853
859
|
dscrds = REXML::XPath.match(doc, "//ART")
|
|
854
860
|
expect(dscrds.size).to eq(NR_EXTENDED_ARTICLES)
|
|
855
861
|
expect(REXML::XPath.match(doc, "//PHAR").count { |x| x.text.match("0000000") }).to eq(0) # 0 is not a valid pharmacode
|
|
856
|
-
expect(REXML::XPath.match(doc, "//PHAR").count { |x| x.text.match(/\d+/) }).to eq 66
|
|
862
|
+
expect(REXML::XPath.match(doc, "//PHAR").count { |x| x.text.match(/\d+/) }).to eq 67 # 66 + the Weleda record
|
|
857
863
|
end
|
|
858
864
|
|
|
859
865
|
it "should have a correct NBR_RECORD in oddb_limitation.xml" do
|
|
@@ -912,7 +918,9 @@ describe Oddb2xml::Builder do
|
|
|
912
918
|
context "testing -e -I 80 option" do
|
|
913
919
|
before(:all) do
|
|
914
920
|
common_run_init
|
|
915
|
-
|
|
921
|
+
# --no-fhir: the assertions below come from the stubbed BAG Preparations.xml;
|
|
922
|
+
# the FHIR stub in mock_downloads is a single bundle (see spec_helper).
|
|
923
|
+
options = Oddb2xml::Options.parse("-e -I 80 --log --no-fhir")
|
|
916
924
|
# @res = buildr_capture(:stdout){ Oddb2xml::Cli.new(options).run }
|
|
917
925
|
@res = Oddb2xml::Cli.new(options).run
|
|
918
926
|
end
|
data/spec/cli_spec.rb
CHANGED
|
@@ -184,7 +184,9 @@ describe Oddb2xml::Cli do
|
|
|
184
184
|
context "when -e and -f dat option is given" do
|
|
185
185
|
before(:all) do
|
|
186
186
|
cleanup_directories_before_run
|
|
187
|
-
|
|
187
|
+
# --no-fhir: the assertions below come from the stubbed BAG Preparations.xml;
|
|
188
|
+
# the FHIR stub in mock_downloads is a single bundle (see spec_helper).
|
|
189
|
+
options = Oddb2xml::Options.parse("-e -f dat --no-fhir")
|
|
188
190
|
@cli = Oddb2xml::Cli.new(options)
|
|
189
191
|
@cli_output = buildr_capture(:stdout) { @cli.run }
|
|
190
192
|
end
|
data/spec/data/transfer.dat
CHANGED
|
@@ -72,4 +72,4 @@
|
|
|
72
72
|
1130361821SEROCYTOL Kolon Supp 3 Stk 002275003960401B080700076800027800132
|
|
73
73
|
1131770177VARILRIX Trockensub c Solv (alt) Fertspr 0.5 ml 004768006650101B080800076800058500102
|
|
74
74
|
1127766692VARILRIX Trockensub c Solv Fertspr 0.5 ml 004767006650101B080800076800058500342
|
|
75
|
-
1111019849Absinthium ethanol Infusum D1 Tropfen 50 ml
|
|
75
|
+
1111019849Absinthium ethanol Infusum D1 Tropfen 50 ml 002000000000300B080160076119161624042
|
data/spec/spec_helper.rb
CHANGED
|
@@ -260,6 +260,15 @@ def mock_downloads
|
|
|
260
260
|
"https://raw.githubusercontent.com/zdavatz/oddb2xml_files/master/weleda_arzneimittel.csv" => "weleda_arzneimittel.csv",
|
|
261
261
|
"https://raw.githubusercontent.com/zdavatz/oddb2xml_files/master/wala_arzneimittel.csv" => "wala_arzneimittel.csv",
|
|
262
262
|
"https://raw.githubusercontent.com/zdavatz/oddb2xml_files/master/bag_sl_group_prices.csv" => "bag_sl_group_prices.csv",
|
|
263
|
+
# The FHIR NDJSON, default SL source for -e/-b since June 2026. It was never
|
|
264
|
+
# stubbed: webmock refused epl.bag.admin.ch, FhirExtractor added 0 items and
|
|
265
|
+
# Builder called Kernel#exit 2, which killed rspec mid-run with a summary
|
|
266
|
+
# line reading "0 failures". One real bundle keeps the extractor non-empty;
|
|
267
|
+
# specs asserting on SL prices/limitations pass --no-fhir and use
|
|
268
|
+
# Preparations.xml instead.
|
|
269
|
+
"https://epl.bag.admin.ch/static/sl/publication/fhir/foph-sl-publication-latest-de.ndjson" => "fhir/cyramza.ndjson",
|
|
270
|
+
"https://epl.bag.admin.ch/static/sl/publication/fhir/foph-sl-publication-latest-fr.ndjson" => "fhir/cyramza.ndjson",
|
|
271
|
+
"https://epl.bag.admin.ch/static/sl/publication/fhir/foph-sl-publication-latest-it.ndjson" => "fhir/cyramza.ndjson",
|
|
263
272
|
"http://www.xn--spezialittenliste-yqb.ch/File.axd?file=XMLPublications.zip" => "XMLPublications.zip",
|
|
264
273
|
"https://www.spezialitaetenliste.ch/File.axd?file=XMLPublications.zip" => "XMLPublications.zip",
|
|
265
274
|
"https://files.refdata.ch/simis-public-prod/Articles/1.0/Refdata.Articles.zip" => "Refdata.Articles.zip",
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: oddb2xml
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 3.0.
|
|
4
|
+
version: 3.0.35
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Yasuhiro Asaka, Zeno R.R. Davatz, Niklaus Giger
|
|
@@ -65,20 +65,6 @@ dependencies:
|
|
|
65
65
|
- - ">="
|
|
66
66
|
- !ruby/object:Gem::Version
|
|
67
67
|
version: 1.18.10
|
|
68
|
-
- !ruby/object:Gem::Dependency
|
|
69
|
-
name: savon
|
|
70
|
-
requirement: !ruby/object:Gem::Requirement
|
|
71
|
-
requirements:
|
|
72
|
-
- - "~>"
|
|
73
|
-
- !ruby/object:Gem::Version
|
|
74
|
-
version: 2.12.0
|
|
75
|
-
type: :runtime
|
|
76
|
-
prerelease: false
|
|
77
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
78
|
-
requirements:
|
|
79
|
-
- - "~>"
|
|
80
|
-
- !ruby/object:Gem::Version
|
|
81
|
-
version: 2.12.0
|
|
82
68
|
- !ruby/object:Gem::Dependency
|
|
83
69
|
name: spreadsheet
|
|
84
70
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -163,20 +149,6 @@ dependencies:
|
|
|
163
149
|
- - ">="
|
|
164
150
|
- !ruby/object:Gem::Version
|
|
165
151
|
version: '0'
|
|
166
|
-
- !ruby/object:Gem::Dependency
|
|
167
|
-
name: httpi
|
|
168
|
-
requirement: !ruby/object:Gem::Requirement
|
|
169
|
-
requirements:
|
|
170
|
-
- - ">="
|
|
171
|
-
- !ruby/object:Gem::Version
|
|
172
|
-
version: '0'
|
|
173
|
-
type: :runtime
|
|
174
|
-
prerelease: false
|
|
175
|
-
version_requirements: !ruby/object:Gem::Requirement
|
|
176
|
-
requirements:
|
|
177
|
-
- - ">="
|
|
178
|
-
- !ruby/object:Gem::Version
|
|
179
|
-
version: '0'
|
|
180
152
|
- !ruby/object:Gem::Dependency
|
|
181
153
|
name: optimist
|
|
182
154
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -604,7 +576,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
604
576
|
requirements:
|
|
605
577
|
- - ">="
|
|
606
578
|
- !ruby/object:Gem::Version
|
|
607
|
-
version: 2.
|
|
579
|
+
version: 3.2.0
|
|
608
580
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
609
581
|
requirements:
|
|
610
582
|
- - ">="
|