fhirsmith 0.11.1 → 0.12.0

package/CHANGELOG.md

@@ -5,6 +5,91 @@ All notable changes to the Health Intersections Node Server will be documented i
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.12.0] - 2026-08-27
+
+### Added
+
+- `$cache-control?mode=check`: ask the server whether a cache-id is still alive. The check is
+  also a keepalive - it resets the cache's idle clock - because a client whose own local cache is
+  absorbing the work is invisible to the server, and its cache would otherwise time out mid-job.
+  An unknown id answers 200 with `valid` = false (not 404), so a client can tell "the server says
+  my cache is gone" from "I could not reach the server"; a live cache also reports `sealed`,
+  `resource-count`, `idle` and the server's idle timeout, so a client can work out how often to
+  check instead of guessing
+- Cache tombstones: the server now records why a cache-id it issued stopped existing - closed by
+  the client, expired after N minutes idle, or cleared with everything else - and says which in
+  the error, with the timings. New `CACHE_ID_CLOSED`, `CACHE_ID_EXPIRED` and `CACHE_ID_CLEARED`
+  messages; `CACHE_ID_UNKNOWN` now means "never issued here", and says that a cache-id is only
+  valid on the server instance and endpoint that issued it (#279)
+- SNOMED CT: MRCM validation of postcoordinated expressions - attribute domain (including the
+  lateralizable body structure rule), range (evaluated as ECL, or as a concrete-value range such
+  as `dec(>#0..)`), cardinality and grouping. A constraint that cannot be resolved leaves the
+  value unchecked rather than rejecting it. The MRCM and lateralizable reference sets are now
+  included in the test SNOMED distribution
+- `$translate`: R4 parameter names are accepted as aliases for the R5 ones (`source`/`target` for
+  the scopes, `targetsystem`, `targetcode`), and R4's `reverse` parameter now does literally what
+  it says - swaps the source and target sides once the parameters are read - while remaining an
+  error in R5+, which names the target concept directly instead
+- `$translate`: reverse translation by naming the target concept (`targetCode`, `targetCoding`,
+  `targetCodeableConcept`), with the source system saying which system the answers come from
+- `$translate`: `ConceptMap.group.element.comment` is read from R6 resources and from the
+  cross-version extension on R5 ones, so a preadopted R5 map and a native R6 map behave the same
+- `$translate`: `originMap` names where a chain of maps started, with every other map consulted
+  along the way reported as a `used-conceptmap`; `group.unmapped` is applied per group, so a
+  mapping in one group no longer suppresses another group's fallback
+- Persistent usage statistics: request counts per module, endpoint and operation are written to a
+  SQLite database every `intervalMinutes`, with both the interval count and the all-time total, so
+  they survive restarts and upgrades. New optional `stats` config block (#255)
+- Support for `ValueSet.compose.property` (R6): a value set can name the properties to return in
+  its own expansion, rather than leaving it to the request
+- Publisher: input validation for GitHub owner and repository, git branch, package id and version -
+  enforced on the server, with matching patterns on the form so the browser objects first
+- Publisher: `large-file-archive` config setting - files the IG Publisher leaves in the web output
+  that are too big for GitHub (>100MB) are moved aside for GitHub-hosted websites, instead of
+  leaving the push to fail
+
+### Changed
+
+- `tx/params.js`: parameter names are now interpreted in exactly one place (`seeParameter`), used
+  by both the request `Parameters` resource and the `valueset-expansion-parameter` extension. A
+  parameter from the request always wins over one embedded in a ValueSet's expansion parameters;
+  accumulating parameters (version rules, designations, properties, supplements) still add
+- An inactive display is now governed by `lenient-display-validation` like every other display
+  check - a warning (and `result` = true) when lenient, an error when not - rather than always
+  being a warning. The display is a designation of the concept, just not a current one
+- tx.fhir.org now loads `fhir.tx.support` rather than `fhir.tx.support.r4`
+
+### Fixed
+
+- `$expand`: `status` was lost from imported property declarations, and a concept carrying the
+  same property more than once had the repeats collapsed to a single value
+- Expansion properties are de-duplicated when they arrive from more than one place (the request,
+  an expansion parameter extension, `compose.property`) - a repeat emitted the property twice and
+  changed the cache key
+- `no-cache=true` never busted the cache: the parameter wrote `uid`, which nothing read, instead of
+  the field the cache key hashes
+- Boolean parameters passed as strings (as they always are on a GET) are now accepted, which
+  revives five parameters that were dead on GET requests
+
+### Tx Conformance Statement
+
+FHIRsmith passed all 2822 HL7 terminology service tests (modes tx.fhir.org+omop+general+snomed, tests v1.9.3, runner v6.10.3)
+
+## [0.11.2] - 2026-08-12
+
+### Fixed
+
+- Update snomed test content
+
+### Fixed
+
+- Fix snomed designation handling for languages
+- Fix for SSRF protection in the validator- 
+
+### Tx Conformance Statement
+
+FHIRsmith passed all 2729 HL7 terminology service tests (modes tx.fhir.org+omop+general+snomed, tests v1.9.3, runner v6.10.1)
+
 ## [0.11.1] - 2026-07-30
 
 ### Added

package/config-template.json

@@ -6,6 +6,18 @@
       "credentials": true
     }
   },
+  // Usage statistics. Request counts are kept per module, per endpoint and per
+  // operation, and written to a SQLite database every intervalMinutes so they
+  // survive restarts and upgrades. Each row carries both the count for that
+  // interval and the all-time total. The whole block is optional.
+  "stats": {
+    // set false to keep statistics in memory only (nothing is written to disk)
+    "enabled": true,
+    // relative paths resolve under the data folder's databases directory
+    "database": "stats.db",
+    // how often the counters are written out
+    "intervalMinutes": 10
+  },
   "modules": {
     "shl": {
       "enabled": false,
@@ -64,7 +76,12 @@
       "database": "/absolute/path/to/publisher.db",
       "sessionSecret": "change-this-to-a-secure-random-string",
       "workspaceRoot": "/absolute/path/to/workspaces",
-      "igPublisherTimeoutMinutes": 60
+      "igPublisherTimeoutMinutes": 60,
+      // where to put files the IG Publisher leaves in the web folder that are too big for GitHub
+      // (>100MB). Only applies to websites whose publish-setup.json has $.website.github = true.
+      // Omit to leave them in place - the push will then fail. Relative paths resolve under the
+      // publisher data folder
+      "large-file-archive": "/absolute/path/to/large-file-archive"
     },
     "npmprojector": {
       "enabled": true,

package/library/folder-content-loader.js

@@ -2,6 +2,8 @@ const fs = require('fs').promises;
 const path = require('path');
 const { PackageContentLoader } = require('./package-manager');
 
+const TERMINOLOGY_TYPES = ['CodeSystem', 'ValueSet', 'ConceptMap'];
+
 /**
  * A content loader that scans a folder for JSON FHIR terminology resources
  * (CodeSystem, ValueSet, ConceptMap). It synthesizes the package layout that
@@ -26,14 +28,27 @@ class FolderContentLoader extends PackageContentLoader {
   }
 
   /**
-   * Replaces the standard initialize: there is no package.json or .index.json
-   * to read, so we synthesize an index from the JSON files in the folder.
+   * The synthesized package name reported by id()/pid().
+   * @returns {string}
    */
-  async initialize() {
-    if (this.loaded) {
-      return;
-    }
+  _packageName() {
+    return 'folder.' + path.basename(this.sourceFolder);
+  }
+
+  /**
+   * Whether a file that can't be parsed, or that isn't a terminology resource,
+   * should be quietly skipped. True for folder scans (the folder may hold all
+   * sorts of things); false when a specific file was nominated.
+   * @returns {boolean}
+   */
+  get _skipInvalidFiles() {
+    return true;
+  }
 
+  /**
+   * Check the source exists and is the right kind of thing.
+   */
+  async _validateSource() {
     try {
       const stat = await fs.stat(this.sourceFolder);
       if (!stat.isDirectory()) {
@@ -45,36 +60,63 @@ class FolderContentLoader extends PackageContentLoader {
       }
       throw err;
     }
+  }
+
+  /**
+   * The file names (relative to sourceFolder) to consider.
+   * @returns {Promise<Array<string>>}
+   */
+  async _candidateFiles() {
+    const names = [];
+    const entries = await fs.readdir(this.sourceFolder, { withFileTypes: true });
+    for (const entry of entries) {
+      if (!entry.isFile()) continue;
+      if (!entry.name.toLowerCase().endsWith('.json')) continue;
+      names.push(entry.name);
+    }
+    return names;
+  }
+
+  /**
+   * Replaces the standard initialize: there is no package.json or .index.json
+   * to read, so we synthesize an index from the JSON files in the source.
+   */
+  async initialize() {
+    if (this.loaded) {
+      return;
+    }
+
+    await this._validateSource();
 
     // Synthesize package metadata so id()/version()/pid() return something usable.
     this.package = {
-      name: 'folder.' + path.basename(this.sourceFolder),
+      name: this._packageName(),
       version: '0.0.0',
       fhirVersions: ['5.0.0']
     };
 
     const files = [];
-    const entries = await fs.readdir(this.sourceFolder, { withFileTypes: true });
-    for (const entry of entries) {
-      if (!entry.isFile()) continue;
-      if (!entry.name.toLowerCase().endsWith('.json')) continue;
-
-      const fullPath = path.join(this.sourceFolder, entry.name);
+    for (const name of await this._candidateFiles()) {
+      const fullPath = path.join(this.sourceFolder, name);
       let json;
       try {
         const content = await fs.readFile(fullPath, 'utf8');
         json = JSON.parse(content);
-      } catch {
-        // Skip unreadable / non-JSON files rather than failing the whole load.
-        continue;
+      } catch (e) {
+        // Skip unreadable / non-JSON files rather than failing the whole load,
+        // unless this exact file was the nominated source.
+        if (this._skipInvalidFiles) continue;
+        throw new Error(`Unable to read FHIR resource from ${fullPath}: ${e.message}`);
       }
 
-      if (!json || typeof json !== 'object') continue;
-      const rt = json.resourceType;
-      if (rt !== 'CodeSystem' && rt !== 'ValueSet' && rt !== 'ConceptMap') continue;
+      const rt = json && typeof json === 'object' ? json.resourceType : null;
+      if (!TERMINOLOGY_TYPES.includes(rt)) {
+        if (this._skipInvalidFiles) continue;
+        throw new Error(`${fullPath} is not a CodeSystem, ValueSet or ConceptMap (resourceType = ${rt || 'none'})`);
+      }
 
       files.push({
-        filename: entry.name,
+        filename: name,
         resourceType: rt,
         id: json.id,
         url: json.url,
@@ -88,4 +130,52 @@ class FolderContentLoader extends PackageContentLoader {
   }
 }
 
-module.exports = { FolderContentLoader };
+/**
+ * A content loader for a single nominated JSON file holding one CodeSystem,
+ * ValueSet or ConceptMap. Everything else works exactly as for a folder source:
+ * the file's own directory is the read-only content root, and the provider
+ * caches live in the supplied cacheFolder.
+ *
+ * Unlike a folder scan, a bad file is an error rather than something to skip -
+ * the config named this file explicitly, so failing silently would just hide
+ * a typo.
+ */
+class FileContentLoader extends FolderContentLoader {
+  /**
+   * @param {string} filePath - The JSON file to load
+   * @param {string} cacheFolder - Folder where Package*Provider DBs may live
+   */
+  constructor(filePath, cacheFolder) {
+    super(path.dirname(filePath), cacheFolder);
+    this.filePath = filePath;
+    this.fileName = path.basename(filePath);
+  }
+
+  _packageName() {
+    return 'file.' + this.fileName.replace(/\.json$/i, '');
+  }
+
+  get _skipInvalidFiles() {
+    return false;
+  }
+
+  async _validateSource() {
+    try {
+      const stat = await fs.stat(this.filePath);
+      if (!stat.isFile()) {
+        throw new Error(`File source path is not a file: ${this.filePath}`);
+      }
+    } catch (err) {
+      if (err.code === 'ENOENT') {
+        throw new Error(`File source path does not exist: ${this.filePath}`);
+      }
+      throw err;
+    }
+  }
+
+  async _candidateFiles() {
+    return [this.fileName];
+  }
+}
+
+module.exports = { FolderContentLoader, FileContentLoader };

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "fhirsmith",
-  "version": "0.11.1",
-  "txVersion": "1.9.1",
+  "version": "0.12.0",
+  "txVersion": "1.9.3",
   "description": "A Node.js server that provides a collection of tools to serve the FHIR ecosystem",
   "main": "server.js",
   "engines": {
@@ -46,7 +46,7 @@
     "express-rate-limit": "^7.4.1",
     "express-session": "^1.19.0",
     "fast-xml-parser": "^5.3.4",
-    "fhir-validator-wrapper": "1.2.2",
+    "fhir-validator-wrapper": "1.3.0",
     "fhirpath": "^4.8.3",
     "fs-extra": "^11.3.3",
     "ini": "^6.0.0",

package/publisher/publisher.js

@@ -1,6 +1,7 @@
 const express = require('express');
 const path = require('path');
 const fs = require('fs');
+const validation = require('./validation');
 const Database = require('sqlite3').Database;
 const bcrypt = require('bcrypt');
 const session = require('express-session');
@@ -8,6 +9,10 @@ const folders = require('../library/folder-setup');
 const escape = require('escape-html');
 const {Utilities} = require("../library/utilities");
 
+// GitHub refuses any file larger than this at the pre-receive hook, and the rejection takes down
+// the whole push rather than just the offending file.
+const GITHUB_MAX_FILE_SIZE = 100 * 1024 * 1024;
+
 
 class PublisherModule {
   constructor(stats) {
@@ -928,6 +933,137 @@ class PublisherModule {
     }
   }
 
+  // The IG Publisher can leave a file in the web folder that GitHub will not accept - typically a
+  // technical correction archive, which is a zip of an entire published version folder and so
+  // carries that folder's own full-ig.zip inside it. GitHub rejects the entire push over one such
+  // file, at the very last step of a publication run. The IG Publisher will keep these out of the
+  // web folder itself from the next release; until then, move them aside here.
+  //
+  // Does nothing unless publisher.large-file-archive says where to put them, and the website is
+  // one that's hosted on GitHub in the first place.
+  async archiveLargeFiles(task, website) {
+    const configured = this.config['large-file-archive'];
+    if (!configured || !this.isGitHubHosted(website)) {
+      return;
+    }
+    const root = website.git_root || website.local_folder;
+    const oversized = this.findLargeFiles(root, GITHUB_MAX_FILE_SIZE);
+    if (oversized.length === 0) {
+      return;
+    }
+
+    const archiveDir = path.isAbsolute(configured)
+        ? configured
+        : folders.filePath('publisher', configured);
+    if (!fs.existsSync(archiveDir)) {
+      fs.mkdirSync(archiveDir, { recursive: true });
+    }
+
+    for (const file of oversized) {
+      const sizeMB = Math.round(fs.statSync(file).size / (1024 * 1024));
+      const dest = this.nonClashingPath(archiveDir, path.basename(file));
+      this.moveFile(file, dest);
+      await this.logTaskMessage(task.id, 'warn', 'Large file ' + path.relative(root, file) + ' (' + sizeMB +
+          ' MB) is over the 100 MB GitHub limit - moved to ' + dest + ' instead of being published');
+    }
+  }
+
+  // Only a website kept in a GitHub repository has a 100MB problem - others can carry files of any
+  // size. $.website.github in the web root's publish-setup.json says which is which; if that can't
+  // be read, or doesn't say, assume not and leave the files where they are.
+  isGitHubHosted(website) {
+    if (!website.local_folder) {
+      return false;
+    }
+    const setupPath = path.join(website.local_folder, 'publish-setup.json');
+    try {
+      const setup = JSON.parse(fs.readFileSync(setupPath, 'utf8'));
+      return !!(setup.website && setup.website.github === true);
+    } catch (err) {
+      this.logger.warn('Could not read ' + setupPath + ' to check whether the website is on GitHub: ' + err.message);
+      return false;
+    }
+  }
+
+  // Files in the web folder that are too big to push. Never descends into .git: a pack file over the
+  // limit is perfectly normal there, and moving it would destroy the repository. Files git is already
+  // ignoring are left alone, since they were never going to be pushed anyway.
+  findLargeFiles(root, limit) {
+    const found = [];
+    const walk = (dir) => {
+      let entries;
+      try {
+        entries = fs.readdirSync(dir, { withFileTypes: true });
+      } catch (err) {
+        this.logger.warn('Large file scan could not read ' + dir + ': ' + err.message);
+        return;
+      }
+      for (const entry of entries) {
+        const full = path.join(dir, entry.name);
+        if (entry.isDirectory()) {
+          if (entry.name !== '.git') {
+            walk(full);
+          }
+        } else if (entry.isFile()) {
+          try {
+            if (fs.statSync(full).size > limit) {
+              found.push(full);
+            }
+          } catch (err) {
+            // the file went away between readdir and stat - nothing left to move
+          }
+        }
+      }
+    };
+    walk(root);
+    return found.length === 0 ? found : this.removeGitIgnored(root, found);
+  }
+
+  removeGitIgnored(root, files) {
+    try {
+      const result = require('child_process').spawnSync('git', ['check-ignore', '--stdin'],
+          { cwd: root, input: files.join('\n'), encoding: 'utf8' });
+      // exit 0: at least one path is ignored. 1: none are. anything else: git could not tell us,
+      // in which case treat everything as pushable rather than leaving a file that breaks the push.
+      if (result.status !== 0 || !result.stdout) {
+        return files;
+      }
+      const ignored = new Set(result.stdout.split('\n').map((s) => s.trim()).filter((s) => s));
+      return files.filter((f) => !ignored.has(f));
+    } catch (err) {
+      this.logger.warn('Could not check the gitignore status of large files: ' + err.message);
+      return files;
+    }
+  }
+
+  nonClashingPath(dir, name) {
+    let candidate = path.join(dir, name);
+    if (!fs.existsSync(candidate)) {
+      return candidate;
+    }
+    const ext = path.extname(name);
+    const stem = path.basename(name, ext);
+    let i = 2;
+    do {
+      candidate = path.join(dir, stem + '-' + i + ext);
+      i++;
+    } while (fs.existsSync(candidate));
+    return candidate;
+  }
+
+  // rename() fails across filesystems, and the archive folder will not always be on the same one
+  moveFile(src, dest) {
+    try {
+      fs.renameSync(src, dest);
+    } catch (err) {
+      if (err.code !== 'EXDEV') {
+        throw err;
+      }
+      fs.copyFileSync(src, dest);
+      fs.unlinkSync(src);
+    }
+  }
+
   async runPublication(task) {
     const website = await this.getWebsite(task.website_id);
     if (!website) {
@@ -1016,6 +1152,9 @@ class PublisherModule {
     // url - can survive into the web tree. Catch that here, before anything is committed.
     await this.verifyPublishedPackage(task, website, draftDir);
 
+    // Step 5c: Move anything too big for GitHub out of the web folder, before it gets staged
+    await this.archiveLargeFiles(task, website);
+
     // Step 6: Commit and push the web folder
     await this.logTaskMessage(task.id, 'info', 'Committing changes to web folder...');
     const gitUrl = 'https://github.com/' + task.github_org + '/' + task.github_repo + '.git';
@@ -1363,23 +1502,23 @@ class PublisherModule {
           content += '</div>';
           content += '<div class="col-md-3">';
           content += '<label for="github_org" class="form-label">GitHub Org</label>';
-          content += '<input type="text" class="form-control" id="github_org" name="github_org" required placeholder="hl7">';
+          content += '<input type="text" class="form-control" id="github_org" name="github_org" required maxlength="39" pattern="' + validation.HTML_PATTERNS.github_org + '" title="Letters, digits and single hyphens" placeholder="hl7">';
           content += '</div>';
           content += '<div class="col-md-3">';
           content += '<label for="github_repo" class="form-label">GitHub Repo</label>';
-          content += '<input type="text" class="form-control" id="github_repo" name="github_repo" required placeholder="fhir-us-core">';
+          content += '<input type="text" class="form-control" id="github_repo" name="github_repo" required maxlength="100" pattern="' + validation.HTML_PATTERNS.github_repo + '" title="Letters, digits, dots, hyphens and underscores" placeholder="fhir-us-core">';
           content += '</div>';
           content += '<div class="col-md-3">';
           content += '<label for="git_branch" class="form-label">Branch</label>';
-          content += '<input type="text" class="form-control" id="git_branch" name="git_branch" required placeholder="main">';
+          content += '<input type="text" class="form-control" id="git_branch" name="git_branch" required maxlength="255" pattern="' + validation.HTML_PATTERNS.git_branch + '" title="A git branch name - slashes are fine, but not spaces, backslashes or any of ~ ^ : ? * [" placeholder="main">';
           content += '</div>';
           content += '<div class="col-md-4">';
           content += '<label for="npm_package_id" class="form-label">NPM Package ID</label>';
-          content += '<input type="text" class="form-control" id="npm_package_id" name="npm_package_id" required placeholder="hl7.fhir.us.core">';
+          content += '<input type="text" class="form-control" id="npm_package_id" name="npm_package_id" required maxlength="128" pattern="' + validation.HTML_PATTERNS.npm_package_id + '" title="Letters, digits, dots, hyphens and underscores" placeholder="hl7.fhir.us.core">';
           content += '</div>';
           content += '<div class="col-md-4">';
           content += '<label for="version" class="form-label">Version</label>';
-          content += '<input type="text" class="form-control" id="version" name="version" required placeholder="6.0.0">';
+          content += '<input type="text" class="form-control" id="version" name="version" required maxlength="64" pattern="' + validation.HTML_PATTERNS.version + '" title="Letters, digits, dots, hyphens and plus signs" placeholder="6.0.0">';
           content += '</div>';
           content += '<div class="col-md-4 d-flex align-items-end">';
           content += '<button type="submit" class="btn btn-primary">Create Task</button>';
@@ -1495,6 +1634,15 @@ class PublisherModule {
       try {
         const {website_id, github_org, github_repo, git_branch, npm_package_id, version} = req.body;
 
+        // Check the input before it goes anywhere near a git command line or a
+        // file name. The browser checks these too (see HTML_PATTERNS), but the
+        // form isn't the only way to reach this route.
+        const problems = validation.validateTaskInput(req.body);
+        if (problems.length > 0) {
+          this.logger.warn('Rejected task creation from user ' + req.session.userId + ': ' + problems.join('; '));
+          return res.status(400).send('Invalid task details: ' + problems.join('; '));
+        }
+
         // Verify user has permission for this website
         const canQueue = await this.userCanQueue(req.session.userId, website_id);
         if (!canQueue) {