fhirsmith 0.13.2 → 0.13.4

package/CHANGELOG.md

@@ -5,6 +5,48 @@ All notable changes to the Health Intersections Node Server will be documented i
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [v0.13.4] - 2026-09-17
+
+### Changed 
+
+- Clean up TxCaps expansion parameter list
+
+### Fixed 
+
+- Fix performance issue with snomed text search
+- clean up release process to avoid version mismatch, and to clean out old dockers
+
+### Tx Conformance Statement
+
+FHIRsmith passed all 3507 HL7 terminology service tests (modes tx.fhir.org+omop+general+snomed+mimetypes+icd-11, tests v1.9.4, runner v6.10.4)
+
+## [v0.13.3] - 2026-09-10
+
+### Fixed
+
+- Fix SNOMED CT display selection on national editions. `_displayRefsetOrder()` knew a display
+  language reference set for only six editions (International, US, UK and the test edition);
+  every other edition fell through to "the first description marked Preferred in any language
+  reference set", which on a multi-language edition means whichever description happened to be
+  stored first. The Belgian edition (11000172109) marks a Preferred synonym in Belgian French
+  (21000172104) and Belgian Dutch (31000172101) as well as in English, so a request that named
+  no `displayLanguage` got French for one concept, Dutch for the next and English for a third.
+  Editions not in the table now default to US then GB English, and the $lookup designation
+  order ranks the edition's display reference set ahead of "preferred anywhere", so the choice
+  no longer depends on import order. An edition with no English language reference set at all
+  still falls back to its own preferred synonym. This changes one tx-ecosystem expectation:
+  `bugs/sct-display-2` recorded "Counselling" for 409063005 on the International edition, which
+  is the GB English preferred term picked only because it is stored before the US one - the same
+  concept already expanded as "Counseling". The expectation is now "Counseling".
+- Fix the language tag on the display designation in $expand. It was hard-coded to `en-US`
+  while its value came from the edition display, so on a multi-language edition a French or
+  Dutch term was published as English - which put it in an implementation guide's English
+  display column and made the real French designation read as a duplicate of it.
+
+### Tx Conformance Statement
+
+FHIRsmith passed all 3507 HL7 terminology service tests (modes tx.fhir.org+omop+general+snomed+mimetypes+icd-11, tests v1.9.5-SNAPSHOT, runner v6.10.4
+
 ## [v0.13.2] - 2026-09-05
 
 ### Added

package/README.md

@@ -259,6 +259,26 @@ Each GitHub Release includes:
     - `ghcr.io/healthintersections/fhirsmith:X.Y.Z`
 - **npm package** published to npmjs.org as `fhirsmith` *(if you add this)*
 
+Release images carry OCI labels, so an image says which release it is without
+having to trust the tag it was pulled under:
+
+```bash
+docker inspect --format '{{json .Config.Labels}}' ghcr.io/healthintersections/fhirsmith:latest
+```
+
+`org.opencontainers.image.version` is the release, `.revision` the commit it was
+built from, and `.source` is what links the package to this repository on GHCR.
+
+### CI Images
+
+Every push to main also publishes `:cibuild` and `:cibuild-<sha>` to the same
+package. These are **not releases** - they are whatever was last merged. The
+Docker Build workflow prunes all but the ten most recent afterwards, so they do
+not bury the current release on the package page. To prune by hand (or to run
+the one-off backfill of images published before that job existed), run the
+**GHCR Cleanup** workflow from the Actions tab with `dry_run` on first, read the
+log, then run it again with `dry_run` off.
+
 ### Creating a Release
 
 GitHub Actions will automatically:
@@ -283,7 +303,18 @@ GitHub Actions will automatically:
 ### Tx Conformance Statement
      {copy content from text-cases-summary.txt}
 ```
-2. Update `package.json` & `package-lock.json` to have the same release version
+2. Update `package.json` & `package-lock.json` to the release version:
+
+```bash
+   npm version --no-git-tag-version X.Y.Z
+   npm install --package-lock-only
+```
+
+   This matters more than it looks. The server reports its version from
+   `package.json` (`server.js`, `tx/tx.js`) - not from the `APP_VERSION` build
+   arg - so a package.json left on the previous number ships a Docker image that
+   calls itself by the wrong release for good. The **Verify Version** job fails
+   the release if `package.json` or `package-lock.json` disagrees with the tag.
 
 3. Commit your changes:
 ```bash

package/config-template.json

@@ -94,7 +94,14 @@
       // (>100MB). Only applies to websites whose publish-setup.json has $.website.github = true.
       // Omit to leave them in place - the push will then fail. Relative paths resolve under the
       // publisher data folder
-      "large-file-archive": "/absolute/path/to/large-file-archive"
+      "large-file-archive": "/absolute/path/to/large-file-archive",
+      // where to clone the IG registry from. Omit for the default,
+      // git@github.com:FHIR/ig-registry.git. Set this to an ssh_config Host alias when the
+      // server should push the registry with a repo-scoped deploy key instead of an account
+      // key - e.g. "git@github-ig-registry:FHIR/ig-registry.git", with a matching Host block
+      // in the service user's ~/.ssh/config carrying IdentityFile and IdentitiesOnly yes.
+      // The alias is what lets ssh tell this remote apart from the website remote.
+      "ig-registry-url": "git@github.com:FHIR/ig-registry.git"
     },
     "npmprojector": {
       "enabled": true,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "fhirsmith",
-  "version": "0.13.2",
+  "version": "0.13.4",
   "txVersion": "1.9.5-SNAPSHOT",
   "description": "A Node.js server that provides a collection of tools to serve the FHIR ecosystem",
   "main": "server.js",
@@ -46,7 +46,7 @@
     "express-rate-limit": "^7.4.1",
     "express-session": "^1.19.0",
     "fast-xml-parser": "^5.3.4",
-    "fhir-validator-wrapper": "1.3.0",
+    "fhir-validator-wrapper": "1.4.0",
     "fhirpath": "^4.8.3",
     "fs-extra": "^11.3.3",
     "ini": "^6.0.0",

package/publisher/publisher.js

@@ -1117,8 +1117,15 @@ class PublisherModule {
     // Step 1: Clone supporting repositories into the task directory
     const registryDir = path.join(taskDir, 'ig-registry');
 
-    await this.runCommand('git', ['clone', 'git@github.com:FHIR/ig-registry.git', registryDir],
-        {}, task.id, 'Cloning ig-registry');
+    // The URL is configurable so that the credential arrangement can live in server config
+    // rather than here. A deployment that pushes ig-registry with a repo-scoped deploy key
+    // rather than an account key points this at an ssh_config Host alias carrying that key,
+    // e.g. git@github-ig-registry:FHIR/ig-registry.git - the alias is the only way ssh can
+    // tell this remote apart from the website remote, both being github.com.
+    const igRegistryUrl = this.config['ig-registry-url'] || 'git@github.com:FHIR/ig-registry.git';
+
+    await this.runCommand('git', ['clone', igRegistryUrl, registryDir],
+        {}, task.id, 'Cloning ig-registry from ' + igRegistryUrl);
 
     // Use website-configured history templates path if provided, otherwise clone the default repo
     let historyDir;

package/tx/cs/cs-snomed.js

@@ -16,6 +16,8 @@ const {ConceptMap} = require("../library/conceptmap");
 const {ECLLexer, ECLParser, ECLNodeType, ECLTokenType} = require("../sct/ecl");
 const {Issue} = require("../library/operation-outcome");
 const {debugLog} = require("../operation-context");
+const {SnomedTextIndex, rankMatches, filterTokens} = require("../sct/text-index");
+const Logger = require("../../library/logger");
 
 // Symbol key under which a resolved ECL filter analysis is memoised directly on
 // the ValueSet compose.include.filter element (`fc`). Symbol-keyed and defined
@@ -28,6 +30,21 @@ const {debugLog} = require("../operation-context");
 // shared - it simply recomputes.
 const SNOMED_FILTER_ANALYSIS = Symbol('snomedFilterAnalysis');
 
+// Description language index -> language code. The index is assigned at import
+// time by SnomedImporter.mapLanguageCode(); this table is its inverse and must
+// be kept in step with it.
+const SCT_LANGUAGE_CODES = {
+  1: 'en',
+  2: 'fr',
+  3: 'nl',
+  4: 'es',
+  5: 'sv',
+  6: 'da',
+  7: 'de',
+  8: 'it',
+  9: 'cs'
+};
+
 // Context kinds matching Pascal enum
 const SnomedProviderContextKind = {
   CODE: 0,
@@ -112,6 +129,9 @@ class SnomedServices {
     this.strings = new SnomedStrings(sharedData.strings);
     this.words = new SnomedWords(sharedData.words);
     this.stems = new SnomedStems(sharedData.stems);
+    // Free-text search index, built on first search (see getTextIndex)
+    this._textIndex = null;
+    this._textIndexPromise = null;
     this.refs = new SnomedReferences(sharedData.refs);
     this.descriptions = new SnomedDescriptions(sharedData.desc);
     this.descriptionIndex = new SnomedDescriptionIndex(sharedData.descRef);
@@ -346,8 +366,20 @@ class SnomedServices {
   // preferred synonym in more than one dialect at once (the International
   // edition ships both US English 509007 and GB English 508004, each marking a
   // different synonym Preferred). The edition default decides which term is
-  // shown. Editions not listed here fall back to "preferred synonym in any
-  // language refset" (the historical behaviour), so no edition regresses.
+  // shown.
+  //
+  // Editions not listed here default to English (US then GB) rather than to
+  // "preferred synonym in any language refset". A national edition carries a
+  // Preferred synonym in each of its own language refsets as well as in English
+  // (e.g. the Belgian edition 11000172109 marks a French term Preferred in
+  // 21000172104 and a Dutch one in 31000172101), so "any refset" resolved to
+  // whichever description happened to be stored first - French for one concept,
+  // Dutch for the next, English for a third. A caller who asked for no
+  // particular language got an arbitrary mix. English is the only defensible
+  // default: it is the one language every edition has, and it is what a request
+  // with no displayLanguage is taken to mean everywhere else in the server. An
+  // edition with no English at all still falls through to the any-refset step
+  // below, so nothing regresses.
   _displayRefsetOrder() {
     if (this._dispOrder) return this._dispOrder;
     const idx = (id) => { const r = this.concepts.findConcept(id); return r.found ? r.index : -1; };
@@ -361,7 +393,7 @@ class SnomedServices {
       '83821000000107': [GB],           // UK Edition
       '999000021000000109': [GB]        // UK Clinical Edition
     };
-    const ids = byEdition[String(this.edition)] || [];
+    const ids = byEdition[String(this.edition)] || [US, GB];
     this._dispOrder = ids.map(idx).filter((i) => i >= 0);
     return this._dispOrder;
   }
@@ -400,59 +432,80 @@ class SnomedServices {
     return false;
   }
 
-  // Return a concept's display: the synonym marked Preferred in the US English
-  // language reference set; failing that the FSN; failing that the first active
-  // description. Previously this returned the first active description outright,
-  // which is only the preferred term by accident of import order.
-  getDisplayName(reference = 0) {
+  // Return a concept's display, together with the language it is actually in:
+  // the synonym marked Preferred in the edition's display language reference
+  // set; failing that a preferred synonym in any language refset; failing that
+  // the FSN; failing that the first active description. Previously this
+  // returned the first active description outright, which is only the preferred
+  // term by accident of import order.
+  //
+  // The language matters as much as the term: the display is emitted as a
+  // designation (DesignationUse.DISPLAY) during $expand, and tagging a French
+  // term as en-US puts it in an IG's English display column and makes the real
+  // French designation look like a duplicate.
+  //
+  // Returns {term, lang} - lang is a language code, or null when no description
+  // could be read.
+  getDisplayNameEx(reference = 0) {
+    const none = { term: '', lang: null };
     try {
       const concept = this.concepts.getConcept(reference);
       const descriptionsRef = concept.descriptions;
 
       if (descriptionsRef === 0) {
-        return '';
+        return none;
       }
 
       const descriptionIndices = this.refs.getReferences(descriptionsRef);
       const K = this._displayConstants();
+      const pick = (description) => ({
+        term: this.strings.getEntry(description.iDesc).trim(),
+        lang: SCT_LANGUAGE_CODES[description.lang] || 'en'
+      });
 
       // 1. Preferred synonym in the edition's default display refset(s), tried
       //    in priority order. This is what disambiguates dialects: on the
       //    International edition a concept may be Preferred in both US and GB
-      //    English, and the edition default (US English) must win.
+      //    English, and the edition default (US English) must win. It is also
+      //    what keeps a multi-language national edition from picking a
+      //    different language for each concept.
       for (const refsetIdx of this._displayRefsetOrder()) {
         for (const descIndex of descriptionIndices) {
           const description = this.descriptions.getDescription(descIndex);
           if (!description.active || description.kind !== K.synonym) continue;
           if (this._descAcceptability(description, refsetIdx) === K.preferred) {
-            return this.strings.getEntry(description.iDesc).trim();
+            return pick(description);
           }
         }
       }
 
       // 2. Fallback: preferred synonym in ANY language refset; then FSN; then
-      //    the first active description. Used for editions without a mapped
-      //    default refset, and for concepts with no preferred synonym there.
-      let fsnTerm = '';
-      let firstActive = '';
+      //    the first active description. Reached only by editions with no
+      //    English language refset at all, and by concepts with no preferred
+      //    synonym in the edition's display refsets.
+      let fsn = null;
+      let firstActive = null;
       for (const descIndex of descriptionIndices) {
         const description = this.descriptions.getDescription(descIndex);
         if (!description.active) continue;
-        const term = this.strings.getEntry(description.iDesc).trim();
-        if (firstActive === '') firstActive = term;
+        if (firstActive === null) firstActive = pick(description);
         if (this._synonymIsPreferred(description)) {
-          return term; // preferred synonym (any English language refset)
+          return pick(description); // preferred synonym (any language refset)
         }
-        if (description.kind === K.fsn && fsnTerm === '') {
-          fsnTerm = term;
+        if (description.kind === K.fsn && fsn === null) {
+          fsn = pick(description);
         }
       }
-      return fsnTerm || firstActive || '';
+      return fsn || firstActive || none;
     } catch (error) {
-      return '';
+      return none;
     }
   }
 
+  getDisplayName(reference = 0) {
+    return this.getDisplayNameEx(reference).term;
+  }
+
   getConceptDescendants(reference) {
     try {
       const allDescRef = this.concepts.getAllDesc(reference);
@@ -1434,16 +1487,104 @@ class SnomedServices {
   };
 
 
-  searchFilter(searchText, includeInactive = false, exactMatch = false) {
-    const result = new SnomedFilterContext();
+  /**
+   * Free-text search over active descriptions.
+   *
+   * Uses the in-memory word index (see tx/sct/text-index.js), built the first
+   * time this edition is searched: every filter token (stop words aside) must
+   * match a word of the concept's active descriptions, either as a prefix or
+   * by sharing its stem. Filters with no indexable token (only 1-character
+   * tokens), or an edition whose index failed to build, fall back to
+   * scanSearch().
+   *
+   * Previously the third parameter was `exactMatch`, which the provider fed
+   * from its unrelated `sort` flag - so $expand with a filter on an
+   * include-all-codes value set matched ANY term by brute-force scan: 10k+
+   * matches for a typical multi-word filter, with the event loop blocked for
+   * the whole scan (tx.fhir.org incident 2026-09-17).
+   *
+   * @param {SearchFilterText} searchText
+   * @param {boolean} includeInactive - include inactive concepts
+   * @param {OperationContext|null} opContext - for yielding / deadline checks
+   * @returns {Promise<SnomedFilterContext>}
+   */
+  async searchFilter(searchText, includeInactive = false, opContext = null) {
+    const filterText = (searchText && searchText.filter ? searchText.filter : '').toLowerCase().trim();
+    const tokens = filterTokens(filterText);
+    if (tokens.length > 0) {
+      const index = await this.getTextIndex(opContext);
+      if (index) {
+        const result = new SnomedFilterContext();
+        const ordinals = index.search(tokens, opContext);
+        if (opContext) {
+          await opContext.checkAndYield('sct:searchFilter');
+        }
+        result.matches = rankMatches(this, index, ordinals, filterText, tokens, includeInactive);
+        return result;
+      }
+    }
+    return await this.scanSearch(filterText, includeInactive, opContext);
+  }
+
+  /**
+   * The edition's text index, building it on first use. The build is shared:
+   * concurrent searches wait on the same promise, and time spent waiting is
+   * not charged to the waiting operation's compute budget. Resolves to null
+   * if the build failed (the next search retries it).
+   * @param {OperationContext|null} opContext
+   * @returns {Promise<SnomedTextIndex|null>}
+   */
+  async getTextIndex(opContext = null) {
+    if (this._textIndex) {
+      return this._textIndex;
+    }
+    if (!this._textIndexPromise) {
+      const started = Date.now();
+      // Logger fetched lazily: importers and tests require this module too
+      const textIndexLog = Logger.getInstance().child({module: 'tx-sct'});
+      this._textIndexPromise = SnomedTextIndex.build(this).then(index => {
+        this._textIndex = index;
+        textIndexLog.info(`SNOMED text index for ${this.versionUri}: ${index.words.length} words, built in ${Date.now() - started}ms`);
+        return index;
+      }, error => {
+        this._textIndexPromise = null;
+        textIndexLog.error(`SNOMED text index build failed for ${this.versionUri}: ${error.message}`);
+        return null;
+      });
+    }
+    if (opContext && typeof opContext.waitFor === 'function') {
+      return await opContext.waitFor(this._textIndexPromise, 'sct:textIndex');
+    }
+    return await this._textIndexPromise;
+  }
 
-    // Simplified search - in full implementation would use stemming and word indexes
-    const searchTerms = searchText.filter.toLowerCase().split(/\s+/);
+  /**
+   * Brute-force scan of every concept's active descriptions: every
+   * whitespace-separated term of the filter must appear (as a substring) in
+   * the same description. Expensive (hundreds of ms on a full edition), so it
+   * yields through opContext.checkAndYield and the compute deadline applies.
+   *
+   * @param {string} filterText - lower-cased, trimmed
+   * @param {boolean} includeInactive
+   * @param {OperationContext|null} opContext
+   * @returns {Promise<SnomedFilterContext>}
+   */
+  async scanSearch(filterText, includeInactive = false, opContext = null) {
+    const result = new SnomedFilterContext();
+    const searchTerms = filterText.split(/\s+/).filter(t => t.length > 0);
     const matches = [];
+    if (searchTerms.length === 0) {
+      result.matches = matches;
+      return result;
+    }
 
-    // Search through all concepts
-    for (let i = 0; i < this.concepts.count(); i++) {
-      const conceptIndex = i * this.concepts.constructor.CONCEPT_SIZE;
+    const conceptCount = this.concepts.count();
+    const conceptSize = this.concepts.constructor.CONCEPT_SIZE;
+    for (let i = 0; i < conceptCount; i++) {
+      if (opContext && (i & 0x3FF) === 0) {
+        await opContext.checkAndYield('sct:searchFilter');
+      }
+      const conceptIndex = i * conceptSize;
 
       try {
         const concept = this.concepts.getConcept(conceptIndex);
@@ -1462,25 +1603,23 @@ class SnomedServices {
           const description = this.descriptions.getDescription(descIndex);
           if (description.active) {
             const term = this.strings.getEntry(description.iDesc).toLowerCase();
-
-            if (exactMatch) {
-              // All search terms must be present
-              matchFound = searchTerms.every(searchTerm => term.includes(searchTerm));
-            } else {
-              // Any search term can match
-              matchFound = searchTerms.some(searchTerm => term.includes(searchTerm));
-            }
-
-            if (matchFound) {
+            if (searchTerms.every(searchTerm => term.includes(searchTerm))) {
+              matchFound = true;
               // Calculate priority based on match quality
-              if (term === searchText.filter.toLowerCase()) {
-                priority = 100; // Exact match
-              } else if (term.startsWith(searchText.filter.toLowerCase())) {
-                priority = 50; // Prefix match
+              let p;
+              if (term === filterText) {
+                p = 100; // Exact match
+              } else if (term.startsWith(filterText)) {
+                p = 50; // Prefix match
               } else {
-                priority = 10; // Contains match
+                p = 10; // Contains match
+              }
+              if (p > priority) {
+                priority = p;
+              }
+              if (priority === 100) {
+                break;
               }
-              break;
             }
           }
         }
@@ -1493,6 +1632,9 @@ class SnomedServices {
           });
         }
       } catch (error) {
+        if (error instanceof Issue || (error && error.abandoned)) {
+          throw error;
+        }
         // Skip problematic concepts
         continue;
       }
@@ -1687,8 +1829,16 @@ class SnomedProvider extends BaseCSServices {
                   if (!prefSyns.has(langCode)) prefSyns.set(langCode, { langCode, use, term });
                 }
               }
-              const display = this.sct.getDisplayName(ctxt.getReference());
-              if (display) displays.addDesignation(true, 'active', 'en-US', null, display);
+              // Tagged with the language the term is actually in, not a
+              // hard-coded en-US: on a multi-language edition the edition
+              // display can be French or Dutch, and mislabelling it as English
+              // is what puts it in an IG's English display column and makes the
+              // real French designation look like a duplicate of it. English
+              // stays 'en-US' (the term comes from the US English language
+              // reference set), so nothing changes for English editions.
+              const display = this.sct.getDisplayNameEx(ctxt.getReference());
+              const displayLang = (!display.lang || display.lang === 'en') ? 'en-US' : display.lang;
+              if (display.term) displays.addDesignation(true, 'active', displayLang, null, display.term);
               // FSNs before synonyms, matching the designation order the published
               // tx-ecosystem expectations were written against (sct/expand-inactive).
               // Which of the two is chosen as the display is settled by
@@ -1697,14 +1847,31 @@ class SnomedProvider extends BaseCSServices {
               for (const d of fsns.values()) displays.addDesignation(false, 'active', d.langCode, d.use, d.term);
               for (const d of prefSyns.values()) displays.addDesignation(false, 'active', d.langCode, d.use, d.term);
             } else {
-              // $lookup: emit every description (preferred synonym first so the
-              // display resolves correctly; order is not otherwise significant).
+              // $lookup: emit every description. Order matters only for the
+              // display: a request with no displayLanguage takes the first
+              // designation Designations._isPreferred() accepts, which is any
+              // SNOMED synonym. Ranking by the edition's display language
+              // reference set first (rather than by "preferred in any refset")
+              // keeps that pick in one language - otherwise a national edition
+              // answers $lookup in French for one concept and Dutch for the
+              // next, depending on description storage order.
+              const K2 = this.sct._displayConstants();
+              const displayRefsets = this.sct._displayRefsetOrder();
+              // Buckets: 0 = preferred in the edition's display refset, 1 =
+              // preferred in some other language refset, 2 = everything else
+              // (FSNs and acceptable synonyms, left in their original order -
+              // Array.sort is stable, so nothing else about the output moves).
+              const rank = (description) => {
+                if (description.kind !== K2.synonym) return 2;
+                for (const refsetIdx of displayRefsets) {
+                  if (this.sct._descAcceptability(description, refsetIdx) === K2.preferred) return 0;
+                }
+                return this.sct._synonymIsPreferred(description) ? 1 : 2;
+              };
               const orderedIndices = descriptionIndices.slice().sort((a, b) => {
                 const da = this.sct.descriptions.getDescription(a);
                 const db = this.sct.descriptions.getDescription(b);
-                const pa = this.sct._synonymIsPreferred(da) ? 0 : 1;
-                const pb = this.sct._synonymIsPreferred(db) ? 0 : 1;
-                return pa - pb;
+                return rank(da) - rank(db);
               });
               for (const descIndex of orderedIndices) {
                 const description = this.sct.descriptions.getDescription(descIndex);
@@ -1733,18 +1900,7 @@ class SnomedProvider extends BaseCSServices {
   }
 
   getLanguageCode(langIndex) {
-    const languageMap = {
-      1: 'en',
-      2: 'fr',
-      3: 'nl',
-      4: 'es',
-      5: 'sv',
-      6: 'da',
-      7: 'de',
-      8: 'it',
-      9: 'cs'
-    };
-    return languageMap[langIndex] || 'en';
+    return SCT_LANGUAGE_CODES[langIndex] || 'en';
   }
 
   // Lookup methods
@@ -2386,8 +2542,10 @@ class SnomedProvider extends BaseCSServices {
   }
 
   // Search filter
+  // eslint-disable-next-line no-unused-vars
   async searchFilter(filterContext, filter, sort) {
-    let f = this.sct.searchFilter(filter, false, sort);
+    // `sort` is not a match-mode flag: results are always ranked.
+    let f = await this.sct.searchFilter(filter, false, this.opContext);
     filterContext.filters.push(f);
     return f;
   }

package/tx/library/extensions.js

@@ -24,8 +24,8 @@ const Extensions = {
     if (resource.jsonObj) {
       resource = resource.jsonObj
     }
-    if (resource.implicitRules) {
-      throw new Issue("error", "business-rule", null, null, 'Cannot process resource "'+name+'" due to the presence of implicit rules @'+place);
+    if (resource.implicitRules && 'https://hl7.org/fhir/implicit-rules/omitted-mandatory-interversion-code' != resource.implicitRules) {
+      throw new Issue("error", "business-rule", null, null, 'Cannot process resource "'+name+'" due to the presence of implicit rules \''+resource.implicitRules+'\' @'+place);
     }
   },
 

package/tx/operation-context.js

@@ -972,6 +972,39 @@ class OperationContext {
     }
   }
 
+  /**
+   * Await something this operation does not itself compute - typically a
+   * shared structure another request is building - without charging the wait
+   * to the compute deadline. The current compute slice is closed before
+   * waiting and a new one opened after; aborts if the client disconnected
+   * meanwhile.
+   *
+   * @param {Promise} promise
+   * @param {string} place - Location identifier for debugging
+   * @returns {Promise<*>} what the promise resolves to
+   */
+  async waitFor(promise, place = 'unknown') {
+    const now = performance.now();
+    this._clock.compute += now - this._clock.sliceStart;
+    let result;
+    try {
+      result = await promise;
+    } finally {
+      const resumed = performance.now();
+      this._clock.sliceStart = resumed;
+      this._clock.lastYield = resumed;
+    }
+    if (this._clock.clientGone) {
+      this.log(`Operation abandoned @ ${place}: client disconnected`);
+      const error = new Issue("error", "too-costly", null,
+          `Operation abandoned at ${place}: the client disconnected before the response was ready`);
+      error.abandoned = true;
+      error.diagnostics = this.diagnostics();
+      throw error;
+    }
+    return result;
+  }
+
   unSeeAll() {
     this.contexts = [];
   }

package/tx/sct/text-index.js

@@ -0,0 +1,419 @@
+'use strict';
+
+/**
+ * In-memory word index for free-text SNOMED CT search ($expand ... filter=).
+ *
+ * The cache files produced by the Pascal importer carry no words/stems
+ * index, and the ones produced by the JS importer carry one keyed on
+ * language-specific stems of every description (inactive ones included). So
+ * rather than depend on what a given cache happens to hold, the index is
+ * built here, in memory, the first time an edition is text-searched:
+ *
+ *  - every alphanumeric token (Unicode letters and digits, lower-cased,
+ *    2+ characters) of every ACTIVE description is a "word"
+ *  - words[] is sorted, so a filter token matches every word it is a prefix
+ *    of (a user who has typed "radioth" finds "radiotherapy")
+ *  - each word is also filed under its Porter stem, so a filter token matches
+ *    the other inflections of the same stem ("fractures" finds "fracture")
+ *  - postings[w] is the ascending list of concept ORDINALS (reference /
+ *    CONCEPT_SIZE) whose active descriptions contain word w
+ *  - wordCounts[ordinal] is the number of distinct words the concept has,
+ *    used to rank concepts whose terms are mostly the filter above ones where
+ *    the filter is a small part of a long term
+ *
+ * A search requires every (non stop-word) filter token to match somewhere in
+ * the concept's active descriptions. For the full International Edition the
+ * build takes a few seconds and the index a few tens of MB; it yields to the
+ * event loop as it goes, and a search takes milliseconds once it exists.
+ */
+
+const natural = require('natural');
+
+const stemmer = natural.PorterStemmer;
+
+// Only the short function words that make a multi-word filter fail to match
+// anything when required. Not "with", "without", "no", "not" - those change
+// the meaning of a clinical term.
+const STOP_WORDS = new Set(['a', 'an', 'and', 'as', 'at', 'by', 'for', 'from', 'in', 'into', 'of', 'on', 'or', 'the', 'to']);
+
+const TOKEN_RE = /[\p{L}\p{N}]+/gu;
+
+const MIN_WORD_LENGTH = 2;
+
+// How long the build runs before giving the event loop a turn (ms).
+const BUILD_SLICE = 20;
+
+// How many of the stem-ranked candidates get the (more expensive)
+// description-level ranking.
+const RANK_REFINE_LIMIT = 200;
+
+/**
+ * Split text into lower-cased word tokens.
+ * @param {string} text
+ * @returns {string[]}
+ */
+function tokenize(text) {
+  if (!text) {
+    return [];
+  }
+  return text.toLowerCase().match(TOKEN_RE) || [];
+}
+
+/**
+ * The tokens of a filter that the index can search on: 2+ characters, and
+ * stop words dropped unless nothing else is left.
+ * @param {string} filterText
+ * @returns {string[]} distinct tokens, in filter order
+ */
+function filterTokens(filterText) {
+  const tokens = [...new Set(tokenize(filterText))].filter(t => t.length >= MIN_WORD_LENGTH);
+  const content = tokens.filter(t => !STOP_WORDS.has(t));
+  return content.length > 0 ? content : tokens;
+}
+
+function yieldToEventLoop() {
+  return new Promise(resolve => setImmediate(resolve));
+}
+
+class SnomedTextIndex {
+  constructor(words, postings, stemWords, wordCounts, conceptSize) {
+    this.words = words;           // sorted string[]
+    this.postings = postings;     // Uint32Array[] parallel to words
+    this.stemWords = stemWords;   // Map stem -> word ids (number[])
+    this.wordCounts = wordCounts; // Uint16Array by concept ordinal
+    this.conceptSize = conceptSize;
+    this.scratch = new Uint8Array(wordCounts.length);
+  }
+
+  /**
+   * Build the index for a loaded edition. Yields to the event loop every
+   * BUILD_SLICE ms of work.
+   * @param {object} sct - SnomedServices
+   * @returns {Promise<SnomedTextIndex>}
+   */
+  static async build(sct) {
+    const concepts = sct.concepts;
+    const conceptSize = concepts.constructor.CONCEPT_SIZE;
+    const conceptCount = concepts.count();
+    const descMaster = sct.descriptions.master;
+
+    const wordIds = new Map();   // word -> provisional id
+    const lists = [];            // provisional id -> number[] of ordinals
+    const lastSeen = [];         // provisional id -> last ordinal added
+    const wordCounts = new Uint16Array(conceptCount);
+
+    let sliceStart = performance.now();
+    for (let ordinal = 0; ordinal < conceptCount; ordinal++) {
+      if ((ordinal & 0xFF) === 0 && performance.now() - sliceStart >= BUILD_SLICE) {
+        await yieldToEventLoop();
+        sliceStart = performance.now();
+      }
+      const descriptionsRef = concepts.getDescriptions(ordinal * conceptSize);
+      if (descriptionsRef === 0) {
+        continue;
+      }
+      let distinct = 0;
+      for (const descIndex of sct.refs.getReferences(descriptionsRef)) {
+        if (descMaster.readUInt8(descIndex + 4) === 0) {
+          continue; // inactive description
+        }
+        const term = sct.strings.getEntry(descMaster.readUInt32LE(descIndex));
+        for (const word of tokenize(term)) {
+          if (word.length < MIN_WORD_LENGTH) {
+            continue;
+          }
+          let id = wordIds.get(word);
+          if (id === undefined) {
+            id = lists.length;
+            wordIds.set(word, id);
+            lists.push([]);
+            lastSeen.push(-1);
+          }
+          if (lastSeen[id] !== ordinal) {
+            lastSeen[id] = ordinal;
+            lists[id].push(ordinal);
+            distinct++;
+          }
+        }
+      }
+      wordCounts[ordinal] = Math.min(distinct, 0xFFFF);
+    }
+
+    // Final layout: words sorted, postings as typed arrays, stems -> words.
+    const words = [...wordIds.keys()].sort();
+    const postings = new Array(words.length);
+    const stemWords = new Map();
+    sliceStart = performance.now();
+    for (let i = 0; i < words.length; i++) {
+      if ((i & 0xFF) === 0 && performance.now() - sliceStart >= BUILD_SLICE) {
+        await yieldToEventLoop();
+        sliceStart = performance.now();
+      }
+      const word = words[i];
+      const id = wordIds.get(word);
+      postings[i] = Uint32Array.from(lists[id]);
+      lists[id] = null;
+      const stem = stemmer.stem(word);
+      let ids = stemWords.get(stem);
+      if (!ids) {
+        ids = [];
+        stemWords.set(stem, ids);
+      }
+      ids.push(i);
+    }
+    return new SnomedTextIndex(words, postings, stemWords, wordCounts, conceptSize);
+  }
+
+  /**
+   * The ids of the words a filter token matches.
+   *
+   * A token always matches the words that share its stem ("fractures" finds
+   * "fracture"). Only the LAST token of the filter also matches the words it
+   * is a prefix of - that one may be a word the user is still typing
+   * ("radioth" -> "radiotherapy"), whereas treating a completed word as a
+   * prefix matches unrelated terms ("anal" -> "analog").
+   *
+   * @param {string} token
+   * @param {boolean} allowPrefix
+   * @returns {number[]}
+   */
+  wordsFor(token, allowPrefix = false) {
+    const ids = new Set();
+    let lo = 0;
+    let hi = this.words.length;
+    while (lo < hi) {
+      const mid = (lo + hi) >>> 1;
+      if (this.words[mid] < token) {
+        lo = mid + 1;
+      } else {
+        hi = mid;
+      }
+    }
+    if (allowPrefix) {
+      for (let i = lo; i < this.words.length && this.words[i].startsWith(token); i++) {
+        ids.add(i);
+      }
+    } else if (lo < this.words.length && this.words[lo] === token) {
+      ids.add(lo);
+    }
+    const stemmed = this.stemWords.get(stemmer.stem(token));
+    if (stemmed) {
+      for (const i of stemmed) {
+        ids.add(i);
+      }
+    }
+    return [...ids];
+  }
+
+  /**
+   * Concept ordinals whose active descriptions match every token.
+   * @param {string[]} tokens - from filterTokens()
+   * @param {object} [opContext] - for deadCheck
+   * @returns {number[]} ascending ordinals
+   */
+  search(tokens, opContext = null) {
+    // Deliberately synchronous: the scratch bitmap below is shared by every
+    // search on this edition, so nothing may interleave with it.
+    if (tokens.length === 0) {
+      return [];
+    }
+    const perToken = tokens.map((token, i) => {
+      const ids = this.wordsFor(token, i === tokens.length - 1);
+      let size = 0;
+      for (const id of ids) {
+        size += this.postings[id].length;
+      }
+      return { ids, size };
+    });
+    perToken.sort((a, b) => a.size - b.size);
+    if (perToken[0].size === 0) {
+      return [];
+    }
+
+    const mark = this.scratch;
+    // Candidates: the union for the rarest token.
+    let candidates;
+    const first = perToken[0].ids;
+    if (first.length === 1) {
+      candidates = Array.from(this.postings[first[0]]);
+    } else {
+      mark.fill(0);
+      for (const id of first) {
+        for (const ordinal of this.postings[id]) {
+          mark[ordinal] = 1;
+        }
+      }
+      candidates = [];
+      for (let o = 0; o < mark.length; o++) {
+        if (mark[o]) {
+          candidates.push(o);
+        }
+      }
+    }
+
+    // Narrow by each further token.
+    for (let t = 1; t < perToken.length && candidates.length > 0; t++) {
+      if (opContext) {
+        opContext.deadCheck('sct:textIndex');
+      }
+      mark.fill(0);
+      for (const id of perToken[t].ids) {
+        for (const ordinal of this.postings[id]) {
+          mark[ordinal] = 1;
+        }
+      }
+      candidates = candidates.filter(o => mark[o] === 1);
+    }
+    return candidates;
+  }
+}
+
+/**
+ * Rank the matched concepts and turn them into SnomedFilterContext matches.
+ *
+ * Coarse rank for all: the share of the concept's words that the filter
+ * accounts for (a concept whose terms are just "appendicitis" beats one that
+ * mentions it in passing). The top RANK_REFINE_LIMIT are then checked
+ * against their active description terms: exact term 100, term starting with
+ * the filter 50, all filter tokens within a single description 30, else 10 -
+ * plus the coarse share as a tie-break.
+ *
+ * @param {object} sct - SnomedServices
+ * @param {SnomedTextIndex} index
+ * @param {number[]} ordinals
+ * @param {string} filterText - lower-cased, trimmed filter
+ * @param {string[]} tokens
+ * @param {boolean} includeInactive
+ * @returns {{index:number, term:bigint, priority:number}[]}
+ */
+function rankMatches(sct, index, ordinals, filterText, tokens, includeInactive) {
+  const size = index.conceptSize;
+  const conceptMaster = sct.concepts.master;
+  const wordCounts = index.wordCounts;
+
+  // Active concepts only (unless asked otherwise). The flags byte is read
+  // straight from the concept record: getConcept() builds an object per
+  // concept, and a one-word filter can match 100k+ of them.
+  const kept = [];
+  for (const ordinal of ordinals) {
+    if (!includeInactive && (conceptMaster.readUInt8(ordinal * size + 8) & 0x0F) !== 0) {
+      continue;
+    }
+    kept.push(ordinal);
+  }
+  if (kept.length === 0) {
+    return [];
+  }
+
+  // Coarse rank: the share of the concept's distinct words that the filter
+  // accounts for - a concept whose terms are little more than the filter
+  // beats one that mentions it in passing. Since the filter is fixed, that
+  // share is decided by the concept's word count, so the best
+  // RANK_REFINE_LIMIT are picked by counting sort on the word count rather
+  // than by sorting every match.
+  const counts = new Map();
+  for (const ordinal of kept) {
+    const c = wordCounts[ordinal] || 1;
+    counts.set(c, (counts.get(c) || 0) + 1);
+  }
+  const distinct = [...counts.keys()].sort((a, b) => a - b);
+  let cutoff = distinct[distinct.length - 1];
+  let running = 0;
+  for (const c of distinct) {
+    running += counts.get(c);
+    if (running >= RANK_REFINE_LIMIT) {
+      cutoff = c;
+      break;
+    }
+  }
+
+  const head = [];
+  const tail = [];
+  for (const ordinal of kept) {
+    if ((wordCounts[ordinal] || 1) <= cutoff) {
+      head.push(ordinal);
+    } else {
+      tail.push(ordinal);
+    }
+  }
+
+  const normalizedFilter = tokenize(filterText).join(' ');
+  const tokenStems = tokens.map(t => stemmer.stem(t));
+  const descMaster = sct.descriptions.master;
+  const scored = head.map(ordinal => {
+    const reference = ordinal * size;
+    const entry = {
+      index: reference,
+      ordinal,
+      base: 10,
+      termLength: Number.MAX_SAFE_INTEGER,
+      share: Math.min(1, tokens.length / (wordCounts[ordinal] || 1))
+    };
+    const descriptionsRef = sct.concepts.getDescriptions(reference);
+    if (!descriptionsRef) {
+      return entry;
+    }
+    for (const descIndex of sct.refs.getReferences(descriptionsRef)) {
+      if (descMaster.readUInt8(descIndex + 4) === 0) {
+        continue;
+      }
+      const termTokens = tokenize(sct.strings.getEntry(descMaster.readUInt32LE(descIndex)));
+      const normalizedTerm = termTokens.join(' ');
+      let base;
+      if (normalizedTerm === normalizedFilter) {
+        base = 100;
+      } else if (normalizedTerm.startsWith(normalizedFilter)) {
+        base = 50;
+      } else if (tokenStems.every((stem, t) => termTokens.some(w => w === tokens[t] || stemmer.stem(w) === stem))) {
+        base = 30;
+      } else {
+        continue;
+      }
+      // Best term wins; between equally good terms, the shorter one - it is
+      // the more specific match ("Diabetes mellitus" over "Diabetes mellitus
+      // screening declined").
+      if (base > entry.base || (base === entry.base && normalizedTerm.length < entry.termLength)) {
+        entry.base = base;
+        entry.termLength = normalizedTerm.length;
+      }
+    }
+    return entry;
+  });
+
+  scored.sort((a, b) =>
+      b.base - a.base ||
+      a.termLength - b.termLength ||
+      b.share - a.share ||
+      a.ordinal - b.ordinal);
+
+  const matches = new Array(scored.length + tail.length);
+  let at = 0;
+  for (const entry of scored) {
+    matches[at++] = {
+      index: entry.index,
+      term: sct.concepts.getConceptId(entry.index),
+      priority: entry.base + entry.share
+    };
+  }
+  // Everything else keeps concept order: these are the matches whose terms are
+  // long enough that the filter is a small part of them, and ranking them
+  // against each other would cost a pass over the whole match set for a
+  // difference no caller sees (count/offset paging stops long before).
+  for (const ordinal of tail) {
+    const reference = ordinal * size;
+    matches[at++] = {
+      index: reference,
+      term: sct.concepts.getConceptId(reference),
+      priority: 10
+    };
+  }
+  return matches;
+}
+
+module.exports = {
+  SnomedTextIndex,
+  rankMatches,
+  filterTokens,
+  tokenize,
+  STOP_WORDS
+};

package/tx/workers/expand.js

@@ -861,6 +861,14 @@ class ValueSetExpander {
                   this.addToTotal();
                 }
               }
+              if (this.pageFilled()) {
+                // No total is reported for this path (noTotal above), so once
+                // the requested page is full there is nothing left to learn
+                // from the rest of the matches - and a text filter on a large
+                // code system can match tens of thousands of them, each
+                // costing a designations/properties lookup.
+                break;
+              }
             }
             this.worker.opContext.log('iterate filters done');
             }
@@ -1792,6 +1800,18 @@ class ValueSetExpander {
     pdv[valueName] = value;
   }
 
+  /**
+   * True when the requested page (offset + count) has been filled, so an
+   * iteration that is not counting a total can stop. Exclusions are handled
+   * after the fact and can remove codes already collected, so a value set
+   * with any exclusion never stops early.
+   * @returns {boolean}
+   */
+  pageFilled() {
+    return this.count > 0 && this.offset > -1 && !this.hasExclusions
+        && this.fullList.length >= this.count + this.offset;
+  }
+
   addToTotal(t = 1) {
     if (this.total > -1 && this.totalStatus != "off") {
       this.total = this.total + t;

package/tx/workers/metadata.js

@@ -429,22 +429,34 @@ class MetadataHandler {
           name: 'tx-resource',
           documentation: 'Additional valuesets needed for evaluation e.g. value sets referred to from the import statement of the value set being expanded'
         },
+        // Parameters the $expand handler reads - see handleTypeLevelExpand in
+        // tx/workers/expand.js and TxParameters.seeParameter in tx/params.js.
+        // Keep this list in step with those when adding a parameter.
         { name: '_incomplete' },
         { name: 'abstract' },
         { name: 'activeOnly' },
         { name: 'check-system-version' },
+        { name: 'check-valueset-version' },
         { name: 'count' },
         { name: 'default-to-latest-version' },
+        { name: 'default-valueset-version' },
+        { name: 'designation' },
         { name: 'displayLanguage' },
         { name: 'excludeNested' },
         { name: 'excludeNotForUI' },
         { name: 'excludePostCoordinated' },
+        { name: 'filter' },
         { name: 'force-system-version' },
+        { name: 'force-valueset-version' },
         { name: 'inactive' },
         { name: 'includeAlternateCodes' },
         { name: 'includeDefinition' },
         { name: 'includeDesignations' },
         { name: 'incomplete-ok' },
+        {
+          name: 'limit',
+          documentation: 'Upper bound on the size of the expansion; the operation fails as too-costly beyond it'
+        },
         { name: 'limitedExpansion' },
         {
           name: 'mode',
@@ -454,11 +466,19 @@ class MetadataHandler {
         { name: 'offset' },
         { name: 'profile' },
         { name: 'property' },
+        {
+          name: 'sort',
+          documentation: 'code | display | design | prop:{name}; prefix with - to reverse. Ignored for hierarchical expansions'
+        },
         { name: 'system-version' },
+        { name: 'url' },
+        { name: 'useSupplement' },
+        { name: 'valueSet' },
         {
           name: 'valueSetMode',
           documentation: '= CHECK_MEMBERSHIP_ONLY | NO_MEMBERSHIP_CHECK'
-        }
+        },
+        { name: 'valueSetVersion' }
       ]
     };
   }