fhirsmith 0.18.2 → 1.0.0

package/ips/readme.md

@@ -100,14 +100,42 @@ documentation section with the endpoint's URL, the CapabilityStatement link and
 the requests it answers - and `/ips/{code}/Bundle/{id}` shows the document.
 The document page's header names the patient, the document's date, when and how it was
 received and its size, with links to the FHIR JSON and to download it (`?download=1`, sent
-as an attachment named `ips-{code}-{id}.json`). Below it, each processing step is shown
-with where it is (tick: done, cross: failed, gear: running, moon: waiting - the step's note,
-or for managers the reason it failed, on hover), and managers get "Process again" and
-"Delete" buttons. The views of a document are tabs, chosen with `?view=`: `narrative` (the
-default - the Composition's narrative followed by each section's, with the section titles),
-`translation` (what the translate step did with each concept, below), `generated` (the
-narrative this server generates, below), `validation` (below), `terminology`, `ai` (the AI assessment, below) and `json`
-(pretty-printed, monospaced).
+as an attachment named `ips-{code}-{id}.json`). Below it, the processing pipeline is shown
+as one connected line: Received, then each step in the order it runs, coloured by where it
+is (tick: done, cross: failed, gear: running, moon: waiting - the step's note, or for
+managers the reason it failed, on hover); each step links to the view that shows its work.
+Managers get "Process again" and "Delete" buttons. The views of a document are tabs, chosen
+with `?view=`: `narrative` (the default; a switch, `&narrative=generated`, chooses between
+the narrative as provided - the Composition's narrative followed by each section's, with
+the section titles - and the narrative this server generates, below), `structure` (the
+document's logical structure, below), `analysis` (what the analysis step found, below), `translation` (what
+the translate step did with each concept, below), `terminology`, `validation` (below),
+`national` (below), `profiles` (the clinical profile checks), `ai` (the AI assessment,
+below) and `source` (the JSON, pretty-printed, monospaced). The old view names `generated`
+and `json` still work. The actions (Edit, the Share buttons, and for managers Process again
+and Delete) are on a line of their own below the pipeline.
+`structure` shows the document as a tree table, in the style of the FHIR specification's
+structure tables: columns Path, Type, Id and Details, with the tree drawn in the Path column
+(each node expands and collapses; Expand all and Collapse all buttons; without scripts
+every row shows). It starts at the Composition: first what the Composition refers to other
+than its sections (subject, author, custodian, attesters, ...), then the sections, nested,
+each with what it refers to (its entries, author, focus). Each reference's path is where it
+is (`entry[0]`, `attester[0].party`), and the reference as given is on hover over the Id;
+each resource expands into the references it makes, and so on (sections start open,
+resources closed). A reference resolves as in a Bundle - by fullUrl, absolute or relative
+to the referring entry's - and `#id` is a resource contained in the referring resource (or
+in the resource that contains it). One that doesn't resolve is shown as external (not in
+document). A resource is expanded once, at the shallowest place it is referred to (so one
+the Composition refers to directly expands there, not inside the subject that also refers to
+it); everywhere else it is shown again, greyed, saying where it is expanded, but doesn't
+expand a second time. Details says what the resource is - a name (a person's, an organization's),
+else its code (`code`, the medication - following `medicationReference` -, `vaccineCode`,
+`type`...), with an Observation's value - and when: the first of the clinical dates (onset,
+occurrence, performed, effective, ...), falling back to the record's own (`recorded`,
+`issued`, `date`, `meta.lastUpdated`). Icons, from `static/assets/images`: the document,
+sections, people (Patient, Practitioner, PractitionerRole, RelatedPerson), organizations,
+medications (Medication and the Medication* resources), other clinical resources, Provenance
+and AuditEvent, references outside the document, and anything else.
 `terminology` lists every CodeableConcept in the document (`ips/terminology.js`): the
 Composition's own, then each section's entries in the IPS section order (Patient Story,
 Problems, Allergies, Medications, ... Results; sections not in that list after, in document
@@ -157,10 +185,10 @@ takes submitted documents in turn, marks each `processing`, runs the pipeline's
 it (`ips/pipeline.js`), and marks it `processed` with the steps' notes, or `error` with the
 message of the step that failed. Each step's own state (`waiting`, `running`, `done` or
 `failed`, with its note or error) is kept in its result row as it goes, which is what the
-document page's step line shows; processing again clears every step's result first. The pipeline, in order, will be: **translate** CodeableConcepts
+document page's step line shows; processing again clears every step's result first. The pipeline, in order, will be: determine the document's **nationality**; **analyse** each resource; **translate** CodeableConcepts
 to the international terminologies; generate the **narrative**; **validate** the document; validate it against the
 **national profiles**; measure its quality with PIQI; check it against the **clinical profiles**; **assess** it with AI. Translation, the narrative,
-validation, the national profiles, the **terminology check**, the clinical profiles check and the AI assessment exist so far (PIQI does not). A step keeps what it finds in one of two places: as a result stored against the
+validation, the analysis, the national profiles, the **terminology check**, the clinical profiles check and the AI assessment exist so far (PIQI does not). A step keeps what it finds in one of two places: as a result stored against the
 document in `document_results` (one row per step - the validation outcome), or in the
 **analysed copy** - a second copy of the document (`documents.analysed_json`) that the steps
 adorn with extensions, so that the master document is never changed. The analysed copy is
@@ -168,16 +196,85 @@ rebuilt from the master on every run, so a document can always be processed agai
 Anything left `processing` when the server stopped is queued again at startup. Managers can
 queue a document again from its page, though not while it is being processed.
 
+### Nationality
+
+The first step (`ips/nationality.js`) works out which nation the document is from, as far as the document says -
+few national IGs give a way to tell, so this is a short list of signs, to grow as more are found:
+
+- **US**: the Bundle claims the US Patient Summary profile (`http://hl7.org/fhir/us/pcs/StructureDefinition/Bundle-us-pcs`),
+  or a US Core extension (`http://hl7.org/fhir/us/core/StructureDefinition/...`) is used anywhere in the document;
+- **AU**: the Bundle claims the AU Patient Summary profile (`http://hl7.org.au/fhir/ps/StructureDefinition/au-ps-bundle`).
+
+The nation is its ISO 3166 two-letter code, or unknown when nothing points to a nation, or the signs point to more
+than one. It is kept as the step's result (with the reasons) and in `documents.nationality`, for the list: the
+project page has a Nation column (the code, or `??`), the document page shows the nation's flag to the left of the
+banner (a black and white globe when it isn't known, or there is no flag for it yet -
+`static/assets/images/flags/{code}.svg`), and the first bullet on the Composition in the analysis view says what was
+decided, and why.
+
+### Analysis
+
+The first step (`ips/analysis.js`), over the document as submitted. It walks the same tree
+as the structure view (`ips/structure.js`) and, for each resource there (once, where it is
+first met; contained resources too) and for the Composition section by section (the
+Composition itself without its sections), keeps a list of findings, by the node's key
+(`e{n}` for entry n, `e{n}#id` for a contained resource, `s:{path}` for a section):
+
+- every extension, at any depth (a primitive's extensions are at the primitive's path; an
+  extension's own parts are not listed), with its definition when the module can find one
+  - the validator is asked (fhir-validator-wrapper's `lookupCanonical()`, the service's
+  `/canonical`, which answers from the validator's context; with `autoLoadPackages`, on by
+  default, the validator finds, downloads and loads the package that defines an extension it
+  doesn't have, as it does while validating), and the answers are cached - or a note that it
+  couldn't be found;
+  modifier extensions likewise, in maroon; `textLink` extensions are left out;
+- `implicitRules`, in maroon;
+- in navy: `active` = false; a status of `entered-in-error` (and a Condition's or
+  AllergyIntolerance's `verificationStatus` of entered-in-error); a status of `unknown`; or no
+  status where the resource requires one (`lifecycleStatus` for a Goal);
+- in dark green: a `subject` (or `patient`) that isn't the document's - not the same
+  resource in the document, nor the same reference, nor the same identifier - and any `focus`;
+- dates in the future, where one isn't expected. Only elements the (R4) type model says are
+  dates (date, dateTime, instant) are looked at - so not an identifier or a note that looks
+  like a date - and not inside extensions. Some dates are about the future by their nature and
+  are never mentioned: the end of a period (a name's, address's or identifier's validity, a
+  prescription's), expiry dates, due dates, and schedules (Timing). Some resources may have
+  future dates: Appointment, AppointmentResponse, ImmunizationRecommendation, Schedule and
+  Slot always; requests and plans (ServiceRequest, MedicationRequest, DeviceRequest,
+  NutritionOrder, CommunicationRequest, SupplyRequest, VisionPrescription, CarePlan, Task, and
+  a Goal by its lifecycleStatus) until they are over (completed, revoked, cancelled,
+  entered-in-error, stopped, failed or rejected); and events still to come - a Procedure in
+  preparation, a planned Encounter, an intended MedicationStatement or DeviceUseStatement. A
+  date with no time zone is in the future only if it is everywhere;
+- a Patient's `link`s; a Composition whose status isn't `final`;
+- for a section: its `emptyReason`, and an author other than that of what contains it (the
+  Composition, or the containing section - which may have inherited its own);
+- whether there is narrative.
+
+With nothing but the narrative to say, a resource is "All OK". The `analysis` view shows the
+structure as a flat table - each node's row as in the structure view but with its full path
+(`Composition.section[0].entry[0].resolve().subject`) - with a row under each that lists the
+findings; a resource already shown is "Analysed above", and a reference to a resource not
+in the document has no rows at all.
+
 ### Translation
 
-The first step. Every CodeableConcept the terminology view lists is checked as the
+The second step. Every CodeableConcept the terminology view lists is checked as the
 terminology check (below) checks it; one that fails - no coding in the element's value set,
 or, for an element with no value set of its own, no international coding - is sent to the
 terminology server to translate: `ConceptMap/$translate` with `sourceCodeableConcept` (the
 concept as submitted) and `targetScope` (the element's value set, or the fallback), the
-value set and what it draws on as `tx-resource`. The server chooses the concept maps. Every
+value set and what it draws on as `tx-resource`. A SNOMED CT coding with no version is sent as a code of
+the edition of the document's nation: when the [nationality](#nationality) step found one, and a national profile
+for that nation names a SNOMED CT edition, the coding goes with `version` = `http://snomed.info/sct/{edition}`
+(the document is not changed, and the translation view says which edition was used). Which edition a code is
+from decides which maps apply - a national edition's own concepts translate to the International Edition. The
+terminology check does not do this: it takes the codes as they are, so a SNOMED CT code with no version passes if
+it is fine as it is, and one whose version names another edition does not. The server chooses the concept maps. Every
 match that comes back (whatever its relationship) is added to the concept in the analysed
-copy as a Coding, unless the concept already has that system and code, with an extension
+copy as a Coding, unless the concept already has that coding - the same system and code, and the same version
+when both give one (SNOMED CT versions compared by edition: the same code in the International Edition is a
+different coding from the one in a national edition, and is what the translation is for) - with an extension
 saying where it came from:
 `http://healthintersections.com.au/fhirsmith/StructureDefinition/ips-translation`, with
 `server` (the terminology server, as configured), `conceptMap` (the match's `originMap`, when
@@ -240,15 +337,16 @@ does (`fhir-validator-wrapper`: the validator CLI jar in the data folder's `bin`
 run as a service on its own `port`, with `txServer` - tx.fhir.org, by default - and the IPS
 package loaded). `jar` names a validator jar of its own instead - a validator built locally from
 org.hl7.fhir.core, say: absolute, or relative to the data folder (`bin/local/validator_cli.jar`) - which
-is used as it is, never downloaded or updated. Keep such a jar out of `bin/validator_cli.jar`: the
-wrapper takes that jar's version from the `.version` file it wrote when it downloaded the release
-there, so a different jar in its place is started as if it were that release (and is replaced by
-the next one). It starts in the background when the module does, and the queue waits for
+is used as it is, never downloaded or updated (the wrapper reads its version from the jar, so it
+is only given the options it knows). Keep such a jar out of `bin/validator_cli.jar`: the wrapper
+keeps that one up to date, so a different jar in its place is replaced by the next release. `autoLoadPackages` (default true) starts the validator with `-autoLoadPackages`, so
+the analysis step's extension lookups can load the package that defines an extension (validator
+7.0.5 and `fhir-validator-wrapper` 1.7.0 or later; older ones just don't look further). It starts in the background when the module does, and the queue waits for
 it. Each document - its analysed copy, so the translations are seen (the validator will
 remark on the translation extension until it is given its definition) - is validated against the IPS Bundle profile
 (`http://hl7.org/fhir/uv/ips/StructureDefinition/Bundle-uv-ips`; set `profiles` to validate
-against something else) with the validator's IPS code checks on (`checkIpsCodes`), and the
-OperationOutcome is stored with the document and shown on
+against something else) - validator 7.0.4 dropped its IPS code checks, so `checkIpsCodes` is no longer
+sent (the validator refuses it) - and the OperationOutcome is stored with the document and shown on
 its `validation` view: the counts, each issue's severity, code, location and message, and
 the raw OperationOutcome. Without a configured, running validator the step fails, and the
 document's status is `error`.
@@ -258,7 +356,10 @@ document's status is `error`.
 The national step runs after validation. It validates the document (the analysed copy, as validation does) against each
 [national profile](#national-profiles) in turn: the national IG's package is loaded into the validator (once - a package
 that has loaded, or has failed to, is remembered until the national profiles are changed), then the document is
-validated against the IG's profile on Bundle, with the same `checkIpsCodes` setting. For each profile the result keeps
+validated against the IG's profile on Bundle. The national profiles' packages are also loaded when the validator
+starts (and when the national profiles are changed), before any document is processed, so that the analysis step -
+which runs first - finds the extensions they define; an extension that couldn't be found is looked up again once a
+package has loaded. For each profile the result keeps
 the OperationOutcome, its counts, and whether the document passed - it passes when the validator reports no errors (or
 fatal issues). A profile whose package can't be loaded, or that the validator can't validate against, is kept with the
 reason and counts as not checked; it doesn't stop the pipeline. With no national profiles there is nothing to do, and no
@@ -437,9 +538,9 @@ As FHIR: `GET /ips/StructureDefinition` is a searchset of them all, and `GET /ip
 
 ## National profiles
 
-A national profile is the profile a national implementation guide defines for an IPS document. Each has four parts:
-the jurisdiction it is for, the IG's package id (e.g. `hl7.fhir.au.ips`), the package version, and the canonical URL
-of the IG's profile on Bundle. National profiles are shared across all projects (`/ips/national-profiles`). Everyone can see them; only the
+A national profile is the profile a national implementation guide defines for an IPS document. Each has five parts:
+the jurisdiction it is for, the IG's package id (e.g. `hl7.fhir.au.ips`), the package version, the canonical URL
+of the IG's profile on Bundle, and - optionally, since not every IG uses SNOMED CT - the SNOMED CT edition the IG takes for granted. National profiles are shared across all projects (`/ips/national-profiles`). Everyone can see them; only the
 administrator adds, edits and deletes them. There is at most one for a package id and version. Every document is
 validated against each of them by the [national profiles step](#national-profile-validation).
 
@@ -450,7 +551,12 @@ lists the terminology module holds (`tx/cs/cs-country.js`, `tx/cs/cs-areacode.js
 (`XK`, `XX`, `ZZ`...) and countries' M49 numeric codes aren't accepted. National profiles made before the jurisdiction was asked
 for show it as not given, until they are saved with one. The package id must be a FHIR package id (lower case letters, digits, `-` and `_`, in two or more parts separated by
 dots), the version a package version (`1.0.0`, `1.1.0-ballot`), and the profile an http or https URL. `GET
-/ips/national-profiles` with `Accept: application/json` returns them as `[{jurisdiction, packageId, version, url}]`.
+/ips/national-profiles` with `Accept: application/json` returns them as `[{jurisdiction, packageId, version, url, snomedEdition}]`
+(`snomedEdition` only when there is one).
+
+The SNOMED CT edition is chosen from a list - the editions in `SNOMED_LINKABLE_EDITIONS` (`tx/sct/editions.js`, the
+editions snomed.info can link to), by name - and kept as the edition's module id (`900000000000207008` for the
+International Edition, `32506021000036107` for the Australian), the id in its edition URI `http://snomed.info/sct/{id}`.
 
 ## FHIRPath functions
 
@@ -498,6 +604,7 @@ shared this way. The QR codes are made with `qrcode-generator`, error correction
 ### Validation view
 
 The view shows everything by default; its "Show" selector (`?level=warning` or
-`?level=error`) hides the lesser issues - the summary line still counts them all.
+`?level=error`) hides the lesser issues - the summary line still counts them all. A national profile's
+validation (the `national` view, one profile) shows errors only by default (`?level=information` for everything).
 `fhir-validator-wrapper` 1.5.0 or later is needed: earlier versions sent the profiles in a
 form the validator service refuses.