fhirsmith 0.18.1 → 1.0.0

package/packages/readme.md

@@ -34,7 +34,10 @@ In summary:
 | `GET /packages/{id}/{version}` | Download a package tarball; or, when the request's `Accept` prefers JSON (ranks it above `*/*`), npm's document for that version. A `FHIR-Package-Download` header always gets the tarball (the HL7 Java tools send it) |
 | `GET /packages/updates` | Versions published since a date |
 | `GET /packages/broken` | Package versions whose dependencies this registry doesn't hold |
+| `GET /packages/-/npm/v1/keys` | The key this registry signs package versions with (see [Signatures](#signatures)) |
+| `GET /packages/-/fhir/v1/signatures/{name}@{version}` | The publisher's signature of a version, as the crawler fetched it (see [Publishers' signatures](#publishers-signatures)) |
 | `GET /packages/npm-check` | Check any registry against the npm registry protocol (a form, in a browser) |
+| `GET /packages/feed-check` | Check a package feed the way the crawler reads it, storing nothing (a form, in a browser); the report is at `/packages/feed-check/{run}` |
 
 Most endpoints return an HTML page instead of JSON when the request's `Accept`
 header contains `text/html`. The JSON is the contract; the HTML is for people.
@@ -92,9 +95,16 @@ In the server config, under `modules.packages`:
 | `crawlToken` | Shared secret for `POST /packages/crawl`. Omit it to disable that endpoint; the scheduled crawler is unaffected. |
 | `crawler.enabled` | Run the crawler at startup and on the schedule. |
 | `crawler.schedule` | Cron expression for crawls. |
+| `crawler.signatures` | Publishers' package signatures: `ignore`, `check` (the default) or `require`. See [Publishers' signatures](#publishers-signatures). |
 | `masterUrl` | The list of feeds to crawl. Defaults to `https://fhir.github.io/ig-registry/package-feeds.json`. |
 | `localFeedDirs` | Optional. Directories feeds may be read from as local files (for testing). |
 | `allowPrivateAddresses` | Optional, for testing only. Lets the crawler and `/npm-check` fetch from private and loopback addresses, which they otherwise refuse (SSRF protection). |
+| `npmSigning.enabled` | Sign package versions (see [Signatures](#signatures)). On unless set to `false`. |
+| `npmSigning.keyFile` | The signing key, an EC P-256 private key in PEM. A relative path is resolved against the database's folder. Default `npm-signing-key.pem`; made (readable only by the server's user) if it isn't there. |
+| `feedCheck.enabled` | `/packages/feed-check`. On unless set to `false`. |
+| `feedCheck.maxItems` | Items checked per run; the rest of the feed isn't read. Default 50. |
+| `feedCheck.maxTotalMB` | Downloaded per run; items after that are listed but their packages aren't fetched. Default 500. |
+| `feedCheck.keepMinutes` | How long a run's report is kept. Default 60. |
 | `npmCheck.enabled` | `/packages/npm-check`. On unless set to `false`. |
 | `npmCheck.maxConcurrentRuns` | Runs at once, across all users; more get a 503. Default 2. |
 | `npmCheck.runsPerWindow` | Runs per client address per 10 minutes; more get a 429. Default 10. |
@@ -122,6 +132,56 @@ are used three ways:
 `/packages/npm-check` also runs them against registry.npmjs.org once a day, and says on
 the page if any fail there.
 
+## Signatures
+
+The server signs every package version it holds, the way the npm registry does, so
+`npm audit signatures` can check packages installed from it. A signature says that this
+registry served this tarball: it is the registry's statement, not the publisher's, so
+it's made for every version, wherever it was crawled from. Each version in a package
+document has `dist.integrity` (the sha512 of the tarball) and `dist.signatures`, the
+ECDSA P-256 signature of `{id}@{version}:{integrity}`; `/packages/-/npm/v1/keys` lists
+the public key.
+
+* The key is in `npmSigning.keyFile`, next to the database unless set otherwise. If
+  there's no key there, one is made at startup. It is no better protected than the
+  database and the configuration: anyone who can change those can change what is
+  served anyway.
+* A version is signed when it's stored. Versions stored before this (or signed with a
+  different key) are signed in the background at startup, a row at a time. Until every
+  version is signed, `/-/npm/v1/keys` is a 404, as if the server didn't sign: npm reports
+  an unsigned version from a registry with keys as an error. `/packages/status` shows the
+  progress (`npmSigning`). If some are left, it tries again after the next crawl.
+* To change the key, replace the file and restart: everything is signed again with the
+  new key, and the keys endpoint is a 404 until that's done.
+* With `npmSigning.enabled` false, versions still get `dist.integrity`, but there are no
+  signatures and no keys.
+
+The signatures and integrity are in the `PackageDist` table, not in `PackageVersions`:
+reading a column stored after the tarball means reading the tarball.
+
+## Checking a feed
+
+`/packages/feed-check` takes a feed's URL and reads it the way the crawler does, without storing
+anything: no packages, signatures or visited pages go into the database, and nothing is written
+to the mirror. For each item it reports the package (downloaded, unpacked with the crawler's
+limits and checked as the crawler checks it before storing: id, semver version, canonical, no
+install scripts, no JavaScript unless it's a template, not notForPublication), whether the
+registry's `package-restrictions` allow it from this feed, its signature (`confirmed`,
+`rejected`, `absent`, or `not checked` when the feed isn't in `package-feeds.json` or lists no
+keys), and whether this registry already holds the version, and if so whether it is the same
+bytes. A feed-level problem (it can't be fetched, isn't RSS, or the server rate limits) is
+reported instead. Next pages are followed up to `feedCheck.maxItems` items.
+
+The signing keys can be given with the check, as JSON: the feed's `signing` block as it would be
+in `package-feeds.json` (or the feed's whole entry). Signatures are then checked against those
+keys instead of the registry's, so a publisher can test a feed and its keys before they are
+registered.
+
+A check runs in the background: the form redirects to `/packages/feed-check/{run}`, which reloads
+every few seconds until the check is done. Every request goes through the crawler's SSRF guard,
+and only http(s) URLs are fetched, so a feed can't make the server read local files. Only one
+check runs at a time, across all users; a request while one is running gets a 503.
+
 ## Crawling
 
 Each crawl reads the master feed list, then each RSS feed in it (Simplifier last).
@@ -148,10 +208,62 @@ stores it.
 
 `/packages/log` shows what the last crawl did, feed by feed and item by item.
 
+### Publishers' signatures
+
+A publisher can sign its packages: the IG Publisher does, when its `fhir-settings.json` has a
+`packageSigningKey` (see `org.hl7.fhir.utilities.npm.NpmPackageSigner` in core). Each signed
+package has a partner file, `{package}.tgz.sigstore.json`, and the feed item links it in a
+`fhir:signature` element. The file is a Sigstore bundle: an ECDSA P-256 signature of a statement
+naming the package (`pkg:npm/{id}@{version}`) and the sha512 of the `.tgz`.
+
+Which keys may sign a feed's packages is in the master list, `package-feeds.json`, on the feed:
+
+```json
+"signing": {
+  "required": false,
+  "keys": [{ "keyid": "SHA256:...", "key": "{base64 public key}", "expires": null }]
+}
+```
+
+`crawler.signatures` says what the crawler does with them, for feeds that list keys:
+
+* `ignore`: nothing.
+* `check` (the default): each package's signature is fetched and checked. One that fails, or
+  can't be fetched, is logged (in the server log and `/packages/log`), and the package is stored
+  anyway, unless the feed has `"required": true`, when it is refused.
+* `require`: as `check`, but an unsigned package, or one whose signature fails, is refused,
+  and feeds that list no keys (Simplifier, for one) aren't read at all - the crawl log says so.
+
+With `ignore` or `check`, feeds that list no keys are read as before, and their packages aren't
+checked. A key that had expired when a
+package was published doesn't count. A package that was stored before it was signed (or before
+the server checked) has its signature checked against the bytes it holds, once, the next time
+the crawler sees it in a feed; it is never refused, since it is already held, but a failure is
+logged and recorded.
+
+The signature file is kept in the `PackageSignatures` table as it was when it was checked, valid
+or not, with the outcome, the key that signed it and when it was checked. The HTML lists of
+packages (search, a package's versions, updates) show each version's signature as `confirmed`,
+`rejected` or `absent` (unsigned, or not checked); hover for the key, or the reason.
+
+In the package document, each version with a signature has `dist.fhir-signature`: `url` (the
+signature file, served at `/packages/-/fhir/v1/signatures/{name}@{version}` exactly as it was
+fetched), `status` (`confirmed` or `rejected`) and, when confirmed, the `keyid`. A client can
+check the signature itself, against the keys in `package-feeds.json`.
+
+It is deliberately not `dist.attestations`. npm (`npm audit signatures`) verifies attestations
+as Sigstore attestations: it wants the signing key listed among the registry's own keys
+(`/-/npm/v1/keys`), and a transparency log (Rekor) entry in the bundle. Publishers' bundles have
+no log entry, and listing a publisher's key with the registry's would let that publisher forge
+the registry's signatures, since npm accepts a registry signature from any listed key. So every
+signed package would fail `npm audit signatures`.
+
 ## Storage
 
 Everything is in the SQLite database. `PackageVersions` holds each version's
-metadata and the tarball itself. `Packages` holds one row per package id, with its
+metadata and the tarball itself. `PackageDist` holds each version's `dist.integrity`
+and npm signature. `PackageSignatures` holds the publisher's signature of each version, as
+the crawler checked it. `Packages` holds one row per package id, with its
 current version and download count. `PackageDependencies` (stored as `id@version`),
 `PackageFHIRVersions` and `PackageURLs` (the canonical URLs of the resources in each
 version) support search.

package/publisher/publisher-template.html

@@ -85,8 +85,6 @@
 		<div id="stripe"> </div>
 		<div class="container">  <!-- container -->
 		<div style="background-color: #ad1f2f; padding: 6px; color: white;">  <!-- container -->
-  <a href="http://www.hl7.org/fhir" style="color: gold" title="Fast Healthcare Interoperability Resources - Home Page"><img border="0" src="/icon-fhir-16.png" style="vertical-align: text-bottom"/> <b>FHIR</b></a> &copy; HL7.org &nbsp;|&nbsp;
-  <a href="https://github.com/HealthIntersections/FHIRsmith/blob/main/README.md" style="color: gold"><img border="0" src="/FHIRsmith16.png" style="vertical-align: text-bottom"/> FHIRsmith</a> [%ver%] &nbsp;|&nbsp;
   <a href="/" style="color: gold"> Server Home</a> &nbsp;|&nbsp;
             <a href="/publisher" style="color: gold">Publisher Home</a> &nbsp;|&nbsp;
             <a href="/publisher/tasks" style="color: gold">Tasks</a> &nbsp;|&nbsp;
@@ -106,7 +104,7 @@
             	<div class="inner-wrapper">
  <div class="col-9">
 
-  <h2><img border="0" src="../FHIRsmith32.png" style="vertical-align: text-bottom"/> [%title%] </h2>
+  <h2><a href="https://github.com/HealthIntersections/FHIRsmith/blob/main/README.md" title="FHIRsmith [%ver%]"><img border="0" src="/FHIRsmith32.png" alt="FHIRsmith Logo" style="vertical-align: text-bottom"/></a> [%title%] </h2>
 
 [%content%]
 
@@ -121,8 +119,7 @@
 		<div class="container">  <!-- container -->
 			<div class="inner-wrapper">
 				<p>
-                    <a href="http://www.hl7.org/fhir" style="color: gold" title="Fast Healthcare Interoperability Resources - Home Page"><img border="0" src="/icon-fhir-16.png" style="vertical-align: text-bottom"/> <b>FHIR</b></a> &copy; HL7.org 2011+.  &nbsp;|&nbsp;
-                    <a href="https://github.com/HealthIntersections/FHIRsmith/blob/main/README.md" style="color: gold"><img border="0" src="/FHIRsmith16.png" style="vertical-align: text-bottom"/> FHIRsmith</a>  [%ver%]  &copy; HealthIntersections.com.au 2023+ &nbsp;|&nbsp;
+                    <a href="https://github.com/HealthIntersections/FHIRsmith/blob/main/README.md" style="color: gold"><img border="0" src="/FHIRsmith16.png" alt="FHIRsmith Logo" style="vertical-align: text-bottom"/> FHIRsmith</a>  [%ver%]  &copy; HealthIntersections.com.au 2023+ &nbsp;|&nbsp;
                      ([%ms%] ms)
                     [%sponsorMessage%]
         </p>