paquette 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4d8ed39d45d99f0d96e971b61bd47529362b0efa35f9b2704134628b2001bfa3
-  data.tar.gz: 765118b95e910e0cac4b34b8ddf841471237ef93c87fc5f59a5b62e45b470a04
+  metadata.gz: d2e249bb3b01d7ddc7e3b78968579b004ea9ff8862895f51cdd40f814e5ab80f
+  data.tar.gz: f7c8e608ad032792fdecfaf2b66e5f0ac975932ce028b3cfdf5066c758401799
 SHA512:
-  metadata.gz: fbd2e25eecb61a6c5e6cc2dba5bce5283a434fd65bfd65ef175f813c3d683342b42ed3d3f7f634943ea4b818df4f096a160a8d0a2dda3923e37e5b0f9d57a1c0
-  data.tar.gz: 6e90af21e71e67d05f2ff3f89a26cbb65913a1ce196e62cc03d31e1d003a11411c5d0ce8feb9ebb97c64fc72a0311315ee41f83a2f32c22224a7500e116e5484
+  metadata.gz: de59c104c34db56212e2c01b0f6feaab8b96f1ec8040c252784ffd4efd47c1bdba89307634617bd574a2ad4cb0558cf36ba8d54f12e75acc559e6b06ae137a07
+  data.tar.gz: ddd5a0a2353161eb730ca6ca99f91eac33e1327fbe200e14965ec0d46768845a12305442634b5b6eddd926985ab5f29d119fa1b54b094ab2afc61a07f58d7d23

data/CHANGELOG.md

@@ -0,0 +1,79 @@
+## Unreleased
+
+## 0.3.0
+
+- Ship signature files that name every external type. A type sord could not resolve — `Rack::Request`, `Gem::Specification`, `File::Stat` and the like — used to land in `rbi/paquette.rbi` and `sig/paquette.rbs` as a `SORD_ERROR_` constant, which no type checker can read; those types are now root-anchored in the YARD tags and come through by name, and every constant carries a type of its own
+- Document every method and attribute with YARD type tags, and cut the code comments down to what a thing does and why the non-obvious parts are the way they are. `rake yard` builds the docs; `rake rbi` and `rake rbs` generate `rbi/paquette.rbi` and `sig/paquette.rbs` from the tags via sord, and the signature files ship with the gem
+- Store, index, download and yank a platform build as its own artifact. `DirectoryGemRepository` keyed a gem by name and version alone, so a push of `nokogiri-1.16.0-java` wanted the plain build's path and was refused 409 "already exists"; a gem is now filed under the filename `Gem::Specification#file_name` gives it, which leaves a plain-ruby gem's path exactly where it was
+- Make `GemAlreadyExists` and `GemYanked` mean "this name, version *and* platform", so a yanked java build neither hides its ruby sibling nor blocks it from being pushed
+- Take `platform` on `DELETE /api/v1/gems/yank`, the param `gem yank --platform` sends. An absent or blank one means `ruby` and yanks only the plain build; a version that already carries the platform (`1.16.0-java`) is accepted too
+- Validate an uploaded gemspec's `platform` before it becomes a filename — a bounded charset that admits the dashes a real platform needs and refuses path separators, a leading dot, `..`, NUL, newlines and invalid UTF-8 (security)
+- Validate the `gem_name`, `version` and `platform` params of a yank, which reached `File.join` unexamined (security)
+- Drop the `/third push/` exclusion from the conformance suite: it runs 550 examples now, up from 104
+- Report a platform gem's real platform, with a bare version beside it, in `/specs.4.8`, `/latest_specs.4.8`, the dependency API, `/api/v1/versions` and search — all five used to advertise `nokogiri-1.16.0-java` as version `"1.16.0-java"` on platform `"ruby"`, which is a version no client can resolve on a platform that is not the gem's
+- Carry a `Gem::Version` rather than a String in the legacy Marshal indexes, as rubygems.org does — `Gem::SpecFetcher` sorts what it unmarshals, and Strings sort `1.10.0` below `1.9.0`
+- Name the latest version of each gem per *platform* in `/latest_specs.4.8`, so a java or arm64-darwin build is not hidden behind the plain-ruby one
+- Serve `/api/v1/dependencies` and `/api/v1/dependencies.json`, which were documented and implemented but had no route; the unsuffixed one answers Marshal, the way Bundler's legacy fallback reads it
+- Report the gem's own authors, summary and platform from `/api/v1/search.json` instead of the placeholders it used to send for every gem
+- Serve `/prerelease_specs.4.8` and `/prerelease_specs.4.8.gz`, which `gem install --pre` and mirroring tools fetch and which used to 404
+- Keep prereleases out of `/specs.4.8` and `/latest_specs.4.8`, which used to list them alongside releases, so a gem with only prereleases no longer has a "latest" release
+- Run RubyGems' own `gem_server_conformance` suite against a live gem server as `rake test:conformance`, with every example Paquette does not pass listed and explained in `test/conformance/rspec_exclusions.rb`
+- Serve `/names` with the format's `---` marker and a terminating newline, and terminate the last line of `/versions` too
+- Answer the compact index endpoints as `text/plain; charset=utf-8` rather than an unqualified `text/plain`, which RFC 2046 reads as US-ASCII
+- Leave `ruby:` out of a compact index line for a gem that constrains no Ruby version, the way `rubygems:` already was, and join a multi-clause Ruby requirement with `&` rather than a comma that would read as another field; the sidecar cache format is bumped so warm caches re-derive it
+- Stop listing a gem in `/names` once its last version has been yanked, which had `/names` announcing a gem whose `/info/` answered 404
+- Serve conditional GETs on `/versions`, `/names` and `/info/`, with a 304 skipping the compact index render entirely
+- Serve single byte ranges on `/versions`, `/names` and `/info/` with `Accept-Ranges` and a `Repr-Digest` of the whole document, so Bundler >= 2.5 can update its compact index by appending the tail instead of refetching it; a range this server will not serve is ignored and answered in full rather than with a 416
+- Derive the `created_at:` of `/versions` from the oldest publication time in the corpus instead of the clock, which makes the document a pure function of the corpus, turns its `ETag` strong, and stops a ranged fetch from failing its digest check on every render
+- Derive HTTP cache validators from the whole repository wrapper stack, and emit none at all when a layer cannot describe itself
+- Never answer a request carrying `Authorization` or a `Cookie`, or one served through a gate or a personalizer, with `Cache-Control: public`, so a shared cache in front cannot replay an authorized download or index to a caller the embedder's gate would have refused; everything else caller-specific is `private` with `Vary: Authorization, Cookie, Accept-Encoding` (security)
+- Answer anonymous requests over an ungated, unpersonalized repository `public`, so a CDN can cache an open registry, and add `shared_caching: false` to both servers for embedders who authorize by IP, mTLS or a header Paquette cannot see
+- Add `gate_key:` to both `ReadGatedRepository` classes, the caller-supplied name a gate needs before it may be cached
+- Add `cache_validator`, `varies_by_caller?` and `gem_checksum` to the repository protocols
+- Serve `.gem` downloads and npm tarballs as immutable, with an `ETag`, `Last-Modified` and `Range` support on Rack's own file serving
+- Answer conditional GETs on npm packuments and dist-tags, and never store `/-/whoami`
+- Fold each `dist-tags.json` mtime into the npm fingerprint as whole nanoseconds rather than a float
+- Document what rack-cache or Rails' Rack::Cache in front of a Paquette server does and does not store
+- Serve only absolute http(s) URLs for `homepage` and the `*_uri` gemspec metadata keys, and for the npm packument `homepage`
+- Name each gem's `required_rubygems_version` in the compact index, and bump the sidecar cache format so warm caches re-derive it
+- Accept every version RubyGems publishes, not just three-segment ones — a gem pushed at `0.2` or `0.17` used to be written to disk and then served to nobody
+- Add `CooldownRepository`, a wrapper serving only versions published longer than a configured interval ago
+- Give `CooldownRepository` a cache validator that moves only when a version cools, a push or yank lands, or the interval changes, so its index documents can answer 304; a custom `published_at:` gets one only with `published_at_validator:`
+- Validate a pushed gemspec before acting on it, closing an arbitrary file write through `spec.name` (security)
+- Refuse newlines and NUL bytes in every spec field the compact index interpolates, so a push cannot forge an index row
+- Refuse any pushed gemspec field longer than 2KB
+- Validate the npm package version, which reached the filesystem as a path component the same way
+- Parse an uploaded gemspec with YAML alias expansion disabled
+- Stream a gem push to disk instead of buffering it in memory, and cap it with `max_push_bytes:` on the server — `Paquette::MAX_PUSH_SIZE_BYTES` (50MB) unless told otherwise. The cap is the server's rule and the server's enforcement: `DirectoryGemRepository#add_gem` takes the payload and nothing else
+- Cap npm request bodies with the same `max_push_bytes:` keyword on `NpmServer` — a publish carries the whole tarball base64-encoded in JSON, and used to be read without a limit
+
+## 0.2.0
+
+- Repack gems in-process instead of shelling out to the `gem` binary
+- Serve correctly when Paquette is mounted under a path prefix
+- Autoload the library and namespace it with flat identifiers
+- Arm the regexp backtracking ceiling from inside the apps instead of a middleware
+- Extract `ReadonlyRepository` out of `ReadGatedRepository`
+- Add `bin/dev`, a read-write gem-only server backed by `tmp/gems`
+- Add an HTML placeholder page for repository roots
+- Verify publishing OTPs, with each server owning its own dialect
+- Add repository fingerprints, route recognition, and selective personalization
+- Add per-request timing instrumentation, and return 400 for bodies that could not be parsed
+- Add a full NPM registry: metadata, tarball serving, publishing, read gating, and licensee personalization
+- Verify npm parity against a real npm client installing from and publishing to Paquette
+- Make repacked gems and written tarballs byte-reproducible
+- Have the compact index name each gem's dependencies, and make `/versions` checksum what `/info/` returns
+- Build gem info lines from sidecar metadata instead of re-parsing each gem
+- Stop the personalizer from deleting directories it did not create
+- Add the `TokenAuthorization` Rack middleware
+- Add a guide for hosting Paquette inside a Rails app
+- Add a GitHub Actions CI workflow
+
+## 0.1.0
+
+- First packaged release: gem repository serving the compact index, `/info/`, `/versions`, and gem downloads
+- Add the RubyGems push and yank endpoints
+- Add read gating so a licensee is only ever handed their own packages
+- Add the gem repacker and personalizer, injecting licensee metadata into the gemspec
+- Route with Mustermann on top of the minuscule app framework
+- Ship the gemspec for gem distribution

data/README.md

@@ -1,26 +1,129 @@
-# Paquette
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="img/paquette-wordmark-logo-on-dark.png">
+  <img src="img/paquette-wordmark-logo.png" alt="Paquette" width="420">
+</picture>
 
-Paquette is a (sligtly unhinged) Rack-based server for libraries. At the moment it serves gems and NPM packages. It is very basic and is made to serve packages gated by a licensing mechanism, which is supposed to be BYO.
+Paquette is a (slightly unhinged) Rack-based server for libraries. At the moment it serves gems and NPM packages. It is designed to be small and embeddable in other Ruby web applications. You allocate a directory with packages and expose it through Paquette, all the while bringing your own authentication, limits, filtering and entitlements.
 
-Things are very in flux at the moment, but it may come in handy.
+In essence, it is a gem/NPM server for distributing commercial packages. I use
+it for https://shop.stanquette.nl and it is pretty neat!
 
-## Setup
+Things are somewhat in flux but Paquette is usable alright.
 
-1. Install dependencies:
-   ```bash
-   bundle install
-   ```
+## Basic setup
 
-2. Start the server:
-   ```bash
-   bundle exec puma
-   ```
+This is a Ruby library so you will need Ruby installed. To try a minimal setup, do a `bundle install` and start `bin/dev`. Try pushing a gem into Paquette and then adding the local server to your `Gemfile` - and then do a `bundle install` in your application for that custom gem.
 
-## Repository gating
+However, that is not how Paquette is primarily meant to be used. It is meant to be integrated into a larger Rails or Rack application, and otherwise bent to your will in various (slightly unhinged) ways. Just a few use cases Paquette was designed for:
 
-Paquette is built on the premise that you can have a corpus of libraries you offer, and deduce - from the `Authorization` HTTP header or by other means - which packages a user may download. Only the packages they have access to get included in the API responses - version lists, checksum lists and so on.
+- Hosting your internal package registry (why get the jFrog and the kitchen sink?)
+- Distributing your packages under specific authentication
+- Distributing your packages with licensing
 
-`Paquette::GemServer` itself is dumb: it takes one repository object and routes every read, push, and yank through it. You build the stack per request by nesting plain constructors — no DSL, no callbacks registered on the server, just wrappers wrapping wrappers. For example, inside a Rack endpoint:
+The key Paquette concepts are:
+
+- The serving app object - this is what poses as a gem/NPM repo. It's a Rack app.
+- The repository object - that's the frontend for your stored packages. The basic implementation just exposes a dir.
+- The repository wrappers - that's how you can control fulfillment from Paquette.
+
+A very basic example: in your Rails app, define a Paquette server with gems inside your app's `storage/`:
+
+```ruby
+# routes.rb
+
+gem_dir = Rails.root.join("storage/gems")
+gem_repo = Paquette::GemServer::DirectoryGemRepository.new(gem_dir)
+gem_server = Paquette::GemServer.new(gem_repo)
+
+# Both package servers require a separate hostname
+constraints(->(req) { req.host.start_with?("gem.") }) do
+  mount gem_server, at: "/"
+end
+```
+
+Call your app with `gem.localhost:3000` (or whichever other port you use) and you should see the Paquette placeholder screen. You can then push gems into your Rails app using the `gem.localhost:3000` source URL. Setup for NPM is similar.
+
+The `Paquette::GemServer::DirectoryGemRepository` is a repository. The `Paquette::GemServer` is the Rack app that serves a repository.
+
+## Repository wrappers
+
+You can make your package server do interesting things by wrapping your `repo`. For example, to disallow gem pushes:
+
+```ruby
+# ...the rest as above
+gem_repo = Paquette::GemServer::ReadonlyRepository.new(gem_repo) # forbids yank and push
+gem_server = Paquette::GemServer.new(gem_repo)
+```
+
+You can also create your own entitlement checks. This allows you to customize which gems get offered to a specific user:
+
+```ruby
+# ...the rest as above
+gem_repo = Paquette::GemServer::ReadGatedRepository.new(gem_repo, gate_key: Current.license.cache_key) do |name:, version: nil|
+  lic = Current.license
+  lic.package_names.include?(name)
+end
+gem_server = Paquette::GemServer.new(gem_repo)
+```
+
+Now the holder of the `license` stored in your `ActiveSupport::Current` for the request will only receive the gems included in their license's `package_names` list. And you can limit version access as well. This affects serving the actual packages, serving the indexes and anything else.
+
+The `gate_key:` (the npm `ReadGatedRepository` takes one too) is optional and is only used for HTTP caching. Broadly speaking, it is the user or license ID which alters the ETag of the response. If you pass none the response won't be cacheable - see [HTTP caching](#http-caching).
+
+### Package personalization
+
+Paquette includes a _personalization_ wrapper which allows every package to be marked with license details of the downlaoder. See it as a "soft-DRM" in the style of Pragmatic Programmers back in the day.
+
+> [!IMPORTANT]
+> Personalizing a package changes its checksum, and the checksum may become unique for every license. Often, this is exactly what you want. However, if you _change_ how you personalize a package for a specific user, their package manager may detect the changed checksum and assume the package has been tampered with. So - if you do personalize, either let your users know that package checksums will change when you alter the personalization flow, or never change how packages get personalized.
+>
+> Personalizing packages thus has security implications.
+
+```ruby
+# ...the rest as above
+gem_repo = Paquette::GemServer::Personalizer.new(
+    gem_repo,
+
+    # Gets injected into the gemspec metadata as "paquette.license_key"
+    license_key: Current.license.serial,
+
+    # The cache's name for "who this gem is baked for" - anything the
+    # personalized contents depend on has to be part of it, or a cached
+    # artifact will be reused after the personalization has changed
+    personalization_key: Current.license.holder_name,
+
+    # Apply blanket edits to a specific magic comment in all Ruby files
+    magic_comment_replacements: {"# license: " => Current.license.serial},
+
+    # Inject files when repackaging the gem, hash of paths to contents
+    files: {"THANK-YOU.txt" => "Thank you for using this library!"}
+)
+gem_server = Paquette::GemServer.new(gem_repo)
+```
+
+All the packages passing through personalization will be repackaged for this specific `personalization_key` and cached. The initial request for the package index may be slow if checksums and sizes have to be provided (that depends on the API endpoint and the registry used).
+
+### Cooldown
+
+There is a _cooldown_ wrapper, which withholds versions that were published less than a given interval ago:
+
+```ruby
+# ...the rest as above
+cooldown_seconds = 7 * 24 * 60 * 60
+gem_repo = Paquette::GemServer::CooldownRepository.new(gem_repo, interval: cooldown_seconds)
+gem_server = Paquette::GemServer.new(gem_repo)
+```
+
+A version that is still cooling does not appear in `/names`, `/versions`, `/info/`, `/specs.4.8`, `/latest_specs.4.8` or `/prerelease_specs.4.8`, and it does not download - it 404s, as if the gem version were not present.
+
+> [!IMPORTANT]
+> By default the publication date is `spec.date`, the date recorded in the gemspec - which is **publisher-controlled and records the build time, not the moment the gem arrived on your server.** A gem built in March and pushed in June is already past a seven-day cooldown when it lands.
+
+The default is `spec.date` because it is a property of the immutable gem bytes.
+
+## Wiring it all together
+
+Since Paquette is very modular and is supposed to be assembled per use case, you will likely want to construct your own Rack application to hold the bits together:
 
 ```ruby
 def call(env)
@@ -44,26 +147,18 @@ def call(env)
 end
 ```
 
-Each line adds one capability:
+The NPM server is assembled exactly the same way, out of the same kind of parts - see [Usage for NPM packages](#usage-for-npm-packages).
 
-- `DirectoryGemRepository` reads `.gem` files from disk and accepts pushes and yanks.
-- `ReadGatedRepository` wraps a repository and filters every name/version through the block — unauthorized gems simply stop existing as far as the server is concerned. It also refuses writes outright: if you gate reads, you are saying this caller is not the right party to mutate the corpus, so `add_gem`/`yank_gem` raise `WriteNotAllowed` (which the server turns into a 403).
-- `Personalizer` wraps a repository and rewrites each served `.gem` on the fly to embed the user's license key.
+## Hostname limitations
 
-Because the wrappers are plain Ruby objects composed at the call site, per-user state (the `user` variable) is captured by ordinary closures. Drop a wrapper to disable that layer; add another by slotting in one more constructor. Whether the server will accept pushes and yanks is decided by what you build — wrap the base repo in `ReadGatedRepository` and writes are blocked; hand the server a bare `DirectoryGemRepository` (or your own wrapper that permits writes) and they go through.
+The server will run on `http://localhost:9292` by default. Note that the NPM registry and the RubyGems registry have to live on separate domains - so they will respond on whichever domain is passed in that has `gem.` or `npm.` as first subdomain. If your OS supports `.localhost` TLDs, you can access `gem.whatever.localhost:9292` and it will respond.
 
-The server will run on `http://localhost:9292` by default. Note that the NPM registry and the Rubygems registry have to live on separate domains - so they will respond on whichever domain is passed in that has `gems.` or `npm.` as first subdomain. If your OS supports `.localhost` TLDs, you can access `gems.whatever.localhost:9292` and it will respond.
+## Publishing gems into Paquette
 
-## Usage for gems
+You can publish gems two ways.
 
-Paquette contains a gem server. This is a separate Rack app which you can use without the NPM server, for example - inside of your Rails app. You can interact with it using `gem` commands, as if it were any other gem server, or by just placing stuff on the filesystem.
-
-### Publishing gems into Paquette
-
-You can publish gems two ways. 
-
-1. Place `.gem` files in the `gems/` directory. The filename should follow the format: `gemname-version.gem`
-2. Actually do a `gem push` - from your shell, do `gem push --host http://gems.localhost:9292 pkg/your-gem-0.1.0.gem` 
+1. Place `.gem` files in a per-gem directory under `gems/`, as `gems/gemname/gemname-version.gem`
+2. Actually do a `gem push` - from your shell, do `gem push --host http://gem.localhost:9292 pkg/your-gem-0.1.0.gem`
 
 Note that Paquette will use whichever auth you wrap it with - that is to say, in the default dev setup - _none._ I told you it is slightly unhinged.
 
@@ -72,19 +167,205 @@ Note that Paquette will use whichever auth you wrap it with - that is to say, in
 To install gems from Paquette, set it as source in your Gemfile - and provide auth for whichever auth mechanism you wrap it with:
 
 ```ruby
-gem "private_algos", source: "https://tok_998218907784:x-oauth-basic@gems.paquette.acme.com"
+gem "private_algos", source: "https://tok_998218907784:x-oauth-basic@gem.paquette.acme.com"
 ```
 
-### API Endpoints
+The RubyGems API in Paquette supports the following endpoints:
 
 - `GET /` - Repository info
-- `GET /api/v1/dependencies` - Gem dependencies
+- `GET /api/v1/dependencies` - Gem dependencies, as Marshal
+- `GET /api/v1/dependencies.json` - the same thing as JSON
 - `GET /api/v1/versions` - Available gem versions
 - `GET /api/v1/names` - Available gem names
 - `GET /api/v1/search.json` - Search gems
+- `GET /specs.4.8`, `GET /specs.4.8.gz` - Legacy Marshal index of every released version
+- `GET /latest_specs.4.8`, `GET /latest_specs.4.8.gz` - The newest release of each gem
+- `GET /prerelease_specs.4.8`, `GET /prerelease_specs.4.8.gz` - Every prerelease version
+- `GET /names`, `GET /versions`, `GET /info/{gemname}` - Compact index
+- `GET /quick/Marshal.4.8/{gemname-version}.gemspec.rz` - Marshalled gemspec
 - `GET /gems/{gemname-version.gem}` - Download gem file
 - `POST /api/v1/gems` - Upload gem (basic implementation)
+- `GET /specs.4.8`, `GET /latest_specs.4.8` (and their `.gz` variants) - the legacy Marshal indexes
+- `GET /names`, `GET /versions`, `GET /info/{gemname}` - the compact index
+
+### Platform gems
+
+A gem built for a platform is stored, downloaded and indexed under the filename RubyGems gives it - `nokogiri-1.16.0-java.gem` - so the same name and version can exist for `ruby`, `java` and `arm64-darwin` side by side, and each is downloaded and yanked on its own.
+
+Where the platform ends up in a response depends on the format, Paquette follows Rubygems conventions.
+
+`gem push` files each build under that same RubyGems filename, so the three builds of one release are three artifacts rather than one pushed three times. What follows from that:
+
+### URL filtering in package metadata
+
+Most of what that endpoints return comes straight out of the uploaded gemspec, so it is written by whoever pushed the gem. Two fields are filtered before they go out: `homepage`, and every `*_uri` key in `metadata`. Each is served only when it is an absolute `http`/`https` URL with a host, and replaced with `""` (or dropped, for a metadata key) when it is not — `gem build` only warns about a `javascript:` or `data:` homepage, so a crafted gem can carry one. The npm packument's top-level `homepage` is filtered the same way; its `repository` and `bugs` are not, because `git://` and `git+ssh://` are legitimate there, and neither are the per-version documents under `versions`, which pass `package.json` through whole.
+
+Everything else — `authors`, `summary`, `description`, `info`, the non-URI keys of `metadata` — is passed through unchanged and is still uploader-controlled text. If you render any of it in a page of your own, escape it there.
+
+## Usage for NPM packages
+
+The NPM server is built the same way as the gem server and out of the same kind of parts: one repository object, wrapped in as many layers as you want, handed to a Rack app.
+
+```ruby
+def call(env)
+  username = env["REMOTE_USER"]
+  user = User.where(login: username).first!
+  packages_dir = Rails.root.join("packages", "npm").to_s
+
+  repo = Paquette::NpmServer::DirectoryNpmRepository.new(packages_dir)
+
+  repo_with_gating = Paquette::NpmServer::ReadGatedRepository.new(repo) do |name:, version: nil|
+    user.license.package_names.include?(name)
+  end
+
+  repo_with_gating_and_personalization = Paquette::NpmServer::Personalizer.new(
+    repo_with_gating,
+    license_key: user.license_key,
+    magic_comment_replacements: {"// paquette_license_info" => user.license_key},
+    files: {"LICENSE.txt" => user.rendered_license}
+  )
+
+  Paquette::NpmServer.new(repo_with_gating_and_personalization).call(env)
+end
+```
+
+The layers mean the same things they do for gems, with a few quirks:
+
+- A gated `latest` follows the newest version the caller is entitled to, not the newest version that exists. Otherwise `npm install pkg` would resolve to a version the very next request refuses to serve.
+- `latest` never points at a prerelease while a stable release exists, which is what npm's own registry does.
+
+### Where packages live on disk
+
+One directory per package, with a scope as an ordinary directory above it:
+
+```
+packages/npm/lodash/lodash-4.17.21.tgz
+packages/npm/@acme/widgets/widgets-1.0.0.tgz
+```
+
+The scope stays on the package name but is dropped from the filename, exactly as in the tarball URLs npm follows (`/@acme/widgets/-/widgets-1.0.0.tgz`).
+
+### Personalization and integrity with NPM packages
+
+npm records a `dist.integrity` hash for every version and refuses to install a tarball whose bytes do not match. A registry that rebuilds a tarball to serve it therefore has to rebuild it to *the same bytes* it published a hash for. Thus the same restriction applies as for Rubygems packages.
+
+Magic comment replacements forces replacements to be on one line. This is done so that if your package contains sourcemaps the offsets of the sourcemap do not shift - and the sourcemap thus won't have to be rewritten.
+
+### Publishing NPM packages into Paquette
+
+1. Place `.tgz` files in the layout above. The filename should be `name-version.tgz`, scope excluded.
+2. Actually do an `npm publish` - from your shell, do `npm publish --registry http://npm.localhost:9292`.
+
+`npm unpublish` works too. An unpublished version leaves a `.tgz.tomb` behind, which stops that exact version being republished with different contents later - the same trick the gem side uses for yanks.
+
+### Consuming packages provided by a Paquette server
+
+Point npm at it and provide auth for whichever mechanism you wrapped it with:
+
+```
+# .npmrc
+@acme:registry=https://npm.paquette.acme.com
+//npm.paquette.acme.com/:_authToken=tok_998218907784
+```
+
+### NPM API Endpoints
+
+- `GET /` - Repository info
+- `GET /-/ping` - Liveness
+- `GET /-/whoami` - The identity your auth wrapper resolved
+- `GET /{package}` - Package metadata document
+- `GET /{package}/-/{name-version.tgz}` - Download tarball
+- `GET /-/package/{package}/dist-tags` - Read dist-tags
+- `PUT /-/package/{package}/dist-tags/{tag}` - Point a dist-tag at a version
+- `PUT /{package}` - Publish
+- `PUT /{package}/-rev/{rev}` - Document update (how `npm unpublish` removes versions)
+- `DELETE /{package}/-rev/{rev}` - Unpublish a package
+- `DELETE /{package}/-/{name-version.tgz}/-rev/{rev}` - Unpublish one version
+
+## HTTP caching
+
+Paquette does not cache anything itself. It emits correct HTTP cache headers and answers conditional requests, and leaves the actual caching to a cache you put in front of it. I usually use `Rack::Cache` which is fairly reliable. You can put a CDN in front of Paquette if you know that it handles the `Vary: Authorization` bits correctly.
+
+Paquette will try to emit sensible ETags for most endpoints, and there is support for partial responses.
+
+### Putting a cache in front
+
+Under plain Rack, with [rack-cache](https://github.com/rtomayko/rack-cache):
+
+```ruby
+# config.ru
+require "rack/cache"
+
+use Rack::Cache,
+  metastore: "file:tmp/cache/rack/meta",
+  entitystore: "file:tmp/cache/rack/body",
+  verbose: false
+
+run Paquette::GemServer.new(gem_repo)
+```
+
+Under Rails, `config.action_dispatch.rack_cache = true` does the same. Put your authorization *below* the cache, next to Paquette, so it runs on every request the cache does not answer.
+
+On an open registry, anonymous downloads are then served out of the cache and index documents are revalidated against Paquette with a cheap 304. Authorized, gated and personalized responses are never stored; each of those requests reaches your gate and Paquette every time, and the client's own cache still revalidates cheaply. If the server-side cost of those is what you are after, cache inside your application - the personalizer already keeps repacked artifacts on disk.
+
+### Caching validators
+
+Paquette allows the repo wrappers to alter the validator (the ETag). The validator itself is derived from the whole wrapper stack, not just from the corpus, wrappers are written to contribute to the ETag in a meaningful way.
+
+A `nil` validator propagates outward, so a `Personalizer` wrapped around a keyless gate emits no `ETag` either.
+
+## The index page
+
+You can customize the user-visible index page of your package server by supplying a Rack app that will serve it, like so:
+
+```ruby
+# Serve our index page
+Paquette::GemServer.new(repo, placeholder_app: Paquette::IndexPage.new("Gems for Stanquette staff. Ask Julik for a token."))
+
+# ...or a redirect
+Paquette::NpmServer.new(repo, placeholder_app: ->(_env) { [302, {"location" => "https://docs.example.com"}, []] })
+Paquette::NpmServer.new(repo, placeholder_app: ->(_env) { [404, {}, []] })  # no root at all
+```
+
+## Mounting under a path prefix (namespaces)
+
+Both servers can be mounted under a prefix, which is how [gem.coop spells a namespace](https://gem.coop/updates/6/). In Rack only a `map` and nothing else:
+
+```ruby
+Rack::Builder.new do
+  map("/@acme") { run Paquette::GemServer.new(acme_repo) }
+  map("/@beta") { run Paquette::GemServer.new(beta_repo) }
+end
+```
+
+A namespace per repository is also a namespace per wrapper stack: each mount can have its own gating and personalization, since it is its own `GemServer` around its own repository.
+
+## Instrumentation
+
+Everything expensive in Paquette is wrapped in a [Measurometer](https://rubygems.org/gems/measurometer) block, so that you know how long things take. Measurometer does nothing until a driver is attached, so you may need to configure it:
+
+```ruby
+Measurometer.drivers << Appsignal
+```
+
+Metric paths that Paquette outputs are all prefixed with `paquette.` - the rest should be fairly self-explanatory.
+
+## Running the tests
+
+```bash
+bundle exec rake test
+```
+
+Most of it is ordinary Ruby, but we also have a few end-to-end tests, that require Docker and Node. You don't need either for using Paquette though.
+
+RubyGems' own compact-index conformance suite runs separately, because it shells out to the `gem_server_conformance` RSpec binary:
+
+```bash
+bundle exec rake test:conformance
+```
+
+It boots a gem server over a bare `DirectoryGemRepository` and drives the suite against it over HTTP. `test/conformance/rspec_exclusions.rb` lists the examples Paquette knowingly does not pass and why — read it before concluding that a green run means full conformance. Without the `gem_server_conformance` gem installed the test skips; set `PAQUETTE_REQUIRE_CONFORMANCE=1` to turn that skip into a failure, which is what CI wants.
 
 ## License
 
-Paquette is offered under the terms of the [O'Sassy license](https://osaasy.dev/) - basically, don't make it into your own product. Use it to sell your libraries. And godspeed!
+Paquette is offered under the terms of the [O'Sassy license](https://osaasy.dev/) - basically, **don't make it into your own product or a service.** Use it to sell your libraries. And godspeed!

data/lib/paquette/cache_validation.rb

@@ -0,0 +1,62 @@
+require "digest"
+
+# How a repository stack names itself to an HTTP cache. Both servers wrap a
+# base repository in per-caller gates and personalizers, so a validator
+# derived from the corpus alone would let one licensee's cached response
+# satisfy another's — the digest rules live here, once, for both servers.
+module Paquette::CacheValidation
+  module_function
+
+  # The stack's validator. Fails closed (nil) for an object that has never
+  # heard of the protocol.
+  #
+  # @param repository [Object]
+  # @return [String, nil]
+  def validator_of(repository)
+    repository.cache_validator if repository.respond_to?(:cache_validator)
+  end
+
+  # Whether two callers asking the stack the same question can get different
+  # answers — which decides whether an anonymous response may be `public`.
+  # True for anything that cannot say: being wrong the other way only costs
+  # a cache hit.
+  #
+  # @param repository [Object]
+  # @return [Boolean]
+  def varies_by_caller?(repository)
+    return true unless repository.respond_to?(:varies_by_caller?)
+
+    repository.varies_by_caller? != false
+  end
+
+  # How a wrapper mixes itself into the validator of what it wraps. nil in
+  # either position gives nil out — the fail-closed rule the whole protocol
+  # rests on. The layer name is in the digest so that gating with key "k"
+  # and personalizing with key "k" cannot land on the same string.
+  #
+  # @param inner_validator [String, nil] the wrapped stack's validator
+  # @param layer [String] a constant naming the wrapping layer
+  # @param key [String, nil] what distinguishes this instance of the layer
+  # @return [String, nil]
+  def derive_validator(inner_validator, layer, key)
+    return nil if inner_validator.nil? || key.nil?
+
+    Digest::SHA256.base64digest([inner_validator, layer, key].join("\0"))
+  end
+
+  # The validator for one response. Returns nil, meaning "emit no ETag",
+  # whenever the stack refused to name itself.
+  #
+  # @param repository [Object]
+  # @param resource_parts [Array<String>] what distinguishes this endpoint's
+  #   body from another's over the same corpus
+  # @param weak [Boolean]
+  # @return [String, nil]
+  def etag_for(repository, *resource_parts, weak: false)
+    base = validator_of(repository)
+    return nil if base.nil?
+
+    tag = %("#{Digest::SHA256.base64digest([base, *resource_parts].join("\0"))}")
+    weak ? "W/#{tag}" : tag
+  end
+end