ldap-rest 1.3.0 → 2.0.1

package/docs/architecture/adr/0002-authentication-dispatch.md

@@ -0,0 +1,43 @@
+# 0002 — Authentication dispatch
+
+## Context
+
+One server can host populations that authenticate differently: administrators
+behind a token, users behind OpenID Connect, machine clients behind HMAC. An
+authentication plugin declares the path prefixes it claims. A request must run
+the authentication that owns its path, and only that one: running two
+unrelated authenticators on one request composes as an AND that no credential
+can satisfy. A request no plugin claims must not silently become open without
+saying so.
+
+## Decision
+
+`DM` mounts one dispatcher layer, once, at the boundary between the protection
+plugins and the routes. For each request it selects the plugins whose prefix
+is the longest match. Several plugins sharing that winning prefix all run, in
+registration order, and every one must let the request through. When no
+plugin claims the path, the unscoped plugins (no prefix) run; when there is
+none, the request passes through unauthenticated, outside any request
+context. A plugin's `afterChain` runs after the whole chain and before the
+route. `runForRequest` then wraps everything downstream of a request an
+authenticator let through, and the dispatcher records which plugins vouched
+for the request in `req.authenticators`.
+
+`warnUnauthenticatedRoutes` lists, at startup, the routes that no
+authentication plugin covers. An unscoped authenticator guards everything and
+silences the warning.
+
+## Consequences
+
+- Two authenticators on one prefix require both credentials. That is
+  legitimate — a session and a token — but it fails closed when accidental.
+- Scoping authentication to a prefix opens every route outside it; the
+  startup warning is the only signal, and it names the routes.
+- `afterChain` is the only hook that sees all the selected authenticators'
+  verdicts, so an answer that depends on several of them belongs there, not
+  in a plugin's own middleware.
+- An authorization plugin can tell whose request it judges from
+  `req.authenticators`; see
+  [authorization composition](0003-authorization-composition.md).
+
+See `src/bin/index.ts` (`dispatchAuth`) and `src/lib/auth/base.ts`.

package/docs/architecture/adr/0003-authorization-composition.md

@@ -0,0 +1,59 @@
+# 0003 — Authorization composition
+
+## Context
+
+Every plugin that judges directory operations registers the same request
+hooks (`ldapsearchrequest`, `ldapaddrequest`, `ldapmodifyrequest`,
+`ldapdeleterequest`, `ldaprenamerequest`). Chained hooks run their
+subscribers in order and stop at the first refusal, so two loaded
+authorization plugins compose as an AND: the first to refuse answers. That
+rule is sound when it is meant. It is a refusal nobody can explain when it is
+not: `authzPerBranch` next to `authzDynamic` judges a machine token by a
+branch configuration written for administrators and refuses a write the
+token's ACLs grant.
+
+## Decision
+
+A plugin with the `authz` role that registers LDAP hooks is a _judge_. It
+judges the requests authenticated by the plugins named in `--authz-for`, or
+everyone by default. A plugin that authenticates as well judges only its own
+requests: its hooks read the credential it verified.
+
+`assertAuthzComposition` runs once every configured plugin is loaded, and
+again for each plugin registered later. It refuses:
+
+- an `authz_for` naming a plugin that is not a loaded authenticator — the
+  plugin would judge nobody. A server-wide `--authz-for` that no loaded plugin
+  reads only warns, as does an `authz_for` on a plugin that also
+  authenticates;
+- two judges whose populations overlap on an authenticator — the same one, or
+  one of them judging everyone — unless `--authz-combine` makes the AND
+  deliberate.
+
+Two judges whose authenticators can run on the same prefix, without sharing an
+authenticator name, are reported, not refused: asking for two credentials is
+already a decision, and a request carrying both is judged by both. A judge on
+`authzPerRoute` gates URLs and registers no LDAP hook, so it is not part of
+this AND — route plus branch is a documented combination.
+
+The request hooks of judges run before every other plugin's request hook and
+`ldapsearchfilter` keeps the load order. All the LDAP hooks compose as an
+AND, including the filters: two filters narrow the result twice.
+
+## Consequences
+
+- A deployment with two branch models on the same population no longer starts
+  unless it says `--authz-combine`; the error names both plugins.
+- Naming authenticators in `--authz-for` requires stable instance names, so
+  the configuration that scopes authentication and the one that scopes
+  authorization must agree.
+- A plugin registered by an embedding host is checked before its routes and
+  hooks exist, so a refused plugin leaves nothing behind.
+- Judges run first, in their load order. The chain stops at the first refusal,
+  so a judge after a refusing one does not run; the refusal is the answer.
+  No judge is skipped while the request is still allowed.
+- A judge that also authenticates judges only its own requests; a request it
+  did not vouch for passes its hooks untouched, because the absence of its
+  credential is not a refusal it makes.
+
+See `src/lib/authz/composition.ts` and `src/config/args.ts`.

package/docs/architecture/adr/0004-fail-closed-request-context.md

@@ -0,0 +1,45 @@
+# 0004 — Fail-closed request context
+
+## Context
+
+An authorization judge decides from the request (`req.user`, its
+authenticators, its path). But a directory operation is often made deep in a
+plugin that never received the request: a provisioning step incremented from a
+hook, a consistency repair, a route helper that searches before it writes.
+Passing the request down every call is easy to forget, and a forgotten request
+used to mean an operation nobody judged — a hole that is invisible until
+someone writes through it.
+
+## Decision
+
+The authentication dispatcher runs everything downstream of an authenticated
+request inside `runForRequest(req, next)`, an `AsyncLocalStorage`. Every
+directory operation defaults to `currentRequest()`, so a plugin that forgets
+the request still gets the caller's identity and is judged as them. Operations
+the server makes for itself run inside `outsideRequest()`, which `ldap.system`
+uses: it is the explicit escape from judgement, not a default.
+
+The log context is a second `AsyncLocalStorage`, kept apart: a line written as
+the server, outside the request, still names the request that made the server
+act. `outsideAnyRequest()` leaves both, for work that outlives a request (a
+pooled connection, a process-wide client).
+
+Hooks are the one place that deliberately leaves the context: the bus runs
+every subscriber inside `outsideRequest()`, so a hook's own writes are the
+server's and not the caller's. A hook that judges or acts for the caller has
+the request in its arguments and names it.
+
+## Consequences
+
+- A plugin that means to act as the server must say so with `ldap.system`;
+  forgetting the request fails closed instead of opening a hole.
+- A test that calls a plugin's directory operations directly must wrap them in
+  `runForRequest` to reproduce what the server does, or get the unjudged
+  behaviour of `ldap.system`.
+- Asynchronous work started inside the request inherits it, so a background
+  task that should not act as the caller has to leave the context first.
+- Errors and logging inherit the request id, which makes a write traceable to
+  the request that caused it.
+
+See `src/lib/requestContext.ts` and `src/lib/ldapActions.ts`
+(`forRequest`, `system`, `currentRequest`).

package/docs/architecture/adr/0005-hook-contract.md

@@ -0,0 +1,54 @@
+# 0005 — Hook contract
+
+## Context
+
+Plugins have to talk to each other without importing each other. A refusal
+has to be able to stop a write; a notification has to be able to run after the
+write, without holding the request open or turning a subscriber's failure into
+the caller's failure. One kind of hook cannot do both.
+
+## Decision
+
+There are two kinds, told apart by what a subscriber answers, and a third
+notification mode:
+
+- **Chained** (`chain(name, value)`). The value goes from one subscriber to
+  the next; the last answer is the result. A subscriber that throws stops the
+  chain and the error travels to the caller: a refusal is a refusal. Several
+  arguments are passed as one tuple.
+- **Notified** (`notify(name, ...args)`). Subscribers run one after the other,
+  outside the request that caused the write, and are not awaited. A failing
+  subscriber is logged with its plugin and hook, counted in the hook failure
+  metrics, and the next one still runs.
+- **Awaited notification** (`notifyAndWait`). As `notify`, but the caller waits
+  for the subscribers. A failure is still logged and swallowed.
+- **Strict notification** (`notifyStrict`) for the rare failure that is part
+  of the answer. Every subscriber runs; the first failure is raised once they
+  all returned.
+
+Every hook is declared in `src/hooks.ts` with its arguments and its answer. A
+plugin subscribing to an undeclared hook does not compile, and `npm run
+check:hooks` keeps the documentation in step. Notified hooks take their
+arguments one by one; chained hooks take one value, a tuple when there are
+several.
+
+The request hooks of authorization plugins run first, in load order. The
+other hooks run in plugin load order.
+
+## Consequences
+
+- A notified hook cannot refuse a write; a plugin that needs to refuse
+  subscribes to the request hook instead.
+- A chained subscriber must answer the value it was given, changed or not: an
+  `undefined` answer becomes the next subscriber's input.
+- A subscriber of a notified hook written `([dn, entry], ctx) => …` compiles
+  without annotation — its parameter takes the declared type, a `string`, and
+  a string can be destructured — and then reads the first character of the
+  DN. The documentation repeats the shape of each hook for this reason.
+- A "done" hook fires once the operation is over, including a modify that had
+  nothing to apply; a write refused by a request hook or by the directory
+  never reaches it. The "end" hook fires all the same, once the done
+  subscribers returned. Keying state by the operation number is what tells
+  two writes of one DN apart.
+
+See [hooks](../../plugin-development/hooks.md) and `src/lib/hookBus.ts`.

package/docs/architecture/adr/0006-search-cache.md

@@ -0,0 +1,49 @@
+# 0006 — Search cache
+
+## Context
+
+Reading one entry by DN is the most frequent operation a route makes: before a
+write, before an authorization decision, to render a page. A directory round
+trip per read is measurable, and a cache in front of it is easy to add and
+hard to keep honest. Serving a stale entry after a write is worse than the
+round trip.
+
+## Decision
+
+`SearchCache` caches non-paged, base-scope searches — single-entry lookups by
+DN, with whatever filter — keyed by base, scope, filter, attributes, options
+and values mode, in an LRU bounded by `--ldap-cache-max`. It is **off by
+default** (`--ldap-cache-ttl=0`): no deployment has run with it on, and
+turning it on for everyone would be a silent behaviour change.
+
+`ldapActions` drops, around every write it makes through this process:
+
+- the reads of the entry written and of its subtree (a rename or a move
+  re-parents a whole branch);
+- the `memberOf` of the members a group change adds or removes;
+- the reads of entries naming the entry renamed or deleted, because the
+  directory's `memberof` and `refint` overlays rewrite them.
+
+Entries are dropped before the directory call as well as after it, and a
+generation counter stops a read that started before the write from storing
+its stale answer afterwards. DNs are compared the way the directory does:
+case, spacing, escapes, aliases and multi-valued RDN order do not matter.
+
+Writes made elsewhere — another replica, an LSC synchronisation, a hand-run
+`ldapmodify` — do not invalidate anything. Each process caches on its own.
+
+## Consequences
+
+- Enable the cache only when this service is the only writer, or when serving
+  an entry up to `--ldap-cache-ttl` seconds old is acceptable.
+- A subtree or one-level read, or a paged search, is never cached; the cache
+  does not accelerate listings. A base-scope read with a filter is cached,
+  keyed by that filter.
+- The cache does not lift a miss: a read of an absent DN throws and caches
+  nothing.
+- The invalidation rules follow the overlays this project documents
+  (`memberof`, `refint`); a deployment with an overlay that rewrites other
+  entries must accept a miss or leave the cache off.
+
+See [configuration](../../usage/configuration.md#the-search-cache) and
+`src/lib/ldap/searchCache.ts`.

package/docs/architecture/adr/0007-trash-and-tombstone.md

@@ -0,0 +1,69 @@
+# 0007 — Trash and tombstone
+
+## Context
+
+Deleting an entry destroys the identifiers it held: a mail address, a login, a
+DN. An administrator who deletes the wrong account cannot undo it, and a
+deleted account's address can be taken by a new one immediately — which, for a
+mail system, means the new account receives the old one's mail. The directory
+has no undelete of its own.
+
+## Decision
+
+Two plugins answer a delete at the `ldapdeleterequest` hook, and a deployment
+picks one per branch:
+
+- **`core/ldap/trash`** moves the entry, by an atomic `modifyDN`, to a trash
+  branch under `--trash-base`, overwriting an earlier entry with the same RDN
+  there; unless `--trash-add-metadata=false`, it then replaces `description`
+  with the deletion date and the original DN, a failure there being logged
+  and not fatal. The native delete is cancelled. `--trash-watched-bases`
+  selects the branches; an empty list watches every branch.
+- **`core/twake/tombstone`** keeps the entry in place: one modify sets the
+  deleted attribute and the lock, the deletion date and the reason when their
+  attributes are configured, and removes the attributes the deployment names.
+  The identifier is reserved until an operator erases the tombstone, once the
+  deletion is old enough or immediately when forced. Erasing removes the entry
+  and its group memberships and announces nothing.
+
+The tombstone moves itself last in the `ldapdeleterequest` and
+`ldapaddrequest` chains (`hookBus.toLast`), so every plugin that may refuse
+has judged. The trash does not: it runs in load order, after the authorization
+judges but possibly before another plugin that can refuse a delete
+(`organizations`, `enterpriseRules`, `james`). A plugin that refuses a delete
+and loads after the trash refuses one that has already been moved. The
+tombstone is written by the server, not as the caller: the caller's delete
+right is what was judged.
+
+SCIM treats a tombstone as gone: its reads carry a not-deleted filter, so
+reads answer 404 and lists omit it; a create with the same identity erases
+the tombstone and creates a new entry. A non-SCIM delete of an existing
+tombstone writes nothing and fires `twakedeletionreplay`, so lifecycle events
+can replay a lost deletion. A SCIM DELETE never gets there: its existence
+check hides the tombstone and answers 404.
+
+The server refuses to start when the tombstone is loaded beside the trash and
+any watched branch overlaps a `--twake-tombstone-dn` pattern: the trash would
+move the entry before the tombstone saw the delete, and no tombstone would be
+written. An empty `--trash-watched-bases` watches every branch, so it overlaps
+any pattern and refuses too.
+
+## Consequences
+
+- A delete is no longer a delete. Code that lists a branch sees what the
+  trash or the tombstone left there, filtered by the plugin that owns the
+  branch.
+- Trash and tombstone cannot watch the same branch; the startup check names
+  the overlap.
+- A deployment that loads the trash must load it before the plugins that can
+  refuse a delete, or accept that a refusal can come after the entry moved.
+- A tombstone cannot be restored: clearing its deleted attribute by hand is
+  not a supported operation.
+- Trash moves the entry, so its DN changes and references to it are not
+  rewritten by the move beyond what the directory's `refint` overlay does.
+- Tombstones are erased by an explicit operator action, so a retention policy
+  has to run it.
+
+See [tombstone](../../usage/plugins/integrations/tombstone.md), the plugin
+list in [plugins](../../usage/plugins/README.md), and
+`src/plugins/ldap/trash.ts`, `src/plugins/twake/tombstone.ts`.

package/docs/architecture/adr/0008-withconfig-views.md

@@ -0,0 +1,58 @@
+# 0008 — Server views for plugins with overrides
+
+## Context
+
+The same plugin module can be loaded several times with different settings:
+`--plugin 'core/auth/token:admins:{…}'` and
+`--plugin 'core/auth/token:users:{…}'`. The plugin constructor receives the
+server, and reads its options from `this.server.config` and `this.config`.
+Those instances must see their own configuration while acting on the one
+server: the same Express app, the same hooks, the same LDAP connection and
+the same plugin registry. A second `DM` would be a second server.
+
+## Decision
+
+`DM.withConfig(config)` returns a view of the server built with
+`Object.create(Object.getPrototypeOf(this))` and `Object.assign(…, this,
+{ config })`. The own properties that hold objects — the app, the hooks, the
+LDAP actions, the plugin registry, the authenticators — are shared by
+reference. The configuration is `{ ...this.config, ...overrides }`, a shallow
+copy: overridden top-level keys are the plugin's alone, nested objects and
+arrays are still shared with the server. The loader constructs a plugin with
+overrides on such a view.
+
+A primitive field would be copied by value into every view, so state the views
+must share lives in an object: the operation counter (`_operations`, read
+through the `operationSequence` accessor) and the composition flag
+(`_compositionState`), as the dispatcher's own state does (`_authState`). A
+view created while the plugins load therefore sees the composition once the
+server has checked it.
+
+The copy is built on the prototype rather than by spreading `{...this}`,
+which copies own properties and leaves the methods behind: a plugin calling
+`this.server.anything()` would throw, and only when configured with
+overrides.
+
+## Consequences
+
+- A plugin loaded with overrides cannot change the server's configuration for
+  anyone else; mutating `this.config` affects its view alone, at the top
+  level.
+- Primitive fields are copied by value, not shared: what must be shared is
+  held in an object, as the operation counter and the composition flag are.
+  A plugin registered after `ready` through a view created while loading is
+  therefore checked against the composition like any other (before [#625] and
+  [#626], the counter and the flag were primitives, and neither held).
+- An own property added to the server after the view was created is not
+  visible to that view. State that must be shared has to live in an object
+  that already exists when the view is made.
+- A plugin that closes over `this.server` keeps the view for its lifetime,
+  which is the point: its options stay its own.
+- Tests may call `withConfig` to exercise option-dependent behaviour without
+  building a second server.
+
+See `src/bin/index.ts` (`withConfig`, `importPlugin`) and
+[plugin options](../../plugin-development/options.md).
+
+[#625]: https://github.com/linagora/ldap-rest/issues/625
+[#626]: https://github.com/linagora/ldap-rest/issues/626

package/docs/client-development/api/rest-api.md

@@ -1365,6 +1365,10 @@ const encoded = encodeURIComponent(dn);
 // Result: uid%3Djohn.doe%2Cou%3Dusers%2Cdc%3Dexample%2Cdc%3Dcom
 ```
 
+A path parameter is percent-decoded once, as RFC 3986 spells it: an
+identifier holding `%` is sent as `%25`, so `100%` is addressed as
+`/api/v1/ldap/users/100%25`.
+
 ### LDAP Attributes are Arrays
 
 LDAP attributes are always returned as arrays, even for single-valued attributes:

package/docs/development/contracts.md

@@ -0,0 +1,119 @@
+# Published contracts
+
+Other code depends on three things ldap-rest publishes. A change to one that
+is not additive breaks that code, so it needs a major release and the pull
+request says so before it merges.
+
+- **Configuration**: the CLI options and `DM_*` variables of
+  [Configuration](../usage/configuration.md) and of each plugin's page.
+- **REST API**: the routes, request bodies and answers of the
+  [REST API](../client-development/api/rest-api.md), their parameters,
+  responses (references resolved), statuses and response headers. The error
+  `code` values are reviewed by hand: the document types them as strings.
+- **Plugin API**: what the npm sub-path exports declare — `ldap-rest`,
+  `./hooks`, `./plugin`, `./ldapactions`, `./expressformatedresponses`, the
+  `./plugin-*`, `./abstract-*` and `./browser-*` entries — and the JSON
+  schema format of the `./schema-*` entries.
+
+[API stability](../README.md#api-stability) says what a minor or patch
+release promises. This page says how a change is detected, and what to do
+when it is not additive.
+
+## Committed snapshots
+
+Two snapshots are generated locally and checked in CI, so a change to a
+contract shows up as a diff instead of a surprise after the release:
+
+- `docs/usage/options.snapshot.json` lists every option's CLI name, plural
+  form, environment variable, type, default, kind, required flag and aliases.
+  Regenerate it with `npm run snapshot:options`; `npm run check:options`, part
+  of `npm run check`, fails when it is stale.
+- `openapi.json` is the REST API document the published reference is built
+  from. Regenerate it with `npm run generate:openapi`; `npm run check:openapi`
+  fails when it is stale.
+- `etc/api/*.api.md` are the API Extractor reports of the five entry points
+  #595 covers first — `ldap-rest`, `./plugin`, `./hooks`, `./ldapactions` and
+  `./expressformatedresponses` — one file per sub-path. Regenerate them with
+  `npm run snapshot:api`; `npm run check:api`, part of `npm run check`, fails
+  when one is stale. A report lists what the entry exports and the shapes of
+  the types those exports reference, including the types the entry does not
+  export itself, so a change to `RouteDefinition` or `OptionSet` shows up even
+  though a plugin cannot name them. The 66 `./plugin-*` entries come with the
+  modularization, see [the plan](modularization-plan.md).
+
+The snapshot records what the parser reads. A narrowed `choices` list, a
+raised `min` or a lowered `max` is checked when a plugin reads its options,
+not by the snapshot, and is reviewed by hand, as is a custom error `code`.
+
+## The pull request gate
+
+The `contracts` job of `.github/workflows/contracts.yml` generates the
+snapshots from the pull request and compares them with the base branch's
+committed ones. It reports:
+
+- additive changes — a new option, route or field — and lets them through;
+- non-additive changes, and fails unless the pull request carries the
+  `breaking` label.
+
+A change is non-additive when it removes or narrows what was published:
+a removed option or environment variable, a changed type, default, bound,
+`const`, enum or required flag, a request value that becomes an enum, a
+request object that stops accepting additional properties, a removed route,
+method, parameter, response status, response header or field, a request field
+that becomes required, a response that may now be null, changed security
+requirements, a removed union variant, a value an alternative of the type
+closes with an enum or a `const` where the type used to accept any, an
+untyped request value that becomes a union, an `allOf` that adds a
+constraint, or a removed declaration in `etc/api/` that a plugin can name —
+the exported one, or a member of an exported namespace. Renaming a type only
+reachable through a signature is not a break: nobody can write its name, and
+the declarations that use it are reported as changed. A changed declaration is
+printed for review and does not fail the job yet: telling an added optional
+member from a narrowed union needs more than a line diff, and comes with the
+`./plugin-*` reports. Renaming an option is a deprecation, not a removal, as
+long as the old CLI name and environment variable stay in its `aliases` and
+`envAliases`. Adding a union variant, a field or an optional parameter is
+additive, and so is a request union with an alternative that still accepts
+everything the base did — in a response the client reads every alternative,
+so a field the others drop is still a removal. A `oneOf` is the exception:
+it refuses a value two alternatives match, so an alternative as wide as the
+base narrows the others away, and the alternatives are not told apart from
+one another — a disjoint `oneOf` therefore asks for the label too.
+Documentation-only edits, such as a summary or description, are ignored.
+
+The job runs the scripts from the pull request, so a change to them is
+reviewed like any other code, as `.github/CODEOWNERS` requires.
+
+The `breaking` label records that the maintainers accepted the break, and the
+`CHANGELOG.md` section must say so: the release gate below reads the section,
+not the label, and refuses to ship the break as a minor or patch release. The
+workflow listens to label events, so adding the label runs the job again; it
+has its own concurrency group, so that event does not cancel the test matrix.
+
+To try the comparison locally against your base branch:
+
+```bash
+npx tsx scripts/contracts.ts --base origin/master
+```
+
+## The release gate
+
+`.github/workflows/release.yml` runs `scripts/check-release.ts` on the tag. A
+non-major tag whose `CHANGELOG.md` section lists `### Breaking Changes` is
+refused. For a 0.x version, a minor release counts as major, as semantic
+versioning lets it break. When the tag has no section of its own, the gate
+reads `## Unreleased`, so a tag cut before the section was renamed is still
+checked; with neither section it fails. When the CHANGELOG lists no earlier
+release, only a first `X.0.0` counts as major, so a trimmed file cannot open
+the gate.
+
+## Deprecating instead of breaking
+
+The window and the rules are in
+[CONTRIBUTING.md](../../CONTRIBUTING.md#deprecating-a-published-contract).
+The mechanics of an option alias are in
+[plugin options](../plugin-development/options.md#deprecating-an-option): the
+old CLI name and environment variable go in `aliases` and `envAliases`, and
+the server warns at startup while it still reads them. An array option's
+plural form has no alias slot: renaming it is a breaking change, so it waits
+for a major release.