ldap-rest 1.1.0 → 1.2.0

package/docs/usage/operations.md

@@ -0,0 +1,193 @@
+# Operations
+
+## Health and readiness
+
+Two routes answer outside the API prefix, without authentication, and are
+never cached:
+
+| Route          | Answers                                                       |
+| -------------- | ------------------------------------------------------------- |
+| `GET /healthz` | `200 {"status":"ok"}` as long as the process serves           |
+| `GET /readyz`  | `200` when the server can serve, `503` when it cannot (below) |
+
+`/readyz` answers a JSON body:
+
+| `status`      | Code | Meaning                                                                                                |
+| ------------- | ---- | ------------------------------------------------------------------------------------------------------ |
+| `starting`    | 503  | The plugins are still loading (a host serving before `ready`; the shipped binary opens the port after) |
+| `stopping`    | 503  | The server was told to stop and finishes what is in flight; a new connection is refused                |
+| `unavailable` | 503  | The directory does not answer a read of its root DSE in 2 s (`ldap: failed`)                           |
+| `ok`          | 200  | Everything answers                                                                                     |
+| `degraded`    | 200  | The directory answers, a plugin reports a resource of its own down                                     |
+
+```json
+{ "status": "degraded", "ldap": "ok", "plugins": { "rabbitmq": "degraded" } }
+```
+
+`plugins` lists the plugins that report their health; `failed` means the
+report threw or did not answer in time. Only the directory takes a replica
+out of rotation: a broken outbound service is visible in the body, and the
+server keeps serving what does not need it.
+
+The directory and the plugins are asked at once, so the answer takes 2 s at
+most. The probe reads the root DSE, which any bound account may: a wrong
+`--ldap-base` is not what it reports. The directory is probed on a connection of its own, bound as the
+server, outside the connection pool and the search cache. One answer serves
+5 seconds whatever the number of probes; the probes arriving meanwhile
+share it.
+
+Both routes are served before the authentication and the rate limiting,
+and `/readyz` names the loaded plugins that report: on an ingress exposing
+the server to the internet, keep the two paths to the cluster.
+
+The Helm chart probes both routes over HTTP (`livenessProbe` and
+`readinessProbe` in `values.yaml`).
+
+### What the shipped plugins report
+
+| Plugin           | `degraded` when                                                                                                                                                                                             |
+| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `storage`        | the backend refuses a `SELECT 1` or a `PING`, or the `file` backend's directory is not a directory this user may write in; the `ldap` backend has no answer of its own, the directory check above covers it |
+| `rabbitmq`       | a broker is configured and no connection to it stands: until the first publish or subscription opens one, and while the client reconnects                                                                   |
+| `james`          | James answers its `GET /healthcheck` with 503, one of its own checks unhealthy, or not within 1.5 s; a James whose checks are only degraded answers 200, so `ok` here                                       |
+| `migrationProxy` | the migration proxy answers its `GET /healthcheck` with an error, or not within 1.5 s                                                                                                                       |
+
+A check that leaves the probe to time out reads as `failed`.
+
+### For plugin authors
+
+A plugin reports its own resources with an optional `health()` method
+returning `ok` or `degraded`, called at most once every 5 seconds; it must
+answer within 2 s and never throw for an ordinary outage. The probe stops
+waiting at 2 s but cancels nothing: a plugin bounds its own call, as
+`james` does with its 1.5 s.
+
+## Request id
+
+Every response the API sends carries an `X-Request-Id` header: an
+identifier the server generates for the request, the same one its writes
+hand to the plugins and its log lines carry. An `X-Request-Id` the client sends is not taken: a
+caller could otherwise make the writes of two requests look like one.
+
+A proxy or an ingress generating an identifier of its own does not pass
+it on either: log the answer's `X-Request-Id` there to match its lines with
+the server's.
+
+Every line logged while a request is served carries `requestId`, the lines
+of the hooks it ran included, as a field of the JSON console output and as
+`[<id>]` after the level in the log file. The line of `core/weblogs` carries
+it too. A line without one was written outside any request: at startup, by
+a scheduled task, by a message consumer, by a connection the server keeps.
+Work shared between requests, a schema load or a readiness answer, logs
+under the request that started it.
+
+A line a hook writes carries `plugin`, the plugin that registered the hook,
+and `op`, the hook's name, the `Hook error` line of a hook that threw
+included: in the JSON console output as fields, in the log file as
+`[<plugin> <op>]` after the request id. A line naming another `plugin`
+itself keeps it, with no `op` from the hook, and reads `[<plugin>]` in the
+file wherever it is written. These are the labels of
+`ldap_rest_hook_failures_total`.
+
+A line carrying an `error`, the reason of a failure, has it as a field of
+the JSON console output and in parentheses after the message in the log
+file; the file keeps no field other than these.
+
+## Metrics
+
+`core/metrics` serves Prometheus metrics on `GET /metrics`, on
+`--metrics-port` (9464): a port of its own, outside the authentication,
+the rate limiting and the request id, to keep to the cluster or the host.
+
+```bash
+--plugin core/metrics --metrics-port 9464
+```
+
+| Metric                                        | Type      | Labels                      | Meaning                                                                                                                                                                                                                                                                                                         |
+| --------------------------------------------- | --------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ldap_rest_http_request_duration_seconds`     | histogram | `method`, `route`, `status` | Time to answer, by route template (`/api/v1/users/:id`; `none` unmatched or answered by a middleware, `core/static` included)                                                                                                                                                                                   |
+| `ldap_rest_ldap_operation_duration_seconds`   | histogram | `operation`, `result`       | Time the directory takes: `bind`, `search`, `add`, `modify`, `rename`, `delete`; `ok`, the LDAP result code of a refusal (`32`: no such entry), `error` when it did not answer                                                                                                                                  |
+| `ldap_rest_ldap_pool_wait_seconds`            | histogram |                             | Time to get a pooled connection, opening one included                                                                                                                                                                                                                                                           |
+| `ldap_rest_ldap_pool_connections`             | gauge     | `state`                     | Pooled connections, `busy` or `idle`                                                                                                                                                                                                                                                                            |
+| `ldap_rest_ldap_pool_waiting`                 | gauge     |                             | Operations waiting for a connection, the pool being full                                                                                                                                                                                                                                                        |
+| `ldap_rest_ldap_pool_max`                     | gauge     |                             | `--ldap-pool-size`                                                                                                                                                                                                                                                                                              |
+| `ldap_rest_ldap_search_cache_total`           | counter   | `result`                    | Cacheable searches, `hit` or `miss`; nothing when the cache is off                                                                                                                                                                                                                                              |
+| `ldap_rest_hook_failures_total`               | counter   | `hook`, `plugin`            | Notification hooks (`*done`, `onLdap*`) that threw, which the server logs and goes on                                                                                                                                                                                                                           |
+| `ldap_rest_outbound_request_duration_seconds` | histogram | `target`, `status`          | Time a service the server calls takes to answer: the calling plugin (`james`, `migrationProxy`, `calendar` or its old name `calendarResources`, `drive`, `cozyProvision`, `clouderyProvision`, `crowdsec`) or the instance provider (`cloudery`, `cozy-stack`); the HTTP status, `error` when it did not answer |
+| `ldap_rest_unhandled_rejections_total`        | counter   |                             | Promise rejections nothing handled; counted by the binary, a host skipping `exitOnCrash()` leaves it at zero                                                                                                                                                                                                    |
+| `process_*`, `nodejs_*`                       |           |                             | The process: CPU, memory, event loop lag, handles, GC                                                                                                                                                                                                                                                           |
+
+A search the caller pages is timed page by page, what the caller does
+between two pages left out; the readiness ping is not timed. The route
+label is the template a route was declared with, never the URL: a label
+per URL would grow without bound; under a router a plugin mounted, the
+mount reads `*`. An outbound call is timed from the moment it is sent,
+after the plugin's own queue, to the answer's headers, each attempt of a
+retried call on its own; a plugin's health check that asks its service,
+James's does, is a call like another. A plugin calling a service through
+`this.fetch` has its calls counted under its name; the OpenID Connect
+client's calls to its provider are made by its library, and not timed.
+
+The instruments come from `prom-client` 15, kept while the server supports
+Node 20, which its successor `@prometheus-io/client` dropped: the
+deprecation notice `npm install` prints is expected.
+
+The port listens on every interface. What to alert on: `/readyz` for a
+replica out of rotation, `result="error"` in
+`ldap_rest_ldap_operation_duration_seconds` (the directory not answering;
+a refusal is a result code), a
+`ldap_rest_ldap_pool_waiting` that stays above zero (raise
+`--ldap-pool-size`), `ldap_rest_hook_failures_total` rising (a
+synchronization that fails only in the log otherwise), `status="error"`
+or the 5xx share in `ldap_rest_outbound_request_duration_seconds` by
+`target` (a service the server depends on failing), the 5xx share of the
+requests.
+
+The Helm chart exposes the port on the Service with `metrics.enabled`, and
+renders a `ServiceMonitor` for the Prometheus operator with
+`metrics.serviceMonitor.enabled`; `core/metrics` still has to be in
+`DM_PLUGINS`.
+
+## Shutdown
+
+On `SIGTERM` or `SIGINT`, the binary stops in order:
+
+1. the listening socket closes: a new connection is refused, and
+   `/readyz` answers 503 `stopping` on the connections still open;
+2. idle keep-alive connections close at once; the others close with the
+   response they carry, sent with `Connection: close`, or are cut when the
+   drain is spent: `--shutdown-timeout` (20 s) less the 5 s kept for the
+   steps below, 1 s at least;
+3. every plugin's `close()` runs, at once: the broker connection, the
+   storage backend;
+4. the directory connections are closed;
+5. the signal ends the process, with its usual code (143 on `SIGTERM`).
+
+The whole takes `--shutdown-timeout` at most: a `close()` that does not
+settle in what remains of the budget is logged and left behind, by
+`shutdown()` itself, so a host gets the same bound, and the directory
+connections are closed anyway. The drain and the closes get 1 s each at
+least, so a budget under 2 s is exceeded. Kubernetes takes the pod out
+of its endpoints as it sends the signal, and the proxies follow a moment
+later: a request routed meanwhile is refused. A `preStop` hook sleeping a
+few seconds, before the signal, closes that window; the chart's `lifecycle`
+value takes one. The sleep and `--shutdown-timeout` together must stay
+under `terminationGracePeriodSeconds`, 30 s by default, which the chart
+leaves to Kubernetes: 5 s and 20 s fit.
+
+A host assembling the server in its own process calls `shutdown()` itself,
+`shutdownOnSignals()` when it wants the same behaviour on a signal, and
+`exitOnCrash()` for the policy below; without the latter, Node's own action
+on an uncaught exception applies.
+
+An exception nothing caught leaves the process in a state nobody knows: it
+is logged, the shutdown above runs, and the process exits with 1, within
+`--shutdown-timeout` whatever hangs, for its supervisor to start it again.
+A promise rejection nothing handled is logged and counted, and the process
+keeps serving.
+
+### For plugin authors
+
+A plugin holding a connection, a timer or a queue implements `close()`,
+called once after the server stopped serving; it must settle, a failure is
+logged and does not hold the others.

package/docs/usage/plugins/README.md

@@ -57,6 +57,7 @@ Connect to external systems:
 | ---------------------------------------------------- | ------------------------------------ |
 | [james-mail](integrations/james-mail.md)             | Apache James synchronization         |
 | [james-mailboxes](integrations/james-mailboxes.md)   | Apache James team mailboxes          |
+| [mail-migration](integrations/mail-migration.md)     | Twake Mail migration proxy           |
 | [calendar](integrations/calendar.md)                 | Twake Calendar resources and users   |
 | [app-accounts](integrations/app-accounts.md)         | Applicative accounts API             |
 | [lifecycle-events](integrations/lifecycle-events.md) | Account lifecycle events to RabbitMQ |
@@ -70,6 +71,7 @@ Connect to external systems:
 | ------------------------------- | ------------------------------------------------------------------- |
 | [static](utilities/static.md)   | Static file server                                                  |
 | [weblogs](utilities/weblogs.md) | HTTP logging                                                        |
+| [metrics](utilities/metrics.md) | Prometheus metrics on a port of their own                           |
 | [storage](utilities/storage.md) | Keyed storage with a deadline, in LDAP, files, PostgreSQL or Valkey |
 
 ## Plugin Dependencies

package/docs/usage/plugins/auth/authz-per-branch.md

@@ -536,7 +536,8 @@ This plugin only handles **authorization**. Combine with an authentication plugi
 
 ```json
 {
-  "error": "check logs"
+  "error": "Internal Server Error",
+  "message": "An error occurred"
 }
 ```
 

package/docs/usage/plugins/integrations/README.md

@@ -4,10 +4,11 @@ Plugins for connecting LDAP-Rest to external systems.
 
 ## Apache James
 
-| Plugin                                | Description                     |
-| ------------------------------------- | ------------------------------- |
-| [james-mail](james-mail.md)           | User and domain synchronization |
-| [james-mailboxes](james-mailboxes.md) | Team mailbox management         |
+| Plugin                                | Description                      |
+| ------------------------------------- | -------------------------------- |
+| [james-mail](james-mail.md)           | User and domain synchronization  |
+| [james-mailboxes](james-mailboxes.md) | Team mailbox management          |
+| [mail-migration](mail-migration.md)   | Migrated-users list of the proxy |
 
 ## Twake
 
@@ -45,6 +46,9 @@ core/twake/james
 core/twake/calendar
   └─ requires: core/ldap/onChange
 
+core/twake/migrationProxy
+  └─ requires: core/ldap/onChange
+
 core/twake/lifecycleEvents
   ├─ requires: core/ldap/onChange
   └─ requires: core/rabbitmq

package/docs/usage/plugins/integrations/mail-migration.md

@@ -0,0 +1,102 @@
+# Mail migration proxy
+
+`core/twake/migrationProxy` keeps the migrated-users list of the
+[Twake Mail migration proxy](https://github.com/linagora/tmail-backend/blob/master/docs/modules/ROOT/pages/migration-proxy/objective.adoc)
+in step with the directory, while a mail platform is moved to Twake Mail.
+
+The proxy sits in front of the old and the new mail backends. It relays
+each IMAP (and, when enabled, submission) connection to the backend hosting
+the authenticated user, and routes each inbound recipient to the backend
+hosting it. An address on its migrated list lives on the new backend; any
+other address lives on the old one. An account created during the migration
+has no mailbox on the old system, so it has to be on the list before its
+first login and its first mail: this plugin puts it there, and keeps the
+addresses of a migrated account on the list as they change.
+
+Load it for the duration of the migration, next to `core/twake/james`, and
+drop it once the old backend is gone. It requires `core/ldap/onChange`.
+
+## Scope
+
+Only the entries under `--migration-proxy-branch` are followed, the
+branches holding the accounts whose addresses the proxy routes: users, and
+the groups whose address is a team mailbox or a mailing list when those are
+created during the migration, as inbound mail for them is routed by
+recipient too. Keep out the branches whose entries carry an address that is
+not their own, the applicative accounts of
+[app-accounts](app-accounts.md) for instance: they would put their owner on
+the list or take it off.
+
+The addresses of an entry are every value of `--mail-attribute` and of
+`--alias-attribute`, the first mail being the one that says where the
+account lives. The proxy folds usernames to lower case, so a change of case
+is no change.
+
+## Behaviour
+
+Through the `onLdapEntryChange` hook, after the write has answered, one
+entry at a time:
+
+- An entry that gains its mail, when created or later, is put on the list
+  with all its addresses: `PUT /migratedUsers/{address}`.
+- An entry that keeps its mail and whose addresses change asks the proxy
+  about its first mail (`HEAD /migratedUsers/{mail}`). When the account is
+  migrated, the addresses it gained are put on the list, then the ones it
+  lost are taken off, once every addition holds; a user the proxy does not
+  know is on the old system, and its change is left to the migration, which
+  puts the new addresses on the list when it moves the mailbox.
+- An entry deleted, or losing its mail, is taken off the list with all its
+  addresses: `DELETE /migratedUsers/{address}`. A tombstone or a trashed
+  entry keeps its address, so it stays on the list until it is erased.
+
+The plugin cannot tell a new account from an existing one whose mail was
+cleared and set again, or from one created anew over its tombstone: those
+are put on the list too. Marking the existing accounts migrated is the
+migration's job, through the proxy's webadmin or its `migration:switch`
+exchange.
+
+### Failures
+
+A call the proxy answers with 500, 502, 503 or 504 is made again,
+`--migration-proxy-retry-attempts` times in all, after
+`--migration-proxy-retry-delay` milliseconds times the number of attempts
+made. A refusal, such as 400 for an address the proxy does not take as a
+username or 401 for a wrong token, and a call the proxy does not answer at
+all, connection refused or reset, are not. Each failed attempt logs a warning, the last failure an
+error naming the address, the call and the entry. Every call is counted in
+the [outbound metrics](../../operations.md#metrics), under the target
+`migrationProxy`.
+
+A failed `PUT` leaves the account routed to the old backend, which has no
+mailbox for it: watch those errors, and put the address on the list by hand
+through the proxy's webadmin. When the proxy has not said whether a renamed
+account is migrated, nothing is changed and the error is logged.
+
+## Where an account lives
+
+`GET /api/v1/users/:user/mail-migration` answers `{ "migrated": true }`
+when the proxy routes the user to the new backend, `{ "migrated": false }`
+otherwise, from the proxy's `HEAD`. The user is found by `uid` across the
+directory, and answered only when the caller may read the entry: another
+one gets 404, as a user that is not there. A user without mail gets 400,
+and a proxy that does not answer 502.
+
+## Readiness
+
+The plugin asks the proxy's `GET /healthcheck` for `/readyz`, and reports
+`degraded` when it answers an error or not within 1.5 s, see
+[what the shipped plugins report](../../operations.md#what-the-shipped-plugins-report).
+
+## Configuration
+
+```bash
+--plugin core/ldap/onChange \
+--plugin core/twake/james \
+--plugin core/twake/migrationProxy \
+--migration-proxy-url http://migration-proxy:8000 \
+--migration-proxy-token "$MIGRATION_PROXY_TOKEN" \
+--migration-proxy-branch "ou=users,dc=example,dc=com"
+```
+
+The options are listed in
+[configuration](../../configuration.md#coretwakemigrationproxy).

package/docs/usage/plugins/integrations/scim.md

@@ -734,10 +734,12 @@ Notable fields:
 - `schemaUrls` — direct download URLs of the schema files SCIM reads
   (`User.json` and `Group.json` under `--schemas-path`, with the
   `default-mapping.json` template beside them when it is there), when a
-  `core/static` plugin serves them, under its own `--static-name` (editor
+  `core/static` plugin serves them merged, through its schema routes, under
+  its own `--static-name` (editor
   UIs typically fetch them to render attribute pickers). Among plugins
   serving different paths, the one registered as `static` is preferred; a
-  path an earlier plugin already serves is left to that one. Absent when
+  path an earlier plugin already serves, lies under or holds, is left to
+  that one. Absent when
   none serves them: read it with a guard.
 - `resourceTypes[].mapping` — the live SCIM↔LDAP mapping used server-side, so
   UI builders can render filter helpers / attribute selectors without second

package/docs/usage/plugins/ldap/bulk-import.md

@@ -174,7 +174,9 @@ mail: jdoe@company.com
 
 ### Fixed Attributes
 
-Attributes marked as `"fixed": true` in the schema are automatically added and should NOT be included in the CSV:
+Attributes marked as `"fixed": true` in the schema are automatically added. A
+row with a value under such a column fails, whatever the case of the column
+name; an empty cell is ignored as any other:
 
 **Schema**:
 

package/docs/usage/plugins/ldap/flat-generic.md

@@ -74,7 +74,9 @@ Each schema file must include an `entity` metadata section and attribute definit
 
 - **name**: Internal name used for hooks (e.g., `ldap{name}add`)
 - **mainAttribute**: DN attribute (e.g., `uid`, `cn`)
-- **objectClass**: LDAP objectClass values
+- **objectClass**: the classes of every entry created, whatever `objectClass`
+  the request carried; the schema's `objectClass` attribute only describes
+  them
 - **singularName**: Used in API paths and messages (e.g., "user")
 - **pluralName**: Used in API paths (e.g., "users")
 - **base**: LDAP base DN, supports config placeholders like `{ldap_base}`
@@ -138,7 +140,8 @@ never what it holds.
 
 Fixed attributes:
 
-- Auto-provisioned with default value on creation
+- Auto-provisioned with default value on creation; `objectClass` takes the
+  classes of the entity metadata instead
 - Cannot be modified after creation
 - Cannot be deleted
 

package/docs/usage/plugins/ldap/on-change.md

@@ -28,7 +28,8 @@ side:
 says who made the write, and through which door:
 
 - `actor`: the caller's `userName`, or its `user` when it has none;
-- `requestId`: generated, the same for every write of one request;
+- `requestId`: the answer's `X-Request-Id`, the same for every write of one
+  request, see [operations](../../operations.md#request-id);
 - `source`: `scim` for the SCIM API, `rest` otherwise.
 
 A write no request is behind, such as a scheduled task's, gets `{}`. The LDAP

package/docs/usage/plugins/ldap/organizations.md

@@ -28,13 +28,13 @@ This plugin manages hierarchical organizations stored as `organizationalUnit` (o
 
 ### Configuration Options
 
-| Argument                                                    | Environment Variable                  | Default                         | Description                                         |
-| ----------------------------------------------------------- | ------------------------------------- | ------------------------------- | --------------------------------------------------- |
-| `--ldap-top-organization`                                   | `DM_LDAP_TOP_ORGANIZATION`            | _Required_                      | DN of the top-level organization                    |
-| `--ldap-organization-class` / `--ldap-organization-classes` | `DM_LDAP_ORGANIZATION_CLASSES`        | `["top", "organizationalUnit"]` | Object classes for organizations                    |
-| `--ldap-organization-link-attribute`                        | `DM_LDAP_ORGANIZATION_LINK_ATTRIBUTE` | `twakeDepartmentLink`           | Attribute for linking users/groups to organizations |
-| `--ldap-organization-path-attribute`                        | `DM_LDAP_ORGANIZATION_PATH_ATTRIBUTE` | `twakeDepartmentPath`           | Attribute for the organizational path               |
-| `--ldap-organization-path-separator`                        | `DM_LDAP_ORGANIZATION_PATH_SEPARATOR` | `" / "`                         | Separator for path components                       |
+| Argument                                                    | Environment Variable                  | Default                                  | Description                                         |
+| ----------------------------------------------------------- | ------------------------------------- | ---------------------------------------- | --------------------------------------------------- |
+| `--ldap-top-organization`                                   | `DM_LDAP_TOP_ORGANIZATION`            | _Required_                               | DN of the top-level organization                    |
+| `--ldap-organization-class` / `--ldap-organization-classes` | `DM_LDAP_ORGANIZATION_CLASSES`        | `top,organizationalUnit,twakeDepartment` | Object classes for organizations                    |
+| `--ldap-organization-link-attribute`                        | `DM_LDAP_ORGANIZATION_LINK_ATTRIBUTE` | `twakeDepartmentLink`                    | Attribute for linking users/groups to organizations |
+| `--ldap-organization-path-attribute`                        | `DM_LDAP_ORGANIZATION_PATH_ATTRIBUTE` | `twakeDepartmentPath`                    | Attribute for the organizational path               |
+| `--ldap-organization-path-separator`                        | `DM_LDAP_ORGANIZATION_PATH_SEPARATOR` | `" / "`                                  | Separator for path components                       |
 
 ### Important Notes
 

package/docs/usage/plugins/utilities/metrics.md

@@ -0,0 +1,21 @@
+# Metrics Plugin
+
+Prometheus metrics of the server, on a port of their own.
+
+## Configuration
+
+```bash
+--plugin core/metrics --metrics-port 9464
+```
+
+`--metrics-port` (`DM_METRICS_PORT`, 9464) is the port `GET /metrics` is
+served on, see [configuration](../../configuration.md#coremetrics).
+
+The port is separate from the API's: nothing on it goes through the
+authentication, the rate limiting or the request id, so it must not be
+reachable from outside the cluster or the host. The Helm chart exposes it
+on the Service and to a Prometheus operator with `metrics.enabled` and
+`metrics.serviceMonitor.enabled`.
+
+What is measured, and how to read it, is in
+[operations](../../operations.md#metrics).

package/docs/usage/plugins/utilities/static.md

@@ -242,12 +242,7 @@ for instance to `/static/index.html`.
 
 ### Example 4: Multiple Static Plugins
 
-Not directly supported, but you can nest directories:
-
-```bash
---plugin core/static \
---static-path ./static
-```
+One plugin serves several applications from sub-directories:
 
 ```
 ./static/
@@ -259,14 +254,27 @@ Not directly supported, but you can nest directories:
     users.json
 ```
 
-Access via:
-
 ```
 /static/docs/index.html
 /static/app/index.html
 /static/schemas/users.json
 ```
 
+A second directory takes a second instance, loaded under a name of its own
+with its own `static_name` and `static_path`
+([`module:name:{json}`](../auth/README.md#serving-several-populations-from-one-server)):
+
+```bash
+--plugin core/static --static-path ./static \
+--plugin 'core/static:console:{"static_name":"console","static_path":"./console"}'
+```
+
+Give them paths that do not nest: `console` and `console/app` would have the
+first one answer requests meant for the second. The plugins that announce a
+schema URL (groups, organizations, flat entities, SCIM) ask every instance,
+the one registered as `static` first, and leave out one whose path lies under
+an earlier one's, or holds it.
+
 ## Security Considerations
 
 ### Path Traversal

package/docs/usage/plugins/utilities/storage.md

@@ -110,6 +110,13 @@ answer "absent" with a warning and writes fail at once instead of waiting.
 does not answer or a package that is not installed; the server starts anyway,
 and every operation tries to connect again.
 
+## Readiness
+
+`GET /readyz` lists the plugin as `degraded` when the backend does not
+answer; the `ldap` backend keeps its records in the directory the probe
+checks on its own, see
+[operations](../../operations.md#what-the-shipped-plugins-report).
+
 ## Expiry
 
 Two rules the backends do not get to reinterpret: