@toa.io/extensions.exposition 1.0.0-alpha.28 → 1.0.0-alpha.283

package/documentation/notes/throttling.md

@@ -0,0 +1,122 @@
+# Decentralized Request Throttling
+
+> **Superseded.** The design below was never implemented. `io:throttle` is now a
+> [GCRA](https://en.wikipedia.org/wiki/Generic_cell_rate_algorithm) deciding in each process, with
+> its state periodically reconciled through Redis by way of the `atom` aspect's `meter`. It
+> answers the same forces the same way — decide locally, reconcile in the background, never fail the
+> request path — and the section below records what it does instead. The problem statement and the
+> forces have not changed.
+
+## Problem
+
+To prevent an unnecessary load, API requests should be throttled when made maliciously or due to an
+error.
+
+## Context
+
+1. The [API Gateway](https://microservices.io/patterns/apigateway.html) of a distributed system runs
+   in multiple instances, each with an in-memory state that does not allow tracking the current
+   quota usage.
+2. Precise quota enforcement (per request, per second) is not critical; the goal is to prevent
+   significant overuse.
+3. Quota configuration is static.
+
+## Forces
+
+1. No per request IO is allowed (centralized solutions do not fit).
+2. In a distributed system, there is no concept of a *global current time*.
+3. Failure to retrieve the quota state should not result in Gateway failure.
+
+## Solution
+
+Implement in-memory quotas in each process, periodically synchronizing them asynchronously using
+Redis.
+
+Consider a basic throttling rule:
+
+1. No more than `MAX_REQUESTS` within `INTERVAL` time for any API route (`KEY`).
+2. If the limit is exceeded, block requests to `KEY` for `COOLDOWN` seconds.
+
+### Concept
+
+1. Divide `INTERVAL` into `N` spans (`N >= 2`).
+2. At the end of each span,
+   atomically increment the value in Redis ([INCRBY](https://redis.io/docs/latest/commands/incrby/))
+   by the number of requests received for each `KEY`, using a key composed of the `KEY` and the
+   current ordinal number of `INTERVAL`
+   in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time).
+3. If the returned value exceeds `MAX_REQUESTS`, block access to `KEY`, remove after `COOLDOWN`.
+4. If the write operation fails and `REQUESTS > MAX_REQUESTS / N` (i.e., the quota is exceeded
+   locally), block access to `KEY`, remove after `COOLDOWN`.
+
+Each API Gateway instance will generate the `KEY` based on its own local time,
+which, in general, will lead to simultaneous writes (from Redis’s local time perspective) to
+different `KEY`s. However, the total contribution of each node to each `KEY` will correspond to the
+actual request rate experienced by that node.
+
+<img src="desync.jpg" alt="Desynchronization" />
+
+Dividing the `INTERVAL` into spans smooths the desynchronization effect.
+
+### Extension
+
+Introduce `nodes`, the approximate number of active API Gateway instances, to improve the algorithm.
+
+1. Initially, `nodes` equals to `1`.
+2. During each `INTERVAL`, sum the total request count for each `KEY`, keep current and previous
+   values.
+3. At the end of each interval:
+
+- Read the value from the `KEY` for the previous interval. At this point, it is assumed that all
+  nodes have switched to the next interval.
+- Update the `nodes` value by dividing response form Redis by the number of `REQUESTS` counted for
+  the previous interval.
+- Set the previous value to the current value.
+- Set the current value to `0`
+
+4. Update point 4 of the [Concept](#concept): `REQUESTS * nodes > MAX_REQUESTS / N`.
+   This will increase the precision of the local quota enforcement.
+5. Add a new step: if `REQUESTS * nodes > MAX_REQUESTS`, block access to `KEY`, remove after
+   `COOLDOWN`.
+
+When nodes are added or removed, the algorithm will adapt in the upcoming intervals.
+
+### Caveats
+
+1. In the worst case scenario (really, really unlikely), the quota is exceeded by `MAX_REQUESTS / N`
+   in a span on each node.
+2. Time desynchronization between nodes should be insignificant for the selected `INTERVAL` (i.e.,
+   `INTERVAL` >> desync). See Extension point 3.
+
+## What was implemented instead
+
+Windows were dropped for a GCRA. A key holds one theoretical arrival time: the moment it would be
+back at zero if nothing more arrived. An admitted request pushes it `interval / requests`
+milliseconds further out, time drags it back, and a request is admitted while it stays within
+`interval` of now. So `requests` is what a key may spend at once and `requests / interval` the rate
+it earns back, with no window edge to burst across and no lockout to time out of — which is why
+`cooldown` is gone.
+
+Deciding reads nothing but a local map, so the request path stays free of I/O, exactly as force 1
+demands. What the other gateways have spent arrives on a timer, as **debt**: the milliseconds a key
+owes, draining at a millisecond a millisecond. Debt is the right thing to send for two reasons.
+
+1. It is a duration, not a moment, so processes exchanging it need not agree on the time — force 2,
+   answered by not needing an answer. The one clock that matters is Redis', read inside the script.
+2. It is additive in admissions, so a process reports only its own increments and Redis keeps the
+   running total. Nothing has to be divided by a number of nodes, which is as well: the gateway
+   cannot observe how many of it there are.
+
+The consequences worth knowing:
+
+- **Reconciling is decoupled from `interval`.** The timer runs at a tenth of `interval`, clamped
+  between 250ms and 2s, so a minute-long budget is no staler than a second-long one. The overshoot
+  is bounded by what the other instances admit within one tick — the caveat that replaces the old
+  note's `MAX_REQUESTS / N` per span.
+- **The cost is one round trip a tick**, not one per key. Every quota in the process flushes into a
+  single script call, which matters because a limiter keyed on `ip` watches as many keys as it has
+  clients.
+- **Redis keys expire on their own**, after the debt on them plus a grace, so nothing sweeps them.
+- **Failure degrades, as force 3 demands.** A tick that cannot reach Redis leaves its debt to the
+  next one rather than losing it; until then the instance throttles on what it alone has seen, and
+  keeps serving.

package/documentation/oauth.md

@@ -0,0 +1,179 @@
+# OAuth 2.1 Authorization Server
+
+An application exposes an authorization server by annotating one, and none without it. A client
+authorized this way holds a token of the identity that consented, and presents it as `Bearer`.
+
+## Annotation
+
+```yaml
+exposition:
+  authorities:
+    local: api.example.com
+  oauth:
+    authorize: https://app.example.com/oauth/authorize
+    resources: ['/mcp/']
+    scopes: [app:notes, app:notes:read]
+    registration: closed
+```
+
+<dl>
+<dt><code>authorize</code></dt>
+<dd>Required. The absolute URL of the consent page. The application serves it; see
+<a href="consent.md">Consent</a>.</dd>
+<dt><code>resources</code></dt>
+<dd>Paths advertised as protected resources, each with a document of its own and named by the
+challenge a request to it is refused with. A token is not yet bound to one, see
+<a href="#audience">Audience</a>.</dd>
+<dt><code>scopes</code></dt>
+<dd>Advertised as what a client may ask for. Each is a <a href="access.md#roles">role</a>, or a
+scope within one.</dd>
+<dt><code>registration</code></dt>
+<dd><code>open</code> advertises the registration endpoint, <code>closed</code> does not.
+Defaults to <code>closed</code>.</dd>
+</dl>
+
+The issuer is the authority's configured host, over `https` unless the host is a loopback one.
+
+## Discovery
+
+Two documents are served, at the paths their specifications fix, to anyone.
+
+`/.well-known/oauth-authorization-server` is
+[RFC 8414](https://www.rfc-editor.org/rfc/rfc8414) metadata:
+
+```yaml
+issuer: https://api.example.com
+authorization_endpoint: https://app.example.com/oauth/authorize
+token_endpoint: https://api.example.com/identity/grants/
+registration_endpoint: https://api.example.com/identity/clients/
+response_types_supported: [code]
+grant_types_supported: [authorization_code]
+code_challenge_methods_supported: [S256]
+token_endpoint_auth_methods_supported: [none]
+client_id_metadata_document_supported: true
+authorization_response_iss_parameter_supported: true
+```
+
+`/.well-known/oauth-protected-resource` is
+[RFC 9728](https://www.rfc-editor.org/rfc/rfc9728) metadata. Each configured resource has one of
+its own, at the path the well-known segment is followed by: `/mcp/` is read at
+`/.well-known/oauth-protected-resource/mcp`.
+
+```yaml
+resource: https://api.example.com/mcp
+authorization_servers: [https://api.example.com]
+bearer_methods_supported: [header]
+```
+
+`/.well-known/openid-configuration` answers with the authorization server metadata, for a client
+that reads only that.
+
+A reply of `401` carries the challenge that names the document:
+
+```http
+401 Unauthorized
+www-authenticate: Bearer resource_metadata="https://api.example.com/.well-known/oauth-protected-resource/mcp", scope="app:notes"
+```
+
+## Clients
+
+A `client_id` that is an `https` URL is a
+[Client ID Metadata Document](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-client-id-metadata-document-00):
+the client publishes what it is at that URL, and the document is read from there. Only origins
+`identity.clients` is configured to trust are read, and an empty list reads none:
+
+```yaml
+configuration:
+  identity.clients:
+    trust:
+      - https://claude.ai
+```
+
+Anything else was registered through
+[RFC 7591](https://www.rfc-editor.org/rfc/rfc7591), which `registration: open` advertises:
+
+```http
+POST /identity/clients/ HTTP/1.1
+content-type: application/json
+
+{"client_name": "Claude", "redirect_uris": ["https://claude.ai/api/mcp/auth_callback"]}
+```
+
+```http
+201 Created
+
+{"client_id": "c8f3a1e4d8c9b2f6a0e5d7c3b1a9f8e2", "client_name": "Claude", …}
+```
+
+A registration is addressed by a hash of what it says, so the same metadata is one client however
+many times it arrives. A registration cannot be changed: a revision is a different client.
+
+Only public clients are supported. `token_endpoint_auth_method` other than `none` is refused.
+
+`redirect_uris` are matched exactly, except for a loopback address, where the port is ignored
+([RFC 8252 §7.3](https://datatracker.ietf.org/doc/html/rfc8252#section-7.3)).
+
+`GET /identity/clients/:id/` returns a client to any authenticated identity, which is what the
+consent page reads.
+
+## Token endpoint
+
+`POST /identity/grants/`, anonymous, `application/x-www-form-urlencoded`.
+
+```http
+POST /identity/grants/ HTTP/1.1
+content-type: application/x-www-form-urlencoded
+
+grant_type=authorization_code&code=SplxlO&code_verifier=dBjftJeZ&redirect_uri=…&client_id=…
+```
+
+```http
+200 OK
+cache-control: no-store
+
+{"access_token": "…", "token_type": "Bearer", "expires_in": 2592000}
+```
+
+`code_verifier` is required: the code was issued against its `S256` hash
+([RFC 7636](https://www.rfc-editor.org/rfc/rfc7636)). A code may be redeemed once, and is spent
+whether or not the verifier matches. Every refusal is `invalid_grant`.
+
+A code is valid for `identity.grants.lifetime` seconds, 60 by default. A token is valid for
+`identity.grants.token` seconds, 30 days by default; `0` is until it is revoked.
+
+## Grants
+
+A grant is what a user allowed a client. One is held per identity per client.
+
+`GET /identity/grants/:identity/` lists them. `DELETE /identity/grants/:identity/:id/` revokes one,
+which disables the key its token was issued under: the token stops being one within
+`identity.tokens.cache.ttl`.
+
+## Tokens
+
+An access token is a token of the identity that consented, carrying the
+[roles](access.md#roles) that identity holds, or the subset the client asked for as `scope`. It is
+presented as `Bearer`, which `identity.federation` also answers to — see
+[Bearer scheme](identity.md#bearer-scheme).
+
+A `scope` a client asks for is not checked against `scopes_supported`: it is granted if the
+consenting identity holds it, and refused as `invalid_scope` otherwise. What is advertised says
+what an application means to offer, not what the server will refuse.
+
+## Audience
+
+`resource` ([RFC 8707](https://www.rfc-editor.org/rfc/rfc8707)) is accepted at both endpoints and
+recorded on the grant. **It does not yet restrict the token.** An access token carries the rights
+of the identity that consented, over every path that identity may reach, whichever resource it was
+asked for.
+
+Restricting it is a `permissions` argument at the `identity.tokens.issue` call in
+`identity.grants.exchange` — `{'/mcp/**': ['*']}` for a resource at `/mcp/` — enforced by
+`permits()`, which is already what reads a token's permissions on every request. It waits on the
+resource that will check the audience itself.
+
+## References
+
+- [OAuth 2.1](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1)
+- [Consent](consent.md), the page an application serves
+- [Features](../features/oauth.grants.feature)

package/documentation/octets.md

@@ -14,26 +14,32 @@ directives under the current RTD Node.
   octets:context: images
 ```
 
-## `octets:store`
+## `octets:put`
 
 Stores the content of the request body into a storage, under the request path with
 specified `content-type`.
 
 If request's `content-type` is not acceptable, or if the request body does not pass
-the [validation](/extensions/storages/readme.md#async-putpath-string-stream-readable-type-typecontrol-maybeentry),
+the [validation](/extensions/storages/readme.md#async-putpath-string-stream-readable-options-options-maybeentry),
 the request is rejected with a `415 Unsupported Media Type` response.
 
-The value of the directive is `null` or an object with the following properties:
+The value of the directive must be `null` (defaults) or an object with the following optional
+properties:
 
+- `location`: a string that represents the path to store the
+  content.
+  If not specified, the path is the same as the request path.
+- `limit`: [maximum size](#stream-size-limit) of the incoming stream.
 - `accept`: a media type or an array of media types that are acceptable.
   If the `accept` property is not specified, any media type is acceptable (which is the default).
 - `workflow`: [workflow](#workflows) to be executed once the content is successfully stored.
+- `trust`: a list of [trusted origins](#downloading-external-content).
 
 ```yaml
 /images:
   octets:context: images
   POST:
-    octets:store:
+    octets:put:
       accept:
         - image/jpeg
         - image/png
@@ -43,31 +49,97 @@ The value of the directive is `null` or an object with the following properties:
         analyze: images.analyze
 ```
 
-Non-standard `content-meta` header can be used
+### Headers
+
+`content-id` header can be used to set the ID of the Entry.
+The value must match the following regular expression `^[a-zA-Z0-9-_]{1,16}$`.
+
+Non-standard `content-attributes` header can be used
 to set initial [metadata](/extensions/storages/readme.md#entry) value for the Entry.
 
-The value of the `content-meta` header is a comma-separated list of key-value string pairs.
+The value of the `content-attributes` header is a comma-separated list of key-value string pairs.
 If no value is provided for a key, the string `true` is used.
 
 ```http
 POST /images/ HTTP/1.1
 content-type: image/jpeg
-content-meta: foo, bar=baz
-content-meta: baz=1
+content-id: example-id
+content-attributes: foo, bar=baz
+content-attributes: baz=1
 ```
 
 ```yaml
-meta:
+attributes:
   foo: 'true'
   bar: 'baz'
   baz: '1'
 ```
 
-If the Entry already exists, the `content-meta` header is ignored.
+If the Entry already exists, the `content-attributes` header is ignored.
+
+### Location
+
+The `location` property can be used to store the content under a different path.
+
+```yaml
+/images:
+  octets:context: images
+  POST:
+    octets:put:
+      location: /archive
+```
+
+Physical storage path is constructed by resolving the `location`
+property [relative](https://datatracker.ietf.org/doc/html/rfc3986#section-5) to the request path.
+
+### Stream size limit
+
+The `limit` property can be used to set the maximum size of the incoming stream in bytes.
+
+The property value can be specified as a number
+(representing bytes) or a string that combines a number with a unit (e.g., `1MB`).
+Both [binary and decimal prefixes](https://en.wikipedia.org/wiki/Binary_prefix) are supported.
+If the prefix or unit is specified _incorrectly_ (e.g., `1mb`),
+it will default to a binary prefix interpretation.
+
+- `1b`, `1B`: 1 byte
+- `1KB`: 1000 bytes
+- `1KiB`: 1024 bytes
+- `1kb`: 1024 bytes
+
+The default value is `64MiB`.
+
+### Downloading external content
+
+The `octets:put` directive can be used to download external content:
+
+```http
+POST /images/ HTTP/1.1
+content-location: https://example.com/image.jpg
+content-length: 0
+```
+
+Requests with `content-location` header must have an empty body (`content-length: 0` header).
+
+Target origin must be allowed by the `trust` property,
+which can contain a list of trusted origins or regular expressions to match the full URL.
+
+URL of the downloaded content is stored in the `origin` property of
+the [Entry](/extensions/storages/readme.md#entry).
+
+```yaml
+/images:
+  octets:context: images
+  POST:
+    octets:put:
+      trust:
+        - https://example.com
+        - ^https://example\.com/[a-z]+\.jpe?g$
+```
 
 ### Response
 
-The response of the `octets:store` directive is the created Entry.
+The response of the `octets:put` directive is the created Entry.
 
 ```
 201 Created
@@ -78,12 +150,13 @@ type: image/jpeg
 created: 1698004822358
 ```
 
-If the `octets:store` directive contains a `workflow`, the response
+If the `octets:put` directive contains a `workflow`, the response
 is [multipart](protocol.md#multipart-types).
 The first part represents the created Entry, which is sent immediately after the BLOB is stored,
 while subsequent parts are results from the workflow endpoints, sent as soon as they are available.
 
-In case a workflow endpoint returns an `Error`, the error part is sent, and the response is closed.
+In case a workflow endpoint returns an `Error`, the error part is sent,
+and the response is closed.
 Error's properties are added to the error part, among with the `step` identifier.
 
 ```
@@ -91,20 +164,33 @@ Error's properties are added to the error part, among with the `step` identifier
 content-type: multipart/yaml; boundary=cut
 
 --cut
+
 id: eecd837c
 type: image/jpeg
 created: 1698004822358
+
 --cut
-optimize: null
+
+step: optimize
+status: completed
+
 --cut
+
+step: resize
 error:
-  step: resize
   code: TOO_SMALL
   message: Image is too small
+status: completed
+
+--cut
+
+step: analyze
+status: exception
+
 --cut--
 ```
 
-## `octets:fetch`
+## `octets:get`
 
 Fetches the content of a stored BLOB corresponding to the request path, and returns it as the
 response body with the corresponding `content-type`, `content-length`
@@ -116,22 +202,18 @@ The value of the directive is an object with the following properties:
 
 - `meta`: `boolean` indicating whether an Entry is accessible.
   Defaults to `false`.
-- `blob`: `boolean` indicating whether the original BLOB is accessible,
-  [BLOB variant](/extensions/storages/readme.md#async-fetchpath-string-maybereadable) must be
-  specified in the path otherwise.
-  Defaults to `true`.
 
 ```yaml
 /images:
   octets:context: images
   /*:
     GET:
-      octets:fetch:
+      octets:get:
         blob: false # prevent access to the original BLOB
         meta: true  # allow access to an Entry
 ```
 
-The `octets:fetch: ~` declaration is equivalent to defaults.
+The `octets:get: ~` declaration is equivalent to defaults.
 
 To access an Entry, the `accept` request header must contain the `octets.entry` subtype
 in
@@ -142,32 +224,6 @@ GET /images/eecd837c HTTP/1.1
 accept: application/vnd.toa.octets.entry+yaml
 ```
 
-## `octets:list`
-
-Lists the entries stored under the request path.
-
-The value of the directive is an object with the following properties:
-
-- `meta`: `boolean` indicating whether the list of Entries is accessible.
-  Defaults to `false`, which means that only entry identifiers are returned.
-
-```yaml
-/images:
-  octets:context: images
-  GET:
-    octets:list:
-      meta: true
-```
-
-The `octets:list: ~` declaration is equivalent to defaults.
-
-To access a list of Entries, the `accept` request header must contain the `octets.entries` subtype:
-
-```http
-GET /images/ HTTP/1.1
-accept: application/vnd.toa.octets.entries+yaml
-```
-
 ## `octets:delete`
 
 Delete the entry corresponding to the request path.
@@ -193,22 +249,6 @@ the entry is deleted.
 
 The error returned by the workflow prevents the deletion of the entry.
 
-## `octets:permute`
-
-Performs
-a [permutation](/extensions/storages/readme.md#async-permutepath-string-ids-string-maybevoid) on the
-entries
-under the request path.
-
-```yaml
-/images:
-  octets:context: images
-  PUT:
-    octets:permute: ~
-```
-
-The request body must be a list of entry identifiers.
-
 ## `octets:workflow`
 
 Execute a [workflow](#workflows) on the entry under the request path.
@@ -227,21 +267,23 @@ A workflow is a list of endpoints to be called.
 The following input will be passed to each endpoint:
 
 ```yaml
+authority: string
 storage: string
 path: string
 entry: Entry
 parameters: Record<string, string> # route parameters
 ```
 
-See [Entry](/extensions/storages/readme.md#entry) and an
-example [workflow step processor](../features/steps/components/octets.tester).
+- [Storages](/extensions/storages/readme.md)
+- [Authorities](authorities.md)
+- Example [workflow step processor](../features/steps/components/octets.tester)
 
 A _workflow unit_ is an object with keys referencing the workflow step identifier, and an endpoint
 as value.
 Steps within a workflow unit are executed in parallel.
 
 ```yaml
-octets:store:
+octets:put:
   workflow:
     resize: images.resize
     analyze: images.analyze
@@ -251,11 +293,22 @@ A workflow can be a single unit, or an array of units.
 If it's an array, the workflow units are executed in sequence.
 
 ```yaml
-octets:store:
+octets:put:
   workflow:
     - optimize: images.optimize   # executed first
     - resize: images.resize       # executed second
       analyze: images.analyze     # executed in parallel with `resize`
 ```
 
-If one of the workflow units returns an error, the execution of the workflow is interrupted.
+If one of the workflow units returns or throws an error,
+the execution of the workflow is interrupted.
+
+### Workflow tasks
+
+A workflow unit which value starts with `task:` prefix will be executed as a Task.
+
+```yaml
+octets:put:
+  workflow:
+    optimize: task:images.optimize
+```

package/documentation/passkeys.md

@@ -0,0 +1,11 @@
+# Web Authentication
+
+- [Standard](https://www.w3.org/TR/webauthn-2/)
+- [Features](../features/passkeys.feature)
+- [Credential management](credentials.md#passkeys)
+
+## Origin
+
+The relying party is the client. The origin a passkey is registered under, and the RP ID derived
+from it, come from the request, so a passkey belongs to the origin of the client that registered
+it, and a client on another origin does not see it.