@toa.io/extensions.exposition 1.0.0-alpha.292 → 1.0.0-alpha.294

package/documentation/notes/sse.md

@@ -1,71 +0,0 @@
-# Note: Server Sent Events
-
-## TL;DR
-
-1. Doesn't exist.
-2. Ridiculous.
-3. Redundant.
-
-## There is no SSE
-
-The initial premise of SSE is to deliver real-time events specifically targeted at a particular user. It's evident that
-to access these events, user authentication is essential, typically implemented through
-an [HTTP Authentication framework](https://datatracker.ietf.org/doc/html/rfc2617).
-
-However, despite this being the most straightforward
-scenario, [it does not work this way](https://github.com/whatwg/html/issues/2177). The only viable way to use SSE is
-to avoid the standard `EventSource` implementation.
-
-## Peculiar event format
-
-### SLAP
-
-[Event stream format](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format)
-defines four fields: `event`, `data`, `id`, `retry`.
-
-In practice, code capable of producing such an object is forced to break the Single Level of Abstraction Principle. This
-requires knowledge of both event structures and connection properties. In real-world systems, these may be handled by
-different processes, such as an Events microservice and an API Gateway.
-
-### Content negotiation
-
-The `data` property is expected to be an arbitrary string, although real-world event data is often structured as an
-object.
-
-Furthermore, since the Event stream format defines its own content type, there is no built-in way to negotiate the
-format of the events.
-
-Additionally, the `data` value is limited to a single line, which hinders the use of
-more [human-friendly formats](https://yaml.org) in a straightforward manner.
-
-## HTTP is enough
-
-Rather than resorting to a non-standard client and a predefined content format, a basic HTTP request can provide a
-solution:
-
-```http
-GET /events/ HTTP/1.1
-accept: application/yaml
-```
-
-```
-200 OK
-content-type: application/yaml
-transfer-encoding: chunked
-
-id: 1
-event: created
-data:
-  foo: bar
-
-id: 2
-event: deleted
-data:
-  bar: baz
-```
-
-Server-side implementation with flow control and stream termination handling is trivial:
-
-```javascript
-eventStream.pipe(response)
-```

package/documentation/notes/throttling.md

@@ -1,122 +0,0 @@
-# Decentralized Request Throttling
-
-> **Superseded.** The design below was never implemented. `io:throttle` is now a
-> [GCRA](https://en.wikipedia.org/wiki/Generic_cell_rate_algorithm) deciding in each process, with
-> its state periodically reconciled through Redis by way of the `atom` aspect's `meter`. It
-> answers the same forces the same way — decide locally, reconcile in the background, never fail the
-> request path — and the section below records what it does instead. The problem statement and the
-> forces have not changed.
-
-## Problem
-
-To prevent an unnecessary load, API requests should be throttled when made maliciously or due to an
-error.
-
-## Context
-
-1. The [API Gateway](https://microservices.io/patterns/apigateway.html) of a distributed system runs
-   in multiple instances, each with an in-memory state that does not allow tracking the current
-   quota usage.
-2. Precise quota enforcement (per request, per second) is not critical; the goal is to prevent
-   significant overuse.
-3. Quota configuration is static.
-
-## Forces
-
-1. No per request IO is allowed (centralized solutions do not fit).
-2. In a distributed system, there is no concept of a _global current time_.
-3. Failure to retrieve the quota state should not result in Gateway failure.
-
-## Solution
-
-Implement in-memory quotas in each process, periodically synchronizing them asynchronously using
-Redis.
-
-Consider a basic throttling rule:
-
-1. No more than `MAX_REQUESTS` within `INTERVAL` time for any API route (`KEY`).
-2. If the limit is exceeded, block requests to `KEY` for `COOLDOWN` seconds.
-
-### Concept
-
-1. Divide `INTERVAL` into `N` spans (`N >= 2`).
-2. At the end of each span,
-   atomically increment the value in Redis ([INCRBY](https://redis.io/docs/latest/commands/incrby/))
-   by the number of requests received for each `KEY`, using a key composed of the `KEY` and the
-   current ordinal number of `INTERVAL`
-   in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time).
-3. If the returned value exceeds `MAX_REQUESTS`, block access to `KEY`, remove after `COOLDOWN`.
-4. If the write operation fails and `REQUESTS > MAX_REQUESTS / N` (i.e., the quota is exceeded
-   locally), block access to `KEY`, remove after `COOLDOWN`.
-
-Each API Gateway instance will generate the `KEY` based on its own local time,
-which, in general, will lead to simultaneous writes (from Redis’s local time perspective) to
-different `KEY`s. However, the total contribution of each node to each `KEY` will correspond to the
-actual request rate experienced by that node.
-
-<img src="desync.jpg" alt="Desynchronization" />
-
-Dividing the `INTERVAL` into spans smooths the desynchronization effect.
-
-### Extension
-
-Introduce `nodes`, the approximate number of active API Gateway instances, to improve the algorithm.
-
-1. Initially, `nodes` equals to `1`.
-2. During each `INTERVAL`, sum the total request count for each `KEY`, keep current and previous
-   values.
-3. At the end of each interval:
-
-- Read the value from the `KEY` for the previous interval. At this point, it is assumed that all
-  nodes have switched to the next interval.
-- Update the `nodes` value by dividing response form Redis by the number of `REQUESTS` counted for
-  the previous interval.
-- Set the previous value to the current value.
-- Set the current value to `0`
-
-4. Update point 4 of the [Concept](#concept): `REQUESTS * nodes > MAX_REQUESTS / N`.
-   This will increase the precision of the local quota enforcement.
-5. Add a new step: if `REQUESTS * nodes > MAX_REQUESTS`, block access to `KEY`, remove after
-   `COOLDOWN`.
-
-When nodes are added or removed, the algorithm will adapt in the upcoming intervals.
-
-### Caveats
-
-1. In the worst case scenario (really, really unlikely), the quota is exceeded by `MAX_REQUESTS / N`
-   in a span on each node.
-2. Time desynchronization between nodes should be insignificant for the selected `INTERVAL` (i.e.,
-   `INTERVAL` >> desync). See Extension point 3.
-
-## What was implemented instead
-
-Windows were dropped for a GCRA. A key holds one theoretical arrival time: the moment it would be
-back at zero if nothing more arrived. An admitted request pushes it `interval / requests`
-milliseconds further out, time drags it back, and a request is admitted while it stays within
-`interval` of now. So `requests` is what a key may spend at once and `requests / interval` the rate
-it earns back, with no window edge to burst across and no lockout to time out of — which is why
-`cooldown` is gone.
-
-Deciding reads nothing but a local map, so the request path stays free of I/O, exactly as force 1
-demands. What the other gateways have spent arrives on a timer, as **debt**: the milliseconds a key
-owes, draining at a millisecond a millisecond. Debt is the right thing to send for two reasons.
-
-1. It is a duration, not a moment, so processes exchanging it need not agree on the time — force 2,
-   answered by not needing an answer. The one clock that matters is Redis', read inside the script.
-2. It is additive in admissions, so a process reports only its own increments and Redis keeps the
-   running total. Nothing has to be divided by a number of nodes, which is as well: the gateway
-   cannot observe how many of it there are.
-
-The consequences worth knowing:
-
-- **Reconciling is decoupled from `interval`.** The timer runs at a tenth of `interval`, clamped
-  between 250ms and 2s, so a minute-long budget is no staler than a second-long one. The overshoot
-  is bounded by what the other instances admit within one tick — the caveat that replaces the old
-  note's `MAX_REQUESTS / N` per span.
-- **The cost is one round trip a tick**, not one per key. Every quota in the process flushes into a
-  single script call, which matters because a limiter keyed on `ip` watches as many keys as it has
-  clients.
-- **Redis keys expire on their own**, after the debt on them plus a grace, so nothing sweeps them.
-- **Failure degrades, as force 3 demands.** A tick that cannot reach Redis leaves its debt to the
-  next one rather than losing it; until then the instance throttles on what it alone has seen, and
-  keeps serving.

package/documentation/oauth.md

@@ -1,180 +0,0 @@
-# OAuth 2.1 Authorization Server
-
-An application exposes an authorization server by annotating one, and none without it. A client
-authorized this way holds a token of the identity that consented, and presents it as `Bearer`.
-
-## Annotation
-
-```yaml
-exposition:
-  authorities:
-    local: api.example.com
-  oauth:
-    authorize: https://app.example.com/oauth/authorize
-    resources: ['/.mcp']
-    scopes: [app:notes, app:notes:read]
-    registration: closed
-```
-
-<dl>
-<dt><code>authorize</code></dt>
-<dd>Required. The absolute URL of the consent page. The application serves it; see
-<a href="consent.md">Consent</a>.</dd>
-<dt><code>resources</code></dt>
-<dd>Paths advertised as protected resources, each with a document of its own and named by the
-challenge a request to it is refused with. <code>/.mcp</code> is where
-<a href="mcp.md">MCP</a> is served. A token is not yet bound to one, see
-<a href="#audience">Audience</a>.</dd>
-<dt><code>scopes</code></dt>
-<dd>Advertised as what a client may ask for. Each is a <a href="access.md#roles">role</a>, or a
-scope within one.</dd>
-<dt><code>registration</code></dt>
-<dd><code>open</code> advertises the registration endpoint, <code>closed</code> does not.
-Defaults to <code>closed</code>.</dd>
-</dl>
-
-The issuer is the authority's configured host, over `https` unless the host is a loopback one.
-
-## Discovery
-
-Two documents are served, at the paths their specifications fix, to anyone.
-
-`/.well-known/oauth-authorization-server` is
-[RFC 8414](https://www.rfc-editor.org/rfc/rfc8414) metadata:
-
-```yaml
-issuer: https://api.example.com
-authorization_endpoint: https://app.example.com/oauth/authorize
-token_endpoint: https://api.example.com/identity/grants/
-registration_endpoint: https://api.example.com/identity/clients/
-response_types_supported: [code]
-grant_types_supported: [authorization_code]
-code_challenge_methods_supported: [S256]
-token_endpoint_auth_methods_supported: [none]
-client_id_metadata_document_supported: true
-authorization_response_iss_parameter_supported: true
-```
-
-`/.well-known/oauth-protected-resource` is
-[RFC 9728](https://www.rfc-editor.org/rfc/rfc9728) metadata. Each configured resource has one of
-its own, at the path the well-known segment is followed by: `/.mcp` is read at
-`/.well-known/oauth-protected-resource/.mcp`.
-
-```yaml
-resource: https://api.example.com/.mcp
-authorization_servers: [https://api.example.com]
-bearer_methods_supported: [header]
-```
-
-`/.well-known/openid-configuration` answers with the authorization server metadata, for a client
-that reads only that.
-
-A reply of `401` carries the challenge that names the document:
-
-```http
-401 Unauthorized
-www-authenticate: Bearer resource_metadata="https://api.example.com/.well-known/oauth-protected-resource/.mcp", scope="app:notes"
-```
-
-## Clients
-
-A `client_id` that is an `https` URL is a
-[Client ID Metadata Document](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-client-id-metadata-document-00):
-the client publishes what it is at that URL, and the document is read from there. Only origins
-`identity.clients` is configured to trust are read, and an empty list reads none:
-
-```yaml
-configuration:
-  identity.clients:
-    trust:
-      - https://claude.ai
-```
-
-Anything else was registered through
-[RFC 7591](https://www.rfc-editor.org/rfc/rfc7591), which `registration: open` advertises:
-
-```http
-POST /identity/clients/ HTTP/1.1
-content-type: application/json
-
-{"client_name": "Claude", "redirect_uris": ["https://claude.ai/api/mcp/auth_callback"]}
-```
-
-```http
-201 Created
-
-{"client_id": "c8f3a1e4d8c9b2f6a0e5d7c3b1a9f8e2", "client_name": "Claude", …}
-```
-
-A registration is addressed by a hash of what it says, so the same metadata is one client however
-many times it arrives. A registration cannot be changed: a revision is a different client.
-
-Only public clients are supported. `token_endpoint_auth_method` other than `none` is refused.
-
-`redirect_uris` are matched exactly, except for a loopback address, where the port is ignored
-([RFC 8252 §7.3](https://datatracker.ietf.org/doc/html/rfc8252#section-7.3)).
-
-`GET /identity/clients/:id/` returns a client to any authenticated identity, which is what the
-consent page reads.
-
-## Token endpoint
-
-`POST /identity/grants/`, anonymous, `application/x-www-form-urlencoded`.
-
-```http
-POST /identity/grants/ HTTP/1.1
-content-type: application/x-www-form-urlencoded
-
-grant_type=authorization_code&code=SplxlO&code_verifier=dBjftJeZ&redirect_uri=…&client_id=…
-```
-
-```http
-200 OK
-cache-control: no-store
-
-{"access_token": "…", "token_type": "Bearer", "expires_in": 2592000}
-```
-
-`code_verifier` is required: the code was issued against its `S256` hash
-([RFC 7636](https://www.rfc-editor.org/rfc/rfc7636)). A code may be redeemed once, and is spent
-whether or not the verifier matches. Every refusal is `invalid_grant`.
-
-A code is valid for `identity.grants.lifetime` seconds, 60 by default. A token is valid for
-`identity.grants.token` seconds, 30 days by default; `0` is until it is revoked.
-
-## Grants
-
-A grant is what a user allowed a client. One is held per identity per client.
-
-`GET /identity/grants/:identity/` lists them. `DELETE /identity/grants/:identity/:id/` revokes one,
-which disables the key its token was issued under: the token stops being one within
-`identity.tokens.cache.ttl`.
-
-## Tokens
-
-An access token is a token of the identity that consented, carrying the
-[roles](access.md#roles) that identity holds, or the subset the client asked for as `scope`. It is
-presented as `Bearer`, which `identity.federation` also answers to — see
-[Bearer scheme](identity.md#bearer-scheme).
-
-A `scope` a client asks for is not checked against `scopes_supported`: it is granted if the
-consenting identity holds it, and refused as `invalid_scope` otherwise. What is advertised says
-what an application means to offer, not what the server will refuse.
-
-## Audience
-
-`resource` ([RFC 8707](https://www.rfc-editor.org/rfc/rfc8707)) is accepted at both endpoints and
-recorded on the grant. **It does not yet restrict the token.** An access token carries the rights
-of the identity that consented, over every path that identity may reach, whichever resource it was
-asked for.
-
-Restricting it is a `permissions` argument at the `identity.tokens.issue` call in
-`identity.grants.exchange` — `{'/.mcp': ['*']}` for a resource at `/.mcp` — enforced by
-`permits()`, which is already what reads a token's permissions on every request. It waits on the
-resource that will check the audience itself.
-
-## References
-
-- [OAuth 2.1](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1)
-- [Consent](consent.md), the page an application serves
-- [Features](../features/oauth.grants.feature)

package/documentation/octets.md

@@ -1,334 +0,0 @@
-# BLOBs
-
-The `octets` directive family implements operations with BLOBs, using
-the [Storages extention](/extensions/storages).
-The most common use case is to handle file uploads, downloads, and processing.
-
-## `octets:context`
-
-Sets the [storage name](/extensions/storages/readme.md#annotation) to be used for the `octets`
-directives under the current RTD Node.
-
-```yaml
-/images:
-  octets:context: images
-```
-
-## `octets:put`
-
-Stores the content of the request body into a storage, under the request path with
-specified `content-type`.
-
-If request's `content-type` is not acceptable, or if the request body does not pass
-the [validation](/extensions/storages/readme.md#async-putpath-string-stream-readable-options-options-maybeentry),
-the request is rejected with a `415 Unsupported Media Type` response.
-
-The value of the directive must be `null` (defaults) or an object with the following optional
-properties:
-
-- `location`: a string that represents the path to store the
-  content.
-  If not specified, the path is the same as the request path.
-- `limit`: [maximum size](#stream-size-limit) of the incoming stream.
-- `accept`: a media type or an array of media types that are acceptable.
-  If the `accept` property is not specified, any media type is acceptable (which is the default).
-- `workflow`: [workflow](#workflows) to be executed once the content is successfully stored.
-- `trust`: a list of [trusted origins](#downloading-external-content).
-
-```yaml
-/images:
-  octets:context: images
-  POST:
-    octets:put:
-      accept:
-        - image/jpeg
-        - image/png
-        - video/*
-      workflow:
-        resize: images.resize
-        analyze: images.analyze
-```
-
-### Headers
-
-`content-id` header can be used to set the ID of the Entry.
-The value must match the following regular expression `^[a-zA-Z0-9-_]{1,16}$`.
-
-Non-standard `content-attributes` header can be used
-to set initial [metadata](/extensions/storages/readme.md#entry) value for the Entry.
-
-The value of the `content-attributes` header is a comma-separated list of key-value string pairs.
-If no value is provided for a key, the string `true` is used.
-
-```http
-POST /images/ HTTP/1.1
-content-type: image/jpeg
-content-id: example-id
-content-attributes: foo, bar=baz
-content-attributes: baz=1
-```
-
-```yaml
-attributes:
-  foo: 'true'
-  bar: 'baz'
-  baz: '1'
-```
-
-If the Entry already exists, the `content-attributes` header is ignored.
-
-### Location
-
-The `location` property can be used to store the content under a different path.
-
-```yaml
-/images:
-  octets:context: images
-  POST:
-    octets:put:
-      location: /archive
-```
-
-Physical storage path is constructed by resolving the `location`
-property [relative](https://datatracker.ietf.org/doc/html/rfc3986#section-5) to the request path.
-
-### Stream size limit
-
-The `limit` property can be used to set the maximum size of the incoming stream in bytes.
-
-The property value can be specified as a number
-(representing bytes) or a string that combines a number with a unit (e.g., `1MB`).
-Both [binary and decimal prefixes](https://en.wikipedia.org/wiki/Binary_prefix) are supported.
-If the prefix or unit is specified _incorrectly_ (e.g., `1mb`),
-it will default to a binary prefix interpretation.
-
-- `1b`, `1B`: 1 byte
-- `1KB`: 1000 bytes
-- `1KiB`: 1024 bytes
-- `1kb`: 1024 bytes
-
-The default value is `64MiB`.
-
-### Downloading external content
-
-The `octets:put` directive can be used to download external content:
-
-```http
-POST /images/ HTTP/1.1
-content-location: https://example.com/image.jpg
-content-length: 0
-```
-
-Requests with `content-location` header must have an empty body (`content-length: 0` header).
-
-Target origin must be allowed by the `trust` property,
-which can contain a list of trusted origins or regular expressions to match the full URL.
-
-URL of the downloaded content is stored in the `origin` property of
-the [Entry](/extensions/storages/readme.md#entry).
-
-```yaml
-/images:
-  octets:context: images
-  POST:
-    octets:put:
-      trust:
-        - https://example.com
-        - ^https://example\.com/[a-z]+\.jpe?g$
-```
-
-### Response
-
-The response of the `octets:put` directive is the created Entry.
-
-```
-201 Created
-content-type: application/yaml
-
-id: eecd837c
-type: image/jpeg
-created: 1698004822358
-```
-
-If the `octets:put` directive contains a `workflow`, the response
-is [multipart](protocol.md#multipart-types).
-The first part represents the created Entry, which is sent immediately after the BLOB is stored,
-while subsequent parts are results from the workflow endpoints, sent as soon as they are available.
-
-In case a workflow endpoint returns an `Error`, the error part is sent,
-and the response is closed.
-Error's properties are added to the error part, among with the `step` identifier.
-
-```
-201 Created
-content-type: multipart/yaml; boundary=cut
-
---cut
-
-id: eecd837c
-type: image/jpeg
-created: 1698004822358
-
---cut
-
-step: optimize
-status: completed
-
---cut
-
-step: resize
-error:
-  code: TOO_SMALL
-  message: Image is too small
-status: completed
-
---cut
-
-step: analyze
-status: exception
-
---cut--
-```
-
-### What a client is told
-
-`octets:put` says what sending a file takes, where everything else about a method is said —
-[`OPTIONS`](introspection.md) and the [tree](discovery.md):
-
-```yaml
-POST:
-  octets:
-    accept: image/jpeg,image/png
-    limit: 64MiB
-    stream: true
-```
-
-`accept` is absent where the directive names no type, and anything may be sent. `limit` is
-written as the declaration writes it, and is what a `413` reports. `stream` says a workflow runs
-on what was stored, so the reply arrives as parts rather than as one object — the first part is
-the entry.
-
-Reading a file and deleting one say nothing: those requests are what they look like.
-
-## `octets:get`
-
-Fetches the content of a stored BLOB corresponding to the request path, and returns it as the
-response body with the corresponding `content-type`, `content-length`
-and `etag` ([conditional GET](https://datatracker.ietf.org/doc/html/rfc2616#section-9.3) is
-also supported).
-The `accept` request header is disregarded.
-
-The value of the directive is an object with the following properties:
-
-- `meta`: `boolean` indicating whether an Entry is accessible.
-  Defaults to `false`.
-
-```yaml
-/images:
-  octets:context: images
-  /*:
-    GET:
-      octets:get:
-        blob: false # prevent access to the original BLOB
-        meta: true # allow access to an Entry
-```
-
-The `octets:get: ~` declaration is equivalent to defaults.
-
-To access an Entry, the `accept` request header must contain the `octets.entry` subtype
-in
-the `toa` [vendor tree](https://datatracker.ietf.org/doc/html/rfc6838#section-3.2):
-
-```http
-GET /images/eecd837c HTTP/1.1
-accept: application/vnd.toa.octets.entry+yaml
-```
-
-## `octets:delete`
-
-Delete the entry corresponding to the request path.
-
-```yaml
-/images:
-  octets:context: images
-  DELETE:
-    octets:delete: ~
-```
-
-The value of the directive may contain a [workflow](#workflows) declaration, to be executed before
-the entry is deleted.
-
-```yaml
-/images:
-  octets:context: images
-  DELETE:
-    octets:delete:
-      workflow:
-        cleanup: images.cleanup
-```
-
-The error returned by the workflow prevents the deletion of the entry.
-
-## `octets:workflow`
-
-Execute a [workflow](#workflows) on the entry under the request path.
-
-```yaml
-/images:
-  /*:
-    DELETE:
-      octets:workflow:
-        archive: images.archive
-```
-
-## Workflows
-
-A workflow is a list of endpoints to be called.
-The following input will be passed to each endpoint:
-
-```yaml
-authority: string
-storage: string
-path: string
-entry: Entry
-parameters: Record<string, string> # route parameters
-```
-
-- [Storages](/extensions/storages/readme.md)
-- [Authorities](authorities.md)
-- Example [workflow step processor](../features/steps/components/octets.tester)
-
-A _workflow unit_ is an object with keys referencing the workflow step identifier, and an endpoint
-as value.
-Steps within a workflow unit are executed in parallel.
-
-```yaml
-octets:put:
-  workflow:
-    resize: images.resize
-    analyze: images.analyze
-```
-
-A workflow can be a single unit, or an array of units.
-If it's an array, the workflow units are executed in sequence.
-
-```yaml
-octets:put:
-  workflow:
-    - optimize: images.optimize # executed first
-    - resize: images.resize # executed second
-      analyze: images.analyze # executed in parallel with `resize`
-```
-
-If one of the workflow units returns or throws an error,
-the execution of the workflow is interrupted.
-
-### Workflow tasks
-
-A workflow unit which value starts with `task:` prefix will be executed as a Task.
-
-```yaml
-octets:put:
-  workflow:
-    optimize: task:images.optimize
-```