@toa.io/extensions.exposition 1.0.0-alpha.28 → 1.0.0-alpha.282

package/documentation/notes/throttling.md

@@ -0,0 +1,122 @@
+# Decentralized Request Throttling
+
+> **Superseded.** The design below was never implemented. `io:throttle` is now a
+> [GCRA](https://en.wikipedia.org/wiki/Generic_cell_rate_algorithm) deciding in each process, with
+> its state periodically reconciled through Redis by way of the `atom` aspect's `meter`. It
+> answers the same forces the same way — decide locally, reconcile in the background, never fail the
+> request path — and the section below records what it does instead. The problem statement and the
+> forces have not changed.
+
+## Problem
+
+To prevent an unnecessary load, API requests should be throttled when made maliciously or due to an
+error.
+
+## Context
+
+1. The [API Gateway](https://microservices.io/patterns/apigateway.html) of a distributed system runs
+   in multiple instances, each with an in-memory state that does not allow tracking the current
+   quota usage.
+2. Precise quota enforcement (per request, per second) is not critical; the goal is to prevent
+   significant overuse.
+3. Quota configuration is static.
+
+## Forces
+
+1. No per request IO is allowed (centralized solutions do not fit).
+2. In a distributed system, there is no concept of a *global current time*.
+3. Failure to retrieve the quota state should not result in Gateway failure.
+
+## Solution
+
+Implement in-memory quotas in each process, periodically synchronizing them asynchronously using
+Redis.
+
+Consider a basic throttling rule:
+
+1. No more than `MAX_REQUESTS` within `INTERVAL` time for any API route (`KEY`).
+2. If the limit is exceeded, block requests to `KEY` for `COOLDOWN` seconds.
+
+### Concept
+
+1. Divide `INTERVAL` into `N` spans (`N >= 2`).
+2. At the end of each span,
+   atomically increment the value in Redis ([INCRBY](https://redis.io/docs/latest/commands/incrby/))
+   by the number of requests received for each `KEY`, using a key composed of the `KEY` and the
+   current ordinal number of `INTERVAL`
+   in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time).
+3. If the returned value exceeds `MAX_REQUESTS`, block access to `KEY`, remove after `COOLDOWN`.
+4. If the write operation fails and `REQUESTS > MAX_REQUESTS / N` (i.e., the quota is exceeded
+   locally), block access to `KEY`, remove after `COOLDOWN`.
+
+Each API Gateway instance will generate the `KEY` based on its own local time,
+which, in general, will lead to simultaneous writes (from Redis’s local time perspective) to
+different `KEY`s. However, the total contribution of each node to each `KEY` will correspond to the
+actual request rate experienced by that node.
+
+<img src="desync.jpg" alt="Desynchronization" />
+
+Dividing the `INTERVAL` into spans smooths the desynchronization effect.
+
+### Extension
+
+Introduce `nodes`, the approximate number of active API Gateway instances, to improve the algorithm.
+
+1. Initially, `nodes` equals to `1`.
+2. During each `INTERVAL`, sum the total request count for each `KEY`, keep current and previous
+   values.
+3. At the end of each interval:
+
+- Read the value from the `KEY` for the previous interval. At this point, it is assumed that all
+  nodes have switched to the next interval.
+- Update the `nodes` value by dividing response form Redis by the number of `REQUESTS` counted for
+  the previous interval.
+- Set the previous value to the current value.
+- Set the current value to `0`
+
+4. Update point 4 of the [Concept](#concept): `REQUESTS * nodes > MAX_REQUESTS / N`.
+   This will increase the precision of the local quota enforcement.
+5. Add a new step: if `REQUESTS * nodes > MAX_REQUESTS`, block access to `KEY`, remove after
+   `COOLDOWN`.
+
+When nodes are added or removed, the algorithm will adapt in the upcoming intervals.
+
+### Caveats
+
+1. In the worst case scenario (really, really unlikely), the quota is exceeded by `MAX_REQUESTS / N`
+   in a span on each node.
+2. Time desynchronization between nodes should be insignificant for the selected `INTERVAL` (i.e.,
+   `INTERVAL` >> desync). See Extension point 3.
+
+## What was implemented instead
+
+Windows were dropped for a GCRA. A key holds one theoretical arrival time: the moment it would be
+back at zero if nothing more arrived. An admitted request pushes it `interval / requests`
+milliseconds further out, time drags it back, and a request is admitted while it stays within
+`interval` of now. So `requests` is what a key may spend at once and `requests / interval` the rate
+it earns back, with no window edge to burst across and no lockout to time out of — which is why
+`cooldown` is gone.
+
+Deciding reads nothing but a local map, so the request path stays free of I/O, exactly as force 1
+demands. What the other gateways have spent arrives on a timer, as **debt**: the milliseconds a key
+owes, draining at a millisecond a millisecond. Debt is the right thing to send for two reasons.
+
+1. It is a duration, not a moment, so processes exchanging it need not agree on the time — force 2,
+   answered by not needing an answer. The one clock that matters is Redis', read inside the script.
+2. It is additive in admissions, so a process reports only its own increments and Redis keeps the
+   running total. Nothing has to be divided by a number of nodes, which is as well: the gateway
+   cannot observe how many of it there are.
+
+The consequences worth knowing:
+
+- **Reconciling is decoupled from `interval`.** The timer runs at a tenth of `interval`, clamped
+  between 250ms and 2s, so a minute-long budget is no staler than a second-long one. The overshoot
+  is bounded by what the other instances admit within one tick — the caveat that replaces the old
+  note's `MAX_REQUESTS / N` per span.
+- **The cost is one round trip a tick**, not one per key. Every quota in the process flushes into a
+  single script call, which matters because a limiter keyed on `ip` watches as many keys as it has
+  clients.
+- **Redis keys expire on their own**, after the debt on them plus a grace, so nothing sweeps them.
+- **Failure degrades, as force 3 demands.** A tick that cannot reach Redis leaves its debt to the
+  next one rather than losing it; until then the instance throttles on what it alone has seen, and
+  keeps serving.

package/documentation/octets.md

@@ -14,26 +14,32 @@ directives under the current RTD Node.
   octets:context: images
 ```
 
-## `octets:store`
+## `octets:put`
 
 Stores the content of the request body into a storage, under the request path with
 specified `content-type`.
 
 If request's `content-type` is not acceptable, or if the request body does not pass
-the [validation](/extensions/storages/readme.md#async-putpath-string-stream-readable-type-typecontrol-maybeentry),
+the [validation](/extensions/storages/readme.md#async-putpath-string-stream-readable-options-options-maybeentry),
 the request is rejected with a `415 Unsupported Media Type` response.
 
-The value of the directive is `null` or an object with the following properties:
+The value of the directive must be `null` (defaults) or an object with the following optional
+properties:
 
+- `location`: a string that represents the path to store the
+  content.
+  If not specified, the path is the same as the request path.
+- `limit`: [maximum size](#stream-size-limit) of the incoming stream.
 - `accept`: a media type or an array of media types that are acceptable.
   If the `accept` property is not specified, any media type is acceptable (which is the default).
 - `workflow`: [workflow](#workflows) to be executed once the content is successfully stored.
+- `trust`: a list of [trusted origins](#downloading-external-content).
 
 ```yaml
 /images:
   octets:context: images
   POST:
-    octets:store:
+    octets:put:
       accept:
         - image/jpeg
         - image/png
@@ -43,31 +49,97 @@ The value of the directive is `null` or an object with the following properties:
         analyze: images.analyze
 ```
 
-Non-standard `content-meta` header can be used
+### Headers
+
+`content-id` header can be used to set the ID of the Entry.
+The value must match the following regular expression `^[a-zA-Z0-9-_]{1,16}$`.
+
+Non-standard `content-attributes` header can be used
 to set initial [metadata](/extensions/storages/readme.md#entry) value for the Entry.
 
-The value of the `content-meta` header is a comma-separated list of key-value string pairs.
+The value of the `content-attributes` header is a comma-separated list of key-value string pairs.
 If no value is provided for a key, the string `true` is used.
 
 ```http
 POST /images/ HTTP/1.1
 content-type: image/jpeg
-content-meta: foo, bar=baz
-content-meta: baz=1
+content-id: example-id
+content-attributes: foo, bar=baz
+content-attributes: baz=1
 ```
 
 ```yaml
-meta:
+attributes:
   foo: 'true'
   bar: 'baz'
   baz: '1'
 ```
 
-If the Entry already exists, the `content-meta` header is ignored.
+If the Entry already exists, the `content-attributes` header is ignored.
+
+### Location
+
+The `location` property can be used to store the content under a different path.
+
+```yaml
+/images:
+  octets:context: images
+  POST:
+    octets:put:
+      location: /archive
+```
+
+Physical storage path is constructed by resolving the `location`
+property [relative](https://datatracker.ietf.org/doc/html/rfc3986#section-5) to the request path.
+
+### Stream size limit
+
+The `limit` property can be used to set the maximum size of the incoming stream in bytes.
+
+The property value can be specified as a number
+(representing bytes) or a string that combines a number with a unit (e.g., `1MB`).
+Both [binary and decimal prefixes](https://en.wikipedia.org/wiki/Binary_prefix) are supported.
+If the prefix or unit is specified _incorrectly_ (e.g., `1mb`),
+it will default to a binary prefix interpretation.
+
+- `1b`, `1B`: 1 byte
+- `1KB`: 1000 bytes
+- `1KiB`: 1024 bytes
+- `1kb`: 1024 bytes
+
+The default value is `64MiB`.
+
+### Downloading external content
+
+The `octets:put` directive can be used to download external content:
+
+```http
+POST /images/ HTTP/1.1
+content-location: https://example.com/image.jpg
+content-length: 0
+```
+
+Requests with `content-location` header must have an empty body (`content-length: 0` header).
+
+Target origin must be allowed by the `trust` property,
+which can contain a list of trusted origins or regular expressions to match the full URL.
+
+URL of the downloaded content is stored in the `origin` property of
+the [Entry](/extensions/storages/readme.md#entry).
+
+```yaml
+/images:
+  octets:context: images
+  POST:
+    octets:put:
+      trust:
+        - https://example.com
+        - ^https://example\.com/[a-z]+\.jpe?g$
+```
 
 ### Response
 
-The response of the `octets:store` directive is the created Entry.
+The response of the `octets:put` directive is the created Entry.
 
 ```
 201 Created
@@ -78,12 +150,13 @@ type: image/jpeg
 created: 1698004822358
 ```
 
-If the `octets:store` directive contains a `workflow`, the response
+If the `octets:put` directive contains a `workflow`, the response
 is [multipart](protocol.md#multipart-types).
 The first part represents the created Entry, which is sent immediately after the BLOB is stored,
 while subsequent parts are results from the workflow endpoints, sent as soon as they are available.
 
-In case a workflow endpoint returns an `Error`, the error part is sent, and the response is closed.
+In case a workflow endpoint returns an `Error`, the error part is sent,
+and the response is closed.
 Error's properties are added to the error part, among with the `step` identifier.
 
 ```
@@ -91,20 +164,33 @@ Error's properties are added to the error part, among with the `step` identifier
 content-type: multipart/yaml; boundary=cut
 
 --cut
+
 id: eecd837c
 type: image/jpeg
 created: 1698004822358
+
 --cut
-optimize: null
+
+step: optimize
+status: completed
+
 --cut
+
+step: resize
 error:
-  step: resize
   code: TOO_SMALL
   message: Image is too small
+status: completed
+
+--cut
+
+step: analyze
+status: exception
+
 --cut--
 ```
 
-## `octets:fetch`
+## `octets:get`
 
 Fetches the content of a stored BLOB corresponding to the request path, and returns it as the
 response body with the corresponding `content-type`, `content-length`
@@ -116,22 +202,18 @@ The value of the directive is an object with the following properties:
 
 - `meta`: `boolean` indicating whether an Entry is accessible.
   Defaults to `false`.
-- `blob`: `boolean` indicating whether the original BLOB is accessible,
-  [BLOB variant](/extensions/storages/readme.md#async-fetchpath-string-maybereadable) must be
-  specified in the path otherwise.
-  Defaults to `true`.
 
 ```yaml
 /images:
   octets:context: images
   /*:
     GET:
-      octets:fetch:
+      octets:get:
         blob: false # prevent access to the original BLOB
         meta: true  # allow access to an Entry
 ```
 
-The `octets:fetch: ~` declaration is equivalent to defaults.
+The `octets:get: ~` declaration is equivalent to defaults.
 
 To access an Entry, the `accept` request header must contain the `octets.entry` subtype
 in
@@ -142,32 +224,6 @@ GET /images/eecd837c HTTP/1.1
 accept: application/vnd.toa.octets.entry+yaml
 ```
 
-## `octets:list`
-
-Lists the entries stored under the request path.
-
-The value of the directive is an object with the following properties:
-
-- `meta`: `boolean` indicating whether the list of Entries is accessible.
-  Defaults to `false`, which means that only entry identifiers are returned.
-
-```yaml
-/images:
-  octets:context: images
-  GET:
-    octets:list:
-      meta: true
-```
-
-The `octets:list: ~` declaration is equivalent to defaults.
-
-To access a list of Entries, the `accept` request header must contain the `octets.entries` subtype:
-
-```http
-GET /images/ HTTP/1.1
-accept: application/vnd.toa.octets.entries+yaml
-```
-
 ## `octets:delete`
 
 Delete the entry corresponding to the request path.
@@ -193,22 +249,6 @@ the entry is deleted.
 
 The error returned by the workflow prevents the deletion of the entry.
 
-## `octets:permute`
-
-Performs
-a [permutation](/extensions/storages/readme.md#async-permutepath-string-ids-string-maybevoid) on the
-entries
-under the request path.
-
-```yaml
-/images:
-  octets:context: images
-  PUT:
-    octets:permute: ~
-```
-
-The request body must be a list of entry identifiers.
-
 ## `octets:workflow`
 
 Execute a [workflow](#workflows) on the entry under the request path.
@@ -227,21 +267,23 @@ A workflow is a list of endpoints to be called.
 The following input will be passed to each endpoint:
 
 ```yaml
+authority: string
 storage: string
 path: string
 entry: Entry
 parameters: Record<string, string> # route parameters
 ```
 
-See [Entry](/extensions/storages/readme.md#entry) and an
-example [workflow step processor](../features/steps/components/octets.tester).
+- [Storages](/extensions/storages/readme.md)
+- [Authorities](authorities.md)
+- Example [workflow step processor](../features/steps/components/octets.tester)
 
 A _workflow unit_ is an object with keys referencing the workflow step identifier, and an endpoint
 as value.
 Steps within a workflow unit are executed in parallel.
 
 ```yaml
-octets:store:
+octets:put:
   workflow:
     resize: images.resize
     analyze: images.analyze
@@ -251,11 +293,22 @@ A workflow can be a single unit, or an array of units.
 If it's an array, the workflow units are executed in sequence.
 
 ```yaml
-octets:store:
+octets:put:
   workflow:
     - optimize: images.optimize   # executed first
     - resize: images.resize       # executed second
       analyze: images.analyze     # executed in parallel with `resize`
 ```
 
-If one of the workflow units returns an error, the execution of the workflow is interrupted.
+If one of the workflow units returns or throws an error,
+the execution of the workflow is interrupted.
+
+### Workflow tasks
+
+A workflow unit which value starts with `task:` prefix will be executed as a Task.
+
+```yaml
+octets:put:
+  workflow:
+    optimize: task:images.optimize
+```

package/documentation/passkeys.md

@@ -0,0 +1,11 @@
+# Web Authentication
+
+- [Standard](https://www.w3.org/TR/webauthn-2/)
+- [Features](../features/passkeys.feature)
+- [Credential management](credentials.md#passkeys)
+
+## Origin
+
+The relying party is the client. The origin a passkey is registered under, and the RP ID derived
+from it, come from the request, so a passkey belongs to the origin of the client that registered
+it, and a client on another origin does not see it.

package/documentation/protocol.md

@@ -4,9 +4,9 @@
 
 The following media types are supported for both requests and responses:
 
-- `application/msgpack` using [msgpackr](https://github.com/kriszyp/msgpackr)
-- `application/yaml` using [js-yaml](https://github.com/nodeca/js-yaml)
 - `application/json`
+- `application/yaml` using [js-yaml](https://github.com/nodeca/js-yaml)
+- `application/msgpack` using [msgpackr](https://github.com/kriszyp/msgpackr)
 - `text/plain`
 
 The response format is determined by content negotiation
@@ -26,7 +26,7 @@ foo: bar
 
 ### Multipart types
 
-Multipart responses are endoded using content negotiation,
+Multipart responses are encoded using content negotiation,
 and the `content-type` of the response is set to one of the custom `multipart/` subtypes,
 corresponding to the type of
 the parts:
@@ -38,6 +38,9 @@ the parts:
 | `multipart/json`    | `application/json`    |
 | `multipart/text`    | `text/plain`          |
 
+Multipart responses are started with a text chunk `ACK`, and finished with a text
+chunk `FIN`.
+
 Example:
 
 ```
@@ -49,10 +52,14 @@ accept: application/yaml
 200 OK
 content-type: multipart/yaml; boundary=cut
 
+--cut
+ACK
 --cut
 foo: bar
 --cut
 baz: qux
+--cut
+FIN
 --cut--
 ```
 
@@ -62,6 +69,30 @@ See also:
 - [Content-Type: multipart](https://learn.microsoft.com/en-us/previous-versions/office/developer/exchange-server-2010/aa493937(v=exchg.140))
   at Microsoft
 
+## HTTP version
+
+The gateway serves HTTP/1.1 by default. `protocol: h2c` serves cleartext HTTP/2 instead.
+
+```yaml
+exposition:
+  protocol: h2c
+  service:
+    annotations:
+      projectcontour.io/upstream-protocol.h2c: "8000"
+```
+
+`h2c` requires an ingress controller that proxies cleartext HTTP/2 upstream, and the
+annotation that controller reads to do so. Contour, Traefik and Envoy-based controllers
+proxy it; ingress-nginx does not. A cleartext HTTP/2 server answers nothing else on the
+same port: there is no ALPN to negotiate with, so a controller that speaks HTTP/1.1 upstream
+reaches nothing.
+
+Under `h2c` a request refused before its body is read is answered and then cancelled with
+`RST_STREAM(NO_ERROR)`. The reply arrives; the upload stops where it is. A client streaming
+a request body sees that body cancelled — in Node, `ERR_STREAM_PREMATURE_CLOSE` on the stream
+it passed, which throws if nothing handles it. A real failure shows in the reply, or in the
+request rejecting because no reply came. A buffered body is unaffected.
+
 ## CORS
 
 [CORS](https://www.w3.org/TR/2020/SPSD-cors-20200602/) is supported,
@@ -75,7 +106,7 @@ The following request headers are allowed:
 - `etag`
 - `if-match`
 - `if-none-match`
-- headers used by the [`vary:embed` directive](vary.md#embeddings)
+- headers used by the [`map` directive family](map.md)
 
 The following response headers are exposed:
 

package/documentation/query.md

@@ -30,6 +30,7 @@ exposition:
         query:
           criteria: state==hot
           sort: rank:desc
+          text: true
           limit: 10
   /latest:
     GET:
@@ -148,6 +149,27 @@ rank: 5
 
 > In case of conflict, path variables override input properties.
 
+## Text search
+
+For entities with `text` indexes, search queries can be enabled using the `search` property.
+
+```yaml
+# manifest.toa.yaml
+
+name: dummies
+
+exposition:
+  /:type:
+    GET:
+      endpoint: observe
+      query:
+        search: true
+```
+
+```yaml
+GET /dummies/?search=some+text+query
+```
+
 ## Omit, limit
 
 `omit` and `limit` properties can declare their default values and allowed boundaries:
@@ -177,6 +199,10 @@ Constant values can be declared using the shortcut:
 limit: 10
 ```
 
+```http
+GET /dummies/?omit=100&limit=10
+```
+
 ## Sort
 
 The `sort` query property defines the result order of Observations within an `objects` scope
@@ -213,6 +239,8 @@ GET /dummies/?sort=timestamp:asc
 
 ## Selectors
 
+![Not implemented](https://img.shields.io/badge/Not_implemented-red)
+
 The `selectors` query property contains a list of Entity properties allowed for a client to use in
 the `criteria` and `sort` query parameters.
 If no value is provided, then no selectors are allowed.
@@ -229,6 +257,21 @@ A list of Entity properties to be included in the Observation result.
 projection: [id, title, timestamp]
 ```
 
+## Parameters
+
+By default, the only query parameters allowed are described above. Arbitrary query parameters
+can be allowed by specifying them in the `parameters` property.
+
+```yaml
+parameters: [foo, bar]
+```
+
+These parameters are embedded in the operation call input, which must be an object.
+
+```http
+GET /dummies/?foo=0&bar=baz
+```
+
 ## Optimistic concurrency control
 
 If an operation returns an object with `_version` property,

package/documentation/tracing.md

@@ -0,0 +1,70 @@
+# Tracing
+
+Each incoming request is processed within a [trace](../../telemetry/readme.md#tracing) —
+a server span is created, and its trace context is propagated to component invocations and
+emitted events, so all log entries and spans across the application caused by the request
+share the same `trace_id`.
+
+## The `ray` response header
+
+Each response contains a `ray` header with the trace ID of the request.
+It can be used to find related log entries, or be attached to a support ticket.
+
+Log entries are stamped with the trace ID regardless of the sampling decision,
+while the trace itself is only recorded if the trace is sampled (see [Sampling](#sampling)).
+
+```
+HTTP/1.1 201 Created
+ray: 0af7651916cd43dd8448eb211c80319c
+```
+
+## Continuing a trace
+
+By default, the trace starts at the gateway. If the request carries a trace context in one of the
+following headers, the trace is continued, that is, the incoming trace ID is used.
+
+### `traceparent`
+
+The [W3C Trace Context](https://www.w3.org/TR/trace-context/#traceparent-header) header, sent by
+clients instrumented with OpenTelemetry-compatible tooling, or by upstream proxies.
+
+```
+traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
+```
+
+In addition to the trace ID, it carries the ID of the client span, which becomes the *parent* of
+the gateway's server span, linking client-side and server-side spans into a single trace tree.
+
+### `ray`
+
+A bare trace ID (32 lowercase hex digits), for clients without tracing instrumentation willing to
+correlate their requests with server-side traces.
+
+```
+ray: 4bf92f3577b34da6a3ce929d0e0e4736
+```
+
+Unlike `traceparent`, it carries no parent span, so the gateway's server span becomes the root of
+the trace.
+
+If both headers are present, `traceparent` takes precedence as it carries more information.
+Headers with invalid values are ignored, and a new trace is started.
+
+## Sampling
+
+The trace *context* (trace ID) is always created and propagated, so the `ray` header and
+`trace_id` in log entries are always present. The sampling decision only determines whether
+the *spans* of the trace are recorded.
+
+The decision is made once, by the process that starts the trace, and is propagated along with
+the trace context (the flags byte of `traceparent`), so a trace is either recorded as a whole
+or not at all. Sampling is configured with the
+[`telemetry.traces` annotation](../../telemetry/readme.md#sampling).
+
+Sending a trace context does not bypass sampling: an incoming `ray` header adopts the trace ID,
+while the recording decision is still made by the gateway. The `sampled` flag of an incoming
+`traceparent` is respected, as the decision belongs to the trace root.
+
+## See also
+
+- [Telemetry extension](../../telemetry/readme.md) — spans, structured logs, `context.span`