@toa.io/extensions.exposition 1.0.0-alpha.26 → 1.0.0-alpha.262

package/documentation/octets.md

@@ -14,26 +14,32 @@ directives under the current RTD Node.
   octets:context: images
 ```
 
-## `octets:store`
+## `octets:put`
 
 Stores the content of the request body into a storage, under the request path with
 specified `content-type`.
 
 If request's `content-type` is not acceptable, or if the request body does not pass
-the [validation](/extensions/storages/readme.md#async-putpath-string-stream-readable-type-typecontrol-maybeentry),
+the [validation](/extensions/storages/readme.md#async-putpath-string-stream-readable-options-options-maybeentry),
 the request is rejected with a `415 Unsupported Media Type` response.
 
-The value of the directive is `null` or an object with the following properties:
+The value of the directive must be `null` (defaults) or an object with the following optional
+properties:
 
+- `location`: a string that represents the path to store the
+  content.
+  If not specified, the path is the same as the request path.
+- `limit`: [maximum size](#stream-size-limit) of the incoming stream.
 - `accept`: a media type or an array of media types that are acceptable.
   If the `accept` property is not specified, any media type is acceptable (which is the default).
 - `workflow`: [workflow](#workflows) to be executed once the content is successfully stored.
+- `trust`: a list of [trusted origins](#downloading-external-content).
 
 ```yaml
 /images:
   octets:context: images
   POST:
-    octets:store:
+    octets:put:
       accept:
         - image/jpeg
         - image/png
@@ -43,31 +49,97 @@ The value of the directive is `null` or an object with the following properties:
         analyze: images.analyze
 ```
 
-Non-standard `content-meta` header can be used
+### Headers
+
+`content-id` header can be used to set the ID of the Entry.
+The value must match the following regular expression `^[a-zA-Z0-9-_]{1,16}$`.
+
+Non-standard `content-attributes` header can be used
 to set initial [metadata](/extensions/storages/readme.md#entry) value for the Entry.
 
-The value of the `content-meta` header is a comma-separated list of key-value string pairs.
+The value of the `content-attributes` header is a comma-separated list of key-value string pairs.
 If no value is provided for a key, the string `true` is used.
 
 ```http
 POST /images/ HTTP/1.1
 content-type: image/jpeg
-content-meta: foo, bar=baz
-content-meta: baz=1
+content-id: example-id
+content-attributes: foo, bar=baz
+content-attributes: baz=1
 ```
 
 ```yaml
-meta:
+attributes:
   foo: 'true'
   bar: 'baz'
   baz: '1'
 ```
 
-If the Entry already exists, the `content-meta` header is ignored.
+If the Entry already exists, the `content-attributes` header is ignored.
+
+### Location
+
+The `location` property can be used to store the content under a different path.
+
+```yaml
+/images:
+  octets:context: images
+  POST:
+    octets:put:
+      location: /archive
+```
+
+Physical storage path is constructed by resolving the `location`
+property [relative](https://datatracker.ietf.org/doc/html/rfc3986#section-5) to the request path.
+
+### Stream size limit
+
+The `limit` property can be used to set the maximum size of the incoming stream in bytes.
+
+The property value can be specified as a number
+(representing bytes) or a string that combines a number with a unit (e.g., `1MB`).
+Both [binary and decimal prefixes](https://en.wikipedia.org/wiki/Binary_prefix) are supported.
+If the prefix or unit is specified _incorrectly_ (e.g., `1mb`),
+it will default to a binary prefix interpretation.
+
+- `1b`, `1B`: 1 byte
+- `1KB`: 1000 bytes
+- `1KiB`: 1024 bytes
+- `1kb`: 1024 bytes
+
+The default value is `64MiB`.
+
+### Downloading external content
+
+The `octets:put` directive can be used to download external content:
+
+```http
+POST /images/ HTTP/1.1
+content-location: https://example.com/image.jpg
+content-length: 0
+```
+
+Requests with `content-location` header must have an empty body (`content-length: 0` header).
+
+Target origin must be allowed by the `trust` property,
+which can contain a list of trusted origins or regular expressions to match the full URL.
+
+URL of the downloaded content is stored in the `origin` property of
+the [Entry](/extensions/storages/readme.md#entry).
+
+```yaml
+/images:
+  octets:context: images
+  POST:
+    octets:put:
+      trust:
+        - https://example.com
+        - ^https://example\.com/[a-z]+\.jpe?g$
+```
 
 ### Response
 
-The response of the `octets:store` directive is the created Entry.
+The response of the `octets:put` directive is the created Entry.
 
 ```
 201 Created
@@ -78,12 +150,13 @@ type: image/jpeg
 created: 1698004822358
 ```
 
-If the `octets:store` directive contains a `workflow`, the response
+If the `octets:put` directive contains a `workflow`, the response
 is [multipart](protocol.md#multipart-types).
 The first part represents the created Entry, which is sent immediately after the BLOB is stored,
 while subsequent parts are results from the workflow endpoints, sent as soon as they are available.
 
-In case a workflow endpoint returns an `Error`, the error part is sent, and the response is closed.
+In case a workflow endpoint returns an `Error`, the error part is sent,
+and the response is closed.
 Error's properties are added to the error part, among with the `step` identifier.
 
 ```
@@ -91,20 +164,33 @@ Error's properties are added to the error part, among with the `step` identifier
 content-type: multipart/yaml; boundary=cut
 
 --cut
+
 id: eecd837c
 type: image/jpeg
 created: 1698004822358
+
 --cut
-optimize: null
+
+step: optimize
+status: completed
+
 --cut
+
+step: resize
 error:
-  step: resize
   code: TOO_SMALL
   message: Image is too small
+status: completed
+
+--cut
+
+step: analyze
+status: exception
+
 --cut--
 ```
 
-## `octets:fetch`
+## `octets:get`
 
 Fetches the content of a stored BLOB corresponding to the request path, and returns it as the
 response body with the corresponding `content-type`, `content-length`
@@ -116,22 +202,18 @@ The value of the directive is an object with the following properties:
 
 - `meta`: `boolean` indicating whether an Entry is accessible.
   Defaults to `false`.
-- `blob`: `boolean` indicating whether the original BLOB is accessible,
-  [BLOB variant](/extensions/storages/readme.md#async-fetchpath-string-maybereadable) must be
-  specified in the path otherwise.
-  Defaults to `true`.
 
 ```yaml
 /images:
   octets:context: images
   /*:
     GET:
-      octets:fetch:
+      octets:get:
         blob: false # prevent access to the original BLOB
         meta: true  # allow access to an Entry
 ```
 
-The `octets:fetch: ~` declaration is equivalent to defaults.
+The `octets:get: ~` declaration is equivalent to defaults.
 
 To access an Entry, the `accept` request header must contain the `octets.entry` subtype
 in
@@ -142,32 +224,6 @@ GET /images/eecd837c HTTP/1.1
 accept: application/vnd.toa.octets.entry+yaml
 ```
 
-## `octets:list`
-
-Lists the entries stored under the request path.
-
-The value of the directive is an object with the following properties:
-
-- `meta`: `boolean` indicating whether the list of Entries is accessible.
-  Defaults to `false`, which means that only entry identifiers are returned.
-
-```yaml
-/images:
-  octets:context: images
-  GET:
-    octets:list:
-      meta: true
-```
-
-The `octets:list: ~` declaration is equivalent to defaults.
-
-To access a list of Entries, the `accept` request header must contain the `octets.entries` subtype:
-
-```http
-GET /images/ HTTP/1.1
-accept: application/vnd.toa.octets.entries+yaml
-```
-
 ## `octets:delete`
 
 Delete the entry corresponding to the request path.
@@ -193,22 +249,6 @@ the entry is deleted.
 
 The error returned by the workflow prevents the deletion of the entry.
 
-## `octets:permute`
-
-Performs
-a [permutation](/extensions/storages/readme.md#async-permutepath-string-ids-string-maybevoid) on the
-entries
-under the request path.
-
-```yaml
-/images:
-  octets:context: images
-  PUT:
-    octets:permute: ~
-```
-
-The request body must be a list of entry identifiers.
-
 ## `octets:workflow`
 
 Execute a [workflow](#workflows) on the entry under the request path.
@@ -227,21 +267,23 @@ A workflow is a list of endpoints to be called.
 The following input will be passed to each endpoint:
 
 ```yaml
+authority: string
 storage: string
 path: string
 entry: Entry
 parameters: Record<string, string> # route parameters
 ```
 
-See [Entry](/extensions/storages/readme.md#entry) and an
-example [workflow step processor](../features/steps/components/octets.tester).
+- [Storages](/extensions/storages/readme.md)
+- [Authorities](authorities.md)
+- Example [workflow step processor](../features/steps/components/octets.tester)
 
 A _workflow unit_ is an object with keys referencing the workflow step identifier, and an endpoint
 as value.
 Steps within a workflow unit are executed in parallel.
 
 ```yaml
-octets:store:
+octets:put:
   workflow:
     resize: images.resize
     analyze: images.analyze
@@ -251,11 +293,22 @@ A workflow can be a single unit, or an array of units.
 If it's an array, the workflow units are executed in sequence.
 
 ```yaml
-octets:store:
+octets:put:
   workflow:
     - optimize: images.optimize   # executed first
     - resize: images.resize       # executed second
       analyze: images.analyze     # executed in parallel with `resize`
 ```
 
-If one of the workflow units returns an error, the execution of the workflow is interrupted.
+If one of the workflow units returns or throws an error,
+the execution of the workflow is interrupted.
+
+### Workflow tasks
+
+A workflow unit which value starts with `task:` prefix will be executed as a Task.
+
+```yaml
+octets:put:
+  workflow:
+    optimize: task:images.optimize
+```

package/documentation/passkeys.md

@@ -0,0 +1,5 @@
+# Web Authentication
+
+- [Standard](https://www.w3.org/TR/webauthn-2/)
+- [Features](../features/passkeys.feature)
+- [Credential management](credentials.md#passkeys)

package/documentation/protocol.md

@@ -4,9 +4,9 @@
 
 The following media types are supported for both requests and responses:
 
-- `application/msgpack` using [msgpackr](https://github.com/kriszyp/msgpackr)
-- `application/yaml` using [js-yaml](https://github.com/nodeca/js-yaml)
 - `application/json`
+- `application/yaml` using [js-yaml](https://github.com/nodeca/js-yaml)
+- `application/msgpack` using [msgpackr](https://github.com/kriszyp/msgpackr)
 - `text/plain`
 
 The response format is determined by content negotiation
@@ -26,7 +26,7 @@ foo: bar
 
 ### Multipart types
 
-Multipart responses are endoded using content negotiation,
+Multipart responses are encoded using content negotiation,
 and the `content-type` of the response is set to one of the custom `multipart/` subtypes,
 corresponding to the type of
 the parts:
@@ -38,6 +38,9 @@ the parts:
 | `multipart/json`    | `application/json`    |
 | `multipart/text`    | `text/plain`          |
 
+Multipart responses are started with a text chunk `ACK`, and finished with a text
+chunk `FIN`.
+
 Example:
 
 ```
@@ -49,10 +52,14 @@ accept: application/yaml
 200 OK
 content-type: multipart/yaml; boundary=cut
 
+--cut
+ACK
 --cut
 foo: bar
 --cut
 baz: qux
+--cut
+FIN
 --cut--
 ```
 
@@ -75,7 +82,7 @@ The following request headers are allowed:
 - `etag`
 - `if-match`
 - `if-none-match`
-- headers used by the [`vary:embed` directive](vary.md#embeddings)
+- headers used by the [`map` directive family](map.md)
 
 The following response headers are exposed:
 

package/documentation/query.md

@@ -30,6 +30,7 @@ exposition:
         query:
           criteria: state==hot
           sort: rank:desc
+          text: true
           limit: 10
   /latest:
     GET:
@@ -148,6 +149,27 @@ rank: 5
 
 > In case of conflict, path variables override input properties.
 
+## Text search
+
+For entities with `text` indexes, search queries can be enabled using the `search` property.
+
+```yaml
+# manifest.toa.yaml
+
+name: dummies
+
+exposition:
+  /:type:
+    GET:
+      endpoint: observe
+      query:
+        search: true
+```
+
+```yaml
+GET /dummies/?search=some+text+query
+```
+
 ## Omit, limit
 
 `omit` and `limit` properties can declare their default values and allowed boundaries:
@@ -177,6 +199,10 @@ Constant values can be declared using the shortcut:
 limit: 10
 ```
 
+```http
+GET /dummies/?omit=100&limit=10
+```
+
 ## Sort
 
 The `sort` query property defines the result order of Observations within an `objects` scope
@@ -213,6 +239,8 @@ GET /dummies/?sort=timestamp:asc
 
 ## Selectors
 
+![Not implemented](https://img.shields.io/badge/Not_implemented-red)
+
 The `selectors` query property contains a list of Entity properties allowed for a client to use in
 the `criteria` and `sort` query parameters.
 If no value is provided, then no selectors are allowed.
@@ -229,6 +257,21 @@ A list of Entity properties to be included in the Observation result.
 projection: [id, title, timestamp]
 ```
 
+## Parameters
+
+By default, the only query parameters allowed are described above. Arbitrary query parameters
+can be allowed by specifying them in the `parameters` property.
+
+```yaml
+parameters: [foo, bar]
+```
+
+These parameters are embedded in the operation call input, which must be an object.
+
+```http
+GET /dummies/?foo=0&bar=baz
+```
+
 ## Optimistic concurrency control
 
 If an operation returns an object with `_version` property,

package/documentation/tracing.md

@@ -0,0 +1,70 @@
+# Tracing
+
+Each incoming request is processed within a [trace](../../telemetry/readme.md#tracing) —
+a server span is created, and its trace context is propagated to component invocations and
+emitted events, so all log entries and spans across the application caused by the request
+share the same `trace_id`.
+
+## The `ray` response header
+
+Each response contains a `ray` header with the trace ID of the request.
+It can be used to find related log entries, or be attached to a support ticket.
+
+Log entries are stamped with the trace ID regardless of the sampling decision,
+while the trace itself is only recorded if the trace is sampled (see [Sampling](#sampling)).
+
+```
+HTTP/1.1 201 Created
+ray: 0af7651916cd43dd8448eb211c80319c
+```
+
+## Continuing a trace
+
+By default, the trace starts at the gateway. If the request carries a trace context in one of the
+following headers, the trace is continued, that is, the incoming trace ID is used.
+
+### `traceparent`
+
+The [W3C Trace Context](https://www.w3.org/TR/trace-context/#traceparent-header) header, sent by
+clients instrumented with OpenTelemetry-compatible tooling, or by upstream proxies.
+
+```
+traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
+```
+
+In addition to the trace ID, it carries the ID of the client span, which becomes the *parent* of
+the gateway's server span, linking client-side and server-side spans into a single trace tree.
+
+### `ray`
+
+A bare trace ID (32 lowercase hex digits), for clients without tracing instrumentation willing to
+correlate their requests with server-side traces.
+
+```
+ray: 4bf92f3577b34da6a3ce929d0e0e4736
+```
+
+Unlike `traceparent`, it carries no parent span, so the gateway's server span becomes the root of
+the trace.
+
+If both headers are present, `traceparent` takes precedence as it carries more information.
+Headers with invalid values are ignored, and a new trace is started.
+
+## Sampling
+
+The trace *context* (trace ID) is always created and propagated, so the `ray` header and
+`trace_id` in log entries are always present. The sampling decision only determines whether
+the *spans* of the trace are recorded.
+
+The decision is made once, by the process that starts the trace, and is propagated along with
+the trace context (the flags byte of `traceparent`), so a trace is either recorded as a whole
+or not at all. Sampling is configured with the
+[`telemetry.traces` annotation](../../telemetry/readme.md#sampling).
+
+Sending a trace context does not bypass sampling: an incoming `ray` header adopts the trace ID,
+while the recording decision is still made by the gateway. The `sampled` flag of an incoming
+`traceparent` is respected, as the decision belongs to the trace root.
+
+## See also
+
+- [Telemetry extension](../../telemetry/readme.md) — spans, structured logs, `context.span`

package/documentation/tree.md

@@ -56,6 +56,19 @@ as it provides a more specific match compared to the generic `/users/:id` route.
 
 The priority of Routes with the same specificity is determined by the order of declaration.
 
+## Route forwarding
+
+A Route can be forwarded to another Route by specifying the destination Route as the value of the
+Route.
+
+```yaml
+/destination/:var: ...
+/static: /destination/hello
+/variables/:bar: /destination/:bar
+```
+
+Forwarding Route variables are mapped to the forwarded Route variables if they have the same name.
+
 ## Methods
 
 Methods are mappings of the HTTP methods to the corresponding operations.

package/features/access.feature

@@ -52,6 +52,56 @@ Feature: Access authorization
       access: granted!
       """
 
+  Scenario: Allow access to anyone
+    Given the annotation:
+      """yaml
+      /:
+        io:output: true
+        auth:anyone: true
+        GET:
+          dev:stub:
+            access: granted!
+      """
+    When the following request is received:
+      """
+      GET / HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic ZGV2ZWxvcGVyOnNlY3JldA==
+      accept: application/yaml
+      """
+    Then the following reply is sent:
+      """
+      200 OK
+      content-type: application/yaml
+
+      access: granted!
+      """
+
+    # request without credentials
+    When the following request is received:
+      """
+      GET / HTTP/1.1
+      host: nex.toa.io
+      accept: application/yaml
+      """
+    Then the following reply is sent:
+      """
+      401 Unauthorized
+      """
+
+    # request with invalid credentials
+    When the following request is received:
+      """
+      GET / HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic bm9uOmV4aXN0ZW50
+      accept: application/yaml
+      """
+    Then the following reply is sent:
+      """
+      401 Unauthorized
+      """
+
   Scenario: Deny access with credentials to a resource with anonymous access
     Given the annotation:
       """yaml
@@ -364,8 +414,8 @@ Feature: Access authorization
     Then the following reply is sent:
       """
       200 OK
-      content-type: application/yaml
       authorization: Token ${{ token }}
+      content-type: application/yaml
 
       access: granted!
       """
@@ -421,12 +471,12 @@ Feature: Access authorization
       """
       403 Forbidden
 
-      Basic authentication scheme is required to access this resource.
+      Basic authentication scheme is required to access this resource
       """
 
   Scenario: Adding a role without required permissions
 
-  Trunk directives should not be applied to the Identity management resources.
+  Trunk directives should not be applied to the Identity management resources
 
     Given the annotation:
       """yaml
@@ -468,9 +518,24 @@ Feature: Access authorization
     Then the following reply is sent:
       """
       200 OK
+      authorization: Token ${{ token }}
 
       identity:
         id: efe3a65ebbee47ed95a73edd911ea328
         roles:
           - developer
       """
+    When the following request is received:
+      """
+      GET /echo/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Token ${{ token }}
+      """
+    Then the following reply is sent:
+      """
+      200 OK
+      """
+    And the reply does not contain:
+      """
+      authorization: Token
+      """