@toa.io/extensions.exposition 1.0.0-alpha.28 → 1.0.0-alpha.283

package/documentation/protocol.md

@@ -4,11 +4,15 @@
 
 The following media types are supported for both requests and responses:
 
-- `application/msgpack` using [msgpackr](https://github.com/kriszyp/msgpackr)
-- `application/yaml` using [js-yaml](https://github.com/nodeca/js-yaml)
 - `application/json`
+- `application/yaml` using [js-yaml](https://github.com/nodeca/js-yaml)
+- `application/msgpack` using [msgpackr](https://github.com/kriszyp/msgpackr)
 - `text/plain`
 
+A request may also be sent as `application/x-www-form-urlencoded`. A form is read and never
+written, so it is not a format a response is negotiated to. A name repeated in a form is read as
+the list it is.
+
 The response format is determined by content negotiation
 using [negotiator](https://github.com/jshttp/negotiator).
 
@@ -26,7 +30,7 @@ foo: bar
 
 ### Multipart types
 
-Multipart responses are endoded using content negotiation,
+Multipart responses are encoded using content negotiation,
 and the `content-type` of the response is set to one of the custom `multipart/` subtypes,
 corresponding to the type of
 the parts:
@@ -38,6 +42,9 @@ the parts:
 | `multipart/json`    | `application/json`    |
 | `multipart/text`    | `text/plain`          |
 
+Multipart responses are started with a text chunk `ACK`, and finished with a text
+chunk `FIN`.
+
 Example:
 
 ```
@@ -49,10 +56,14 @@ accept: application/yaml
 200 OK
 content-type: multipart/yaml; boundary=cut
 
+--cut
+ACK
 --cut
 foo: bar
 --cut
 baz: qux
+--cut
+FIN
 --cut--
 ```
 
@@ -62,6 +73,30 @@ See also:
 - [Content-Type: multipart](https://learn.microsoft.com/en-us/previous-versions/office/developer/exchange-server-2010/aa493937(v=exchg.140))
   at Microsoft
 
+## HTTP version
+
+The gateway serves HTTP/1.1 by default. `protocol: h2c` serves cleartext HTTP/2 instead.
+
+```yaml
+exposition:
+  protocol: h2c
+  service:
+    annotations:
+      projectcontour.io/upstream-protocol.h2c: "8000"
+```
+
+`h2c` requires an ingress controller that proxies cleartext HTTP/2 upstream, and the
+annotation that controller reads to do so. Contour, Traefik and Envoy-based controllers
+proxy it; ingress-nginx does not. A cleartext HTTP/2 server answers nothing else on the
+same port: there is no ALPN to negotiate with, so a controller that speaks HTTP/1.1 upstream
+reaches nothing.
+
+Under `h2c` a request refused before its body is read is answered and then cancelled with
+`RST_STREAM(NO_ERROR)`. The reply arrives; the upload stops where it is. A client streaming
+a request body sees that body cancelled — in Node, `ERR_STREAM_PREMATURE_CLOSE` on the stream
+it passed, which throws if nothing handles it. A real failure shows in the reply, or in the
+request rejecting because no reply came. A buffered body is unaffected.
+
 ## CORS
 
 [CORS](https://www.w3.org/TR/2020/SPSD-cors-20200602/) is supported,
@@ -75,7 +110,7 @@ The following request headers are allowed:
 - `etag`
 - `if-match`
 - `if-none-match`
-- headers used by the [`vary:embed` directive](vary.md#embeddings)
+- headers used by the [`map` directive family](map.md)
 
 The following response headers are exposed:
 

package/documentation/query.md

@@ -30,6 +30,7 @@ exposition:
         query:
           criteria: state==hot
           sort: rank:desc
+          text: true
           limit: 10
   /latest:
     GET:
@@ -148,6 +149,27 @@ rank: 5
 
 > In case of conflict, path variables override input properties.
 
+## Text search
+
+For entities with `text` indexes, search queries can be enabled using the `search` property.
+
+```yaml
+# manifest.toa.yaml
+
+name: dummies
+
+exposition:
+  /:type:
+    GET:
+      endpoint: observe
+      query:
+        search: true
+```
+
+```yaml
+GET /dummies/?search=some+text+query
+```
+
 ## Omit, limit
 
 `omit` and `limit` properties can declare their default values and allowed boundaries:
@@ -177,6 +199,10 @@ Constant values can be declared using the shortcut:
 limit: 10
 ```
 
+```http
+GET /dummies/?omit=100&limit=10
+```
+
 ## Sort
 
 The `sort` query property defines the result order of Observations within an `objects` scope
@@ -213,6 +239,8 @@ GET /dummies/?sort=timestamp:asc
 
 ## Selectors
 
+![Not implemented](https://img.shields.io/badge/Not_implemented-red)
+
 The `selectors` query property contains a list of Entity properties allowed for a client to use in
 the `criteria` and `sort` query parameters.
 If no value is provided, then no selectors are allowed.
@@ -229,6 +257,21 @@ A list of Entity properties to be included in the Observation result.
 projection: [id, title, timestamp]
 ```
 
+## Parameters
+
+By default, the only query parameters allowed are described above. Arbitrary query parameters
+can be allowed by specifying them in the `parameters` property.
+
+```yaml
+parameters: [foo, bar]
+```
+
+These parameters are embedded in the operation call input, which must be an object.
+
+```http
+GET /dummies/?foo=0&bar=baz
+```
+
 ## Optimistic concurrency control
 
 If an operation returns an object with `_version` property,

package/documentation/tracing.md

@@ -0,0 +1,70 @@
+# Tracing
+
+Each incoming request is processed within a [trace](../../telemetry/readme.md#tracing) —
+a server span is created, and its trace context is propagated to component invocations and
+emitted events, so all log entries and spans across the application caused by the request
+share the same `trace_id`.
+
+## The `ray` response header
+
+Each response contains a `ray` header with the trace ID of the request.
+It can be used to find related log entries, or be attached to a support ticket.
+
+Log entries are stamped with the trace ID regardless of the sampling decision,
+while the trace itself is only recorded if the trace is sampled (see [Sampling](#sampling)).
+
+```
+HTTP/1.1 201 Created
+ray: 0af7651916cd43dd8448eb211c80319c
+```
+
+## Continuing a trace
+
+By default, the trace starts at the gateway. If the request carries a trace context in one of the
+following headers, the trace is continued, that is, the incoming trace ID is used.
+
+### `traceparent`
+
+The [W3C Trace Context](https://www.w3.org/TR/trace-context/#traceparent-header) header, sent by
+clients instrumented with OpenTelemetry-compatible tooling, or by upstream proxies.
+
+```
+traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
+```
+
+In addition to the trace ID, it carries the ID of the client span, which becomes the *parent* of
+the gateway's server span, linking client-side and server-side spans into a single trace tree.
+
+### `ray`
+
+A bare trace ID (32 lowercase hex digits), for clients without tracing instrumentation willing to
+correlate their requests with server-side traces.
+
+```
+ray: 4bf92f3577b34da6a3ce929d0e0e4736
+```
+
+Unlike `traceparent`, it carries no parent span, so the gateway's server span becomes the root of
+the trace.
+
+If both headers are present, `traceparent` takes precedence as it carries more information.
+Headers with invalid values are ignored, and a new trace is started.
+
+## Sampling
+
+The trace *context* (trace ID) is always created and propagated, so the `ray` header and
+`trace_id` in log entries are always present. The sampling decision only determines whether
+the *spans* of the trace are recorded.
+
+The decision is made once, by the process that starts the trace, and is propagated along with
+the trace context (the flags byte of `traceparent`), so a trace is either recorded as a whole
+or not at all. Sampling is configured with the
+[`telemetry.traces` annotation](../../telemetry/readme.md#sampling).
+
+Sending a trace context does not bypass sampling: an incoming `ray` header adopts the trace ID,
+while the recording decision is still made by the gateway. The `sampled` flag of an incoming
+`traceparent` is respected, as the decision belongs to the trace root.
+
+## See also
+
+- [Telemetry extension](../../telemetry/readme.md) — spans, structured logs, `context.span`

package/documentation/tree.md

@@ -56,6 +56,19 @@ as it provides a more specific match compared to the generic `/users/:id` route.
 
 The priority of Routes with the same specificity is determined by the order of declaration.
 
+## Route forwarding
+
+A Route can be forwarded to another Route by specifying the destination Route as the value of the
+Route.
+
+```yaml
+/destination/:var: ...
+/static: /destination/hello
+/variables/:bar: /destination/:bar
+```
+
+Forwarding Route variables are mapped to the forwarded Route variables if they have the same name.
+
 ## Methods
 
 Methods are mappings of the HTTP methods to the corresponding operations.

package/features/access.feature

@@ -52,6 +52,56 @@ Feature: Access authorization
       access: granted!
       """
 
+  Scenario: Allow access to anyone
+    Given the annotation:
+      """yaml
+      /:
+        io:output: true
+        auth:anyone: true
+        GET:
+          dev:stub:
+            access: granted!
+      """
+    When the following request is received:
+      """
+      GET / HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic ZGV2ZWxvcGVyOnNlY3JldA==
+      accept: application/yaml
+      """
+    Then the following reply is sent:
+      """
+      200 OK
+      content-type: application/yaml
+
+      access: granted!
+      """
+
+    # request without credentials
+    When the following request is received:
+      """
+      GET / HTTP/1.1
+      host: nex.toa.io
+      accept: application/yaml
+      """
+    Then the following reply is sent:
+      """
+      401 Unauthorized
+      """
+
+    # request with invalid credentials
+    When the following request is received:
+      """
+      GET / HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic bm9uOmV4aXN0ZW50
+      accept: application/yaml
+      """
+    Then the following reply is sent:
+      """
+      401 Unauthorized
+      """
+
   Scenario: Deny access with credentials to a resource with anonymous access
     Given the annotation:
       """yaml
@@ -364,8 +414,8 @@ Feature: Access authorization
     Then the following reply is sent:
       """
       200 OK
-      content-type: application/yaml
       authorization: Token ${{ token }}
+      content-type: application/yaml
 
       access: granted!
       """
@@ -421,12 +471,12 @@ Feature: Access authorization
       """
       403 Forbidden
 
-      Basic authentication scheme is required to access this resource.
+      Basic authentication scheme is required to access this resource
       """
 
   Scenario: Adding a role without required permissions
 
-  Trunk directives should not be applied to the Identity management resources.
+  Trunk directives should not be applied to the Identity management resources
 
     Given the annotation:
       """yaml
@@ -468,9 +518,79 @@ Feature: Access authorization
     Then the following reply is sent:
       """
       200 OK
+      authorization: Token ${{ token }}
 
       identity:
         id: efe3a65ebbee47ed95a73edd911ea328
         roles:
           - developer
       """
+    When the following request is received:
+      """
+      GET /echo/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Token ${{ token }}
+      """
+    Then the following reply is sent:
+      """
+      200 OK
+      """
+    And the reply does not contain:
+      """
+      authorization: Token
+      """
+
+  Scenario: Delegation does not violate input restrictions
+    Given the `identity.roles` database contains:
+      | _id                              | identity                         | role      |
+      | 775a648d054e4ce1a65f8f17e5b51803 | efe3a65ebbee47ed95a73edd911ea328 | developer |
+    # `auth:` is declared before `io:`, and the identity it embeds is the server's
+    # own addition — what `io:input` restricts is the body the client sent
+    And the `echo` is running with the following manifest:
+      """yaml
+      exposition:
+        /:
+          POST:
+            auth:rule:
+              role: developer
+              delegate: identity
+            io:input: [foo]
+            io:output: true
+            endpoint: echo
+      """
+    When the following request is received:
+      """
+      POST /echo/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic ZGV2ZWxvcGVyOnNlY3JldA==
+      accept: application/yaml
+      content-type: application/yaml
+
+      foo: bar
+      """
+    Then the following reply is sent:
+      """
+      201 Created
+
+      foo: bar
+      identity:
+        id: efe3a65ebbee47ed95a73edd911ea328
+      """
+    # and a property the client is not allowed to send is still refused
+    When the following request is received:
+      """
+      POST /echo/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic ZGV2ZWxvcGVyOnNlY3JldA==
+      accept: application/yaml
+      content-type: application/yaml
+
+      foo: bar
+      baz: qux
+      """
+    Then the following reply is sent:
+      """
+      400 Bad Request
+
+      'Unexpected input: baz'
+      """

package/features/auth.assert.feature

@@ -0,0 +1,55 @@
+Feature: Identity assertion
+
+  Scenario: Assert Identity
+    Given the `identity.basic` database contains:
+      | _id                              | authority | username  | password                                                     | _version |
+      | efe3a65ebbee47ed95a73edd911ea328 | nex       | developer | $2b$10$ZRSKkgZoGnrcTNA5w5eCcu3pxDzdTduhteVYXcp56AaNcilNkwJ.O | 1        |
+    And the `echo` is running with the following manifest:
+      """yaml
+      exposition:
+        /:
+          GET:
+            auth:assert: true
+            anyone: true
+            io:output: true
+            endpoint: ping
+      """
+
+    # existent identity
+    When the following request is received:
+      """
+      GET /echo/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic ZGV2ZWxvcGVyOnNlY3JldA==
+      """
+    Then the following reply is sent:
+      """
+      200 OK
+      authorization: Token ${{ developer.token }}
+      """
+
+    # new identity
+    When the following request is received:
+      """
+      GET /echo/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic dXNlcjpwYSQkdzByZCE=
+      """
+    Then the following reply is sent:
+      """
+      201 Created
+      authorization: Token ${{ user.token }}
+      """
+
+    # wrong password for the credentials just created
+    When the following request is received:
+      """
+      GET /echo/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic dXNlcjpwYXNz
+      accept: application/yaml
+      """
+    Then the following reply is sent:
+      """
+      401 Unauthorized
+      """

package/features/auth.bearer.feature

@@ -0,0 +1,120 @@
+@security
+Feature: More than one provider claims Bearer
+
+  An OpenID `id_token` and a token this gateway issued are both presented as `Bearer`,
+  and only their own provider can verify either. The gateway asks each in turn.
+
+  Background:
+    Given local IDP is running
+    And the `identity.basic` database contains:
+      | _id                              | authority | username  | password                                                     |
+      | efe3a65ebbee47ed95a73edd911ea328 | nex       | developer | $2b$10$ZRSKkgZoGnrcTNA5w5eCcu3pxDzdTduhteVYXcp56AaNcilNkwJ.O |
+    And the annotation:
+      """yaml
+      /:
+        io:output: true
+        /hello/:id:
+          auth:id: id
+          GET:
+            dev:stub: Hello
+      """
+
+  Scenario: A token this gateway issued is presented as Bearer
+    When the following request is received:
+      """
+      GET /hello/efe3a65ebbee47ed95a73edd911ea328/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Basic ZGV2ZWxvcGVyOnNlY3JldA==
+      accept: text/plain
+      """
+    Then the following reply is sent:
+      """
+      200 OK
+      authorization: Token ${{ token }}
+      content-type: text/plain
+
+      Hello
+      """
+    When the following request is received:
+      """
+      GET /hello/efe3a65ebbee47ed95a73edd911ea328/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Bearer ${{ token }}
+      accept: text/plain
+      """
+    Then the following reply is sent:
+      """
+      200 OK
+      content-type: text/plain
+
+      Hello
+      """
+    # the token was answered by its own provider, so nothing rotates it
+    And the reply does not contain:
+      """
+      authorization: Token
+      """
+
+  Scenario: A Bearer this gateway did not issue is answered by federation
+    Given the IDP token for Stranger is issued
+    When the following request is received:
+      """
+      GET /hello/efe3a65ebbee47ed95a73edd911ea328/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Bearer ${{ Stranger.id_token }}
+      accept: text/plain
+      """
+    # `identity.tokens` declines it, and federation rejects an issuer it does not trust
+    Then the following reply is sent:
+      """
+      401 Unauthorized
+      """
+
+  Scenario: A malformed Bearer is rejected, not an error
+    When the following request is received:
+      """
+      GET /hello/efe3a65ebbee47ed95a73edd911ea328/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Bearer not.a.token
+      accept: text/plain
+      """
+    Then the following reply is sent:
+      """
+      401 Unauthorized
+      """
+
+  Scenario: Both kinds of Bearer against one authority
+    Given the `identity.federation` configuration:
+      """yaml
+      trust:
+        - iss: http://localhost:44444
+          aud: test
+      """
+    And the IDP token for User is issued
+    When the following request is received:
+      """
+      GET /identity/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Bearer ${{ User.id_token }}
+      accept: application/yaml
+      """
+    Then the following reply is sent:
+      """
+      200 OK
+      authorization: Token ${{ federated }}
+
+      id: ${{ User.id }}
+      """
+    When the following request is received:
+      """
+      GET /identity/ HTTP/1.1
+      host: nex.toa.io
+      authorization: Bearer ${{ federated }}
+      accept: application/yaml
+      """
+    Then the following reply is sent:
+      """
+      200 OK
+
+      id: ${{ User.id }}
+      """