@toa.io/extensions.exposition 1.0.0-alpha.28 → 1.0.0-alpha.283

package/documentation/identity.md

@@ -6,6 +6,9 @@ identifier** of an individual, organization, application or device.
 To prove its Identity, the request originator must provide a valid _credentials_ that are associated
 with that Identity.
 
+Persistent authentication methods and their management resources are described in
+[Persistent credentials](credentials.md).
+
 Identity is intrinsically linked to credentials, as an Identity is established only when the first
 set of credentials for that Identity is created.
 In other words, the creation of credentials marks the inception of an Identity.
@@ -38,23 +41,51 @@ See [`identity.basic` component](components.md#basic-credentials).
 
 ### Token scheme
 
-Tokens issued by the Authentication system. These tokens are [PASETO](https://paseto.io).
+Tokens issued by the Authentication system. New tokens are compact
+[JSON Web Encryption (JWE)](https://www.rfc-editor.org/rfc/rfc7516) values using direct symmetric
+encryption (`alg: dir`) and AES-256-GCM (`enc: A256GCM`). Legacy PASETO V3.local tokens remain
+accepted during migration and are always replaced on successful authentication.
 
 ```http
-Authrization: Token v4.local.eyJzdWIiOiJqb2hu...
+Authorization: Token eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIiwidHlwIjoiSldUIiwia2lkIjo...
 ```
 
 The `Token` is the **primary** authentication scheme.
 If request originators use an alternative authentication scheme, they will receive a response
-containing `Token`credentials and will be required to switch to the `Token` scheme for any
+containing `Token` credentials and will be required to switch to the `Token` scheme for any
 subsequent requests.
-Continued use of other authentication schemes will result in temporary blocking of requests.
 
-See [`identity.tokens` component](components.md#stateless-tokens).
+See [`identity.tokens` component](components.md#local-tokens).
+
+### Failed authentications
+
+Credentials that are rejected can be metered per client address, whatever the scheme. An address
+may fail `attempts` at once and earns them back at `attempts` per `interval` seconds; a request
+carrying credentials from an address with nothing left is answered `429 Too Many Requests` with
+`Retry-After`. Nothing is metered unless the exposition annotation carries `bouncer`;
+`attempts` and `interval` default to 20 and 60. The address is the one the request context
+resolved, see [Client address](ip.md): `bouncer` requires `ip`, in the same environment, and a
+request without an address is not metered.
+
+```yaml
+# context.toa.yaml
+
+exposition:
+  ip: cf-connecting-ip
+  bouncer:
+    attempts: 20
+    interval: 60
+```
+
+The count is shared by the gateway replicas through
+[atomicity](../../../connectors/atomicity/readme.md) and decided in each process; nothing is read on
+the request path.
 
 ### Bearer scheme
 
-OpenID tokens issued by trusted providers.
+OpenID tokens issued by trusted providers, and access tokens issued by
+[this authorization server](oauth.md) to OAuth clients. Both are presented the same way, and each
+provider says which credentials are its own; a credential neither claims is refused.
 For more information, refer
 to [OpenID Connect Core 1.0](https://openid.net/specs/openid-connect-core-1_0.html),
 [RFC6750](https://datatracker.ietf.org/doc/html/rfc6750).
@@ -63,8 +94,8 @@ to [OpenID Connect Core 1.0](https://openid.net/specs/openid-connect-core-1_0.ht
 Authorization: Bearer eyJhbGciOiJIUzI1...
 ```
 
-Trusted providers are specified using the `identity.federation` property within the configuration
-annotation.
+Trusted providers are specified using the `identity.federation` configuration. Provider metadata
+and signing keys are discovered using OpenID Connect discovery and JWKS.
 
 ```yaml
 # context.toa.yaml
@@ -73,16 +104,97 @@ configuration:
   identity.federation:
     trust:
       - iss: https://accounts.google.com
-        aud:
-          - <GOOGLE_CLIENT_ID>
+        aud: <GOOGLE_CLIENT_ID>
+
+      - iss: https://appleid.apple.com
+        aud: <APPLE_CLIENT_ID>
+        secret: <APPLE_CLIENT_SECRET> # enables Authorization Code Flow
+    principal:
+      authority: example
+      iss: https://accounts.google.com
+      sub: 4218230498234
+    assert: true
+```
+
+`principal` specifies the authority and the values of the `iss` and `sub` claims of an Identity
+that will be granted with a `system` role. The same subject in another authority is ordinary
+credentials.
+
+`assert` indicates whether the Identity should be implicitly created when valid credentials for a
+non-existent Identity are provided (default `true`).
+
+### Authorization Code Flow
 
+[OAuth 2.0 RFC 6749, section 4.1](https://datatracker.ietf.org/doc/html/rfc6749#section-4.1)
+
+```
+GET /identity/
+authorization: Code <credentials>
+```
+
+`<credentials>` is a base64-encoded JSON containing the following properties:
+
+```yaml
+code: authorization code
+iss: code issuer
+for: redirect URI
+```
+
+Trust configuration for the issuer requires either `secret` or `signature`
+to enable the Authorization Code Flow.
+
+> If `aud` is an array, the first value is used.
+
+```yaml
+# context.toa.yaml
+configuration:
+  identity.federation:
+    trust:
+      - iss: https://accounts.google.com
+        aud: 1045282659797-n705sf85j4b2rodtpdn43od43tvseiet.apps.googleusercontent.com
+        secret: $GOOGLE_CLIENT_SECRET
       - iss: https://appleid.apple.com
+        aud: io.toa.services.id
+        signature:
+          iss: team-id
+          kid: key-id
+          key: $APPLE_PRIVATE_KEY # PKCS8 PEM, as in the .p8 file
+```
+
+### OTP scheme
+
+One-time passwords.
+
+Passwords can be issued by calling `identity.otp.issue` operation, with the following input:
+
+```yaml
+authority: string
+username: string
+```
+
+The reply will contain the `code` property of type `string` formed as a random 6-digit number,
+valid for 60 seconds by default.
+
+```yaml
+code: 123456
+```
+
+OTP can be used with `OTP` authentication formatted as `base64(username:password)`.
+
+```
+GET /identity/ HTTP/1.1
+authentication: OTP dXNlcm5hbWU6MTIzNDU2
+```
+
+OTP expiration time and the number of failed attempts a username is allowed within it can be
+configured using the `identity.otp` configuration.
 
-      - iss: private.entity
-        secrets:
-          HS384:
-            key0: <THE-SECRET-STRING-FOR-HS384>
-            key1: <THE-SECRET-STRING-FOR-HS384> # selected by `kid` in the JWT header
+```yaml
+# context.toa.yaml
+configuration:
+  identity.otp:
+    lifetime: 60 # seconds
+    attempts: 5
 ```
 
 ## Identity inception
@@ -98,7 +210,10 @@ name: users
 
 entity:
   schema:
-    name: string
+    type: object
+    properties:
+      name:
+        type: string
 
 exposition:
   /:
@@ -110,7 +225,7 @@ exposition:
 The value of the `auth:incept` directive refers to the name of the response property that will be
 returned by the `POST` operation, containing the created entity identifier.
 
-A request with Identity inception must contain (non-existent) credentials that will be associated
+A request with Identity inception may contain (non-existent) credentials that will be associated
 with the created Identity.
 
 ```http
@@ -132,6 +247,39 @@ id: 2428c31ecb6e4a51a24ef52f0c4181b9
 As a result of processing the above request, the provided Basic credentials associated with the
 Identity `2428c31ecb6e4a51a24ef52f0c4181b9` are created.
 
+> `auth:incept` directive may have a `null` value, which means that the Identity will be created
+> without any associated entity.
+
+Inception is supported for `Basic` and `Bearer` authentication schemes.
+
+Inception goes ahead when the request carries no credentials, or credentials the scheme verified
+and no Identity owns, which its `authenticate` operation reports as `NOT_FOUND`. Credentials
+rejected for any other reason, a wrong password for one, are answered with `401 Unauthorized`
+before the endpoint is called.
+
+## Identity assertion
+
+`auth:assert` directive is used to ensure that given credentials are associated with an existing
+Identity or to create a new Identity if it does not exist.
+The directive itself does not allow or deny access to the requested resource.
+
+> Used authentication scheme must support inception.
+
+```yaml
+  /accounts/echo:
+    auth:assert: true
+    auth:anyone: true
+    endpoint: echo
+```
+
+```http
+GET /accounts/echo/
+authorization: Basic new-or-existent-credentials
+```
+
+If new Identity is created and endpoint returns a successful response, the status code `201 Created`
+is returned.
+
 ## FAQ
 
 <dl>

package/documentation/introspection.md

@@ -0,0 +1,82 @@
+# Resource introspection
+
+Any resource can be introspected by sending an `OPTIONS` request to the resource's path.
+The response will contain the resource's input and output schemas for each supported method.
+
+Introspection properties:
+
+- `route` route parameters
+- `query` query parameters
+- `input` input schema
+- `output` output schema
+- `errors` error codes
+
+```http
+OPTIONS /pots/:id/ HTTP/1.1
+accept: application/yaml
+```
+
+```http
+200 OK
+Allow: GET, POST, OPTIONS
+
+GET:
+  route:
+    id:
+      type: string
+      pattern: ^[a-fA-F0-9]{32}$
+  output:
+    type: array
+    items:
+      type: object
+      properties:
+        title:
+          type: string
+          maxLength: 64
+        volume:
+          type: number
+          exclusiveMinimum: 0
+          maximum: 1000
+        temperature:
+          type: number
+          exclusiveMinimum: 0
+          maximum: 300
+      additionalProperties: false
+      required:
+        - id
+        - title
+        - volume
+POST:
+  route:
+    id:
+      type: string
+      pattern: ^[a-fA-F0-9]{32}$
+  input:
+    type: object
+    properties:
+      title:
+        type: string
+        maxLength: 64
+      temperature:
+        type: number
+        exclusiveMinimum: 0
+        maximum: 300
+      volume:
+        type: number
+        exclusiveMinimum: 0
+        maximum: 1000
+    additionalProperties: false
+    required:
+      - title
+      - volume
+  output:
+    type: object
+    properties:
+      id:
+        type: string
+        pattern: ^[a-fA-F0-9]{32}$
+    additionalProperties: false
+  errors:
+    - NO_WAY
+    - WONT_CREATE
+```

package/documentation/io.md

@@ -1,9 +1,9 @@
 # I/O restrictions
 
 The Exposition comes with `io` directives to control access to the operation's input and output
-properties.
+properties, and to let an operation state the status of its reply.
 
-## `io:input`
+## Input
 
 The `io:input` optional directive contains a list of properties that are allowed to be specified in
 the request body.
@@ -22,7 +22,7 @@ not in the list, the request will be rejected with a `400` status code.
 > Therefore, `io:input` is only applicable to operations which input is an object or an
 > array of objects.
 
-## `io:output`
+## Output
 
 The `io:output` mandatory directive contains a list of properties that are allowed to be included in
 the response body.
@@ -53,4 +53,109 @@ GET:
   io:output: false
 ```
 
-Output restrictions are not applied to stream responses and errors.
+Output restrictions are not applied to stream responses, nor to a reply the gateway built out of an
+exception — an operation that returns an error is answered with a code and a message of the
+gateway's own, and a list of permitted properties has nothing to say about those. A reply the
+operation returned is restricted whatever status it carries, see [Status](#status).
+
+## Status
+
+The `io:status` optional directive names a property of the reply that carries the status of the
+response. The property is removed from the body.
+
+```yaml
+POST:
+  endpoint: register
+  io:status: status
+  io:output: [client_id, client_name, error, error_description]
+```
+
+```javascript
+return { status: 400, error: 'invalid_redirect_uri' }
+```
+
+```http
+400 Bad Request
+
+error: invalid_redirect_uri
+```
+
+A reply that does not carry the property is answered with the status it would have been. A property
+that is not a number is an error.
+
+This is for an outcome the operation knows and the transport does not — a protocol that states its
+error as a body rather than as a status of its own, or a request that created where it might have
+updated. Such an operation returns no error: both replies are its output, and `io:output` permits
+the properties of both. An operation that does return an error is unaffected — that reply is the
+gateway's, and is not restricted.
+
+## Throttling
+
+The `io:throttle` directive limits the rate of requests meeting the specified criteria.
+
+```yaml
+exposition:
+  /:
+    io:throttle:
+      key:
+        - route
+        - ip
+      condition:
+        status: 404
+      requests: 500
+      interval: 30
+```
+
+Requests are metered per `key`. A key may spend `requests` at once, and earns them back at a rate of
+`requests` per `interval` seconds. A request carrying a key with nothing left to spend is answered
+`429 Too Many Requests`, with a `Retry-After` saying how long until one would be admitted again.
+`interval` is in seconds.
+
+So `requests` is the burst, and `requests / interval` the rate it is repaid at. There is no window to
+save a budget up in and spend twice across the edge of, and no lockout to sit out: a key that has
+overspent is admitted again as soon as it has earned a single request back, and a client that keeps
+asking meanwhile is not penalised for it.
+
+`requests` is a budget for the **whole group** of gateway instances, not for one of them. Instances
+reconcile through Redis on a timer, so they converge on one number rather than each enforcing the
+limit separately. Between two of those, an instance goes on what it alone has spent — so the group
+can overshoot by what the other instances admit within that window, and by no more. The window is a
+tenth of `interval`, between a quarter of a second and two seconds, whatever `interval` is. Precise
+per-request enforcement is not what this is for.
+
+An instance that cannot reach Redis keeps throttling on what it has seen itself and keeps serving,
+and reports what it could not the next time it gets through.
+
+### Key components
+
+What a request is metered against. Give one, or a list — a list keys on the combination, so
+`[route, ip]` meters each address separately on each route.
+
+- `ip` — the client address, as the request context resolved it. See [Client address](ip.md) for
+  where it is read from and why.
+- `path` — the path the request came in on, `/users/1`.
+- `route` — the route as declared, `/users/:id`. Every path matching the route shares one budget,
+  which `path` cannot do: keyed on `path`, walking ids is a way around the limit.
+- `identity` — the authenticated identity. Requests carrying none share a single budget between
+  them, so pair it with `ip` if that matters.
+- `segment: <name>` — the value bound to a named route segment, `1` for `:id` in `/users/:id`.
+
+### Conditions
+
+What is metered, as opposed to what it is metered against. A request that fails a condition is served
+and costs nothing. Conditions are evaluated against the response, so they cannot decide whether to
+refuse a request — only whether it is charged for, once it has been answered.
+
+- `status: <code>` — charge only for responses with this status.
+
+```yaml
+io:throttle:
+  key: ip
+  condition:
+    status: 404
+  requests: 20
+  interval: 600
+```
+
+An address may probe for twenty missing paths at once, and thereafter for one every thirty seconds.
+Requests that find something are served and never counted against it.

package/documentation/ip.md

@@ -0,0 +1,24 @@
+# Client address
+
+The address a request comes from is resolved once, into the request context, and read from there
+by whatever keys on it: the meter of [failed authentications](identity.md#failed-authentications)
+and the `ip` key of [`io:throttle`](io.md#key-components).
+
+```yaml
+# context.toa.yaml
+
+exposition:
+  ip: cf-connecting-ip
+```
+
+`ip` names the header the address is read from. In production the gateway sits behind a load
+balancer, an ingress or a CDN: the connection it accepts comes from that machine, so every client
+shares one address and nothing tells them apart. The edge is the one that saw the client connect,
+and it writes the client's address into a header of its own, `CF-Connecting-IP`, `X-Real-IP` or
+`X-Forwarded-For`, depending on the infrastructure. A client may send any of these headers too, so
+the gateway trusts none of them on its own: the deployment names the one its edge overwrites, and
+where the header holds a list, the last value is the one the edge appended.
+
+The connection's address is never used. Without `ip`, or for a request that does not carry the
+header, the request has no address: the bouncer and a throttle keyed on `ip` let it through and
+say so in the log, once per hundred seconds rather than once per request.

package/documentation/map.md

@@ -0,0 +1,115 @@
+# HTTP context mapping
+
+The `map` directive family is used to map HTTP request parts to operation call input properties.
+
+[Features](../features/map.feature).
+
+## TL;DR
+
+```yaml
+exposition:
+  /:group:
+    languages: [en, fr]   # supported languages
+    GET:
+      map:authority: hostname # request authority (e.g., hostname)
+      map:language: lang      # requested language
+      map:headers: # raw header values
+        token: x-access-token
+      map:buffer: body # unparsed request body
+      map:segments: # route parameters
+        group: group
+      map:claims: # Bearer token claims
+        address: email
+        verified: email_verified
+      endpoint: observe
+```
+
+The operation input type must be an object.
+If the input already contains the specified keys, they will be overwritten.
+
+## Authority
+
+The `map:authority` directive maps the [authority identifier](authorities.md) to an operation call
+input property specified by the directive value.
+
+### Language
+
+The `map:language` mapping sets the [most matching](https://github.com/jshttp/negotiator) language
+code based on the `accept-language` request header and a list of supported languages defined by
+the `map:languages` directive, and also adds `accept-language` to the `Vary` HTTP response header
+value.
+
+If none of the supported languages match, the first supported language is used.
+
+> `map:languages` has a shorthand form: `languages: [en, fr]`.
+
+## Header values
+
+The `map:headers` directive maps the values of HTTP request headers to operation call input
+properties.
+The value of the directive is a map where keys are the names of the input properties and values are
+the names of the HTTP request headers.
+
+The names of these headers are then included in the `Vary` HTTP response header
+and [Access-Control-Allow-Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers)
+of the [CORS](protocol.md#cors).
+
+[Multiple header fields](https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2) are combined
+as a comma-separated list.
+
+## Raw body
+
+The `map:buffer` directive maps the unparsed request body to an operation call input property.
+The directive value is the name of the input property.
+
+```yaml
+/webhook:
+  POST:
+    map:buffer: body
+    map:headers:
+      signature: x-signature
+    endpoint: verify
+```
+
+The body is read from the request stream as raw bytes and delivered as a UTF-8 string.
+
+Body parsing is bypassed, so the route does not depend on a supported `Content-Type` or format decoding. The request stream is consumed; a route with `map:buffer` cannot also receive a parsed body from the same request.
+
+## Route parameters
+
+The `map:segments` directive maps the values of route parameters to operation call input properties.
+The value of the directive is a map where keys are the names of the input properties, and values are
+the names of the route parameters.
+
+Parameter name may be prefixed with `~`
+to indicate that the parameter should not be available to the
+remaining directives or used as criteria in the operation call.
+
+```yaml
+/:id/:tag:
+  POST:
+    map:segments:
+      id: id
+      tag: ~tag
+    endpoint: create
+```
+
+Wildcard routes are supported by using `**` as the parameter name.
+
+```yaml
+/path/to/**:
+  GET:
+    map:segments:
+      path: '**'
+    endpoint: observe
+```
+
+## Bearer token claims
+
+The `map:claims` directive maps the values of
+the [token claims](https://datatracker.ietf.org/doc/html/rfc7519#section-4).
+The value of the directive is a map where keys are the names of the input properties and values are
+the names of the claims.
+
+If the claim is not present in the token or the request is not authenticated using
+the [`Bearer` scheme](identity.md#bearer-scheme), the input properties are not set.

package/documentation/notes/peers.md

@@ -0,0 +1,59 @@
+# Distributed Peer Discovery
+
+## Problem
+
+Modulo partitioning algorithm `taks.id % replicas == index` requires
+to know the number of task processing instances running in the cluster,
+and the own index of the current instance.
+
+## Forces
+
+1. Static configuration is not an option (due to dynamic scaling / failover).
+2. In a distributed system, there is no concept of a *global current time*.
+
+## Solution
+
+An algorithm that emits `(index, replicas)` once per `interval` seconds, using common Redis
+key and atomic increment.
+
+Define the following parameters:
+
+- `name`: a name of the task processing (e.g. `mail-sender`)
+- `interval`: indexing interval that is deliberately greater than the expected clock skew among
+  instances
+
+At the start of each `interval` in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time):
+
+1. Calculate an ordinal number of the current interval: `number = ceil(now() / interval)`
+2. Compose a `key` as `{name}:{number}`
+3. Atomically increment a `key` in Redis ([INCR](https://redis.io/docs/latest/commands/incr/))
+4. If `index` is defined (see 5)
+
+- get the value of previous key `{name}:{number-1}` as `replicas`
+- if the `replicas` is defined, emit `(index, replicas)` **algorithm result**
+
+5. Store the response (3) in `index`.
+
+## Safe index transition
+
+If the `index` or `replicas` changes, the algorithm consumer must stop consuming new tasks
+and execute _safe index transition_, to prevent task duplication or loss.
+
+The transition can be implemented in a manner similar to the described algorithm, using a dedicated
+`{name}:transition` key.
+However, this process is considered outside the scope of this document.
+
+## Extension
+
+If the system clocks are too precisely synchronized (skew is less than a round-trip time to Redis),
+this may result in continuous [index transitions](#safe-index-transition).
+
+To mitigate this, the algorithm can be extended with a random delay:
+
+1. Before starting the algorithm, define a random constant clock skew, significantly smaller than
+   `interval`: `skew = random() * (interval / 2)`
+2. Start the algorithm at each `interval + skew`.
+
+## Caveats
+
+- The first result will become known between `interval` and `interval × 2` seconds.