@toa.io/extensions.exposition 1.0.0-alpha.22 → 1.0.0-alpha.220

package/documentation/identity.md

@@ -63,8 +63,7 @@ to [OpenID Connect Core 1.0](https://openid.net/specs/openid-connect-core-1_0.ht
 Authorization: Bearer eyJhbGciOiJIUzI1...
 ```
 
-Trusted providers are specified using the `identity.federation` property within the configuration
-annotation.
+Trusted providers are specified using the `identity.federation`  configuration.
 
 ```yaml
 # context.toa.yaml
@@ -73,16 +72,99 @@ configuration:
   identity.federation:
     trust:
       - iss: https://accounts.google.com
-        aud:
-          - <GOOGLE_CLIENT_ID>
+        aud: <GOOGLE_CLIENT_ID>
 
       - iss: https://appleid.apple.com
+        aud: <APPLE_CLIENT_ID>
+        secret: <APPLE_CLIENT_SECRET> # enables Authorization Code Flow
 
       - iss: private.entity
         secrets:
           HS384:
             key0: <THE-SECRET-STRING-FOR-HS384>
             key1: <THE-SECRET-STRING-FOR-HS384> # selected by `kid` in the JWT header
+    principal:
+      iss: https://accounts.google.com
+      sub: 4218230498234
+    implicit: true
+```
+
+`principal` specifies the values of the `iss` and `sub` claims of an Identity that will be granted
+with a `system` role.
+
+`implicit` indicates whether the Identity should be implicitly created when a valid token for a
+non-existent Identity is provided (default `false`).
+
+### Authorization Code Flow
+
+[OAuth 2.0 RFC 6749, section 4.1](https://datatracker.ietf.org/doc/html/rfc6749#section-4.1)
+
+```
+GET /identity/
+authorization: Code <credentials>
+```
+
+`<credentials>` is a base64-encoded JSON containing the following properties:
+
+```yaml
+code: authorization code
+iss: code issuer
+for: redirect URI
+```
+
+Trust configuration for the issuer requires `aud` and either `secret` or `signature`
+values to enable the Authorization Code Flow.
+
+> If `aud` is an array, the first value is used.
+
+```yaml
+# context.toa.yaml
+configuration:
+  identity.federation:
+    trust:
+      - iss: https://accounts.google.com
+        aud: 1045282659797-n705sf85j4b2rodtpdn43od43tvseiet.apps.googleusercontent.com
+        secret: $GOOGLE_CLIENT_SECRET
+      - iss: https://appleid.apple.com
+        aud: io.toa.services.id
+        signature:
+          iss: team-id
+          kid: key-id
+          key: $APPLE_PRIVATE_KEY
+```
+
+### OTP scheme
+
+One-time passwords.
+
+Passwords can be issued by calling `identity.otp.issue` operation, with the following input:
+
+```yaml
+authority: string
+username: string
+```
+
+The reply will contain the `code` property of type `string` formed as a random 6-digit number,
+valid for 60 seconds by default.
+
+```yaml
+code: 123456
+```
+
+OTP can be used with `OTP` authentication formatted as `base64(username:password)`.
+
+```
+GET /identity/ HTTP/1.1
+authentication: OTP dXNlcm5hbWU6MTIzNDU2
+```
+
+OTP expiration time can be configured using the `identity.otp` configuration.
+
+```yaml
+# context.toa.yaml
+configuration:
+  identity.otp:
+    lifetime: 60 # seconds
 ```
 
 ## Identity inception
@@ -110,7 +192,7 @@ exposition:
 The value of the `auth:incept` directive refers to the name of the response property that will be
 returned by the `POST` operation, containing the created entity identifier.
 
-A request with Identity inception must contain (non-existent) credentials that will be associated
+A request with Identity inception may contain (non-existent) credentials that will be associated
 with the created Identity.
 
 ```http
@@ -132,6 +214,34 @@ id: 2428c31ecb6e4a51a24ef52f0c4181b9
 As a result of processing the above request, the provided Basic credentials associated with the
 Identity `2428c31ecb6e4a51a24ef52f0c4181b9` are created.
 
+> `auth:incept` directive may have a `null` value, which means that the Identity will be created
+> without any associated entity.
+
+Inception is supported for `Basic` and `Bearer` authentication schemes.
+
+## Identity assertion
+
+`auth:assert` directive is used to ensure that given credentials are associated with an existing
+Identity or to create a new Identity if it does not exist.
+The directive itself does not allow or deny access to the requested resource.
+
+> Used authentication scheme must support inception.
+
+```yaml
+  /accounts/echo:
+    auth:assert: true
+    auth:anyone: true
+    endpoint: echo
+```
+
+```http
+GET /accounts/echo/
+authorization: Basic new-or-existent-credentials
+```
+
+If new Identity is created and endpoint returns a successful response, the status code `201 Created`
+is returned.
+
 ## FAQ
 
 <dl>

package/documentation/introspection.md

@@ -0,0 +1,82 @@
+# Resource introspection
+
+Any resource can be introspected by sending an `OPTIONS` request to the resource's path.
+The response will contain the resource's input and output schemas for each supported method.
+
+Introspection properties:
+
+- `route` route parameters
+- `query` query parameters
+- `input` input schema
+- `output` output schema
+- `errors` error codes
+
+```http
+OPTIONS /pots/:id/ HTTP/1.1
+accept: application/yaml
+```
+
+```http
+200 OK
+Allow: GET, POST, OPTIONS
+
+GET:
+  route:
+    id:
+      type: string
+      pattern: ^[a-fA-F0-9]{32}$
+  output:
+    type: array
+    items:
+      type: object
+      properties:
+        title:
+          type: string
+          maxLength: 64
+        volume:
+          type: number
+          exclusiveMinimum: 0
+          maximum: 1000
+        temperature:
+          type: number
+          exclusiveMinimum: 0
+          maximum: 300
+      additionalProperties: false
+      required:
+        - id
+        - title
+        - volume
+POST:
+  route:
+    id:
+      type: string
+      pattern: ^[a-fA-F0-9]{32}$
+  input:
+    type: object
+    properties:
+      title:
+        type: string
+        maxLength: 64
+      temperature:
+        type: number
+        exclusiveMinimum: 0
+        maximum: 300
+      volume:
+        type: number
+        exclusiveMinimum: 0
+        maximum: 1000
+    additionalProperties: false
+    required:
+      - title
+      - volume
+  output:
+    type: object
+    properties:
+      id:
+        type: string
+        pattern: ^[a-fA-F0-9]{32}$
+    additionalProperties: false
+  errors:
+    - NO_WAY
+    - WONT_CREATE
+```

package/documentation/io.md

@@ -3,7 +3,7 @@
 The Exposition comes with `io` directives to control access to the operation's input and output
 properties.
 
-## `io:input`
+## Input
 
 The `io:input` optional directive contains a list of properties that are allowed to be specified in
 the request body.
@@ -22,7 +22,7 @@ not in the list, the request will be rejected with a `400` status code.
 > Therefore, `io:input` is only applicable to operations which input is an object or an
 > array of objects.
 
-## `io:output`
+## Output
 
 The `io:output` mandatory directive contains a list of properties that are allowed to be included in
 the response body.
@@ -54,3 +54,43 @@ GET:
 ```
 
 Output restrictions are not applied to stream responses and errors.
+
+## Throttling
+
+The `io:throttle` directive is used to limit the rate of the requests meeting the specified
+criteria.
+See [algorithm description](notes/throttling.md).
+
+```yaml
+exposition:
+  /:
+    io:throttle:
+      key:
+        - route
+        - ip
+        - identity
+        - segment: id
+        - header: x-real-ip
+        - status: [4xx, 5xx]
+      requests: 500
+      interval: 30
+      cooldown: 30
+      status: 429
+```
+
+Once the rate limit is reached, the server will block the requests with the specified `status`
+code (429 by default) until the `cooldown` period expires.
+
+> Currently, only `header` and `status` key components are supported.
+
+### Request key components
+
+Request components used to track quota usage and block requests:
+
+- `header`: name of the header (or a list of header names), which value(s) will be used
+
+### Response key components
+
+Response components used only to track quota usage:
+
+- `status`: response status code (or a list of status codes)

package/documentation/map.md

@@ -0,0 +1,115 @@
+# HTTP context mapping
+
+The `map` directive family is used to map HTTP request parts to operation call input properties.
+
+[Features](../features/map.feature).
+
+## TL;DR
+
+```yaml
+exposition:
+  /:group:
+    languages: [en, fr]   # supported languages
+    GET:
+      map:authority: hostname # request authority (e.g., hostname)
+      map:language: lang      # requested language
+      map:headers: # raw header values
+        token: x-access-token
+      map:buffer: body # unparsed request body
+      map:segments: # route parameters
+        group: group
+      map:claims: # Bearer token claims
+        address: email
+        verified: email_verified
+      endpoint: observe
+```
+
+The operation input type must be an object.
+If the input already contains the specified keys, they will be overwritten.
+
+## Authority
+
+The `map:authority` directive maps the [authority identifier](authorities.md) to an operation call
+input property specified by the directive value.
+
+### Language
+
+The `map:language` mapping sets the [most matching](https://github.com/jshttp/negotiator) language
+code based on the `accept-language` request header and a list of supported languages defined by
+the `map:languages` directive, and also adds `accept-language` to the `Vary` HTTP response header
+value.
+
+If none of the supported languages match, the first supported language is used.
+
+> `map:languages` has a shorthand form: `languages: [en, fr]`.
+
+## Header values
+
+The `map:headers` directive maps the values of HTTP request headers to operation call input
+properties.
+The value of the directive is a map where keys are the names of the input properties and values are
+the names of the HTTP request headers.
+
+The names of these headers are then included in the `Vary` HTTP response header
+and [Access-Control-Allow-Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers)
+of the [CORS](protocol.md#cors).
+
+[Multiple header fields](https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2) are combined
+as a comma-separated list.
+
+## Raw body
+
+The `map:buffer` directive maps the unparsed request body to an operation call input property.
+The directive value is the name of the input property.
+
+```yaml
+/webhook:
+  POST:
+    map:buffer: body
+    map:headers:
+      signature: x-signature
+    endpoint: verify
+```
+
+The body is read from the request stream as raw bytes and delivered as a UTF-8 string.
+
+Body parsing is bypassed, so the route does not depend on a supported `Content-Type` or format decoding. The request stream is consumed; a route with `map:buffer` cannot also receive a parsed body from the same request.
+
+## Route parameters
+
+The `map:segments` directive maps the values of route parameters to operation call input properties.
+The value of the directive is a map where keys are the names of the input properties, and values are
+the names of the route parameters.
+
+Parameter name may be prefixed with `~`
+to indicate that the parameter should not be available to the
+remaining directives or used as criteria in the operation call.
+
+```yaml
+/:id/:tag:
+  POST:
+    map:segments:
+      id: id
+      tag: ~tag
+    endpoint: create
+```
+
+Wildcard routes are supported by using `**` as the parameter name.
+
+```yaml
+/path/to/**:
+  GET:
+    map:segments:
+      path: '**'
+    endpoint: observe
+```
+
+## Bearer token claims
+
+The `map:claims` directive maps the values of
+the [token claims](https://datatracker.ietf.org/doc/html/rfc7519#section-4).
+The value of the directive is a map where keys are the names of the input properties and values are
+the names of the claims.
+
+If the claim is not present in the token or the request is not authenticated using
+the [`Bearer` scheme](identity.md#bearer-scheme), the input properties are not set.

package/documentation/notes/peers.md

@@ -0,0 +1,59 @@
+# Distributed Peer Discovery
+
+## Problem
+
+Modulo partitioning algorithm `taks.id % replicas == index` requires
+to know the number of task processing instances running in the cluster,
+and the own index of the current instance.
+
+## Forces
+
+1. Static configuration is not an option (due to dynamic scaling / failover).
+2. In a distributed system, there is no concept of a *global current time*.
+
+## Solution
+
+An algorithm that emits `(index, replicas)` once per `interval` seconds, using common Redis
+key and atomic increment.
+
+Define the following parameters:
+
+- `name`: a name of the task processing (e.g. `mail-sender`)
+- `interval`: indexing interval that is deliberately greater than the expected clock skew among
+  instances
+
+At the start of each `interval` in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time):
+
+1. Calculate an ordinal number of the current interval: `number = ceil(now() / interval)`
+2. Compose a `key` as `{name}:{number}`
+3. Atomically increment a `key` in Redis ([INCR](https://redis.io/docs/latest/commands/incr/))
+4. If `index` is defined (see 5)
+
+- get the value of previous key `{name}:{number-1}` as `replicas`
+- if the `replicas` is defined, emit `(index, replicas)` **algorithm result**
+
+5. Store the response (3) in `index`.
+
+## Safe index transition
+
+If the `index` or `replicas` changes, the algorithm consumer must stop consuming new tasks
+and execute _safe index transition_, to prevent task duplication or loss.
+
+The transition can be implemented in a manner similar to the described algorithm, using a dedicated
+`{name}:transition` key.
+However, this process is considered outside the scope of this document.
+
+## Extension
+
+If the system clocks are too precisely synchronized (skew is less than a round-trip time to Redis),
+this may result in continuous [index transitions](#safe-index-transition).
+
+To mitigate this, the algorithm can be extended with a random delay:
+
+1. Before starting the algorithm, define a random constant clock skew, significantly smaller than
+   `interval`: `skew = random() * (interval / 2)`
+2. Start the algorithm at each `interval + skew`.
+
+## Caveats
+
+- The first result will become known between `interval` and `interval × 2` seconds.

package/documentation/notes/throttling.md

@@ -0,0 +1,82 @@
+# Decentralized Request Throttling
+
+## Problem
+
+To prevent an unnecessary load, API requests should be throttled when made maliciously or due to an
+error.
+
+## Context
+
+1. The [API Gateway](https://microservices.io/patterns/apigateway.html) of a distributed system runs
+   in multiple instances, each with an in-memory state that does not allow tracking the current
+   quota usage.
+2. Precise quota enforcement (per request, per second) is not critical; the goal is to prevent
+   significant overuse.
+3. Quota configuration is static.
+
+## Forces
+
+1. No per request IO is allowed (centralized solutions do not fit).
+2. In a distributed system, there is no concept of a *global current time*.
+3. Failure to retrieve the quota state should not result in Gateway failure.
+
+## Solution
+
+Implement in-memory quotas in each process, periodically synchronizing them asynchronously using
+Redis.
+
+Consider a basic throttling rule:
+
+1. No more than `MAX_REQUESTS` within `INTERVAL` time for any API route (`KEY`).
+2. If the limit is exceeded, block requests to `KEY` for `COOLDOWN` seconds.
+
+### Concept
+
+1. Divide `INTERVAL` into `N` spans (`N >= 2`).
+2. At the end of each span,
+   atomically increment the value in Redis ([INCRBY](https://redis.io/docs/latest/commands/incrby/))
+   by the number of requests received for each `KEY`, using a key composed of the `KEY` and the
+   current ordinal number of `INTERVAL`
+   in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time).
+3. If the returned value exceeds `MAX_REQUESTS`, block access to `KEY`, remove after `COOLDOWN`.
+4. If the write operation fails and `REQUESTS > MAX_REQUESTS / N` (i.e., the quota is exceeded
+   locally), block access to `KEY`, remove after `COOLDOWN`.
+
+Each API Gateway instance will generate the `KEY` based on its own local time,
+which, in general, will lead to simultaneous writes (from Redis’s local time perspective) to
+different `KEY`s. However, the total contribution of each node to each `KEY` will correspond to the
+actual request rate experienced by that node.
+
+<img src="desync.jpg" alt="Desynchronization" />
+
+Dividing the `INTERVAL` into spans smooths the desynchronization effect.
+
+### Extension
+
+Introduce `nodes`, the approximate number of active API Gateway instances, to improve the algorithm.
+
+1. Initially, `nodes` equals to `1`.
+2. During each `INTERVAL`, sum the total request count for each `KEY`, keep current and previous
+   values.
+3. At the end of each interval:
+
+- Read the value from the `KEY` for the previous interval. At this point, it is assumed that all
+  nodes have switched to the next interval.
+- Update the `nodes` value by dividing response form Redis by the number of `REQUESTS` counted for
+  the previous interval.
+- Set the previous value to the current value.
+- Set the current value to `0`
+
+4. Update point 4 of the [Concept](#concept): `REQUESTS * nodes > MAX_REQUESTS / N`.
+   This will increase the precision of the local quota enforcement.
+5. Add a new step: if `REQUESTS * nodes > MAX_REQUESTS`, block access to `KEY`, remove after
+   `COOLDOWN`.
+
+When nodes are added or removed, the algorithm will adapt in the upcoming intervals.
+
+### Caveats
+
+1. In the worst case scenario (really, really unlikely), the quota is exceeded by `MAX_REQUESTS / N`
+   in a span on each node.
+2. Time desynchronization between nodes should be insignificant for the selected `INTERVAL` (i.e.,
+   `INTERVAL` >> desync). See Extension point 3.