@toa.io/extensions.exposition 1.0.0-alpha.17 → 1.0.0-alpha.171

package/documentation/identity.md

@@ -1,36 +1,30 @@
 # Identity
 
 Identity is the fundamental entity within an authentication system that represents the **unique
-identifier** of an
-individual, organization, application or device.
+identifier** of an individual, organization, application or device.
 
-In order to prove its Identity, the request originator must provide a valid _credentials_ that are
-associated with that
-Identity.
+To prove its Identity, the request originator must provide a valid _credentials_ that are associated
+with that Identity.
 
 Identity is intrinsically linked to credentials, as an Identity is established only when the first
-set of credentials
-for that Identity is created.
+set of credentials for that Identity is created.
 In other words, the creation of credentials marks the inception of an Identity.
 Once the last credentials are removed from the Identity, it ceases to exist.
 Without credentials, there is no basis for defining or asserting an Identity.
 
 ## Authentication
 
-The Authenticaiton system resolves provided credentials to an Identity using one of the supported
-authentication
-schemes.
+The Authentication system resolves provided credentials to an Identity using one of the supported
+authentication schemes.
 
 The Authentication is request-agnostic, meaning it does not depend on the specific URL being
-requested or the content of
-the request body.
+requested or the content of the request body.
 The only information it handles is the value of the `Authorization` header.
 
-> Except for its own [management resources](#persistent-credentials).
+> Except for its own [management resources](components.md).
 
 If the provided credentials are not valid or not associated with an Identity, then Authentication
-interrupts request
-processing and responds with an authentication error.
+interrupts request processing and responds with an authentication error.
 
 ### Basic scheme
 
@@ -52,8 +46,8 @@ Authrization: Token v4.local.eyJzdWIiOiJqb2hu...
 
 The `Token` is the **primary** authentication scheme.
 If request originators use an alternative authentication scheme, they will receive a response
-containing `Token`
-credentials and will be required to switch to the `Token` scheme for any subsequent requests.
+containing `Token`credentials and will be required to switch to the `Token` scheme for any
+subsequent requests.
 Continued use of other authentication schemes will result in temporary blocking of requests.
 
 See [`identity.tokens` component](components.md#stateless-tokens).
@@ -69,7 +63,7 @@ to [OpenID Connect Core 1.0](https://openid.net/specs/openid-connect-core-1_0.ht
 Authorization: Bearer eyJhbGciOiJIUzI1...
 ```
 
-Trusted providers are specified using the `identity.federation` property within the configuration annotation.
+Trusted providers are specified using the `identity.federation`  configuration.
 
 ```yaml
 # context.toa.yaml
@@ -77,17 +71,100 @@ Trusted providers are specified using the `identity.federation` property within
 configuration:
   identity.federation:
     trust:
-      - issuer: https://accounts.google.com
-        audience:
-          - <GOOGLE_CLIENT_ID>
+      - iss: https://accounts.google.com
+        aud: <GOOGLE_CLIENT_ID>
 
-      - issuer: https://appleid.apple.com
+      - iss: https://appleid.apple.com
+        aud: <APPLE_CLIENT_ID>
+        secret: <APPLE_CLIENT_SECRET> # enables Authorization Code Flow
 
-      - issuer: private.entity
+      - iss: private.entity
         secrets:
           HS384:
             key0: <THE-SECRET-STRING-FOR-HS384>
             key1: <THE-SECRET-STRING-FOR-HS384> # selected by `kid` in the JWT header
+    principal:
+      iss: https://accounts.google.com
+      sub: 4218230498234
+    implicit: true
+```
+
+`principal` specifies the values of the `iss` and `sub` claims of an Identity that will be granted
+with a `system` role.
+
+`implicit` indicates whether the Identity should be implicitly created when a valid token for a
+non-existent Identity is provided (default `false`).
+
+### Authorization Code Flow
+
+[OAuth 2.0 RFC 6749, section 4.1](https://datatracker.ietf.org/doc/html/rfc6749#section-4.1)
+
+```
+GET /identity/
+authorization: Code <credentials>
+```
+
+`<credentials>` is a base64-encoded JSON containing the following properties:
+
+```yaml
+code: authorization code
+iss: code issuer
+for: redirect URI
+```
+
+Trust configuration for the issuer requires `aud` and either `secret` or `signature`
+values to enable the Authorization Code Flow.
+
+> If `aud` is an array, the first value is used.
+
+```yaml
+# context.toa.yaml
+configuration:
+  identity.federation:
+    trust:
+      - iss: https://accounts.google.com
+        aud: 1045282659797-n705sf85j4b2rodtpdn43od43tvseiet.apps.googleusercontent.com
+        secret: $GOOGLE_CLIENT_SECRET
+      - iss: https://appleid.apple.com
+        aud: io.toa.services.id
+        signature:
+          iss: team-id
+          kid: key-id
+          key: $APPLE_PRIVATE_KEY
+```
+
+### OTP scheme
+
+One-time passwords.
+
+Passwords can be issued by calling `identity.otp.issue` operation, with the following input:
+
+```yaml
+authority: string
+username: string
+```
+
+The reply will contain the `code` property of type `string` formed as a random 6-digit number,
+valid for 60 seconds by default.
+
+```yaml
+code: 123456
+```
+
+OTP can be used with `OTP` authentication formatted as `base64(username:password)`.
+
+```
+GET /identity/ HTTP/1.1
+authentication: OTP dXNlcm5hbWU6MTIzNDU2
+```
+
+OTP expiration time can be configured using the `identity.otp` configuration.
+
+```yaml
+# context.toa.yaml
+configuration:
+  identity.otp:
+    lifetime: 60 # seconds
 ```
 
 ## Identity inception
@@ -115,7 +192,7 @@ exposition:
 The value of the `auth:incept` directive refers to the name of the response property that will be
 returned by the `POST` operation, containing the created entity identifier.
 
-A request with Identity inception must contain (non-existent) credentials that will be associated
+A request with Identity inception may contain (non-existent) credentials that will be associated
 with the created Identity.
 
 ```http
@@ -137,6 +214,34 @@ id: 2428c31ecb6e4a51a24ef52f0c4181b9
 As a result of processing the above request, the provided Basic credentials associated with the
 Identity `2428c31ecb6e4a51a24ef52f0c4181b9` are created.
 
+> `auth:incept` directive may have a `null` value, which means that the Identity will be created
+> without any associated entity.
+
+Inception is supported for `Basic` and `Bearer` authentication schemes.
+
+## Identity assertion
+
+`auth:assert` directive is used to ensure that given credentials are associated with an existing
+Identity or to create a new Identity if it does not exist.
+The directive itself does not allow or deny access to the requested resource.
+
+> Used authentication scheme must support inception.
+
+```yaml
+  /accounts/echo:
+    auth:assert: true
+    auth:anyone: true
+    endpoint: echo
+```
+
+```http
+GET /accounts/echo/
+authorization: Basic new-or-existent-credentials
+```
+
+If new Identity is created and endpoint returns a successful response, the status code `201 Created`
+is returned.
+
 ## FAQ
 
 <dl>

package/documentation/introspection.md

@@ -0,0 +1,82 @@
+# Resource introspection
+
+Any resource can be introspected by sending an `OPTIONS` request to the resource's path.
+The response will contain the resource's input and output schemas for each supported method.
+
+Introspection properties:
+
+- `route` route parameters
+- `query` query parameters
+- `input` input schema
+- `output` output schema
+- `errors` error codes
+
+```http
+OPTIONS /pots/:id/ HTTP/1.1
+accept: application/yaml
+```
+
+```http
+200 OK
+Allow: GET, POST, OPTIONS
+
+GET:
+  route:
+    id:
+      type: string
+      pattern: ^[a-fA-F0-9]{32}$
+  output:
+    type: array
+    items:
+      type: object
+      properties:
+        title:
+          type: string
+          maxLength: 64
+        volume:
+          type: number
+          exclusiveMinimum: 0
+          maximum: 1000
+        temperature:
+          type: number
+          exclusiveMinimum: 0
+          maximum: 300
+      additionalProperties: false
+      required:
+        - id
+        - title
+        - volume
+POST:
+  route:
+    id:
+      type: string
+      pattern: ^[a-fA-F0-9]{32}$
+  input:
+    type: object
+    properties:
+      title:
+        type: string
+        maxLength: 64
+      temperature:
+        type: number
+        exclusiveMinimum: 0
+        maximum: 300
+      volume:
+        type: number
+        exclusiveMinimum: 0
+        maximum: 1000
+    additionalProperties: false
+    required:
+      - title
+      - volume
+  output:
+    type: object
+    properties:
+      id:
+        type: string
+        pattern: ^[a-fA-F0-9]{32}$
+    additionalProperties: false
+  errors:
+    - NO_WAY
+    - WONT_CREATE
+```

package/documentation/io.md

@@ -3,7 +3,7 @@
 The Exposition comes with `io` directives to control access to the operation's input and output
 properties.
 
-## `io:input`
+## Input
 
 The `io:input` optional directive contains a list of properties that are allowed to be specified in
 the request body.
@@ -22,7 +22,7 @@ not in the list, the request will be rejected with a `400` status code.
 > Therefore, `io:input` is only applicable to operations which input is an object or an
 > array of objects.
 
-## `io:output`
+## Output
 
 The `io:output` mandatory directive contains a list of properties that are allowed to be included in
 the response body.
@@ -54,3 +54,43 @@ GET:
 ```
 
 Output restrictions are not applied to stream responses and errors.
+
+## Throttling
+
+The `io:throttle` directive is used to limit the rate of the requests meeting the specified
+criteria.
+See [algorithm description](notes/throttling.md).
+
+```yaml
+exposition:
+  /:
+    io:throttle:
+      key:
+        - route
+        - ip
+        - identity
+        - segment: id
+        - header: x-real-ip
+        - status: [4xx, 5xx]
+      requests: 500
+      interval: 30
+      cooldown: 30
+      status: 429
+```
+
+Once the rate limit is reached, the server will block the requests with the specified `status`
+code (429 by default) until the `cooldown` period expires.
+
+> Currently, only `header` and `status` key components are supported.
+
+### Request key components
+
+Request components used to track quota usage and block requests:
+
+- `header`: name of the header (or a list of header names), which value(s) will be used
+
+### Response key components
+
+Response components used only to track quota usage:
+
+- `status`: response status code (or a list of status codes)

package/documentation/map.md

@@ -0,0 +1,96 @@
+# HTTP context mapping
+
+The `map` directive family is used to map HTTP request parts to operation call input properties.
+
+[Features](../features/map.feature).
+
+## TL;DR
+
+```yaml
+exposition:
+  /:group:
+    languages: [en, fr]   # supported languages
+    GET:
+      map:authority: hostname # request authority (e.g., hostname)
+      map:language: lang      # requested language
+      map:headers: # raw header values
+        token: x-access-token
+      map:segments: # route parameters
+        group: group
+      map:claims: # Bearer token claims
+        address: email
+        verified: email_verified
+      endpoint: observe
+```
+
+The operation input type must be an object.
+If the input already contains the specified keys, they will be overwritten.
+
+## Authority
+
+The `map:authority` directive maps the [authority identifier](authorities.md) to an operation call
+input property specified by the directive value.
+
+### Language
+
+The `map:language` mapping sets the [most matching](https://github.com/jshttp/negotiator) language
+code based on the `accept-language` request header and a list of supported languages defined by
+the `map:languages` directive, and also adds `accept-language` to the `Vary` HTTP response header
+value.
+
+If none of the supported languages match, the first supported language is used.
+
+> `map:languages` has a shorthand form: `languages: [en, fr]`.
+
+## Header values
+
+The `map:headers` directive maps the values of HTTP request headers to operation call input
+properties.
+The value of the directive is a map where keys are the names of the input properties and values are
+the names of the HTTP request headers.
+
+The names of these headers are then included in the `Vary` HTTP response header
+and [Access-Control-Allow-Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers)
+of the [CORS](protocol.md#cors).
+
+[Multiple header fields](https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2) are combined
+as a comma-separated list.
+
+## Route parameters
+
+The `map:segments` directive maps the values of route parameters to operation call input properties.
+The value of the directive is a map where keys are the names of the input properties, and values are
+the names of the route parameters.
+
+Parameter name may be prefixed with `~`
+to indicate that the parameter should not be available to the
+remaining directives or used as criteria in the operation call.
+
+```yaml
+/:id/:tag:
+  POST:
+    map:segments:
+      id: id
+      tag: ~tag
+    endpoint: create
+```
+
+Wildcard routes are supported by using `**` as the parameter name.
+
+```yaml
+/path/to/**:
+  GET:
+    map:segments:
+      path: '**'
+    endpoint: observe
+```
+
+## Bearer token claims
+
+The `map:claims` directive maps the values of
+the [token claims](https://datatracker.ietf.org/doc/html/rfc7519#section-4).
+The value of the directive is a map where keys are the names of the input properties and values are
+the names of the claims.
+
+If the claim is not present in the token or the request is not authenticated using
+the [`Bearer` scheme](identity.md#bearer-scheme), the input properties are not set.

package/documentation/notes/peers.md

@@ -0,0 +1,59 @@
+# Distributed Peer Discovery
+
+## Problem
+
+Modulo partitioning algorithm `taks.id % replicas == index` requires
+to know the number of task processing instances running in the cluster,
+and the own index of the current instance.
+
+## Forces
+
+1. Static configuration is not an option (due to dynamic scaling / failover).
+2. In a distributed system, there is no concept of a *global current time*.
+
+## Solution
+
+An algorithm that emits `(index, replicas)` once per `interval` seconds, using common Redis
+key and atomic increment.
+
+Define the following parameters:
+
+- `name`: a name of the task processing (e.g. `mail-sender`)
+- `interval`: indexing interval that is deliberately greater than the expected clock skew among
+  instances
+
+At the start of each `interval` in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time):
+
+1. Calculate an ordinal number of the current interval: `number = ceil(now() / interval)`
+2. Compose a `key` as `{name}:{number}`
+3. Atomically increment a `key` in Redis ([INCR](https://redis.io/docs/latest/commands/incr/))
+4. If `index` is defined (see 5)
+
+- get the value of previous key `{name}:{number-1}` as `replicas`
+- if the `replicas` is defined, emit `(index, replicas)` **algorithm result**
+
+5. Store the response (3) in `index`.
+
+## Safe index transition
+
+If the `index` or `replicas` changes, the algorithm consumer must stop consuming new tasks
+and execute _safe index transition_, to prevent task duplication or loss.
+
+The transition can be implemented in a manner similar to the described algorithm, using a dedicated
+`{name}:transition` key.
+However, this process is considered outside the scope of this document.
+
+## Extension
+
+If the system clocks are too precisely synchronized (skew is less than a round-trip time to Redis),
+this may result in continuous [index transitions](#safe-index-transition).
+
+To mitigate this, the algorithm can be extended with a random delay:
+
+1. Before starting the algorithm, define a random constant clock skew, significantly smaller than
+   `interval`: `skew = random() * (interval / 2)`
+2. Start the algorithm at each `interval + skew`.
+
+## Caveats
+
+- The first result will become known between `interval` and `interval × 2` seconds.

package/documentation/notes/throttling.md

@@ -0,0 +1,82 @@
+# Decentralized Request Throttling
+
+## Problem
+
+To prevent an unnecessary load, API requests should be throttled when made maliciously or due to an
+error.
+
+## Context
+
+1. The [API Gateway](https://microservices.io/patterns/apigateway.html) of a distributed system runs
+   in multiple instances, each with an in-memory state that does not allow tracking the current
+   quota usage.
+2. Precise quota enforcement (per request, per second) is not critical; the goal is to prevent
+   significant overuse.
+3. Quota configuration is static.
+
+## Forces
+
+1. No per request IO is allowed (centralized solutions do not fit).
+2. In a distributed system, there is no concept of a *global current time*.
+3. Failure to retrieve the quota state should not result in Gateway failure.
+
+## Solution
+
+Implement in-memory quotas in each process, periodically synchronizing them asynchronously using
+Redis.
+
+Consider a basic throttling rule:
+
+1. No more than `MAX_REQUESTS` within `INTERVAL` time for any API route (`KEY`).
+2. If the limit is exceeded, block requests to `KEY` for `COOLDOWN` seconds.
+
+### Concept
+
+1. Divide `INTERVAL` into `N` spans (`N >= 2`).
+2. At the end of each span,
+   atomically increment the value in Redis ([INCRBY](https://redis.io/docs/latest/commands/incrby/))
+   by the number of requests received for each `KEY`, using a key composed of the `KEY` and the
+   current ordinal number of `INTERVAL`
+   in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time).
+3. If the returned value exceeds `MAX_REQUESTS`, block access to `KEY`, remove after `COOLDOWN`.
+4. If the write operation fails and `REQUESTS > MAX_REQUESTS / N` (i.e., the quota is exceeded
+   locally), block access to `KEY`, remove after `COOLDOWN`.
+
+Each API Gateway instance will generate the `KEY` based on its own local time,
+which, in general, will lead to simultaneous writes (from Redis’s local time perspective) to
+different `KEY`s. However, the total contribution of each node to each `KEY` will correspond to the
+actual request rate experienced by that node.
+
+<img src="desync.jpg" alt="Desynchronization" />
+
+Dividing the `INTERVAL` into spans smooths the desynchronization effect.
+
+### Extension
+
+Introduce `nodes`, the approximate number of active API Gateway instances, to improve the algorithm.
+
+1. Initially, `nodes` equals to `1`.
+2. During each `INTERVAL`, sum the total request count for each `KEY`, keep current and previous
+   values.
+3. At the end of each interval:
+
+- Read the value from the `KEY` for the previous interval. At this point, it is assumed that all
+  nodes have switched to the next interval.
+- Update the `nodes` value by dividing response form Redis by the number of `REQUESTS` counted for
+  the previous interval.
+- Set the previous value to the current value.
+- Set the current value to `0`
+
+4. Update point 4 of the [Concept](#concept): `REQUESTS * nodes > MAX_REQUESTS / N`.
+   This will increase the precision of the local quota enforcement.
+5. Add a new step: if `REQUESTS * nodes > MAX_REQUESTS`, block access to `KEY`, remove after
+   `COOLDOWN`.
+
+When nodes are added or removed, the algorithm will adapt in the upcoming intervals.
+
+### Caveats
+
+1. In the worst case scenario (really, really unlikely), the quota is exceeded by `MAX_REQUESTS / N`
+   in a span on each node.
+2. Time desynchronization between nodes should be insignificant for the selected `INTERVAL` (i.e.,
+   `INTERVAL` >> desync). See Extension point 3.