@toa.io/extensions.exposition 1.0.0-alpha.2 → 1.0.0-alpha.200

package/documentation/introspection.md

@@ -0,0 +1,82 @@
+# Resource introspection
+
+Any resource can be introspected by sending an `OPTIONS` request to the resource's path.
+The response will contain the resource's input and output schemas for each supported method.
+
+Introspection properties:
+
+- `route` route parameters
+- `query` query parameters
+- `input` input schema
+- `output` output schema
+- `errors` error codes
+
+```http
+OPTIONS /pots/:id/ HTTP/1.1
+accept: application/yaml
+```
+
+```http
+200 OK
+Allow: GET, POST, OPTIONS
+
+GET:
+  route:
+    id:
+      type: string
+      pattern: ^[a-fA-F0-9]{32}$
+  output:
+    type: array
+    items:
+      type: object
+      properties:
+        title:
+          type: string
+          maxLength: 64
+        volume:
+          type: number
+          exclusiveMinimum: 0
+          maximum: 1000
+        temperature:
+          type: number
+          exclusiveMinimum: 0
+          maximum: 300
+      additionalProperties: false
+      required:
+        - id
+        - title
+        - volume
+POST:
+  route:
+    id:
+      type: string
+      pattern: ^[a-fA-F0-9]{32}$
+  input:
+    type: object
+    properties:
+      title:
+        type: string
+        maxLength: 64
+      temperature:
+        type: number
+        exclusiveMinimum: 0
+        maximum: 300
+      volume:
+        type: number
+        exclusiveMinimum: 0
+        maximum: 1000
+    additionalProperties: false
+    required:
+      - title
+      - volume
+  output:
+    type: object
+    properties:
+      id:
+        type: string
+        pattern: ^[a-fA-F0-9]{32}$
+    additionalProperties: false
+  errors:
+    - NO_WAY
+    - WONT_CREATE
+```

package/documentation/io.md

@@ -0,0 +1,96 @@
+# I/O restrictions
+
+The Exposition comes with `io` directives to control access to the operation's input and output
+properties.
+
+## Input
+
+The `io:input` optional directive contains a list of properties that are allowed to be specified in
+the request body.
+
+```yaml
+POST:
+  endpoint: create
+  io:input: [name, location]
+```
+
+The list must be a valid subset of the operation's input properties.
+
+If `io:input` is specified and the request body is not an object, or contains properties that are
+not in the list, the request will be rejected with a `400` status code.
+
+> Therefore, `io:input` is only applicable to operations which input is an object or an
+> array of objects.
+
+## Output
+
+The `io:output` mandatory directive contains a list of properties that are allowed to be included in
+the response body.
+
+```yaml
+GET:
+  endpoint: observe
+  io:output: [name, location]
+```
+
+When an operation does not return an object (e.g., a primitive or a stream), or an object is dynamic
+and its properties are not known in advance, `io:output` may have a value of `true` to disable
+output restrictions.
+
+```yaml
+GET:
+  endpoint: proxy
+  io:output: true
+```
+
+If a method declaration lacks `io:output` directive, it will trigger a warning, and its
+response will consistently be empty.
+If this behavior is intended, a `false` value can be employed to suppress warnings.
+
+```yaml
+GET:
+  endpoint: conceal
+  io:output: false
+```
+
+Output restrictions are not applied to stream responses and errors.
+
+## Throttling
+
+The `io:throttle` directive is used to limit the rate of the requests meeting the specified
+criteria.
+See [algorithm description](notes/throttling.md).
+
+```yaml
+exposition:
+  /:
+    io:throttle:
+      key:
+        - route
+        - ip
+        - identity
+        - segment: id
+        - header: x-real-ip
+        - status: [4xx, 5xx]
+      requests: 500
+      interval: 30
+      cooldown: 30
+      status: 429
+```
+
+Once the rate limit is reached, the server will block the requests with the specified `status`
+code (429 by default) until the `cooldown` period expires.
+
+> Currently, only `header` and `status` key components are supported.
+
+### Request key components
+
+Request components used to track quota usage and block requests:
+
+- `header`: name of the header (or a list of header names), which value(s) will be used
+
+### Response key components
+
+Response components used only to track quota usage:
+
+- `status`: response status code (or a list of status codes)

package/documentation/map.md

@@ -0,0 +1,96 @@
+# HTTP context mapping
+
+The `map` directive family is used to map HTTP request parts to operation call input properties.
+
+[Features](../features/map.feature).
+
+## TL;DR
+
+```yaml
+exposition:
+  /:group:
+    languages: [en, fr]   # supported languages
+    GET:
+      map:authority: hostname # request authority (e.g., hostname)
+      map:language: lang      # requested language
+      map:headers: # raw header values
+        token: x-access-token
+      map:segments: # route parameters
+        group: group
+      map:claims: # Bearer token claims
+        address: email
+        verified: email_verified
+      endpoint: observe
+```
+
+The operation input type must be an object.
+If the input already contains the specified keys, they will be overwritten.
+
+## Authority
+
+The `map:authority` directive maps the [authority identifier](authorities.md) to an operation call
+input property specified by the directive value.
+
+### Language
+
+The `map:language` mapping sets the [most matching](https://github.com/jshttp/negotiator) language
+code based on the `accept-language` request header and a list of supported languages defined by
+the `map:languages` directive, and also adds `accept-language` to the `Vary` HTTP response header
+value.
+
+If none of the supported languages match, the first supported language is used.
+
+> `map:languages` has a shorthand form: `languages: [en, fr]`.
+
+## Header values
+
+The `map:headers` directive maps the values of HTTP request headers to operation call input
+properties.
+The value of the directive is a map where keys are the names of the input properties and values are
+the names of the HTTP request headers.
+
+The names of these headers are then included in the `Vary` HTTP response header
+and [Access-Control-Allow-Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers)
+of the [CORS](protocol.md#cors).
+
+[Multiple header fields](https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2) are combined
+as a comma-separated list.
+
+## Route parameters
+
+The `map:segments` directive maps the values of route parameters to operation call input properties.
+The value of the directive is a map where keys are the names of the input properties, and values are
+the names of the route parameters.
+
+Parameter name may be prefixed with `~`
+to indicate that the parameter should not be available to the
+remaining directives or used as criteria in the operation call.
+
+```yaml
+/:id/:tag:
+  POST:
+    map:segments:
+      id: id
+      tag: ~tag
+    endpoint: create
+```
+
+Wildcard routes are supported by using `**` as the parameter name.
+
+```yaml
+/path/to/**:
+  GET:
+    map:segments:
+      path: '**'
+    endpoint: observe
+```
+
+## Bearer token claims
+
+The `map:claims` directive maps the values of
+the [token claims](https://datatracker.ietf.org/doc/html/rfc7519#section-4).
+The value of the directive is a map where keys are the names of the input properties and values are
+the names of the claims.
+
+If the claim is not present in the token or the request is not authenticated using
+the [`Bearer` scheme](identity.md#bearer-scheme), the input properties are not set.

package/documentation/notes/peers.md

@@ -0,0 +1,59 @@
+# Distributed Peer Discovery
+
+## Problem
+
+Modulo partitioning algorithm `taks.id % replicas == index` requires
+to know the number of task processing instances running in the cluster,
+and the own index of the current instance.
+
+## Forces
+
+1. Static configuration is not an option (due to dynamic scaling / failover).
+2. In a distributed system, there is no concept of a *global current time*.
+
+## Solution
+
+An algorithm that emits `(index, replicas)` once per `interval` seconds, using common Redis
+key and atomic increment.
+
+Define the following parameters:
+
+- `name`: a name of the task processing (e.g. `mail-sender`)
+- `interval`: indexing interval that is deliberately greater than the expected clock skew among
+  instances
+
+At the start of each `interval` in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time):
+
+1. Calculate an ordinal number of the current interval: `number = ceil(now() / interval)`
+2. Compose a `key` as `{name}:{number}`
+3. Atomically increment a `key` in Redis ([INCR](https://redis.io/docs/latest/commands/incr/))
+4. If `index` is defined (see 5)
+
+- get the value of previous key `{name}:{number-1}` as `replicas`
+- if the `replicas` is defined, emit `(index, replicas)` **algorithm result**
+
+5. Store the response (3) in `index`.
+
+## Safe index transition
+
+If the `index` or `replicas` changes, the algorithm consumer must stop consuming new tasks
+and execute _safe index transition_, to prevent task duplication or loss.
+
+The transition can be implemented in a manner similar to the described algorithm, using a dedicated
+`{name}:transition` key.
+However, this process is considered outside the scope of this document.
+
+## Extension
+
+If the system clocks are too precisely synchronized (skew is less than a round-trip time to Redis),
+this may result in continuous [index transitions](#safe-index-transition).
+
+To mitigate this, the algorithm can be extended with a random delay:
+
+1. Before starting the algorithm, define a random constant clock skew, significantly smaller than
+   `interval`: `skew = random() * (interval / 2)`
+2. Start the algorithm at each `interval + skew`.
+
+## Caveats
+
+- The first result will become known between `interval` and `interval × 2` seconds.

package/documentation/notes/throttling.md

@@ -0,0 +1,82 @@
+# Decentralized Request Throttling
+
+## Problem
+
+To prevent an unnecessary load, API requests should be throttled when made maliciously or due to an
+error.
+
+## Context
+
+1. The [API Gateway](https://microservices.io/patterns/apigateway.html) of a distributed system runs
+   in multiple instances, each with an in-memory state that does not allow tracking the current
+   quota usage.
+2. Precise quota enforcement (per request, per second) is not critical; the goal is to prevent
+   significant overuse.
+3. Quota configuration is static.
+
+## Forces
+
+1. No per request IO is allowed (centralized solutions do not fit).
+2. In a distributed system, there is no concept of a *global current time*.
+3. Failure to retrieve the quota state should not result in Gateway failure.
+
+## Solution
+
+Implement in-memory quotas in each process, periodically synchronizing them asynchronously using
+Redis.
+
+Consider a basic throttling rule:
+
+1. No more than `MAX_REQUESTS` within `INTERVAL` time for any API route (`KEY`).
+2. If the limit is exceeded, block requests to `KEY` for `COOLDOWN` seconds.
+
+### Concept
+
+1. Divide `INTERVAL` into `N` spans (`N >= 2`).
+2. At the end of each span,
+   atomically increment the value in Redis ([INCRBY](https://redis.io/docs/latest/commands/incrby/))
+   by the number of requests received for each `KEY`, using a key composed of the `KEY` and the
+   current ordinal number of `INTERVAL`
+   in [Unix epoch](https://en.wikipedia.org/wiki/Unix_time).
+3. If the returned value exceeds `MAX_REQUESTS`, block access to `KEY`, remove after `COOLDOWN`.
+4. If the write operation fails and `REQUESTS > MAX_REQUESTS / N` (i.e., the quota is exceeded
+   locally), block access to `KEY`, remove after `COOLDOWN`.
+
+Each API Gateway instance will generate the `KEY` based on its own local time,
+which, in general, will lead to simultaneous writes (from Redis’s local time perspective) to
+different `KEY`s. However, the total contribution of each node to each `KEY` will correspond to the
+actual request rate experienced by that node.
+
+<img src="desync.jpg" alt="Desynchronization" />
+
+Dividing the `INTERVAL` into spans smooths the desynchronization effect.
+
+### Extension
+
+Introduce `nodes`, the approximate number of active API Gateway instances, to improve the algorithm.
+
+1. Initially, `nodes` equals to `1`.
+2. During each `INTERVAL`, sum the total request count for each `KEY`, keep current and previous
+   values.
+3. At the end of each interval:
+
+- Read the value from the `KEY` for the previous interval. At this point, it is assumed that all
+  nodes have switched to the next interval.
+- Update the `nodes` value by dividing response form Redis by the number of `REQUESTS` counted for
+  the previous interval.
+- Set the previous value to the current value.
+- Set the current value to `0`
+
+4. Update point 4 of the [Concept](#concept): `REQUESTS * nodes > MAX_REQUESTS / N`.
+   This will increase the precision of the local quota enforcement.
+5. Add a new step: if `REQUESTS * nodes > MAX_REQUESTS`, block access to `KEY`, remove after
+   `COOLDOWN`.
+
+When nodes are added or removed, the algorithm will adapt in the upcoming intervals.
+
+### Caveats
+
+1. In the worst case scenario (really, really unlikely), the quota is exceeded by `MAX_REQUESTS / N`
+   in a span on each node.
+2. Time desynchronization between nodes should be insignificant for the selected `INTERVAL` (i.e.,
+   `INTERVAL` >> desync). See Extension point 3.