integreat 0.8.0-beta.8 → 0.8.0-rc.0

package/README.md

@@ -7,64 +7,66 @@ An integration layer for node.js.
 [![Coverage Status](https://coveralls.io/repos/github/integreat-io/integreat/badge.svg?branch=master)](https://coveralls.io/github/integreat-io/integreat?branch=master)
 [![Maintainability](https://api.codeclimate.com/v1/badges/a5bd9841a47ff9f74577/maintainability)](https://codeclimate.com/github/integreat-io/integreat/maintainability)
 
-**Note:** We're still changing the api rather drastically from release to
-release. We encourage trying it out and experimenting with Integreat, and we
-highly appreciate feedback, but know that anything might change.
-
-The basic idea of Integreat is to make it easy to define a set of data services
-and expose them through a well defined interface, to abstract away the specifics
-of each service, and map their data to defined schemas.
-
-This is done through:
-
-- adapters, that does all the hard work of communicating with the different
-  services
-- a definition format, for setting up each service with the right adapter and
-  parameters
-- a `dispatch()` function that sends actions to the right adapters via internal
-  action handlers
-
-It is possible to set up Integreat to treat one service as a store/buffer for
-other services, and schedule syncs between the store and the other services.
-
-Finally, there will be different interface modules available, that will plug
-into the `dispatch()` function and offer other ways of reaching data from the
-services – such as out of the box REST or GraphQL APSs.
+**Note:** We're closing in on a more stable version, but there might still be
+a few changes coming before v1.0. We encourage trying out and experimenting with
+Integreat, and we highly appreciate feedback, but know that some things might
+still change.
+
+The basic idea of Integreat is to make it easy to define how to send data to and
+receive data from a set of [**services**](#services), and expose them through a
+well defined interface, abstracting away the specifics of each service.
+
+There are a few concepts that makes this possible:
+
+- [**Transporters**](#transporters) and [**adapters**](#adapters) speak the
+  language of different types of services and standards of data exchange, and
+  does the basic translation to and from the structures used by Integreat. You
+  deal with familiar JavasScript objects, arrays, and primitive data types,
+  regardless of what the service expects.
+- [**Mutation pipelines**](#mutations) let you define how the data coming from
+  or going to a service should be transformed. This includes changing the overal
+  structure, renaming properties, transforming and filtering values with
+  transformer functions, etc. You may also provide your own transformer
+  functions.
+- [**Schemas**](#schemas) serve as a common normalization of data between
+  services. You define your own schemas and mutate data to and from them,
+  enabling inter-service sharing of data. If you have data in one schema, you
+  may send it to any service where you have set up the right mutations for this
+  schema, again abstracting away all service details.
+
+All configuration is done through basic JSON-friendly structures, and you define
+your services with different endpoints, mutation pipelines, authentication
+schemes, etc.
+
+Your configuration is spun up as an Integreat instance. To send and retrieve
+data, you dispatch [**actions**](#actions) to your instance and get
+[**response**](#action-response) objects back. You may define [jobs](#jobs) to
+run simple actions or longer "flows" consisting of several actions with
+conditions and logic. You may also configure [queues](#queues) to have actions
+run in sequence or on a later time.
 
 ```
-            _________________
-           |    Integreat    |
-           |                 |
-           |        |-> Adapter <-> Service
-Action -> Dispatch -|        |
-           |        |-> Adapter <-> Service
-           |                 |
-           |_________________|
+                   ____________________________________________________
+                  |                                                   |
+                  |                Integreat instance                 |
+Action ----|      |                                                   |
+           |-> Dispatch <-> Schema <-> Mutation <-> Adapter <-> Transporter <-> Service
+Response <-|      |                                                   |
+                  |___________________________________________________|
 ```
 
-Data from the services is retrieved, normalized, and mapped by the adapter, and
-returned asynchronously back to the code that initiated the action. Actions for
-fetching data will be executed right away.
-
-Actions that update data on services will reversely map and serialize the data
-before it is sent to a service. These actions may be queued or scheduled, by
-setting up Integreat with the supplied queue middleware.
-
-Integreat comes with a [standard data format](#the-data-format), which is the
-only format that will be exposed to the code dispatching the actions. The
-mapping, normalizing, and serializing will happing to and from this format,
-according to the defined schemas and mapping rules.
-
-To deal with security and permissions, Integreat has a built-in concept of an
-ident. Other authentication schemes may be mapped to Integreat's ident scheme,
-to provide data security from a service to another service or to the dispatched
+To deal with security and permissions, Integreat has a concept of an ident.
+Other authentication schemes may be mapped to Integreat's ident scheme, to
+provide data security from a service to another service or to the dispatched
 action. A ground principle is that nothing that enters Integreat from an
 authenticated service, will leave Integreat unauthenticated. What this means,
 though, depends on how you define your services.
 
+# Usage
+
 ## Install
 
-Requires node v8.6.
+Requires node v18.
 
 Install from npm:
 
@@ -72,802 +74,1066 @@ Install from npm:
 npm install integreat
 ```
 
-## Hello world
+You will probably also need some [transporters](#transporters) and
+[adapters](#adapters), and the basic transformers in
+[`integreat-transformers`](https://github.com/integreat-io/integreat-transformers).
 
-The hello world example of Integreat, would look something like this:
+## Basic example
+
+The following is the "hello world" example of Integreat. As most hello world
+examples, this is a bit too trivial a use case to demonstrate the real
+usefulness of Integreat, but it shows you the simplest setup possible.
+
+Here, we fetch cat facts from the API endpoint
+'https://cat-fact.herokuapp.com/facts', which returns data in JSON and requires
+no authentication. The returned list of facts are mutated and cast to the `fact`
+schema. We only fetch data _from_ the service, and no data is sent _to_ it.
 
 ```javascript
 import Integreat from 'integreat'
 import httpTransporter from 'integreat-transporter-http'
+import jsonAdapter from 'integreat-adapter-json'
 
 const schemas = [
   {
-    id: 'message',
-    service: 'helloworld',
-    shape: { text: 'string' },
+    id: 'fact', // The id of the schema
+    shape: {
+      // The fields of the type
+      id: 'string', // An id field will always be included, but we define it here for readability
+      text: 'string', // The text of the cat fact
+      createdAt: 'date', // The created date (`createdAt` and `updatedAt` will always be dates)
+    },
+    access: { allow: 'all' }, // No access restrictions
   },
 ]
 
 const services = [
   {
-    id: 'helloworld',
-    transporter: 'http',
-    endpoints: [{ options: { uri: 'https://api.helloworld.io/json' } }],
-    mappings: {
-      message: { text: 'message' },
+    id: 'catfact', // The id of the service
+    transporter: 'http', // Use the http transporter
+    adapters: ['json'], // Run the request and the response through the json adapter
+    options: {
+      // Options for the transporter
+      uri: 'https://cat-fact.herokuapp.com/facts', // Only the uri is needed here
     },
+    endpoints: [
+      {
+        match: { action: 'GET', type: 'fact' }, // Match to a GET action for type 'fact'
+        mutation: {
+          $direction: 'from', // We're mutating data _from_ the service
+          // Here we're mutating `response.data` and "setting it back" where we found it ...
+          'response.data': [
+            'response.data[]',
+            {
+              $iterate: true, // Mutate each item in an array
+              id: '_id', // The id is called `_id` the data from the service
+              text: 'text', // text is called `text`
+              createdAt: 'createdAt', // Creation date is called `createdAt`
+            },
+          ],
+        },
+      },
+    ],
   },
 ]
 
-const transporters = { http: httpTransporter }
-
-const great = Integreat.create({ schemas, services }, { transporters })
-await great.listen() // Only needed for transporter listening to incoming requests
-const action = { type: 'GET', payload: { type: 'message' } }
+// Create the Integreat instance from our definitions and provide the
+// transporters and adapters we require.
+const great = Integreat.create(
+  { schemas, services },
+  { transporters: { http: httpTransporter }, adapters: { json: jsonAdapter } }
+)
 
-great.dispatch(action).then((data) => console.log(data.text))
-//--> Hello world
+// Prepare an action to fetch all cat facts from the service `catfact`
+const action = { type: 'GET', payload: { type: 'fact', service: 'catfact' } }
 
-await great.close()
+// Dispatch the action and get the response
+const response = await great.dispatch(action)
 ```
 
-As most hello world examples, this is a bit too trivial a use case to
-demonstrate the real usefulness of Integreat, but shows the simplest setup
-possible.
-
-The example requires an imagined api at 'https://api.helloworld.io/json',
-returning the following json data:
+The `response` object will look like this:
 
-```json
-{
-  "message": "Hello world"
-}
-```
-
-## Schema definitions
-
-To do anything with Integreat, you need to define one or more schemas. They
-describe the data you expected to get out of Integreat. A type will be
-associated with a service, which is used to retrieve data for the type, unless
-another service is specified.
-
-```
+```javascript
 {
-  id: <string>,
-  plural: <string>,
-  service: <serviceId>,
-  attributes: {
-    <attrId>: {
-      type: <string>,
-      default: <object>
-    }
-  },
-  relationships: {
-    <relId>: {
-      type: <string>,
-      default: <object>,
-      query: <query params>
-    }
-  },
-  auth: <auth def>
+   status: 'ok',
+   data: [
+    {
+      id: '58e008780aac31001185ed05',
+      $type: 'fact',
+      text: 'Owning a cat can reduce the risk of stroke and heart attack by a third.',
+      createdAt: new Date('2018-03-29T20:20:03.844Z')
+    },
+    // ...
+  ]
 }
 ```
 
-The convention is to use singular mode for the `id`. The `plural` property is
-optional, but it's good practice to set it to the plural mode of the `id`, as
-some interfaces may use it. For instance,
-[`integreat-api-json`](https://github.com/integreat-io/integreat-api-json) uses
-it to build a RESTful endpoint structure, and will append an _s_ to `id` if
-`plural` is not set – which may be weird in some cases.
-
-### Attributes
-
-Each attribute is defined with an id, which may contain only alphanumeric
-characters, and may not start with a digit. This id is used to reference the
-attribute.
-
-The `type` defaults to `string`. Other options are `integer`, `float`,
-`boolean`, and `date`. Data from Integreat will be cast to corresponding
-JavaScript types.
-
-The `default` value will be used when a data service does not provide this value.
-Default is `null`.
+# Integreat concepts
 
-### Relationships
-
-Relationship is defined in the same way as attributes, but with one important
-difference: The `type` property refers to other Integreat schemas. E.g. a
-schema for an article may have a relationship called `author`, with
-`type: 'user'`, referring to the schema with id `user`. `type` is required on
-relationships.
-
-The `default` property sets a default value for the relationship, in the same
-way as for attributes, but note that this value should be a valid id for an item
-of the type the relationship refers to.
-
-Finally, relationships have a `query` property, which is used to retrieve items
-for this relationship. In many cases, a service may not have data that maps to
-id(s) for a relationship directly, and this is the typical use case for this
-property.
-
-The `query` property is an object with key/value pairs, where the key is the id
-of a field (an attribute, a relationship, or `id`) on the schema the relationship
-refers to, and the value is the id of field on this schema.
-
-Example schema with a query definition:
-
-```
-{
-  id: 'user',
-  ...
-  relationships: {
-    articles: {type: 'article', query: {author: 'id'}}
-  }
-}
-```
+As mentioned in the introduction, the building blocks of Integreat are services,
+transporters and adapters, mutation pipelines, and schemas.
 
-In this case, the `articles` relationship on the `user` schema may be fetched by
-querying for all items of type `article`, where the `author` field equals the
-`id` of the `user` item in question.
+## Services
 
-### Authorization
+A service is the API, database, FTP server, queue, etc. that you want to get
+data from and/or set data to. We pass on a set of service definitions to
+Integreat, specifying what transporter, adapters, authentication schemas it
+requires, in adition to defining the different endpoints available on the
+service, how they should be called, and how data should be mutated in each case.
 
-Set the `access` property to enforce permission checking on the schema. This
-applies to any service that provides this schema.
+We'll get back to the details of all of this in turn, but first we want to
+highlight how central the concept of a service is to Integreat. Basically, in
+Integreat "everything is a service". A simple REST/JSON API is a service, a
+database is a service, and everything external you want to communicate with are
+services. Want to set up a queue to handle actions one by one? That's a service.
+Want to cache data in a memory store? That's a service. Want to schedule actions
+to run on intervals? That's a service to. By simply defining services and their
+specifics, you may set up a variety of different types of configurations with
+the same few building blocks. This is very powerful as soon as you get into the
+right mindset.
 
-The simplest access type `auth`, which means that anyone can do anything with
-the data of this schema, as long as they are authenticated.
+Services are configured by service definitions, and tells Integreat how to
+fetch data from a service, how to mutate this data to schemas, and how to send
+data back to the service.
 
-Example of a schema with an access rule:
+The service definition object includes the transporter id, adapter ids, any
+authentication method, the endpoints for fetching from and sending to the
+service, mutations that data to all endpoints will pass through, and options
+for transporters, adapters, etc.
 
 ```javascript
 {
-  id: 'entry',
-  attributes: {...},
-  relationships: {...},
-  access: 'auth'
-}
-```
-
-To signal that the schema really has no need for authorization, use `all`.
-This is not the same as not setting the `auth` prop, as `all` will override
-Integreat's principle of not letting authorized data out of Integreat without
-an authorization rule. In a way, you can say that `all` is an authorization
-rule, but it allows anybody to access the data, even the unauthenticated.
-
-The last of the simpler access types, is `none`, which will simly give no one
-access, no matter who they are.
-
-For a more fine-grained rules, set `access` to an access definition.
-
-## Service definitions
-
-Service definitions are at the core of Integreat, as they define the services to
-fetch data from, how to map this data to a set of items to make available
-through Integreat's data api, and how to send data back to the service.
-
-A service definition object defines the adapter, any authentication method, the
-endpoints for fetching from and sending to the service, and mappings to the
-supported schemas (attributes and relationships):
-
-```
-{
-  id: <string>,
-  adapter: <string>,
+  id: <service id>,
+  transporter: <transporter id>,
+  adapters: [<adapter id>, <adapter id>, ...],
   auth: <auth id>,
   meta: <type id>,
   options: {...},
+  mutation: <mutation pipeline>,
   endpoints: [
     <endpoint definition>,
     ...
-  ],
-  mappings: {
-    <schema id>: <mapping definition | mapping id>,
-    ...
-  }
+  ]
 }
 ```
 
 Service definitions are passed to Integreat on creation through the
-`Integreat.create()` function. There is no way to change services after
-creation.
+`Integreat.create()` function. There is no way to change service defintions
+after creation.
 
-See [mapping definition](#mapping-definition) for a description of the
-relationship between services and mappings, and the `mappings` property.
+See [mutations](#mutations) for a description of how to define the mutation
+pipeline for a service.
 
 The `auth` property should normally be set to the id of an
-[auth definition](#service-authentication) if the service requires authentication.
-In cases where the service is authenticated by other means, e.g. by including
-username and password in the uri, set the `auth` property to `true` to signal
-that this is an authenticated service.
-
-### Endpoint definition
+[auth definition](#service-authentication), if the service requires
+authentication. In cases where the service is authenticated by other means, e.g.
+by including username and password in the uri, set the `auth` property to `true`
+to signal that this is an authenticated service.
+
+### Endpoints
+
+A service will have at least one endpoint, but often there will be several. An
+endpoint is a definition of one of the ways Integreat may interact with a
+service. You decide how you want to set up the endpoints and what is the right
+"endpoint design" for a service, but there might be one endpoint for each
+operation that can be done with a type of data.
+
+For example, let's say you have a simple REST API with blog articles and
+authors. There will most likely be an endpoint to fetch all (or some) articles,
+one endpoint for fetching one article by id, one endpoint for creating an
+article, one for updating an article, and so on. And you'll have similar
+endpoints for authors, one endpoint for fetching all, one for fetching one by
+id, one endpoint for creating an author, etc. As this is REST, each endpoint
+will address a different combination of urls and http verbs (through the
+transporter).
+
+As another example, you may be accessing a database of articles and authors
+directly. The configuration details will be very different than for a REST API,
+but you'll probably have the same endpoints, fetching all articles, fetching
+one, creating, updating, and the same all over for users. Instead of urls and
+http verbs, as for REST, these endpoints will address different databases and
+different database operations (through the transporter).
+
+> Note: This is not to say that Integreat requires you to set up endpoints
+> exactly as described in these examples, it might be that you would like to set
+> up an endpoint that handles many of these cases. The intention here is just to
+> give you an understanding of what an endpoint is in Integreat.
+
+When you dispatch an action, Integreat will figure out what service and what
+endpoint to send the action to. The target service is often specified in the
+action payload with the `service` property, but if not, the default service of
+the schema specified with the payload `type` property, will be used.
+
+The matching to an endpoint is done by finding the endpoint whose `match` object
+matches the action with most accuracy. The rules of the endpoint matching is
+describe in more details [below](#match-properties).
+
+Here's the format of an endpoint definition:
 
-```
+```javascript
 {
-  id: <string>,
+  id: <endpoint id>,
   match: {
-    type: <string>,
-    scope: <'collection'|'member'>,
+    type: <schema id>,
+    scope: <'collection'|'member'|'members'|'all'>,
     action: <action type>,
     params: {...},
-    filters: []
+    incoming: <boolean>,
+    filters: {...}
   },
-  validate: [],
   mutation: <mutation pipeline>,
-  sendNoDefaults: <boolean>,
-  returnNoDefaults: <boolean>,
-  mappings: {
-    <schema id>: <mapping definition | mapping id>,
-  },
-  options: {...}
+  options: {...},
+  allowRawRequest: <boolean>,
+  allowRawResponse: <boolean>
 }
 ```
 
+All of these properties are optional. An empty endpoint defintion object will
+match anything, pass on the action to the transporter untouched, and relay any
+response coming back. This might be what you need, but often you'll want to
+specify a few things:
+
+- `id`: The endpoint may have an id, which you may use to specify that you want
+  an action to go to this particular id. However, most of the time you'll set
+  up the `match` object so that Integreat will decide what endpoint to use for
+  the action you dispatch.
+- `match`: The match object is used to decide the right endpoint for an action.
+  More one this in the [Match properties](#match-properties) section.
+- `mutation`: A mutation pipeline for the endpoint. The pipeline is run for both
+  actions going to a service and the response coming back, so keep this in mind
+  when you set up this pipeline. See [Mutation pipelines](#mutation-pipelines)
+  for more on how to define the mutation.
+- `options`: This object will be passed on to the transporter and may contain
+  any properties that are meaningful to the transporter. You may also add other
+  properties for use in your mutations, but keep in mind that they will be sent
+  to the transporter. The endpoint `options` object is merged with the service
+  `options` object. Endpoint properties will override equally named service
+  properties, but this is done through deep merging, so child objects will be
+  merged as well.
+- `allowRawRequest`: When set to `true`, payload `data` sent to this endpoint
+  will not by cast automatically nor will an error be returned if the data is
+  not typed.
+- `allowRawResponse`: When set to `true`, response `data` coming from this
+  endpoint will not by cast automatically nor will an error be returned if the
+  data is not typed.
+
 #### Match properties
 
 An endpoint may specify none or more of the following match properties:
 
-- `id`: An action payload may include an `endpoint` property, that will be
-  matched against this `id`. For actions with an endpoint id, no other matching
-  properties will be considered
 - `type`: When set, the endpoint will only be used for actions with the
-  specified schema type (not to be confused with the action type)
-- `scope`: May be `member` or `collection`, to specify that the endpoint should
-  be used to request one item (member) or an entire collection of items.
-  Setting this to `member` will require an `id` property in the action payload.
-  Not setting this property signals an endpoint that will work for both
+  specified schema type (the schema's id). `type` may also be an array of types,
+  matching any one of the schemas in the list.
+- `scope`: May be `member`, `members`, `collection`, or `all`, to specify that
+  the endpoint should be used to request one item (member) by id, several items
+  by ids (members), or an entire collection of items. Setting this to `member`
+  or `members` will only match actions with a payload `id` property, and the
+  `id` should be an array of ids for `members`. Not setting this property, or
+  setting it to `all`, signals an endpoint that will work for all scopes.
 - `action`: May be set to the type string of an action. The endpoint will match
-  only actions of this type
-
-Endpoints are matched to an action by picking the matching endpoint with highest
-level of specificity. E.g., for a GET action asking for resources of type
-`entry`, an endpoint with `action: 'GET'` and `type: 'entry'` is picked over an
-endpoint matching all GET actions.
-
-Properties are matched in the order they are listed above, so that when two
-endpoints matches – e.g. one with a scope and the other with an action, the one
-matching with scope is picked. When two endpoints are equally specified with the
-same match properties specified, the first one is used.
-
-All match properties except `id` may be specified with an array of matching
-values, so that an endpoint may match more cases. However, when two endpoints
-match on a property specified as an array on one and as a single value on the
-other, the one with the single value is picked.
+  only actions of this type. When this is not specified, any action type will
+  match. `action` may also be a list of action types, matching any of these.
+- `params`: This object should list all params that this endpoint supports. A
+  param in this context is any property on the action payload except `type`,
+  `id`, or `data`. Use the param name as key on this object and set the value to
+  `true` if it is required, and `false` if it is optional. When matching
+  endpoints, an action will only match if it has all the required params, and in
+  case several match, the endpoint with more specified params will be preferred.
+- `incoming`: If this is `true`, it will only match incoming actions, if `false`
+  only outgoing, and if not set, it will match both.
+- `filters`: The filter object specifies a set of tests that needs to match an
+  action. The key of the object is a full dot notation path for the action
+  object, e.g. `payload.onlyArchived` and the value is a
+  [JSON Schema Validation object](https://json-schema.org/draft/2020-12/json-schema-validation.html).
+
+> Editor's note: Describe what incoming actions are, and give more details on
+> filters.
+
+There might be cases where several endpoints match an action, and in these cases
+the endpoint with the highest level of specificity will be used. E.g., for a
+`GET` action asking for resources of type `entry`, an endpoint with both
+`action: 'GET'` and `type: 'entry'` is picked over an endpoint matching all
+`GET` actions regardless of type. For `params` and `filters` this is decided by
+the highest number of properties on these objects.
+
+The order of the endpoints in the `endpoints` list matters only when two
+endpoints are equally specified with the same match properties specified. Then
+the first one is used.
 
 When no match properties are set, the endpoint will match any actions, as long
 as no other endpoints match.
 
-#### Params property
-
-An endpoint may accept properties, and indicate this by listing them on the
-`params` object, with the value set to `true` for required params. All
-properties are treated as strings.
+Finally, if an action specifies the endpoint id with the `endpoint`
+[payload property](#payload-properties), this overrides all else, and the
+endpoint with the id is used regardless of how the match object would apply.
 
-An endpoint is only used for actions where all the required parameters are
-present.
+Example service definition with endpoint match object:
 
-Example service definition with endpoint parameters:
-
-```
+```javascript
 {
   id: 'entries',
-  adapter: 'json',
+  transporter: 'http',
   endpoints: [
     {
-      params: {
-        author: true,
-        archive: false
+      match: {
+        type: 'entry',
+        action: 'GET',
+        scope: 'collection',
+        params: {
+          author: true,
+          archive: false
+        }
       },
       options: {
-        uri: 'https://example.api.com/1.0/{author}/{type}_log{?archive}'
+        uri: 'https://example.api.com/1.0/{author}/{type}_log?archive={archive}'
       }
     }
   ],
-  ...
+  // ...
 }
 ```
 
-Some params are always available and does not need to be specified in `params`,
-unless to define them as required:
-
-- `id`: The item id from the action payload or from the data property (if it is
-  an object and not an array). Required in endpoints with `scope: 'member'`, not
-  included for `scope: 'collection'`, and optional when scope is not set.
-- `type`: The item type from the action payload or from the data property (if it
-  is an object and not an array).
-- `typePlural`: The plural form of the type, gotten from the corresponding
-  schema's `plural` prop – or by adding an 's' to the type is `plural` is not
-  set.
-
-#### Options property
+### Service authentication
 
-Unlike the match properties, the `options` property is required. This should be
-an object with properties to be passed to the adapter as part of a request. The
-props are completely adapter specific, so that each adapter can dictate what
-kind of information it will need, but there are a set of recommended props to
-use when they are relevant:
-
-- `uri`: A uri template, where e.g. `{id}` will be placed with the value of the
-  parameter `id` from the action payload. For a full specification of the template
-  format, see
-  [Integreat URI Template](https://github.com/integreat-io/great-uri-template).
-
-- `path`: A [path](#paths) into the data, specific for this endpoint. It will
-  usually point to an array, in which the items can be found, but as mappings may
-  have their own `path`, the endpoint path may point to an object from where the
-  different mapping paths point to different arrays.
-
-- `method`: An adapter specific keyword, to tell the adapter which method of
-  transportation to use. For adapters based on http, the options will typically
-  be `PUT`, `POST`, etc. The method specified on the endpoint will override any
-  method provided elsewhere. As an example, the `SET` action will use the `PUT`
-  method as default, but only if no method is specified on the endpoint.
-
-## Mapping definition
+This definition format is used to authenticate with a service:
 
-```
+```javascript
 {
-  id: <string>,
-  type: <typeId>,
-  path: <string>,
-  attributes: {
-    <attrKey>: {
-      path: <string>,
-      transform: <transform pipeline>
-    }
-  },
-  relationships: {
-    <relKey>: {
-      path: <string>,
-      transform: <transform pipeline>
-    }
-  },
-  toService: {
-    path: <string>,
-    transform: <transform pipeline>
+  id: <id>,
+  authenticator: <authenticator id>,
+  options: {
+    // ...
   }
-  qualifier: <string>,
-  transform: <transform pipeline>,
-  filterFrom: <filter pipeline>,
-  filterTo: <filter pipeline>
 }
 ```
 
-`id`, `createdAt`, or `updatedAt` should be defined as attributes on
-the `attributes` property, but will be moved to the item on mapping. If any of
-these are not defined, default values will be used; a UUID for `id` and the
-current timestamp for `createdAt` and `updatedAt`.
-
-Data from the service may come in a different format than what is
-[required by Integreat](<(#the-data-format)>), so specify a [`path`](#paths) to
-point to the right value for each attribute and relationship. These values will
-be cast to the right schema after all mapping, mutating, and transforming is
-done. The value of each attribute or relationship should be in a format that can
-be coerced to the type defined in the schema. The `transform` pipeline may be
-used to accomplish this, but it is sufficient to return something that can be
-cast to the right type. E.g. returning `'3'` for an integer is okay, as
-Integreat will cast it with `parseInt()`.
-
-The `param` property is an alternative to specifying a `path`, and refers to a
-param passed to the `retreive` method. Instead of retrieving a value from the
-service data, an attribute or relationship with `param` will get its value from
-the corresponding parameter. When sending data _to_ a service, this
-attribute/relationship will be disregarded.
-
-Most of the time, your `attributes` and `relationships` definitions will only
-have the `path` property, so providing the `path` string instead of an object
-is a useful shorthand for this. I.e. `{title: 'article.headline'}` translates to
-`{title: {path: 'article.headline'}}`.
-
-The `toService` section behaves just like `attributes` and `relationships`,
-except that it is only used when mapping _to_ a service. This is where you'll
-put mappings that don't have a corresponding field in the schema you map
-to/from, e.g. root paths like `^params.service`.
-
-Mappings relate to both services and schemas, as the thing that binds them
-together, but it is the service definition that "owns" them. The `mappings`
-object on a service definition contains the ids of all relevant schemas as keys,
-and the value for these keys are either a mapping definition object or a mapping
-id. The `type` property of a mapping definition is optional when defined
-directly in a service definition, but should match the key if it is set.
-
-In some cases you may be able to reuse a mapping for several services or several
-types, by referring to the mapping id from several service definitions.
-
-### Paths
-
-Mappings, attributes, and relationships all have an optional `path` property,
-for specifying what part of the data from the service to return in each case.
-(Endpoints may also have a `path` property, but not all adapters support this.)
-
-The `path` properties use a dot notation with array brackets.
-
-For example, with this data returned from the service ...
+- `id`: The id used to reference this authentication, especially from the
+  [service definition](#services).
+- `authenticator`: The id of an authenticator used to authenticate requests.
+  Integreat comes with a few basic ones built in, and there are others
+  available.
+- `options`: An object of values meaningful to the authenticator. See the
+  documentation of each authenticator to learn how it should be configured.
 
-```javascript
-const data = {
-  sections: [
-    {
-      title: 'First section',
-      articles: {
-        items: [
-          {id: 'article1', title: 'The title', body: {content: 'The text'}}
-        ],
-        ...
-      }
-    }
-  ]
-}
-```
+Available authenticators:
 
-... a valid path to retrieve all `items` of the first instance of `sections`
-would be `'sections[0].articles.items[]'` and to get the content of each item
-`'body.content'`.
+- `options`: Will pass on the options as authentication, so whatever you provide
+  here is the authentication. What options to provide, then, is depending on
+  what the relevant transporter requires. This is built into Integreat.
+- `token`: A simple way of authenticating with a given token. For HTTP requests,
+  the token will be provided as a `Authorization` header, and a configurable
+  prefix like `Basic` or `Bearer`. This is built into Integreat.
+- [`jwt`](https://github.com/integreat-io/integreat-authenticator-jwt): Will
+  generate and encode a JavaScript Web Token (JWT) based on the options.
+- [`oauth2`](https://github.com/integreat-io/integreat-authenticator-oauth2):
+  Will run the balett of calling different OAuth2 endpoints and receive a token
+  based on the provided options.
 
-The bracket notation offers some possibilities for filtering arrays:
+### Configuring service metadata
 
-- `[]` - Matches _all_ items in an array
-- `[0]` - Matches the item at index 0
-- `[1:3]` - Matches all items from index 1 to 3, not including 3.
-- `[-1]` - Matches the last item in the array.
-- `[id="ent1"]` - Matches the item with an id equal to `'ent1'`
+Integreat supports getting and setting metadata for a service. The most common
+use of this is to keep track of when data of a certain type was last synced.
 
-The bracket notation also offers two options for objects:
+Some services may have support for storing their own metadata, but usually you
+set up a dedicated service for storing other services' metadata. A few different
+pieces goes into setting up a meta store:
 
-- `[keys]` - Matches all keys on an object
-- `[values]` - Matches all values for the object's keys
+- A meta schema with the fields available as metadata
+- A service for storing metadata, with an endpoint suporting the metadata schema
+- Possible a metadata mutation for the metadata service
 
-When mapping data _to_ the service, the paths are used to reconstruct the data
-format the service expects. Only properties included in the paths will be
-created, so any additional properties must be set by a transform function or the
-adapter.
+When all of this is set up, you activate the metadata on the service the
+metadata will be stored for, by setting the `meta` property to the id of the
+schema defining the metadata fields. The `service` set on the schema will tell
+Integreat what service to get and set the metadata from/to.
 
-Arrays are reconstructed with any object or value at the first index, unless a
-single, non-negative index is specified in the path.
+The schema will look something like this:
 
-Prefix a path with `^` to map with values from the mapping object, e.g.
-`^params.service`.
+```javascript
+{
+  id: 'meta', // You may give it any id you'd like and reference it on the `meta` prop on the service
+  service: <id of service handling the metadata>,
+  shape: {
+    <metadataKey>: <type string>,
+    // ...
+  }
+}
+```
 
-You may optionally supply alternative paths by providing an array of paths. If
-the first one does not match any properties in the data, the next path is tried,
-and so on.
+To get or set metadata, use [`GET_META`](#get_meta) and [`SET_META`](#set_meta)
+with the service you are getting metadata from as the `service`. Integreat will
+figure out the rest.
 
-### Qualifiers
+## Transporters
 
-When a service returns data for several schemas, Integreat needs a way to
-recognize which schema to use for each item in the data. For some services,
-the different schemas may be find on different paths in the data, so
-specifying different paths on each mapping is sufficient. But when all items
-are returned in one array, for instance, you need to specify qualifiers for
-the mappings.
+A transporter handles all the details of sending and receiving data to and from
+a service. When dispatching an action to a service, the action will be handled
+in a relevant manner for the type of service the transporter supports, e.g.
+sending an http requrest for the HTTP transporter, or doing a query to a
+database for the MongoDb transporter. Some transporters may also support
+listening to a service, e.g. the HTTP transporter listing for incoming requests
+or the MQTT transporter subscribing to events on a topic.
 
-A qualifier is simply a path with an expression that will evaluate to true or
-false. If a mapping has qualifiers, it will only be applied to data that
-satisfies all its qualifiers. Qualifiers are applied to the data at the
-mapping's path, before it is mapped and transformed.
+Integreat has transporters for some common cases, and more may come:
 
-An example of two mappings with qualifiers:
+- [Bull](https://github.com/integreat-io/integreat-transporter-bull)
+- [FTP](https://github.com/integreat-io/integreat-transporter-ftp)
+- [HTTP](https://github.com/integreat-io/integreat-transporter-http)
+- [MongoDb](https://github.com/integreat-io/integreat-transporter-mongodb)
+- [MQTT](https://github.com/integreat-io/integreat-transporter-mqtt)
+- [Redis](https://github.com/integreat-io/integreat-transporter-redis)
 
-```
-...
-mappings: {
-  entry: {
-    attributes: {...},
-    qualifier: 'type="entry"'
-  },
-  admin: {
-    attributes: {...},
-    qualifier: [
-      'type="account"',
-      'permissions.roles[]="admin"'
-    ]
-  }
-}
-```
+You may write your own transporters if your case is not covered by any of these.
+Documentation on developing transporters are coming.
 
-When a qualifier points to an array, the qualifier returns true when at least
-one of the items in the array satisfies the condition.
+Integreat will handle the transporters based on you configurations, but there
+are some specifics to each transporter, like HTTP needing an `uri` option or
+MongoDb needing a `collection` option. See the documentation of each transporter
+for more.
 
-### Configuring metadata
+## Adapters
 
-If a service may send and receive metadata, set the `meta` property to the id of
-a schema defining the metadata as attributes.
+Adapters are working together with transporters to prepare the incoming and
+outgoing data in accordance with the type of services they support.
+
+As an example, the HTTP transporter will return data from a response as a
+string, since there is no common way to treat the response body. So for a JSON
+API, you will configure the JSON adapter to make sure the data from the
+mutations are sent as a JSON string, and that the JSON comming back from the
+service is parsed before mutation starts. For a service using XML, you would
+instead set up the XML adapter, and perhaps also the SOAP adapter, to again
+stringify and parse the data going back and forth.
+
+The MongoDb transporter, on the other hand, does not require any adapters, as
+documents from the database will always come as arrays and object, and may be
+fed directly into the mutation pipelines.
+
+Integreat currently have the following adapters:
+
+- [CSV](https://github.com/integreat-io/integreat-adapter-csv)
+- [JSON](https://github.com/integreat-io/integreat-adapter-json)
+- [SOAP](https://github.com/integreat-io/integreat-adapter-soap)
+- [Url encoded form data](https://github.com/integreat-io/integreat-adapter-form)
+- [XML](https://github.com/integreat-io/integreat-adapter-xml)
+
+You may write your own adapters as well, and documentation on this is coming.
+
+## Mutations
+
+Both on the service and on endpoints, you define mutation pipelines. The service
+mutation is run before the endpoint mutation for data coming from a service, and
+in the oposite order when going to a service.
+
+A nice, but sometimes complicated, thing about mutations, is that they are run
+in both directions. They are by default defined for mutating data coming _from_
+a service, and will be run in reverse for data going _to_ a service. In some
+cases this reversing of the pipeline will work as expected without modifications
+-- you define the mutation pipeline for data coming _from_ the service, and the
+reversed pipeline works _to_ as well. But many times you need to make
+adjustments and sometimes you'll have to have separate steps based on the
+direction. We'll get into more details in the following.
+
+A mutation pipeline consists of one or more steps that the data will go through,
+before coming out on the other in the desired shape. It helps picturing this as
+an actual pipeline. After each step, data will be in a different shape, and this
+is the input to the next step.
+
+You define a pipeline in Integreat with an array, although for a pipeline with
+only one step, you may skip the array for simplicity.
+
+Each step may be one of the following:
+
+- [**A dot notation path**](#dot-notation-paths), e.g. `path.to.data`. The data
+  at that path will be extracted, and will be provided as the data to the next
+  step in the pipeline. When going in reverse, the data will be set on that path
+  instead.
+- **A mutation object** is an object that basically describes the object you
+  want as a result, where the keys are dot notation paths and the values are
+  mutation pipelines. Each pipeline on the mutation object will be run on the
+  data, and then set on the path, resulting in an object that will be passed on
+  to the next step.
+- **A transform object** letting you run a transformer function on the data,
+  e.g. `{ $transform: 'number' }` to transform the value into a number, or
+  `undefined` if not possible.
+- **A filter object** that will run a transformer function on the data and
+  filter away any items not resulting in a truthy value. As an example,
+  `{ $filter: 'boolean' }` will filter away anything that is not convertable to
+  `true` in JS rules. When applied to an array, you'll get an array where items
+  are filtered away. For an object or a plain value, filtering away will means
+  `undefined` is passed on to the next step in the pipeline.
+- **An if object** that runs a `then` pipeline if the provided pipeline returns
+  truthy, and the `else` pipeline if it returns falsy.
+- **A cast object**, e.g. `{ $cast: 'author' }` that casts the data into a
+  schema, removing all properties that is not part of the shape of the schema,
+  and transforming all values to the expected types or `undefined` if not
+  possible.
+
+### Dot notation paths
+
+At its most basic, a dot notation path is just a property key, like `content`.
+You may dive into a data structure by adding a key from the next level,
+separated by a dot, like `content.articles`. With an object like this:
 
-```
+```javascript
 {
-  id: 'meta',
-  service: <id of service handling the metadata>,
-  attributes: {
-    <metadataKey>: {
-      type: <string>
-    }
+  content: {
+    articles: [{ id: '1' }, { id: '2' }],
+    authors: [{ id: 'john' }]
   }
 }
 ```
 
-The `service` property on the type defines the service that holds metadata for
-this type. In some cases the service you're defining metadata for and the service
-handling these metadata will be the same, but it is possible to let a service
-handle other services' metadata. If you're getting data from a read-only service,
-but need to, for instance, set the `lastSyncedAt` metadata for this store,
-you'll set up a service as a store for this (the store may also hold other types
-of data). Then the read-only store will be defined with `meta='meta'`, and the
-`meta` schema will have `service='store'`.
-
-It will usually make no sense to specify default values for metadata.
+... the path `content.articles` will give you the array
+`[{ id: '1' }, { id: '2' }]`.
 
-As with other data received and sent to services, make sure to include endpoints
-for the service that will hold the metadata, matching the `GET_META` and
-`SET_META` actions, or the schema defining the metadata. The way you set up
-these endpoints will depend on your service.
+You may add brackets to the path to traverse into arrays, e.g.
+`content.articles[0]` will give you the object `{ id: '1' }`, and
+`content.articles[0].id` will give you `'1'`.
 
-Also define a [mapping](#mapping-definition) between this schema and the
-service. You may leave out `attributes` and `relationships` definitions and the
-service will receive the metadata in Integreat's standard format:
+Empty brackets, like `content.articles[]` will ensure that you get an array
+back. If the data at the path is an array, this will return the same as
+`content.articles`, but if the path returns an object or a plain value, it will
+be returned in an array.
 
-```
-{
-  id: <serviceId>,
-  type: <meta type>,
-  createdAt: <date>,
-  updatedAt: <date>,
-  attributes: {
-    <key>: <value>
-  }
-}
-```
+When mapping data _to_ a service, the paths are used to reconstruct the data
+format the service expects. Only properties included in the paths will be
+created.
 
-Finally, if a service will not have metadata, simply set `meta` to null or skip
-it all together.
+Arrays are reconstructed with any object or value at the first index, unless a
+single, non-negative index is specified in the path.
 
-## Idents and security rules
+You may use a carret `^` to go one level up -- to the parent -- in the data
+(after going down), so after `content.articles`, the path `^.authors` will
+return `[{ id: 'john' }]`. Arrays count as one level, so after
+`content.articles[0]` you will need to go up twice like so: `^.^.authors`.
 
-An ident in Integreat is basically an id unique to one participant in the
-security scheme. It is represented by an object, that may also have other
-properties to describe the ident's permissions, or to make it possible to map
-identities in other systems, to an Integreat ident.
+A double carret `^^` takes you to the top -- the root -- so after
+`content.articles[0].id`, `^^.content.authors` returns `[{ id: 'john' }]`.
 
-Example ident:
+Carret notations -- parents and roots -- does not currently work in reverse, but
+they might in a future version.
 
-```javascript
-{
-  id: 'ident1',
-  tokens: ['facebook|12345', 'twitter|23456'],
-  roles: ['admin']
-}
-```
+### Jobs
 
-The actual value of the `id` is irrelevant to Integreat, as long as it is a
-string with A-Z, a-z, 0-9, \_, and -, and it's unique within one Integreat
-configuration. This means that mapped value from services may be used as ident
-ids, but be careful to set this up right.
+> Editor's note: Write this.
 
-`tokens` are other values that may identify this ident. E.g., an api that uses
-Twitter OAuth to identify it's users, may provide the `'twitter|23456'` token in
-the example above, which will be replaced with this ident when it enters
-Integreat.
+## Schemas
 
-`roles` are an example of how idents are given permissions. The roles are
-custom defined per setup, and may be mapped to roles from other systems. When
-setting the auth rules for a service, roles may be used to require that
-the request to get data of this schema, an ident with the role `admin` must
-be provided.
+A central idea to Integreat, is that any integration has two sides; the getting
+of data from one service and the sending of data to another. Instead of setting
+up an integration directly from A to B, you have a schema in middle, and
+configure how data from A will be mutated to a schema, and then have data in
+that schema will be mutated and sent to B.
 
-Idents may be supplied with an action on the `meta.ident` property. It's up to
-the code dispatching an action to get hold of the properties of an ident in a
-secure way. Once Integreat receives an ident, it will assume this is accurate
-information and uphold its part of the security agreement and only return data
-and execute actions that the ident have permissions for.
+This is a useful abstraction, and if you ever need to change one side, you can
+do so without involving the other side. If you need to switch out service B with
+service C, you can do so without involving the configuration of service A, or
+you can send data to both B and C, using the same setup for service A.
 
-### Access rules
+To be clear, you can setup flows without schemas in Integreat, but at the loss
+of this flexibility and maintainability.
 
-Access rules are defined with properties telling Integreat which rights to
-require when performing different actions with a given schema. It may be set
-as a overall right to do anything with a schema, or it may be specified on the
-different action types available: GET, SET, and DELETE, including actions that
-start with these verbs.
+A schema describe the data you expected to get out of Integreat, or send through
+it. You basically define the fields and their types, and may then cast data to
+that shape. Note that data on an action for a specified type, will be
+automatically cast to that type.
 
-Note that this applies to the actual actions being sent to a service – some
-actions will never reach a service, but will trigger other actions, and access
-will be granted or refused to each of these actions as they reach the service
-interface, but not to the triggering action.
+```javascript
+{
+  id: <schema id>,
+  plural: <the id in plural>,
+  service: <the default service for this schema>,
+  shape: {
+    <fieldId>: <field type>,
+    <fieldId>: {
+      $type: <field type>,
+      default: <default value>
+      const: <value that will override any other value>
+    },
+  },
+  access: <access def>
+}
+```
 
-An access definition for letting all authorized idents to GET, but requiring
-the role `admin` for SETs:
+- `id`: The id of the schema, used to reference it in actions (the payload
+  `type`), when casting to the schema with `{ $type: '<schema id>' }`, and to
+  signal what schema a data object is cast to (the `$type` prop on typed data
+  items). The convention is to use singular mode for the `id`, e.g. if your
+  defining a schema for articles, you would give it the id `'article'`.
+- `plural`: When the plural of `id` is not simply a matter of adding an `'s'`,
+  you may specify the plural mode here. E.g. `id: 'entry'` would have
+  `plural: 'entries'`. This is not used by Integreat right now, but it may be
+  used in the future for error messages, generating APIs from schemas, etc.
+- `service`: You may specify a default service for the schema when it makes
+  sense. This allows you to dispatch an action for a type without specifying the
+  target service, e.g. `{ type: 'GET', payload: { type: 'article' } }`, and have
+  Integreat use the default service. This is a way of hiding configuration
+  details from the code dispatching the actions, and you may also change the
+  default service without changing the dispatching code if need be. You may
+  always override this by specifying a `service` on the action payload.
+- `shape`: This is where you define all the fields, see
+  [the section below](#the-shape-of-a-schema).
+- `generateId`: Set this to `true` to generate a unique id for the `id` field
+  when the data being cast does not provide an `id`. Default is `false`, which
+  will just set `id: null`
+- `access`: Integreat lets you define authorization schemes per schema. All use
+  of data cast to a schema will then be controlled by the rules you set here.
+  See [Access rules](#access-rules) below for details on these rules. Note that
+  `access` is optional, but when you get data from a service where any form of
+  authentication is used to access the data, you will not be able to do anything
+  with the data unless you cast it to a schema with `access` set up (or
+  specifically says that you allow raw data from that endpoint).
+
+### The shape of a schema
+
+The shape is defined by an object where each key is the id of a field, which may
+contain only alphanumeric characters, and may not start with a digit. A schema
+cannot have the same id as a primitive type (see list below).
+
+The values on this object define the types of the fields and a few other
+optional features:
 
 ```javascript
 {
-  id: 'access1',
-  actions: {
-    GET: {allow: 'auth'},
-    SET: {role: 'admin'}
-  }
+  $type: <field type>,
+  default: <default value>
+  const: <value that will override any other value>
 }
 ```
 
-To use these access rules, set the definition object directly on the `access`
-property, of an schema, or set `access: 'access1'` on the relevant schema(s).
-The `id` is only needed in the latter case.
+The `$type` prop sets the type of the field. The available primitive types, are
+`string`, `integer`, `float` (or `number`), `boolean`, and `date`. A field may
+also have another schema as its type, in which case the id of the schema is set
+in `$type`. An example can be an
+`article` schema with an `author` field of type `user`, referring to a schema
+with id `user`. When casting the `article`, data on the `author` prop will be
+cast with the `user` schema.
+
+The `default` value will be used when the field is `undefined`, `null`, or not
+preset in data object being cast to this schema. If `default` is set to a
+function, the function will be run with no argument, and the returned value is
+used as the default value. When no `default` is given, `undefined` is used.
+
+The `const` value override any value you provide to the field. It may be useful
+if you want a field to always have a fixed value. Just as for `default`, you may
+set it to a function, in which case the function will be run without arguments
+and the returned value will be used.
 
-**Note:** Referring to access rules by id is not implemented yet.
+If both `const` and `default` are set, `const` will be used.
 
-For rules that treat every action the same, set the props on the access object
-directly. This will also define a default for actions not defined specifically.
+When only setting the field type, you don't need to provide the entire object,
+you can just provide the type string.
 
-In the example above, no one will be allowed to DELETE. A better way to
-achieve what we aimed for above, could be:
+Example schema:
 
 ```javascript
 {
-  id: 'access2',
-  role: 'admin',
-  actions: {
-    GET: {allow: 'auth'}
-  }
+  id: 'article',
+  shape: {
+    id: 'string', // Not needed, as it is always provided, but it's good to include for clarity
+    title: { $type: 'string', default: 'Unnamed article' },
+    text: 'string',
+    readCount: 'integer',
+    archived: { $type: 'boolean', default: false },
+    rating: 'float',
+    createdAt: 'date',
+    updatedAt: 'date'
+  },
+  access: 'all'
 }
 ```
 
-In this example, all actions are allowed for admins, but anyone else that is
-authenticated may GET.
+Note that if you provide the `id` field, it should be set to type `'string'` or
+Integreat will throw. The same happens if you set `createdAt` or `updatedAt` to
+anything else than the type `'date'`. If you don't include these fields,
+Integreat will include the `id` for you, but not `createdAt` or `updatedAt`.
 
-Available rule props:
+### Typed data
 
-- `role` - Authorize only idents with this role. May be an array of strings
-  (array is not implemented).
-- `ident` - Authorize only idents with this precise id. May be an array (array
-  is not implemented).
-- `roleFromField` - Specify the field name (attribute or relationship) on
-  the schema, that will hold the role value. When authorizing a data item with
-  an ident, the field value on the item must match a role on the ident.
-- `identFromField` - The same as `roleFromField`, but for an ident id.
-- `allow` - Set to `all`, `auth`, or `none`, to give access to everybody, only
-  the authenticated, or no one at all. This is also available in short form –
-  use this string instead of a access rule object.
-
-Another example, intended for authorizing only the ident matching an account:
+When data is cast to a schema, the data will be in the following format:
 
-```javascript
+```
 {
-  id: 'accountAccess',
-  identFromField: 'id'
+  id: <string>,
+  $type: <schema>,
+  createdAt: <date>,
+  updatedAt: <date>,
+  <key>: <value>,
+  <key>: { id: <string>, $ref: <schema> },
+  <key: [{ id: <string>, $type: <schema>, ... }],
+  ...
 }
 ```
 
-When used with e.g. an `account` schema, given that the id of the account is
-used as ident id, only an ident with the same id as the account, will have
-access to it.
+- `id`: The id is mandatory and created by Integreat when it is not included in
+  the schema. If you don't map anything to the id prop, it will be set to
+  `null`, unless the schema is set up with `generateId: true`, in which case a
+  universally unique id will be generated for you.
+- `$type`: Set to the id of the schema by Integreat. This is a signal that the
+  data has been cast.
+- `createdAt`: This is not mandatory, but has special meaning. When a schema has
+  a `createdAt` field, but the date is not set in the data, it will be set to
+  the same as `updatedAt` (if provided) or to the current date/time.
+- `updatedAt`: Just as `createdAt`, this is not mandatory. When a schema has
+  an `updatedAt` field, and the date is not set in the data, it will be set to
+  the same as `createdAt` (if provided) or the current date/time.
+- `<key>`: Then follows the values of all the fields specified in the schema.
+  Any value not provided in the data will be set to their default value or
+  `undefined`. Fields with other schemas as their type, will be an object. If
+  only the id is provided in the data, the `{ id: <string>, $ref: <schema id> }`
+  format will be used, with `$ref` being the id of the field type schema. When
+  more data is provided, Integreat will cast it to the target schema and provide
+  the entire data object, or array of objects, with the relevant `$type`.
 
-### Persisting idents
+### Access rules
 
-A security scheme with no way of storing the permissions given to each ident,
-is of little value. (The only case where this would suffice, is when every
-relevant service provided the same ident id, and authorization where done on the
-ident id only.)
+Set the `access` property on a schema to enforce permission checking. This
+applies to any service that provides data in this schema.
 
-Unsurprisingly, Integreat uses schemas and services to store idents. In the
-definition object passed to `Integreat.create()`, set the id of the schema to
-use with idents, on `ident.schema`.
+The simplest access rule is `auth`, which means that anyone can do anything with
+the data of this schema, as long as they are authenticated. Being authenticated
+in this context, means that the dispatched action has an `ident` in the `meta`
+object. See [the section on idents](#idents) for more on this.
 
-In addition, you may define what fields (attributes or relationships) will
-match the different props on an ident:
+Example of a schema with an access rule:
 
 ```javascript
 {
-  ...,
-  ident: {
-    type: 'account',
-    props: {
-      id: 'id',
-      roles: 'groups',
-      tokens: 'tokens'
-    }
-  }
+  id: 'article',
+  shape: {
+    // ...
+  },
+  access: 'auth'
 }
 ```
 
-When the prop and the field has the same name, it may be omitted, though it
-doesn't hurt to specify it anyway – for clarity. The service still have the
-final word, as any field that is not defined on the schema, will not survive
-casting.
-
-Note that in the example above, the `id` of the data will be used as the ident
-`id`. When the id is not suited for this, you will need another field on the
-schema that may act as the ident id. In cases where you need to transform the
-id from the data in some way, this must be set up as a separate field and the
-mapping definition will dictate how to transform it. In most cases, the `id`
-will do, though.
-
-The `service` specified on the schema, will be where the ident are stored,
-although that's not a precise way of putting it. The ident is never stored, but
-a data item of the specified schema is. The point is just that the ident
-system will get the relevant data item and get the relevant fields from it. In
-the same way, when storing an ident, a data item of the specified type is
-updated with props from the ident – and then sent to the service.
+To signal that the schema really has no need for authorization, use `all`.
+This is not the same as not setting the `auth` prop, as `all` will override
+Integreat's principle of not letting authorized data out of Integreat without
+an access rule. `all` allows anybody to access the data, even the
+unauthenticated.
+
+On the other end of the spectrum, `none` will allow no one to access data cast
+to this schema, no matter who they are.
+
+For more fine-grained rules, set `access` to an access definition object with
+rules telling Integreat which rights to require when performing different
+actions with a given schema. These rules apply to the [idents](#idents) set on
+the action `meta` object.
+
+The following access rule props are available:
+
+- `allow`: Set to `all`, `auth`, or `none`, to give access to everybody, only
+  the authenticated, or no one at all. This is what we describe in short form
+  above, where we provided this string instead of a access rule object.
+- `role`: Authorize only idents with this `role`. May also be an array.
+- `ident`: Authorize only idents with this precise `id`. May also be an array.
+- `roleFromField`: Same as `role`, except the role is gotten from a field in the
+  schema. When authorizing data cast to this schema, the value of the role field
+  needs to be identical to (one of) the role(s) of the ident.
+- `identFromField` - The same as `roleFromField`, but for an ident id.
 
-For some setups, this requires certain endpoints to be defined on the service.
-To match a token with an ident, the service must have an endpoint that matches
-actions like this:
+In addition, you may override the general access rules of a schema with specific
+rules for a type of action, by setting an `action` object with access rules for
+action types. Here's an example of an access definition for allowing all
+authorized idents to `GET` data in a certain shema, requiring the role `admin`
+for `SET`s, and disallowing all other actions with the general rule
+`allow: 'none'`:
 
 ```javascript
 {
-  type: 'GET',
-  payload: {
-    type: 'account',
-    params: {tokens: 'twitter|23456'}
+  id: 'article',
+  shape: {
+    // ...
+  },
+  access: {
+    allow: 'none',
+    actions: {
+      GET: { allow: 'auth' },
+      SET: { role: 'admin' }
+    }
   }
 }
 ```
 
-In this case, `account` is the schema mapped to idents, and the `tokens`
-property on the ident is mapped to the `tokens` field on the schema.
+Note that these action specific rules only applies to actions being sent to a
+service. Some actions will never reach a service, but will instead trigger other
+actions, and access will be granted or rejected only for the actions that are
+about to be sent to a service. E.g. when you dispatch a `SYNC` action, it starts
+off by dispatching one or more `GET` actions. The `SYNC` action is not subjected
+to any access rules, but the `GET` actions are, and so the `SYNC` will fail if
+one of the `GET` is rejected.
 
-To make Integreat complete idents on actions with the persisted data, set it up
-with the `completeIdent` middleware:
+Another example, intended for authorizing only the ident matching a user:
 
 ```javascript
-const great = Integreat.create(defs, resources, [
-  integreat.middleware.completeIdent,
-])
+{
+  id: 'user',
+  shape: {
+    // ...
+  },
+  access: { identFromField: 'id' }
+}
 ```
 
-This middleware will intercept any action with `meta.ident` and replace it with
-the ident item loaded from the designated schema. If the ident has an `id`,
-the ident with this id is loaded, otherwise a `withToken` is used to load the
-ident with the specified token. If no ident is found, the original ident is
-kept.
+Here, only actions where the ident id is the same as the id of the user data,
+will be allowed. This means that authenticated users (idents) may only
+only access their own user data.
 
 ## Actions
 
-Actions are serializable objects that are dispatched to Integreat, and may be
-queued when appropriate. It is a key point that they are serializable, as they
-allows them to be put in a database persisted queue and be picked up of another
-Intergreat instance in another process.
+Actions are serializable objects that are dispatched to Integreat. It is a
+important that they are serializable, as this allows them to, for instance, be
+put in a database persisted queue and be picked up of another Intergreat
+instance in another process. Note that `Date` objects are considered
+serializable, as they are converted to ISO date strings when needed.
 
 An action looks like this:
 
-```
+```javascript
 {
-  type: <actionType>,
-  payload: <payload>,
-  meta: <meta properties>
+  type: <action type>,
+  payload: <payload object>,
+  meta: <meta object>
 }
 ```
 
-`type` is one [of the action types](#available-actions) that comes with
-Integreat and `payload` are data for this action.
-
-The `meta` object is for properties that does not belong in the payload. You may
-add your own properties here, but be aware that some properties are already
-used by Integreat, and more may be added in the future.
+- `type`: This is the id of one [of the action handlers](#available-action-handlers)
+  that comes with Integreat, e.g. `GET`. When you dispatch an action, it is
+  handed off to this handler (after some inital preperation). You may write
+  your own action handlers as well.
+- `payload`: Holds parameters and [data](#typed-data) for this action. There are
+  some reserved [payload properties](#payload-properties), and the rest will be
+  made available to you in the mutation pipeline.
+- `meta`: Holds information about the action that does not belong in the
+  payload, like the ident of the user dispatching, action id, etc. There are
+  some reserved [meta properties](#meta-properties), but you may add your own
+  here too.
+
+When an action is dispatched, it returns a [response object](#action-response)
+with status, data, error message, etc.
+
+Note that in a mutation pipeline, action handler, or middleware, the
+response object is provided as a fourth property on the action. You will most
+likely meet this at least when setting up mutations.
+
+### Payload properties
+
+The payload is, together with the action `type`, a description to Integreat and
+the service of what to do. A design principle of Integreat has been to have as
+little specifics in these payload, so actions may be discpatched to service
+without knowing how the service works. This is not always possible, at least not
+yet, but it's a good principle to follow, also when you configure services and
+plan what props need to be sent in the action payload.
+
+You may set any properties on the payload, and they will be be available to you
+in the service endpoint match and in the service mutations. Some properties have
+special meanings, though, and you should avoid using them for anything else:
+
+- `type`: The type of the data the action sends and/or receives. This refers to
+  the `id` of a schema, and you will usually want to set this. Data provided
+  in the payload `data` and response `data` will be cast to this schema. If
+  you're dealing with several types in one action, you may set an array of
+  types, but will have to cast the data in the mutation yourself. Integreat will
+  validate that the data you send and receive is indeed of that type, and will
+  give you an auth error if not. (See
+  [`allowRawRequest` and `allowRawResponse` on endpoints](#endpoints) for an
+  exception.)
+- `id`: You provide an id when you want to address a specific data item, usually
+  when you want to fetch one data item with an action like
+  `{ type: 'GET', payload: { type: 'article', id: '12345' } }`. You may also
+  supply an array of ids to fetch several data items by id. When setting data,
+  the id will instead be specified in the `data` when appropriate.
+- `data`: The data to send to a service. This may be any data that makes sense
+  to the service, but will usually be a [typed data object](#typed-data) or an
+  array of typed data objects, where the adjustments for the service happens in
+  service mutations.
+- `service`: The `id` of the service to send this action to. If not specified,
+  Integreat will try and find the right service from the `type`.
+- `targetService`: An alias of `service`.
+- `sourceService`: When data comes from a different service and has not been
+  mutated and cast yet, the `sourceService` property will tell Integreat to run
+  the data through the source service configuration before passing the action on
+  to an action handler. An example may be data coming in through an API, where
+  the API is configured as a service in Integreat. Note that this property is
+  usually set by transporters in their `listen()` methods, but you may also set
+  it directly on the action when it makes sense.
+- `endpoint`: Set this to the `id` of a service endpoint when you want to
+  override the endpoint match rules of Integreat. This should only be used when
+  it is really necessary. Normally, you should instead design the match
+  properties to match the correct actions.
+
+For services that support pagination, i.e. fetching data in several rounds, one
+page at a time, the following properties may be supported:
+
+- `pageSize`: The number of data items to fetch in one request to the service.
+  By specifying a page size, you signal that you would like to use pagination,
+  and without it all other pagination properties will be disregarded. You will
+  get the number of data items you specify (or less, if there are not that many
+  items), and may then go on to dispatch an action for the next page. See
+  [pagination](#pagination) for more
+- `pageOffset`: The number of data items to "skip" before returning the number
+  of items specified in `pageSize`. If you ask for 500 items, the first action
+  should have `pageOffset: 0` (or not specified), the next action
+  `pageOffset: 500`, then `pageOffset: 1000`, and so on.
+- `page`: The index of the page to fetch. Unlike most other indexes, this starts
+  with `1` being the first page. The effect is the same as `pageOffset`, it's
+  just a different way of specifying it. `page: 1` is the same as
+  `pageOffset: 0`, and `page: 2` is the same as `pageOffset: 500`, given a
+  `pageSize: 500`. Integreat will actually calculate both before sending it to
+  the transporter, as different types of services support different types of
+  pagination.
+- `pageAfter`: As an alternative to specifying the number of items to skip, you
+  may ask for the items after the item with the id you provide as `pageAfter`.
+  If the last item of the first page is `'12345'`, you may set
+  `pageAfter: '12345'` to get the next page.
+- `pageBefore`: This works the same as `pageAfter`, except it is intended for
+  when your going backward and fetching a number items _before_ the id you
+  provide.
+- `pageId`: Some services and/or transporters will return an id for the next
+  page, as an alternative to the other properties mentioned above. You then
+  apply this id as `pageId` when dispatching the action for the next page. Note
+  that this id may hold internal logic from the transporter, but you should
+  never rely on this logic and simply use it as an id.
+
+> **Important note:** Pagination has to be supported by the service and your
+> service configuration, and sometimes also the transporter. Integreat prepares
+> and passes on these pagination properties, but if the service disregards them,
+> there is little Integreat can do – except limiting the number of items
+> returned. It's up to you to figure out how to configure pagination for a
+> service, but youshould use these pagination properties to support it, to make
+> this predictable. It also lets you use actions such as `GET_ALL`, that support
+> pagination.
+
+Finally, there are some properties that has no special meaning to Integreat
+itself, but that may be set on incoming actions from transporters. These should
+ideally be used in the same way or avoided:
+
+- `contentType`: A keyword for the type of content in the `data` property. E.g.
+  `application/json` or `text/plain`.
+- `headers`: An object of header information, given as key/value pairs. The
+  value may be a string or an array of strings. This may be HTTP headers or any
+  other type of header information that makes sense to a service.
+- `hostname`: The host name that incoming request was sent to. For HTTP, this
+  will be the domain name the request was sent to.
+- `method`: The method of the incoming request. The HTTP transporter will set
+  this to `GET`, `POST`, `PUT`, etc. from the incoming request.
+- `path`: The path from the incoming request. For the HTTP transporter, this
+  will be the part of the url after the domain name, like
+  `'/v1.0/articles/12345'`.
+- `port`: The port number of the incoming request.
+- `queryParams`: An object of query params from the incoming request, usually
+  key/value pairs where the value is a string or an array of strings. For HTTP,
+  this will be the part after the question mark.
+
+### Meta properties
+
+The action meta object is for information about an action that does not directly
+define the action itself. The difference may be subtle in some cases, but the
+general rule is a piece of information affects how the action is run, it should
+be in the payload. E.g. the type of items to fetch is in the payload, while the
+time the action was dispatched would go in the meta.
+
+This rule does not always hold, e.g. for information on the user dispatching the
+action in `ident` on the meta object. Different idents may result in different
+data being returned from the service, but still the action to perform is the
+same, so it makes sense to have the ident in the meta object.
+
+You may set your own meta properties, but in most cases you'll probably rather
+set payload properties.
 
 Current meta properties reserved by Integreat:
 
-- `id`: Assigning the action an id. Will be picked up when queueing.
-- `queue`: Signals that an action may be queued. May be `true` or a timestamp
-- `queuedAt`: Timestamp for when the action was pushed to the queue
-- `schedule`: A [schedule definition](#schedule-definition)
-- `ident`: The ident to authorize the action with
+- `ident`: The ident to authorize the action with. May hold an `id`, `roles`,
+  `tokens`, and a few other options. See
+  [the section on idents](#idents-and-security-rules).
+- `id`: The id of the action itself. You may set this yourself or let Integreat
+  generate a universally unique id for you. Useful for logging and may be used
+  by queues.
+- `cid`: A correlation id that may be used to group actions that belong
+  together, primarily for logging purposes. You may set this yourself or
+  Integreat will set it to the same as the `id`. Some Integreat action handlers
+  will dispatch sub actions using the `cid` from the original action.
+- `dispatchedAt`: Timestamp for when the action was dispatched (set by
+  Integreat).
+- `queue`: Signals to Integreat that an action may be queued. Set to `true` when
+  you want the action to be queued, but executed as soon as possible. Set to a
+  UNIX timestamp (number) to schedule for a later time. If no queue is set up,
+  the action will be dispatched right away. More on this under
+  [the section on queues](#queue).
+- `queuedAt`: Timestamp for when the action was pushed to the queue (set by
+  Integreat).
+- `options`: Used for passing the processed service endpoint options object to
+  a transporter. The `options` object is available through mutations, so that
+  you may modify it futher before it goes to the transporter.
+- `authorized`: An internal flag signaling that the action has been authorized.
+  Will be removed from any dispatched actions.
+
+### Action response
+
+When you dispatch an action, you will get a response object back in this format:
 
-### Returned responses from actions
-
-Retrieving from a service will return an Intgreat response object of the
-following format:
-
-```
+```javascript
 {
-  status: <statusCode>,
-  data: <object>
-  error: <string>
+  status: <status code>,
+  data: <data from the service, usually mutated>,
+  error: <error message>,
+  warning: <warning message>,
+  access: <ident of the user>,
+  paging: <pagination objects>,
+  params: <key/value pairs>,
+  headers: <key/value pairs>,
+  responses: <array of sub-responses when relevant>,
+  meta: <meta object>
 }
 ```
 
-The `status` will be one of the following status codes:
+- `status`: The status of the action. Will be `ok` when everything went well,
+  see [list of status codes](#status-codes) below for more.
+- `data`: Any data returned from the service, after being modified by the
+  mutation pipelines from your service and endpoint configuration. It will be
+  cast to [typed data](#typed-data) through the schema specified by the payload
+  `type`, if it is set to a single type and the endpoint `allowRawResponse` is
+  not set to `true`.
+- `error`: All error statuses (i.e. not `ok` or `queued`) will return an error
+  message, some may include error messages from the service.
+- `warning`: When the action was successful, but there still was something you
+  should know, the warning message is where you'll get noticed. An example is
+  when you get an array of data items, but some of them was removed due to the
+  access of the ident on the action.
+- `paging`: For services and transporters that support
+  [pagination](#pagination), this object will hold information about how to get
+  the next or previous page, in a `next` or `prev` object. These objects are
+  essentially the payloads you need to dispatch (with the same action `type` and
+  meta), to get the next or previous page. If there is no next or previous page,
+  the corresponding prop will not be set on the `paging` object. When pagination
+  is not relevant or used, the `paging` object may be missing completely.
+- `params`: Integreat never sets this, but you may set it in your mutations to
+  provide parameters from a service that does not belong in the `data`.
+- `headers`: Integreat never sets this, but you may set it in your mutations to
+  provide header key/value pairs from a service. Typically used when this is a
+  response to an incoming request that support headers, like HTTP do.
+- `responses`: In some cases, an action will run several sub-actions, like
+  `SYNC` or `RUN`. The action handlers _may_ then provide an array of all the
+  sub-response objects here.
+
+> Editor's note: Do we return access in responses?
+> Editor's note: Do we return meta in responses?
+> Editor's note: Is it correct that queues return the id in the data?
+
+When the status is `queued`, the id of the queued action may found in
+`response.data.id`. This is the id assigned by the queue, and not necessarily
+the same as `action.meta.id`.
+
+### Status codes
+
+The `status` or the action response will be one of the following status codes:
 
 - `ok`: Everything is well, data is returned as expected
-- `queued`: The action has been queued
-- `notfound`: Tried to access a resource/endpoint that does not exist
-- `noaction`: The action did nothing
+- `queued`: The action has been queued. This is regarded as a success status
+- `noaction`: The action did nothing, e.g. when a `SYNC` action has no data to
+  sync
+- `notfound`: Tried to get or modify a resource that does not exist
 - `timeout`: The attempt to perform the action timed out
 - `autherror`: An authentication request failed
 - `noaccess`: Authentication is required or the provided auth is not enough
@@ -875,121 +1141,121 @@ The `status` will be one of the following status codes:
 - `badresponse`: Response data is not as expected
 - `error`: Any other error
 
-On `ok` status, the retrieved data will be set on the `data` property. This will
-usually be mapped data in [Integreat's data format](#the-data-format), but
-essentially, the data format depends on which action it comes from.
+### Idents
 
-When the status is `queued`, the id of the queued action may found in
-`response.data.id`. This is the id assigned by the queue, but it is expected
-that queues will use `action.meta.id` when present.
-
-In case of any other status than `ok` or `queued`, there will be no `data`, and
-instead the `error` property will be set to an error message, usually returned
-from the adapter.
-
-`data` and `error` will never be set at the same time.
-
-The same principles applies when an action is sending data or performing an
-action other than receiving data. On success, the returned `status` will be
-`ok`, and the `data` property will hold whatever the adapter returns. There is
-no guaranty on the returned data format in these cases.
-
-### The data format
+An ident in Integreat is basically an id unique to one participant in the
+security scheme. It is represented by an object that may also have other
+properties to describe the ident's permissions, or to make it possible to match
+to identities in other services.
 
-Items will be in the following format:
+Example ident:
 
-```
+```javascript
 {
-  id: <string>,
-  type: <schema>,
-  createdAt: <date>,
-  updatedAt: <date>,
-  attributes: {
-    <attrKey>: <value>,
-    ...
-  },
-  relationships: {
-    <relKey>: {id: <string>, type: <schema>},
-    <relKey: [{id: <string>, type: <schema>, ...],
-    ...
-  }
+  id: 'ident1',
+  tokens: ['facebook|12345', 'twitter|23456'],
+  roles: ['admin']
 }
 ```
 
-`id`, `type`, `createdAt`, and `updatedAt` are mandatory and created by
-Integreat even when there are no mappings to these fields. In the future,
-`createdAt` and `updatedAt` may become properties of `attributes`, but will
-still be treated in the same way as now.
-
-### Available actions
+- `id`: A unique string identifying the ident. The actual value is irrelevant to
+  Integreat, as long as it is a string with A-Z, a-z, 0-9, \_, and -, and it's
+  unique within one Integreat configuration. This means that mapped values from
+  services may be used as ident ids, as long as they are unique among these
+  services.
+- `tokens`: A list of values that may identify this ident in other services. For
+  example, an api that uses Twitter OAuth to identify its users, may provide
+  the `'twitter|23456'` token in the example above, which will be replaced with
+  this ident when it enters Integreat.
+- `roles`: A list of roles or permissions given to this ident. The roles are
+  custom defined per setup, and may be mapped to roles from other systems. When
+  setting the auth rules for a schema, you specify required rules so that to get
+  data cast in this schema, an ident with e.g. the role `admin` must be
+  provided.
+
+Actions are authenticated by setting an ident on the `meta.ident` property. It's
+up to the code dispatching an action to get hold of the properties of an ident
+in a secure way. Once Integreat receives an ident through a dispatch, it will
+assume this is accurate information and uphold its part of the security
+agreement and only return data and execute actions that the ident have
+permissions for.
+
+Note that it's possible to set up
+[the `completeIdent` middleware](#completeIdent-middleware) for combining
+information from the authenticator with user information held e.g. in a
+database.
+
+### Available action handlers
 
 #### `GET`
 
-Get items from a service. Returned in the `data` property is an array of mapped
-object, in [Integreat's data format](#the-data-format).
+Get data from a service. You receive the data on the `data` property, after it
+has been run through your service and endpoint mutations.
 
-Example GET action:
+Example GET action to a collection of data items:
 
 ```javascript
 {
   type: 'GET',
-  payload: {
-    type: 'entry'
-  }
+  payload: { type: 'article' }
 }
 ```
 
-In the example above, the service is inferred from the payload `type` property.
-Override this by supplying the id of a service as a `service` property.
-
 By providing an `id` property on `payload`, the item with the given id and type
-is fetched, if it exists.
+is fetched, if it exists:
 
-The endpoint will be picked according to the matching properties, unless an
-endpoint id is supplied as an `endpoint` property of `payload`.
+```javascript
+{
+  type: 'GET',
+  payload: { type: 'article', id: '12345' }
+}
+```
 
-By default, the returned data will be cast with default values, but set
-`returnNoDefaults: true` on the action payload to get only values mapped from
-the service data.
+See [the section on payload properties](#payload-properties) for more properties
+that may be used with the `GET` action.
 
-#### `GET_UNMAPPED`
+#### `GET_ALL`
 
-Get data from a service without applying the mapping rules. Returned in the
-`data` property is an array of normalized objects in the format retrieved from
-the service. The data is not mapped in any way, and the only thing guarantied, is
-that this is a JavaScript object.
+Will run as many `GET` actions as needed to the get all available pages of data.
 
-This action does not require a `type`, unlike the `GET` action, as it won't
-lookup mappings for any given type. The only reason to include a `type` in the
-payload, would be if the endpoint uri requires a `type` parameter.
+The action ...
 
-Furthermore, a `service` property is required, as there is no `type` to infer
-from.
+```javascript
+{
+  type: 'GET_ALL',
+  payload: { type: 'article', pageSize: 500 }
+}
+```
 
-Example GET action:
+... will dispatch the following action is sequence:
 
 ```javascript
 {
-  type: 'GET_UNMAPPED',
-  payload: {
-    service: 'store',
-    endpoint: 'get'
-  }
+  type: 'GET',
+  payload: { type: 'article', pageSize: 500 }
+}
+```
+
+```javascript
+{
+  type: 'GET',
+  payload: { type: 'article', pageSize: 500, pageOffset: 500 }
 }
 ```
 
-The endpoint will be picked according to the matching properties, unless an
-endpoint id is supplied as an `endpoint` property of `payload`.
+... and so on, until we get no more data.
+
+See [the section on pagination](#pagination) for more on the paging properties.
 
 #### `GET_META`
 
-Get metadata for a service. Normal endpoint matching is applied, but it's
-common practice to define an endpoint matching the `GET_META` action.
+Get metadata for a service. See
+[the section on metadata](#configuring-service-metadata) for how to set this up.
 
-The action returns an object with a `data` property, which contains the `service`
-(the service id) and `meta` object with the metadata set as properties.
+The `data` of the response from this aciton contains the `service` (the service
+id) and `meta` object with the metadata set as properties.
 
-Example GET_META action:
+Example `GET_META` action:
 
 ```javascript
 {
@@ -1020,55 +1286,43 @@ If the action has no `keys`, all metadata set on the service will be retrieved.
 The `keys` property may be an array of keys to retrieve several in one request,
 or a single key.
 
-Note that the service must be set up to handle metadata. See
-[Configuring metadata](#configuring-metadata) for more.
-
 #### `SET`
 
-Send data to a service. Returned in the `data` property is the data that was sent
-to the service – casted, but not mapped to the service.
+Send data to a service. The data to send is provided in the payload `data`
+property. Recomended practice is to provide the data as
+[typed data](#typed-data), i.e. data objects cast to a schema, and let
+mutations on the service endpoint modify it to the format the service expects.
 
-The data to send is provided in the payload `data` property, and must given as
-an array of objects in [Integreat's data format](#the-data-format).
+Any data coming back from the service, will be provided on `response.data` and
+may be mutated through service endpoint mutations, just as for [`GET`](#get)
+actions.
 
-Example SET action:
+Example `SET` action:
 
 ```javascript
 {
   type: 'SET',
   payload: {
-    service: 'store',
+    type: 'article',
     data: [
-      {id: 'ent1', type: 'entry'},
-      {id: 'ent5', type: 'entry'}
+      { id: '12345', $type: 'article', title: 'First article' },
+      { id: '12346', $type: 'article', title: 'Second article' }
     ]
   }
 }
 ```
 
-In the example above, the `service` is specified in the payload. Specifying a
-`type` to infer the service from is also possible, but not recommended, as it
-may be removed in future versions of Integreat.
-
-The endpoint will be picked according to the matching properties, unless an
-endpoint id is supplied as an `endpoint` property of `payload`.
-
-To send only fields mapped from the action data to the service, set
-`sendNoDefaults: true` on the endpoint config to cast the data going to the
-service without using default values. This will not affect the data coming back
-from the action, but set `returnNoDefaults: true` to leave defaults out of the
-response data.
-
 #### `SET_META`
 
-Set metadata on a service. Returned in the `data` property is whatever the
-adapter returns. Normal endpoint matching is used, but it's common practice to
-set up an endpoint matching the `SET_META` action.
+Set metadata on a service. The payload should contain the `service` to get
+metadata for (the service id), and a `meta` object, with all metadata to set as
+properties.
 
-The payload should contain the `service` to get metadata for (the service id), and
-a `meta` object, with all metadata to set as properties.
+Any data coming back from the service, will be provided on `response.data` and
+may be mutated through service endpoint mutations, just as for [`GET`](#get)
+actions.
 
-Example SET_META action:
+Example `SET_META` action:
 
 ```javascript
 {
@@ -1083,37 +1337,35 @@ Example SET_META action:
 ```
 
 Note that the service must be set up to handle metadata. See
-[Configuring metadata](#configuring-metadata) for more.
+[Configuring metadata](#configuring-service-metadata) for more.
 
 #### `DELETE` / `DEL`
 
-Delete data for several items from a service. Returned in the `data` property is
-whatever the adapter returns.
+Delete one or more items from a service. Set the data for the items to delete,
+in the payload `data` property as an array of [typed data](#typed-data).
+In most cases, you only need to provide the `id` and the `$type`, but the way
+you set up the service may require more properties.
 
-The data for the items to delete, is provided in the payload `data` property,
-and must given as an array of objects in
-[Integreat's data format](#the-data-format), but note that the attributes and
-relationships are not required.
+Any data coming back from the service, will be provided on `response.data` and
+may be mutated through service endpoint mutations, just as for [`GET`](#get)
+actions.
 
-Example DELETE action:
+Example `DELETE` action:
 
 ```javascript
 {
   type: 'DELETE',
   payload: {
-    service: 'store',
+    type: 'article',
     data: [
-      {id: 'ent1', type: 'entry'},
-      {id: 'ent5', type: 'entry'}
+      { id: '12345', $type: 'article' },
+      { id: '12346', $type: 'article' }
     ]
   }
 }
 ```
 
-In the example above, the `service` is specified in the payload. Specifying a
-`type` to infer the service from is also possible.
-
-Example DELETE action for one item:
+You may also `DELETE` one item like this:
 
 ```javascript
 {
@@ -1125,63 +1377,84 @@ Example DELETE action for one item:
 }
 ```
 
-The endpoint will be picked according to the matching properties, unless an
-endpoint id is supplied as an `endpoint` property of `payload`.
+`DEL` is a shorthand for `DELETE`.
 
-The method used for the request defaults to `POST` when `data` is set, and
-`DELETE` for the `id` and `type` option, but may be overridden on the endpoint.
+#### `RUN`
 
-`DEL` is a shorthand for `DELETE`.
+The `RUN` action will run jobs provided to `Integreat.create()` in the jobs
+definitions. These jobs will then run other actions or series of action, also
+called "flows".
+
+Only one payload property is required – the `jobId`, which refers to a job in
+the jobs definitions. Any other properties on the payload will be passed on as
+input to the job.
+
+An action for running the `archiveOutdated` job:
+
+```javascript
+{
+  type: 'RUN',
+  payload: { jobId: 'archiveOutdated' }
+}
+```
+
+See [the section on jobs](#jobs) for more on how to configure jobs.
 
 #### `SYNC`
 
-The `SYNC` action will retrieve items from one service and set them on another.
+The `SYNC` action will `GET` items from one service and `SET` them on another.
 There are different options for how to retrieve items, ranging from a crude
 retrieval of all items on every sync, to a more fine grained approach where only
-items that have been updated since last sync, will be synced.
+items that have been updated or created since last sync, will be synced.
 
 The simplest action definition would look like this, where all items would be
 retrieved from the service and set on the target:
 
-```
+```javascript
 {
   type: 'SYNC',
   payload: {
-    from: <serviceId | params>,
-    to: <serviceId | params>,
-    type: <itemType>,
-    retrieve: 'all'
+    type: <item type>,
+    retrieve: 'all',
+    from: <service id | payload>,
+    to: <service id | payload>
   }
 }
 ```
 
-The action will dispatch a 'GET' action right away, and then immediately
+The action will dispatch a `GET` action right away, and then immediately
 dispatch a `SET_META` action to update the `lastSyncedAt` date on the service.
-The actions to update the target is added to the queue.
+The `SET` actions to update the target service is added to the queue if one is
+configured.
 
 To retrieve only new items, change the `retrieve` property to `updated`. In
-this case, the action will get the `lastSyncedAt` from the `from` service, and
-get only newer items, by passing it the `updatedAfter` param. The action will
-also filter out older items, in case the service does not support `updatedAfter`.
+this case, the action will dispatch `GET_META` to get the `lastSyncedAt` from
+the `from` service, and get only newer items, by passing it the `updatedAfter`
+param. The action will also filter out older items, in case the service does not
+support `updatedAfter`.
+
+By setting `retrieve` to `created`, you accomplish the same, but with
+`createdAfter`.
 
 If you need to include more params in the actions to get from the `from` service
 or set to the `to` service, you may provide a params object for the `from` or
-`to` props, with the service id set as a `service` param.
+`to` props, with the service id set as a `service` param. You may also provide
+different action types than `GET` and `SET`, by setting the `action` prop on
+the `from` or `to` objects respectively.
+
+> There are more options than these, and the documentation will be updated to
+> include them later.
 
 #### `EXPIRE`
 
-With an endpoint for getting expired items, the `EXPIRE` action will fetch
-these and delete them from the service. The endpoint may include param for the
-current time, either as microseconds since Januar 1, 1970 UTC with param
-`{timestamp}` or as the current time in the extended ISO 8601 format
-(`YYYY-MM-DDThh:mm:ss.sssZ`) with the `{isodate}` param. To get a time in the
-future instead, set `msFromNow` to a positive number of milliseconds to add
-to the current time, or set `msFromNow` to a negative number to a time in the
-past.
+> Note: This action will change before we reach v1.0.
 
-Here's a typical action definition:
+The `EXPIRE` action will `GET` expired data items from a service, and the then
+`DELETE` them.
 
-```
+Here's an example of an `EXPIRE` action:
+
+```javascript
 {
   type: 'EXPIRE',
   payload: {
@@ -1193,21 +1466,40 @@ Here's a typical action definition:
 }
 ```
 
-This will get and map items of type `entry` from the `getExpired` endpoint on
-the service `store`, and delete them from the same service. Only an endpoint
-specified with an id is allowed, as the consequence of delete all items received
-from the wrong endpoint could be quite severe.
+The `endpoint` property is required for this action, and needs to specify a
+service endpoint used to fetch expired items. The action dispatched to this
+endpoint will have a `timestamp` property with the current time as microseconds
+since epoc (Januar 1, 1970 UTC), and `isodate` as the current time in the
+extended ISO 8601 format(`YYYY-MM-DDThh:mm:ss.sssZ`).
+
+To have `timestamp` and `isodate` be a time in the future instead, set
+`msFromNow` to a positive number of milliseconds. This will be added to the
+current time. To have a time in the past, use a negative number for `msFromNow`.
 
-Example endpoint uri template for `getExpired` (from a CouchDB service):
+#### `SERVICE`
+
+A `SERVICE` action will be sent directly to the specified service without any
+intervention by Integreat. This allows for running specialized actions on the
+service that goes beyond what Integreat supports. It's up to each transporter to
+support such actions, describe what they'll do, and define their payload
+properties.
+
+An example of an action that will tell a
+[Bull](https://github.com/integreat-io/integreat-transporter-bull) queue to
+clean out all completed jobs more than a week old:
 
 ```javascript
 {
-  uri: '/_design/fns/_view/expired?include_docs=true{&endkey=timestamp}',
-  path: 'rows[].doc'
+  type: 'SERVICE',
+  payload: {
+    type: 'cleanCompleted',
+    targetService: 'bullService',
+    olderThanMs: 604800000
+  }
 }
 ```
 
-### Custom actions
+### Write your own action handlers
 
 You may write your own action handlers to handle dispatched actions just like
 the built-in types.
@@ -1215,190 +1507,252 @@ the built-in types.
 Action handler signature:
 
 ```javascript
-function (payload, {dispatch, services, schemas, getService}) { ... }
+async function (action, { dispatch, getService, setProgress, options }) { ... }
 ```
 
-An action handler may dispatch new actions with the `dispatch()` method. These
-will be passed through the middleware chain just like any other action, so it's
-for instance possible to queue actions from an action handler by setting
-`action.meta.queue = true`.
-
-The `services` and `schemas` arguments provide all services and schemas set on
-objects with their ids as keys.
-
-Finally, `getService()` is a convenience method that will return the relevant
-service object when you provide it with a type. An optional second argument may
-be set to a service id, in which case the service object with this id will be
-returned.
-
-Custom actions are supplied to an Integreat instance on setup, by providing an
-object with the key set to the action type your handler will be responsible for,
-and the handler function as the value.
+- `action`: This is the dispatched action after it has been modified a bit
+  by the `dispatch()` method and possible after running an incoming mutation on
+  it. The modifications include cleaning up alias fields (e.g. `service` will be
+  set as `targetService`), removing sensitive or forbidden fields, and setting a
+  few default or internal fields (like the `dispatchedAt` meta).
+- `dispatch`: From the handler, you may dispatch your own sub actions to the
+  provided `dispatch()` method. Note that this is an "internal dispatch method",
+  so it will return an action with the `response` object on it, instead of just
+  the `response` object. It's good practice to set the `cid` meta prop for the
+  actions you dispatch, to the `cid` meta prop on the `action` you're handling.
+  You should also use the same `ident` unless you have very good reasons to do
+  otherwise, to make sure you don't create security holes.
+- `getService`: This is a convenience method that will return the relevant
+  service object when you provide it with a type and optional a service id. With
+  a service id, you'll get the service with that id, with only the type, you'll
+  get the default service for that type. E.g.: `getService('article')`.
+- `setProgress`: For long running tasks, you may want to set the progress along
+  the way. Progress is specified as a number between `0` and `1`, e.g.
+  `setProgress(.5)` to signal that you're halfway through. When the your handler
+  is finished, the progress will automatically be set to `1`. This may be used
+  by queue implementations etc., to give progress feedback to users and to know
+  the action has not gone stale.
+- `options`: This is an object with a few settings: `queueService` is the id of
+  the service set up as the default queue, and `identConfig` is the config
+  object used for mapping ident schemas to ids, roles, and tokens (see
+  [the `completeIdent` middleware](#completeIdent-middleware)).
+
+Your action handler must return the `action` it received with a `response`
+object set on it. If your handler just relays to another action handler, it may
+reuse the `response` from that handler, but in many cases it will be more
+correct to generate your own response, possibly based on what the actions you
+dispatch returns.
+
+You provide your custom actions to Integreat on setup, by providing an object
+with the key set to the action type your handler will be responsible for, and
+the handler function as the value:
 
 ```javascript
 const actions = {
-  `MYACTION`: function (payload, {dispatch}) { ... }
+  `MY_ACTION`: async function myAction (action, { dispatch }) { ... }
 }
-const great = Integreat.create(defs, {schemas, services, mappings, actions})
+const great = Integreat.create(defs, { schemas, services, mappings, actions })
 ```
 
-Note that if a custom action handler is added with an action type that is
-already use by one of Integreat's built-in action handlers, the custom handler
+Note that if you set up your custom action handler with an action type that is
+already used by one of Integreat's built-in action handlers, the custom handler
 will have precedence. So be careful when you choose an action type, if your
 intention is not to replace an existing action handler.
 
-## Adapters
+## Queues
 
-Interface:
+As everything else in Integreat, a queue is also a service. You configure a
+queue service, e.g.
+[`integreat-transporter-bull`](https://github.com/integreat-io/integreat-transporter-bull),
+and set its service id on the `queueService` property of the definition object
+you give to `Integreat.create()`:
 
-- `prepareEndpoint(endpointOptions, [serviceOptions])`
-- `async send(request)`
-- `async normalize(data, request)`
-- `async serialize(data, request)`
+```javascript
+import bullQueue from `integreat-transporter-bull`
+
+const services = [
+  {
+    id: 'queue',
+    transporter: 'bull',
+    // ...
+  }
+]
+const transporters = {
+  bull: bullQueue
+}
 
-Available adapters:
+const great = Integreat.create(
+  { services, queueService: 'queue' },
+  { transporters }
+)
+```
 
-- `json` (built in)
-- [`couchdb`](https://github.com/integreat-io/integreat-adapter-couchdb)
+To queue an action instead of dispatching it right away, you set `queue: true`
+on the `meta` object. If everything is set up correctly, Integreat will push the
+action to the queue. When the action is later pulled from the queue, it will be
+dispatched again, but without the `queue` property.
 
-## Service authentication
+You may also set the meta `queue` property to a Unix timestamp, and if the queue
+transporter supports it, it will be run at this time instead of being processed
+as soon as it is next in line in the queue.
 
-This definition format is used to authenticate with a service:
+When a queue is not set up, a dispatched action with `queue: true` will just be
+run right away as a normal action.
 
-```
-{
-  id: <id>,
-  authenticator: <authenticator id>,
-  options: {
-    ...
-  }
-}
-```
+You may also use queues directly, by dispatching to it as a server and getting
+incoming actions from its `listen()` method. In that case, it's just as any
+other service with no need for any special handling.
 
-At runtime, the specified authenticator is used to authenticate requests. The
-authenticator is given the `options` payload.
+> Queueing actions are actually done through an action handler, but this handler
+> is not available from outside Integreat.
 
-## Pipeline functions
+## Middleware
 
-- Item `transform(item)`
-- Item `filter(item)`
-- Attribute `transform(value)`
+Integreat supports middleware, and there are two different middleware
+"pipelines":
 
-Built in transformers:
+- The first one is run on dispatched actions. The action goes through the
+  middleware before the action handler takes over, but after the incoming
+  mutations have been run. Because of this, given that you have set up the
+  services with mutation and casting to schemas, you should always be dealing
+  with [typed data](#typed-data) in the middleware.
+- The action then passes through a second middleware "pipeline" just before it
+  is sent to the service. This happens _after_ all mutations have been run, so
+  you will be dealing with the data as it is sent to the service. Incoming
+  actions from a service also pass through this middleware on the way in,
+  _before_ it is mutated, giving you access to the data as it comes from the
+  service.
 
-- `not` - inverts a boolean value going from or to a service
-- `hash` - converts any string(ish) value to a SHA256 hash in base64 (with the
-  url-unfriendly characters +, /, and = replaced with -, \_, and ~)
+To set up a logger of what we recieve from and send to a service, you'll use the
+second middleware "pipeline", while a logger of dispatched actions would be
+placed in the first.
 
-### Schedule definition
+When actions pass through middleware, they may modifiy the actions as
+appropriate, and you will have middleware that modifies (e.g. the
+[`completeIdent` middleware](#completeident-middleware)), and others that just
+monitors what's coming through (e.g. a logger).
 
-**Note:** This will likely be removed in version 0.8.
+Middelware is passed to Integreat like this:
 
+```javascript
+const great = Integreat.create(
+  defs,
+  resources,
+  [
+    // Dispatch middleware
+  ],
+  [
+    // Service middleware
+  ]
+)
 ```
-{
-  schedule: <schedule>,
-  action: <action definition>,
-}
+
+### `completeIdent` middleware
+
+If your access rules are based only on the information received from an
+authenticator, you don't need the following. You will always get an id and
+potentially some other fields, like roles.
+
+But when you need to match the ident id from the authenticator with user
+information held somewhere else, e.g. in a database, you need to configure a
+user schema and set up a service to fetch this information.
+
+Integreat uses schemas and services to store idents. In the definition object
+passed to `Integreat.create()`, you may provide an `identConfig` property with
+a definition object looking something like this:
+
+```javascript
+const great = Integreat.create(
+  {
+    // ...,
+    identConfig: {
+      type: 'user',
+      props: {
+        id: 'id',
+        roles: 'groups',
+        tokens: 'tokens',
+      },
+    },
+  },
+  {
+    // ...
+  }
+)
 ```
 
-The `schedule` format is directly borrowed from
-[Later](http://bunkat.github.io/later/schedules.html) (also accepts the basic or
-composite schedule formats on the `schedule` property, as well as the [text
-format](http://bunkat.github.io/later/parsers.html#text)).
-
-The following time periods are supported:
-
-- `s`: Seconds in a minute (0-59)
-- `m`: Minutes in an hour (0-59)
-- `h`: Hours in a day (0-23)
-- `t`: Time of the day, as seconds since midnight (0-86399)
-- `D`: Days of the month (1-maximum number of days in the month, 0 to specifies
-  last day of month)
-- `d`: Days of the week (1-7, starting with Sunday)
-- `dc`: Days of week count, (1-maximum weeks of the month, 0 specifies last in
-  the month). Use together with `d` to get first Wednesday every month, etc.
-- `dy`: Days of year (1 to maximum number of days in the year, 0
-  specifies last day of year).
-- `wm`: Weeks of the month (1-maximum number of weeks in the month, 0 for last
-  week of the month.). First week of the month is the week containing the 1st, and
-  weeks start on Sunday.
-- `wy`: [ISO weeks of the year](http://en.wikipedia.org/wiki/ISO_week_date)
-  (1-maximum number of ISO weeks in the year, 0 is the last ISO week of the year).
-- `M`: Months of the year (1-12)
-- `Y`: Years (1970-2099)
-
-See Later's documentation on
-[time periods](http://bunkat.github.io/later/time-periods.html) for more.
-
-Example schedule running an action at 2 am every weekday:
+- `type`: This is the id of the schema used for getting ident data. This schema
+  needs to have a `service` specified.
+- `props`: You may provide alternative field names for the `id`, `roles`, and
+  `tokens` for an ident in the schema specified on `type`. When the prop and the
+  field has the same name, it may be omitted, though it doesn't hurt to specif
+  it anyway for clarity.
+
+Note that in the example above, the `id` of the data will be used as the ident
+`id`. When the id is not suited for this, you will need another field on the
+schema that may act as the ident id. In cases where you need to transform the
+id from the data in some way, this must be set up as a separate field and the
+mutation will dictate how to transform it. In most cases, the `id` will do,
+though.
+
+For some setups, this requires certain endpoints to be defined on the service.
+To match a token with an ident, the service must have an endpoint that matches
+actions like this:
 
 ```javascript
 {
-  schedule: {d: [2,3,4,5,6], h: [2]},
-  action: {
-    type: 'SYNC',
-    payload: {
-      from: 'src1',
-      to: 'src2',
-      type: 'entry'
-    }
+  type: 'GET',
+  payload: {
+    type: 'user',
+    tokens: 'twitter|23456'
   }
 }
 ```
 
-To run an action every hour, use `{m: [0]}` or simply `'every hour'`.
+In this case, `user` is the schema mapped to idents, and the `tokens`
+property on the ident is mapped to the `tokens` field on the schema.
+
+To make Integreat complete idents on actions with the persisted data, set it up
+with the `completeIdent` middleware:
+
+```javascript
+const great = Integreat.create(defs, resources, [
+  Integreat.middleware.completeIdent,
+])
+```
+
+This middleware will intercept any action with `meta.ident` and replace it with
+the ident item loaded from the designated schema. If the ident has an `id`,
+the ident with this id is loaded, otherwise a `withToken` is used to load the
+ident with the specified token. If no ident is found, the original ident is
+kept.
 
 ## Writing middleware
 
 You may write middleware to intercept dispatched actions. This may be useful
-for logging, debugging, and features like action replay. Also, Integreat's
-queue feature is written as a middleware.
+for logging, debugging, and situations where you need to make adjustments to
+certain actions.
 
 A middleware is a function that accepts a `next()` function as only argument,
 and returns an async function that will be called with the action on dispatch.
 The returned function is expected to call `next()` with the action, and return
 the result from the `next()` function, but is not required to do so. The only
-requirement is that the functions returns a valid Integreat response object.
+requirement is that the functions returns a valid Integreat action object,
+most likely with a `response` object on it.
 
 Example implementation of a very simple logger middleware:
 
 ```javascript
 const logger = (next) => async (action) => {
   console.log('Dispatch was called with action', action)
-  const response = await next(action)
-  console.log('Dispatch completed with response', response)
-  return response
+  const responseAction = await next(action)
+  console.log('Dispatch completed with response', responseAction.response)
+  return responseAction
 }
 ```
 
-## Queue
-
-Integreat comes with a generic queue interface at `integreat.queue`, that must
-be setup with a specific queue implementation, for instance
-[`integreat-queue-redis`](https://github.com/integreat-io/integreat-queue-redis).
-
-The queue interface is a middleware, that will intercept any dispatched action
-with `action.meta.queue` set to `true` or a timestamp, and direct it to the
-queue. When the action is later pulled from the queue, it will be dispatched
-again, but without the `action.meta.queue` property.
-
-If a dispatched action has a schedule definition at `action.meta.schedule`, it
-will be queued for the next timestamp defined by the schedule.
-
-To setup Integreat with a queue:
-
-```javascript
-const queue = Integreat.queue(redisQueue(options))
-const great = Integreat.create(defs, resources, [queue.middleware])
-queue.setDispatch(great.dispatch)
-```
-
-`queue.middleware` is the middleware, while `queue.setDispatch` must be called
-to tell the queue interface where to dispatch actions pulled from the queue.
-
 ## Debugging
 
 Run Integreat with env variable `DEBUG=great`, to receive debug messages.
 
-Some sub modules sends debug messages with the `great:` prefix, so use
-`DEBUG=great,great:*` to catch these as well.
+Some sub modules sends debug messages with the `integreat:` prefix, so use
+`DEBUG=great,integreat:*` to catch these as well.