@toa.io/extensions.chronicles 1.0.0-alpha.332 → 1.0.0-alpha.335

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (82) hide show
  1. package/LICENSE +22 -0
  2. package/components/chronicles.encounters/manifest.toa.yaml +108 -0
  3. package/components/{chronicles.ingestion → chronicles.encounters}/operations/anonymous.js +1 -1
  4. package/components/chronicles.encounters/operations/describe.js +4 -0
  5. package/components/{chronicles.ingestion → chronicles.encounters}/operations/identified.js +1 -1
  6. package/components/{chronicles.ingestion → chronicles.encounters}/types/toa.d.ts +13 -0
  7. package/components/chronicles.queries/manifest.toa.yaml +39 -15
  8. package/components/chronicles.queries/operations/define.js +1 -1
  9. package/components/chronicles.queries/types/toa.d.ts +8 -3
  10. package/components/{chronicles.facts → chronicles.relay}/manifest.toa.yaml +14 -3
  11. package/components/chronicles.relay/operations/encounter.js +4 -0
  12. package/components/{chronicles.facts → chronicles.relay}/operations/record.js +1 -1
  13. package/components/{chronicles.facts → chronicles.relay}/types/toa.d.ts +5 -0
  14. package/guides/clickhouse.md +19 -0
  15. package/guides/log.md +52 -0
  16. package/package.json +8 -4
  17. package/readme.md +459 -142
  18. package/store/clickhouse.scope.sql +29 -0
  19. package/store/clickhouse.sql +99 -0
  20. package/transpiled/Client.d.ts +19 -6
  21. package/transpiled/Client.js +89 -37
  22. package/transpiled/Client.js.map +1 -1
  23. package/transpiled/Factory.d.ts +13 -5
  24. package/transpiled/Factory.js +32 -23
  25. package/transpiled/Factory.js.map +1 -1
  26. package/transpiled/Provider.d.ts +76 -25
  27. package/transpiled/Provider.js.map +1 -1
  28. package/transpiled/Publisher.d.ts +50 -0
  29. package/transpiled/Publisher.js +169 -0
  30. package/transpiled/Publisher.js.map +1 -0
  31. package/transpiled/Reader.d.ts +15 -3
  32. package/transpiled/Reader.js +98 -14
  33. package/transpiled/Reader.js.map +1 -1
  34. package/transpiled/Recorder.d.ts +21 -11
  35. package/transpiled/Recorder.js +49 -24
  36. package/transpiled/Recorder.js.map +1 -1
  37. package/transpiled/Relay.d.ts +14 -0
  38. package/transpiled/Relay.js +27 -0
  39. package/transpiled/Relay.js.map +1 -0
  40. package/transpiled/Sender.d.ts +8 -4
  41. package/transpiled/Sender.js +27 -10
  42. package/transpiled/Sender.js.map +1 -1
  43. package/transpiled/Service.d.ts +26 -0
  44. package/transpiled/Service.js +70 -0
  45. package/transpiled/Service.js.map +1 -0
  46. package/transpiled/agent.d.ts +16 -0
  47. package/transpiled/agent.js +48 -0
  48. package/transpiled/agent.js.map +1 -0
  49. package/transpiled/clickhouse/Client.d.ts +5 -8
  50. package/transpiled/clickhouse/Client.js +12 -14
  51. package/transpiled/clickhouse/Client.js.map +1 -1
  52. package/transpiled/clickhouse/Provider.d.ts +20 -18
  53. package/transpiled/clickhouse/Provider.js +97 -72
  54. package/transpiled/clickhouse/Provider.js.map +1 -1
  55. package/transpiled/clickhouse/ddl.d.ts +3 -20
  56. package/transpiled/clickhouse/ddl.js +6 -79
  57. package/transpiled/clickhouse/ddl.js.map +1 -1
  58. package/transpiled/clickhouse/views.d.ts +11 -0
  59. package/transpiled/clickhouse/views.js +97 -0
  60. package/transpiled/clickhouse/views.js.map +1 -0
  61. package/transpiled/contract.d.ts +12 -0
  62. package/transpiled/contract.js +58 -0
  63. package/transpiled/contract.js.map +1 -0
  64. package/transpiled/index.d.ts +9 -3
  65. package/transpiled/settings.d.ts +6 -4
  66. package/transpiled/settings.js +27 -14
  67. package/transpiled/settings.js.map +1 -1
  68. package/transpiled/store.d.ts +3 -3
  69. package/transpiled/store.js +3 -3
  70. package/transpiled/store.js.map +1 -1
  71. package/components/chronicles.ingestion/manifest.toa.yaml +0 -60
  72. package/transpiled/Facts.d.ts +0 -9
  73. package/transpiled/Facts.js +0 -16
  74. package/transpiled/Facts.js.map +0 -1
  75. package/transpiled/Scribe.d.ts +0 -25
  76. package/transpiled/Scribe.js +0 -61
  77. package/transpiled/Scribe.js.map +0 -1
  78. package/transpiled/Writer.d.ts +0 -14
  79. package/transpiled/Writer.js +0 -50
  80. package/transpiled/Writer.js.map +0 -1
  81. /package/components/{chronicles.facts → chronicles.encounters}/types/index.d.ts +0 -0
  82. /package/components/{chronicles.ingestion → chronicles.relay}/types/index.d.ts +0 -0
package/readme.md CHANGED
@@ -1,81 +1,171 @@
1
1
  # Toa Chronicles
2
2
 
3
- A log of what happens in a context, kept in a store the context names. Components write to it by
4
- the events they name and the facts they send, browsers by a route of the gateway, and it is read
5
- by an [HTTP API](#reading-the-log) — which the gateway also publishes as [MCP](#over-mcp) tools.
6
- The service `chronicles-scribe` writes the log, and is what the store is provisioned by and read
7
- through.
3
+ A log of what happens in a context. Components write to it by the events they declare and the
4
+ facts they send, clients by a route of the gateway; the service `chronicles-relay` publishes every
5
+ row of it to an analytics broker, and reads it back from a store, by an
6
+ [HTTP API](#reading-the-log) — which the gateway also publishes as [MCP](#over-mcp) tools.
8
7
 
9
- ## The log
10
-
11
- Two tables.
8
+ Writing and reading meet in nothing Toa does: what takes the rows off the broker into the store,
9
+ and who may read what there, is the [store's](#the-store) own setup.
12
10
 
13
- **`events`** holds a row for everything that happened, from three sources:
11
+ ## The log
14
12
 
15
- | source | `label` | `entity`, `version` | `payload` |
16
- | ------------------------------------------------ | --------------------- | ---------------------------------------- | ---------------------- |
17
- | an [event](#events-and-states) a component names | `<component>.<event>` | the entity that changed, and its version | what the event renders |
18
- | a [fact](#facts) an operation sends | `<component>.<name>` | `NULL` | what was sent |
19
- | an event a [browser](#from-a-browser) sends | `agent.<name>` | `NULL` | what the browser sent |
13
+ Three tables, each row of which has the `scope` of the context that wrote it — the context's name,
14
+ followed by `TOA_SUFFIX` where one is given — and a reader sees the rows of its own scope.
20
15
 
21
- An order placed, a search made and a page left are three rows:
16
+ **`events`** holds a row for everything a component did: an [event](#events-and-states) it
17
+ declares, under `<component>.<event>`, with the entity that changed and its version; and a
18
+ [fact](#facts) it sends, under `<component>.<name>`, with neither.
22
19
 
23
- | `label` | `component` | `entity` | `version` | `payload` |
24
- | ----------------------- | ---------------------- | -------- | --------- | --------------------------------- |
25
- | `store.orders.created` | `store.orders` | `9c1f…` | `0` | `{ total: 120, status: "new" }` |
26
- | `store.search.searched` | `store.search` | `NULL` | `NULL` | `{ query: "teapot", results: 3 }` |
27
- | `agent.page_left` | `chronicles.ingestion` | `NULL` | `NULL` | `{ path: "/pricing" }` |
20
+ | `label` | `component` | `entity` | `version` | `payload` |
21
+ | ----------------------- | -------------- | -------- | --------- | --------------------------------- |
22
+ | `store.orders.created` | `store.orders` | `9c1f…` | `0` | `{ total: 120, status: "new" }` |
23
+ | `store.search.searched` | `store.search` | `NULL` | `NULL` | `{ query: "teapot", results: 3 }` |
28
24
 
29
25
  Every row also has `id`, the same however often it is delivered, so a row is kept once;
30
- `identity`, whose it is, where something [says so](#identity); `ip` and `country`, of the
31
- browser that [sent it](#from-a-browser); `time`, when it happened; `received`, when the store wrote
32
- it; and `trace`, the traceparent of the call that caused it.
26
+ `identity`, whose it is, where something [says so](#identity); `time`, when it
27
+ [happened](#time); and `trace`, the traceparent of the call that caused it.
33
28
 
34
- **`states`** holds each entity as it stands: for every component that names a
29
+ **`states`** holds each entity as it stands: for every component that declares a
35
30
  [state](#events-and-states), a row per entity — `component`, `entity`, `version`, `identity`,
36
- `time`, `received` and `payload` — which a higher version replaces.
31
+ `time` and `payload` — which a higher version replaces. It is the current state and
32
+ nothing before it: what an entity was yesterday is read from its events, so an event a question
33
+ of history depends on has to carry what that question asks.
34
+
35
+ **`encounters`** holds what [clients send](#encounters): `id`, `name`, `time`, `received`,
36
+ `visitor`, `session`, `identity`, `ip`, `country`, `origin`, `agent`, `browser`, `os`, `device` and
37
+ `payload`.
37
38
 
38
39
  Nothing is ordered: order by `time`, or by `version` within one entity.
39
40
 
41
+ ### Time
42
+
43
+ Every time is the one its source says — when it happened — and none is when the store wrote it:
44
+
45
+ | | `time` |
46
+ | --------- | ----------------------------------------------------------------------------------------- |
47
+ | an event | the payload's [timestamp](#when-a-change-was-made), or when the component made the change |
48
+ | a state | likewise |
49
+ | a fact | when the operation sent it |
50
+ | encounter | when the gateway received it, less how long before sending it the client says it happened |
51
+
52
+ So a change the relay takes an hour late is written at the time it was made, and the hours it
53
+ was away fill in as it catches up: a count of a recent period may still grow. An encounter's
54
+ `received` is when the gateway received its batch.
55
+
40
56
  ## Events and states
41
57
 
58
+ A component declares what it logs, and nothing else of it is written:
59
+
42
60
  ```yaml
43
61
  # manifest.toa.yaml
44
62
  chronicles:
45
- events: [created, paid] # what happened: rows of `events`
46
- state: snapshot # the entity as it stands: its row of `states`
63
+ events:
64
+ paid:
65
+ description: An order's payment completed
66
+ properties:
67
+ total:
68
+ type: integer
69
+ description: The amount paid, in cents
70
+ shipping:
71
+ type: object
72
+ description: Where the order goes
73
+ properties:
74
+ country:
75
+ type: string
76
+ description: ISO 3166 alpha-2
77
+ state:
78
+ event: snapshot
79
+ description: An order as it stands
80
+ properties:
81
+ status:
82
+ type: string
83
+ enum: [new, paid, shipped]
84
+ description: Where the order is in its life
47
85
  ```
48
86
 
49
- Each label is an event of the component — its own, or the prototype's `created`, `updated`,
50
- `deleted` and `sync`. An event is written as it renders: where its `condition` does not hold,
51
- nothing is written; where it has a `payload`, what the payload returns is what is written. A field
52
- a payload leaves out does not reach the store. A list is the `events` alone:
87
+ Each label under `events` is an event of the component — its own, or the prototype's `created`,
88
+ `updated`, `deleted` and `sync` — and `state.event` is the one whose payload is the entity. An
89
+ event is written as it renders: where its `condition` does not hold, nothing is written; where it
90
+ has a `payload`, what the payload returns is what is written. An event that exists for the log
91
+ alone is an event file like any other, and nothing else has to consume it.
92
+
93
+ ### The contract
94
+
95
+ `description` says what the label means, and `properties` what its payload holds, as
96
+ [JSON Schema](https://json-schema.org) of the types [below](#types). Every property, at every
97
+ depth, has a `description`; a declaration that leaves one out, or uses what has no type there, is
98
+ refused when the component is read.
99
+
100
+ What is written is the declared properties of the payload, at every depth, and nothing else: a
101
+ property the declaration leaves out — an email, a document, a field added to the entity since —
102
+ does not reach the store. A payload whose declared property is of another type is not written,
103
+ and is kept in the parking queue of its consumer.
104
+
105
+ A state's payload carries `id` and `VERSION`, which say which entity it is and which version: one
106
+ that does not is refused, and parked likewise. A row of `events` takes its `entity` and `version`
107
+ from the payload's `id` and `VERSION` where it carries them, and is written without them where it
108
+ does not. These, the [timestamp](#when-a-change-was-made) and the [identity](#identity) are read
109
+ before the contract applies, and are not declared. An event that is both is named in both.
110
+
111
+ A committed change is written however long the relay or the broker is away, and once however often it is
112
+ delivered. One committed before the relay first ran in a deployment is not.
113
+
114
+ ### Types
115
+
116
+ | declared | read as |
117
+ | ---------------------------------------- | ------------------------ |
118
+ | `string` | `String` |
119
+ | `string` with `enum` | `LowCardinality(String)` |
120
+ | `integer` | `Int64` |
121
+ | `number` | `Float64` |
122
+ | `boolean` | `Bool` |
123
+ | `string`, `format: date-time` | `DateTime64(3, 'UTC')` |
124
+ | `string`, `format: date` | `Date32` |
125
+ | `string`, `format: time` | `Time64(3)` |
126
+ | `string`, `format: uuid`, `ipv4`, `ipv6` | `UUID`, `IPv4`, `IPv6` |
127
+ | `integer`, `format: int32` | `Int32` |
128
+ | `number`, `format: float` | `Float32` |
129
+ | any other `format` | as its `type` |
130
+ | `object` with `properties` | `Tuple` of them, by name |
131
+ | `array` with `items` of any of these | `Array` of it |
132
+
133
+ ### Reading what is declared
134
+
135
+ `describe` tells, of every declared field of a label or a state, its type and how to read it from
136
+ the table in that type:
53
137
 
54
138
  ```yaml
55
- # manifest.toa.yaml
56
- chronicles: [created, paid]
139
+ - label: store.orders.paid
140
+ fields:
141
+ - path: total
142
+ type: Int64
143
+ description: The amount paid, in cents
144
+ read: JSONExtract(ifNull(toJSONString(payload.total), 'null'), 'Nullable(Int64)')
57
145
  ```
58
146
 
59
- The state is any event whose payload is the entity, `sync` among them. An event that exists for
60
- the log alone is an event file like any other, and nothing else has to consume it:
147
+ ```sql
148
+ SELECT JSONExtract(ifNull(toJSONString(payload.total), 'null'), 'Nullable(Int64)') AS total, count()
149
+ FROM events FINAL WHERE label = 'store.orders.paid' GROUP BY total
150
+ ```
61
151
 
62
- ```typescript
63
- // events/snapshot.ts
64
- export function payload(event: Event) {
65
- const { email, address, ...rest } = event.state
152
+ A value that is not of its type reads as `NULL`; the row keeps it as it was written. A contract
153
+ that changes changes what `describe` tells, and the rows written before it are read the same way.
66
154
 
67
- return rest
68
- }
69
- ```
155
+ ### When a change was made
70
156
 
71
- A state's payload carries `id` and `VERSION`, which say which entity it is and which version: one
72
- that does not is refused, and kept in the parking queue of its consumer rather than written. A row
73
- of `events` takes its `entity`, `version` and `time` from the payload's `id`, `VERSION` and
74
- `UPDATED` where it carries them, and is written without them, at the time it was taken, where it
75
- does not. An event that is both is named in both.
157
+ `time` is the payload's `UPDATED`, which the entity as it stands carries, where it is a number,
158
+ and where it is not, the moment the component made the change, which every event carries:
76
159
 
77
- A committed change is written however long the scribe is away, and once however often it is
78
- delivered. One committed before the scribe first ran in a deployment is not.
160
+ ```yaml
161
+ # manifest.toa.yaml
162
+ chronicles:
163
+ timestamp: placed # the property of the payloads that says when, in milliseconds
164
+ events: …
165
+ ```
166
+
167
+ A component names another property, or none with `timestamp: ~`, when the moment of the change
168
+ is always the time. One property serves every event of the component and its state.
79
169
 
80
170
  ### Identity
81
171
 
@@ -84,9 +174,8 @@ delivered. One committed before the scribe first ran in a deployment is not.
84
174
  ```yaml
85
175
  # manifest.toa.yaml
86
176
  chronicles:
87
- events: [created, paid]
88
- state: snapshot
89
177
  identity: customer # the property of the payloads that names whose a row is
178
+ events: …
90
179
  ```
91
180
 
92
181
  A component names another property, or none with `identity: ~`. One property serves every event of
@@ -95,12 +184,29 @@ the component names is one of its entity.
95
184
 
96
185
  ## Facts
97
186
 
187
+ ```yaml
188
+ # manifest.toa.yaml
189
+ chronicles:
190
+ facts:
191
+ searched:
192
+ description: A customer searched the catalogue
193
+ properties:
194
+ query:
195
+ type: string
196
+ description: What was typed
197
+ results:
198
+ type: integer
199
+ description: How many products were found
200
+ ```
201
+
98
202
  ```typescript
99
203
  await context.chronicles.send('searched', { query: input.query, results: results.length })
100
204
  ```
101
205
 
102
- The fact is a row of `events` under `<component>.searched`, with what was sent as its payload.
103
- `send` resolves once the broker has it, and it then waits for the scribe as an event does.
206
+ The fact is a row of `events` under `<component>.searched`, with what was sent of its contract as
207
+ its payload. `send` throws for a name the component does not declare, or a payload a declared
208
+ property of which is of another type; it resolves once the broker has it, and it then waits for
209
+ the relay as an event does.
104
210
 
105
211
  An operation that knows whose the fact is says so; the extension does not know who called it:
106
212
 
@@ -112,28 +218,29 @@ await context.chronicles.send(
112
218
  )
113
219
  ```
114
220
 
115
- `context.chronicles` is given to a component that declares `chronicles`. One that only sends
116
- declares it empty:
117
-
118
- ```yaml
119
- # manifest.toa.yaml
120
- chronicles: []
121
- ```
221
+ `context.chronicles` is given to a component that declares `chronicles`.
122
222
 
123
- ## From a browser
223
+ ## Encounters
124
224
 
125
- Served where the context says `ingest: true`, at two routes: one of an identity, and one of no one.
225
+ What clients send: pages viewed, buttons pressed, anything a client tells of itself. Served where
226
+ the context says `encounters: true`, at two routes: one of an identity, and one of no one. Who may
227
+ send to which is in [Access](#access).
126
228
 
127
- ### `POST /chronicles/ingestion/:identity/`
229
+ ### `POST /chronicles/encounters/:identity/`
128
230
 
129
231
  What a signed-in client sends, as the identity it is.
130
232
 
131
233
  ```http
132
- POST /chronicles/ingestion/9c1f0a6d5b7e4f3a8d2c1b0a9e8f7d6c/ HTTP/1.1
234
+ POST /chronicles/encounters/9c1f0a6d5b7e4f3a8d2c1b0a9e8f7d6c/ HTTP/1.1
133
235
  authorization: Token …
134
236
  content-type: application/json
135
237
 
136
- { "events": [{ "id": "0192f3a1-…", "name": "page_viewed", "time": 1761200000000, "payload": { "path": "/pricing" } }] }
238
+ {
239
+ "sent": 1761200004000,
240
+ "visitor": "7f3a…",
241
+ "session": "01c2…",
242
+ "events": [{ "id": "0192f3a1-…", "name": "page_viewed", "time": 1761200000000, "payload": { "path": "/pricing" } }]
243
+ }
137
244
  ```
138
245
 
139
246
  ```http
@@ -142,67 +249,143 @@ content-type: application/json
142
249
  accepted: 1
143
250
  ```
144
251
 
145
- The token is the identity's, or the request is answered `403`: the rows are that identity's,
146
- whatever the body says. Throttled to 60 requests a minute per identity.
252
+ The rows are that identity's, whatever the body says.
147
253
 
148
- ### `POST /chronicles/ingestion/`
254
+ ### `POST /chronicles/encounters/`
149
255
 
150
256
  What anyone sends, before signing in or without it. The same body, and rows of no identity; a body
151
257
  that names one is refused.
152
258
 
153
- ```http
154
- POST /chronicles/ingestion/ HTTP/1.1
155
- content-type: application/json
156
-
157
- { "events": [{ "id": "0192f3a2-…", "name": "page_left", "time": 1761200000000 }] }
158
- ```
159
-
160
- It takes no credentials, and is throttled to 60 requests a minute per address. A form whose
161
- `events` is their JSON is read as the JSON body, so `navigator.sendBeacon`, which sends no
162
- `authorization`, can send a batch here as the page closes:
259
+ A form whose `events` is their JSON, beside `sent`, `visitor` and `session`, is read as the JSON
260
+ body, so `navigator.sendBeacon`, which sends no `authorization`, can send a batch here as the page
261
+ closes:
163
262
 
164
263
  ```javascript
165
264
  navigator.sendBeacon(
166
- '/chronicles/ingestion/',
167
- new URLSearchParams({ events: JSON.stringify(events) })
265
+ '/chronicles/encounters/',
266
+ new URLSearchParams({
267
+ sent: Date.now(),
268
+ visitor,
269
+ session,
270
+ events: JSON.stringify(events)
271
+ })
168
272
  )
169
273
  ```
170
274
 
171
- A signed-in client sends with `fetch(…, { keepalive: true })` instead. Anyone can send anything: a
172
- `user` in a payload is what the client said.
275
+ A signed-in client sends with `fetch(…, { keepalive: true })` instead.
173
276
 
174
277
  ### The body
175
278
 
176
- `{ events }`, at most 100 of them, each a row of `events` under `agent.<name>`:
279
+ `{ sent, visitor?, session?, events }`:
280
+
281
+ | property | |
282
+ | --------- | ---------------------------------------------------------------------------------------------- |
283
+ | `sent` | when the client sends the batch, in milliseconds, by its clock |
284
+ | `visitor` | the client's own id of itself, kept across sessions; at most 64 characters, or `null` for none |
285
+ | `session` | the id of one period of the client's activity; at most 64 characters, or `null` for none |
286
+ | `events` | at most 100 of them |
287
+
288
+ Each event:
177
289
 
178
290
  | property | |
179
291
  | --------- | ------------------------------------------------------------------- |
180
292
  | `id` | the client's own, at most 64 characters; a retry sends the same one |
181
293
  | `name` | lowercase letters, digits and underscores from a letter, at most 64 |
182
- | `time` | when it happened, in milliseconds, by the client's clock |
294
+ | `time` | when it happened, in milliseconds, by the client's clock, by `sent` |
183
295
  | `payload` | an object, at most 16 KiB of JSON |
184
296
 
297
+ The client keeps its `visitor` — in `localStorage`, say — and sends the same one before signing
298
+ in and after. The rows of the route of an identity have both, so the visitors of an identity are
299
+ a query. A visitor is not a person: one identity may have many, and a browser that clears its
300
+ storage is a new one.
301
+
185
302
  A request is answered once its events are written. A batch one event of which breaks any of the
186
303
  above is written not at all, and answered `422` with the code `INVALID`, or `OVERSIZED` for a
187
304
  payload too large, and the `index` of the event in `cause`.
188
305
 
306
+ A payload is written as it was sent, whatever its shape: a client is deployed once and runs for
307
+ months, so a shape that changes is a new `name` — `checkout_started_v2` — and the old one stays as
308
+ old clients send it.
309
+
310
+ A client's clock is its own, and may be off by anything; what it measures between two of its
311
+ moments is not. So an encounter's `time` is when the gateway received the batch, less `sent`
312
+ minus its own `time`: the gateway's clock, at the distance the client measured. A clock that is
313
+ wrong moves nothing, and a batch kept offline for an hour is written an hour back. A row is kept
314
+ and expires by `received`.
315
+
189
316
  ### Address and country
190
317
 
191
- Every row a browser sent has the `ip` and the `country` of the request, as the gateway reads them
192
- from the headers [`exposition.ip` and `exposition.country`](/extensions/exposition/documentation/ip.md)
318
+ Every encounter has the `ip` and the `country` of the request, as the gateway reads them from the
319
+ headers [`exposition.ip` and `exposition.country`](/extensions/exposition/documentation/ip.md)
193
320
  name, and `NULL` where it has none; a body that names either is overwritten. An address is
194
- personal data, and the log erases nothing, so of an address the log keeps its network — a `/24` of
195
- IPv4, a `/48` of IPv6 — unless the annotation says `ip: full`.
321
+ personal data, and the log erases nothing, so of an address the log keeps its network — a `/24`
322
+ of IPv4, a `/48` of IPv6 — unless the annotation says `ip: full`.
323
+
324
+ ### Origin and agent
325
+
326
+ Every encounter has the `origin` of the page that sent it and the client's `agent`, from the
327
+ request's `Origin` and `User-Agent` headers, and `NULL` where it has none; a body that names either
328
+ is overwritten, and a body cannot say one the request does not have. A browser sends `Origin` with
329
+ a request to another origin, and with every `POST`, so it tells which of the context's sites a
330
+ batch came from.
331
+
332
+ What the agent says the client is, is three columns of its own, by name and without versions:
333
+
334
+ | | |
335
+ | --------- | ----------------------------------------------------------------------------- |
336
+ | `browser` | `Edge`, `Opera`, `Samsung Internet`, `Firefox`, `Chrome`, `Safari`, or `NULL` |
337
+ | `os` | `Windows`, `iOS`, `macOS`, `Android`, `ChromeOS`, `Linux`, or `NULL` |
338
+ | `device` | `bot`, `tablet`, `mobile` or `desktop`, and `NULL` where there is no agent |
339
+
340
+ `bot` is a crawler, a headless browser or a program — what a question of who visits leaves out.
341
+ Safari on an iPad says it is a Mac, and is a `desktop` running `macOS` here. A version is read from
342
+ `agent` where a question needs one, knowing that browsers freeze what they say of the system:
343
+ Safari on iOS 26 says iOS 18, and Chrome on Android says Android 10.
344
+
345
+ ### What the encounters are
346
+
347
+ What a client sends is told in the configuration of `chronicles.encounters`, by name:
348
+
349
+ ```yaml
350
+ # context.toa.yaml
351
+ configuration:
352
+ chronicles.encounters:
353
+ page_viewed:
354
+ description: A page was shown
355
+ fields:
356
+ path: The URL path, without the query
357
+ referrer: The page the visitor came from
358
+ ```
359
+
360
+ It is what `describe` tells an agent of each encounter's fields, and what whoever writes a client
361
+ reads:
362
+
363
+ ```http
364
+ GET /chronicles/encounters/ HTTP/1.1
365
+ authorization: Token …
366
+ accept: application/yaml
367
+ ```
368
+
369
+ ```http
370
+ 200 OK
371
+
372
+ page_viewed:
373
+ description: A page was shown
374
+ fields:
375
+ path: The URL path, without the query
376
+ referrer: The page the visitor came from
377
+ ```
378
+
379
+ It documents, and holds nothing back: a name it does not tell is written as any other.
196
380
 
197
381
  ## Reading the log
198
382
 
199
- Reading takes the role `system:chronicles:query`, and views take `system:chronicles:views`;
200
- `system:chronicles` grants both. A refusal is answered `422` with `{ code, cause }`.
383
+ A refusal is answered `422` with `{ code, cause }`. Who may read is in [Access](#access).
201
384
 
202
385
  ### `GET /chronicles/queries/`
203
386
 
204
- What the log holds: the dialect queries are written in, the labels of `events`, the components
205
- with a state, and the views.
387
+ What the log holds: a guide to reading it, the labels of `events`, the components
388
+ with a state, the encounters, and the views.
206
389
 
207
390
  ```http
208
391
  GET /chronicles/queries/ HTTP/1.1
@@ -213,29 +396,81 @@ accept: application/yaml
213
396
  ```http
214
397
  200 OK
215
398
 
216
- dialect: ClickHouse SQL; read `events` and `states` with FINAL
399
+ guide: |
400
+ # Reading the log
401
+
402
+ The log is three tables, read with `FINAL`.
403
+ …
217
404
  labels:
218
- - label: store.orders.created
405
+ - label: store.orders.paid
406
+ description: An order's payment completed
219
407
  rows: 1204
220
408
  first: 2026-10-01 08:12:03.000
221
409
  last: 2026-10-24 17:40:51.000
222
- paths: [VERSION, id, status, total]
410
+ fields:
411
+ - path: total
412
+ type: Int64
413
+ description: The amount paid, in cents
414
+ read: JSONExtract(ifNull(toJSONString(payload.total), 'null'), 'Nullable(Int64)')
415
+ - path: shipping
416
+ type: Tuple(country String)
417
+ description: Where the order goes
418
+ read: JSONExtract(toJSONString(payload.^shipping), 'Tuple(country Nullable(String))')
419
+ - path: shipping.country
420
+ type: String
421
+ description: ISO 3166 alpha-2
223
422
  states:
224
423
  - component: store.orders
424
+ description: An order as it stands
225
425
  entities: 1180
226
- paths: [VERSION, id, status, total]
426
+ last: 2026-10-24 17:40:51.000
427
+ fields:
428
+ - path: status
429
+ type: LowCardinality(String)
430
+ description: Where the order is in its life
431
+ read: JSONExtract(ifNull(toJSONString(payload.status), 'null'), 'LowCardinality(Nullable(String))')
432
+ encounters:
433
+ - name: page_viewed
434
+ description: A page was shown
435
+ rows: 88123
436
+ first: 2026-10-01 00:00:14.000
437
+ last: 2026-10-24 17:41:02.000
438
+ received: 2026-10-24 17:41:03.020
439
+ fields:
440
+ - path: path
441
+ types: [String]
442
+ description: The URL path, without the query
443
+ - path: utm
444
+ types: [Int64, String]
227
445
  views:
228
- - name: daily
229
- sql: SELECT toDate(time) AS day, count() AS orders FROM events FINAL GROUP BY day
446
+ - name: paid_daily
447
+ read: shop.paid_daily
448
+ title: Orders paid per day
449
+ description: Orders whose payment completed, counted by the day of `time`
450
+ sql: SELECT toDate(time) AS day, count() AS paid FROM events FINAL WHERE label = 'store.orders.paid' GROUP BY day
230
451
  refresh: 3600
452
+ refreshed: 2026-10-24 17:00:03
231
453
  ```
232
454
 
233
- `rows` counts what the store holds, a row delivered twice possibly twice; `paths` are the fields of
234
- the latest payloads.
455
+ `guide` is how to read the log — what its tables and columns mean, and what to be careful of —
456
+ and how its store is read, in the store's dialect: what an agent reads before its first query.
457
+ Toa writes it; what is particular to an application is in the descriptions of what it declares
458
+ and documents, and in [`exposition.mcp.instructions`](/extensions/exposition/documentation/mcp.md).
459
+
460
+ A label and a state are told by their declaration, with each field's type and how to
461
+ [read](#reading-what-is-declared) it in that type; a field of a field is read inside the one that
462
+ holds it. A label declared and not yet written is told with no rows. An encounter is
463
+ told by what was sent: the fields of the latest payloads, the types they were sent as — two
464
+ types of one field are two versions of a client — and the description the
465
+ [configuration](#what-the-encounters-are) gives.
466
+
467
+ `rows` counts what the store holds, a row delivered twice possibly twice. `first` and `last` are
468
+ by `time`, the latest change of a state too; an encounter's `received` is the newest batch the
469
+ gateway received, which tells whether clients still send.
235
470
 
236
471
  ### `POST /chronicles/queries/`
237
472
 
238
- One statement, in the dialect.
473
+ One statement, in the store's dialect, as the `guide` says.
239
474
 
240
475
  ```http
241
476
  POST /chronicles/queries/ HTTP/1.1
@@ -243,7 +478,7 @@ authorization: Token …
243
478
  accept: application/yaml
244
479
  content-type: application/yaml
245
480
 
246
- sql: SELECT payload.status AS status, count() AS n FROM states FINAL GROUP BY status
481
+ sql: SELECT status, count() AS n FROM store_orders GROUP BY status
247
482
  ```
248
483
 
249
484
  ```http
@@ -251,7 +486,7 @@ sql: SELECT payload.status AS status, count() AS n FROM states FINAL GROUP BY st
251
486
 
252
487
  columns:
253
488
  - name: status
254
- type: Dynamic
489
+ type: Nullable(String)
255
490
  - name: n
256
491
  type: UInt64
257
492
  rows:
@@ -260,24 +495,26 @@ rows:
260
495
  truncated: false
261
496
  ```
262
497
 
263
- A query reads the log and nothing else, and cannot write. It runs for at most 30 seconds and
264
- answers at most 10,000 rows: at the rows it answers what it has, with `truncated: true`. It leaves
265
- `FORMAT` out.
498
+ A query reads the log and nothing else, and cannot write. It runs for at most 30 seconds, answers
499
+ at most 10,000 rows and 64 MiB, and a reader runs at most 4 queries at once: past the rows it
500
+ answers what it has, with `truncated: true`; past the rest it is refused. It leaves `FORMAT` out.
266
501
 
267
502
  | code | when |
268
503
  | --------- | ----------------------------------------------------------------------- |
269
504
  | `REFUSED` | the store refused the statement; `cause.message` is what the store said |
270
505
 
271
- ### `PUT /chronicles/queries/views/:name`
506
+ ### `PUT /chronicles/queries/views/:name/`
272
507
 
273
508
  A view over the log, by a statement a query could run. Defining a name again replaces the view.
274
509
 
275
510
  ```http
276
- PUT /chronicles/queries/views/daily/ HTTP/1.1
511
+ PUT /chronicles/queries/views/paid_daily/ HTTP/1.1
277
512
  authorization: Token …
278
513
  content-type: application/yaml
279
514
 
280
- sql: SELECT toDate(time) AS day, count() AS orders FROM events FINAL GROUP BY day
515
+ title: Orders paid per day
516
+ description: Orders whose payment completed, counted by the day of `time`
517
+ sql: SELECT toDate(time) AS day, count() AS paid FROM events FINAL WHERE label = 'store.orders.paid' GROUP BY day
281
518
  refresh: 3600
282
519
  ```
283
520
 
@@ -285,9 +522,13 @@ refresh: 3600
285
522
  204 No Content
286
523
  ```
287
524
 
288
- Without `refresh`, a view is computed when it is read. With `refresh`, in seconds and at least
289
- `3600`, it is computed on that cycle and read as a table. A name is lowercase letters, digits and
290
- underscores, and a context has at most 50 views. A query reads a view by its name.
525
+ `title`, at most 100 characters, and `description` tell an agent that finds the view what it
526
+ counts. Without `refresh`, a view is computed when it is read. With `refresh`, in seconds and at
527
+ least `3600`, it is computed on that cycle and read as a table, and `describe` tells when it last
528
+ was. A name is lowercase letters, digits and underscores, and a context has at most 50 views. A
529
+ view is kept in the database named after the context's scope, and a query reads it by that name:
530
+ `shop.paid_daily`. Defining one takes a store whose user may create views there; one that may not
531
+ is refused.
291
532
 
292
533
  | code | when |
293
534
  | ----------- | -------------------------------------------------------------------- |
@@ -295,10 +536,10 @@ underscores, and a context has at most 50 views. A query reads a view by its nam
295
536
  | `RESERVED` | the name is one of the log's tables |
296
537
  | `EXHAUSTED` | the context has 50 views, and this is not one of them |
297
538
 
298
- ### `DELETE /chronicles/queries/views/:name`
539
+ ### `DELETE /chronicles/queries/views/:name/`
299
540
 
300
541
  ```http
301
- DELETE /chronicles/queries/views/daily/ HTTP/1.1
542
+ DELETE /chronicles/queries/views/paid_daily/ HTTP/1.1
302
543
  authorization: Token …
303
544
  ```
304
545
 
@@ -315,55 +556,131 @@ authorization: Token …
315
556
  Where the context publishes [MCP](/extensions/exposition/documentation/mcp.md), each of these
316
557
  routes is a tool, under the name the gateway gives a route, and taking what the route takes:
317
558
 
318
- | tool | route |
319
- | --------------------------------------- | ---------------------------------------- |
320
- | `chronicles.queries.GET` | `GET /chronicles/queries/` |
321
- | `chronicles.queries.POST` | `POST /chronicles/queries/` |
322
- | `chronicles.queries.views._name.PUT` | `PUT /chronicles/queries/views/:name` |
323
- | `chronicles.queries.views._name.DELETE` | `DELETE /chronicles/queries/views/:name` |
559
+ | tool | route |
560
+ | --------------------------------------- | ----------------------------------------- |
561
+ | `chronicles.queries.GET` | `GET /chronicles/queries/` |
562
+ | `chronicles.queries.POST` | `POST /chronicles/queries/` |
563
+ | `chronicles.queries.views._name.PUT` | `PUT /chronicles/queries/views/:name/` |
564
+ | `chronicles.queries.views._name.DELETE` | `DELETE /chronicles/queries/views/:name/` |
565
+ | `chronicles.encounters.GET` | `GET /chronicles/encounters/` |
566
+
567
+ A refusal is a tool result with `isError`.
568
+
569
+ ## Access
570
+
571
+ | route | who may call | limit |
572
+ | ----------------------------------------- | ------------------------------------ | --------------------------- |
573
+ | `POST /chronicles/encounters/` | anyone, with no credentials | 60 a minute per address |
574
+ | `POST /chronicles/encounters/:identity/` | the identity, by its own credentials | 60 a minute per identity |
575
+ | `GET /chronicles/encounters/` | `system:chronicles:encounters` | |
576
+ | `GET /chronicles/queries/` | `system:chronicles:query` | |
577
+ | `POST /chronicles/queries/` | `system:chronicles:query` | the [reader's](#the-reader) |
578
+ | `PUT /chronicles/queries/views/:name/` | `system:chronicles:views` | 50 views |
579
+ | `DELETE /chronicles/queries/views/:name/` | `system:chronicles:views` | |
580
+
581
+ A caller without what a route takes is answered `403`, and past its limit `429`. Over
582
+ [MCP](#over-mcp), a tool is listed to a caller that holds its role, and to no one else.
583
+
584
+ ### Roles
324
585
 
325
- A tool is listed to a caller that holds its role, and a refusal is a tool result with `isError`.
586
+ | | |
587
+ | ------------------------------ | ------------------------------------------------------------------------------------------- |
588
+ | `system:chronicles:query` | reads every row of the log: the identities, the networks, the origins and agents of clients |
589
+ | `system:chronicles:views` | defines and drops the views every reader reads; a view can read only what a query can |
590
+ | `system:chronicles:encounters` | reads what the configuration documents of encounters, and no row |
591
+ | `system:chronicles` | all three |
592
+
593
+ `query` is access to personal data, and is granted as that. `views` changes what other readers are
594
+ told a view counts, and changes nothing of the log. `encounters` is for whoever writes a client.
595
+ A role is granted as any [role](/extensions/exposition/documentation/access.md) is.
596
+
597
+ ### What a client cannot say
598
+
599
+ A client sends events and nothing that says who or where it is: `identity` is the path's, which the
600
+ credentials have to be the identity's; `ip`, `country`, `origin` and `agent` are what the gateway
601
+ read off the request, and a body that names one is overwritten, or refused where it names
602
+ `identity` on the route of no one. Anything else in a payload is what the client said — a `user`
603
+ there is not a user — so a question that depends on who sent an encounter asks `identity`.
604
+
605
+ ### The reader
606
+
607
+ A query runs as the context's user of the store, which reads what the [store](#the-store) lets it
608
+ — in the reference setup, the rows of its scope and the views of its database — and is never let
609
+ lift its limits there. Toa asks a query to be read-only besides, so a query can change nothing even
610
+ of what its user may: only `define` and `drop` create and remove views.
326
611
 
327
612
  ## Deployment
328
613
 
329
614
  ```yaml
330
615
  # context.toa.yaml
331
616
  chronicles:
617
+ broker: amqp://analytics.internal
332
618
  store:
333
619
  provider: clickhouse
334
- url: https://ch.example.com:8443/shop
620
+ url: http://analytics.internal:8123
335
621
  retention: 730
336
- ingest: true
622
+ encounters: true
337
623
  ```
338
624
 
339
- | | |
340
- | ----------- | ----------------------------------------------------------- |
341
- | `store` | the provider that keeps the log, and its address |
342
- | `retention` | days a row of `events` is kept; absent, kept |
343
- | `ingest` | the browser route; absent, there is none |
344
- | `ip` | `full` keeps a browser's whole address; absent, its network |
625
+ | | |
626
+ | ------------ | ----------------------------------------------------------------------------------- |
627
+ | `broker` | where the rows are published |
628
+ | `store` | where they are read: the provider, and the address of its server |
629
+ | `retention` | days a row of `events` and of `encounters` is kept; absent, kept |
630
+ | `encounters` | the routes clients send to; absent, there are none |
631
+ | `ip` | `full` keeps a client's whole address; absent, its network |
632
+ | `resources` | what the service may take, as [`resources`](/migrations/262.md) is stated elsewhere |
345
633
 
346
634
  ```shell
347
- $ toa conceal chronicles-store username=… password=… reader=…
635
+ $ toa conceal chronicles broker_username=… broker_password=… store_username=… store_password=…
348
636
  ```
349
637
 
350
- The user provisions the store when the service starts. It creates the reader, the user queries run
351
- as, with `reader` as its password.
638
+ The service `chronicles-relay` alone is given the secret.
639
+
640
+ The events the components declare travel where the context's `amqp` says, as every event does, and
641
+ the relay takes them from there. A fact and an encounter reach it as tasks. Each is acknowledged
642
+ once the broker has confirmed its row, so while the broker is away they wait where they are, and
643
+ are published once it is back; the relay starts and is ready without it.
644
+
645
+ ## The store
646
+
647
+ What Toa publishes and what it reads; what takes the one into the other is not Toa's.
648
+
649
+ **Published:** one JSON row a message, persistent, to the exchange `chronicles` (`direct`) of the
650
+ broker, under the key of its table — `events`, `states` or `encounters`. A row has the columns of
651
+ its table:
652
+
653
+ | table | row |
654
+ | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
655
+ | `events` | `scope`, `id`, `label`, `component`, `time`, `entity`, `version`, `identity`, `trace`, `payload`, `expires` |
656
+ | `states` | `scope`, `component`, `entity`, `version`, `identity`, `time`, `payload` |
657
+ | `encounters` | `scope`, `id`, `name`, `time`, `received`, `visitor`, `session`, `identity`, `ip`, `country`, `origin`, `agent`, `browser`, `os`, `device`, `payload`, `expires` |
658
+
659
+ A time is `YYYY-MM-DD hh:mm:ss.sss`, in UTC; `expires` is `time` — an encounter's `received` — and
660
+ `retention` days, or `2106-01-01 00:00:00` where there is none. A row may be delivered more than
661
+ once, with the same `id`. The broker's user needs to declare and to write the exchange `chronicles`,
662
+ and nothing else.
352
663
 
353
- The events the components name travel where the context's `amqp` says, as every event does.
664
+ **Read:** `events`, `states` and `encounters`, with those columns, in the default database of the
665
+ `store` user, and a database named after the scope for its views.
354
666
 
355
- ## ClickHouse
667
+ The relay writes the scope of its context into every row, and nothing else does: the broker tells
668
+ the store nothing of who published a row. What keeps a context from reading another's rows is the
669
+ store's.
356
670
 
357
- `url` is ClickHouse's HTTP address, and its path names the database, which is created where it does
358
- not exist. The user needs `CREATE`, `INSERT` and `access_management`; the reader is
359
- `<database>_reader`.
671
+ ### ClickHouse
360
672
 
361
- `events` and `states` are replaced by key in the background, so a query reads them with `FINAL`,
362
- which collapses what was delivered twice:
673
+ [`store/clickhouse.sql`](./store/clickhouse.sql) sets a store up: the database `chronicles` with the
674
+ three tables, which are replaced by key in the background — so a query reads them with `FINAL`,
675
+ which collapses what was delivered twice — and a RabbitMQ engine table and a view for each, which
676
+ take the rows off the broker. [`store/clickhouse.scope.sql`](./store/clickhouse.scope.sql) adds the
677
+ user of a context: its row policy, its limits and its database. A user with no row policy reads
678
+ every scope, which is how a platform reads the contexts it runs:
363
679
 
364
680
  ```sql
365
- SELECT payload.total::Float64, count() FROM events FINAL WHERE label = 'store.orders.paid' GROUP BY 1
366
- SELECT entity, payload FROM states FINAL WHERE component = 'store.orders'
681
+ SELECT scope, count() FROM encounters FINAL GROUP BY scope
682
+ SELECT visitor, groupUniqArray(identity) FROM encounters FINAL WHERE identity IS NOT NULL GROUP BY visitor
367
683
  ```
368
684
 
369
- A view with `refresh` is a refreshable materialized view.
685
+ A view with `refresh` is a refreshable materialized view, computed as its definer — the context's
686
+ user — and so of its scope.