@toa.io/extensions.chronicles 0.0.0-stage → 1.0.0-alpha.333
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +22 -0
- package/components/chronicles.encounters/manifest.toa.yaml +108 -0
- package/components/chronicles.encounters/operations/anonymous.js +4 -0
- package/components/chronicles.encounters/operations/describe.js +4 -0
- package/components/chronicles.encounters/operations/identified.js +4 -0
- package/components/chronicles.encounters/types/index.d.ts +3 -0
- package/components/chronicles.encounters/types/toa.d.ts +36 -0
- package/components/chronicles.queries/manifest.toa.yaml +139 -0
- package/components/chronicles.queries/operations/define.js +3 -0
- package/components/chronicles.queries/operations/describe.js +3 -0
- package/components/chronicles.queries/operations/drop.js +3 -0
- package/components/chronicles.queries/operations/query.js +3 -0
- package/components/chronicles.queries/types/index.d.ts +3 -0
- package/components/chronicles.queries/types/toa.d.ts +49 -0
- package/components/chronicles.relay/manifest.toa.yaml +43 -0
- package/components/chronicles.relay/operations/encounter.js +4 -0
- package/components/chronicles.relay/operations/record.js +4 -0
- package/components/chronicles.relay/types/index.d.ts +3 -0
- package/components/chronicles.relay/types/toa.d.ts +24 -0
- package/guides/clickhouse.md +19 -0
- package/guides/log.md +52 -0
- package/package.json +39 -4
- package/readme.md +686 -0
- package/store/clickhouse.scope.sql +29 -0
- package/store/clickhouse.sql +99 -0
- package/transpiled/Client.d.ts +32 -0
- package/transpiled/Client.js +173 -0
- package/transpiled/Client.js.map +1 -0
- package/transpiled/Factory.d.ts +24 -0
- package/transpiled/Factory.js +62 -0
- package/transpiled/Factory.js.map +1 -0
- package/transpiled/Provider.d.ts +142 -0
- package/transpiled/Provider.js +4 -0
- package/transpiled/Provider.js.map +1 -0
- package/transpiled/Publisher.d.ts +50 -0
- package/transpiled/Publisher.js +169 -0
- package/transpiled/Publisher.js.map +1 -0
- package/transpiled/Reader.d.ts +28 -0
- package/transpiled/Reader.js +166 -0
- package/transpiled/Reader.js.map +1 -0
- package/transpiled/Recorder.d.ts +38 -0
- package/transpiled/Recorder.js +107 -0
- package/transpiled/Recorder.js.map +1 -0
- package/transpiled/Relay.d.ts +14 -0
- package/transpiled/Relay.js +27 -0
- package/transpiled/Relay.js.map +1 -0
- package/transpiled/Sender.d.ts +25 -0
- package/transpiled/Sender.js +86 -0
- package/transpiled/Sender.js.map +1 -0
- package/transpiled/Service.d.ts +26 -0
- package/transpiled/Service.js +70 -0
- package/transpiled/Service.js.map +1 -0
- package/transpiled/address.d.ts +7 -0
- package/transpiled/address.js +27 -0
- package/transpiled/address.js.map +1 -0
- package/transpiled/agent.d.ts +16 -0
- package/transpiled/agent.js +48 -0
- package/transpiled/agent.js.map +1 -0
- package/transpiled/clickhouse/Client.d.ts +35 -0
- package/transpiled/clickhouse/Client.js +71 -0
- package/transpiled/clickhouse/Client.js.map +1 -0
- package/transpiled/clickhouse/Provider.d.ts +35 -0
- package/transpiled/clickhouse/Provider.js +169 -0
- package/transpiled/clickhouse/Provider.js.map +1 -0
- package/transpiled/clickhouse/ddl.d.ts +9 -0
- package/transpiled/clickhouse/ddl.js +15 -0
- package/transpiled/clickhouse/ddl.js.map +1 -0
- package/transpiled/clickhouse/views.d.ts +11 -0
- package/transpiled/clickhouse/views.js +97 -0
- package/transpiled/clickhouse/views.js.map +1 -0
- package/transpiled/contract.d.ts +12 -0
- package/transpiled/contract.js +58 -0
- package/transpiled/contract.js.map +1 -0
- package/transpiled/index.d.ts +15 -0
- package/transpiled/index.js +2 -0
- package/transpiled/index.js.map +1 -0
- package/transpiled/settings.d.ts +13 -0
- package/transpiled/settings.js +48 -0
- package/transpiled/settings.js.map +1 -0
- package/transpiled/store.d.ts +5 -0
- package/transpiled/store.js +11 -0
- package/transpiled/store.js.map +1 -0
- package/README.md +0 -3
package/readme.md
ADDED
|
@@ -0,0 +1,686 @@
|
|
|
1
|
+
# Toa Chronicles
|
|
2
|
+
|
|
3
|
+
A log of what happens in a context. Components write to it by the events they declare and the
|
|
4
|
+
facts they send, clients by a route of the gateway; the service `chronicles-relay` publishes every
|
|
5
|
+
row of it to an analytics broker, and reads it back from a store, by an
|
|
6
|
+
[HTTP API](#reading-the-log) — which the gateway also publishes as [MCP](#over-mcp) tools.
|
|
7
|
+
|
|
8
|
+
Writing and reading meet in nothing Toa does: what takes the rows off the broker into the store,
|
|
9
|
+
and who may read what there, is the [store's](#the-store) own setup.
|
|
10
|
+
|
|
11
|
+
## The log
|
|
12
|
+
|
|
13
|
+
Three tables, each row of which has the `scope` of the context that wrote it — the context's name,
|
|
14
|
+
followed by `TOA_SUFFIX` where one is given — and a reader sees the rows of its own scope.
|
|
15
|
+
|
|
16
|
+
**`events`** holds a row for everything a component did: an [event](#events-and-states) it
|
|
17
|
+
declares, under `<component>.<event>`, with the entity that changed and its version; and a
|
|
18
|
+
[fact](#facts) it sends, under `<component>.<name>`, with neither.
|
|
19
|
+
|
|
20
|
+
| `label` | `component` | `entity` | `version` | `payload` |
|
|
21
|
+
| ----------------------- | -------------- | -------- | --------- | --------------------------------- |
|
|
22
|
+
| `store.orders.created` | `store.orders` | `9c1f…` | `0` | `{ total: 120, status: "new" }` |
|
|
23
|
+
| `store.search.searched` | `store.search` | `NULL` | `NULL` | `{ query: "teapot", results: 3 }` |
|
|
24
|
+
|
|
25
|
+
Every row also has `id`, the same however often it is delivered, so a row is kept once;
|
|
26
|
+
`identity`, whose it is, where something [says so](#identity); `time`, when it
|
|
27
|
+
[happened](#time); and `trace`, the traceparent of the call that caused it.
|
|
28
|
+
|
|
29
|
+
**`states`** holds each entity as it stands: for every component that declares a
|
|
30
|
+
[state](#events-and-states), a row per entity — `component`, `entity`, `version`, `identity`,
|
|
31
|
+
`time` and `payload` — which a higher version replaces. It is the current state and
|
|
32
|
+
nothing before it: what an entity was yesterday is read from its events, so an event a question
|
|
33
|
+
of history depends on has to carry what that question asks.
|
|
34
|
+
|
|
35
|
+
**`encounters`** holds what [clients send](#encounters): `id`, `name`, `time`, `received`,
|
|
36
|
+
`visitor`, `session`, `identity`, `ip`, `country`, `origin`, `agent`, `browser`, `os`, `device` and
|
|
37
|
+
`payload`.
|
|
38
|
+
|
|
39
|
+
Nothing is ordered: order by `time`, or by `version` within one entity.
|
|
40
|
+
|
|
41
|
+
### Time
|
|
42
|
+
|
|
43
|
+
Every time is the one its source says — when it happened — and none is when the store wrote it:
|
|
44
|
+
|
|
45
|
+
| | `time` |
|
|
46
|
+
| --------- | ----------------------------------------------------------------------------------------- |
|
|
47
|
+
| an event | the payload's [timestamp](#when-a-change-was-made), or when the component made the change |
|
|
48
|
+
| a state | likewise |
|
|
49
|
+
| a fact | when the operation sent it |
|
|
50
|
+
| encounter | when the gateway received it, less how long before sending it the client says it happened |
|
|
51
|
+
|
|
52
|
+
So a change the relay takes an hour late is written at the time it was made, and the hours it
|
|
53
|
+
was away fill in as it catches up: a count of a recent period may still grow. An encounter's
|
|
54
|
+
`received` is when the gateway received its batch.
|
|
55
|
+
|
|
56
|
+
## Events and states
|
|
57
|
+
|
|
58
|
+
A component declares what it logs, and nothing else of it is written:
|
|
59
|
+
|
|
60
|
+
```yaml
|
|
61
|
+
# manifest.toa.yaml
|
|
62
|
+
chronicles:
|
|
63
|
+
events:
|
|
64
|
+
paid:
|
|
65
|
+
description: An order's payment completed
|
|
66
|
+
properties:
|
|
67
|
+
total:
|
|
68
|
+
type: integer
|
|
69
|
+
description: The amount paid, in cents
|
|
70
|
+
shipping:
|
|
71
|
+
type: object
|
|
72
|
+
description: Where the order goes
|
|
73
|
+
properties:
|
|
74
|
+
country:
|
|
75
|
+
type: string
|
|
76
|
+
description: ISO 3166 alpha-2
|
|
77
|
+
state:
|
|
78
|
+
event: snapshot
|
|
79
|
+
description: An order as it stands
|
|
80
|
+
properties:
|
|
81
|
+
status:
|
|
82
|
+
type: string
|
|
83
|
+
enum: [new, paid, shipped]
|
|
84
|
+
description: Where the order is in its life
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Each label under `events` is an event of the component — its own, or the prototype's `created`,
|
|
88
|
+
`updated`, `deleted` and `sync` — and `state.event` is the one whose payload is the entity. An
|
|
89
|
+
event is written as it renders: where its `condition` does not hold, nothing is written; where it
|
|
90
|
+
has a `payload`, what the payload returns is what is written. An event that exists for the log
|
|
91
|
+
alone is an event file like any other, and nothing else has to consume it.
|
|
92
|
+
|
|
93
|
+
### The contract
|
|
94
|
+
|
|
95
|
+
`description` says what the label means, and `properties` what its payload holds, as
|
|
96
|
+
[JSON Schema](https://json-schema.org) of the types [below](#types). Every property, at every
|
|
97
|
+
depth, has a `description`; a declaration that leaves one out, or uses what has no type there, is
|
|
98
|
+
refused when the component is read.
|
|
99
|
+
|
|
100
|
+
What is written is the declared properties of the payload, at every depth, and nothing else: a
|
|
101
|
+
property the declaration leaves out — an email, a document, a field added to the entity since —
|
|
102
|
+
does not reach the store. A payload whose declared property is of another type is not written,
|
|
103
|
+
and is kept in the parking queue of its consumer.
|
|
104
|
+
|
|
105
|
+
A state's payload carries `id` and `VERSION`, which say which entity it is and which version: one
|
|
106
|
+
that does not is refused, and parked likewise. A row of `events` takes its `entity` and `version`
|
|
107
|
+
from the payload's `id` and `VERSION` where it carries them, and is written without them where it
|
|
108
|
+
does not. These, the [timestamp](#when-a-change-was-made) and the [identity](#identity) are read
|
|
109
|
+
before the contract applies, and are not declared. An event that is both is named in both.
|
|
110
|
+
|
|
111
|
+
A committed change is written however long the relay or the broker is away, and once however often it is
|
|
112
|
+
delivered. One committed before the relay first ran in a deployment is not.
|
|
113
|
+
|
|
114
|
+
### Types
|
|
115
|
+
|
|
116
|
+
| declared | read as |
|
|
117
|
+
| ---------------------------------------- | ------------------------ |
|
|
118
|
+
| `string` | `String` |
|
|
119
|
+
| `string` with `enum` | `LowCardinality(String)` |
|
|
120
|
+
| `integer` | `Int64` |
|
|
121
|
+
| `number` | `Float64` |
|
|
122
|
+
| `boolean` | `Bool` |
|
|
123
|
+
| `string`, `format: date-time` | `DateTime64(3, 'UTC')` |
|
|
124
|
+
| `string`, `format: date` | `Date32` |
|
|
125
|
+
| `string`, `format: time` | `Time64(3)` |
|
|
126
|
+
| `string`, `format: uuid`, `ipv4`, `ipv6` | `UUID`, `IPv4`, `IPv6` |
|
|
127
|
+
| `integer`, `format: int32` | `Int32` |
|
|
128
|
+
| `number`, `format: float` | `Float32` |
|
|
129
|
+
| any other `format` | as its `type` |
|
|
130
|
+
| `object` with `properties` | `Tuple` of them, by name |
|
|
131
|
+
| `array` with `items` of any of these | `Array` of it |
|
|
132
|
+
|
|
133
|
+
### Reading what is declared
|
|
134
|
+
|
|
135
|
+
`describe` tells, of every declared field of a label or a state, its type and how to read it from
|
|
136
|
+
the table in that type:
|
|
137
|
+
|
|
138
|
+
```yaml
|
|
139
|
+
- label: store.orders.paid
|
|
140
|
+
fields:
|
|
141
|
+
- path: total
|
|
142
|
+
type: Int64
|
|
143
|
+
description: The amount paid, in cents
|
|
144
|
+
read: JSONExtract(ifNull(toJSONString(payload.total), 'null'), 'Nullable(Int64)')
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
```sql
|
|
148
|
+
SELECT JSONExtract(ifNull(toJSONString(payload.total), 'null'), 'Nullable(Int64)') AS total, count()
|
|
149
|
+
FROM events FINAL WHERE label = 'store.orders.paid' GROUP BY total
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
A value that is not of its type reads as `NULL`; the row keeps it as it was written. A contract
|
|
153
|
+
that changes changes what `describe` tells, and the rows written before it are read the same way.
|
|
154
|
+
|
|
155
|
+
### When a change was made
|
|
156
|
+
|
|
157
|
+
`time` is the payload's `UPDATED`, which the entity as it stands carries, where it is a number,
|
|
158
|
+
and where it is not, the moment the component made the change, which every event carries:
|
|
159
|
+
|
|
160
|
+
```yaml
|
|
161
|
+
# manifest.toa.yaml
|
|
162
|
+
chronicles:
|
|
163
|
+
timestamp: placed # the property of the payloads that says when, in milliseconds
|
|
164
|
+
events: …
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
A component names another property, or none with `timestamp: ~`, when the moment of the change
|
|
168
|
+
is always the time. One property serves every event of the component and its state.
|
|
169
|
+
|
|
170
|
+
### Identity
|
|
171
|
+
|
|
172
|
+
`identity` is the payload's `identity`, where it is a string:
|
|
173
|
+
|
|
174
|
+
```yaml
|
|
175
|
+
# manifest.toa.yaml
|
|
176
|
+
chronicles:
|
|
177
|
+
identity: customer # the property of the payloads that names whose a row is
|
|
178
|
+
events: …
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
A component names another property, or none with `identity: ~`. One property serves every event of
|
|
182
|
+
the component and its state, and a payload that leaves it out is a row of no identity. A property
|
|
183
|
+
the component names is one of its entity.
|
|
184
|
+
|
|
185
|
+
## Facts
|
|
186
|
+
|
|
187
|
+
```yaml
|
|
188
|
+
# manifest.toa.yaml
|
|
189
|
+
chronicles:
|
|
190
|
+
facts:
|
|
191
|
+
searched:
|
|
192
|
+
description: A customer searched the catalogue
|
|
193
|
+
properties:
|
|
194
|
+
query:
|
|
195
|
+
type: string
|
|
196
|
+
description: What was typed
|
|
197
|
+
results:
|
|
198
|
+
type: integer
|
|
199
|
+
description: How many products were found
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
```typescript
|
|
203
|
+
await context.chronicles.send('searched', { query: input.query, results: results.length })
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
The fact is a row of `events` under `<component>.searched`, with what was sent of its contract as
|
|
207
|
+
its payload. `send` throws for a name the component does not declare, or a payload a declared
|
|
208
|
+
property of which is of another type; it resolves once the broker has it, and it then waits for
|
|
209
|
+
the relay as an event does.
|
|
210
|
+
|
|
211
|
+
An operation that knows whose the fact is says so; the extension does not know who called it:
|
|
212
|
+
|
|
213
|
+
```typescript
|
|
214
|
+
await context.chronicles.send(
|
|
215
|
+
'searched',
|
|
216
|
+
{ query: input.query },
|
|
217
|
+
{ identity: input.customer }
|
|
218
|
+
)
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
`context.chronicles` is given to a component that declares `chronicles`.
|
|
222
|
+
|
|
223
|
+
## Encounters
|
|
224
|
+
|
|
225
|
+
What clients send: pages viewed, buttons pressed, anything a client tells of itself. Served where
|
|
226
|
+
the context says `encounters: true`, at two routes: one of an identity, and one of no one. Who may
|
|
227
|
+
send to which is in [Access](#access).
|
|
228
|
+
|
|
229
|
+
### `POST /chronicles/encounters/:identity/`
|
|
230
|
+
|
|
231
|
+
What a signed-in client sends, as the identity it is.
|
|
232
|
+
|
|
233
|
+
```http
|
|
234
|
+
POST /chronicles/encounters/9c1f0a6d5b7e4f3a8d2c1b0a9e8f7d6c/ HTTP/1.1
|
|
235
|
+
authorization: Token …
|
|
236
|
+
content-type: application/json
|
|
237
|
+
|
|
238
|
+
{
|
|
239
|
+
"sent": 1761200004000,
|
|
240
|
+
"visitor": "7f3a…",
|
|
241
|
+
"session": "01c2…",
|
|
242
|
+
"events": [{ "id": "0192f3a1-…", "name": "page_viewed", "time": 1761200000000, "payload": { "path": "/pricing" } }]
|
|
243
|
+
}
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
```http
|
|
247
|
+
202 Accepted
|
|
248
|
+
|
|
249
|
+
accepted: 1
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
The rows are that identity's, whatever the body says.
|
|
253
|
+
|
|
254
|
+
### `POST /chronicles/encounters/`
|
|
255
|
+
|
|
256
|
+
What anyone sends, before signing in or without it. The same body, and rows of no identity; a body
|
|
257
|
+
that names one is refused.
|
|
258
|
+
|
|
259
|
+
A form whose `events` is their JSON, beside `sent`, `visitor` and `session`, is read as the JSON
|
|
260
|
+
body, so `navigator.sendBeacon`, which sends no `authorization`, can send a batch here as the page
|
|
261
|
+
closes:
|
|
262
|
+
|
|
263
|
+
```javascript
|
|
264
|
+
navigator.sendBeacon(
|
|
265
|
+
'/chronicles/encounters/',
|
|
266
|
+
new URLSearchParams({
|
|
267
|
+
sent: Date.now(),
|
|
268
|
+
visitor,
|
|
269
|
+
session,
|
|
270
|
+
events: JSON.stringify(events)
|
|
271
|
+
})
|
|
272
|
+
)
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
A signed-in client sends with `fetch(…, { keepalive: true })` instead.
|
|
276
|
+
|
|
277
|
+
### The body
|
|
278
|
+
|
|
279
|
+
`{ sent, visitor?, session?, events }`:
|
|
280
|
+
|
|
281
|
+
| property | |
|
|
282
|
+
| --------- | ---------------------------------------------------------------------------------------------- |
|
|
283
|
+
| `sent` | when the client sends the batch, in milliseconds, by its clock |
|
|
284
|
+
| `visitor` | the client's own id of itself, kept across sessions; at most 64 characters, or `null` for none |
|
|
285
|
+
| `session` | the id of one period of the client's activity; at most 64 characters, or `null` for none |
|
|
286
|
+
| `events` | at most 100 of them |
|
|
287
|
+
|
|
288
|
+
Each event:
|
|
289
|
+
|
|
290
|
+
| property | |
|
|
291
|
+
| --------- | ------------------------------------------------------------------- |
|
|
292
|
+
| `id` | the client's own, at most 64 characters; a retry sends the same one |
|
|
293
|
+
| `name` | lowercase letters, digits and underscores from a letter, at most 64 |
|
|
294
|
+
| `time` | when it happened, in milliseconds, by the client's clock, by `sent` |
|
|
295
|
+
| `payload` | an object, at most 16 KiB of JSON |
|
|
296
|
+
|
|
297
|
+
The client keeps its `visitor` — in `localStorage`, say — and sends the same one before signing
|
|
298
|
+
in and after. The rows of the route of an identity have both, so the visitors of an identity are
|
|
299
|
+
a query. A visitor is not a person: one identity may have many, and a browser that clears its
|
|
300
|
+
storage is a new one.
|
|
301
|
+
|
|
302
|
+
A request is answered once its events are written. A batch one event of which breaks any of the
|
|
303
|
+
above is written not at all, and answered `422` with the code `INVALID`, or `OVERSIZED` for a
|
|
304
|
+
payload too large, and the `index` of the event in `cause`.
|
|
305
|
+
|
|
306
|
+
A payload is written as it was sent, whatever its shape: a client is deployed once and runs for
|
|
307
|
+
months, so a shape that changes is a new `name` — `checkout_started_v2` — and the old one stays as
|
|
308
|
+
old clients send it.
|
|
309
|
+
|
|
310
|
+
A client's clock is its own, and may be off by anything; what it measures between two of its
|
|
311
|
+
moments is not. So an encounter's `time` is when the gateway received the batch, less `sent`
|
|
312
|
+
minus its own `time`: the gateway's clock, at the distance the client measured. A clock that is
|
|
313
|
+
wrong moves nothing, and a batch kept offline for an hour is written an hour back. A row is kept
|
|
314
|
+
and expires by `received`.
|
|
315
|
+
|
|
316
|
+
### Address and country
|
|
317
|
+
|
|
318
|
+
Every encounter has the `ip` and the `country` of the request, as the gateway reads them from the
|
|
319
|
+
headers [`exposition.ip` and `exposition.country`](/extensions/exposition/documentation/ip.md)
|
|
320
|
+
name, and `NULL` where it has none; a body that names either is overwritten. An address is
|
|
321
|
+
personal data, and the log erases nothing, so of an address the log keeps its network — a `/24`
|
|
322
|
+
of IPv4, a `/48` of IPv6 — unless the annotation says `ip: full`.
|
|
323
|
+
|
|
324
|
+
### Origin and agent
|
|
325
|
+
|
|
326
|
+
Every encounter has the `origin` of the page that sent it and the client's `agent`, from the
|
|
327
|
+
request's `Origin` and `User-Agent` headers, and `NULL` where it has none; a body that names either
|
|
328
|
+
is overwritten, and a body cannot say one the request does not have. A browser sends `Origin` with
|
|
329
|
+
a request to another origin, and with every `POST`, so it tells which of the context's sites a
|
|
330
|
+
batch came from.
|
|
331
|
+
|
|
332
|
+
What the agent says the client is, is three columns of its own, by name and without versions:
|
|
333
|
+
|
|
334
|
+
| | |
|
|
335
|
+
| --------- | ----------------------------------------------------------------------------- |
|
|
336
|
+
| `browser` | `Edge`, `Opera`, `Samsung Internet`, `Firefox`, `Chrome`, `Safari`, or `NULL` |
|
|
337
|
+
| `os` | `Windows`, `iOS`, `macOS`, `Android`, `ChromeOS`, `Linux`, or `NULL` |
|
|
338
|
+
| `device` | `bot`, `tablet`, `mobile` or `desktop`, and `NULL` where there is no agent |
|
|
339
|
+
|
|
340
|
+
`bot` is a crawler, a headless browser or a program — what a question of who visits leaves out.
|
|
341
|
+
Safari on an iPad says it is a Mac, and is a `desktop` running `macOS` here. A version is read from
|
|
342
|
+
`agent` where a question needs one, knowing that browsers freeze what they say of the system:
|
|
343
|
+
Safari on iOS 26 says iOS 18, and Chrome on Android says Android 10.
|
|
344
|
+
|
|
345
|
+
### What the encounters are
|
|
346
|
+
|
|
347
|
+
What a client sends is told in the configuration of `chronicles.encounters`, by name:
|
|
348
|
+
|
|
349
|
+
```yaml
|
|
350
|
+
# context.toa.yaml
|
|
351
|
+
configuration:
|
|
352
|
+
chronicles.encounters:
|
|
353
|
+
page_viewed:
|
|
354
|
+
description: A page was shown
|
|
355
|
+
fields:
|
|
356
|
+
path: The URL path, without the query
|
|
357
|
+
referrer: The page the visitor came from
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
It is what `describe` tells an agent of each encounter's fields, and what whoever writes a client
|
|
361
|
+
reads:
|
|
362
|
+
|
|
363
|
+
```http
|
|
364
|
+
GET /chronicles/encounters/ HTTP/1.1
|
|
365
|
+
authorization: Token …
|
|
366
|
+
accept: application/yaml
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
```http
|
|
370
|
+
200 OK
|
|
371
|
+
|
|
372
|
+
page_viewed:
|
|
373
|
+
description: A page was shown
|
|
374
|
+
fields:
|
|
375
|
+
path: The URL path, without the query
|
|
376
|
+
referrer: The page the visitor came from
|
|
377
|
+
```
|
|
378
|
+
|
|
379
|
+
It documents, and holds nothing back: a name it does not tell is written as any other.
|
|
380
|
+
|
|
381
|
+
## Reading the log
|
|
382
|
+
|
|
383
|
+
A refusal is answered `422` with `{ code, cause }`. Who may read is in [Access](#access).
|
|
384
|
+
|
|
385
|
+
### `GET /chronicles/queries/`
|
|
386
|
+
|
|
387
|
+
What the log holds: a guide to reading it, the labels of `events`, the components
|
|
388
|
+
with a state, the encounters, and the views.
|
|
389
|
+
|
|
390
|
+
```http
|
|
391
|
+
GET /chronicles/queries/ HTTP/1.1
|
|
392
|
+
authorization: Token …
|
|
393
|
+
accept: application/yaml
|
|
394
|
+
```
|
|
395
|
+
|
|
396
|
+
```http
|
|
397
|
+
200 OK
|
|
398
|
+
|
|
399
|
+
guide: |
|
|
400
|
+
# Reading the log
|
|
401
|
+
|
|
402
|
+
The log is three tables, read with `FINAL`.
|
|
403
|
+
…
|
|
404
|
+
labels:
|
|
405
|
+
- label: store.orders.paid
|
|
406
|
+
description: An order's payment completed
|
|
407
|
+
rows: 1204
|
|
408
|
+
first: 2026-10-01 08:12:03.000
|
|
409
|
+
last: 2026-10-24 17:40:51.000
|
|
410
|
+
fields:
|
|
411
|
+
- path: total
|
|
412
|
+
type: Int64
|
|
413
|
+
description: The amount paid, in cents
|
|
414
|
+
read: JSONExtract(ifNull(toJSONString(payload.total), 'null'), 'Nullable(Int64)')
|
|
415
|
+
- path: shipping
|
|
416
|
+
type: Tuple(country String)
|
|
417
|
+
description: Where the order goes
|
|
418
|
+
read: JSONExtract(toJSONString(payload.^shipping), 'Tuple(country Nullable(String))')
|
|
419
|
+
- path: shipping.country
|
|
420
|
+
type: String
|
|
421
|
+
description: ISO 3166 alpha-2
|
|
422
|
+
states:
|
|
423
|
+
- component: store.orders
|
|
424
|
+
description: An order as it stands
|
|
425
|
+
entities: 1180
|
|
426
|
+
last: 2026-10-24 17:40:51.000
|
|
427
|
+
fields:
|
|
428
|
+
- path: status
|
|
429
|
+
type: LowCardinality(String)
|
|
430
|
+
description: Where the order is in its life
|
|
431
|
+
read: JSONExtract(ifNull(toJSONString(payload.status), 'null'), 'LowCardinality(Nullable(String))')
|
|
432
|
+
encounters:
|
|
433
|
+
- name: page_viewed
|
|
434
|
+
description: A page was shown
|
|
435
|
+
rows: 88123
|
|
436
|
+
first: 2026-10-01 00:00:14.000
|
|
437
|
+
last: 2026-10-24 17:41:02.000
|
|
438
|
+
received: 2026-10-24 17:41:03.020
|
|
439
|
+
fields:
|
|
440
|
+
- path: path
|
|
441
|
+
types: [String]
|
|
442
|
+
description: The URL path, without the query
|
|
443
|
+
- path: utm
|
|
444
|
+
types: [Int64, String]
|
|
445
|
+
views:
|
|
446
|
+
- name: paid_daily
|
|
447
|
+
read: shop.paid_daily
|
|
448
|
+
title: Orders paid per day
|
|
449
|
+
description: Orders whose payment completed, counted by the day of `time`
|
|
450
|
+
sql: SELECT toDate(time) AS day, count() AS paid FROM events FINAL WHERE label = 'store.orders.paid' GROUP BY day
|
|
451
|
+
refresh: 3600
|
|
452
|
+
refreshed: 2026-10-24 17:00:03
|
|
453
|
+
```
|
|
454
|
+
|
|
455
|
+
`guide` is how to read the log — what its tables and columns mean, and what to be careful of —
|
|
456
|
+
and how its store is read, in the store's dialect: what an agent reads before its first query.
|
|
457
|
+
Toa writes it; what is particular to an application is in the descriptions of what it declares
|
|
458
|
+
and documents, and in [`exposition.mcp.instructions`](/extensions/exposition/documentation/mcp.md).
|
|
459
|
+
|
|
460
|
+
A label and a state are told by their declaration, with each field's type and how to
|
|
461
|
+
[read](#reading-what-is-declared) it in that type; a field of a field is read inside the one that
|
|
462
|
+
holds it. A label declared and not yet written is told with no rows. An encounter is
|
|
463
|
+
told by what was sent: the fields of the latest payloads, the types they were sent as — two
|
|
464
|
+
types of one field are two versions of a client — and the description the
|
|
465
|
+
[configuration](#what-the-encounters-are) gives.
|
|
466
|
+
|
|
467
|
+
`rows` counts what the store holds, a row delivered twice possibly twice. `first` and `last` are
|
|
468
|
+
by `time`, the latest change of a state too; an encounter's `received` is the newest batch the
|
|
469
|
+
gateway received, which tells whether clients still send.
|
|
470
|
+
|
|
471
|
+
### `POST /chronicles/queries/`
|
|
472
|
+
|
|
473
|
+
One statement, in the store's dialect, as the `guide` says.
|
|
474
|
+
|
|
475
|
+
```http
|
|
476
|
+
POST /chronicles/queries/ HTTP/1.1
|
|
477
|
+
authorization: Token …
|
|
478
|
+
accept: application/yaml
|
|
479
|
+
content-type: application/yaml
|
|
480
|
+
|
|
481
|
+
sql: SELECT status, count() AS n FROM store_orders GROUP BY status
|
|
482
|
+
```
|
|
483
|
+
|
|
484
|
+
```http
|
|
485
|
+
201 Created
|
|
486
|
+
|
|
487
|
+
columns:
|
|
488
|
+
- name: status
|
|
489
|
+
type: Nullable(String)
|
|
490
|
+
- name: n
|
|
491
|
+
type: UInt64
|
|
492
|
+
rows:
|
|
493
|
+
- [new, 31]
|
|
494
|
+
- [paid, 1149]
|
|
495
|
+
truncated: false
|
|
496
|
+
```
|
|
497
|
+
|
|
498
|
+
A query reads the log and nothing else, and cannot write. It runs for at most 30 seconds, answers
|
|
499
|
+
at most 10,000 rows and 64 MiB, and a reader runs at most 4 queries at once: past the rows it
|
|
500
|
+
answers what it has, with `truncated: true`; past the rest it is refused. It leaves `FORMAT` out.
|
|
501
|
+
|
|
502
|
+
| code | when |
|
|
503
|
+
| --------- | ----------------------------------------------------------------------- |
|
|
504
|
+
| `REFUSED` | the store refused the statement; `cause.message` is what the store said |
|
|
505
|
+
|
|
506
|
+
### `PUT /chronicles/queries/views/:name/`
|
|
507
|
+
|
|
508
|
+
A view over the log, by a statement a query could run. Defining a name again replaces the view.
|
|
509
|
+
|
|
510
|
+
```http
|
|
511
|
+
PUT /chronicles/queries/views/paid_daily/ HTTP/1.1
|
|
512
|
+
authorization: Token …
|
|
513
|
+
content-type: application/yaml
|
|
514
|
+
|
|
515
|
+
title: Orders paid per day
|
|
516
|
+
description: Orders whose payment completed, counted by the day of `time`
|
|
517
|
+
sql: SELECT toDate(time) AS day, count() AS paid FROM events FINAL WHERE label = 'store.orders.paid' GROUP BY day
|
|
518
|
+
refresh: 3600
|
|
519
|
+
```
|
|
520
|
+
|
|
521
|
+
```http
|
|
522
|
+
204 No Content
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
`title`, at most 100 characters, and `description` tell an agent that finds the view what it
|
|
526
|
+
counts. Without `refresh`, a view is computed when it is read. With `refresh`, in seconds and at
|
|
527
|
+
least `3600`, it is computed on that cycle and read as a table, and `describe` tells when it last
|
|
528
|
+
was. A name is lowercase letters, digits and underscores, and a context has at most 50 views. A
|
|
529
|
+
view is kept in the database named after the context's scope, and a query reads it by that name:
|
|
530
|
+
`shop.paid_daily`. Defining one takes a store whose user may create views there; one that may not
|
|
531
|
+
is refused.
|
|
532
|
+
|
|
533
|
+
| code | when |
|
|
534
|
+
| ----------- | -------------------------------------------------------------------- |
|
|
535
|
+
| `REFUSED` | the statement is not one a query could run; `cause.message` says why |
|
|
536
|
+
| `RESERVED` | the name is one of the log's tables |
|
|
537
|
+
| `EXHAUSTED` | the context has 50 views, and this is not one of them |
|
|
538
|
+
|
|
539
|
+
### `DELETE /chronicles/queries/views/:name/`
|
|
540
|
+
|
|
541
|
+
```http
|
|
542
|
+
DELETE /chronicles/queries/views/paid_daily/ HTTP/1.1
|
|
543
|
+
authorization: Token …
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
```http
|
|
547
|
+
204 No Content
|
|
548
|
+
```
|
|
549
|
+
|
|
550
|
+
| code | when |
|
|
551
|
+
| ----------- | --------------------- |
|
|
552
|
+
| `NOT_FOUND` | there is no such view |
|
|
553
|
+
|
|
554
|
+
### Over MCP
|
|
555
|
+
|
|
556
|
+
Where the context publishes [MCP](/extensions/exposition/documentation/mcp.md), each of these
|
|
557
|
+
routes is a tool, under the name the gateway gives a route, and taking what the route takes:
|
|
558
|
+
|
|
559
|
+
| tool | route |
|
|
560
|
+
| --------------------------------------- | ----------------------------------------- |
|
|
561
|
+
| `chronicles.queries.GET` | `GET /chronicles/queries/` |
|
|
562
|
+
| `chronicles.queries.POST` | `POST /chronicles/queries/` |
|
|
563
|
+
| `chronicles.queries.views._name.PUT` | `PUT /chronicles/queries/views/:name/` |
|
|
564
|
+
| `chronicles.queries.views._name.DELETE` | `DELETE /chronicles/queries/views/:name/` |
|
|
565
|
+
| `chronicles.encounters.GET` | `GET /chronicles/encounters/` |
|
|
566
|
+
|
|
567
|
+
A refusal is a tool result with `isError`.
|
|
568
|
+
|
|
569
|
+
## Access
|
|
570
|
+
|
|
571
|
+
| route | who may call | limit |
|
|
572
|
+
| ----------------------------------------- | ------------------------------------ | --------------------------- |
|
|
573
|
+
| `POST /chronicles/encounters/` | anyone, with no credentials | 60 a minute per address |
|
|
574
|
+
| `POST /chronicles/encounters/:identity/` | the identity, by its own credentials | 60 a minute per identity |
|
|
575
|
+
| `GET /chronicles/encounters/` | `system:chronicles:encounters` | |
|
|
576
|
+
| `GET /chronicles/queries/` | `system:chronicles:query` | |
|
|
577
|
+
| `POST /chronicles/queries/` | `system:chronicles:query` | the [reader's](#the-reader) |
|
|
578
|
+
| `PUT /chronicles/queries/views/:name/` | `system:chronicles:views` | 50 views |
|
|
579
|
+
| `DELETE /chronicles/queries/views/:name/` | `system:chronicles:views` | |
|
|
580
|
+
|
|
581
|
+
A caller without what a route takes is answered `403`, and past its limit `429`. Over
|
|
582
|
+
[MCP](#over-mcp), a tool is listed to a caller that holds its role, and to no one else.
|
|
583
|
+
|
|
584
|
+
### Roles
|
|
585
|
+
|
|
586
|
+
| | |
|
|
587
|
+
| ------------------------------ | ------------------------------------------------------------------------------------------- |
|
|
588
|
+
| `system:chronicles:query` | reads every row of the log: the identities, the networks, the origins and agents of clients |
|
|
589
|
+
| `system:chronicles:views` | defines and drops the views every reader reads; a view can read only what a query can |
|
|
590
|
+
| `system:chronicles:encounters` | reads what the configuration documents of encounters, and no row |
|
|
591
|
+
| `system:chronicles` | all three |
|
|
592
|
+
|
|
593
|
+
`query` is access to personal data, and is granted as that. `views` changes what other readers are
|
|
594
|
+
told a view counts, and changes nothing of the log. `encounters` is for whoever writes a client.
|
|
595
|
+
A role is granted as any [role](/extensions/exposition/documentation/access.md) is.
|
|
596
|
+
|
|
597
|
+
### What a client cannot say
|
|
598
|
+
|
|
599
|
+
A client sends events and nothing that says who or where it is: `identity` is the path's, which the
|
|
600
|
+
credentials have to be the identity's; `ip`, `country`, `origin` and `agent` are what the gateway
|
|
601
|
+
read off the request, and a body that names one is overwritten, or refused where it names
|
|
602
|
+
`identity` on the route of no one. Anything else in a payload is what the client said — a `user`
|
|
603
|
+
there is not a user — so a question that depends on who sent an encounter asks `identity`.
|
|
604
|
+
|
|
605
|
+
### The reader
|
|
606
|
+
|
|
607
|
+
A query runs as the context's user of the store, which reads what the [store](#the-store) lets it
|
|
608
|
+
— in the reference setup, the rows of its scope and the views of its database — and is never let
|
|
609
|
+
lift its limits there. Toa asks a query to be read-only besides, so a query can change nothing even
|
|
610
|
+
of what its user may: only `define` and `drop` create and remove views.
|
|
611
|
+
|
|
612
|
+
## Deployment
|
|
613
|
+
|
|
614
|
+
```yaml
|
|
615
|
+
# context.toa.yaml
|
|
616
|
+
chronicles:
|
|
617
|
+
broker: amqp://analytics.internal
|
|
618
|
+
store:
|
|
619
|
+
provider: clickhouse
|
|
620
|
+
url: http://analytics.internal:8123
|
|
621
|
+
retention: 730
|
|
622
|
+
encounters: true
|
|
623
|
+
```
|
|
624
|
+
|
|
625
|
+
| | |
|
|
626
|
+
| ------------ | ----------------------------------------------------------------------------------- |
|
|
627
|
+
| `broker` | where the rows are published |
|
|
628
|
+
| `store` | where they are read: the provider, and the address of its server |
|
|
629
|
+
| `retention` | days a row of `events` and of `encounters` is kept; absent, kept |
|
|
630
|
+
| `encounters` | the routes clients send to; absent, there are none |
|
|
631
|
+
| `ip` | `full` keeps a client's whole address; absent, its network |
|
|
632
|
+
| `resources` | what the service may take, as [`resources`](/migrations/262.md) is stated elsewhere |
|
|
633
|
+
|
|
634
|
+
```shell
|
|
635
|
+
$ toa conceal chronicles broker_username=… broker_password=… store_username=… store_password=…
|
|
636
|
+
```
|
|
637
|
+
|
|
638
|
+
The service `chronicles-relay` alone is given the secret.
|
|
639
|
+
|
|
640
|
+
The events the components declare travel where the context's `amqp` says, as every event does, and
|
|
641
|
+
the relay takes them from there. A fact and an encounter reach it as tasks. Each is acknowledged
|
|
642
|
+
once the broker has confirmed its row, so while the broker is away they wait where they are, and
|
|
643
|
+
are published once it is back; the relay starts and is ready without it.
|
|
644
|
+
|
|
645
|
+
## The store
|
|
646
|
+
|
|
647
|
+
What Toa publishes and what it reads; what takes the one into the other is not Toa's.
|
|
648
|
+
|
|
649
|
+
**Published:** one JSON row a message, persistent, to the exchange `chronicles` (`direct`) of the
|
|
650
|
+
broker, under the key of its table — `events`, `states` or `encounters`. A row has the columns of
|
|
651
|
+
its table:
|
|
652
|
+
|
|
653
|
+
| table | row |
|
|
654
|
+
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
655
|
+
| `events` | `scope`, `id`, `label`, `component`, `time`, `entity`, `version`, `identity`, `trace`, `payload`, `expires` |
|
|
656
|
+
| `states` | `scope`, `component`, `entity`, `version`, `identity`, `time`, `payload` |
|
|
657
|
+
| `encounters` | `scope`, `id`, `name`, `time`, `received`, `visitor`, `session`, `identity`, `ip`, `country`, `origin`, `agent`, `browser`, `os`, `device`, `payload`, `expires` |
|
|
658
|
+
|
|
659
|
+
A time is `YYYY-MM-DD hh:mm:ss.sss`, in UTC; `expires` is `time` — an encounter's `received` — and
|
|
660
|
+
`retention` days, or `2106-01-01 00:00:00` where there is none. A row may be delivered more than
|
|
661
|
+
once, with the same `id`. The broker's user needs to declare and to write the exchange `chronicles`,
|
|
662
|
+
and nothing else.
|
|
663
|
+
|
|
664
|
+
**Read:** `events`, `states` and `encounters`, with those columns, in the default database of the
|
|
665
|
+
`store` user, and a database named after the scope for its views.
|
|
666
|
+
|
|
667
|
+
The relay writes the scope of its context into every row, and nothing else does: the broker tells
|
|
668
|
+
the store nothing of who published a row. What keeps a context from reading another's rows is the
|
|
669
|
+
store's.
|
|
670
|
+
|
|
671
|
+
### ClickHouse
|
|
672
|
+
|
|
673
|
+
[`store/clickhouse.sql`](./store/clickhouse.sql) sets a store up: the database `chronicles` with the
|
|
674
|
+
three tables, which are replaced by key in the background — so a query reads them with `FINAL`,
|
|
675
|
+
which collapses what was delivered twice — and a RabbitMQ engine table and a view for each, which
|
|
676
|
+
take the rows off the broker. [`store/clickhouse.scope.sql`](./store/clickhouse.scope.sql) adds the
|
|
677
|
+
user of a context: its row policy, its limits and its database. A user with no row policy reads
|
|
678
|
+
every scope, which is how a platform reads the contexts it runs:
|
|
679
|
+
|
|
680
|
+
```sql
|
|
681
|
+
SELECT scope, count() FROM encounters FINAL GROUP BY scope
|
|
682
|
+
SELECT visitor, groupUniqArray(identity) FROM encounters FINAL WHERE identity IS NOT NULL GROUP BY visitor
|
|
683
|
+
```
|
|
684
|
+
|
|
685
|
+
A view with `refresh` is a refreshable materialized view, computed as its definer — the context's
|
|
686
|
+
user — and so of its scope.
|