@pmndre/pine 0.35.1 → 0.36.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -47,14 +47,14 @@ scans", or `pine unsubscribe main scans`.
47
47
  "<name>": {
48
48
  "type": "folder | mqtt | webhook | ftp | telegram | http | command | feed | ...",
49
49
  "description": "one line, shown to agents and in listings",
50
- "...": "the options of the type, see below"
50
+ "...": "the options of the type"
51
51
  }
52
52
  }
53
53
  ```
54
54
 
55
55
  | Key | Default | Meaning |
56
56
  |---|---|---|
57
- | `type` | | One of the built-in types: the seven everyday ones below, and the ones in [source-types.md](source-types.md) (`feed`, `calendar`, `stream`, `mail`, `slack`, `matrix`, `discord`, `smtp`, `syslog`, `system`, `presence`, `modbus`, `serial`, `onvif`). Or a type of your own registered under `sourceTypes` (see "A source type of your own"). Or `use` instead: the path of a module, for a one-off. |
57
+ | `type` | | One of the built-in types, each described in [source-types.md](source-types.md): `folder`, `mqtt`, `webhook`, `ftp`, `telegram`, `http`, `command`, `feed`, `calendar`, `stream`, `mail`, `slack`, `matrix`, `discord`, `smtp`, `syslog`, `onvif`, `modbus`, `serial`, `presence`, `system`. Or a type of your own registered under `sourceTypes` (see "A source type of your own"). Or `use` instead: the path of a module, for a one-off. |
58
58
  | `description` | | What it reports. Agents choose what to subscribe to by this line, so write it for them. |
59
59
  | `agents` | all | In `pine.json` only: the agents that may subscribe, e.g. `["house"]`. |
60
60
  | `trust` | `"untrusted"` | Who its events speak for. See "Trust" below. |
@@ -68,7 +68,8 @@ a global one is the one that agent means by that name. Secrets go into `.env` an
68
68
  as `${NAME}`, as everywhere in the configuration; when an agent sets a source up, pine does that
69
69
  by itself (see "Secrets" below).
70
70
 
71
- Every type is a module that says which options it takes, as a schema. The tables below, the
71
+ Every type is a module that says which options it takes, as a schema. The tables in
72
+ [source-types.md](source-types.md), the
72
73
  `add_source` tool an agent uses and the check of what an agent passes all come from that one
73
74
  description, so they cannot drift apart, and a type of your own is offered to agents exactly
74
75
  like the built-in ones.
@@ -80,513 +81,25 @@ others, and routes can match them.
80
81
  What should happen at a time ("every morning at seven", "remind me at five") is not a source
81
82
  but a schedule: [events.md](events.md), "Schedules".
82
83
 
83
- ## The built-in types
84
-
85
- These are the seven everyday types. Fourteen more, for feeds, calendars, services that push,
86
- mail, chat, devices and the computer itself, are in [source-types.md](source-types.md),
87
- together with a table of which type reaches which service.
88
-
89
- ### folder
90
-
91
- A folder: reports every new or changed file once it has stopped growing.
92
-
93
- ```json
94
- "scans": { "type": "folder", "dir": "~/Scans", "mask": "*.pdf" }
95
- ```
96
-
97
- | Option | Default | |
98
- |---|---|---|
99
- | `dir` | | The folder. `~` and paths relative to the configuration file work. |
100
- | `mask` | | Only files whose name matches, as in a shell: `*.pdf`, `*.{png,jpg}`, `report-??.csv`. Several are separated by commas. Capitals do not matter. |
101
- | `pattern` | | Only files whose full path matches this regular expression, for what a mask cannot say (`/src/.*\\.ts$`). Capitals do not matter. |
102
- | `ignore` | version control, dependencies, editor leftovers | Never files whose path matches this one. The default skips `.git`, `node_modules`, `.DS_Store`, build folders, and files ending in `~`, `.swp`, `.tmp`, `.part`. |
103
- | `settle` | `"1.5s"` | How long a file must stay unchanged before it counts as complete. |
104
- | `recursive` | `true` | Also folders inside it. |
105
-
106
- Event `added`: `{ path, file, size, images? }`. For jpg, png, webp and gif under 10 MB the
107
- picture is shown to the agent if its model can look at images.
108
-
109
- ### mqtt
110
-
111
- An MQTT broker: reports the messages on the topics you name.
112
-
113
- ```json
114
- "printer": {
115
- "type": "mqtt", "url": "mqtts://192.168.1.50:8883", "username": "bblp", "password": "${BAMBU_ACCESS_CODE}",
116
- "topics": ["device/+/report"], "insecureTls": true, "minInterval": "30s",
117
- "trust": "owner", "description": "status reports of the 3D printer"
118
- }
119
- ```
120
-
121
- | Option | Default | |
122
- |---|---|---|
123
- | `url` | | `mqtt://`, `mqtts://`, `ws://` or `wss://` address of the broker. |
124
- | `topics` | | One MQTT topic filter or a list. |
125
- | `publish` | none | Topics an agent may send to, to act on a device: `["zigbee2mqtt/+/set"]`. Without it the source only listens. See "Sending through a source". |
126
- | `username`, `password`, `clientId` | | As the broker wants them. |
127
- | `minInterval` | | Drop messages that follow the last one on the same topic sooner than this. Use it for devices that report every second. |
128
- | `insecureTls` | `false` | Accept a certificate that cannot be verified, as devices with self-made ones need. |
129
-
130
- Event `message`: `{ topic, value }`, where `value` is the parsed JSON, or the text.
131
-
132
- The example is a Bambu printer in LAN mode: it is its own broker on port 8883, the user is
133
- `bblp`, the password is the access code from the printer's screen.
134
-
135
- ### webhook
136
-
137
- A web address other services can call: `POST http://<pine's control address>/hook/<name>`.
138
-
139
- ```json
140
- "github": { "type": "webhook", "secret": "${GITHUB_HOOK_SECRET}", "description": "pushes and issues of my repositories" }
141
- ```
142
-
143
- | Option | |
144
- |---|---|
145
- | `secret` | The caller proves itself with it: `Authorization: Bearer <secret>`, or a signature of the body (HMAC SHA-256 in `X-Hub-Signature-256` or `X-Signature`, as GitHub sends it). |
146
- | `open` | `true` instead of a secret: accept anyone who can reach the port. Only for a port nobody else can reach. |
147
-
148
- Event `received`: `{ body, headers }`. `body` is the parsed JSON, or `{ text }`. Of the headers
149
- only a few that say what kind of request it is are kept (`content-type`, `user-agent`,
150
- `x-github-event` and the like), never the secret.
151
-
152
- The address has the source's name as it is in the event topics: `/hook/<name>` for a source in
153
- `pine.json`, `/hook/<name>@<agent>` for one in an agent's own `agent.json` (which is where a
154
- source that an agent sets up goes, unless it is for all agents).
155
-
156
- It needs the control API (`http` in `pine.json`), which listens on `127.0.0.1` by default. A
157
- service on the internet reaches it only through a tunnel or reverse proxy that you set up. While
158
- nobody is subscribed the address answers "not found".
159
-
160
- ### ftp
161
-
162
- An FTP drop: a small FTP server that accepts uploads and nothing else. For cameras, doorbells,
163
- scanners and other devices whose way of sending a file is "upload to FTP server".
164
-
165
- ```json
166
- "door": { "type": "ftp", "port": 2121, "user": "cam", "password": "${CAMERA_FTP_PASSWORD}", "allow": ["192.168.1.40"], "description": "pictures from the front door camera when it sees motion" }
167
- ```
168
-
169
- | Option | Default | |
170
- |---|---|---|
171
- | `port`, `user`, `password` | | What you enter in the device. The password needs 8 characters or more. When an agent sets the source up, `user` is `device` unless you name one, and a password you do not give is made by pine. |
172
- | `host` | `"0.0.0.0"` | The address to listen on. Devices on your network have to reach it. |
173
- | `allow` | everyone | Only these clients: a full address (`"192.168.1.40"`), or a prefix that ends in a dot (`"192.168.2."`). |
174
- | `pasvPorts` | any | `[from, to]`: the ports for data connections, if a firewall has to know them. |
175
- | `maxFileMb` | `50` | Largest file accepted. |
176
- | `keepDays` | `7` | Received files are deleted after this many days. |
177
-
178
- Event `upload`: `{ file, path, size, from, images? }`. `file` is the name the device gave it,
179
- `path` is where it lies now (`data/sources/<name>/files/`, or `data/sources/<name>@<agent>/files/`
180
- for an agent's own source). A picture is shown to the agent with the event. A video or any
181
- other file is not: the agent reads it from `path` (see "Files a source received").
182
-
183
- It speaks plain FTP, as these devices do, so the password crosses your network unencrypted. It
184
- only receives: nothing can be listed, read, changed or deleted through it, and every file lands
185
- under a name of pine's choosing in the source's own folder. Set the device to passive mode.
186
-
187
- ### telegram
188
-
189
- A Telegram bot: you write to it from wherever you are, and the agent that subscribed answers
190
- there. It is how you reach your agents when you are away, and how they reach you.
191
-
192
- > Watch Telegram. You are my assistant there, and what you want to tell me while I am away
193
- > goes to that chat.
194
-
195
- The agent calls `add_source`. The dialog you confirm has a field for the bot's token: you type
196
- or paste it there, where it is not shown and does not pass through the agent or its model (see
197
- "Secrets" below). Then the agent tells you the one remaining step, which ties the bot to your account:
198
-
199
- 1. Open the bot in Telegram and write anything to it.
200
- 2. The bot answers that account with a six-digit code.
201
- 3. pine shows a question on your computer that names the account and has a field for the code.
202
- Type the code from your phone and press enter.
203
-
204
- From then on that account is the only one the bot listens to. The code is the point: it shows
205
- that the person at pine's keyboard holds that Telegram account. A stranger who finds the bot
206
- can make the question appear, but does not see your screen, and you do not have their code; a
207
- key pressed by accident cannot say yes. An account you deny is not put to you again, and only
208
- one question is open at a time.
209
-
210
- Anyone can write to a bot; that is how Telegram works. What they write reaches no agent. Before
211
- you have paired, an unknown account gets the code message (and a line if the code that was
212
- typed was wrong) and nothing else. After you have,
213
- unknown accounts get no answer at all and cause no question.
214
-
215
- ```json
216
- "telegram": { "type": "telegram", "token": "${PINE_SOURCE_TELEGRAM_MAIN_TOKEN}", "trust": "owner", "description": "my Telegram" }
217
- ```
218
-
219
- | Option | Default | |
220
- |---|---|---|
221
- | `token` | | The bot's token. Create a bot by writing `/newbot` to @BotFather in Telegram; it answers with the token. Kept in `.env`. |
222
- | `allow` | pairing | Telegram user ids that may talk to the bot, if you know yours. Without it the bot is paired as described above. With several ids, what the agent sends unasked and the approval requests go to the first. |
223
- | `approvals` | `true` | Approval requests are also sent to that chat, with buttons to answer them: Allow, Deny and, where a yes can be remembered for the conversation, Always. `/allow` and `/deny` work too (with the request's id when more than one is open). A question the agent asks with its `question` tool arrives the same way when it offers answers to pick from, with one button for each and one for "No answer"; an answer in your own words, and a question without answers to pick from, you type in the terminal UI. A request that needs a secret typed is never sent there. |
224
- | `apiBase` | Telegram's | The address of a Bot API server of your own. |
225
-
226
- Event `message`: `{ text, name, username, chatId, messageId, images? }`. A photo you send (up
227
- to 10 MB) is shown to the model. The bot listens only in its private chat with you: what is
228
- written in a group it was added to reaches no agent.
229
-
230
- What makes it different from the other types:
231
-
232
- - **It is a conversation.** Your message reaches the agent as your message, not as a report
233
- about one, in the conversation that subscribed. The answer is sent back to the chat and is
234
- also in that conversation in the terminal UI, so both show the same thread.
235
- - **It speaks for you.** Telegram vouches for the account and you proved it is yours, so what
236
- it writes counts as your own words: full tools, no "outside content" mark. That is why
237
- pairing takes a code and why the set-up is always put to you.
238
- - **It can send.** The agent writes to you on its own with the `send` tool, with a picture or
239
- a video if it has one to show, and without a question to you; see below.
240
- - **Messages are not held back.** The ten-second pause between events that protects a model from
241
- a chatty device does not apply unless the subscription sets a `throttle` itself.
242
-
243
- Who the bot listens to is said wherever sources are listed: `pine sources`, `/sources` in the
244
- terminal UI, and the agent's `sources` tool show "listens to nobody yet", or the account you
245
- paired with its name and Telegram id. A bot never learns a phone number; the account's id is
246
- what Telegram vouches for. To pair with another account, remove the source and set it up
247
- again: the pairing goes with the source.
248
-
249
- Messages are fetched by long polling: one request that Telegram answers the moment something
250
- arrives. Nothing listens on a port, so it works behind any router. Only one program can poll a
251
- bot at a time; a second one makes both miss messages.
252
-
253
- ### http
254
-
255
- A web address that is looked at again and again. For what cannot call you: a price, a version,
256
- a status page, a device on your network that has a state page.
257
-
258
- > Watch https://shop.example/fridge-3000 and tell me when the price drops below 500.
259
-
260
- ```json
261
- "price": { "type": "http", "url": "https://shop.example/fridge-3000", "match": "Price: ([0-9.,]+)", "every": "30m", "description": "the price of the fridge" }
262
- ```
263
-
264
- | Option | Default | |
265
- |---|---|---|
266
- | `url` | | The address, `https://...` (`http://` for a device on the local network). |
267
- | `every` | `"5m"` | How often to look. At least `"30s"` when an agent sets it up, and at least `"2m"` with `render`. |
268
- | `path` | | For an answer in JSON: watch only this field, `"data.price"` or `"items.0.state"`. `path`, `values`, `match` and `select` are four ways to say what counts in the answer; a source takes one of them. |
269
- | `match` | | For a page or text: watch only what this regular expression finds, its first group if it has one. It is looked for in what a reader of the page sees, then in the page's source. |
270
- | `select` | | For a page: watch only the text of the first element this CSS selector matches, `".product .price"` or `"#stock > span"`. With `render` it is any selector a browser reads. |
271
- | `attr` | | With `select`: the value of this attribute of the element instead of its text, `"content"` for a meta tag, `"data-price"`. |
272
- | `render` | `false` | `true` for a page that a script fills in after loading: every look opens the page in pine's browser, without a window, and takes what a reader sees then. See "A page that a script fills in". Not together with `path`, `values`, `method`, `body`, `headers`, `token` or `tokenHeader`. |
273
- | `values` | | For an answer in JSON: several fields at once, each under a name of your own: `{ "price": "data.price", "stock": "data.stock.count" }`. Twenty at most. |
274
- | `method` | `"GET"` | `"POST"` for an address that is asked by sending something. Nothing else: a source only looks. |
275
- | `body` | | With `POST`: what to send, as text. Sent as JSON when it is JSON. Not for secrets. |
276
- | `headers` | | Headers to send, `{ "accept": "application/json" }`. Not for secrets. |
277
- | `token` | | An access token or key, sent as `Authorization: Bearer <token>`. Kept in `.env`. |
278
- | `tokenHeader` | | Send the token as this header instead, as it is: `"x-api-key"`, `"cookie"`. |
279
- | `local` | `false` | The address is on your own network under a name that does not say so (`nas.home`, `ha.lan`). Not needed for an IP address, `localhost` or a `.local` name. |
280
-
281
- | Event | | |
282
- |---|---|---|
283
- | `changed` | `{ url, value, previous, number?, previousNumber? }` | With `path`, `match` or `select`: the value now and before. |
284
- | `changed` | `{ url, values, previous, changed }` | With `values`: all the fields now and before, and the names of those that differ. |
285
- | `changed` | `{ url, added, removed, more? }` | Without: the lines that are new and the lines that are gone, up to 30 of each. |
286
- | `down` | `{ url, error, looks }` | After three failed looks in a row, once. |
287
- | `up` | `{ url }` | When it answers again. |
288
-
289
- Without `path`, `values`, `match` or `select` the whole answer counts. For a page that is its text as a reader
290
- sees it: scripts, styles and markup are left out, so a counter in a script does not report a
291
- change at every look. A page that shows the time still does; give it a `match`.
292
-
293
- #### One element of a page
294
-
295
- `match` finds a value by the text around it; `select` finds it by where it stands in the page,
296
- which holds better when the words around a price change with the language or the season:
297
-
298
- ```json
299
- "fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "select": ".product-detail .price", "every": "30m", "description": "the price of the fridge" }
300
- ```
301
-
302
- A selector is made of tags (`span`), ids (`#total`), classes (`.price`), attributes (`[itemprop]`,
303
- `[itemprop=price]`, also `~=`, `^=`, `$=`, `*=`), put together (`span.price`, `li.item.sale`) and
304
- nested with a space (somewhere inside) or `>` (directly inside); a comma separates alternatives.
305
- Nothing with a colon (`:first-child`, `:not(...)`) and no `+` or `~`: such a selector is refused
306
- with what it takes instead. The first element in the page that matches counts. Many shops also
307
- state the price for machines, which changes less often than the layout:
308
-
309
- ```json
310
- "fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "select": "meta[property=product:price:amount]", "attr": "content" }
311
- ```
312
-
313
- The page is read as the server sends it. What a script fills in after loading is not
314
- there: for such a page set `render`, or watch the address the script itself asks (see "A
315
- page that a script fills in").
316
-
317
- #### A page that a script fills in
318
-
319
- Many pages arrive almost empty, and a script of the page then fetches the price or the state
320
- and writes it in. A plain look reads the page as the server sends it, so it finds the
321
- placeholder ("loading", an empty element, nothing at all) and never the value. With `render`
322
- the look is made by a browser:
323
-
324
- > The price on https://shop.example/fridge-3000 only shows up after the page has loaded. Watch
325
- > it and tell me when it drops.
326
-
327
- ```json
328
- "fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "render": true, "select": ".product-detail .price", "every": "30m", "description": "the price of the fridge" }
329
- ```
330
-
331
- At every look pine starts its browser (the one the browser tools use, which `pine init` and
332
- `pine browser-install` fetch), without a window and with an empty profile, opens the address,
333
- waits until the page has loaded and then until it has stopped asking for things (five seconds
334
- at most), and takes what a reader sees. Then the browser is closed. What counts is chosen as
335
- without `render`:
336
-
337
- | With | What counts |
338
- |---|---|
339
- | `select` | The text of the first element the selector matches, as the browser shows it. The browser reads the selector, so everything CSS has works: `li:nth-child(2) .price`, `.offer:not(.sold-out)`, `h1 + p`. No element matches: the value is `null`. |
340
- | `select` and `attr` | The value of that attribute of the element. |
341
- | `match` | What the regular expression finds in the text a reader sees. The page's source is not looked at. |
342
- | neither | The whole text of the page, line by line; an event names the lines that are new and gone. |
343
-
344
- The events are the same, and an amount arrives as `number` as described below.
345
-
346
- **What it costs.** A plain look is one request and takes a moment. A rendered look starts a
347
- browser, runs the page's scripts and loads everything the page loads: a second or two and a
348
- few hundred megabytes of memory while it lasts, at every look. So:
349
-
350
- - `every` is at least `"2m"` when an agent sets the source up.
351
- - Two browsers run at a time, for all sources together. A look that finds them busy waits.
352
- - A look may take 45 seconds. After that it is ended and counts as failed.
353
- - There is no "nothing new" answer (see below): the page is loaded in full each time.
354
-
355
- **When the plain look is the better choice.** A script that fills a page in gets the value
356
- from somewhere, usually an address that answers JSON. If you can find it (the network tab of
357
- your browser's developer tools shows what the page asks), watch that address without `render`
358
- and with `path`: it is faster, it can be asked every 30 seconds, a number arrives as a number,
359
- it can take a `token`, and it does not break when the shop changes its layout. Use `render`
360
- when there is no such address, when it cannot be asked from outside the page, or when what you
361
- want is what the page shows.
362
-
363
- **What does not go with it.** A set-up with one of these and `render` is refused, with the
364
- reason:
365
-
366
- | Option | Why not |
367
- |---|---|
368
- | `path`, `values` | They read an answer in JSON. There is no page to render. Leave `render` out. |
369
- | `method`, `body` | A rendered page is asked the way a reader's browser asks it. |
370
- | `headers`, `token`, `tokenHeader` | The browser would send them with every request the page makes, to every address the page loads from. So a page behind a login cannot be watched this way. |
371
- | `local: true` | See below: a name that has to be looked up is not opened on your network. |
372
-
373
- **What the page can reach.** The page's own scripts run in that browser, as they would in
374
- yours, and the dialog you confirm says so. What they can reach is narrower than in yours:
375
-
376
- - Every connection of that browser goes through the same guard as the browser tools'
377
- ([security.md](security.md)). The public internet is reachable. Your own network and this
378
- computer are not: not for the page, and not for anything the page loads or its scripts ask.
379
- - One exception, which is the page itself: an address that says by itself that it is local
380
- (`localhost`, a `.local` name, an IP address in a private range) is opened, and the browser
381
- then reaches exactly that host and port and nothing else on your network. The dialog names
382
- it.
383
- - A name that only turns out to be local when it is looked up (`nas.home`, `ha.lan`) is not
384
- opened with `render`, also not with `local: true`: what such a name points at can change,
385
- and a page's scripts would run there. Give the device by its IP address.
386
- - pine's own control port is never opened.
387
- - The look counts only if the browser ends up on the host that was set up. A page that sends
388
- it on to another host is a failed look that names the host.
389
- - The browser keeps nothing from one look to the next (no cookies, no logins), downloads
390
- nothing, and grants the page no permission (camera, location, notifications). A window the
391
- page opens is closed.
392
-
393
- What such a source reports is outside content, like everything an `http` source reports.
394
-
395
- **When a look fails.** A failed look is retried with a growing pause, and `pine sources` shows
396
- since when it fails and why; after three in a row the source reports `down`, and `up` when the
397
- page opens again.
398
-
399
- | The status says | What it means |
400
- |---|---|
401
- | `pine's browser is not installed on this computer` | Type `pine browser-install` into a terminal. `pine init` and `pine update` do it too. |
402
- | `the page did not finish loading within 45 s` | The page is slow, or never finishes. |
403
- | `"select" is not a CSS selector the browser can read` | The selector has a mistake in it. |
404
- | `it sends on to <host>, which is not the address that was confirmed` | The page moved. Set the source up with the address it leads to, if that is the right one. |
405
- | `<host> leads to an address on your own network` | The name points into your network. With `render` only an address that says so itself is opened there. |
406
- | `it answered 403` (or another number) | The site turned the browser away. Some sites do that to a browser without a window. |
407
- | `the browser could not load it (...)` | No connection, an invalid certificate, or the like; the words in brackets are the browser's. |
408
-
409
- `found nothing with the selector` is not a failed look: the page opened and no element matched.
410
- If the value is there in your own browser, the selector is wrong, or the page fills the
411
- element in later than the look waits.
412
-
413
- #### An amount as a number
414
-
415
- A value that reads as a number or an amount, and nothing else, also arrives as `number` (and
416
- the one before as `previousNumber`), so that a subscription can compare it:
417
-
418
- ```json
419
- "subscribe": [{ "source": "fridge", "when": { "payload.number": { "lt": 500 } }, "note": "Tell me: the fridge is under 500 now." }]
420
- ```
421
-
422
- | The value | `number` |
423
- |---|---|
424
- | `12.5`, `"12.5"`, `"12,5"`, `"87 %"`, `"-3,5 °C"`, `"3 GB"` | 12.5, 12.5, 12.5, 87, -3.5, 3 |
425
- | `"$1,299.00"`, `"1.299,00 €"`, `"1 299,00 kr"`, `"1'299.50"` | 1299, 1299, 1299, 1299.5 |
426
- | `"1,299"`, `"1.299"` | 1299 and 1.299: as a program writes numbers |
427
- | `"1.299 €"`, `"1,299 €"` | 1299 and 1.299: next to `€`, `zł`, `kr` or `Kč` a dot groups thousands and a comma is the decimal point |
428
- | `"1.2.3"`, `"12:30"`, `"2026-05-01"`, `"about 5"`, `"sold out"` | none |
429
-
430
- The one case that can be read two ways is a single separator followed by exactly three digits,
431
- and the last two rows are how it is decided. Where that is not what the page means, take the
432
- digits apart with `match` (`"([0-9]+)[.,][0-9]{3} "`) and compare the text.
433
-
434
- #### Several fields, and an address that is asked with POST
435
-
436
- ```json
437
- "fridge": {
438
- "type": "http", "url": "https://shop.example/graphql", "method": "POST",
439
- "body": "{\"query\": \"{ product(id: 7) { price stock { count } } }\"}",
440
- "values": { "price": "data.product.price", "stock": "data.product.stock.count" },
441
- "every": "30m", "description": "price and stock of the fridge"
442
- }
443
- ```
444
-
445
- One event says which of the fields changed: `{ "values": { "price": 479, "stock": 3 },
446
- "previous": { "price": 499, "stock": 3 }, "changed": ["price"] }`. A field that is not in the
447
- answer is `null`. Since these are the values of the JSON itself, numbers are numbers and
448
- `when` compares them directly: `{ "payload.values.stock": { "gt": 0 } }`.
449
-
450
- A `POST` is for asking, and what it sends is shown to you in full before the source is set up:
451
- a source repeats it on a clock, with nobody there.
452
-
453
- #### Nothing new
454
-
455
- An address that names a version of its answer (`ETag`, `Last-Modified`) is asked "anything new
456
- since?" from the second look on, and answers with one line when there is not. That is a look
457
- like any other, only cheaper for both sides. `feed` and `calendar` ask the same way. A look
458
- with `render` loads the page in full every time.
459
-
460
- #### Which addresses it asks
461
-
462
- An address on your own network or this computer is said to be one in the dialog you confirm.
463
- Any other address has to stay on the public internet: if its name one day leads into your
464
- network, the look fails and says so instead of fetching from there.
465
-
466
- A redirect is followed only on the same host. One that leads to another host is a failed look
467
- that says where it leads: you confirmed one address, and the token is for that one. An answer
468
- is read up to 1 MB. A key that is part of the address (`?apikey=...`) is not a secret to pine:
469
- it is stored and shown like the rest of the address.
470
-
471
- Refused when it is set up: a user name or password in the address, `path` and `match`
472
- together, and a header that carries a secret (`authorization`, `cookie`), which belongs into
473
- `token`. Also more than one of `path`, `values`, `match` and `select`; `attr` without
474
- `select`; a selector that cannot be read; a `method` other than `GET` and `POST`; a `body`
475
- without `POST` or longer than 20000 characters; `values` that are empty, more than twenty, or
476
- under a name that is not a plain word. One look may take 20 seconds and follow five redirects.
477
-
478
- ### command
479
-
480
- A command that is run again and again. For what only a command can tell: free disk space, a
481
- build, a branch that moved, the end of a log, whether a machine answers.
482
- A command that keeps running and prints a line whenever something happens is followed instead
483
- of run again and again: see "Following a command that keeps running" below.
484
-
485
- > Tell me when the disk is more than 90% full.
486
-
487
- ```json
488
- "disk": { "type": "command", "run": "df -h /", "match": "([0-9]+)%", "every": "10m", "description": "how full the disk is" }
489
- ```
490
-
491
- | Option | Default | |
492
- |---|---|---|
493
- | `run` | | The command line. bash runs it from the workspace of the agent the source belongs to. |
494
- | `every` | `"1m"` | How often. At least `"5s"` when an agent sets it up. |
495
- | `match` | | Watch only what this regular expression finds in what the command prints. |
496
- | `timeout` | `"30s"` | How long one run may take before it is stopped. |
497
-
498
- Event `changed`: `{ command, code, previousCode?, value, previous }` with `match`, or
499
- `{ command, code, previousCode?, added, removed, more? }` without. `code` is the exit code;
500
- `previousCode` is there when it changed, so `test -f build/done` or `ping -c1 nas` report by
501
- how they end.
502
-
503
- The command runs **like a command in the shell of an agent in `safe` mode**: in the
504
- agent's sandbox, with its private home folder and without your environment, also for an agent
505
- that works in `auto` mode and has your whole account in your conversations (nobody is there to answer for what a
506
- command on a clock does). It reaches the agent's own folders and nothing else: no network
507
- unless `sandbox.network` is set, none of your home folder beyond `sandbox.read`, none of
508
- pine's files. A source of all agents runs as the main agent. You are shown the
509
- command line before it is set up, every time. A command that cannot be started or runs out of
510
- time is a failed look (see below), not an event. `timeout` is at least `"1s"`, and a command
511
- line of more than 2000 characters is refused (put it into a script). If the agent's sandbox is
512
- switched off, the command runs without one, as its shell does.
513
-
514
- #### Following a command that keeps running
515
-
516
- Some commands tell by themselves when something happens: `tail -F` a log, `log stream`,
517
- `docker events`, `kubectl get pods -w`, the client of a broker or a database. With `follow`
518
- the command is started once and kept running, and every line it prints is an event as it
519
- comes. This is the type for every protocol that has a command line client and no source type
520
- of its own.
521
-
522
- > Tell me when the backup log says ERROR.
523
-
524
- ```json
525
- "backup": { "type": "command", "run": "tail -n 0 -F ~/Library/Logs/backup.log", "follow": true, "match": "ERROR (.*)", "description": "errors in the backup log" }
526
- ```
527
-
528
- | Option | Default | |
529
- |---|---|---|
530
- | `follow` | `false` | `true`: start the command once, keep it running, report each line it prints. `every` and `timeout` are not used then. |
531
- | `perMinute` | `30` | With `follow`: report at most this many lines a minute. The rest are counted, and the next line that is reported carries the count as `dropped`. |
532
- | `match` | | With `follow`: only lines in which this regular expression finds something. Its first group is the line's `value`. |
533
-
534
- | Event | | |
535
- |---|---|---|
536
- | `line` | `{ command, line, value?, dropped? }` | One line the command printed. `value` is what `match` found, or the parsed line when it is JSON. |
537
- | `ended` | `{ command, code?, error?, ran }` | The command ended (`ran`: after how many seconds, `error`: the end of what it wrote as errors). It is started again, after a pause that grows while it keeps ending at once. A command that ends the moment it is started is reported once, not at every try. |
538
-
539
- | What to follow | `run` |
540
- |---|---|
541
- | A log file | `tail -n 0 -F /path/app.log` (`-n 0`: only what is written from now on; `-F`: also after the file was rotated) |
542
- | The Mac's own log | `log stream --style ndjson --predicate 'subsystem == "com.apple.TimeMachine"'` |
543
- | Docker | `docker events --format '{{json .}}'` |
544
- | Kubernetes | `kubectl get pods -w -o json`, `kubectl get events -w` |
545
- | systemd's journal, on another machine | `ssh nas journalctl -f -p warning -o json` |
546
- | PostgreSQL `LISTEN` | a few lines of script around a client library that print each notification as a line |
547
- | Redis | `redis-cli --raw subscribe alerts` |
548
- | NATS, Kafka, RabbitMQ | `nats sub 'orders.>'`, `kcat -C -b broker -t topic -o end`, `amqp-consume -q queue cat` |
549
- | A folder on another machine | `ssh nas inotifywait -m -e close_write /volume1/scans` |
550
-
551
- A line is cleaned of the colour codes a terminal would act on and cut at 2000 characters; a
552
- line that is JSON also arrives parsed, so a subscription's `when` can look into it
553
- (`{"payload.value.status": "die"}` for a container that stopped).
554
-
555
- The command runs in the same sandbox as one that is run again and again: the agent's own, as
556
- in `safe` mode. That is why most of the examples above need something from you first: the log
557
- file has to be readable by the agent (`sandbox.read`), and a client that talks to the network
558
- needs `sandbox.network`. The dialog shows the command line, and that it will keep running.
559
-
560
- On Linux `follow` is refused. The sandbox there cannot forbid a running command to make a file
561
- that is read as instructions; pine takes such a file away after a command has ended, and a
562
- followed command does not end. Run it again and again instead (`every`, with a `match`), or
563
- have the command's lines sent to a `syslog` or `webhook` source.
564
-
565
- When the source stops (the last subscriber left, or pine stops), the command and everything
566
- it started is ended with it.
567
-
568
- ### Looking again and again
569
-
570
- `http`, `command`, `feed`, `calendar`, `modbus` and `presence` cannot be told when something
571
- happens; they go and look. Three things hold for every source that does:
572
-
573
- - **The first look is the starting point.** Setting a source up reports nothing. Only what is
574
- different at a later look does.
575
- - **A look that finds nothing new costs nothing.** No event, nothing stored, no model woken.
576
- This is the difference to waking an agent every few minutes to check: an agent's turn is
577
- spent only when there is something to act on.
578
- - **A failed look is not an event.** The pause before the next one doubles with each failure in
579
- a row, up to 15 minutes (or the source's own interval, if that is longer), and goes back to
580
- normal with the first look that works.
581
-
582
- How a polling source is doing is said wherever sources are listed (`pine sources`, `/sources`,
583
- the agent's `sources` tool): `last looked 14:02:11`, or `failing since 14:02:11 (4 looks): it
584
- answered 503`, followed by what it found. What it found is repeated there only when it is a
585
- number or a single word; anything longer is only counted, because that line is read by agents
586
- without the mark that outside content gets.
587
-
588
- Two looks of one source never run at the same time, and each waits a little shorter or longer
589
- than its interval, so that sources set to the same interval do not all fire together.
84
+ ## What all types have in common
85
+
86
+ Every type there is, with its options, its events and an entry to copy, is in
87
+ [source-types.md](source-types.md), which starts with a table of which type is for what.
88
+ Whatever the type:
89
+
90
+ - **The start is the starting point.** What is there when a source starts (the entries of a
91
+ feed, the mail in a mailbox, the state of a device) is not reported. Only what comes after is.
92
+ - **What is reported is bounded.** A text is cut to a few thousand characters, and most types
93
+ that are told things report at most so many events a minute (`perMinute` where it can be
94
+ set); what was left out is counted in the next event's `dropped`.
95
+ - **An address is the one you confirmed.** A public address has to stay on the public
96
+ internet: if its name one day leads into your network, the source fails and says so. A local
97
+ address is said to be local in the dialog. A redirect to another host is not followed, and a
98
+ token goes only to the host it was given for.
99
+ - **A source that also sends, sends only where you listed.** `sendTo` of `mail`, `slack`,
100
+ `matrix` and `discord`, `publish` of `mqtt`. Without the list nothing can be sent.
101
+ - **How it is doing is said where sources are listed** (`pine sources`, `/sources`, the
102
+ agent's `sources` tool): connected since when, last looked when, failing since when and why.
590
103
 
591
104
  ## Sending through a source
592
105
 
@@ -598,6 +111,9 @@ tool says which can and what each takes:
598
111
  | `telegram` | A message to you, in your chat with the bot, with a picture, a video or a sound file if the agent has one to show | `text`, and optionally `file`: the path of a picture, video or sound file in its workspace |
599
112
  | `mqtt` | A message to a topic on the source's `publish` list, to act on a device | `data`: `{ "topic": "lamp/set", "message": "on" }`, optionally with `retain` and `qos` |
600
113
 
114
+ `mail`, `slack`, `matrix` and `discord` send too, to the addresses, channels and rooms on
115
+ their `sendTo` list; what each takes is in its section of [source-types.md](source-types.md).
116
+
601
117
  > When the print is done, tell me on Telegram and switch the enclosure light off.
602
118
 
603
119
  Where things may go is the source's to decide, not the agent's: Telegram sends to your chat and
@@ -737,7 +253,7 @@ When a conversation subscribes to a source its agent's main conversation already
737
253
  both are woken by each event, and the agent tells you so.
738
254
 
739
255
  `add_source` takes a name, a type, a one-line description and the options of that type, the
740
- same ones the tables above list:
256
+ same ones [source-types.md](source-types.md) lists for it:
741
257
 
742
258
  ```json
743
259
  { "name": "papers", "type": "folder", "description": "what the scanner saves",
@@ -749,6 +265,12 @@ with a note the subscription is made in the same step), `throttle`, `own` (`true
749
265
  you said it is your own device or folder: its events are then trusted like your messages, and
750
266
  you are always asked) and `ask` (secret options you are to type into the dialog).
751
267
 
268
+ An agent is told seven everyday types with their options: `folder`, `mqtt`, `webhook`, `ftp`,
269
+ `telegram`, `http` and `command`. The other types it knows by name and purpose, and what one
270
+ takes it looks up in this manual (a search for "source type" and the name finds a part made
271
+ from the type's own description), or learns from the answer to a call that leaves a needed
272
+ option out.
273
+
752
274
  The type checks them. An option it does not have, a missing one or a wrong kind of value is
753
275
  answered with what the type takes, before anything is written or you are asked anything. The
754
276
  type also says what the source would do, and whether that is something new:
@@ -864,13 +386,39 @@ The same key in `pine.json` is the global scope.
864
386
  | Key | Default | |
865
387
  |---|---|---|
866
388
  | `note` | | What is to be done when woken, and what you want to hear about. Shown with every event. |
867
- | `when` | | Only events that match. Same filter as in [routes](events.md): keys are paths such as `payload.file`, values are compared, or `{ "regex": ... }`, `{ "gt": ... }`, `{ "in": [...] }`. |
389
+ | `when` | | Only events that match: keys are paths such as `payload.file`, values say what each has to be. See "Being woken only for what counts" below. The same filter as in [routes](events.md). |
868
390
  | `throttle` | `"10s"` | After one event, ignore the source for this long. Protects the model from a chatty device; `"0s"` turns it off. |
869
391
  | `debounce` | | Wait for this long without a new event, then handle only the last one. |
870
392
  | `session` | `main` (`on-<source>` with `separate`) | The conversation that listens. Leaving it out is the agent scope. `"new"`: a fresh conversation for every event, closed when it is answered ("A fresh conversation for every event"). |
871
393
  | `separate` | `false` | `true`: do not deliver into a conversation of yours, but into one of its own (`on-<source>`), with the reduced tool set for sources that are not yours. See "Trust". |
872
394
  | `prompt` | | Replaces the standard text for the event. Templates as in routes: `{{payload.file}}`. |
873
395
 
396
+ ### Being woken only for what counts
397
+
398
+ A subscription can carry a filter, `when`, so that the agent's turn is spent only on the
399
+ events that matter. It names fields of the event and what each has to be:
400
+
401
+ ```json
402
+ { "source": "price", "when": { "payload.number": { "lt": 500 } }, "note": "Tell me: the fridge is under 500." }
403
+ ```
404
+
405
+ | In `when` | Holds when the field |
406
+ |---|---|
407
+ | `"payload.kind": "dm"` | equals the value |
408
+ | `{ "in": ["err", "crit"] }` | is one of these |
409
+ | `{ "gt": 90 }`, `{ "gte": 90 }`, `{ "lt": 5 }`, `{ "lte": 5 }` | is a number above, at least, below, at most |
410
+ | `{ "contains": "backup" }` | is a text or a list that contains it |
411
+ | `{ "regex": "\\.pdf$" }` | matches the regular expression |
412
+ | `{ "exists": true }` | is there at all |
413
+ | `{ "ne": "ok" }` | is anything but the value |
414
+
415
+ All fields named have to hold. The agent sets `when` with its `subscribe` tool; in a file it
416
+ is a key of the subscription. Many types also narrow at the source (`match`, `keywords`,
417
+ `topics`, `severity`): what is filtered there is not even an event. `throttle` and `debounce`
418
+ of a subscription keep a source that reports in bursts from waking the agent for each one,
419
+ and a subscription with `"session": "new"` gives every event a conversation of its own
420
+ ("A fresh conversation for every event").
421
+
874
422
  ## What the agent is woken with
875
423
 
876
424
  The event arrives as the next message in the conversation that is subscribed: the one it was
@@ -896,6 +444,20 @@ reported, followed by the agent's answer; if no tab has that conversation open,
896
444
  when you are not looking, the agent uses its `send` tool with a source that can send, Telegram for example: say so in the note
897
445
  or in its `AGENTS.md`.
898
446
 
447
+ ### What an event looks like
448
+
449
+ Every report is an event on pine's bus:
450
+
451
+ | Field | |
452
+ |---|---|
453
+ | `topic` | `source/<name>/<event>`, or `source/<name>@<agent>/<event>` for an agent's own source. `<event>` is the word the type's section in [source-types.md](source-types.md) names: `entry`, `message`, `changed`. |
454
+ | `payload` | The fields that section lists for the event. |
455
+ | `trust` | `"untrusted"` unless the source is marked as yours. The agent reads the event as data, not as instructions. |
456
+ | `ts`, `id` | When it was reported, and its number in the event log. |
457
+
458
+ `pine events --topic 'source/#'` shows them as they come, and `pine events --topic
459
+ 'source/releases/#'` those of one source.
460
+
899
461
  ### Files a source received
900
462
 
901
463
  A camera uploads a picture or a video clip, a scanner a PDF, you send a photo through
@@ -1067,7 +629,7 @@ The start function gets a `source` object:
1067
629
  | `on(pattern, handler)` | Look at events of the system, read only: `agent/+/started`, `approval/requested`. `pattern` is one topic pattern or a list. |
1068
630
  | `approvals` | `ask(summary, why)` puts a question of the source's own to you in pine's UI; `request(summary, why, inputs)` asks with fields you fill in and returns `{ allow, by, values }`; `pending()` and `decide(id, allow, by, always?)` let it pass on your answer to open requests; `choose(id, n, by)` answers an agent's question (a request with `tool: "question"`) with the n-th of the answers it offers in `choices`, counted from 0. |
1069
631
  | `key` | What it is called in events: its name, or `name@agent` for an agent's own. |
1070
- | `poll(every, look)` | Look at something again and again: `look(signal)` runs once at the start and then every `every` (`"30s"`, `"5m"`), never twice at once. Throw from it for a failed look; pine backs off and shows it in the status. See "Looking again and again". |
632
+ | `poll(every, look)` | Look at something again and again: `look(signal)` runs once at the start and then every `every` (`"30s"`, `"5m"`), never twice at once. Throw from it for a failed look; pine backs off and shows it in the status. See "Looking again and again" in [source-types.md](source-types.md). |
1071
633
  | `changed(value, slot?)` | Compare a value with the one of the last look and remember it. Returns `{ previous }` when it differs, nothing when it is the same or the first look. |
1072
634
  | `exec(command, opts?)` | Run a shell command in the sandbox of the agent the source belongs to (the main agent for a source of all agents). Returns `{ code, output, truncated }`; throws when it cannot start or takes longer than `opts.timeoutMs` (30 s). |
1073
635
  | `kv.get`, `kv.set` | Private storage that survives restarts. |