@pmndre/pine 0.35.1 → 0.35.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -6,10 +6,6 @@ and how secrets are kept is in [sources.md](sources.md). This file is the catalo
6
6
  type there is, the options it takes, the events it reports with their fields, and the services
7
7
  and devices each one reaches, with an entry to copy for each.
8
8
 
9
- The seven everyday types (`folder`, `mqtt`, `webhook`, `ftp`, `telegram`, `http`, `command`)
10
- are described in [sources.md](sources.md), "The built-in types". The others are described
11
- here.
12
-
13
9
  ## Which type for what
14
10
 
15
11
  Two kinds of type exist. One **is told**: it listens on a port or holds a connection open, and
@@ -46,6 +42,29 @@ What should happen at a time is not a source but a schedule ([events.md](events.
46
42
  "Schedules"). A `calendar` source reports that an entry was added or moved; "ten minutes
47
43
  before it starts" is a schedule the agent sets when it is told of the entry.
48
44
 
45
+ ### Looking again and again
46
+
47
+ `http`, `command`, `feed`, `calendar`, `modbus` and `presence` cannot be told when something
48
+ happens; they go and look. Three things hold for every source that does:
49
+
50
+ - **The first look is the starting point.** Setting a source up reports nothing. Only what is
51
+ different at a later look does.
52
+ - **A look that finds nothing new costs nothing.** No event, nothing stored, no model woken.
53
+ This is the difference to waking an agent every few minutes to check: an agent's turn is
54
+ spent only when there is something to act on.
55
+ - **A failed look is not an event.** The pause before the next one doubles with each failure in
56
+ a row, up to 15 minutes (or the source's own interval, if that is longer), and goes back to
57
+ normal with the first look that works.
58
+
59
+ How a polling source is doing is said wherever sources are listed (`pine sources`, `/sources`,
60
+ the agent's `sources` tool): `last looked 14:02:11`, or `failing since 14:02:11 (4 looks): it
61
+ answered 503`, followed by what it found. What it found is repeated there only when it is a
62
+ number or a single word; anything longer is only counted, because that line is read by agents
63
+ without the mark that outside content gets.
64
+
65
+ Two looks of one source never run at the same time, and each waits a little shorter or longer
66
+ than its interval, so that sources set to the same interval do not all fire together.
67
+
49
68
  ## Which type for which service
50
69
 
51
70
  | Service or device | Type | How |
@@ -83,99 +102,502 @@ before it starts" is a schedule the agent sets when it is told of the entry.
83
102
 
84
103
  ## How every type is used
85
104
 
86
- ### Ask the agent
87
-
88
105
  Say what you want watched. The agent picks the type, pine shows you what the source will do,
89
106
  and you confirm:
90
107
 
91
108
  > Tell me on Telegram when pmndre/pine has a new release.
92
109
 
93
- The agent calls `add_source` with `type: "feed"`, the address, a description and a note for
94
- itself. The dialog says "read the feed at https://github.com/pmndre/pine/releases.atom
95
- every 15m and report each new entry". A password, token or key is never typed into the conversation: the
96
- agent leaves it out, and pine asks you for it in a dialog of its own and keeps it in `.env`
97
- ([sources.md](sources.md), "Secrets").
110
+ Or write the entry yourself, under `sources` in `pine.json` or in an agent's `agent.json`, with
111
+ the options of the type next to `type`, and subscribe an agent to it. Each type below has an
112
+ entry to copy.
113
+
114
+ What holds whatever the type is in [sources.md](sources.md): the keys every source has
115
+ ("Configuring sources"), what all types have in common, subscribing and its filter `when`
116
+ ("Being woken only for what counts"), how a password or token is kept ("Secrets"), what an
117
+ event looks like, and whose words an event counts as ("Trust").
118
+
119
+ ## The types
120
+
121
+ ### folder
122
+
123
+ A folder: reports every new or changed file once it has stopped growing.
124
+
125
+ ```json
126
+ "scans": { "type": "folder", "dir": "~/Scans", "mask": "*.pdf" }
127
+ ```
128
+
129
+ | Option | Default | |
130
+ |---|---|---|
131
+ | `dir` | | The folder. `~` and paths relative to the configuration file work. |
132
+ | `mask` | | Only files whose name matches, as in a shell: `*.pdf`, `*.{png,jpg}`, `report-??.csv`. Several are separated by commas. Capitals do not matter. |
133
+ | `pattern` | | Only files whose full path matches this regular expression, for what a mask cannot say (`/src/.*\\.ts$`). Capitals do not matter. |
134
+ | `ignore` | version control, dependencies, editor leftovers | Never files whose path matches this one. The default skips `.git`, `node_modules`, `.DS_Store`, build folders, and files ending in `~`, `.swp`, `.tmp`, `.part`. |
135
+ | `settle` | `"1.5s"` | How long a file must stay unchanged before it counts as complete. |
136
+ | `recursive` | `true` | Also folders inside it. |
98
137
 
99
- An agent is told the everyday types with their options. The other types it knows by name and
100
- purpose, and what one takes it looks up in this manual (a search for "source type" and the
101
- name finds a part made from the type's own description), or learns from the answer to a call
102
- that leaves a needed option out.
138
+ Event `added`: `{ path, file, size, images? }`. For jpg, png, webp and gif under 10 MB the
139
+ picture is shown to the agent if its model can look at images.
103
140
 
104
- ### In the configuration
141
+ ### mqtt
105
142
 
106
- A source is an entry under `sources` in `pine.json` (every agent can subscribe) or in an
107
- agent's `agent.json` (only that agent can). The options of the type lie next to `type`:
143
+ An MQTT broker: reports the messages on the topics you name.
108
144
 
109
145
  ```json
110
- "sources": {
111
- "releases": { "type": "feed", "url": "https://github.com/pmndre/pine/releases.atom", "every": "30m", "description": "new releases of pine" }
146
+ "printer": {
147
+ "type": "mqtt", "url": "mqtts://192.168.1.50:8883", "username": "bblp", "password": "${BAMBU_ACCESS_CODE}",
148
+ "topics": ["device/+/report"], "insecureTls": true, "minInterval": "30s",
149
+ "trust": "owner", "description": "status reports of the 3D printer"
112
150
  }
113
151
  ```
114
152
 
115
- The keys every source has, whatever its type (`description`, `agents`, `trust`, `start`,
116
- `enabled`), are in [sources.md](sources.md), "Configuring sources". Then an agent subscribes:
117
- in a conversation ("watch the releases"), or with `pine subscribe main releases`.
153
+ | Option | Default | |
154
+ |---|---|---|
155
+ | `url` | | `mqtt://`, `mqtts://`, `ws://` or `wss://` address of the broker. |
156
+ | `topics` | | One MQTT topic filter or a list. |
157
+ | `publish` | none | Topics an agent may send to, to act on a device: `["zigbee2mqtt/+/set"]`. Without it the source only listens. See "Sending through a source" in [sources.md](sources.md). |
158
+ | `username`, `password`, `clientId` | | As the broker wants them. |
159
+ | `minInterval` | | Drop messages that follow the last one on the same topic sooner than this. Use it for devices that report every second. |
160
+ | `insecureTls` | `false` | Accept a certificate that cannot be verified, as devices with self-made ones need. |
118
161
 
119
- ### What an event looks like
162
+ Event `message`: `{ topic, value }`, where `value` is the parsed JSON, or the text.
120
163
 
121
- Every report is an event on pine's bus:
164
+ The example is a Bambu printer in LAN mode: it is its own broker on port 8883, the user is
165
+ `bblp`, the password is the access code from the printer's screen.
122
166
 
123
- | Field | |
167
+ ### webhook
168
+
169
+ A web address other services can call: `POST http://<pine's control address>/hook/<name>`.
170
+
171
+ ```json
172
+ "github": { "type": "webhook", "secret": "${GITHUB_HOOK_SECRET}", "description": "pushes and issues of my repositories" }
173
+ ```
174
+
175
+ | Option | |
124
176
  |---|---|
125
- | `topic` | `source/<name>/<event>`, or `source/<name>@<agent>/<event>` for an agent's own source. `<event>` is the word in the tables below: `entry`, `message`, `changed`. |
126
- | `payload` | The fields in the tables below. |
127
- | `trust` | `"untrusted"` unless the source is marked as yours. The agent reads the event as data, not as instructions. |
128
- | `ts`, `id` | When it was reported, and its number in the event log. |
177
+ | `secret` | The caller proves itself with it: `Authorization: Bearer <secret>`, or a signature of the body (HMAC SHA-256 in `X-Hub-Signature-256` or `X-Signature`, as GitHub sends it). |
178
+ | `open` | `true` instead of a secret: accept anyone who can reach the port. Only for a port nobody else can reach. |
179
+
180
+ Event `received`: `{ body, headers }`. `body` is the parsed JSON, or `{ text }`. Of the headers
181
+ only a few that say what kind of request it is are kept (`content-type`, `user-agent`,
182
+ `x-github-event` and the like), never the secret.
183
+
184
+ The address has the source's name as it is in the event topics: `/hook/<name>` for a source in
185
+ `pine.json`, `/hook/<name>@<agent>` for one in an agent's own `agent.json` (which is where a
186
+ source that an agent sets up goes, unless it is for all agents).
129
187
 
130
- `pine events --topic 'source/#'` shows them as they come, and `pine events --topic
131
- 'source/releases/#'` those of one source. The agent receives the event as a message with the
132
- note of its subscription in front ([sources.md](sources.md), "What the agent is woken with").
188
+ It needs the control API (`http` in `pine.json`), which listens on `127.0.0.1` by default. A
189
+ service on the internet reaches it only through a tunnel or reverse proxy that you set up. While
190
+ nobody is subscribed the address answers "not found".
133
191
 
134
- ### Being woken only for what counts
192
+ ### ftp
135
193
 
136
- A subscription can carry a filter, `when`, so that the agent's turn is spent only on the
137
- events that matter. It names fields of the event and what each has to be:
194
+ An FTP drop: a small FTP server that accepts uploads and nothing else. For cameras, doorbells,
195
+ scanners and other devices whose way of sending a file is "upload to FTP server".
138
196
 
139
197
  ```json
140
- { "source": "price", "when": { "payload.number": { "lt": 500 } }, "note": "Tell me: the fridge is under 500." }
198
+ "door": { "type": "ftp", "port": 2121, "user": "cam", "password": "${CAMERA_FTP_PASSWORD}", "allow": ["192.168.1.40"], "description": "pictures from the front door camera when it sees motion" }
141
199
  ```
142
200
 
143
- | In `when` | Holds when the field |
201
+ | Option | Default | |
202
+ |---|---|---|
203
+ | `port`, `user`, `password` | | What you enter in the device. The password needs 8 characters or more. When an agent sets the source up, `user` is `device` unless you name one, and a password you do not give is made by pine. |
204
+ | `host` | `"0.0.0.0"` | The address to listen on. Devices on your network have to reach it. |
205
+ | `allow` | everyone | Only these clients: a full address (`"192.168.1.40"`), or a prefix that ends in a dot (`"192.168.2."`). |
206
+ | `pasvPorts` | any | `[from, to]`: the ports for data connections, if a firewall has to know them. |
207
+ | `maxFileMb` | `50` | Largest file accepted. |
208
+ | `keepDays` | `7` | Received files are deleted after this many days. |
209
+
210
+ Event `upload`: `{ file, path, size, from, images? }`. `file` is the name the device gave it,
211
+ `path` is where it lies now (`data/sources/<name>/files/`, or `data/sources/<name>@<agent>/files/`
212
+ for an agent's own source). A picture is shown to the agent with the event. A video or any
213
+ other file is not: the agent reads it from `path` (see "Files a source received" in
214
+ [sources.md](sources.md)).
215
+
216
+ It speaks plain FTP, as these devices do, so the password crosses your network unencrypted. It
217
+ only receives: nothing can be listed, read, changed or deleted through it, and every file lands
218
+ under a name of pine's choosing in the source's own folder. Set the device to passive mode.
219
+
220
+ ### telegram
221
+
222
+ A Telegram bot: you write to it from wherever you are, and the agent that subscribed answers
223
+ there. It is how you reach your agents when you are away, and how they reach you.
224
+
225
+ > Watch Telegram. You are my assistant there, and what you want to tell me while I am away
226
+ > goes to that chat.
227
+
228
+ The agent calls `add_source`. The dialog you confirm has a field for the bot's token: you type
229
+ or paste it there, where it is not shown and does not pass through the agent or its model (see
230
+ "Secrets" in [sources.md](sources.md)). Then the agent tells you the one remaining step, which ties the bot to your account:
231
+
232
+ 1. Open the bot in Telegram and write anything to it.
233
+ 2. The bot answers that account with a six-digit code.
234
+ 3. pine shows a question on your computer that names the account and has a field for the code.
235
+ Type the code from your phone and press enter.
236
+
237
+ From then on that account is the only one the bot listens to. The code is the point: it shows
238
+ that the person at pine's keyboard holds that Telegram account. A stranger who finds the bot
239
+ can make the question appear, but does not see your screen, and you do not have their code; a
240
+ key pressed by accident cannot say yes. An account you deny is not put to you again, and only
241
+ one question is open at a time.
242
+
243
+ Anyone can write to a bot; that is how Telegram works. What they write reaches no agent. Before
244
+ you have paired, an unknown account gets the code message (and a line if the code that was
245
+ typed was wrong) and nothing else. After you have,
246
+ unknown accounts get no answer at all and cause no question.
247
+
248
+ ```json
249
+ "telegram": { "type": "telegram", "token": "${PINE_SOURCE_TELEGRAM_MAIN_TOKEN}", "trust": "owner", "description": "my Telegram" }
250
+ ```
251
+
252
+ | Option | Default | |
253
+ |---|---|---|
254
+ | `token` | | The bot's token. Create a bot by writing `/newbot` to @BotFather in Telegram; it answers with the token. Kept in `.env`. |
255
+ | `allow` | pairing | Telegram user ids that may talk to the bot, if you know yours. Without it the bot is paired as described above. With several ids, what the agent sends unasked and the approval requests go to the first. |
256
+ | `approvals` | `true` | Approval requests are also sent to that chat, with buttons to answer them: Allow, Deny and, where a yes can be remembered for the conversation, Always. `/allow` and `/deny` work too (with the request's id when more than one is open). A question the agent asks with its `question` tool arrives the same way when it offers answers to pick from, with one button for each and one for "No answer"; an answer in your own words, and a question without answers to pick from, you type in the terminal UI. A request that needs a secret typed is never sent there. |
257
+ | `apiBase` | Telegram's | The address of a Bot API server of your own. |
258
+
259
+ Event `message`: `{ text, name, username, chatId, messageId, images? }`. A photo you send (up
260
+ to 10 MB) is shown to the model. The bot listens only in its private chat with you: what is
261
+ written in a group it was added to reaches no agent.
262
+
263
+ What makes it different from the other types:
264
+
265
+ - **It is a conversation.** Your message reaches the agent as your message, not as a report
266
+ about one, in the conversation that subscribed. The answer is sent back to the chat and is
267
+ also in that conversation in the terminal UI, so both show the same thread.
268
+ - **It speaks for you.** Telegram vouches for the account and you proved it is yours, so what
269
+ it writes counts as your own words: full tools, no "outside content" mark. That is why
270
+ pairing takes a code and why the set-up is always put to you.
271
+ - **It can send.** The agent writes to you on its own with the `send` tool, with a picture or
272
+ a video if it has one to show, and without a question to you; see "Sending through a
273
+ source" in [sources.md](sources.md).
274
+ - **Messages are not held back.** The ten-second pause between events that protects a model from
275
+ a chatty device does not apply unless the subscription sets a `throttle` itself.
276
+
277
+ Who the bot listens to is said wherever sources are listed: `pine sources`, `/sources` in the
278
+ terminal UI, and the agent's `sources` tool show "listens to nobody yet", or the account you
279
+ paired with its name and Telegram id. A bot never learns a phone number; the account's id is
280
+ what Telegram vouches for. To pair with another account, remove the source and set it up
281
+ again: the pairing goes with the source.
282
+
283
+ Messages are fetched by long polling: one request that Telegram answers the moment something
284
+ arrives. Nothing listens on a port, so it works behind any router. Only one program can poll a
285
+ bot at a time; a second one makes both miss messages.
286
+
287
+ ### http
288
+
289
+ A web address that is looked at again and again. For what cannot call you: a price, a version,
290
+ a status page, a device on your network that has a state page.
291
+
292
+ > Watch https://shop.example/fridge-3000 and tell me when the price drops below 500.
293
+
294
+ ```json
295
+ "price": { "type": "http", "url": "https://shop.example/fridge-3000", "match": "Price: ([0-9.,]+)", "every": "30m", "description": "the price of the fridge" }
296
+ ```
297
+
298
+ | Option | Default | |
299
+ |---|---|---|
300
+ | `url` | | The address, `https://...` (`http://` for a device on the local network). |
301
+ | `every` | `"5m"` | How often to look. At least `"30s"` when an agent sets it up, and at least `"2m"` with `render`. |
302
+ | `path` | | For an answer in JSON: watch only this field, `"data.price"` or `"items.0.state"`. `path`, `values`, `match` and `select` are four ways to say what counts in the answer; a source takes one of them. |
303
+ | `match` | | For a page or text: watch only what this regular expression finds, its first group if it has one. It is looked for in what a reader of the page sees, then in the page's source. |
304
+ | `select` | | For a page: watch only the text of the first element this CSS selector matches, `".product .price"` or `"#stock > span"`. With `render` it is any selector a browser reads. |
305
+ | `attr` | | With `select`: the value of this attribute of the element instead of its text, `"content"` for a meta tag, `"data-price"`. |
306
+ | `render` | `false` | `true` for a page that a script fills in after loading: every look opens the page in pine's browser, without a window, and takes what a reader sees then. See "A page that a script fills in". Not together with `path`, `values`, `method`, `body`, `headers`, `token` or `tokenHeader`. |
307
+ | `values` | | For an answer in JSON: several fields at once, each under a name of your own: `{ "price": "data.price", "stock": "data.stock.count" }`. Twenty at most. |
308
+ | `method` | `"GET"` | `"POST"` for an address that is asked by sending something. Nothing else: a source only looks. |
309
+ | `body` | | With `POST`: what to send, as text. Sent as JSON when it is JSON. Not for secrets. |
310
+ | `headers` | | Headers to send, `{ "accept": "application/json" }`. Not for secrets. |
311
+ | `token` | | An access token or key, sent as `Authorization: Bearer <token>`. Kept in `.env`. |
312
+ | `tokenHeader` | | Send the token as this header instead, as it is: `"x-api-key"`, `"cookie"`. |
313
+ | `local` | `false` | The address is on your own network under a name that does not say so (`nas.home`, `ha.lan`). Not needed for an IP address, `localhost` or a `.local` name. |
314
+
315
+ | Event | | |
316
+ |---|---|---|
317
+ | `changed` | `{ url, value, previous, number?, previousNumber? }` | With `path`, `match` or `select`: the value now and before. |
318
+ | `changed` | `{ url, values, previous, changed }` | With `values`: all the fields now and before, and the names of those that differ. |
319
+ | `changed` | `{ url, added, removed, more? }` | Without: the lines that are new and the lines that are gone, up to 30 of each. |
320
+ | `down` | `{ url, error, looks }` | After three failed looks in a row, once. |
321
+ | `up` | `{ url }` | When it answers again. |
322
+
323
+ Without `path`, `values`, `match` or `select` the whole answer counts. For a page that is its text as a reader
324
+ sees it: scripts, styles and markup are left out, so a counter in a script does not report a
325
+ change at every look. A page that shows the time still does; give it a `match`.
326
+
327
+ #### One element of a page
328
+
329
+ `match` finds a value by the text around it; `select` finds it by where it stands in the page,
330
+ which holds better when the words around a price change with the language or the season:
331
+
332
+ ```json
333
+ "fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "select": ".product-detail .price", "every": "30m", "description": "the price of the fridge" }
334
+ ```
335
+
336
+ A selector is made of tags (`span`), ids (`#total`), classes (`.price`), attributes (`[itemprop]`,
337
+ `[itemprop=price]`, also `~=`, `^=`, `$=`, `*=`), put together (`span.price`, `li.item.sale`) and
338
+ nested with a space (somewhere inside) or `>` (directly inside); a comma separates alternatives.
339
+ Nothing with a colon (`:first-child`, `:not(...)`) and no `+` or `~`: such a selector is refused
340
+ with what it takes instead. The first element in the page that matches counts. Many shops also
341
+ state the price for machines, which changes less often than the layout:
342
+
343
+ ```json
344
+ "fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "select": "meta[property=product:price:amount]", "attr": "content" }
345
+ ```
346
+
347
+ The page is read as the server sends it. What a script fills in after loading is not
348
+ there: for such a page set `render`, or watch the address the script itself asks (see "A
349
+ page that a script fills in").
350
+
351
+ #### A page that a script fills in
352
+
353
+ Many pages arrive almost empty, and a script of the page then fetches the price or the state
354
+ and writes it in. A plain look reads the page as the server sends it, so it finds the
355
+ placeholder ("loading", an empty element, nothing at all) and never the value. With `render`
356
+ the look is made by a browser:
357
+
358
+ > The price on https://shop.example/fridge-3000 only shows up after the page has loaded. Watch
359
+ > it and tell me when it drops.
360
+
361
+ ```json
362
+ "fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "render": true, "select": ".product-detail .price", "every": "30m", "description": "the price of the fridge" }
363
+ ```
364
+
365
+ At every look pine starts its browser (the one the browser tools use, which `pine init` and
366
+ `pine browser-install` fetch), without a window and with an empty profile, opens the address,
367
+ waits until the page has loaded and then until it has stopped asking for things (five seconds
368
+ at most), and takes what a reader sees. Then the browser is closed. What counts is chosen as
369
+ without `render`:
370
+
371
+ | With | What counts |
372
+ |---|---|
373
+ | `select` | The text of the first element the selector matches, as the browser shows it. The browser reads the selector, so everything CSS has works: `li:nth-child(2) .price`, `.offer:not(.sold-out)`, `h1 + p`. No element matches: the value is `null`. |
374
+ | `select` and `attr` | The value of that attribute of the element. |
375
+ | `match` | What the regular expression finds in the text a reader sees. The page's source is not looked at. |
376
+ | neither | The whole text of the page, line by line; an event names the lines that are new and gone. |
377
+
378
+ The events are the same, and an amount arrives as `number` as described below.
379
+
380
+ **What it costs.** A plain look is one request and takes a moment. A rendered look starts a
381
+ browser, runs the page's scripts and loads everything the page loads: a second or two and a
382
+ few hundred megabytes of memory while it lasts, at every look. So:
383
+
384
+ - `every` is at least `"2m"` when an agent sets the source up.
385
+ - Two browsers run at a time, for all sources together. A look that finds them busy waits.
386
+ - A look may take 45 seconds. After that it is ended and counts as failed.
387
+ - There is no "nothing new" answer (see below): the page is loaded in full each time.
388
+
389
+ **When the plain look is the better choice.** A script that fills a page in gets the value
390
+ from somewhere, usually an address that answers JSON. If you can find it (the network tab of
391
+ your browser's developer tools shows what the page asks), watch that address without `render`
392
+ and with `path`: it is faster, it can be asked every 30 seconds, a number arrives as a number,
393
+ it can take a `token`, and it does not break when the shop changes its layout. Use `render`
394
+ when there is no such address, when it cannot be asked from outside the page, or when what you
395
+ want is what the page shows.
396
+
397
+ **What does not go with it.** A set-up with one of these and `render` is refused, with the
398
+ reason:
399
+
400
+ | Option | Why not |
144
401
  |---|---|
145
- | `"payload.kind": "dm"` | equals the value |
146
- | `{ "in": ["err", "crit"] }` | is one of these |
147
- | `{ "gt": 90 }`, `{ "gte": 90 }`, `{ "lt": 5 }`, `{ "lte": 5 }` | is a number above, at least, below, at most |
148
- | `{ "contains": "backup" }` | is a text or a list that contains it |
149
- | `{ "regex": "\\.pdf$" }` | matches the regular expression |
150
- | `{ "exists": true }` | is there at all |
151
- | `{ "ne": "ok" }` | is anything but the value |
152
-
153
- All fields named have to hold. The agent sets `when` with its `subscribe` tool; in a file it
154
- is a key of the subscription. Many types also narrow at the source (`match`, `keywords`,
155
- `topics`, `severity`): what is filtered there is not even an event. `throttle` and `debounce`
156
- of a subscription keep a source that reports in bursts from waking the agent for each one,
157
- and a subscription with `"session": "new"` gives every event a conversation of its own
158
- ([sources.md](sources.md), "A fresh conversation for every event").
159
-
160
- ### What all types have in common
161
-
162
- - **Nothing runs until somebody is subscribed.** A configured source opens no port and no
163
- connection. The first subscription starts it, the last one to end stops it.
164
- - **The start is the starting point.** What is there when a source starts (the entries of a
165
- feed, the mail in a mailbox, the state of a device) is not reported. Only what comes after is.
166
- - **What is reported is bounded.** A text is cut to a few thousand characters, and most types
167
- that are told things report at most so many events a minute (`perMinute` where it can be
168
- set); what was left out is counted in the next event's `dropped`.
169
- - **An address is the one you confirmed.** A public address has to stay on the public
170
- internet: if its name one day leads into your network, the source fails and says so. A local
171
- address is said to be local in the dialog. A redirect to another host is not followed, and a
172
- token goes only to the host it was given for.
173
- - **A source that also sends, sends only where you listed.** `sendTo` of `mail`, `slack`,
174
- `matrix` and `discord`, `publish` of `mqtt`. Without the list nothing can be sent.
175
- - **How it is doing is said where sources are listed** (`pine sources`, `/sources`, the
176
- agent's `sources` tool): connected since when, last looked when, failing since when and why.
402
+ | `path`, `values` | They read an answer in JSON. There is no page to render. Leave `render` out. |
403
+ | `method`, `body` | A rendered page is asked the way a reader's browser asks it. |
404
+ | `headers`, `token`, `tokenHeader` | The browser would send them with every request the page makes, to every address the page loads from. So a page behind a login cannot be watched this way. |
405
+ | `local: true` | See below: a name that has to be looked up is not opened on your network. |
406
+
407
+ **What the page can reach.** The page's own scripts run in that browser, as they would in
408
+ yours, and the dialog you confirm says so. What they can reach is narrower than in yours:
409
+
410
+ - Every connection of that browser goes through the same guard as the browser tools'
411
+ ([security.md](security.md)). The public internet is reachable. Your own network and this
412
+ computer are not: not for the page, and not for anything the page loads or its scripts ask.
413
+ - One exception, which is the page itself: an address that says by itself that it is local
414
+ (`localhost`, a `.local` name, an IP address in a private range) is opened, and the browser
415
+ then reaches exactly that host and port and nothing else on your network. The dialog names
416
+ it.
417
+ - A name that only turns out to be local when it is looked up (`nas.home`, `ha.lan`) is not
418
+ opened with `render`, also not with `local: true`: what such a name points at can change,
419
+ and a page's scripts would run there. Give the device by its IP address.
420
+ - pine's own control port is never opened.
421
+ - The look counts only if the browser ends up on the host that was set up. A page that sends
422
+ it on to another host is a failed look that names the host.
423
+ - The browser keeps nothing from one look to the next (no cookies, no logins), downloads
424
+ nothing, and grants the page no permission (camera, location, notifications). A window the
425
+ page opens is closed.
426
+
427
+ What such a source reports is outside content, like everything an `http` source reports.
428
+
429
+ **When a look fails.** A failed look is retried with a growing pause, and `pine sources` shows
430
+ since when it fails and why; after three in a row the source reports `down`, and `up` when the
431
+ page opens again.
432
+
433
+ | The status says | What it means |
434
+ |---|---|
435
+ | `pine's browser is not installed on this computer` | Type `pine browser-install` into a terminal. `pine init` and `pine update` do it too. |
436
+ | `the page did not finish loading within 45 s` | The page is slow, or never finishes. |
437
+ | `"select" is not a CSS selector the browser can read` | The selector has a mistake in it. |
438
+ | `it sends on to <host>, which is not the address that was confirmed` | The page moved. Set the source up with the address it leads to, if that is the right one. |
439
+ | `<host> leads to an address on your own network` | The name points into your network. With `render` only an address that says so itself is opened there. |
440
+ | `it answered 403` (or another number) | The site turned the browser away. Some sites do that to a browser without a window. |
441
+ | `the browser could not load it (...)` | No connection, an invalid certificate, or the like; the words in brackets are the browser's. |
177
442
 
178
- ## The types
443
+ `found nothing with the selector` is not a failed look: the page opened and no element matched.
444
+ If the value is there in your own browser, the selector is wrong, or the page fills the
445
+ element in later than the look waits.
446
+
447
+ #### An amount as a number
448
+
449
+ A value that reads as a number or an amount, and nothing else, also arrives as `number` (and
450
+ the one before as `previousNumber`), so that a subscription can compare it:
451
+
452
+ ```json
453
+ "subscribe": [{ "source": "fridge", "when": { "payload.number": { "lt": 500 } }, "note": "Tell me: the fridge is under 500 now." }]
454
+ ```
455
+
456
+ | The value | `number` |
457
+ |---|---|
458
+ | `12.5`, `"12.5"`, `"12,5"`, `"87 %"`, `"-3,5 °C"`, `"3 GB"` | 12.5, 12.5, 12.5, 87, -3.5, 3 |
459
+ | `"$1,299.00"`, `"1.299,00 €"`, `"1 299,00 kr"`, `"1'299.50"` | 1299, 1299, 1299, 1299.5 |
460
+ | `"1,299"`, `"1.299"` | 1299 and 1.299: as a program writes numbers |
461
+ | `"1.299 €"`, `"1,299 €"` | 1299 and 1.299: next to `€`, `zł`, `kr` or `Kč` a dot groups thousands and a comma is the decimal point |
462
+ | `"1.2.3"`, `"12:30"`, `"2026-05-01"`, `"about 5"`, `"sold out"` | none |
463
+
464
+ The one case that can be read two ways is a single separator followed by exactly three digits,
465
+ and the last two rows are how it is decided. Where that is not what the page means, take the
466
+ digits apart with `match` (`"([0-9]+)[.,][0-9]{3} "`) and compare the text.
467
+
468
+ #### Several fields, and an address that is asked with POST
469
+
470
+ ```json
471
+ "fridge": {
472
+ "type": "http", "url": "https://shop.example/graphql", "method": "POST",
473
+ "body": "{\"query\": \"{ product(id: 7) { price stock { count } } }\"}",
474
+ "values": { "price": "data.product.price", "stock": "data.product.stock.count" },
475
+ "every": "30m", "description": "price and stock of the fridge"
476
+ }
477
+ ```
478
+
479
+ One event says which of the fields changed: `{ "values": { "price": 479, "stock": 3 },
480
+ "previous": { "price": 499, "stock": 3 }, "changed": ["price"] }`. A field that is not in the
481
+ answer is `null`. Since these are the values of the JSON itself, numbers are numbers and
482
+ `when` compares them directly: `{ "payload.values.stock": { "gt": 0 } }`.
483
+
484
+ A `POST` is for asking, and what it sends is shown to you in full before the source is set up:
485
+ a source repeats it on a clock, with nobody there.
486
+
487
+ #### Nothing new
488
+
489
+ An address that names a version of its answer (`ETag`, `Last-Modified`) is asked "anything new
490
+ since?" from the second look on, and answers with one line when there is not. That is a look
491
+ like any other, only cheaper for both sides. `feed` and `calendar` ask the same way. A look
492
+ with `render` loads the page in full every time.
493
+
494
+ #### Which addresses it asks
495
+
496
+ An address on your own network or this computer is said to be one in the dialog you confirm.
497
+ Any other address has to stay on the public internet: if its name one day leads into your
498
+ network, the look fails and says so instead of fetching from there.
499
+
500
+ A redirect is followed only on the same host. One that leads to another host is a failed look
501
+ that says where it leads: you confirmed one address, and the token is for that one. An answer
502
+ is read up to 1 MB. A key that is part of the address (`?apikey=...`) is not a secret to pine:
503
+ it is stored and shown like the rest of the address.
504
+
505
+ Refused when it is set up: a user name or password in the address, `path` and `match`
506
+ together, and a header that carries a secret (`authorization`, `cookie`), which belongs into
507
+ `token`. Also more than one of `path`, `values`, `match` and `select`; `attr` without
508
+ `select`; a selector that cannot be read; a `method` other than `GET` and `POST`; a `body`
509
+ without `POST` or longer than 20000 characters; `values` that are empty, more than twenty, or
510
+ under a name that is not a plain word. One look may take 20 seconds and follow five redirects.
511
+
512
+ ### command
513
+
514
+ A command that is run again and again. For what only a command can tell: free disk space, a
515
+ build, a branch that moved, the end of a log, whether a machine answers.
516
+ A command that keeps running and prints a line whenever something happens is followed instead
517
+ of run again and again: see "Following a command that keeps running" below.
518
+
519
+ > Tell me when the disk is more than 90% full.
520
+
521
+ ```json
522
+ "disk": { "type": "command", "run": "df -h /", "match": "([0-9]+)%", "every": "10m", "description": "how full the disk is" }
523
+ ```
524
+
525
+ | Option | Default | |
526
+ |---|---|---|
527
+ | `run` | | The command line. bash runs it from the workspace of the agent the source belongs to. |
528
+ | `every` | `"1m"` | How often. At least `"5s"` when an agent sets it up. |
529
+ | `match` | | Watch only what this regular expression finds in what the command prints. |
530
+ | `timeout` | `"30s"` | How long one run may take before it is stopped. |
531
+
532
+ Event `changed`: `{ command, code, previousCode?, value, previous }` with `match`, or
533
+ `{ command, code, previousCode?, added, removed, more? }` without. `code` is the exit code;
534
+ `previousCode` is there when it changed, so `test -f build/done` or `ping -c1 nas` report by
535
+ how they end.
536
+
537
+ The command runs **like a command in the shell of an agent in `safe` mode**: in the
538
+ agent's sandbox, with its private home folder and without your environment, also for an agent
539
+ that works in `auto` mode and has your whole account in your conversations (nobody is there to answer for what a
540
+ command on a clock does). It reaches the agent's own folders and nothing else: no network
541
+ unless `sandbox.network` is set, none of your home folder beyond `sandbox.read`, none of
542
+ pine's files. A source of all agents runs as the main agent. You are shown the
543
+ command line before it is set up, every time. A command that cannot be started or runs out of
544
+ time is a failed look (see "Looking again and again"), not an event. `timeout` is at least `"1s"`, and a command
545
+ line of more than 2000 characters is refused (put it into a script). If the agent's sandbox is
546
+ switched off, the command runs without one, as its shell does.
547
+
548
+ #### Following a command that keeps running
549
+
550
+ Some commands tell by themselves when something happens: `tail -F` a log, `log stream`,
551
+ `docker events`, `kubectl get pods -w`, the client of a broker or a database. With `follow`
552
+ the command is started once and kept running, and every line it prints is an event as it
553
+ comes. This is the type for every protocol that has a command line client and no source type
554
+ of its own.
555
+
556
+ > Tell me when the backup log says ERROR.
557
+
558
+ ```json
559
+ "backup": { "type": "command", "run": "tail -n 0 -F ~/Library/Logs/backup.log", "follow": true, "match": "ERROR (.*)", "description": "errors in the backup log" }
560
+ ```
561
+
562
+ | Option | Default | |
563
+ |---|---|---|
564
+ | `follow` | `false` | `true`: start the command once, keep it running, report each line it prints. `every` and `timeout` are not used then. |
565
+ | `perMinute` | `30` | With `follow`: report at most this many lines a minute. The rest are counted, and the next line that is reported carries the count as `dropped`. |
566
+ | `match` | | With `follow`: only lines in which this regular expression finds something. Its first group is the line's `value`. |
567
+
568
+ | Event | | |
569
+ |---|---|---|
570
+ | `line` | `{ command, line, value?, dropped? }` | One line the command printed. `value` is what `match` found, or the parsed line when it is JSON. |
571
+ | `ended` | `{ command, code?, error?, ran }` | The command ended (`ran`: after how many seconds, `error`: the end of what it wrote as errors). It is started again, after a pause that grows while it keeps ending at once. A command that ends the moment it is started is reported once, not at every try. |
572
+
573
+ | What to follow | `run` |
574
+ |---|---|
575
+ | A log file | `tail -n 0 -F /path/app.log` (`-n 0`: only what is written from now on; `-F`: also after the file was rotated) |
576
+ | The Mac's own log | `log stream --style ndjson --predicate 'subsystem == "com.apple.TimeMachine"'` |
577
+ | Docker | `docker events --format '{{json .}}'` |
578
+ | Kubernetes | `kubectl get pods -w -o json`, `kubectl get events -w` |
579
+ | systemd's journal, on another machine | `ssh nas journalctl -f -p warning -o json` |
580
+ | PostgreSQL `LISTEN` | a few lines of script around a client library that print each notification as a line |
581
+ | Redis | `redis-cli --raw subscribe alerts` |
582
+ | NATS, Kafka, RabbitMQ | `nats sub 'orders.>'`, `kcat -C -b broker -t topic -o end`, `amqp-consume -q queue cat` |
583
+ | A folder on another machine | `ssh nas inotifywait -m -e close_write /volume1/scans` |
584
+
585
+ A line is cleaned of the colour codes a terminal would act on and cut at 2000 characters; a
586
+ line that is JSON also arrives parsed, so a subscription's `when` can look into it
587
+ (`{"payload.value.status": "die"}` for a container that stopped).
588
+
589
+ The command runs in the same sandbox as one that is run again and again: the agent's own, as
590
+ in `safe` mode. That is why most of the examples above need something from you first: the log
591
+ file has to be readable by the agent (`sandbox.read`), and a client that talks to the network
592
+ needs `sandbox.network`. The dialog shows the command line, and that it will keep running.
593
+
594
+ On Linux `follow` is refused. The sandbox there cannot forbid a running command to make a file
595
+ that is read as instructions; pine takes such a file away after a command has ended, and a
596
+ followed command does not end. Run it again and again instead (`every`, with a `match`), or
597
+ have the command's lines sent to a `syslog` or `webhook` source.
598
+
599
+ When the source stops (the last subscriber left, or pine stops), the command and everything
600
+ it started is ended with it.
179
601
 
180
602
  ### feed
181
603
 
@@ -552,7 +974,7 @@ turned app passwords off.
552
974
  the addresses on `sendTo` and nobody else; the type checks that itself, whatever a call names.
553
975
  With one address on the list, `to` can be left out. Mail can reach anybody, so unlike a
554
976
  Telegram message to you it counts as information leaving: after outside content the agent asks
555
- before it sends (see "Sending through a source"). The text goes as plain text; no files.
977
+ before it sends (see "Sending through a source" in [sources.md](sources.md)). The text goes as plain text; no files.
556
978
 
557
979
  Refused when it is set up: a `host` with a scheme, port or path in it, `"security": "none"` for
558
980
  a server that is not local, a `from` entry that is neither an address nor `@domain`, `sendTo`
@@ -893,7 +1315,7 @@ answered the moment something arrives, so nothing listens on a port.
893
1315
  | `mentions` | | User ids (`@paul:matrix.org`): only messages that mention one of them. |
894
1316
  | `keywords` | | Only messages that contain one of these words. Capitals do not matter. |
895
1317
  | `ignore` | | User ids that are never reported. What the account itself writes never is. |
896
- | `sendTo` | none | Room ids an agent may write to. Without it the source only listens. See "Sending through a source". |
1318
+ | `sendTo` | none | Room ids an agent may write to. Without it the source only listens. See "Sending through a source" in [sources.md](sources.md). |
897
1319
  | `local` | `false` | The homeserver is on your own network under a name that does not say so. |
898
1320
 
899
1321
  Event `message`: `{ text, user, room, roomName?, kind, id, mention?, keyword?, dropped? }`.
@@ -952,7 +1374,7 @@ to Discord's gateway, so nothing listens on a port.
952
1374
  | `keywords` | | In servers, only messages that contain one of these words. Capitals do not matter. With `mentions` as well, either is enough. |
953
1375
  | `dms` | `true` | Direct messages to the bot count, without a mention or a keyword. |
954
1376
  | `bots` | `false` | What other bots and webhooks write counts too. |
955
- | `sendTo` | none | Channel ids an agent may write to. Without it the source only listens. See "Sending through a source". |
1377
+ | `sendTo` | none | Channel ids an agent may write to. Without it the source only listens. See "Sending through a source" in [sources.md](sources.md). |
956
1378
  | `apiBase` | `https://discord.com/api/v10` | A Discord-compatible server of your own: reached over https, or on this machine. |
957
1379
 
958
1380
  Ids are long numbers. With "Developer Mode" on (User Settings, Advanced), a right click on a
@@ -1440,10 +1862,10 @@ On Linux and Windows a `system` source cannot be set up. What a command can tell
1440
1862
 
1441
1863
  | Wanted | Instead |
1442
1864
  |---|---|
1443
- | A page that a script fills in after loading | It has no type of its own: `http` with `render` has a browser make the look ([sources.md](sources.md), "A page that a script fills in"). Where the page's script asks an address that answers JSON, watching that with `path` is lighter. |
1865
+ | A page that a script fills in after loading | It has no type of its own: `http` with `render` has a browser make the look (see "A page that a script fills in" under `http`). Where the page's script asks an address that answers JSON, watching that with `path` is lighter. |
1444
1866
  | A calendar's "ten minutes before" | A schedule the agent sets when `calendar` reports the entry ([events.md](events.md), "Schedules"). |
1445
1867
  | Bluetooth sensors and beacons | A bridge that publishes them to MQTT (Theengs Gateway, OpenMQTTGateway), and the `mqtt` type. |
1446
1868
  | iMessage, WhatsApp | No type. Signal works through signal-cli and `stream`; Telegram, Slack, Matrix and Discord have types. |
1447
1869
  | KNX, Z-Wave, Zigbee, HomeKit, Matter | Through the hub that speaks them (Home Assistant, zigbee2mqtt, zwave-js-ui), by `mqtt` or `stream`. |
1448
- | Something with a command line client and nothing else | `command` with `follow` ([sources.md](sources.md), "Following a command that keeps running"). |
1870
+ | Something with a command line client and nothing else | `command` with `follow` (see "Following a command that keeps running" under `command`). |
1449
1871
  | Something of your own | A source type of your own is a small module ([sources.md](sources.md), "A source type of your own"); the main agent can write and register one when you ask it. |