@pmndre/pine 0.35.0 → 0.35.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/changes.json +12 -15
- package/docs/SKILL.md +4 -3
- package/docs/references/events.md +1 -1
- package/docs/references/operations.md +1 -1
- package/docs/references/setup.md +1 -1
- package/docs/references/source-types.md +498 -76
- package/docs/references/sources.md +75 -513
- package/docs/references/tui.md +4 -2
- package/package.json +1 -1
- package/src/cli.js +1 -1
- package/src/manual.d.ts +1 -1
- package/src/tools.js +1 -1
- package/src/tui/app.js +33 -8
|
@@ -6,10 +6,6 @@ and how secrets are kept is in [sources.md](sources.md). This file is the catalo
|
|
|
6
6
|
type there is, the options it takes, the events it reports with their fields, and the services
|
|
7
7
|
and devices each one reaches, with an entry to copy for each.
|
|
8
8
|
|
|
9
|
-
The seven everyday types (`folder`, `mqtt`, `webhook`, `ftp`, `telegram`, `http`, `command`)
|
|
10
|
-
are described in [sources.md](sources.md), "The built-in types". The others are described
|
|
11
|
-
here.
|
|
12
|
-
|
|
13
9
|
## Which type for what
|
|
14
10
|
|
|
15
11
|
Two kinds of type exist. One **is told**: it listens on a port or holds a connection open, and
|
|
@@ -46,6 +42,29 @@ What should happen at a time is not a source but a schedule ([events.md](events.
|
|
|
46
42
|
"Schedules"). A `calendar` source reports that an entry was added or moved; "ten minutes
|
|
47
43
|
before it starts" is a schedule the agent sets when it is told of the entry.
|
|
48
44
|
|
|
45
|
+
### Looking again and again
|
|
46
|
+
|
|
47
|
+
`http`, `command`, `feed`, `calendar`, `modbus` and `presence` cannot be told when something
|
|
48
|
+
happens; they go and look. Three things hold for every source that does:
|
|
49
|
+
|
|
50
|
+
- **The first look is the starting point.** Setting a source up reports nothing. Only what is
|
|
51
|
+
different at a later look does.
|
|
52
|
+
- **A look that finds nothing new costs nothing.** No event, nothing stored, no model woken.
|
|
53
|
+
This is the difference to waking an agent every few minutes to check: an agent's turn is
|
|
54
|
+
spent only when there is something to act on.
|
|
55
|
+
- **A failed look is not an event.** The pause before the next one doubles with each failure in
|
|
56
|
+
a row, up to 15 minutes (or the source's own interval, if that is longer), and goes back to
|
|
57
|
+
normal with the first look that works.
|
|
58
|
+
|
|
59
|
+
How a polling source is doing is said wherever sources are listed (`pine sources`, `/sources`,
|
|
60
|
+
the agent's `sources` tool): `last looked 14:02:11`, or `failing since 14:02:11 (4 looks): it
|
|
61
|
+
answered 503`, followed by what it found. What it found is repeated there only when it is a
|
|
62
|
+
number or a single word; anything longer is only counted, because that line is read by agents
|
|
63
|
+
without the mark that outside content gets.
|
|
64
|
+
|
|
65
|
+
Two looks of one source never run at the same time, and each waits a little shorter or longer
|
|
66
|
+
than its interval, so that sources set to the same interval do not all fire together.
|
|
67
|
+
|
|
49
68
|
## Which type for which service
|
|
50
69
|
|
|
51
70
|
| Service or device | Type | How |
|
|
@@ -83,99 +102,502 @@ before it starts" is a schedule the agent sets when it is told of the entry.
|
|
|
83
102
|
|
|
84
103
|
## How every type is used
|
|
85
104
|
|
|
86
|
-
### Ask the agent
|
|
87
|
-
|
|
88
105
|
Say what you want watched. The agent picks the type, pine shows you what the source will do,
|
|
89
106
|
and you confirm:
|
|
90
107
|
|
|
91
108
|
> Tell me on Telegram when pmndre/pine has a new release.
|
|
92
109
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
110
|
+
Or write the entry yourself, under `sources` in `pine.json` or in an agent's `agent.json`, with
|
|
111
|
+
the options of the type next to `type`, and subscribe an agent to it. Each type below has an
|
|
112
|
+
entry to copy.
|
|
113
|
+
|
|
114
|
+
What holds whatever the type is in [sources.md](sources.md): the keys every source has
|
|
115
|
+
("Configuring sources"), what all types have in common, subscribing and its filter `when`
|
|
116
|
+
("Being woken only for what counts"), how a password or token is kept ("Secrets"), what an
|
|
117
|
+
event looks like, and whose words an event counts as ("Trust").
|
|
118
|
+
|
|
119
|
+
## The types
|
|
120
|
+
|
|
121
|
+
### folder
|
|
122
|
+
|
|
123
|
+
A folder: reports every new or changed file once it has stopped growing.
|
|
124
|
+
|
|
125
|
+
```json
|
|
126
|
+
"scans": { "type": "folder", "dir": "~/Scans", "mask": "*.pdf" }
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
| Option | Default | |
|
|
130
|
+
|---|---|---|
|
|
131
|
+
| `dir` | | The folder. `~` and paths relative to the configuration file work. |
|
|
132
|
+
| `mask` | | Only files whose name matches, as in a shell: `*.pdf`, `*.{png,jpg}`, `report-??.csv`. Several are separated by commas. Capitals do not matter. |
|
|
133
|
+
| `pattern` | | Only files whose full path matches this regular expression, for what a mask cannot say (`/src/.*\\.ts$`). Capitals do not matter. |
|
|
134
|
+
| `ignore` | version control, dependencies, editor leftovers | Never files whose path matches this one. The default skips `.git`, `node_modules`, `.DS_Store`, build folders, and files ending in `~`, `.swp`, `.tmp`, `.part`. |
|
|
135
|
+
| `settle` | `"1.5s"` | How long a file must stay unchanged before it counts as complete. |
|
|
136
|
+
| `recursive` | `true` | Also folders inside it. |
|
|
98
137
|
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
name finds a part made from the type's own description), or learns from the answer to a call
|
|
102
|
-
that leaves a needed option out.
|
|
138
|
+
Event `added`: `{ path, file, size, images? }`. For jpg, png, webp and gif under 10 MB the
|
|
139
|
+
picture is shown to the agent if its model can look at images.
|
|
103
140
|
|
|
104
|
-
###
|
|
141
|
+
### mqtt
|
|
105
142
|
|
|
106
|
-
|
|
107
|
-
agent's `agent.json` (only that agent can). The options of the type lie next to `type`:
|
|
143
|
+
An MQTT broker: reports the messages on the topics you name.
|
|
108
144
|
|
|
109
145
|
```json
|
|
110
|
-
"
|
|
111
|
-
"
|
|
146
|
+
"printer": {
|
|
147
|
+
"type": "mqtt", "url": "mqtts://192.168.1.50:8883", "username": "bblp", "password": "${BAMBU_ACCESS_CODE}",
|
|
148
|
+
"topics": ["device/+/report"], "insecureTls": true, "minInterval": "30s",
|
|
149
|
+
"trust": "owner", "description": "status reports of the 3D printer"
|
|
112
150
|
}
|
|
113
151
|
```
|
|
114
152
|
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
153
|
+
| Option | Default | |
|
|
154
|
+
|---|---|---|
|
|
155
|
+
| `url` | | `mqtt://`, `mqtts://`, `ws://` or `wss://` address of the broker. |
|
|
156
|
+
| `topics` | | One MQTT topic filter or a list. |
|
|
157
|
+
| `publish` | none | Topics an agent may send to, to act on a device: `["zigbee2mqtt/+/set"]`. Without it the source only listens. See "Sending through a source" in [sources.md](sources.md). |
|
|
158
|
+
| `username`, `password`, `clientId` | | As the broker wants them. |
|
|
159
|
+
| `minInterval` | | Drop messages that follow the last one on the same topic sooner than this. Use it for devices that report every second. |
|
|
160
|
+
| `insecureTls` | `false` | Accept a certificate that cannot be verified, as devices with self-made ones need. |
|
|
118
161
|
|
|
119
|
-
|
|
162
|
+
Event `message`: `{ topic, value }`, where `value` is the parsed JSON, or the text.
|
|
120
163
|
|
|
121
|
-
|
|
164
|
+
The example is a Bambu printer in LAN mode: it is its own broker on port 8883, the user is
|
|
165
|
+
`bblp`, the password is the access code from the printer's screen.
|
|
122
166
|
|
|
123
|
-
|
|
167
|
+
### webhook
|
|
168
|
+
|
|
169
|
+
A web address other services can call: `POST http://<pine's control address>/hook/<name>`.
|
|
170
|
+
|
|
171
|
+
```json
|
|
172
|
+
"github": { "type": "webhook", "secret": "${GITHUB_HOOK_SECRET}", "description": "pushes and issues of my repositories" }
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
| Option | |
|
|
124
176
|
|---|---|
|
|
125
|
-
| `
|
|
126
|
-
| `
|
|
127
|
-
|
|
128
|
-
|
|
177
|
+
| `secret` | The caller proves itself with it: `Authorization: Bearer <secret>`, or a signature of the body (HMAC SHA-256 in `X-Hub-Signature-256` or `X-Signature`, as GitHub sends it). |
|
|
178
|
+
| `open` | `true` instead of a secret: accept anyone who can reach the port. Only for a port nobody else can reach. |
|
|
179
|
+
|
|
180
|
+
Event `received`: `{ body, headers }`. `body` is the parsed JSON, or `{ text }`. Of the headers
|
|
181
|
+
only a few that say what kind of request it is are kept (`content-type`, `user-agent`,
|
|
182
|
+
`x-github-event` and the like), never the secret.
|
|
183
|
+
|
|
184
|
+
The address has the source's name as it is in the event topics: `/hook/<name>` for a source in
|
|
185
|
+
`pine.json`, `/hook/<name>@<agent>` for one in an agent's own `agent.json` (which is where a
|
|
186
|
+
source that an agent sets up goes, unless it is for all agents).
|
|
129
187
|
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
188
|
+
It needs the control API (`http` in `pine.json`), which listens on `127.0.0.1` by default. A
|
|
189
|
+
service on the internet reaches it only through a tunnel or reverse proxy that you set up. While
|
|
190
|
+
nobody is subscribed the address answers "not found".
|
|
133
191
|
|
|
134
|
-
###
|
|
192
|
+
### ftp
|
|
135
193
|
|
|
136
|
-
|
|
137
|
-
|
|
194
|
+
An FTP drop: a small FTP server that accepts uploads and nothing else. For cameras, doorbells,
|
|
195
|
+
scanners and other devices whose way of sending a file is "upload to FTP server".
|
|
138
196
|
|
|
139
197
|
```json
|
|
140
|
-
{ "
|
|
198
|
+
"door": { "type": "ftp", "port": 2121, "user": "cam", "password": "${CAMERA_FTP_PASSWORD}", "allow": ["192.168.1.40"], "description": "pictures from the front door camera when it sees motion" }
|
|
141
199
|
```
|
|
142
200
|
|
|
143
|
-
|
|
|
201
|
+
| Option | Default | |
|
|
202
|
+
|---|---|---|
|
|
203
|
+
| `port`, `user`, `password` | | What you enter in the device. The password needs 8 characters or more. When an agent sets the source up, `user` is `device` unless you name one, and a password you do not give is made by pine. |
|
|
204
|
+
| `host` | `"0.0.0.0"` | The address to listen on. Devices on your network have to reach it. |
|
|
205
|
+
| `allow` | everyone | Only these clients: a full address (`"192.168.1.40"`), or a prefix that ends in a dot (`"192.168.2."`). |
|
|
206
|
+
| `pasvPorts` | any | `[from, to]`: the ports for data connections, if a firewall has to know them. |
|
|
207
|
+
| `maxFileMb` | `50` | Largest file accepted. |
|
|
208
|
+
| `keepDays` | `7` | Received files are deleted after this many days. |
|
|
209
|
+
|
|
210
|
+
Event `upload`: `{ file, path, size, from, images? }`. `file` is the name the device gave it,
|
|
211
|
+
`path` is where it lies now (`data/sources/<name>/files/`, or `data/sources/<name>@<agent>/files/`
|
|
212
|
+
for an agent's own source). A picture is shown to the agent with the event. A video or any
|
|
213
|
+
other file is not: the agent reads it from `path` (see "Files a source received" in
|
|
214
|
+
[sources.md](sources.md)).
|
|
215
|
+
|
|
216
|
+
It speaks plain FTP, as these devices do, so the password crosses your network unencrypted. It
|
|
217
|
+
only receives: nothing can be listed, read, changed or deleted through it, and every file lands
|
|
218
|
+
under a name of pine's choosing in the source's own folder. Set the device to passive mode.
|
|
219
|
+
|
|
220
|
+
### telegram
|
|
221
|
+
|
|
222
|
+
A Telegram bot: you write to it from wherever you are, and the agent that subscribed answers
|
|
223
|
+
there. It is how you reach your agents when you are away, and how they reach you.
|
|
224
|
+
|
|
225
|
+
> Watch Telegram. You are my assistant there, and what you want to tell me while I am away
|
|
226
|
+
> goes to that chat.
|
|
227
|
+
|
|
228
|
+
The agent calls `add_source`. The dialog you confirm has a field for the bot's token: you type
|
|
229
|
+
or paste it there, where it is not shown and does not pass through the agent or its model (see
|
|
230
|
+
"Secrets" in [sources.md](sources.md)). Then the agent tells you the one remaining step, which ties the bot to your account:
|
|
231
|
+
|
|
232
|
+
1. Open the bot in Telegram and write anything to it.
|
|
233
|
+
2. The bot answers that account with a six-digit code.
|
|
234
|
+
3. pine shows a question on your computer that names the account and has a field for the code.
|
|
235
|
+
Type the code from your phone and press enter.
|
|
236
|
+
|
|
237
|
+
From then on that account is the only one the bot listens to. The code is the point: it shows
|
|
238
|
+
that the person at pine's keyboard holds that Telegram account. A stranger who finds the bot
|
|
239
|
+
can make the question appear, but does not see your screen, and you do not have their code; a
|
|
240
|
+
key pressed by accident cannot say yes. An account you deny is not put to you again, and only
|
|
241
|
+
one question is open at a time.
|
|
242
|
+
|
|
243
|
+
Anyone can write to a bot; that is how Telegram works. What they write reaches no agent. Before
|
|
244
|
+
you have paired, an unknown account gets the code message (and a line if the code that was
|
|
245
|
+
typed was wrong) and nothing else. After you have,
|
|
246
|
+
unknown accounts get no answer at all and cause no question.
|
|
247
|
+
|
|
248
|
+
```json
|
|
249
|
+
"telegram": { "type": "telegram", "token": "${PINE_SOURCE_TELEGRAM_MAIN_TOKEN}", "trust": "owner", "description": "my Telegram" }
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
| Option | Default | |
|
|
253
|
+
|---|---|---|
|
|
254
|
+
| `token` | | The bot's token. Create a bot by writing `/newbot` to @BotFather in Telegram; it answers with the token. Kept in `.env`. |
|
|
255
|
+
| `allow` | pairing | Telegram user ids that may talk to the bot, if you know yours. Without it the bot is paired as described above. With several ids, what the agent sends unasked and the approval requests go to the first. |
|
|
256
|
+
| `approvals` | `true` | Approval requests are also sent to that chat, with buttons to answer them: Allow, Deny and, where a yes can be remembered for the conversation, Always. `/allow` and `/deny` work too (with the request's id when more than one is open). A question the agent asks with its `question` tool arrives the same way when it offers answers to pick from, with one button for each and one for "No answer"; an answer in your own words, and a question without answers to pick from, you type in the terminal UI. A request that needs a secret typed is never sent there. |
|
|
257
|
+
| `apiBase` | Telegram's | The address of a Bot API server of your own. |
|
|
258
|
+
|
|
259
|
+
Event `message`: `{ text, name, username, chatId, messageId, images? }`. A photo you send (up
|
|
260
|
+
to 10 MB) is shown to the model. The bot listens only in its private chat with you: what is
|
|
261
|
+
written in a group it was added to reaches no agent.
|
|
262
|
+
|
|
263
|
+
What makes it different from the other types:
|
|
264
|
+
|
|
265
|
+
- **It is a conversation.** Your message reaches the agent as your message, not as a report
|
|
266
|
+
about one, in the conversation that subscribed. The answer is sent back to the chat and is
|
|
267
|
+
also in that conversation in the terminal UI, so both show the same thread.
|
|
268
|
+
- **It speaks for you.** Telegram vouches for the account and you proved it is yours, so what
|
|
269
|
+
it writes counts as your own words: full tools, no "outside content" mark. That is why
|
|
270
|
+
pairing takes a code and why the set-up is always put to you.
|
|
271
|
+
- **It can send.** The agent writes to you on its own with the `send` tool, with a picture or
|
|
272
|
+
a video if it has one to show, and without a question to you; see "Sending through a
|
|
273
|
+
source" in [sources.md](sources.md).
|
|
274
|
+
- **Messages are not held back.** The ten-second pause between events that protects a model from
|
|
275
|
+
a chatty device does not apply unless the subscription sets a `throttle` itself.
|
|
276
|
+
|
|
277
|
+
Who the bot listens to is said wherever sources are listed: `pine sources`, `/sources` in the
|
|
278
|
+
terminal UI, and the agent's `sources` tool show "listens to nobody yet", or the account you
|
|
279
|
+
paired with its name and Telegram id. A bot never learns a phone number; the account's id is
|
|
280
|
+
what Telegram vouches for. To pair with another account, remove the source and set it up
|
|
281
|
+
again: the pairing goes with the source.
|
|
282
|
+
|
|
283
|
+
Messages are fetched by long polling: one request that Telegram answers the moment something
|
|
284
|
+
arrives. Nothing listens on a port, so it works behind any router. Only one program can poll a
|
|
285
|
+
bot at a time; a second one makes both miss messages.
|
|
286
|
+
|
|
287
|
+
### http
|
|
288
|
+
|
|
289
|
+
A web address that is looked at again and again. For what cannot call you: a price, a version,
|
|
290
|
+
a status page, a device on your network that has a state page.
|
|
291
|
+
|
|
292
|
+
> Watch https://shop.example/fridge-3000 and tell me when the price drops below 500.
|
|
293
|
+
|
|
294
|
+
```json
|
|
295
|
+
"price": { "type": "http", "url": "https://shop.example/fridge-3000", "match": "Price: ([0-9.,]+)", "every": "30m", "description": "the price of the fridge" }
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
| Option | Default | |
|
|
299
|
+
|---|---|---|
|
|
300
|
+
| `url` | | The address, `https://...` (`http://` for a device on the local network). |
|
|
301
|
+
| `every` | `"5m"` | How often to look. At least `"30s"` when an agent sets it up, and at least `"2m"` with `render`. |
|
|
302
|
+
| `path` | | For an answer in JSON: watch only this field, `"data.price"` or `"items.0.state"`. `path`, `values`, `match` and `select` are four ways to say what counts in the answer; a source takes one of them. |
|
|
303
|
+
| `match` | | For a page or text: watch only what this regular expression finds, its first group if it has one. It is looked for in what a reader of the page sees, then in the page's source. |
|
|
304
|
+
| `select` | | For a page: watch only the text of the first element this CSS selector matches, `".product .price"` or `"#stock > span"`. With `render` it is any selector a browser reads. |
|
|
305
|
+
| `attr` | | With `select`: the value of this attribute of the element instead of its text, `"content"` for a meta tag, `"data-price"`. |
|
|
306
|
+
| `render` | `false` | `true` for a page that a script fills in after loading: every look opens the page in pine's browser, without a window, and takes what a reader sees then. See "A page that a script fills in". Not together with `path`, `values`, `method`, `body`, `headers`, `token` or `tokenHeader`. |
|
|
307
|
+
| `values` | | For an answer in JSON: several fields at once, each under a name of your own: `{ "price": "data.price", "stock": "data.stock.count" }`. Twenty at most. |
|
|
308
|
+
| `method` | `"GET"` | `"POST"` for an address that is asked by sending something. Nothing else: a source only looks. |
|
|
309
|
+
| `body` | | With `POST`: what to send, as text. Sent as JSON when it is JSON. Not for secrets. |
|
|
310
|
+
| `headers` | | Headers to send, `{ "accept": "application/json" }`. Not for secrets. |
|
|
311
|
+
| `token` | | An access token or key, sent as `Authorization: Bearer <token>`. Kept in `.env`. |
|
|
312
|
+
| `tokenHeader` | | Send the token as this header instead, as it is: `"x-api-key"`, `"cookie"`. |
|
|
313
|
+
| `local` | `false` | The address is on your own network under a name that does not say so (`nas.home`, `ha.lan`). Not needed for an IP address, `localhost` or a `.local` name. |
|
|
314
|
+
|
|
315
|
+
| Event | | |
|
|
316
|
+
|---|---|---|
|
|
317
|
+
| `changed` | `{ url, value, previous, number?, previousNumber? }` | With `path`, `match` or `select`: the value now and before. |
|
|
318
|
+
| `changed` | `{ url, values, previous, changed }` | With `values`: all the fields now and before, and the names of those that differ. |
|
|
319
|
+
| `changed` | `{ url, added, removed, more? }` | Without: the lines that are new and the lines that are gone, up to 30 of each. |
|
|
320
|
+
| `down` | `{ url, error, looks }` | After three failed looks in a row, once. |
|
|
321
|
+
| `up` | `{ url }` | When it answers again. |
|
|
322
|
+
|
|
323
|
+
Without `path`, `values`, `match` or `select` the whole answer counts. For a page that is its text as a reader
|
|
324
|
+
sees it: scripts, styles and markup are left out, so a counter in a script does not report a
|
|
325
|
+
change at every look. A page that shows the time still does; give it a `match`.
|
|
326
|
+
|
|
327
|
+
#### One element of a page
|
|
328
|
+
|
|
329
|
+
`match` finds a value by the text around it; `select` finds it by where it stands in the page,
|
|
330
|
+
which holds better when the words around a price change with the language or the season:
|
|
331
|
+
|
|
332
|
+
```json
|
|
333
|
+
"fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "select": ".product-detail .price", "every": "30m", "description": "the price of the fridge" }
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
A selector is made of tags (`span`), ids (`#total`), classes (`.price`), attributes (`[itemprop]`,
|
|
337
|
+
`[itemprop=price]`, also `~=`, `^=`, `$=`, `*=`), put together (`span.price`, `li.item.sale`) and
|
|
338
|
+
nested with a space (somewhere inside) or `>` (directly inside); a comma separates alternatives.
|
|
339
|
+
Nothing with a colon (`:first-child`, `:not(...)`) and no `+` or `~`: such a selector is refused
|
|
340
|
+
with what it takes instead. The first element in the page that matches counts. Many shops also
|
|
341
|
+
state the price for machines, which changes less often than the layout:
|
|
342
|
+
|
|
343
|
+
```json
|
|
344
|
+
"fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "select": "meta[property=product:price:amount]", "attr": "content" }
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
The page is read as the server sends it. What a script fills in after loading is not
|
|
348
|
+
there: for such a page set `render`, or watch the address the script itself asks (see "A
|
|
349
|
+
page that a script fills in").
|
|
350
|
+
|
|
351
|
+
#### A page that a script fills in
|
|
352
|
+
|
|
353
|
+
Many pages arrive almost empty, and a script of the page then fetches the price or the state
|
|
354
|
+
and writes it in. A plain look reads the page as the server sends it, so it finds the
|
|
355
|
+
placeholder ("loading", an empty element, nothing at all) and never the value. With `render`
|
|
356
|
+
the look is made by a browser:
|
|
357
|
+
|
|
358
|
+
> The price on https://shop.example/fridge-3000 only shows up after the page has loaded. Watch
|
|
359
|
+
> it and tell me when it drops.
|
|
360
|
+
|
|
361
|
+
```json
|
|
362
|
+
"fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "render": true, "select": ".product-detail .price", "every": "30m", "description": "the price of the fridge" }
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
At every look pine starts its browser (the one the browser tools use, which `pine init` and
|
|
366
|
+
`pine browser-install` fetch), without a window and with an empty profile, opens the address,
|
|
367
|
+
waits until the page has loaded and then until it has stopped asking for things (five seconds
|
|
368
|
+
at most), and takes what a reader sees. Then the browser is closed. What counts is chosen as
|
|
369
|
+
without `render`:
|
|
370
|
+
|
|
371
|
+
| With | What counts |
|
|
372
|
+
|---|---|
|
|
373
|
+
| `select` | The text of the first element the selector matches, as the browser shows it. The browser reads the selector, so everything CSS has works: `li:nth-child(2) .price`, `.offer:not(.sold-out)`, `h1 + p`. No element matches: the value is `null`. |
|
|
374
|
+
| `select` and `attr` | The value of that attribute of the element. |
|
|
375
|
+
| `match` | What the regular expression finds in the text a reader sees. The page's source is not looked at. |
|
|
376
|
+
| neither | The whole text of the page, line by line; an event names the lines that are new and gone. |
|
|
377
|
+
|
|
378
|
+
The events are the same, and an amount arrives as `number` as described below.
|
|
379
|
+
|
|
380
|
+
**What it costs.** A plain look is one request and takes a moment. A rendered look starts a
|
|
381
|
+
browser, runs the page's scripts and loads everything the page loads: a second or two and a
|
|
382
|
+
few hundred megabytes of memory while it lasts, at every look. So:
|
|
383
|
+
|
|
384
|
+
- `every` is at least `"2m"` when an agent sets the source up.
|
|
385
|
+
- Two browsers run at a time, for all sources together. A look that finds them busy waits.
|
|
386
|
+
- A look may take 45 seconds. After that it is ended and counts as failed.
|
|
387
|
+
- There is no "nothing new" answer (see below): the page is loaded in full each time.
|
|
388
|
+
|
|
389
|
+
**When the plain look is the better choice.** A script that fills a page in gets the value
|
|
390
|
+
from somewhere, usually an address that answers JSON. If you can find it (the network tab of
|
|
391
|
+
your browser's developer tools shows what the page asks), watch that address without `render`
|
|
392
|
+
and with `path`: it is faster, it can be asked every 30 seconds, a number arrives as a number,
|
|
393
|
+
it can take a `token`, and it does not break when the shop changes its layout. Use `render`
|
|
394
|
+
when there is no such address, when it cannot be asked from outside the page, or when what you
|
|
395
|
+
want is what the page shows.
|
|
396
|
+
|
|
397
|
+
**What does not go with it.** A set-up with one of these and `render` is refused, with the
|
|
398
|
+
reason:
|
|
399
|
+
|
|
400
|
+
| Option | Why not |
|
|
144
401
|
|---|---|
|
|
145
|
-
| `
|
|
146
|
-
| `
|
|
147
|
-
| `
|
|
148
|
-
| `
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
-
|
|
165
|
-
|
|
166
|
-
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
402
|
+
| `path`, `values` | They read an answer in JSON. There is no page to render. Leave `render` out. |
|
|
403
|
+
| `method`, `body` | A rendered page is asked the way a reader's browser asks it. |
|
|
404
|
+
| `headers`, `token`, `tokenHeader` | The browser would send them with every request the page makes, to every address the page loads from. So a page behind a login cannot be watched this way. |
|
|
405
|
+
| `local: true` | See below: a name that has to be looked up is not opened on your network. |
|
|
406
|
+
|
|
407
|
+
**What the page can reach.** The page's own scripts run in that browser, as they would in
|
|
408
|
+
yours, and the dialog you confirm says so. What they can reach is narrower than in yours:
|
|
409
|
+
|
|
410
|
+
- Every connection of that browser goes through the same guard as the browser tools'
|
|
411
|
+
([security.md](security.md)). The public internet is reachable. Your own network and this
|
|
412
|
+
computer are not: not for the page, and not for anything the page loads or its scripts ask.
|
|
413
|
+
- One exception, which is the page itself: an address that says by itself that it is local
|
|
414
|
+
(`localhost`, a `.local` name, an IP address in a private range) is opened, and the browser
|
|
415
|
+
then reaches exactly that host and port and nothing else on your network. The dialog names
|
|
416
|
+
it.
|
|
417
|
+
- A name that only turns out to be local when it is looked up (`nas.home`, `ha.lan`) is not
|
|
418
|
+
opened with `render`, also not with `local: true`: what such a name points at can change,
|
|
419
|
+
and a page's scripts would run there. Give the device by its IP address.
|
|
420
|
+
- pine's own control port is never opened.
|
|
421
|
+
- The look counts only if the browser ends up on the host that was set up. A page that sends
|
|
422
|
+
it on to another host is a failed look that names the host.
|
|
423
|
+
- The browser keeps nothing from one look to the next (no cookies, no logins), downloads
|
|
424
|
+
nothing, and grants the page no permission (camera, location, notifications). A window the
|
|
425
|
+
page opens is closed.
|
|
426
|
+
|
|
427
|
+
What such a source reports is outside content, like everything an `http` source reports.
|
|
428
|
+
|
|
429
|
+
**When a look fails.** A failed look is retried with a growing pause, and `pine sources` shows
|
|
430
|
+
since when it fails and why; after three in a row the source reports `down`, and `up` when the
|
|
431
|
+
page opens again.
|
|
432
|
+
|
|
433
|
+
| The status says | What it means |
|
|
434
|
+
|---|---|
|
|
435
|
+
| `pine's browser is not installed on this computer` | Type `pine browser-install` into a terminal. `pine init` and `pine update` do it too. |
|
|
436
|
+
| `the page did not finish loading within 45 s` | The page is slow, or never finishes. |
|
|
437
|
+
| `"select" is not a CSS selector the browser can read` | The selector has a mistake in it. |
|
|
438
|
+
| `it sends on to <host>, which is not the address that was confirmed` | The page moved. Set the source up with the address it leads to, if that is the right one. |
|
|
439
|
+
| `<host> leads to an address on your own network` | The name points into your network. With `render` only an address that says so itself is opened there. |
|
|
440
|
+
| `it answered 403` (or another number) | The site turned the browser away. Some sites do that to a browser without a window. |
|
|
441
|
+
| `the browser could not load it (...)` | No connection, an invalid certificate, or the like; the words in brackets are the browser's. |
|
|
177
442
|
|
|
178
|
-
|
|
443
|
+
`found nothing with the selector` is not a failed look: the page opened and no element matched.
|
|
444
|
+
If the value is there in your own browser, the selector is wrong, or the page fills the
|
|
445
|
+
element in later than the look waits.
|
|
446
|
+
|
|
447
|
+
#### An amount as a number
|
|
448
|
+
|
|
449
|
+
A value that reads as a number or an amount, and nothing else, also arrives as `number` (and
|
|
450
|
+
the one before as `previousNumber`), so that a subscription can compare it:
|
|
451
|
+
|
|
452
|
+
```json
|
|
453
|
+
"subscribe": [{ "source": "fridge", "when": { "payload.number": { "lt": 500 } }, "note": "Tell me: the fridge is under 500 now." }]
|
|
454
|
+
```
|
|
455
|
+
|
|
456
|
+
| The value | `number` |
|
|
457
|
+
|---|---|
|
|
458
|
+
| `12.5`, `"12.5"`, `"12,5"`, `"87 %"`, `"-3,5 °C"`, `"3 GB"` | 12.5, 12.5, 12.5, 87, -3.5, 3 |
|
|
459
|
+
| `"$1,299.00"`, `"1.299,00 €"`, `"1 299,00 kr"`, `"1'299.50"` | 1299, 1299, 1299, 1299.5 |
|
|
460
|
+
| `"1,299"`, `"1.299"` | 1299 and 1.299: as a program writes numbers |
|
|
461
|
+
| `"1.299 €"`, `"1,299 €"` | 1299 and 1.299: next to `€`, `zł`, `kr` or `Kč` a dot groups thousands and a comma is the decimal point |
|
|
462
|
+
| `"1.2.3"`, `"12:30"`, `"2026-05-01"`, `"about 5"`, `"sold out"` | none |
|
|
463
|
+
|
|
464
|
+
The one case that can be read two ways is a single separator followed by exactly three digits,
|
|
465
|
+
and the last two rows are how it is decided. Where that is not what the page means, take the
|
|
466
|
+
digits apart with `match` (`"([0-9]+)[.,][0-9]{3} "`) and compare the text.
|
|
467
|
+
|
|
468
|
+
#### Several fields, and an address that is asked with POST
|
|
469
|
+
|
|
470
|
+
```json
|
|
471
|
+
"fridge": {
|
|
472
|
+
"type": "http", "url": "https://shop.example/graphql", "method": "POST",
|
|
473
|
+
"body": "{\"query\": \"{ product(id: 7) { price stock { count } } }\"}",
|
|
474
|
+
"values": { "price": "data.product.price", "stock": "data.product.stock.count" },
|
|
475
|
+
"every": "30m", "description": "price and stock of the fridge"
|
|
476
|
+
}
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
One event says which of the fields changed: `{ "values": { "price": 479, "stock": 3 },
|
|
480
|
+
"previous": { "price": 499, "stock": 3 }, "changed": ["price"] }`. A field that is not in the
|
|
481
|
+
answer is `null`. Since these are the values of the JSON itself, numbers are numbers and
|
|
482
|
+
`when` compares them directly: `{ "payload.values.stock": { "gt": 0 } }`.
|
|
483
|
+
|
|
484
|
+
A `POST` is for asking, and what it sends is shown to you in full before the source is set up:
|
|
485
|
+
a source repeats it on a clock, with nobody there.
|
|
486
|
+
|
|
487
|
+
#### Nothing new
|
|
488
|
+
|
|
489
|
+
An address that names a version of its answer (`ETag`, `Last-Modified`) is asked "anything new
|
|
490
|
+
since?" from the second look on, and answers with one line when there is not. That is a look
|
|
491
|
+
like any other, only cheaper for both sides. `feed` and `calendar` ask the same way. A look
|
|
492
|
+
with `render` loads the page in full every time.
|
|
493
|
+
|
|
494
|
+
#### Which addresses it asks
|
|
495
|
+
|
|
496
|
+
An address on your own network or this computer is said to be one in the dialog you confirm.
|
|
497
|
+
Any other address has to stay on the public internet: if its name one day leads into your
|
|
498
|
+
network, the look fails and says so instead of fetching from there.
|
|
499
|
+
|
|
500
|
+
A redirect is followed only on the same host. One that leads to another host is a failed look
|
|
501
|
+
that says where it leads: you confirmed one address, and the token is for that one. An answer
|
|
502
|
+
is read up to 1 MB. A key that is part of the address (`?apikey=...`) is not a secret to pine:
|
|
503
|
+
it is stored and shown like the rest of the address.
|
|
504
|
+
|
|
505
|
+
Refused when it is set up: a user name or password in the address, `path` and `match`
|
|
506
|
+
together, and a header that carries a secret (`authorization`, `cookie`), which belongs into
|
|
507
|
+
`token`. Also more than one of `path`, `values`, `match` and `select`; `attr` without
|
|
508
|
+
`select`; a selector that cannot be read; a `method` other than `GET` and `POST`; a `body`
|
|
509
|
+
without `POST` or longer than 20000 characters; `values` that are empty, more than twenty, or
|
|
510
|
+
under a name that is not a plain word. One look may take 20 seconds and follow five redirects.
|
|
511
|
+
|
|
512
|
+
### command
|
|
513
|
+
|
|
514
|
+
A command that is run again and again. For what only a command can tell: free disk space, a
|
|
515
|
+
build, a branch that moved, the end of a log, whether a machine answers.
|
|
516
|
+
A command that keeps running and prints a line whenever something happens is followed instead
|
|
517
|
+
of run again and again: see "Following a command that keeps running" below.
|
|
518
|
+
|
|
519
|
+
> Tell me when the disk is more than 90% full.
|
|
520
|
+
|
|
521
|
+
```json
|
|
522
|
+
"disk": { "type": "command", "run": "df -h /", "match": "([0-9]+)%", "every": "10m", "description": "how full the disk is" }
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
| Option | Default | |
|
|
526
|
+
|---|---|---|
|
|
527
|
+
| `run` | | The command line. bash runs it from the workspace of the agent the source belongs to. |
|
|
528
|
+
| `every` | `"1m"` | How often. At least `"5s"` when an agent sets it up. |
|
|
529
|
+
| `match` | | Watch only what this regular expression finds in what the command prints. |
|
|
530
|
+
| `timeout` | `"30s"` | How long one run may take before it is stopped. |
|
|
531
|
+
|
|
532
|
+
Event `changed`: `{ command, code, previousCode?, value, previous }` with `match`, or
|
|
533
|
+
`{ command, code, previousCode?, added, removed, more? }` without. `code` is the exit code;
|
|
534
|
+
`previousCode` is there when it changed, so `test -f build/done` or `ping -c1 nas` report by
|
|
535
|
+
how they end.
|
|
536
|
+
|
|
537
|
+
The command runs **like a command in the shell of an agent in `safe` mode**: in the
|
|
538
|
+
agent's sandbox, with its private home folder and without your environment, also for an agent
|
|
539
|
+
that works in `auto` mode and has your whole account in your conversations (nobody is there to answer for what a
|
|
540
|
+
command on a clock does). It reaches the agent's own folders and nothing else: no network
|
|
541
|
+
unless `sandbox.network` is set, none of your home folder beyond `sandbox.read`, none of
|
|
542
|
+
pine's files. A source of all agents runs as the main agent. You are shown the
|
|
543
|
+
command line before it is set up, every time. A command that cannot be started or runs out of
|
|
544
|
+
time is a failed look (see "Looking again and again"), not an event. `timeout` is at least `"1s"`, and a command
|
|
545
|
+
line of more than 2000 characters is refused (put it into a script). If the agent's sandbox is
|
|
546
|
+
switched off, the command runs without one, as its shell does.
|
|
547
|
+
|
|
548
|
+
#### Following a command that keeps running
|
|
549
|
+
|
|
550
|
+
Some commands tell by themselves when something happens: `tail -F` a log, `log stream`,
|
|
551
|
+
`docker events`, `kubectl get pods -w`, the client of a broker or a database. With `follow`
|
|
552
|
+
the command is started once and kept running, and every line it prints is an event as it
|
|
553
|
+
comes. This is the type for every protocol that has a command line client and no source type
|
|
554
|
+
of its own.
|
|
555
|
+
|
|
556
|
+
> Tell me when the backup log says ERROR.
|
|
557
|
+
|
|
558
|
+
```json
|
|
559
|
+
"backup": { "type": "command", "run": "tail -n 0 -F ~/Library/Logs/backup.log", "follow": true, "match": "ERROR (.*)", "description": "errors in the backup log" }
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
| Option | Default | |
|
|
563
|
+
|---|---|---|
|
|
564
|
+
| `follow` | `false` | `true`: start the command once, keep it running, report each line it prints. `every` and `timeout` are not used then. |
|
|
565
|
+
| `perMinute` | `30` | With `follow`: report at most this many lines a minute. The rest are counted, and the next line that is reported carries the count as `dropped`. |
|
|
566
|
+
| `match` | | With `follow`: only lines in which this regular expression finds something. Its first group is the line's `value`. |
|
|
567
|
+
|
|
568
|
+
| Event | | |
|
|
569
|
+
|---|---|---|
|
|
570
|
+
| `line` | `{ command, line, value?, dropped? }` | One line the command printed. `value` is what `match` found, or the parsed line when it is JSON. |
|
|
571
|
+
| `ended` | `{ command, code?, error?, ran }` | The command ended (`ran`: after how many seconds, `error`: the end of what it wrote as errors). It is started again, after a pause that grows while it keeps ending at once. A command that ends the moment it is started is reported once, not at every try. |
|
|
572
|
+
|
|
573
|
+
| What to follow | `run` |
|
|
574
|
+
|---|---|
|
|
575
|
+
| A log file | `tail -n 0 -F /path/app.log` (`-n 0`: only what is written from now on; `-F`: also after the file was rotated) |
|
|
576
|
+
| The Mac's own log | `log stream --style ndjson --predicate 'subsystem == "com.apple.TimeMachine"'` |
|
|
577
|
+
| Docker | `docker events --format '{{json .}}'` |
|
|
578
|
+
| Kubernetes | `kubectl get pods -w -o json`, `kubectl get events -w` |
|
|
579
|
+
| systemd's journal, on another machine | `ssh nas journalctl -f -p warning -o json` |
|
|
580
|
+
| PostgreSQL `LISTEN` | a few lines of script around a client library that print each notification as a line |
|
|
581
|
+
| Redis | `redis-cli --raw subscribe alerts` |
|
|
582
|
+
| NATS, Kafka, RabbitMQ | `nats sub 'orders.>'`, `kcat -C -b broker -t topic -o end`, `amqp-consume -q queue cat` |
|
|
583
|
+
| A folder on another machine | `ssh nas inotifywait -m -e close_write /volume1/scans` |
|
|
584
|
+
|
|
585
|
+
A line is cleaned of the colour codes a terminal would act on and cut at 2000 characters; a
|
|
586
|
+
line that is JSON also arrives parsed, so a subscription's `when` can look into it
|
|
587
|
+
(`{"payload.value.status": "die"}` for a container that stopped).
|
|
588
|
+
|
|
589
|
+
The command runs in the same sandbox as one that is run again and again: the agent's own, as
|
|
590
|
+
in `safe` mode. That is why most of the examples above need something from you first: the log
|
|
591
|
+
file has to be readable by the agent (`sandbox.read`), and a client that talks to the network
|
|
592
|
+
needs `sandbox.network`. The dialog shows the command line, and that it will keep running.
|
|
593
|
+
|
|
594
|
+
On Linux `follow` is refused. The sandbox there cannot forbid a running command to make a file
|
|
595
|
+
that is read as instructions; pine takes such a file away after a command has ended, and a
|
|
596
|
+
followed command does not end. Run it again and again instead (`every`, with a `match`), or
|
|
597
|
+
have the command's lines sent to a `syslog` or `webhook` source.
|
|
598
|
+
|
|
599
|
+
When the source stops (the last subscriber left, or pine stops), the command and everything
|
|
600
|
+
it started is ended with it.
|
|
179
601
|
|
|
180
602
|
### feed
|
|
181
603
|
|
|
@@ -552,7 +974,7 @@ turned app passwords off.
|
|
|
552
974
|
the addresses on `sendTo` and nobody else; the type checks that itself, whatever a call names.
|
|
553
975
|
With one address on the list, `to` can be left out. Mail can reach anybody, so unlike a
|
|
554
976
|
Telegram message to you it counts as information leaving: after outside content the agent asks
|
|
555
|
-
before it sends (see "Sending through a source"). The text goes as plain text; no files.
|
|
977
|
+
before it sends (see "Sending through a source" in [sources.md](sources.md)). The text goes as plain text; no files.
|
|
556
978
|
|
|
557
979
|
Refused when it is set up: a `host` with a scheme, port or path in it, `"security": "none"` for
|
|
558
980
|
a server that is not local, a `from` entry that is neither an address nor `@domain`, `sendTo`
|
|
@@ -893,7 +1315,7 @@ answered the moment something arrives, so nothing listens on a port.
|
|
|
893
1315
|
| `mentions` | | User ids (`@paul:matrix.org`): only messages that mention one of them. |
|
|
894
1316
|
| `keywords` | | Only messages that contain one of these words. Capitals do not matter. |
|
|
895
1317
|
| `ignore` | | User ids that are never reported. What the account itself writes never is. |
|
|
896
|
-
| `sendTo` | none | Room ids an agent may write to. Without it the source only listens. See "Sending through a source". |
|
|
1318
|
+
| `sendTo` | none | Room ids an agent may write to. Without it the source only listens. See "Sending through a source" in [sources.md](sources.md). |
|
|
897
1319
|
| `local` | `false` | The homeserver is on your own network under a name that does not say so. |
|
|
898
1320
|
|
|
899
1321
|
Event `message`: `{ text, user, room, roomName?, kind, id, mention?, keyword?, dropped? }`.
|
|
@@ -952,7 +1374,7 @@ to Discord's gateway, so nothing listens on a port.
|
|
|
952
1374
|
| `keywords` | | In servers, only messages that contain one of these words. Capitals do not matter. With `mentions` as well, either is enough. |
|
|
953
1375
|
| `dms` | `true` | Direct messages to the bot count, without a mention or a keyword. |
|
|
954
1376
|
| `bots` | `false` | What other bots and webhooks write counts too. |
|
|
955
|
-
| `sendTo` | none | Channel ids an agent may write to. Without it the source only listens. See "Sending through a source". |
|
|
1377
|
+
| `sendTo` | none | Channel ids an agent may write to. Without it the source only listens. See "Sending through a source" in [sources.md](sources.md). |
|
|
956
1378
|
| `apiBase` | `https://discord.com/api/v10` | A Discord-compatible server of your own: reached over https, or on this machine. |
|
|
957
1379
|
|
|
958
1380
|
Ids are long numbers. With "Developer Mode" on (User Settings, Advanced), a right click on a
|
|
@@ -1440,10 +1862,10 @@ On Linux and Windows a `system` source cannot be set up. What a command can tell
|
|
|
1440
1862
|
|
|
1441
1863
|
| Wanted | Instead |
|
|
1442
1864
|
|---|---|
|
|
1443
|
-
| A page that a script fills in after loading | It has no type of its own: `http` with `render` has a browser make the look (
|
|
1865
|
+
| A page that a script fills in after loading | It has no type of its own: `http` with `render` has a browser make the look (see "A page that a script fills in" under `http`). Where the page's script asks an address that answers JSON, watching that with `path` is lighter. |
|
|
1444
1866
|
| A calendar's "ten minutes before" | A schedule the agent sets when `calendar` reports the entry ([events.md](events.md), "Schedules"). |
|
|
1445
1867
|
| Bluetooth sensors and beacons | A bridge that publishes them to MQTT (Theengs Gateway, OpenMQTTGateway), and the `mqtt` type. |
|
|
1446
1868
|
| iMessage, WhatsApp | No type. Signal works through signal-cli and `stream`; Telegram, Slack, Matrix and Discord have types. |
|
|
1447
1869
|
| KNX, Z-Wave, Zigbee, HomeKit, Matter | Through the hub that speaks them (Home Assistant, zigbee2mqtt, zwave-js-ui), by `mqtt` or `stream`. |
|
|
1448
|
-
| Something with a command line client and nothing else | `command` with `follow` (
|
|
1870
|
+
| Something with a command line client and nothing else | `command` with `follow` (see "Following a command that keeps running" under `command`). |
|
|
1449
1871
|
| Something of your own | A source type of your own is a small module ([sources.md](sources.md), "A source type of your own"); the main agent can write and register one when you ask it. |
|