@pmndre/pine 0.35.1 → 0.36.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/changes.json +12 -16
- package/docs/SKILL.md +4 -3
- package/docs/references/events.md +1 -1
- package/docs/references/operations.md +1 -1
- package/docs/references/setup.md +1 -1
- package/docs/references/source-types.md +553 -82
- package/docs/references/sources.md +75 -513
- package/package.json +1 -1
- package/src/cli.js +1 -1
- package/src/manual.d.ts +1 -1
- package/src/sources/slack.d.ts +9 -1
- package/src/sources/slack.js +297 -13
- package/src/tools.js +1 -1
- package/src/tui/app.js +1 -1
|
@@ -6,10 +6,6 @@ and how secrets are kept is in [sources.md](sources.md). This file is the catalo
|
|
|
6
6
|
type there is, the options it takes, the events it reports with their fields, and the services
|
|
7
7
|
and devices each one reaches, with an entry to copy for each.
|
|
8
8
|
|
|
9
|
-
The seven everyday types (`folder`, `mqtt`, `webhook`, `ftp`, `telegram`, `http`, `command`)
|
|
10
|
-
are described in [sources.md](sources.md), "The built-in types". The others are described
|
|
11
|
-
here.
|
|
12
|
-
|
|
13
9
|
## Which type for what
|
|
14
10
|
|
|
15
11
|
Two kinds of type exist. One **is told**: it listens on a port or holds a connection open, and
|
|
@@ -46,6 +42,29 @@ What should happen at a time is not a source but a schedule ([events.md](events.
|
|
|
46
42
|
"Schedules"). A `calendar` source reports that an entry was added or moved; "ten minutes
|
|
47
43
|
before it starts" is a schedule the agent sets when it is told of the entry.
|
|
48
44
|
|
|
45
|
+
### Looking again and again
|
|
46
|
+
|
|
47
|
+
`http`, `command`, `feed`, `calendar`, `modbus` and `presence` cannot be told when something
|
|
48
|
+
happens; they go and look. Three things hold for every source that does:
|
|
49
|
+
|
|
50
|
+
- **The first look is the starting point.** Setting a source up reports nothing. Only what is
|
|
51
|
+
different at a later look does.
|
|
52
|
+
- **A look that finds nothing new costs nothing.** No event, nothing stored, no model woken.
|
|
53
|
+
This is the difference to waking an agent every few minutes to check: an agent's turn is
|
|
54
|
+
spent only when there is something to act on.
|
|
55
|
+
- **A failed look is not an event.** The pause before the next one doubles with each failure in
|
|
56
|
+
a row, up to 15 minutes (or the source's own interval, if that is longer), and goes back to
|
|
57
|
+
normal with the first look that works.
|
|
58
|
+
|
|
59
|
+
How a polling source is doing is said wherever sources are listed (`pine sources`, `/sources`,
|
|
60
|
+
the agent's `sources` tool): `last looked 14:02:11`, or `failing since 14:02:11 (4 looks): it
|
|
61
|
+
answered 503`, followed by what it found. What it found is repeated there only when it is a
|
|
62
|
+
number or a single word; anything longer is only counted, because that line is read by agents
|
|
63
|
+
without the mark that outside content gets.
|
|
64
|
+
|
|
65
|
+
Two looks of one source never run at the same time, and each waits a little shorter or longer
|
|
66
|
+
than its interval, so that sources set to the same interval do not all fire together.
|
|
67
|
+
|
|
49
68
|
## Which type for which service
|
|
50
69
|
|
|
51
70
|
| Service or device | Type | How |
|
|
@@ -83,99 +102,502 @@ before it starts" is a schedule the agent sets when it is told of the entry.
|
|
|
83
102
|
|
|
84
103
|
## How every type is used
|
|
85
104
|
|
|
86
|
-
### Ask the agent
|
|
87
|
-
|
|
88
105
|
Say what you want watched. The agent picks the type, pine shows you what the source will do,
|
|
89
106
|
and you confirm:
|
|
90
107
|
|
|
91
108
|
> Tell me on Telegram when pmndre/pine has a new release.
|
|
92
109
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
110
|
+
Or write the entry yourself, under `sources` in `pine.json` or in an agent's `agent.json`, with
|
|
111
|
+
the options of the type next to `type`, and subscribe an agent to it. Each type below has an
|
|
112
|
+
entry to copy.
|
|
113
|
+
|
|
114
|
+
What holds whatever the type is in [sources.md](sources.md): the keys every source has
|
|
115
|
+
("Configuring sources"), what all types have in common, subscribing and its filter `when`
|
|
116
|
+
("Being woken only for what counts"), how a password or token is kept ("Secrets"), what an
|
|
117
|
+
event looks like, and whose words an event counts as ("Trust").
|
|
118
|
+
|
|
119
|
+
## The types
|
|
120
|
+
|
|
121
|
+
### folder
|
|
122
|
+
|
|
123
|
+
A folder: reports every new or changed file once it has stopped growing.
|
|
124
|
+
|
|
125
|
+
```json
|
|
126
|
+
"scans": { "type": "folder", "dir": "~/Scans", "mask": "*.pdf" }
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
| Option | Default | |
|
|
130
|
+
|---|---|---|
|
|
131
|
+
| `dir` | | The folder. `~` and paths relative to the configuration file work. |
|
|
132
|
+
| `mask` | | Only files whose name matches, as in a shell: `*.pdf`, `*.{png,jpg}`, `report-??.csv`. Several are separated by commas. Capitals do not matter. |
|
|
133
|
+
| `pattern` | | Only files whose full path matches this regular expression, for what a mask cannot say (`/src/.*\\.ts$`). Capitals do not matter. |
|
|
134
|
+
| `ignore` | version control, dependencies, editor leftovers | Never files whose path matches this one. The default skips `.git`, `node_modules`, `.DS_Store`, build folders, and files ending in `~`, `.swp`, `.tmp`, `.part`. |
|
|
135
|
+
| `settle` | `"1.5s"` | How long a file must stay unchanged before it counts as complete. |
|
|
136
|
+
| `recursive` | `true` | Also folders inside it. |
|
|
98
137
|
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
name finds a part made from the type's own description), or learns from the answer to a call
|
|
102
|
-
that leaves a needed option out.
|
|
138
|
+
Event `added`: `{ path, file, size, images? }`. For jpg, png, webp and gif under 10 MB the
|
|
139
|
+
picture is shown to the agent if its model can look at images.
|
|
103
140
|
|
|
104
|
-
###
|
|
141
|
+
### mqtt
|
|
105
142
|
|
|
106
|
-
|
|
107
|
-
agent's `agent.json` (only that agent can). The options of the type lie next to `type`:
|
|
143
|
+
An MQTT broker: reports the messages on the topics you name.
|
|
108
144
|
|
|
109
145
|
```json
|
|
110
|
-
"
|
|
111
|
-
"
|
|
146
|
+
"printer": {
|
|
147
|
+
"type": "mqtt", "url": "mqtts://192.168.1.50:8883", "username": "bblp", "password": "${BAMBU_ACCESS_CODE}",
|
|
148
|
+
"topics": ["device/+/report"], "insecureTls": true, "minInterval": "30s",
|
|
149
|
+
"trust": "owner", "description": "status reports of the 3D printer"
|
|
112
150
|
}
|
|
113
151
|
```
|
|
114
152
|
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
153
|
+
| Option | Default | |
|
|
154
|
+
|---|---|---|
|
|
155
|
+
| `url` | | `mqtt://`, `mqtts://`, `ws://` or `wss://` address of the broker. |
|
|
156
|
+
| `topics` | | One MQTT topic filter or a list. |
|
|
157
|
+
| `publish` | none | Topics an agent may send to, to act on a device: `["zigbee2mqtt/+/set"]`. Without it the source only listens. See "Sending through a source" in [sources.md](sources.md). |
|
|
158
|
+
| `username`, `password`, `clientId` | | As the broker wants them. |
|
|
159
|
+
| `minInterval` | | Drop messages that follow the last one on the same topic sooner than this. Use it for devices that report every second. |
|
|
160
|
+
| `insecureTls` | `false` | Accept a certificate that cannot be verified, as devices with self-made ones need. |
|
|
161
|
+
|
|
162
|
+
Event `message`: `{ topic, value }`, where `value` is the parsed JSON, or the text.
|
|
118
163
|
|
|
119
|
-
|
|
164
|
+
The example is a Bambu printer in LAN mode: it is its own broker on port 8883, the user is
|
|
165
|
+
`bblp`, the password is the access code from the printer's screen.
|
|
120
166
|
|
|
121
|
-
|
|
167
|
+
### webhook
|
|
122
168
|
|
|
123
|
-
|
|
169
|
+
A web address other services can call: `POST http://<pine's control address>/hook/<name>`.
|
|
170
|
+
|
|
171
|
+
```json
|
|
172
|
+
"github": { "type": "webhook", "secret": "${GITHUB_HOOK_SECRET}", "description": "pushes and issues of my repositories" }
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
| Option | |
|
|
176
|
+
|---|---|
|
|
177
|
+
| `secret` | The caller proves itself with it: `Authorization: Bearer <secret>`, or a signature of the body (HMAC SHA-256 in `X-Hub-Signature-256` or `X-Signature`, as GitHub sends it). |
|
|
178
|
+
| `open` | `true` instead of a secret: accept anyone who can reach the port. Only for a port nobody else can reach. |
|
|
179
|
+
|
|
180
|
+
Event `received`: `{ body, headers }`. `body` is the parsed JSON, or `{ text }`. Of the headers
|
|
181
|
+
only a few that say what kind of request it is are kept (`content-type`, `user-agent`,
|
|
182
|
+
`x-github-event` and the like), never the secret.
|
|
183
|
+
|
|
184
|
+
The address has the source's name as it is in the event topics: `/hook/<name>` for a source in
|
|
185
|
+
`pine.json`, `/hook/<name>@<agent>` for one in an agent's own `agent.json` (which is where a
|
|
186
|
+
source that an agent sets up goes, unless it is for all agents).
|
|
187
|
+
|
|
188
|
+
It needs the control API (`http` in `pine.json`), which listens on `127.0.0.1` by default. A
|
|
189
|
+
service on the internet reaches it only through a tunnel or reverse proxy that you set up. While
|
|
190
|
+
nobody is subscribed the address answers "not found".
|
|
191
|
+
|
|
192
|
+
### ftp
|
|
193
|
+
|
|
194
|
+
An FTP drop: a small FTP server that accepts uploads and nothing else. For cameras, doorbells,
|
|
195
|
+
scanners and other devices whose way of sending a file is "upload to FTP server".
|
|
196
|
+
|
|
197
|
+
```json
|
|
198
|
+
"door": { "type": "ftp", "port": 2121, "user": "cam", "password": "${CAMERA_FTP_PASSWORD}", "allow": ["192.168.1.40"], "description": "pictures from the front door camera when it sees motion" }
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
| Option | Default | |
|
|
202
|
+
|---|---|---|
|
|
203
|
+
| `port`, `user`, `password` | | What you enter in the device. The password needs 8 characters or more. When an agent sets the source up, `user` is `device` unless you name one, and a password you do not give is made by pine. |
|
|
204
|
+
| `host` | `"0.0.0.0"` | The address to listen on. Devices on your network have to reach it. |
|
|
205
|
+
| `allow` | everyone | Only these clients: a full address (`"192.168.1.40"`), or a prefix that ends in a dot (`"192.168.2."`). |
|
|
206
|
+
| `pasvPorts` | any | `[from, to]`: the ports for data connections, if a firewall has to know them. |
|
|
207
|
+
| `maxFileMb` | `50` | Largest file accepted. |
|
|
208
|
+
| `keepDays` | `7` | Received files are deleted after this many days. |
|
|
209
|
+
|
|
210
|
+
Event `upload`: `{ file, path, size, from, images? }`. `file` is the name the device gave it,
|
|
211
|
+
`path` is where it lies now (`data/sources/<name>/files/`, or `data/sources/<name>@<agent>/files/`
|
|
212
|
+
for an agent's own source). A picture is shown to the agent with the event. A video or any
|
|
213
|
+
other file is not: the agent reads it from `path` (see "Files a source received" in
|
|
214
|
+
[sources.md](sources.md)).
|
|
215
|
+
|
|
216
|
+
It speaks plain FTP, as these devices do, so the password crosses your network unencrypted. It
|
|
217
|
+
only receives: nothing can be listed, read, changed or deleted through it, and every file lands
|
|
218
|
+
under a name of pine's choosing in the source's own folder. Set the device to passive mode.
|
|
219
|
+
|
|
220
|
+
### telegram
|
|
221
|
+
|
|
222
|
+
A Telegram bot: you write to it from wherever you are, and the agent that subscribed answers
|
|
223
|
+
there. It is how you reach your agents when you are away, and how they reach you.
|
|
224
|
+
|
|
225
|
+
> Watch Telegram. You are my assistant there, and what you want to tell me while I am away
|
|
226
|
+
> goes to that chat.
|
|
227
|
+
|
|
228
|
+
The agent calls `add_source`. The dialog you confirm has a field for the bot's token: you type
|
|
229
|
+
or paste it there, where it is not shown and does not pass through the agent or its model (see
|
|
230
|
+
"Secrets" in [sources.md](sources.md)). Then the agent tells you the one remaining step, which ties the bot to your account:
|
|
231
|
+
|
|
232
|
+
1. Open the bot in Telegram and write anything to it.
|
|
233
|
+
2. The bot answers that account with a six-digit code.
|
|
234
|
+
3. pine shows a question on your computer that names the account and has a field for the code.
|
|
235
|
+
Type the code from your phone and press enter.
|
|
236
|
+
|
|
237
|
+
From then on that account is the only one the bot listens to. The code is the point: it shows
|
|
238
|
+
that the person at pine's keyboard holds that Telegram account. A stranger who finds the bot
|
|
239
|
+
can make the question appear, but does not see your screen, and you do not have their code; a
|
|
240
|
+
key pressed by accident cannot say yes. An account you deny is not put to you again, and only
|
|
241
|
+
one question is open at a time.
|
|
242
|
+
|
|
243
|
+
Anyone can write to a bot; that is how Telegram works. What they write reaches no agent. Before
|
|
244
|
+
you have paired, an unknown account gets the code message (and a line if the code that was
|
|
245
|
+
typed was wrong) and nothing else. After you have,
|
|
246
|
+
unknown accounts get no answer at all and cause no question.
|
|
247
|
+
|
|
248
|
+
```json
|
|
249
|
+
"telegram": { "type": "telegram", "token": "${PINE_SOURCE_TELEGRAM_MAIN_TOKEN}", "trust": "owner", "description": "my Telegram" }
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
| Option | Default | |
|
|
253
|
+
|---|---|---|
|
|
254
|
+
| `token` | | The bot's token. Create a bot by writing `/newbot` to @BotFather in Telegram; it answers with the token. Kept in `.env`. |
|
|
255
|
+
| `allow` | pairing | Telegram user ids that may talk to the bot, if you know yours. Without it the bot is paired as described above. With several ids, what the agent sends unasked and the approval requests go to the first. |
|
|
256
|
+
| `approvals` | `true` | Approval requests are also sent to that chat, with buttons to answer them: Allow, Deny and, where a yes can be remembered for the conversation, Always. `/allow` and `/deny` work too (with the request's id when more than one is open). A question the agent asks with its `question` tool arrives the same way when it offers answers to pick from, with one button for each and one for "No answer"; an answer in your own words, and a question without answers to pick from, you type in the terminal UI. A request that needs a secret typed is never sent there. |
|
|
257
|
+
| `apiBase` | Telegram's | The address of a Bot API server of your own. |
|
|
258
|
+
|
|
259
|
+
Event `message`: `{ text, name, username, chatId, messageId, images? }`. A photo you send (up
|
|
260
|
+
to 10 MB) is shown to the model. The bot listens only in its private chat with you: what is
|
|
261
|
+
written in a group it was added to reaches no agent.
|
|
262
|
+
|
|
263
|
+
What makes it different from the other types:
|
|
264
|
+
|
|
265
|
+
- **It is a conversation.** Your message reaches the agent as your message, not as a report
|
|
266
|
+
about one, in the conversation that subscribed. The answer is sent back to the chat and is
|
|
267
|
+
also in that conversation in the terminal UI, so both show the same thread.
|
|
268
|
+
- **It speaks for you.** Telegram vouches for the account and you proved it is yours, so what
|
|
269
|
+
it writes counts as your own words: full tools, no "outside content" mark. That is why
|
|
270
|
+
pairing takes a code and why the set-up is always put to you.
|
|
271
|
+
- **It can send.** The agent writes to you on its own with the `send` tool, with a picture or
|
|
272
|
+
a video if it has one to show, and without a question to you; see "Sending through a
|
|
273
|
+
source" in [sources.md](sources.md).
|
|
274
|
+
- **Messages are not held back.** The ten-second pause between events that protects a model from
|
|
275
|
+
a chatty device does not apply unless the subscription sets a `throttle` itself.
|
|
276
|
+
|
|
277
|
+
Who the bot listens to is said wherever sources are listed: `pine sources`, `/sources` in the
|
|
278
|
+
terminal UI, and the agent's `sources` tool show "listens to nobody yet", or the account you
|
|
279
|
+
paired with its name and Telegram id. A bot never learns a phone number; the account's id is
|
|
280
|
+
what Telegram vouches for. To pair with another account, remove the source and set it up
|
|
281
|
+
again: the pairing goes with the source.
|
|
282
|
+
|
|
283
|
+
Messages are fetched by long polling: one request that Telegram answers the moment something
|
|
284
|
+
arrives. Nothing listens on a port, so it works behind any router. Only one program can poll a
|
|
285
|
+
bot at a time; a second one makes both miss messages.
|
|
286
|
+
|
|
287
|
+
### http
|
|
288
|
+
|
|
289
|
+
A web address that is looked at again and again. For what cannot call you: a price, a version,
|
|
290
|
+
a status page, a device on your network that has a state page.
|
|
291
|
+
|
|
292
|
+
> Watch https://shop.example/fridge-3000 and tell me when the price drops below 500.
|
|
293
|
+
|
|
294
|
+
```json
|
|
295
|
+
"price": { "type": "http", "url": "https://shop.example/fridge-3000", "match": "Price: ([0-9.,]+)", "every": "30m", "description": "the price of the fridge" }
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
| Option | Default | |
|
|
299
|
+
|---|---|---|
|
|
300
|
+
| `url` | | The address, `https://...` (`http://` for a device on the local network). |
|
|
301
|
+
| `every` | `"5m"` | How often to look. At least `"30s"` when an agent sets it up, and at least `"2m"` with `render`. |
|
|
302
|
+
| `path` | | For an answer in JSON: watch only this field, `"data.price"` or `"items.0.state"`. `path`, `values`, `match` and `select` are four ways to say what counts in the answer; a source takes one of them. |
|
|
303
|
+
| `match` | | For a page or text: watch only what this regular expression finds, its first group if it has one. It is looked for in what a reader of the page sees, then in the page's source. |
|
|
304
|
+
| `select` | | For a page: watch only the text of the first element this CSS selector matches, `".product .price"` or `"#stock > span"`. With `render` it is any selector a browser reads. |
|
|
305
|
+
| `attr` | | With `select`: the value of this attribute of the element instead of its text, `"content"` for a meta tag, `"data-price"`. |
|
|
306
|
+
| `render` | `false` | `true` for a page that a script fills in after loading: every look opens the page in pine's browser, without a window, and takes what a reader sees then. See "A page that a script fills in". Not together with `path`, `values`, `method`, `body`, `headers`, `token` or `tokenHeader`. |
|
|
307
|
+
| `values` | | For an answer in JSON: several fields at once, each under a name of your own: `{ "price": "data.price", "stock": "data.stock.count" }`. Twenty at most. |
|
|
308
|
+
| `method` | `"GET"` | `"POST"` for an address that is asked by sending something. Nothing else: a source only looks. |
|
|
309
|
+
| `body` | | With `POST`: what to send, as text. Sent as JSON when it is JSON. Not for secrets. |
|
|
310
|
+
| `headers` | | Headers to send, `{ "accept": "application/json" }`. Not for secrets. |
|
|
311
|
+
| `token` | | An access token or key, sent as `Authorization: Bearer <token>`. Kept in `.env`. |
|
|
312
|
+
| `tokenHeader` | | Send the token as this header instead, as it is: `"x-api-key"`, `"cookie"`. |
|
|
313
|
+
| `local` | `false` | The address is on your own network under a name that does not say so (`nas.home`, `ha.lan`). Not needed for an IP address, `localhost` or a `.local` name. |
|
|
314
|
+
|
|
315
|
+
| Event | | |
|
|
316
|
+
|---|---|---|
|
|
317
|
+
| `changed` | `{ url, value, previous, number?, previousNumber? }` | With `path`, `match` or `select`: the value now and before. |
|
|
318
|
+
| `changed` | `{ url, values, previous, changed }` | With `values`: all the fields now and before, and the names of those that differ. |
|
|
319
|
+
| `changed` | `{ url, added, removed, more? }` | Without: the lines that are new and the lines that are gone, up to 30 of each. |
|
|
320
|
+
| `down` | `{ url, error, looks }` | After three failed looks in a row, once. |
|
|
321
|
+
| `up` | `{ url }` | When it answers again. |
|
|
322
|
+
|
|
323
|
+
Without `path`, `values`, `match` or `select` the whole answer counts. For a page that is its text as a reader
|
|
324
|
+
sees it: scripts, styles and markup are left out, so a counter in a script does not report a
|
|
325
|
+
change at every look. A page that shows the time still does; give it a `match`.
|
|
326
|
+
|
|
327
|
+
#### One element of a page
|
|
328
|
+
|
|
329
|
+
`match` finds a value by the text around it; `select` finds it by where it stands in the page,
|
|
330
|
+
which holds better when the words around a price change with the language or the season:
|
|
331
|
+
|
|
332
|
+
```json
|
|
333
|
+
"fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "select": ".product-detail .price", "every": "30m", "description": "the price of the fridge" }
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
A selector is made of tags (`span`), ids (`#total`), classes (`.price`), attributes (`[itemprop]`,
|
|
337
|
+
`[itemprop=price]`, also `~=`, `^=`, `$=`, `*=`), put together (`span.price`, `li.item.sale`) and
|
|
338
|
+
nested with a space (somewhere inside) or `>` (directly inside); a comma separates alternatives.
|
|
339
|
+
Nothing with a colon (`:first-child`, `:not(...)`) and no `+` or `~`: such a selector is refused
|
|
340
|
+
with what it takes instead. The first element in the page that matches counts. Many shops also
|
|
341
|
+
state the price for machines, which changes less often than the layout:
|
|
342
|
+
|
|
343
|
+
```json
|
|
344
|
+
"fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "select": "meta[property=product:price:amount]", "attr": "content" }
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
The page is read as the server sends it. What a script fills in after loading is not
|
|
348
|
+
there: for such a page set `render`, or watch the address the script itself asks (see "A
|
|
349
|
+
page that a script fills in").
|
|
350
|
+
|
|
351
|
+
#### A page that a script fills in
|
|
352
|
+
|
|
353
|
+
Many pages arrive almost empty, and a script of the page then fetches the price or the state
|
|
354
|
+
and writes it in. A plain look reads the page as the server sends it, so it finds the
|
|
355
|
+
placeholder ("loading", an empty element, nothing at all) and never the value. With `render`
|
|
356
|
+
the look is made by a browser:
|
|
357
|
+
|
|
358
|
+
> The price on https://shop.example/fridge-3000 only shows up after the page has loaded. Watch
|
|
359
|
+
> it and tell me when it drops.
|
|
360
|
+
|
|
361
|
+
```json
|
|
362
|
+
"fridge": { "type": "http", "url": "https://shop.example/fridge-3000", "render": true, "select": ".product-detail .price", "every": "30m", "description": "the price of the fridge" }
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
At every look pine starts its browser (the one the browser tools use, which `pine init` and
|
|
366
|
+
`pine browser-install` fetch), without a window and with an empty profile, opens the address,
|
|
367
|
+
waits until the page has loaded and then until it has stopped asking for things (five seconds
|
|
368
|
+
at most), and takes what a reader sees. Then the browser is closed. What counts is chosen as
|
|
369
|
+
without `render`:
|
|
370
|
+
|
|
371
|
+
| With | What counts |
|
|
372
|
+
|---|---|
|
|
373
|
+
| `select` | The text of the first element the selector matches, as the browser shows it. The browser reads the selector, so everything CSS has works: `li:nth-child(2) .price`, `.offer:not(.sold-out)`, `h1 + p`. No element matches: the value is `null`. |
|
|
374
|
+
| `select` and `attr` | The value of that attribute of the element. |
|
|
375
|
+
| `match` | What the regular expression finds in the text a reader sees. The page's source is not looked at. |
|
|
376
|
+
| neither | The whole text of the page, line by line; an event names the lines that are new and gone. |
|
|
377
|
+
|
|
378
|
+
The events are the same, and an amount arrives as `number` as described below.
|
|
379
|
+
|
|
380
|
+
**What it costs.** A plain look is one request and takes a moment. A rendered look starts a
|
|
381
|
+
browser, runs the page's scripts and loads everything the page loads: a second or two and a
|
|
382
|
+
few hundred megabytes of memory while it lasts, at every look. So:
|
|
383
|
+
|
|
384
|
+
- `every` is at least `"2m"` when an agent sets the source up.
|
|
385
|
+
- Two browsers run at a time, for all sources together. A look that finds them busy waits.
|
|
386
|
+
- A look may take 45 seconds. After that it is ended and counts as failed.
|
|
387
|
+
- There is no "nothing new" answer (see below): the page is loaded in full each time.
|
|
388
|
+
|
|
389
|
+
**When the plain look is the better choice.** A script that fills a page in gets the value
|
|
390
|
+
from somewhere, usually an address that answers JSON. If you can find it (the network tab of
|
|
391
|
+
your browser's developer tools shows what the page asks), watch that address without `render`
|
|
392
|
+
and with `path`: it is faster, it can be asked every 30 seconds, a number arrives as a number,
|
|
393
|
+
it can take a `token`, and it does not break when the shop changes its layout. Use `render`
|
|
394
|
+
when there is no such address, when it cannot be asked from outside the page, or when what you
|
|
395
|
+
want is what the page shows.
|
|
396
|
+
|
|
397
|
+
**What does not go with it.** A set-up with one of these and `render` is refused, with the
|
|
398
|
+
reason:
|
|
399
|
+
|
|
400
|
+
| Option | Why not |
|
|
124
401
|
|---|---|
|
|
125
|
-
| `
|
|
126
|
-
| `
|
|
127
|
-
| `
|
|
128
|
-
| `
|
|
402
|
+
| `path`, `values` | They read an answer in JSON. There is no page to render. Leave `render` out. |
|
|
403
|
+
| `method`, `body` | A rendered page is asked the way a reader's browser asks it. |
|
|
404
|
+
| `headers`, `token`, `tokenHeader` | The browser would send them with every request the page makes, to every address the page loads from. So a page behind a login cannot be watched this way. |
|
|
405
|
+
| `local: true` | See below: a name that has to be looked up is not opened on your network. |
|
|
406
|
+
|
|
407
|
+
**What the page can reach.** The page's own scripts run in that browser, as they would in
|
|
408
|
+
yours, and the dialog you confirm says so. What they can reach is narrower than in yours:
|
|
409
|
+
|
|
410
|
+
- Every connection of that browser goes through the same guard as the browser tools'
|
|
411
|
+
([security.md](security.md)). The public internet is reachable. Your own network and this
|
|
412
|
+
computer are not: not for the page, and not for anything the page loads or its scripts ask.
|
|
413
|
+
- One exception, which is the page itself: an address that says by itself that it is local
|
|
414
|
+
(`localhost`, a `.local` name, an IP address in a private range) is opened, and the browser
|
|
415
|
+
then reaches exactly that host and port and nothing else on your network. The dialog names
|
|
416
|
+
it.
|
|
417
|
+
- A name that only turns out to be local when it is looked up (`nas.home`, `ha.lan`) is not
|
|
418
|
+
opened with `render`, also not with `local: true`: what such a name points at can change,
|
|
419
|
+
and a page's scripts would run there. Give the device by its IP address.
|
|
420
|
+
- pine's own control port is never opened.
|
|
421
|
+
- The look counts only if the browser ends up on the host that was set up. A page that sends
|
|
422
|
+
it on to another host is a failed look that names the host.
|
|
423
|
+
- The browser keeps nothing from one look to the next (no cookies, no logins), downloads
|
|
424
|
+
nothing, and grants the page no permission (camera, location, notifications). A window the
|
|
425
|
+
page opens is closed.
|
|
426
|
+
|
|
427
|
+
What such a source reports is outside content, like everything an `http` source reports.
|
|
428
|
+
|
|
429
|
+
**When a look fails.** A failed look is retried with a growing pause, and `pine sources` shows
|
|
430
|
+
since when it fails and why; after three in a row the source reports `down`, and `up` when the
|
|
431
|
+
page opens again.
|
|
432
|
+
|
|
433
|
+
| The status says | What it means |
|
|
434
|
+
|---|---|
|
|
435
|
+
| `pine's browser is not installed on this computer` | Type `pine browser-install` into a terminal. `pine init` and `pine update` do it too. |
|
|
436
|
+
| `the page did not finish loading within 45 s` | The page is slow, or never finishes. |
|
|
437
|
+
| `"select" is not a CSS selector the browser can read` | The selector has a mistake in it. |
|
|
438
|
+
| `it sends on to <host>, which is not the address that was confirmed` | The page moved. Set the source up with the address it leads to, if that is the right one. |
|
|
439
|
+
| `<host> leads to an address on your own network` | The name points into your network. With `render` only an address that says so itself is opened there. |
|
|
440
|
+
| `it answered 403` (or another number) | The site turned the browser away. Some sites do that to a browser without a window. |
|
|
441
|
+
| `the browser could not load it (...)` | No connection, an invalid certificate, or the like; the words in brackets are the browser's. |
|
|
129
442
|
|
|
130
|
-
`
|
|
131
|
-
|
|
132
|
-
|
|
443
|
+
`found nothing with the selector` is not a failed look: the page opened and no element matched.
|
|
444
|
+
If the value is there in your own browser, the selector is wrong, or the page fills the
|
|
445
|
+
element in later than the look waits.
|
|
133
446
|
|
|
134
|
-
|
|
447
|
+
#### An amount as a number
|
|
135
448
|
|
|
136
|
-
A
|
|
137
|
-
|
|
449
|
+
A value that reads as a number or an amount, and nothing else, also arrives as `number` (and
|
|
450
|
+
the one before as `previousNumber`), so that a subscription can compare it:
|
|
138
451
|
|
|
139
452
|
```json
|
|
140
|
-
{ "source": "
|
|
453
|
+
"subscribe": [{ "source": "fridge", "when": { "payload.number": { "lt": 500 } }, "note": "Tell me: the fridge is under 500 now." }]
|
|
141
454
|
```
|
|
142
455
|
|
|
143
|
-
|
|
|
456
|
+
| The value | `number` |
|
|
144
457
|
|---|---|
|
|
145
|
-
| `"
|
|
146
|
-
| `
|
|
147
|
-
| `
|
|
148
|
-
| `
|
|
149
|
-
| `
|
|
150
|
-
| `{ "exists": true }` | is there at all |
|
|
151
|
-
| `{ "ne": "ok" }` | is anything but the value |
|
|
152
|
-
|
|
153
|
-
All fields named have to hold. The agent sets `when` with its `subscribe` tool; in a file it
|
|
154
|
-
is a key of the subscription. Many types also narrow at the source (`match`, `keywords`,
|
|
155
|
-
`topics`, `severity`): what is filtered there is not even an event. `throttle` and `debounce`
|
|
156
|
-
of a subscription keep a source that reports in bursts from waking the agent for each one,
|
|
157
|
-
and a subscription with `"session": "new"` gives every event a conversation of its own
|
|
158
|
-
([sources.md](sources.md), "A fresh conversation for every event").
|
|
159
|
-
|
|
160
|
-
### What all types have in common
|
|
161
|
-
|
|
162
|
-
- **Nothing runs until somebody is subscribed.** A configured source opens no port and no
|
|
163
|
-
connection. The first subscription starts it, the last one to end stops it.
|
|
164
|
-
- **The start is the starting point.** What is there when a source starts (the entries of a
|
|
165
|
-
feed, the mail in a mailbox, the state of a device) is not reported. Only what comes after is.
|
|
166
|
-
- **What is reported is bounded.** A text is cut to a few thousand characters, and most types
|
|
167
|
-
that are told things report at most so many events a minute (`perMinute` where it can be
|
|
168
|
-
set); what was left out is counted in the next event's `dropped`.
|
|
169
|
-
- **An address is the one you confirmed.** A public address has to stay on the public
|
|
170
|
-
internet: if its name one day leads into your network, the source fails and says so. A local
|
|
171
|
-
address is said to be local in the dialog. A redirect to another host is not followed, and a
|
|
172
|
-
token goes only to the host it was given for.
|
|
173
|
-
- **A source that also sends, sends only where you listed.** `sendTo` of `mail`, `slack`,
|
|
174
|
-
`matrix` and `discord`, `publish` of `mqtt`. Without the list nothing can be sent.
|
|
175
|
-
- **How it is doing is said where sources are listed** (`pine sources`, `/sources`, the
|
|
176
|
-
agent's `sources` tool): connected since when, last looked when, failing since when and why.
|
|
458
|
+
| `12.5`, `"12.5"`, `"12,5"`, `"87 %"`, `"-3,5 °C"`, `"3 GB"` | 12.5, 12.5, 12.5, 87, -3.5, 3 |
|
|
459
|
+
| `"$1,299.00"`, `"1.299,00 €"`, `"1 299,00 kr"`, `"1'299.50"` | 1299, 1299, 1299, 1299.5 |
|
|
460
|
+
| `"1,299"`, `"1.299"` | 1299 and 1.299: as a program writes numbers |
|
|
461
|
+
| `"1.299 €"`, `"1,299 €"` | 1299 and 1.299: next to `€`, `zł`, `kr` or `Kč` a dot groups thousands and a comma is the decimal point |
|
|
462
|
+
| `"1.2.3"`, `"12:30"`, `"2026-05-01"`, `"about 5"`, `"sold out"` | none |
|
|
177
463
|
|
|
178
|
-
|
|
464
|
+
The one case that can be read two ways is a single separator followed by exactly three digits,
|
|
465
|
+
and the last two rows are how it is decided. Where that is not what the page means, take the
|
|
466
|
+
digits apart with `match` (`"([0-9]+)[.,][0-9]{3} "`) and compare the text.
|
|
467
|
+
|
|
468
|
+
#### Several fields, and an address that is asked with POST
|
|
469
|
+
|
|
470
|
+
```json
|
|
471
|
+
"fridge": {
|
|
472
|
+
"type": "http", "url": "https://shop.example/graphql", "method": "POST",
|
|
473
|
+
"body": "{\"query\": \"{ product(id: 7) { price stock { count } } }\"}",
|
|
474
|
+
"values": { "price": "data.product.price", "stock": "data.product.stock.count" },
|
|
475
|
+
"every": "30m", "description": "price and stock of the fridge"
|
|
476
|
+
}
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
One event says which of the fields changed: `{ "values": { "price": 479, "stock": 3 },
|
|
480
|
+
"previous": { "price": 499, "stock": 3 }, "changed": ["price"] }`. A field that is not in the
|
|
481
|
+
answer is `null`. Since these are the values of the JSON itself, numbers are numbers and
|
|
482
|
+
`when` compares them directly: `{ "payload.values.stock": { "gt": 0 } }`.
|
|
483
|
+
|
|
484
|
+
A `POST` is for asking, and what it sends is shown to you in full before the source is set up:
|
|
485
|
+
a source repeats it on a clock, with nobody there.
|
|
486
|
+
|
|
487
|
+
#### Nothing new
|
|
488
|
+
|
|
489
|
+
An address that names a version of its answer (`ETag`, `Last-Modified`) is asked "anything new
|
|
490
|
+
since?" from the second look on, and answers with one line when there is not. That is a look
|
|
491
|
+
like any other, only cheaper for both sides. `feed` and `calendar` ask the same way. A look
|
|
492
|
+
with `render` loads the page in full every time.
|
|
493
|
+
|
|
494
|
+
#### Which addresses it asks
|
|
495
|
+
|
|
496
|
+
An address on your own network or this computer is said to be one in the dialog you confirm.
|
|
497
|
+
Any other address has to stay on the public internet: if its name one day leads into your
|
|
498
|
+
network, the look fails and says so instead of fetching from there.
|
|
499
|
+
|
|
500
|
+
A redirect is followed only on the same host. One that leads to another host is a failed look
|
|
501
|
+
that says where it leads: you confirmed one address, and the token is for that one. An answer
|
|
502
|
+
is read up to 1 MB. A key that is part of the address (`?apikey=...`) is not a secret to pine:
|
|
503
|
+
it is stored and shown like the rest of the address.
|
|
504
|
+
|
|
505
|
+
Refused when it is set up: a user name or password in the address, `path` and `match`
|
|
506
|
+
together, and a header that carries a secret (`authorization`, `cookie`), which belongs into
|
|
507
|
+
`token`. Also more than one of `path`, `values`, `match` and `select`; `attr` without
|
|
508
|
+
`select`; a selector that cannot be read; a `method` other than `GET` and `POST`; a `body`
|
|
509
|
+
without `POST` or longer than 20000 characters; `values` that are empty, more than twenty, or
|
|
510
|
+
under a name that is not a plain word. One look may take 20 seconds and follow five redirects.
|
|
511
|
+
|
|
512
|
+
### command
|
|
513
|
+
|
|
514
|
+
A command that is run again and again. For what only a command can tell: free disk space, a
|
|
515
|
+
build, a branch that moved, the end of a log, whether a machine answers.
|
|
516
|
+
A command that keeps running and prints a line whenever something happens is followed instead
|
|
517
|
+
of run again and again: see "Following a command that keeps running" below.
|
|
518
|
+
|
|
519
|
+
> Tell me when the disk is more than 90% full.
|
|
520
|
+
|
|
521
|
+
```json
|
|
522
|
+
"disk": { "type": "command", "run": "df -h /", "match": "([0-9]+)%", "every": "10m", "description": "how full the disk is" }
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
| Option | Default | |
|
|
526
|
+
|---|---|---|
|
|
527
|
+
| `run` | | The command line. bash runs it from the workspace of the agent the source belongs to. |
|
|
528
|
+
| `every` | `"1m"` | How often. At least `"5s"` when an agent sets it up. |
|
|
529
|
+
| `match` | | Watch only what this regular expression finds in what the command prints. |
|
|
530
|
+
| `timeout` | `"30s"` | How long one run may take before it is stopped. |
|
|
531
|
+
|
|
532
|
+
Event `changed`: `{ command, code, previousCode?, value, previous }` with `match`, or
|
|
533
|
+
`{ command, code, previousCode?, added, removed, more? }` without. `code` is the exit code;
|
|
534
|
+
`previousCode` is there when it changed, so `test -f build/done` or `ping -c1 nas` report by
|
|
535
|
+
how they end.
|
|
536
|
+
|
|
537
|
+
The command runs **like a command in the shell of an agent in `safe` mode**: in the
|
|
538
|
+
agent's sandbox, with its private home folder and without your environment, also for an agent
|
|
539
|
+
that works in `auto` mode and has your whole account in your conversations (nobody is there to answer for what a
|
|
540
|
+
command on a clock does). It reaches the agent's own folders and nothing else: no network
|
|
541
|
+
unless `sandbox.network` is set, none of your home folder beyond `sandbox.read`, none of
|
|
542
|
+
pine's files. A source of all agents runs as the main agent. You are shown the
|
|
543
|
+
command line before it is set up, every time. A command that cannot be started or runs out of
|
|
544
|
+
time is a failed look (see "Looking again and again"), not an event. `timeout` is at least `"1s"`, and a command
|
|
545
|
+
line of more than 2000 characters is refused (put it into a script). If the agent's sandbox is
|
|
546
|
+
switched off, the command runs without one, as its shell does.
|
|
547
|
+
|
|
548
|
+
#### Following a command that keeps running
|
|
549
|
+
|
|
550
|
+
Some commands tell by themselves when something happens: `tail -F` a log, `log stream`,
|
|
551
|
+
`docker events`, `kubectl get pods -w`, the client of a broker or a database. With `follow`
|
|
552
|
+
the command is started once and kept running, and every line it prints is an event as it
|
|
553
|
+
comes. This is the type for every protocol that has a command line client and no source type
|
|
554
|
+
of its own.
|
|
555
|
+
|
|
556
|
+
> Tell me when the backup log says ERROR.
|
|
557
|
+
|
|
558
|
+
```json
|
|
559
|
+
"backup": { "type": "command", "run": "tail -n 0 -F ~/Library/Logs/backup.log", "follow": true, "match": "ERROR (.*)", "description": "errors in the backup log" }
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
| Option | Default | |
|
|
563
|
+
|---|---|---|
|
|
564
|
+
| `follow` | `false` | `true`: start the command once, keep it running, report each line it prints. `every` and `timeout` are not used then. |
|
|
565
|
+
| `perMinute` | `30` | With `follow`: report at most this many lines a minute. The rest are counted, and the next line that is reported carries the count as `dropped`. |
|
|
566
|
+
| `match` | | With `follow`: only lines in which this regular expression finds something. Its first group is the line's `value`. |
|
|
567
|
+
|
|
568
|
+
| Event | | |
|
|
569
|
+
|---|---|---|
|
|
570
|
+
| `line` | `{ command, line, value?, dropped? }` | One line the command printed. `value` is what `match` found, or the parsed line when it is JSON. |
|
|
571
|
+
| `ended` | `{ command, code?, error?, ran }` | The command ended (`ran`: after how many seconds, `error`: the end of what it wrote as errors). It is started again, after a pause that grows while it keeps ending at once. A command that ends the moment it is started is reported once, not at every try. |
|
|
572
|
+
|
|
573
|
+
| What to follow | `run` |
|
|
574
|
+
|---|---|
|
|
575
|
+
| A log file | `tail -n 0 -F /path/app.log` (`-n 0`: only what is written from now on; `-F`: also after the file was rotated) |
|
|
576
|
+
| The Mac's own log | `log stream --style ndjson --predicate 'subsystem == "com.apple.TimeMachine"'` |
|
|
577
|
+
| Docker | `docker events --format '{{json .}}'` |
|
|
578
|
+
| Kubernetes | `kubectl get pods -w -o json`, `kubectl get events -w` |
|
|
579
|
+
| systemd's journal, on another machine | `ssh nas journalctl -f -p warning -o json` |
|
|
580
|
+
| PostgreSQL `LISTEN` | a few lines of script around a client library that print each notification as a line |
|
|
581
|
+
| Redis | `redis-cli --raw subscribe alerts` |
|
|
582
|
+
| NATS, Kafka, RabbitMQ | `nats sub 'orders.>'`, `kcat -C -b broker -t topic -o end`, `amqp-consume -q queue cat` |
|
|
583
|
+
| A folder on another machine | `ssh nas inotifywait -m -e close_write /volume1/scans` |
|
|
584
|
+
|
|
585
|
+
A line is cleaned of the colour codes a terminal would act on and cut at 2000 characters; a
|
|
586
|
+
line that is JSON also arrives parsed, so a subscription's `when` can look into it
|
|
587
|
+
(`{"payload.value.status": "die"}` for a container that stopped).
|
|
588
|
+
|
|
589
|
+
The command runs in the same sandbox as one that is run again and again: the agent's own, as
|
|
590
|
+
in `safe` mode. That is why most of the examples above need something from you first: the log
|
|
591
|
+
file has to be readable by the agent (`sandbox.read`), and a client that talks to the network
|
|
592
|
+
needs `sandbox.network`. The dialog shows the command line, and that it will keep running.
|
|
593
|
+
|
|
594
|
+
On Linux `follow` is refused. The sandbox there cannot forbid a running command to make a file
|
|
595
|
+
that is read as instructions; pine takes such a file away after a command has ended, and a
|
|
596
|
+
followed command does not end. Run it again and again instead (`every`, with a `match`), or
|
|
597
|
+
have the command's lines sent to a `syslog` or `webhook` source.
|
|
598
|
+
|
|
599
|
+
When the source stops (the last subscriber left, or pine stops), the command and everything
|
|
600
|
+
it started is ended with it.
|
|
179
601
|
|
|
180
602
|
### feed
|
|
181
603
|
|
|
@@ -552,7 +974,7 @@ turned app passwords off.
|
|
|
552
974
|
the addresses on `sendTo` and nobody else; the type checks that itself, whatever a call names.
|
|
553
975
|
With one address on the list, `to` can be left out. Mail can reach anybody, so unlike a
|
|
554
976
|
Telegram message to you it counts as information leaving: after outside content the agent asks
|
|
555
|
-
before it sends (see "Sending through a source"). The text goes as plain text; no files.
|
|
977
|
+
before it sends (see "Sending through a source" in [sources.md](sources.md)). The text goes as plain text; no files.
|
|
556
978
|
|
|
557
979
|
Refused when it is set up: a `host` with a scheme, port or path in it, `"security": "none"` for
|
|
558
980
|
a server that is not local, a `from` entry that is neither an address nor `@domain`, `sendTo`
|
|
@@ -628,7 +1050,7 @@ What you write yourself is never reported to you.
|
|
|
628
1050
|
| `ignore` | | User ids that are never reported: a colleague's bot account, a channel's chatterbox. An app's id (`B...`) is taken too. What the token's own account writes (with a user token: you) is left out without being listed. |
|
|
629
1051
|
| `sendTo` | none | Where an agent may write, by id only: channels (`"C0123ABCD"`) and people (`"U0123ABCD"`, a direct message). Without it the source only listens. Needs `token`. See "Sending" below. |
|
|
630
1052
|
| `perMinute` | `60` | At most this many reports a minute, 1 to 600. What comes on top is counted and dropped, and the next report says how many were (`dropped`). |
|
|
631
|
-
| `history` | `true` | With a `token`: an agent may read what was written earlier where the source listens (`read_source`),
|
|
1053
|
+
| `history` | `true` | With a `token`: an agent may read what was written earlier where the source listens (`read_source`), an answer in a thread comes with the messages it answers, and what was written while the source was not connected is asked for afterwards. `false` turns all three off. |
|
|
632
1054
|
| `context` | `3` | How many earlier messages of a thread come with an answer in it, the thread's first message among them. `0` for none, `10` at most. |
|
|
633
1055
|
| `maxMb` | `10` | Files that come with a message are fetched up to this size (50 at most). A larger one is named in the event, without a `path`. Needs `token`. |
|
|
634
1056
|
| `keepDays` | `7` | Fetched files are deleted after this many days. |
|
|
@@ -649,7 +1071,7 @@ shown where the sources are listed: `there is no channel named "opps" that the a
|
|
|
649
1071
|
|
|
650
1072
|
| `events` | Event | Fields | When |
|
|
651
1073
|
|---|---|---|---|
|
|
652
|
-
| `"message"` | `message` | `{ text, user, userName?, channel, channelName?, kind, ts, thread?, reply?, bot?, mention?, keyword?, from?, files?, images?, dropped? }` | Somebody wrote something that counts. |
|
|
1074
|
+
| `"message"` | `message` | `{ text, user, userName?, channel, channelName?, kind, ts, thread?, reply?, bot?, mention?, keyword?, from?, files?, images?, late?, dropped? }` | Somebody wrote something that counts. |
|
|
653
1075
|
| `"reaction"` | `reaction` | `{ reaction, added, user, userName?, channel, channelName?, ts, author?, authorName? }` | Somebody put an emoji on a message (`added: true`) or took it away (`false`). `ts` is the message, `author` who wrote it. |
|
|
654
1076
|
| `"edit"` | `edited` | `{ text, previous?, user, userName?, channel, channelName?, kind, ts, mention?, keyword?, from? }` | A message was changed. `previous` is what it said before. The preview Slack puts under a link is not an edit. |
|
|
655
1077
|
| `"edit"` | `deleted` | `{ channel, channelName?, ts, previous? }` | A message was deleted. `previous` is what it said. |
|
|
@@ -658,6 +1080,7 @@ shown where the sources are listed: `there is no channel named "opps" that the a
|
|
|
658
1080
|
| `"pin"` | `pin` | `{ added, channel, channelName?, ts?, user, userName? }` | A message was pinned (`added: true`) or unpinned. |
|
|
659
1081
|
| `"command"` | `command` | `{ command, text, user, userName?, channel, channelName? }` | Somebody typed a slash command of your app: `command` is `"/deploy"`, `text` what followed it. Whoever typed it reads "Passed on." |
|
|
660
1082
|
| `"other"` | `event` | `{ type, data }` | Any other event your app is subscribed to. `type` is Slack's name for it, `data` the event as Slack sent it, cut at 4000 characters. |
|
|
1083
|
+
| always | `gap` | `{ text, from, to, seconds, found, looked, complete, missing? }` | The source was not connected for a minute or longer, and is again. `text` says it in a sentence. `complete: false` means that not everything from that time is known, and `missing` lists why. See "After a time without a connection". |
|
|
661
1084
|
|
|
662
1085
|
Any event can carry `dropped`: how many reports were left out before it, when more than
|
|
663
1086
|
`perMinute` counted in a minute.
|
|
@@ -674,6 +1097,7 @@ The fields of a `message`:
|
|
|
674
1097
|
| `thread`, `reply` | For an answer in a thread: `reply: true`, and `thread` is the time stamp of the thread. `thread` is what `send` takes to answer in that thread. |
|
|
675
1098
|
| `context` | For an answer in a thread: what it answers, as a list of `{ user, userName?, text, ts }`, oldest first. The thread's first message and the last ones before the answer, `context` of them (3). Not there with `history: false`, without a `token`, or when the app may not read history. |
|
|
676
1099
|
| `bot` | `true` when an app or integration posted it. |
|
|
1100
|
+
| `late` | `true` when it was written while the source was not connected, and pine asked Slack for it afterwards. `ts` says when it was written. |
|
|
677
1101
|
| `why` | Why it is reported, a word for each reason: `"dm"` (written to you directly), `"mention"`, `"keyword"`, `"from"` (its sender is on `from`), or `"channel"` when it is none of these and simply was written where the source listens. A list, since a message can be several: a subscription picks with `"payload.why": { "contains": "mention" }`. |
|
|
678
1102
|
| `mention`, `keyword` | Which entry of `mentions` it mentions (for `"me"`: the id), and which word on `keywords` it contains. |
|
|
679
1103
|
| `from` | `true` when its sender is on `from`. |
|
|
@@ -739,13 +1163,16 @@ On api.slack.com/apps, "Create New App", "From scratch":
|
|
|
739
1163
|
| names of people | | `users:read` |
|
|
740
1164
|
| names of channels, and `channels` by name | | `channels:read`, and `groups:read` for private channels |
|
|
741
1165
|
| the files of a message | | `files:read` |
|
|
1166
|
+
| what was written while it was not connected, in conversations it has seen a message in | | the `...:history` scope of that kind of conversation, which being told of messages there needs anyway |
|
|
1167
|
+
| the same in conversations it has not seen a message in yet (somebody who writes to you for the first time) | | `im:read` for direct conversations; without a list of `channels` also `channels:read`, `groups:read` and `mpim:read` |
|
|
742
1168
|
| posting a message | | `chat:write` (and `chat:write.public` to post in a public channel the bot is not in) |
|
|
743
1169
|
| a direct message to a person on `sendTo` | | `chat:write`, `im:write` |
|
|
744
1170
|
| an emoji on a message | | `reactions:write` |
|
|
745
1171
|
| sending a file | | `files:write` |
|
|
746
1172
|
|
|
747
1173
|
A scope that is missing does not stop the source, with one exception: a name is left out, a
|
|
748
|
-
file has no `path`,
|
|
1174
|
+
file has no `path`, a `send` says which scope Slack wants, and after a time without a
|
|
1175
|
+
connection the `gap` event names what could not be looked at. The exception is a channel
|
|
749
1176
|
given by its name, which cannot be found without `channels:read`.
|
|
750
1177
|
|
|
751
1178
|
A **bot token** (`xoxb-`) sees the channels the bot was invited to (`/invite @yourapp`) and
|
|
@@ -816,9 +1243,9 @@ there writes /invite and its name`.
|
|
|
816
1243
|
|
|
817
1244
|
If the connection ends it is made again, as described under `stream` ("Staying connected").
|
|
818
1245
|
Slack itself closes it now and then and says so beforehand; pine connects again at once.
|
|
819
|
-
Slack is pinged every few seconds, so a line that died without a word is noticed.
|
|
820
|
-
|
|
821
|
-
to a call, pine waits as long as Slack says and asks again.
|
|
1246
|
+
Slack is pinged every few seconds, so a line that died without a word is noticed. A machine
|
|
1247
|
+
that slept does not trust the line it finds open when it wakes: pine connects again. When
|
|
1248
|
+
Slack says "slow down" to a call, pine waits as long as Slack says and asks again.
|
|
822
1249
|
|
|
823
1250
|
Where the sources are listed, the line says how things stand: `connected since 14:02:11.
|
|
824
1251
|
listening to 2 channels and direct messages, for messages, reactions`, or `cannot connect
|
|
@@ -826,6 +1253,50 @@ since 14:02:11 (3 attempts): Slack refused apps.connections.open: invalid_auth`.
|
|
|
826
1253
|
that Slack does not take, and a channel name it does not know, are why a source does not
|
|
827
1254
|
start at all.
|
|
828
1255
|
|
|
1256
|
+
#### After a time without a connection
|
|
1257
|
+
|
|
1258
|
+
Slack does not send again what was written while nobody was connected: the machine was
|
|
1259
|
+
asleep, the network was gone, pine was not running. So pine notes when the connection last
|
|
1260
|
+
worked (also across a restart), and when it is connected again it asks Slack for what was
|
|
1261
|
+
written since, where the source listens and nowhere else:
|
|
1262
|
+
|
|
1263
|
+
- Each message that counts is reported like any other, with `late: true`, oldest first. An
|
|
1264
|
+
answer in a thread is found when the thread began among the last 50 messages of its channel.
|
|
1265
|
+
- After a minute or more, a `gap` event comes first:
|
|
1266
|
+
|
|
1267
|
+
```json
|
|
1268
|
+
{ "text": "Slack was not connected from 2026-03-02 14:02:11 to 2026-03-02 14:07:30 (5 minutes). 2 messages written in that time follow, marked \"late\". It looked through all 14 conversations it listens to.",
|
|
1269
|
+
"from": "2026-03-02 14:02:11", "to": "2026-03-02 14:07:30", "seconds": 319, "found": 2, "looked": 14, "complete": true }
|
|
1270
|
+
```
|
|
1271
|
+
|
|
1272
|
+
`complete: false` means that the time is not fully known, and `missing` says why, each reason
|
|
1273
|
+
a sentence. An agent that tells you what came in passes that on, and does not say "nothing
|
|
1274
|
+
new" for a time it could not look at:
|
|
1275
|
+
|
|
1276
|
+
| `missing` says | Why | What helps |
|
|
1277
|
+
|---|---|---|
|
|
1278
|
+
| `direct conversations this source had not seen a message in were not looked at (the app lacks the scope im:read)` | pine knows a conversation from the first message it saw in it. Which others there are, it has to ask Slack, and the app may not. The same for `public channels` (`channels:read`), `private channels` (`groups:read`) and `group conversations` (`mpim:read`) when the source has no list of `channels`. | Add the scope to the Slack app (for a user token among the User Token Scopes) and install the app again. |
|
|
1279
|
+
| `1 direct conversation could not be read (the app lacks the scope im:history)` | The app may not read what was written earlier in that kind of conversation. | Add that scope. |
|
|
1280
|
+
| `3 conversations could not be read (not_in_channel)` | Slack refused, in its own word, or could not be reached. | For `not_in_channel`: invite the app to the channel. |
|
|
1281
|
+
| `this source has no token, so it could not ask Slack what was written in that time` | Asking needs the `token`. | Set the source up again with a `token`. |
|
|
1282
|
+
| `this source was set up not to read what was written earlier ("history": false)` | | |
|
|
1283
|
+
| `2 more messages from that time were left out, since at most 60 are reported in a minute` | `perMinute`. | The agent reads the rest with `read_source`. |
|
|
1284
|
+
| `what else happened in that time (reactions, edits) cannot be asked for afterwards` | Slack keeps messages, and not who reacted, joined or edited when. A source that reports such things always says this. | |
|
|
1285
|
+
| `what the people on "from" wrote outside the listed channels was not looked for` | With a list of `channels`, only those and direct conversations are asked for. | |
|
|
1286
|
+
| `12 more conversations were not looked at: 200 is the most at once` | | |
|
|
1287
|
+
|
|
1288
|
+
A gap shorter than a minute (Slack renewing the connection, which it does now and then and
|
|
1289
|
+
which takes a moment) is made up for quietly: pine asks in the listed `channels` and in the conversations
|
|
1290
|
+
it has seen a message in, 40 at most, and reports no `gap`. After a longer one it also asks
|
|
1291
|
+
Slack which conversations there are, 200 at most. Reactions, edits and the other kinds are
|
|
1292
|
+
never made up for.
|
|
1293
|
+
|
|
1294
|
+
Where the sources are listed, the last gap is said too: `connected since 14:07:30. listening
|
|
1295
|
+
to ... Not connected from 2026-03-02 14:02:11 to 2026-03-02 14:07:30: not everything from
|
|
1296
|
+
then is known`. A subscription that picks with `when` (`"payload.why": { "contains":
|
|
1297
|
+
"mention" }`) is not woken by a `gap`, which has no `why`: an agent that is asked what came
|
|
1298
|
+
in looks at that line.
|
|
1299
|
+
|
|
829
1300
|
#### Coming from the old Slack monitor
|
|
830
1301
|
|
|
831
1302
|
The OpenClaw plugin `openclaw-slack-monitor`:
|
|
@@ -893,7 +1364,7 @@ answered the moment something arrives, so nothing listens on a port.
|
|
|
893
1364
|
| `mentions` | | User ids (`@paul:matrix.org`): only messages that mention one of them. |
|
|
894
1365
|
| `keywords` | | Only messages that contain one of these words. Capitals do not matter. |
|
|
895
1366
|
| `ignore` | | User ids that are never reported. What the account itself writes never is. |
|
|
896
|
-
| `sendTo` | none | Room ids an agent may write to. Without it the source only listens. See "Sending through a source". |
|
|
1367
|
+
| `sendTo` | none | Room ids an agent may write to. Without it the source only listens. See "Sending through a source" in [sources.md](sources.md). |
|
|
897
1368
|
| `local` | `false` | The homeserver is on your own network under a name that does not say so. |
|
|
898
1369
|
|
|
899
1370
|
Event `message`: `{ text, user, room, roomName?, kind, id, mention?, keyword?, dropped? }`.
|
|
@@ -952,7 +1423,7 @@ to Discord's gateway, so nothing listens on a port.
|
|
|
952
1423
|
| `keywords` | | In servers, only messages that contain one of these words. Capitals do not matter. With `mentions` as well, either is enough. |
|
|
953
1424
|
| `dms` | `true` | Direct messages to the bot count, without a mention or a keyword. |
|
|
954
1425
|
| `bots` | `false` | What other bots and webhooks write counts too. |
|
|
955
|
-
| `sendTo` | none | Channel ids an agent may write to. Without it the source only listens. See "Sending through a source". |
|
|
1426
|
+
| `sendTo` | none | Channel ids an agent may write to. Without it the source only listens. See "Sending through a source" in [sources.md](sources.md). |
|
|
956
1427
|
| `apiBase` | `https://discord.com/api/v10` | A Discord-compatible server of your own: reached over https, or on this machine. |
|
|
957
1428
|
|
|
958
1429
|
Ids are long numbers. With "Developer Mode" on (User Settings, Advanced), a right click on a
|
|
@@ -1440,10 +1911,10 @@ On Linux and Windows a `system` source cannot be set up. What a command can tell
|
|
|
1440
1911
|
|
|
1441
1912
|
| Wanted | Instead |
|
|
1442
1913
|
|---|---|
|
|
1443
|
-
| A page that a script fills in after loading | It has no type of its own: `http` with `render` has a browser make the look (
|
|
1914
|
+
| A page that a script fills in after loading | It has no type of its own: `http` with `render` has a browser make the look (see "A page that a script fills in" under `http`). Where the page's script asks an address that answers JSON, watching that with `path` is lighter. |
|
|
1444
1915
|
| A calendar's "ten minutes before" | A schedule the agent sets when `calendar` reports the entry ([events.md](events.md), "Schedules"). |
|
|
1445
1916
|
| Bluetooth sensors and beacons | A bridge that publishes them to MQTT (Theengs Gateway, OpenMQTTGateway), and the `mqtt` type. |
|
|
1446
1917
|
| iMessage, WhatsApp | No type. Signal works through signal-cli and `stream`; Telegram, Slack, Matrix and Discord have types. |
|
|
1447
1918
|
| KNX, Z-Wave, Zigbee, HomeKit, Matter | Through the hub that speaks them (Home Assistant, zigbee2mqtt, zwave-js-ui), by `mqtt` or `stream`. |
|
|
1448
|
-
| Something with a command line client and nothing else | `command` with `follow` (
|
|
1919
|
+
| Something with a command line client and nothing else | `command` with `follow` (see "Following a command that keeps running" under `command`). |
|
|
1449
1920
|
| Something of your own | A source type of your own is a small module ([sources.md](sources.md), "A source type of your own"); the main agent can write and register one when you ask it. |
|