pikuri-thunderbird 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/DESIGN.md +460 -0
- data/README.md +66 -0
- data/lib/pikuri/thunderbird/calendar.rb +372 -0
- data/lib/pikuri/thunderbird/calendar_create.rb +126 -0
- data/lib/pikuri/thunderbird/calendar_read.rb +130 -0
- data/lib/pikuri/thunderbird/calendar_search.rb +145 -0
- data/lib/pikuri/thunderbird/compose_guard.rb +281 -0
- data/lib/pikuri/thunderbird/contact_search.rb +90 -0
- data/lib/pikuri/thunderbird/database_snapshot.rb +155 -0
- data/lib/pikuri/thunderbird/date_helpers.rb +93 -0
- data/lib/pikuri/thunderbird/extension.rb +196 -0
- data/lib/pikuri/thunderbird/gloda/contacts.rb +229 -0
- data/lib/pikuri/thunderbird/gloda/mail.rb +216 -0
- data/lib/pikuri/thunderbird/gloda.rb +153 -0
- data/lib/pikuri/thunderbird/ical_line.rb +130 -0
- data/lib/pikuri/thunderbird/ics_event.rb +131 -0
- data/lib/pikuri/thunderbird/launcher.rb +76 -0
- data/lib/pikuri/thunderbird/mail_compose.rb +92 -0
- data/lib/pikuri/thunderbird/mail_read.rb +73 -0
- data/lib/pikuri/thunderbird/mail_search.rb +131 -0
- data/lib/pikuri/thunderbird/mailto_uri.rb +58 -0
- data/lib/pikuri/thunderbird/profile.rb +201 -0
- data/lib/pikuri-thunderbird.rb +76 -0
- data/prompts/thunderbird.txt +5 -0
- metadata +114 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: bd969845e2b1269b1be2163e7c670cb1fcf04821f9128bc8b89b487e5c4be828
|
|
4
|
+
data.tar.gz: 6441b467c55043738ad0475c6a3a55bbfdcc1c13c1b5bee7af3cb41d72498f99
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 41c4d6b2bab01c68bd8d3ac522f7f2bef92b17f5c101caa71375129a2d1d1dc4f34ff055781b548e117803c29b5312d5ef50fae2312888a4baaea5939e929f4e
|
|
7
|
+
data.tar.gz: '06927ce1a93609bc8541f87a6725801688a0387b1b88b8febac84b5c5c94ca192ab1c8405850b60e5120be71f0c6335e8436ef18254c07ea5d4ebb8f9cf78fc3'
|
data/DESIGN.md
ADDED
|
@@ -0,0 +1,460 @@
|
|
|
1
|
+
# pikuri-thunderbird — design & threat model
|
|
2
|
+
|
|
3
|
+
Audience: someone extending this gem or auditing its security posture.
|
|
4
|
+
Rationale for individual classes lives in their YARD headers; this file
|
|
5
|
+
covers the cross-cutting design and the threat model. The in-flight design
|
|
6
|
+
notes and still-open questions live in `ideas/thunderbird.md`.
|
|
7
|
+
|
|
8
|
+
## What it does
|
|
9
|
+
|
|
10
|
+
Gives a pikuri agent read-only access to the user's **local** Thunderbird
|
|
11
|
+
data — stored mail and cached calendars — by reading Thunderbird's own
|
|
12
|
+
on-disk profile. It never speaks IMAP/POP3/SMTP or CalDAV itself; it reads
|
|
13
|
+
files Thunderbird already wrote.
|
|
14
|
+
|
|
15
|
+
- **Mail** rides Thunderbird's Gloda index (`global-messages-db.sqlite`),
|
|
16
|
+
which has already MIME-decoded, HTML-stripped, and split every message
|
|
17
|
+
into body / subject / attachment-names / author / recipients. pikuri
|
|
18
|
+
treats it as a pre-decoded corpus.
|
|
19
|
+
- **Calendar** reads the `calendar-data/*.sqlite` stores, whose events are
|
|
20
|
+
normalized columns (no ICS parser for the common case).
|
|
21
|
+
|
|
22
|
+
## Threat model: the lethal trifecta
|
|
23
|
+
|
|
24
|
+
Email is the textbook lethal trifecta. The three legs:
|
|
25
|
+
|
|
26
|
+
1. **Private data** — the mailbox and calendars. Inherently present.
|
|
27
|
+
2. **Untrusted content** — every message body, subject, and attachment name
|
|
28
|
+
is authored by whoever sent the mail, i.e. potentially an attacker.
|
|
29
|
+
Inherently present.
|
|
30
|
+
3. **Egress** — the ability to send data out.
|
|
31
|
+
|
|
32
|
+
**The default surface removes leg 3 entirely.** The five tools
|
|
33
|
+
`thunderbird_mail_search`, `thunderbird_mail_read`,
|
|
34
|
+
`thunderbird_contact_search`, `thunderbird_calendar_search`, and
|
|
35
|
+
`thunderbird_calendar_read` are all inbound-only, and they are all `Extension`
|
|
36
|
+
gives unless a host opts into an outbound leg. An agent wired with just these (plus, say, a calculator) has legs 1
|
|
37
|
+
and 2 but no way to send anything anywhere, so a prompt injection buried in a
|
|
38
|
+
message body has no exfiltration channel. That is the "trifecta broken by
|
|
39
|
+
construction" posture `bin/pikuri-corpus` and `bin/pikuri-memory` demo.
|
|
40
|
+
|
|
41
|
+
The bundled `bin/pikuri-thunderbird` deliberately goes one step further — it
|
|
42
|
+
wires both outbound `allow_` flags to demo the *whole* surface — so it does
|
|
43
|
+
carry leg 3. But that leg is human-gated: compose can only open a Thunderbird
|
|
44
|
+
compose window the user Sends themselves (below), so there is an egress leg yet
|
|
45
|
+
no *autonomous* egress. A poisoned message can influence a draft, but cannot make it leave the
|
|
46
|
+
machine without the user clicking Send on a window they can see. So the demo's
|
|
47
|
+
trifecta is *intact-but-gated*, not severed — an honest smoke-alarm, not a
|
|
48
|
+
firewall.
|
|
49
|
+
|
|
50
|
+
Defense in depth on leg 2: **Spam/Junk/Trash are not searched.** On the
|
|
51
|
+
Gloda path this is *free* — Gloda already assigns those folders indexing
|
|
52
|
+
priority `-1` (never indexed), so the search backend structurally cannot
|
|
53
|
+
return them. This attenuates but does **not** sever leg 2: a *targeted*
|
|
54
|
+
injection lands in the Inbox (a reply, a newsletter, a calendar invite), not
|
|
55
|
+
in Spam. Record it as a mitigation, never a boundary. Honest wrinkle
|
|
56
|
+
(confirmed on a live box): Gmail's account-level Trash folders are `-1`, but
|
|
57
|
+
the `[Gmail]/Trash` *label* folder is indexed at priority 20, so a handful of
|
|
58
|
+
trashed Gmail messages can surface — the free skip is nearly, not perfectly,
|
|
59
|
+
complete.
|
|
60
|
+
|
|
61
|
+
## The snapshot mechanism
|
|
62
|
+
|
|
63
|
+
Thunderbird holds its SQLite DBs open with `locking_mode=EXCLUSIVE`, so a
|
|
64
|
+
plain read fails "database is locked" while it runs, and a lock-bypassing
|
|
65
|
+
`immutable=1` read of the live file risks a torn read. pikuri never queries
|
|
66
|
+
the live file. Instead {Pikuri::Thunderbird::DatabaseSnapshot} takes a **bracketed
|
|
67
|
+
`cp --reflink=auto`**: it reads the SQLite change counter (header bytes
|
|
68
|
+
24–27) and checks for a rollback `-journal` before and after the copy, and
|
|
69
|
+
accepts the copy only if the counter didn't move and no journal was present
|
|
70
|
+
at either edge (otherwise it discards and retries). Reflink makes the copy a
|
|
71
|
+
near-free metadata-only extent share on a copy-on-write filesystem.
|
|
72
|
+
|
|
73
|
+
The validated change counter doubles as a freshness stamp: each query
|
|
74
|
+
compares the live counter against it and re-copies (and, for mail, rebuilds
|
|
75
|
+
the FTS5 index) only when it moved. One signal drives both consistency and
|
|
76
|
+
freshness.
|
|
77
|
+
|
|
78
|
+
**Mail search index.** Gloda's own full-text index (`messagesText`) is an
|
|
79
|
+
fts3 virtual table behind Mozilla's custom `mozporter` tokenizer, which the
|
|
80
|
+
`sqlite3` gem cannot open. So pikuri builds its *own* ephemeral FTS5 index
|
|
81
|
+
over the decoded columns (`messagesText_content`) inside the snapshot copy —
|
|
82
|
+
external-content, so it stores only the index, not a second copy of the
|
|
83
|
+
text. Rebuilt (~1 s) only when the Gloda generation moves.
|
|
84
|
+
|
|
85
|
+
### Privacy caveat
|
|
86
|
+
|
|
87
|
+
The snapshot is the decoded mailbox/calendar landing **outside** the
|
|
88
|
+
Thunderbird profile, in a `0700` dir under `~/.cache/pikuri/thunderbird/`.
|
|
89
|
+
This is a second at-rest copy — but at the *same trust level* as the source:
|
|
90
|
+
the Gloda and calendar DBs are already unencrypted plaintext on disk in the
|
|
91
|
+
profile, and the copy is user-owned with the same permissions. It is not a
|
|
92
|
+
new exposure class. The copy is `Pikuri::Finalizers`-reaped on a clean exit;
|
|
93
|
+
a crash can strand it until it's next overwritten or the dir is cleared
|
|
94
|
+
(re-derivable data, low harm).
|
|
95
|
+
|
|
96
|
+
The credential stores that share the profile directory (`logins.*`,
|
|
97
|
+
`key4.db`, `cert9.db`, `openpgp.sqlite`, …) are **never** opened — the
|
|
98
|
+
backends touch only `global-messages-db.sqlite` and `calendar-data/*.sqlite`.
|
|
99
|
+
|
|
100
|
+
## Outbound is a human-gated v2, never an autonomous send
|
|
101
|
+
|
|
102
|
+
Composing a mail or creating a calendar event is egress (a send, an invite,
|
|
103
|
+
a CalDAV sync). The design invariant, locked in before any of it ships:
|
|
104
|
+
|
|
105
|
+
> **pikuri-thunderbird performs no network egress and no autonomous send.**
|
|
106
|
+
> Every outbound action is a hand-off to Thunderbird's own UI — a pre-filled
|
|
107
|
+
> compose window (via a `mailto:` URI) or an `.ics` import wizard — where the
|
|
108
|
+
> *human* reviews and commits. pikuri writes nothing to Thunderbird's stores
|
|
109
|
+
> and sends nothing itself. No send tool ever ships.
|
|
110
|
+
>
|
|
111
|
+
> **And no attachment ever ships.** The outbound surface carries only fields
|
|
112
|
+
> that *render literally*: a plain-text body, a subject, addresses. No
|
|
113
|
+
> attachment of any kind (`.png`, `.zip`, `.txt` — the extension is
|
|
114
|
+
> irrelevant), no HTML body, no iCalendar `ATTACH` property.
|
|
115
|
+
|
|
116
|
+
This converts egress from *agent-automatic* to *human-action-required*. Note
|
|
117
|
+
what that does and does not buy — the leg is **downgraded, not removed**, and
|
|
118
|
+
the threat-model section above is the authority: once an `allow_` flag is on,
|
|
119
|
+
leg 3 is present. Only the default surface (both flags off) severs it.
|
|
120
|
+
|
|
121
|
+
What the downgrade rests on is narrow and worth naming, because it's the one
|
|
122
|
+
property a rubber-stamping human can't destroy: **the human sees the exact
|
|
123
|
+
bytes that leave.** The compose window shows the literal recipient, subject
|
|
124
|
+
and body that will be sent; the `.ics` wizard shows the event that will be
|
|
125
|
+
imported. Contrast a shell agent that asks "run `curl …`?" — there the human
|
|
126
|
+
approves a *program*, and what it actually uploads was never on screen. So a
|
|
127
|
+
poisoned message can influence a draft, but every byte of that draft was
|
|
128
|
+
displayed to someone before it moved.
|
|
129
|
+
|
|
130
|
+
That is a real reduction and not a boundary: it kills *silent* exfil — every
|
|
131
|
+
byte was on a screen — and leaves the covert channel, a secret spelled into
|
|
132
|
+
prose the reader skims past, sent as a reply to the injector (see "What the
|
|
133
|
+
body guards do *not* close" below).
|
|
134
|
+
|
|
135
|
+
**Resist the temptation to call the survivors low-bandwidth.** They aren't.
|
|
136
|
+
A first-letter acrostic carries roughly one letter per word, and a secret is
|
|
137
|
+
already text — so an agent *transcribes* it rather than computing anything.
|
|
138
|
+
Price the channel that way, since this gem wires no shell and no calculator:
|
|
139
|
+
a pasted `id_ed25519` is its ~400 base64 characters, so ~400 words; a whole
|
|
140
|
+
RSA PEM ~1600. A thread, not a wall — and the mailbox is exactly where
|
|
141
|
+
someone pasted the key "just to have it" five years ago. (An agent that
|
|
142
|
+
*does* hold local compute prices it lower still, by the secret's entropy —
|
|
143
|
+
~55 words for that ed25519 key; a network-severed sandbox cuts egress, never
|
|
144
|
+
arithmetic. See `book/os-assistant.md` §"Why 'too big to hide' is not a
|
|
145
|
+
defense".) The bytes-visible property is worth stating because it is
|
|
146
|
+
structural; "the payload is too big to hide" is not, and shouldn't be
|
|
147
|
+
offered as reassurance.
|
|
148
|
+
|
|
149
|
+
### Why no attachments — it's the same invariant as plain-text-only
|
|
150
|
+
|
|
151
|
+
An attachment is the cleanest way to defeat the whole gate. A key encoded in
|
|
152
|
+
the low bits of a `.png` is invisible to any reader: the human approves a
|
|
153
|
+
*filename* and a thumbnail, never the bytes. That is exactly the shape of a
|
|
154
|
+
shell agent asking "run `curl …`?" — an approved *reference* to content
|
|
155
|
+
nobody saw — and it would demote this hand-off from "human sees the payload"
|
|
156
|
+
to the same class as an autonomous send, no matter how careful the human is.
|
|
157
|
+
|
|
158
|
+
So "plain text only" and "no attachments" are not two policies but one:
|
|
159
|
+
**the outbound surface may contain only fields whose rendering is their
|
|
160
|
+
content.** HTML fails it (`<span style="display:none">`), attachments fail it,
|
|
161
|
+
and so would any richer field added later. Today this is enforced by
|
|
162
|
+
construction rather than by filtering — {Pikuri::Thunderbird::MailtoUri}
|
|
163
|
+
*builds* an allowlist of five parameters (`to`/`cc`/`bcc`/`subject`/`body`),
|
|
164
|
+
so there is no `attach=` to strip, and {Pikuri::Thunderbird::IcsEvent} emits
|
|
165
|
+
no `ATTACH` property. Enforced-by-absence is fragile against a future
|
|
166
|
+
"improvement" — `mailto:?attach=` was a real Thunderbird feature — which is
|
|
167
|
+
why the invariant is written down rather than left implied by the code.
|
|
168
|
+
|
|
169
|
+
**Compose is shipped, opt-in.** `thunderbird_mail_compose`
|
|
170
|
+
({Pikuri::Thunderbird::MailCompose}) is wired only when a host passes
|
|
171
|
+
`Extension.new(allow_compose_mail: true)` — off by default, so an unopted wiring has no
|
|
172
|
+
egress leg *by construction*. `bin/pikuri-thunderbird` opts in (to demo the
|
|
173
|
+
full surface). Turning compose on re-introduces leg 3, but only as a
|
|
174
|
+
human-committed hand-off: the tool opens a
|
|
175
|
+
`mailto:` compose window ({Pikuri::Thunderbird::MailtoUri} percent-encodes every
|
|
176
|
+
field so a body can't smuggle `&bcc=`), the body is plain-text-only (no hidden
|
|
177
|
+
content, no remote beacons), and a mechanical recipient guard
|
|
178
|
+
({Pikuri::Thunderbird::ComposeGuard}) runs first: a suspicious *body* fails
|
|
179
|
+
closed; suspicious/non-ASCII/punycode *recipients* are dropped for the human to
|
|
180
|
+
retype; a never-corresponded-with recipient domain (per
|
|
181
|
+
{Pikuri::Thunderbird::Gloda#domains_seen}) is flagged, not blocked; and a body
|
|
182
|
+
carrying a long base64 *blob* earns a located soft-warn (see
|
|
183
|
+
{Pikuri::Thunderbird::ComposeGuard#body_blob_note}). The launch
|
|
184
|
+
is detached (`setsid --fork`) so closing the agent never kills the user's
|
|
185
|
+
Thunderbird.
|
|
186
|
+
|
|
187
|
+
**What the body guards do *not* close.** They raise attention on the lazy exfil
|
|
188
|
+
— a control-byte/homoglyph body fails closed, a base64/hex/base32 dump earns a
|
|
189
|
+
note — but
|
|
190
|
+
they are not a barrier against an *adaptive* injector, who splits the blob with
|
|
191
|
+
whitespace or spells it in prose for free (the same injection payload carries
|
|
192
|
+
the evasion). The sharpest surviving channel is **reply-to-injector**: the
|
|
193
|
+
poisoning message's From-domain is already in Gloda's identities graph (which
|
|
194
|
+
counts received mail, both directions), so a reply to the attacker draws no
|
|
195
|
+
novelty flag. The check is also per-*domain*, so a never-seen address at a
|
|
196
|
+
domain you have mailed before is silent as well.
|
|
197
|
+
|
|
198
|
+
**What each guard really buys is a rung, not a wall.** Every check here
|
|
199
|
+
narrows the *form* the payload may take, never the payload itself, so an
|
|
200
|
+
adaptive injector simply steps down: fail the body on control bytes,
|
|
201
|
+
zero-width joiners and homoglyphs, and it drops to plain ASCII; note a
|
|
202
|
+
base64 dump and it moves to hex, then base32, then splits the run below the
|
|
203
|
+
threshold; catch those and what remains is **linguistic steganography** — the
|
|
204
|
+
secret spelled out by the first letters of an innocuous sentence ("I think
|
|
205
|
+
dogs are super cute"), by word choice, by sentence count. That bottom rung
|
|
206
|
+
reads as ordinary prose to the guards *and* to the human, which is where an
|
|
207
|
+
injector who has read this file starts. It is slower, and slow is not a
|
|
208
|
+
barrier: `book/os-assistant.md` §"Why 'too big to hide' is not a defense"
|
|
209
|
+
prices what fits. The human-gated, irreversible-send compose window — not the
|
|
210
|
+
guard — is the backstop all of this leans on.
|
|
211
|
+
|
|
212
|
+
**Create-event is shipped, opt-in.** `thunderbird_calendar_create`
|
|
213
|
+
({Pikuri::Thunderbird::CalendarCreate}) is wired only when a host passes
|
|
214
|
+
`Extension.new(allow_create_calendar_event: true)`. Same human-gated shape:
|
|
215
|
+
it builds an event `.ics` ({Pikuri::Thunderbird::IcsEvent} RFC-5545-escapes
|
|
216
|
+
every field so a description can't inject a second `VEVENT`), stages it where a
|
|
217
|
+
snap-confined Thunderbird can read it ({Pikuri::Thunderbird::Profile#outbox_dir}),
|
|
218
|
+
and hands it to TB's import wizard, where the human picks the destination
|
|
219
|
+
calendar (local-vs-CalDAV = their no-egress-vs-egress choice) and confirms.
|
|
220
|
+
Both outbound tools launch via the shared {Pikuri::Thunderbird::Launcher}. The
|
|
221
|
+
full v2 rationale + open questions are in `ideas/thunderbird.md`.
|
|
222
|
+
|
|
223
|
+
## Discovery
|
|
224
|
+
|
|
225
|
+
{Pikuri::Thunderbird::Profile} never guesses: it resolves one profile
|
|
226
|
+
unambiguously (parsing `profiles.ini`), raises with a fix if two install
|
|
227
|
+
roots both hold a profile (a stale apt profile left after an apt→snap
|
|
228
|
+
migration), or returns `nil` (tools simply not registered) when Thunderbird
|
|
229
|
+
isn't installed. An explicit `profile_dir:` override bypasses discovery.
|
|
230
|
+
Linux snap and apt roots are supported; flatpak is out of scope.
|
|
231
|
+
|
|
232
|
+
## Storage landscape (empirically confirmed)
|
|
233
|
+
|
|
234
|
+
The reference data the design rests on, probed against a live snap install
|
|
235
|
+
(Thunderbird 152.0, Linux, 2026-07). It's here — not in the class yardocs —
|
|
236
|
+
because it spans the whole gem and is expensive to re-derive (it needs a real,
|
|
237
|
+
populated Thunderbird). The per-symbol *why* lives in the class yardocs; this
|
|
238
|
+
is the shared evidence base, and the ground truth a v2 builder will need.
|
|
239
|
+
|
|
240
|
+
Caveat: all of this is one real install (snap; Gmail + iCloud IMAP; CalDAV
|
|
241
|
+
calendars). The code is otherwise tested only against synthetic fixtures built
|
|
242
|
+
to this schema — a first run against a real profile is still the true smoke
|
|
243
|
+
test.
|
|
244
|
+
|
|
245
|
+
### Profile & install roots
|
|
246
|
+
|
|
247
|
+
- `profiles.ini` is a small INI; the active profile is the `Default=1`
|
|
248
|
+
section (`Path`, relative when `IsRelative=1`). The confirmed real file has
|
|
249
|
+
**no `[Install…]` sections** (Firefox's per-install mechanism), so the plain
|
|
250
|
+
`Default=1` scan suffices.
|
|
251
|
+
- Roots: **snap** `~/snap/thunderbird/common/.thunderbird/` (confirmed),
|
|
252
|
+
**apt** `~/.thunderbird/` (recalled). Flatpak
|
|
253
|
+
(`~/.var/app/org.mozilla.Thunderbird/…`) is out of scope.
|
|
254
|
+
- `.parentlock` at the profile root is Thunderbird's running-instance lock — a
|
|
255
|
+
cheap "is TB open?" probe.
|
|
256
|
+
|
|
257
|
+
### Mail: the Gloda index (`global-messages-db.sqlite`) — the v1 source
|
|
258
|
+
|
|
259
|
+
A complete, pre-decoded search index. Tables that matter:
|
|
260
|
+
|
|
261
|
+
- **`messagesText`** — an FTS3 virtual table over `body`, `subject`,
|
|
262
|
+
`attachmentNames`, `author`, `recipients` (already MIME-decoded,
|
|
263
|
+
HTML-stripped, attachment *names* extracted). Its backing store is the plain
|
|
264
|
+
table **`messagesText_content(docid, c0body, c1subject, c2attachmentNames,
|
|
265
|
+
c3author, c4recipients)`**.
|
|
266
|
+
- **`messages`** — `id` (= FTS `docid`), `folderID`, `messageKey`,
|
|
267
|
+
`conversationID`, `date` (**PRTime = µs since epoch**, ÷1e6 for Unix
|
|
268
|
+
seconds, second granularity), `headerMessageID` (the RFC822 `Message-ID`),
|
|
269
|
+
**`deleted`**, `jsonAttributes`, `notability`.
|
|
270
|
+
- **`folderLocations`** — `id`, `folderURI`, `dirtyStatus`, `name`,
|
|
271
|
+
**`indexingPriority`**: `-1` = never indexed (Gloda's sentinel), else `20`
|
|
272
|
+
(normal) / `50` (INBOX) / `60` (Sent). Resolves a hit's `folderID` → folder.
|
|
273
|
+
- **`contacts` / `identities` / `contactAttributes`** — the email↔name
|
|
274
|
+
correspondence graph. `contacts` is **dead for resolution** — `frecency` /
|
|
275
|
+
`popularity` are 0 for every row and `name` is unreliable (often a bare
|
|
276
|
+
address or junk), so {Pikuri::Thunderbird::Gloda::Contacts#resolve} reads the
|
|
277
|
+
*decoded* `c3author` / `c4recipients` columns instead (name + address
|
|
278
|
+
together, with a natural per-message frequency). `identities(kind, value)` —
|
|
279
|
+
where `kind='email'`, `value` the bare address — *is* used, by
|
|
280
|
+
{Gloda::Contacts#domains_seen} for the recipient-novelty warn.
|
|
281
|
+
- **`attributeDefinitions` / `messageAttributes`** — extensible per-message
|
|
282
|
+
attributes; ids seen: 57 `tag`, 58 `star`, 59 `read`, 60 `repliedTo`,
|
|
283
|
+
61 `forwarded`. (Relevant to churn — see the change counter below.)
|
|
284
|
+
- Ignore: `imConversations*`, `ext_mimeTypes`.
|
|
285
|
+
|
|
286
|
+
**The mozporter sharp edge.** Both FTS tables use Mozilla's custom `mozporter`
|
|
287
|
+
tokenizer, so a stock SQLite (CLI *or* the `sqlite3` gem) cannot `… MATCH …`
|
|
288
|
+
against them — it errors *"unknown tokenizer: mozporter"*. Worse, the gem's
|
|
289
|
+
precompiled SQLite (3.53.2) is built **FTS5-only**, so it can't even *open*
|
|
290
|
+
Gloda's fts3 `messagesText` (`no such module: fts3`). Registering a mozporter
|
|
291
|
+
tokenizer `.so` was tested and is a dead end on the default stack (it needs a
|
|
292
|
+
bespoke FTS3-compiled SQLite build, not just a loadable tokenizer). Escape:
|
|
293
|
+
read the plain `messagesText_content` columns directly (openable fine) and
|
|
294
|
+
build our own FTS5 over them — which is exactly what {Pikuri::Thunderbird::Gloda}
|
|
295
|
+
does. (`c0body LIKE '%invoice%'` scanned cleanly, 234 hits, on the real box.)
|
|
296
|
+
|
|
297
|
+
**Freshness/consistency: the change counter.** `journal_mode=delete` (classic
|
|
298
|
+
rollback journal, no WAL), `user_version=30` (a drift anchor). The file change
|
|
299
|
+
counter (header bytes 24–27) is authoritative per commit here, bumped only on
|
|
300
|
+
unlock-after-write (a read never moves it). Measured: it bumps on *any* Gloda
|
|
301
|
+
write — **including read/star/tag flips during triage** (marking one message
|
|
302
|
+
unread advanced it +1), not only new-mail arrivals — because Gloda indexes
|
|
303
|
+
those as attributes. Consequence for the FTS5 rebuild: it re-fires during
|
|
304
|
+
ordinary triage, not just after a poll (a ~1 s papercut; a future optimization
|
|
305
|
+
gates the rebuild on a content triple `(max(id), count(*), sum(deleted))`
|
|
306
|
+
instead of the raw counter). The `-journal` is **transient, not persistently
|
|
307
|
+
retained** (probed: present in 1 of 3 idle samples, exactly while the counter
|
|
308
|
+
was mid-bump), so its presence is a valid "write in flight → retry" signal for
|
|
309
|
+
the snapshot bracket.
|
|
310
|
+
|
|
311
|
+
**Gmail label fan-out (why dedup is mandatory).** Gmail exposes each label as
|
|
312
|
+
an IMAP folder, so a message with N labels yields N Gloda rows.
|
|
313
|
+
**`[Gmail]/All Mail` is indexed** (`indexingPriority` 20, **5974 messages** —
|
|
314
|
+
the largest folder, vs. 8239 distinct Message-IDs / 9654 total rows → ~1415
|
|
315
|
+
duplicate rows). Search dedups by `headerMessageID` before the top-N cap. The
|
|
316
|
+
7 `-1` never-indexed folders are exactly Trash/Spam/Junk/Unsent across the
|
|
317
|
+
accounts — so the "skip Spam" leg-2 attenuation is free on the Gloda path —
|
|
318
|
+
**except `[Gmail]/Trash` is indexed at prio 20** (5 msgs), so that skip is
|
|
319
|
+
nearly, not perfectly, complete. (Also seen: 1571 rows with a null/blank
|
|
320
|
+
`folderID` — irrelevant to Gloda-only reads, which take the body from
|
|
321
|
+
`c0body`.)
|
|
322
|
+
|
|
323
|
+
### Mail: mbox on disk (the unbuilt fallback's substrate)
|
|
324
|
+
|
|
325
|
+
Not used by v1 (Gloda serves it fully); recorded for the deferred family-B
|
|
326
|
+
fallback. Mail is **mbox** (extensionless files, `file(1)` "Mailbox text"),
|
|
327
|
+
under `ImapMail/<server>/` (IMAP, subfolders nest via `<Folder>.sbd/`) and
|
|
328
|
+
`Mail/` (Local Folders, Feeds); each mbox is shadowed by a `<Folder>.msf`
|
|
329
|
+
**Mork** summary (ignorable — Gloda has the decoded text + folder map). A
|
|
330
|
+
correct mbox reader would need: CRLF line endings; the dash-prefixed
|
|
331
|
+
`From - <ctime>` separator (not classic `From <addr>`); server-defined folder
|
|
332
|
+
names (Gmail INBOX is uppercase — never hardcode); and honoring the
|
|
333
|
+
`X-Mozilla-Status`/`Status2` **deleted-but-not-compacted** bit. `messageKey`
|
|
334
|
+
is *not* a universal seek offset — IMAP UID for `ImapMail/`, byte offset only
|
|
335
|
+
for local `Mail/` — which is why a Gloda→mbox hybrid read was rejected.
|
|
336
|
+
|
|
337
|
+
### Calendar stores (`calendar-data/`)
|
|
338
|
+
|
|
339
|
+
Three sibling DBs: **`cache.sqlite`** (cached network/CalDAV calendars),
|
|
340
|
+
**`local.sqlite`** (`storage`-type local calendars), `deleted.sqlite`
|
|
341
|
+
(`cal_deleted_items` sync tombstones). **The data gate:** a CalDAV calendar
|
|
342
|
+
keeps events **in memory only** unless "Offline Support" (`cache.enabled` in
|
|
343
|
+
`prefs.js`) is on — so all three can be empty while the UI shows full
|
|
344
|
+
calendars. Enabling it populated `cache.sqlite` to **62 MB / 5414 events** and
|
|
345
|
+
it mirrors `local.sqlite`'s schema exactly (`cal_calendar_schema_version=23`;
|
|
346
|
+
`user_version` is 0 — gate on the table, not the pragma). This is why the
|
|
347
|
+
calendar tools register regardless and self-explain the empty state.
|
|
348
|
+
|
|
349
|
+
`cal_events` is normalized columns, not opaque ICS:
|
|
350
|
+
|
|
351
|
+
- `cal_id` (the calendar's registry UUID → `prefs.js`
|
|
352
|
+
`calendar.registry.<uuid>.{name,type,uri,cache.enabled}` for human name +
|
|
353
|
+
`storage`/`caldav` type), `id` (event UID), `title`.
|
|
354
|
+
- `event_start`/`event_end` — **PRTime µs of the UTC instant** (confirmed on
|
|
355
|
+
5414 events: sane 2010→2031 histogram; times read as business hours only
|
|
356
|
+
under UTC→local; user-confirmed against the live UI — a 10:00 meeting reads
|
|
357
|
+
10:00 local, not 07:00 UTC). Displayed via the `_tz` column.
|
|
358
|
+
- `event_start_tz`/`event_end_tz` — a tzid string, and the two **can differ**
|
|
359
|
+
(cross-zone events). Valid forms, all meaningful: Olson names
|
|
360
|
+
(`Europe/Helsinki` dominant), literal **`floating`** (wall-clock; every
|
|
361
|
+
all-day event uses it), **`UTC`**, rare fixed-offset **`GMT+0300`**. Default
|
|
362
|
+
defensively even though no null was observed.
|
|
363
|
+
- `flags` = calStorageCalendar's `CAL_ITEM_FLAG` bitfield (verify against
|
|
364
|
+
comm-central at v2 impl): 2=attendees, 4=properties, **8=all-day**,
|
|
365
|
+
16=recurrence, 32=exceptions, 64=attachments, 128=relations, 256=alarms,
|
|
366
|
+
512=recurrence-id-all-day. **All-day = bit 8** — airtight (a clean diagonal:
|
|
367
|
+
126 rows, `flags&8` ⟺ midnight-aligned start ⟺ `tz=floating`, zero
|
|
368
|
+
off-diagonal). All-day DTEND is iCal **exclusive** (1-day event spans exactly
|
|
369
|
+
1 day), so render `[start, end − 1 day]` and don't zone-convert.
|
|
370
|
+
- `ical_status`, `priority`, `privacy`, recurrence fields. **`recurrence_id`**
|
|
371
|
+
is the fold key: a recurring series is stored as a *master* row
|
|
372
|
+
(`recurrence_id` NULL, `flags & 16` set, RRULE in `cal_recurrence`) **plus one
|
|
373
|
+
row per modified occurrence** ("exception") — same event UID, distinct
|
|
374
|
+
`recurrence_id` (the occurrence's PRTime), and *no* `flags & 16` bit. So a
|
|
375
|
+
per-row recurrence filter on `flags & 16 = 0` leaks every occurrence under
|
|
376
|
+
"single" and floods plain search with the same event; `Calendar#search` folds
|
|
377
|
+
rows by UID (`#fold_series`) and classifies the *series* recurring if any row
|
|
378
|
+
is a master or an occurrence.
|
|
379
|
+
|
|
380
|
+
Companions: `cal_todos` (VTODO, same shape); `cal_properties(item_id, key,
|
|
381
|
+
value)` (DESCRIPTION, LOCATION, …); `cal_recurrence`/`cal_attendees`/
|
|
382
|
+
`cal_alarms`/`cal_relations`/`cal_attachments` (iCal-shaped `icalString`
|
|
383
|
+
fragments — parse with the `icalendar` gem only if wanted).
|
|
384
|
+
|
|
385
|
+
### SQLite locking
|
|
386
|
+
|
|
387
|
+
Thunderbird opens its DBs with `locking_mode=EXCLUSIVE`, so a plain
|
|
388
|
+
`-readonly` open fails *"database is locked (5)"* while it runs — **even for
|
|
389
|
+
WAL stores** (the calendar DBs are WAL, and WAL readers are still shut out).
|
|
390
|
+
`immutable=1` is the universal lock-bypass; for a WAL store it reads
|
|
391
|
+
as-of-last-checkpoint (Thunderbird checkpoints aggressively — right after the
|
|
392
|
+
62 MB sync the `-wal` held ~1 MB and `immutable=1` saw all 5414 events). This
|
|
393
|
+
is why {Pikuri::Thunderbird::DatabaseSnapshot} works against a *copy*, not the live
|
|
394
|
+
file. (WAL caveat for calendar freshness: a WAL DB's change counter bumps only
|
|
395
|
+
on checkpoint, so calendar staleness should eventually pair the counter with
|
|
396
|
+
`-wal` size+mtime — deferred; v1 uses the counter.)
|
|
397
|
+
|
|
398
|
+
### Platform confinement (snap `home` interface)
|
|
399
|
+
|
|
400
|
+
Mapped by staging a probe `.ics` in three spots and opening each in a
|
|
401
|
+
snap-confined Thunderbird:
|
|
402
|
+
|
|
403
|
+
| Location | Readable by snap TB? |
|
|
404
|
+
|---|---|
|
|
405
|
+
| snap-owned area (`~/snap/thunderbird/common/…`) | **yes** |
|
|
406
|
+
| hidden home dir (`~/.cache/…`) | **no** |
|
|
407
|
+
| non-hidden home (`~/…`) | **yes** |
|
|
408
|
+
|
|
409
|
+
The recalled dotfile restriction is real: the `home` interface grants
|
|
410
|
+
non-hidden `$HOME` files, not dotfiles. **Sharp UX edge:** a blocked path does
|
|
411
|
+
*not* error — the `.ics` import wizard opens and reports "no events available"
|
|
412
|
+
(it read an empty file) — so staging correctness can't lean on error
|
|
413
|
+
detection. Consequence: the v1 snapshot lives in hidden `~/.cache` (read by
|
|
414
|
+
*pikuri*, unconfined — fine), but a v2 `.ics` hand-off must stage where the
|
|
415
|
+
confined TB reads (snap-owned area, or a non-hidden `$HOME` dir); the two dirs
|
|
416
|
+
can't be shared on snap.
|
|
417
|
+
|
|
418
|
+
### Hand-off mechanics — for the v2 egress tools
|
|
419
|
+
|
|
420
|
+
- **Compose → `mailto:` URI (built).** `thunderbird 'mailto:to@…?cc=…&bcc=…&subject=…&body=First%0ASecond'`
|
|
421
|
+
opens a pre-filled compose window, percent-decoding cleanly (`%0A`→newline,
|
|
422
|
+
`%20`→space), populating subject + cc + **bcc**, reusing the running
|
|
423
|
+
instance. Chosen over `-compose`, which **cannot do multiline** (`body='A%0AB'`
|
|
424
|
+
rendered the literal `A%0AB`). Because the transport is a URI, every field
|
|
425
|
+
value must be percent-encoded so an attacker-authored body can't smuggle a
|
|
426
|
+
hidden `&bcc=` — see {Pikuri::Thunderbird::MailtoUri}. pikuri launches it
|
|
427
|
+
detached (`setsid --fork`) so the agent's exit-sweep can't SIGTERM the user's
|
|
428
|
+
Thunderbird, and so a not-yet-running Thunderbird doesn't block the tool on
|
|
429
|
+
the GUI.
|
|
430
|
+
- **Create event → `.ics` import (built).** `thunderbird <path.ics>` opens TB's
|
|
431
|
+
"Import Calendar file" wizard as a new tab (running-instance reuse): a 4-step
|
|
432
|
+
flow ending in **Confirm**, with a **target-calendar picker** at step 3.
|
|
433
|
+
**No silent import** — Cancel leaves nothing behind. So TB's UI owns the
|
|
434
|
+
commit *and* the human picks local-vs-CalDAV (i.e. no-egress-vs-egress). The
|
|
435
|
+
`.ics` is built by {Pikuri::Thunderbird::IcsEvent} (RFC-5545 field escaping is
|
|
436
|
+
the injection defense), staged in {Pikuri::Thunderbird::Profile#outbox_dir}
|
|
437
|
+
(confined-readable, **not** the hidden `~/.cache` snapshot dir — see § "Platform
|
|
438
|
+
confinement"), and handed off by the shared {Pikuri::Thunderbird::Launcher}
|
|
439
|
+
(the same detached `setsid --fork` launch compose uses).
|
|
440
|
+
|
|
441
|
+
Both are the human-gated hand-off shape the no-send seam requires: pikuri
|
|
442
|
+
opens a window, the human reviews and commits.
|
|
443
|
+
|
|
444
|
+
### Files never touched
|
|
445
|
+
|
|
446
|
+
The guarantee is an **allowlist, not a blocklist**: the backends open only
|
|
447
|
+
`global-messages-db.sqlite` and `calendar-data/*.sqlite` (plus their snapshot
|
|
448
|
+
copies), and discovery reads `profiles.ini`. Nothing else in the profile
|
|
449
|
+
directory is ever opened — so no enumeration of what-to-avoid is load-bearing,
|
|
450
|
+
and a wrong name below changes nothing.
|
|
451
|
+
|
|
452
|
+
For orientation, the profile dir (the same dir as the Gloda file) also holds
|
|
453
|
+
the credential/secret stores pikuri stays clear of: the NSS key/cert databases
|
|
454
|
+
`key4.db` + `cert9.db`, saved passwords `logins.json` + `logins-backup.json`,
|
|
455
|
+
and Thunderbird's OpenPGP key material (including
|
|
456
|
+
`encrypted-openpgp-passphrase.txt`). These names are **recalled from the
|
|
457
|
+
Firefox/Thunderbird profile layout, not probe-confirmed and not exhaustive**
|
|
458
|
+
(verify against a real profile if it ever matters — but it doesn't, given the
|
|
459
|
+
allowlist). Browser/telemetry cruft (`places.sqlite`, `cookies.sqlite`,
|
|
460
|
+
`favicons.sqlite`, `storage/`, `datareporting/`, …) is likewise untouched.
|
data/README.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# pikuri-thunderbird
|
|
2
|
+
|
|
3
|
+
Search and read your **local** Thunderbird mail and calendars from a pikuri
|
|
4
|
+
agent — reading Thunderbird's own on-disk profile, never touching the
|
|
5
|
+
network. Part of the [pikuri](https://codeberg.org/mvysny/pikuri) family.
|
|
6
|
+
|
|
7
|
+
The default surface is read-only and egress-free: four inbound tools
|
|
8
|
+
(`thunderbird_mail_search` / `thunderbird_mail_read` /
|
|
9
|
+
`thunderbird_calendar_search` / `thunderbird_calendar_read`). An agent wired
|
|
10
|
+
with only these has your private mailbox and untrusted message content but no
|
|
11
|
+
way to send anything out — the lethal trifecta is broken by construction.
|
|
12
|
+
|
|
13
|
+
pikuri never sends mail on its own. If you turn on the opt-in
|
|
14
|
+
`thunderbird_mail_compose` tool, the most it can do is **open a pre-filled
|
|
15
|
+
Thunderbird compose window** — you read it, and *you* click Send. It draws the
|
|
16
|
+
draft in plain text (no hidden HTML, no tracking images), refuses a body laced
|
|
17
|
+
with invisible characters, drops look-alike recipient addresses for you to
|
|
18
|
+
retype, and warns you when you're about to mail a domain you've never
|
|
19
|
+
corresponded with. See [`DESIGN.md`](DESIGN.md).
|
|
20
|
+
|
|
21
|
+
## Requirements
|
|
22
|
+
|
|
23
|
+
- Linux, with Thunderbird installed (snap or apt).
|
|
24
|
+
- For mail search: Thunderbird's global search index (Gloda) enabled — the
|
|
25
|
+
default. For calendar: events must be cached to disk (a local calendar, or
|
|
26
|
+
a network calendar with "Offline Support" turned on).
|
|
27
|
+
|
|
28
|
+
## Try it
|
|
29
|
+
|
|
30
|
+
```sh
|
|
31
|
+
./bin/pikuri-thunderbird "what did Alice say about the invoice last month?"
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
It auto-discovers your active Thunderbird profile. Point it elsewhere with
|
|
35
|
+
`--profile-dir /path/to/profile`. This demo wires the full surface, including
|
|
36
|
+
the compose hand-off — ask it to reply to a message and it opens a pre-filled
|
|
37
|
+
Thunderbird window for you to send (it never sends by itself). The demo talks
|
|
38
|
+
to a local llama.cpp server by default; put your LLM connection in
|
|
39
|
+
`~/.pikuri-examples-config.yaml` (`llm_server`, `llm_model`, `llm_api_key`).
|
|
40
|
+
|
|
41
|
+
## Wire it into your own agent
|
|
42
|
+
|
|
43
|
+
```ruby
|
|
44
|
+
require 'pikuri-thunderbird'
|
|
45
|
+
|
|
46
|
+
agent = Pikuri::Agent.new(transport:, system_prompt:) do |c|
|
|
47
|
+
c.add_extension Pikuri::Thunderbird::Extension.new
|
|
48
|
+
end
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Mail tools register only when the Gloda index is present; calendar tools
|
|
52
|
+
register whenever a profile is found (and guide you to enable Offline Support
|
|
53
|
+
if nothing is cached yet). To add an outbound hand-off, pass
|
|
54
|
+
`Extension.new(allow_compose_mail: true)` (draft a message) or
|
|
55
|
+
`allow_create_calendar_event: true` (draft a calendar event) — each stays off
|
|
56
|
+
unless you ask for it, and your `system_prompt` should then tell the model it
|
|
57
|
+
may draft for you to send/import (an inbound-only prompt that forbids sending
|
|
58
|
+
will keep the tools unused).
|
|
59
|
+
|
|
60
|
+
## How it works
|
|
61
|
+
|
|
62
|
+
pikuri copies a consistent snapshot of Thunderbird's SQLite databases (so it
|
|
63
|
+
never fights Thunderbird for the lock), builds its own fast full-text index
|
|
64
|
+
over the already-decoded mail, and serves ranked search + full reads with no
|
|
65
|
+
MIME parsing. Design and threat model: [`DESIGN.md`](DESIGN.md). In-flight
|
|
66
|
+
design notes: `ideas/thunderbird.md` in the repo.
|