@substrat-run/kernel 0.133.0 → 0.134.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/attachment-extractor.d.ts +180 -0
- package/dist/attachment-extractor.d.ts.map +1 -0
- package/dist/attachment-extractor.js +230 -0
- package/dist/attachment-extractor.js.map +1 -0
- package/dist/attachment-text.d.ts +283 -0
- package/dist/attachment-text.d.ts.map +1 -0
- package/dist/attachment-text.js +497 -0
- package/dist/attachment-text.js.map +1 -0
- package/dist/idempotency.d.ts +4 -3
- package/dist/idempotency.d.ts.map +1 -1
- package/dist/idempotency.js +15 -7
- package/dist/idempotency.js.map +1 -1
- package/dist/index.d.ts +8 -4
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +4 -2
- package/dist/index.js.map +1 -1
- package/dist/job-run.d.ts.map +1 -1
- package/dist/job-run.js +3 -2
- package/dist/job-run.js.map +1 -1
- package/dist/list-index.d.ts +2 -1
- package/dist/list-index.d.ts.map +1 -1
- package/dist/list-index.js +9 -0
- package/dist/list-index.js.map +1 -1
- package/dist/platform-sweep.d.ts +9 -21
- package/dist/platform-sweep.d.ts.map +1 -1
- package/dist/platform-sweep.js +30 -1
- package/dist/platform-sweep.js.map +1 -1
- package/dist/scope-host.d.ts +89 -1
- package/dist/scope-host.d.ts.map +1 -1
- package/dist/scope-host.js +75 -1
- package/dist/scope-host.js.map +1 -1
- package/dist/search-index.d.ts +7 -0
- package/dist/search-index.d.ts.map +1 -1
- package/dist/search-index.js +7 -0
- package/dist/search-index.js.map +1 -1
- package/dist/subject-redaction.d.ts +151 -6
- package/dist/subject-redaction.d.ts.map +1 -1
- package/dist/subject-redaction.js +212 -8
- package/dist/subject-redaction.js.map +1 -1
- package/package.json +2 -2
|
@@ -0,0 +1,283 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Attachment content search (#1575): the text an attachment's bytes carry, extracted off
|
|
3
|
+
* the upload by a job and indexed in the scope, keyed by attachment id.
|
|
4
|
+
*
|
|
5
|
+
* `ctx.search` answers over an entity's declared columns. An attachment had nothing but
|
|
6
|
+
* its filename there, because nothing ever read its bytes. This is the reading, and the
|
|
7
|
+
* place the result lands. It is kernel-owned for the reasons the entity index is (see
|
|
8
|
+
* `search-index.ts`): no module may write `_substrat_attachments`'s neighbourhood, a
|
|
9
|
+
* kernel-maintained index cannot be desynchronised by a writer that forgets it, and the
|
|
10
|
+
* permission gate is already the kernel's.
|
|
11
|
+
*
|
|
12
|
+
* ## The flow
|
|
13
|
+
*
|
|
14
|
+
* 1. **Upload** writes the attachment row, a `pending` text row and a job run
|
|
15
|
+
* (`ATTACHMENT_TEXT_MODULE` / `ATTACHMENT_TEXT_JOB`, instance = the attachment id), all
|
|
16
|
+
* in the upload's own transaction. Nothing is extracted on the upload path, so an
|
|
17
|
+
* extraction failure cannot fail an upload: the bytes and the record have landed
|
|
18
|
+
* whatever happens next.
|
|
19
|
+
* 2. **The job driver** (`runDueJobs`, which the scope sweeper calls when a deployment
|
|
20
|
+
* opts into `runJobs`) runs `attachmentTextJob`: read the record, read the bytes, run
|
|
21
|
+
* the host's extractor for its type (K-43), write the outcome. Each adapter supplies
|
|
22
|
+
* the handler itself for this key, so no deployment has to register it.
|
|
23
|
+
* 3. **Search** is `ScopeAttachments.search`, authorized before it matches (below).
|
|
24
|
+
*
|
|
25
|
+
* ## Reading the bytes needs no grant, and that is deliberate
|
|
26
|
+
*
|
|
27
|
+
* Extraction is a derivation the kernel owns, like the entity index's triggers reading a
|
|
28
|
+
* module's rows. Reading through `getSystemAttachments` instead would need
|
|
29
|
+
* `system:<module>` to hold every target's read key, and a CP-less vertical has no way to
|
|
30
|
+
* seat that grant short of declaring a schedule it does not have. The gate that decides
|
|
31
|
+
* what a PERSON learns is at search time, and it is the same check `list` and `open` make.
|
|
32
|
+
*
|
|
33
|
+
* ## The search gate, and what it refuses to leak
|
|
34
|
+
*
|
|
35
|
+
* A search AUTHORIZES FIRST, from the scope and the caller alone, and only then matches
|
|
36
|
+
* (`searchAttachments`). The owners the caller may read — the check `open` makes, the
|
|
37
|
+
* target's `readPermission` on the owning entity — are decided before the term is looked
|
|
38
|
+
* at, and handed to the query, so `ORDER BY … LIMIT` runs over readable rows only. A match
|
|
39
|
+
* the caller cannot open neither appears nor takes a slot, however many there are: the
|
|
40
|
+
* page a caller gets is the page they would get if those attachments did not exist. And
|
|
41
|
+
* because nothing is examined and then dropped, there is no scan bound whose cutoff — or
|
|
42
|
+
* whose running time — depends on hidden rows. Three more choices follow from the same
|
|
43
|
+
* rule, each because the alternative would leak.
|
|
44
|
+
*
|
|
45
|
+
* - **No count and no "more" marker.** Either would count matches the caller cannot see.
|
|
46
|
+
* - **No score, and newest-first order instead of relevance.** FTS5's `bm25` weighs a term
|
|
47
|
+
* by how many documents in the WHOLE index contain it, including the ones the caller
|
|
48
|
+
* cannot open. Exposing the score, or ordering a multi-term result by it, would let a
|
|
49
|
+
* caller infer how often a term occurs in attachments they may not read. Ordering by
|
|
50
|
+
* attachment id (a ULID, so upload time) depends only on rows the caller can see.
|
|
51
|
+
* - **No snippet.** A snippet is text from the attachment, and producing one for a denied
|
|
52
|
+
* row would be the leak itself.
|
|
53
|
+
*
|
|
54
|
+
* The bound that remains is on authorization, not on matches: a caller without scope-level
|
|
55
|
+
* read on a target type has its owners checked one by one, at most
|
|
56
|
+
* `ATTACHMENT_SEARCH_OWNER_MAX` of them, and past that the search is refused with a stable
|
|
57
|
+
* reason (`ATTACHMENT_SEARCH_TOO_MANY_OWNERS`). Whether it refuses depends on the scope's
|
|
58
|
+
* owner count and the caller's rights — the same answer for every term. What still scales
|
|
59
|
+
* with every match is the FTS index's own read of its posting lists, which is the same work
|
|
60
|
+
* whoever asks.
|
|
61
|
+
*
|
|
62
|
+
* ## Where the text lives, and where it does not go
|
|
63
|
+
*
|
|
64
|
+
* Both tables carry the `_substrat_search_` prefix, which is what keeps them out of every
|
|
65
|
+
* scope dump (`isSearchIndexTable`). That is the point, not a convenience: a dump carries
|
|
66
|
+
* an attachment's METADATA row and never its bytes, so the text extracted from those bytes
|
|
67
|
+
* stays where the bytes are. A restore or a preview carry that brings an attachment row
|
|
68
|
+
* back re-queues its extraction (`reconcileAttachmentText`), which re-reads the bytes if
|
|
69
|
+
* they are still there — the same scope — and records a legible failure where they are
|
|
70
|
+
* not, as in a fork into a new scope. The lake streams the outbox only, so the text never
|
|
71
|
+
* reaches it either.
|
|
72
|
+
*
|
|
73
|
+
* Erasure: `shredSubject` does not reach attachments — they carry no subject and their
|
|
74
|
+
* bytes are not sealed per subject — so there is no shred path for the text to outlive.
|
|
75
|
+
* What does make bytes unreadable is deleting the attachment row (`remove`, or any other
|
|
76
|
+
* delete), and a trigger on `_substrat_attachments` removes the text row with it, which
|
|
77
|
+
* takes its index entries out through the index's own triggers.
|
|
78
|
+
*/
|
|
79
|
+
import { type AttachmentRecord, type Decision, type EntityRef, type ModuleId, type PermissionKey } from '@substrat-run/contracts';
|
|
80
|
+
import type { JobHandler } from './job-run.js';
|
|
81
|
+
import { type ScopedSql } from './scope-host.js';
|
|
82
|
+
import { type AttachmentExtractor, type AttachmentTextBounds, type ExtractionOutcome } from './attachment-extractor.js';
|
|
83
|
+
/** The job's module: the kernel itself, which owns the index. Parses as a `ModuleId`. */
|
|
84
|
+
export declare const ATTACHMENT_TEXT_MODULE: ModuleId;
|
|
85
|
+
/** The job's name. With the module, the key every adapter supplies a handler for. */
|
|
86
|
+
export declare const ATTACHMENT_TEXT_JOB = "attachment-text";
|
|
87
|
+
/** Is this run one of the kernel's extraction runs? */
|
|
88
|
+
export declare function isAttachmentTextRun(run: {
|
|
89
|
+
module_id: string;
|
|
90
|
+
job: string;
|
|
91
|
+
}): boolean;
|
|
92
|
+
/**
|
|
93
|
+
* Refuse a job registered under the kernel's own module id — every adapter's `registerJob`
|
|
94
|
+
* calls this first.
|
|
95
|
+
*
|
|
96
|
+
* The kernel's jobs are each host's own: it supplies their handlers at dispatch, bound to
|
|
97
|
+
* the scope it is driving, and dispatch looks there before the registry. A handler
|
|
98
|
+
* registered under the kernel's id would therefore never run, and say nothing — the one
|
|
99
|
+
* outcome a registration must not have. The whole id is reserved, not only today's job
|
|
100
|
+
* name, so a later kernel job cannot be shadowed by a registration that predates it.
|
|
101
|
+
*/
|
|
102
|
+
export declare function assertJobRegistrable(moduleId: string, name: string): void;
|
|
103
|
+
/**
|
|
104
|
+
* The text rows and their index, as both adapters build them.
|
|
105
|
+
*
|
|
106
|
+
* Shared for the reason `JOB_RUN_DDL` is: `lint:spine-ddl` compares what each adapter's
|
|
107
|
+
* `KERNEL_DDL` executes, and one definition keeps the two the same shape.
|
|
108
|
+
*
|
|
109
|
+
* **The double underscore is a reservation.** A module's derived index is named
|
|
110
|
+
* `_substrat_search_<slug(module)>_<slug(entity)>`, and `slug` can never produce a leading
|
|
111
|
+
* underscore, so no module's index can ever collide with these two names.
|
|
112
|
+
*
|
|
113
|
+
* **External content.** The index stores terms and points at the text row's `rid`; the
|
|
114
|
+
* text is stored once, in the row. The three triggers keep the index in step with the
|
|
115
|
+
* row however it is written, and the fourth removes the row when its attachment goes.
|
|
116
|
+
* Interpolated into each adapter's `KERNEL_DDL` after `_substrat_attachments`, which the
|
|
117
|
+
* fourth trigger names.
|
|
118
|
+
*
|
|
119
|
+
* **Only a body reaches the index.** Most rows never carry one — every upload starts
|
|
120
|
+
* `pending`, and an image or a PDF stays bodiless — so the index triggers are guarded on
|
|
121
|
+
* it: a NULL body would index no term and still cost the index's own bookkeeping writes,
|
|
122
|
+
* inside the upload's transaction. An update that leaves the body as it was (a replayed
|
|
123
|
+
* extraction) re-tokenizes nothing.
|
|
124
|
+
*/
|
|
125
|
+
export declare const ATTACHMENT_TEXT_DDL = "\n CREATE TABLE IF NOT EXISTS _substrat_search__attachment_text (\n -- The rowid alias the index points at. Never exposed: the attachment id is the key.\n rid INTEGER PRIMARY KEY,\n attachment_id TEXT NOT NULL UNIQUE,\n -- 'pending' | 'indexed' | 'empty' | 'unsupported' | 'failed'. Only 'indexed' has a body.\n status TEXT NOT NULL,\n -- Which extractor ran ('text', 'html', 'docx', 'xlsx', 'pptx'). NULL while pending,\n -- and for a type no extractor reads.\n extractor TEXT,\n -- The extracted text, normalized and capped. NULL unless indexed.\n body TEXT,\n -- UTF-8 bytes of body. NULL unless indexed.\n body_bytes INTEGER,\n -- 1 when body was cut at the per-attachment bound.\n truncated INTEGER NOT NULL DEFAULT 0,\n -- Why there is no body: the unsupported type, the damaged file, the bound exceeded.\n -- Never quotes the file.\n detail TEXT,\n updated_at TEXT NOT NULL\n );\n CREATE VIRTUAL TABLE IF NOT EXISTS _substrat_search__attachments USING fts5(\n body, content='_substrat_search__attachment_text', content_rowid='rid', tokenize='unicode61'\n );\n CREATE TRIGGER IF NOT EXISTS _substrat_search__attachment_text_ai\n AFTER INSERT ON _substrat_search__attachment_text WHEN new.body IS NOT NULL BEGIN\n INSERT INTO _substrat_search__attachments(rowid, body) VALUES (new.rid, new.body);\n END;\n CREATE TRIGGER IF NOT EXISTS _substrat_search__attachment_text_ad\n AFTER DELETE ON _substrat_search__attachment_text WHEN old.body IS NOT NULL BEGIN\n INSERT INTO _substrat_search__attachments(_substrat_search__attachments, rowid, body)\n VALUES ('delete', old.rid, old.body);\n END;\n CREATE TRIGGER IF NOT EXISTS _substrat_search__attachment_text_au\n AFTER UPDATE ON _substrat_search__attachment_text WHEN old.body IS NOT new.body BEGIN\n INSERT INTO _substrat_search__attachments(_substrat_search__attachments, rowid, body)\n SELECT 'delete', old.rid, old.body WHERE old.body IS NOT NULL;\n INSERT INTO _substrat_search__attachments(rowid, body)\n SELECT new.rid, new.body WHERE new.body IS NOT NULL;\n END;\n CREATE TRIGGER IF NOT EXISTS _substrat_attachments_text_ad\n AFTER DELETE ON _substrat_attachments BEGIN\n DELETE FROM _substrat_search__attachment_text WHERE attachment_id = old.id;\n END;\n";
|
|
126
|
+
/** Where an attachment's text is. */
|
|
127
|
+
export type AttachmentTextStatus = 'pending' | 'indexed' | 'empty' | 'unsupported' | 'failed';
|
|
128
|
+
/** An attachment's extraction state, as `readAttachmentText` reports it. */
|
|
129
|
+
export interface AttachmentTextState {
|
|
130
|
+
attachmentId: string;
|
|
131
|
+
status: AttachmentTextStatus;
|
|
132
|
+
extractor: string | null;
|
|
133
|
+
/** UTF-8 bytes indexed; null unless `indexed`. */
|
|
134
|
+
bytes: number | null;
|
|
135
|
+
truncated: boolean;
|
|
136
|
+
/** Why there is no text — or, for a `failed` derived from the run, the run's last error. */
|
|
137
|
+
detail: string | null;
|
|
138
|
+
updatedAt: string;
|
|
139
|
+
}
|
|
140
|
+
/**
|
|
141
|
+
* Queue one attachment's extraction: a `pending` row and a coalesced run, inside the
|
|
142
|
+
* caller's transaction. An existing text row is left as it is: re-extracting an indexed
|
|
143
|
+
* attachment keeps its current text searchable until the new outcome replaces it.
|
|
144
|
+
*/
|
|
145
|
+
export declare function enqueueAttachmentText(sql: ScopedSql, attachmentId: string, runId: string, at: string): void;
|
|
146
|
+
/**
|
|
147
|
+
* Write one extraction outcome — only while the attachment still exists.
|
|
148
|
+
*
|
|
149
|
+
* A run reads the bytes outside any lock, so the attachment can be removed while it
|
|
150
|
+
* extracts. The remove's trigger has already taken the text row away; an unguarded upsert
|
|
151
|
+
* would put the text of a deleted attachment straight back. So the existence check and
|
|
152
|
+
* the write are one statement, and the return says which happened.
|
|
153
|
+
*
|
|
154
|
+
* Idempotent: the same outcome written twice leaves one row and one set of index entries,
|
|
155
|
+
* because an upsert on a present row is an UPDATE, whose trigger replaces the old terms
|
|
156
|
+
* with the new ones (and touches nothing when the body did not change).
|
|
157
|
+
*/
|
|
158
|
+
export declare function recordAttachmentText(sql: ScopedSql, attachmentId: string, outcome: ExtractionOutcome, at: string): boolean;
|
|
159
|
+
/**
|
|
160
|
+
* After a restore or a fork: drop text whose attachment the load did not bring back, and
|
|
161
|
+
* queue extraction for every attachment that has no text row.
|
|
162
|
+
*
|
|
163
|
+
* The text tables are not in a dump, so a load leaves them as they were: an in-place
|
|
164
|
+
* restore keeps the text of attachments that exist on both sides (an attachment id names
|
|
165
|
+
* one write-once object, so that text is still the text of those bytes), and holds rows
|
|
166
|
+
* for attachments the restore rewound away, which must go. A fork starts with none.
|
|
167
|
+
* Every attachment row without text is queued; its run re-reads the bytes, and where they
|
|
168
|
+
* are not reachable — a fork, whose objects stay under the source scope's key — the
|
|
169
|
+
* outcome is a `failed` row that says so.
|
|
170
|
+
*/
|
|
171
|
+
export declare function reconcileAttachmentText(sql: ScopedSql, mintId: () => string, at: string): {
|
|
172
|
+
removed: number;
|
|
173
|
+
queued: number;
|
|
174
|
+
};
|
|
175
|
+
/**
|
|
176
|
+
* An attachment's extraction state, or null when none was ever recorded — an attachment
|
|
177
|
+
* uploaded before extraction existed, or one a fork carried without its text.
|
|
178
|
+
*
|
|
179
|
+
* Reads the spine through `ctx.sql`, which module code may (it may never write it). Like
|
|
180
|
+
* `readTimeline`, it checks no permission: the caller does, first, as it would before
|
|
181
|
+
* `open`.
|
|
182
|
+
*
|
|
183
|
+
* A `pending` row whose latest run has FAILED is reported as `failed`, with the run's
|
|
184
|
+
* error. That run will not be retried, so "pending" would be a promise nobody keeps; the
|
|
185
|
+
* row itself is left alone so a re-queue picks it up.
|
|
186
|
+
*/
|
|
187
|
+
export declare function readAttachmentText(ctx: {
|
|
188
|
+
readonly sql: Pick<ScopedSql, 'query'>;
|
|
189
|
+
}, attachmentId: string): AttachmentTextState | null;
|
|
190
|
+
/**
|
|
191
|
+
* The most owners a search will check one by one, for a caller without scope-level read
|
|
192
|
+
* on their type. Past it the search is REFUSED (`ATTACHMENT_SEARCH_TOO_MANY_OWNERS`), never
|
|
193
|
+
* truncated: a truncated owner set would drop readable hits silently, by an order the
|
|
194
|
+
* caller cannot see.
|
|
195
|
+
*/
|
|
196
|
+
export declare const ATTACHMENT_SEARCH_OWNER_MAX = 2000;
|
|
197
|
+
/** The `forbidden` reason a refused search carries — stable, so a UI can explain it. */
|
|
198
|
+
export declare const ATTACHMENT_SEARCH_TOO_MANY_OWNERS = "attachment_search_too_many_owners";
|
|
199
|
+
/** The gate a search runs through: the declared targets, and `ctx.check` as the caller. */
|
|
200
|
+
export interface AttachmentSearchGate {
|
|
201
|
+
/** The read key `open` checks, per declared target entity type. */
|
|
202
|
+
readonly targets: ReadonlyMap<string, {
|
|
203
|
+
readonly read: PermissionKey;
|
|
204
|
+
}>;
|
|
205
|
+
/** `ctx.check`, as the surface's principal — with no entity, at the scope. */
|
|
206
|
+
check(permission: PermissionKey, entity?: EntityRef): Promise<Decision>;
|
|
207
|
+
}
|
|
208
|
+
/**
|
|
209
|
+
* The owners a narrowed caller is checked over: those of the given types that HAVE text,
|
|
210
|
+
* one more than the cap so a refusal can tell "at the cap" from "past it". Walks the
|
|
211
|
+
* `(entity_type, entity_id)` index on `_substrat_attachments`. Params: types (JSON array),
|
|
212
|
+
* limit.
|
|
213
|
+
*/
|
|
214
|
+
export declare const ATTACHMENT_SEARCH_OWNERS_SQL = "SELECT DISTINCT a.entity_type, a.entity_id\n FROM _substrat_attachments a\n JOIN _substrat_search__attachment_text t ON t.attachment_id = a.id\n WHERE t.body IS NOT NULL AND a.entity_type IN (SELECT value FROM json_each(?))\n ORDER BY a.entity_type, a.entity_id\n LIMIT ?";
|
|
215
|
+
/**
|
|
216
|
+
* The match, restricted to readable owners BEFORE the order and the limit. A wide type is
|
|
217
|
+
* one `IN` over the JSON array of types; a narrowed owner is a row-value `IN` over JSON
|
|
218
|
+
* pairs, which SQLite answers from an ephemeral index on the subquery rather than per
|
|
219
|
+
* pair — one bound JSON array, never a `?` per owner, since a Durable Object refuses the
|
|
220
|
+
* 101st parameter. Params: match, wide types (JSON array), owners (JSON array of
|
|
221
|
+
* `[entityType, entityId]`), limit.
|
|
222
|
+
*/
|
|
223
|
+
export declare const ATTACHMENT_SEARCH_SQL = "SELECT a.id, a.entity_type, a.entity_id, a.filename, a.content_type, a.size,\n a.sha256, a.visibility, a.created_by, a.created_at\n FROM _substrat_search__attachments\n JOIN _substrat_search__attachment_text t ON t.rid = _substrat_search__attachments.rowid\n JOIN _substrat_attachments a ON a.id = t.attachment_id\n WHERE _substrat_search__attachments MATCH ?\n AND (a.entity_type IN (SELECT value FROM json_each(?))\n OR (a.entity_type, a.entity_id) IN\n (SELECT json_extract(value, '$[0]'), json_extract(value, '$[1]') FROM json_each(?)))\n ORDER BY a.id DESC\n LIMIT ?";
|
|
224
|
+
/**
|
|
225
|
+
* Search extracted text: attachments the caller may open, newest first, at most `limit`.
|
|
226
|
+
*
|
|
227
|
+
* The readable owners are decided first (`readableOwners`) and handed to the query, so the
|
|
228
|
+
* `ORDER BY … LIMIT` runs over readable rows only: a match the caller cannot open neither
|
|
229
|
+
* appears nor takes a slot, however many there are or how new. Nothing is examined and then
|
|
230
|
+
* dropped, so there is no scan bound whose cutoff could depend on hidden rows.
|
|
231
|
+
*
|
|
232
|
+
* The term is judged before anything else, so a too-short term refuses at no cost. The FTS
|
|
233
|
+
* table is not aliased: its own name is the hidden column `MATCH` reads (`searchQuery`).
|
|
234
|
+
*/
|
|
235
|
+
export declare function searchAttachments(sql: Pick<ScopedSql, 'query'>, gate: AttachmentSearchGate, term: string, limit: number): Promise<AttachmentRecord[]>;
|
|
236
|
+
/** One `_substrat_attachments` row, as SELECTed. */
|
|
237
|
+
export interface AttachmentRowShape {
|
|
238
|
+
readonly id: string;
|
|
239
|
+
readonly entity_type: string;
|
|
240
|
+
readonly entity_id: string;
|
|
241
|
+
readonly filename: string;
|
|
242
|
+
readonly content_type: string;
|
|
243
|
+
readonly size: number;
|
|
244
|
+
readonly sha256: string;
|
|
245
|
+
readonly visibility: string;
|
|
246
|
+
readonly created_by: string;
|
|
247
|
+
readonly created_at: string;
|
|
248
|
+
}
|
|
249
|
+
/** The metadata fact an attachment row records — parsed, so a malformed row throws here. */
|
|
250
|
+
export declare function attachmentRecordOfRow(row: AttachmentRowShape): AttachmentRecord;
|
|
251
|
+
/**
|
|
252
|
+
* What the extraction job needs from its adapter: no permission gate on any of it, because
|
|
253
|
+
* extraction is the kernel's own derivation (this file's header says why).
|
|
254
|
+
*/
|
|
255
|
+
export interface AttachmentTextSource {
|
|
256
|
+
/** The record, or null when the attachment no longer exists. */
|
|
257
|
+
record(attachmentId: string): Promise<AttachmentRecord | null>;
|
|
258
|
+
/**
|
|
259
|
+
* The bytes, or null when the blob store does not hold them. A THROW is a transient
|
|
260
|
+
* failure (the store is unreachable) and the run retries; null is a fact about this
|
|
261
|
+
* attachment and is recorded as such.
|
|
262
|
+
*/
|
|
263
|
+
bytes(record: AttachmentRecord): Promise<Uint8Array | null>;
|
|
264
|
+
/** Write an outcome; false when the attachment was removed meanwhile. */
|
|
265
|
+
write(attachmentId: string, outcome: ExtractionOutcome): Promise<boolean>;
|
|
266
|
+
}
|
|
267
|
+
/**
|
|
268
|
+
* The extraction job: one pass per attachment, done when its outcome is written.
|
|
269
|
+
*
|
|
270
|
+
* Everything decidable without the bytes is decided first — no extractor for the type, a
|
|
271
|
+
* recorded size over the input bound — so those files are never fetched. Bytes that are
|
|
272
|
+
* gone, or that no longer match the recorded hash, are recorded `failed`: retrying cannot
|
|
273
|
+
* bring them back. Only a throw from the store is retried, by the driver's backoff; what
|
|
274
|
+
* the EXTRACTOR does, it answers for in its outcome (`runAttachmentExtractor`).
|
|
275
|
+
*
|
|
276
|
+
* The extractors are the host's, handed in (K-43): the kernel parses no format itself.
|
|
277
|
+
*
|
|
278
|
+
* No `step()`: the pass is one idempotent unit (read, extract, upsert), so a replay after
|
|
279
|
+
* a crash simply does it again, and a step memo would only park the extracted text in
|
|
280
|
+
* `_substrat_job_steps`, which a dump carries.
|
|
281
|
+
*/
|
|
282
|
+
export declare function attachmentTextJob(source: AttachmentTextSource, extractors: readonly AttachmentExtractor[], bounds?: AttachmentTextBounds): JobHandler;
|
|
283
|
+
//# sourceMappingURL=attachment-text.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"attachment-text.d.ts","sourceRoot":"","sources":["../src/attachment-text.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA6EG;AACH,OAAO,EAIL,KAAK,gBAAgB,EACrB,KAAK,QAAQ,EACb,KAAK,SAAS,EACd,KAAK,QAAQ,EACb,KAAK,aAAa,EACnB,MAAM,yBAAyB,CAAC;AACjC,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,cAAc,CAAC;AAC/C,OAAO,EAAoB,KAAK,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAEnE,OAAO,EAOL,KAAK,mBAAmB,EACxB,KAAK,oBAAoB,EACzB,KAAK,iBAAiB,EACvB,MAAM,2BAA2B,CAAC;AAEnC,yFAAyF;AACzF,eAAO,MAAM,sBAAsB,EAA6B,QAAQ,CAAC;AAEzE,qFAAqF;AACrF,eAAO,MAAM,mBAAmB,oBAAoB,CAAC;AAErD,uDAAuD;AACvD,wBAAgB,mBAAmB,CAAC,GAAG,EAAE;IAAE,SAAS,EAAE,MAAM,CAAC;IAAC,GAAG,EAAE,MAAM,CAAA;CAAE,GAAG,OAAO,CAEpF;AAED;;;;;;;;;GASG;AACH,wBAAgB,oBAAoB,CAAC,QAAQ,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG,IAAI,CAOzE;AAED;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,eAAO,MAAM,mBAAmB,6xEA4C/B,CAAC;AAEF,qCAAqC;AACrC,MAAM,MAAM,oBAAoB,GAAG,SAAS,GAAG,SAAS,GAAG,OAAO,GAAG,aAAa,GAAG,QAAQ,CAAC;AAE9F,4EAA4E;AAC5E,MAAM,WAAW,mBAAmB;IAClC,YAAY,EAAE,MAAM,CAAC;IACrB,MAAM,EAAE,oBAAoB,CAAC;IAC7B,SAAS,EAAE,MAAM,GAAG,IAAI,CAAC;IACzB,kDAAkD;IAClD,KAAK,EAAE,MAAM,GAAG,IAAI,CAAC;IACrB,SAAS,EAAE,OAAO,CAAC;IACnB,4FAA4F;IAC5F,MAAM,EAAE,MAAM,GAAG,IAAI,CAAC;IACtB,SAAS,EAAE,MAAM,CAAC;CACnB;AAwCD;;;;GAIG;AACH,wBAAgB,qBAAqB,CAAC,GAAG,EAAE,SAAS,EAAE,YAAY,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,IAAI,CAQ3G;AAED;;;;;;;;;;;GAWG;AACH,wBAAgB,oBAAoB,CAClC,GAAG,EAAE,SAAS,EACd,YAAY,EAAE,MAAM,EACpB,OAAO,EAAE,iBAAiB,EAC1B,EAAE,EAAE,MAAM,GACT,OAAO,CAwBT;AAED;;;;;;;;;;;GAWG;AACH,wBAAgB,uBAAuB,CACrC,GAAG,EAAE,SAAS,EACd,MAAM,EAAE,MAAM,MAAM,EACpB,EAAE,EAAE,MAAM,GACT;IAAE,OAAO,EAAE,MAAM,CAAC;IAAC,MAAM,EAAE,MAAM,CAAA;CAAE,CAgBrC;AAED;;;;;;;;;;;GAWG;AACH,wBAAgB,kBAAkB,CAChC,GAAG,EAAE;IAAE,QAAQ,CAAC,GAAG,EAAE,IAAI,CAAC,SAAS,EAAE,OAAO,CAAC,CAAA;CAAE,EAC/C,YAAY,EAAE,MAAM,GACnB,mBAAmB,GAAG,IAAI,CAmC5B;AAID;;;;;GAKG;AACH,eAAO,MAAM,2BAA2B,OAAQ,CAAC;AAEjD,wFAAwF;AACxF,eAAO,MAAM,iCAAiC,sCAAsC,CAAC;AAErF,2FAA2F;AAC3F,MAAM,WAAW,oBAAoB;IACnC,mEAAmE;IACnE,QAAQ,CAAC,OAAO,EAAE,WAAW,CAAC,MAAM,EAAE;QAAE,QAAQ,CAAC,IAAI,EAAE,aAAa,CAAA;KAAE,CAAC,CAAC;IACxE,8EAA8E;IAC9E,KAAK,CAAC,UAAU,EAAE,aAAa,EAAE,MAAM,CAAC,EAAE,SAAS,GAAG,OAAO,CAAC,QAAQ,CAAC,CAAC;CACzE;AA4DD;;;;;GAKG;AACH,eAAO,MAAM,4BAA4B,2RAK/B,CAAC;AAEX;;;;;;;GAOG;AACH,eAAO,MAAM,qBAAqB,smBAUxB,CAAC;AAEX;;;;;;;;;;GAUG;AACH,wBAAsB,iBAAiB,CACrC,GAAG,EAAE,IAAI,CAAC,SAAS,EAAE,OAAO,CAAC,EAC7B,IAAI,EAAE,oBAAoB,EAC1B,IAAI,EAAE,MAAM,EACZ,KAAK,EAAE,MAAM,GACZ,OAAO,CAAC,gBAAgB,EAAE,CAAC,CAY7B;AAED,oDAAoD;AACpD,MAAM,WAAW,kBAAkB;IACjC,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,YAAY,EAAE,MAAM,CAAC;IAC9B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;IACxB,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;IAC5B,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;IAC5B,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;CAC7B;AAED,4FAA4F;AAC5F,wBAAgB,qBAAqB,CAAC,GAAG,EAAE,kBAAkB,GAAG,gBAAgB,CAY/E;AAID;;;GAGG;AACH,MAAM,WAAW,oBAAoB;IACnC,gEAAgE;IAChE,MAAM,CAAC,YAAY,EAAE,MAAM,GAAG,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC,CAAC;IAC/D;;;;OAIG;IACH,KAAK,CAAC,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAAC,UAAU,GAAG,IAAI,CAAC,CAAC;IAC5D,yEAAyE;IACzE,KAAK,CAAC,YAAY,EAAE,MAAM,EAAE,OAAO,EAAE,iBAAiB,GAAG,OAAO,CAAC,OAAO,CAAC,CAAC;CAC3E;AAUD;;;;;;;;;;;;;;GAcG;AACH,wBAAgB,iBAAiB,CAC/B,MAAM,EAAE,oBAAoB,EAC5B,UAAU,EAAE,SAAS,mBAAmB,EAAE,EAC1C,MAAM,GAAE,oBAAqD,GAC5D,UAAU,CAgCZ"}
|