@happyvertical/smrt-ingestion 0.0.0-stage → 0.55.5

package/AGENTS.md

@@ -0,0 +1,33 @@
+# @happyvertical/smrt-ingestion
+
+Durable foundation for ADR 0004. `src/models.ts` owns all ingestion tables;
+`src/server.ts` owns authenticated service boundaries and lifecycle primitives.
+`src/execution.ts` owns authoritative review/apply and imports server-only principals.
+See [execution integration](agents/execution.md) for host transaction guarantees,
+catalog contracts, deployment dependencies and continuation bindings.
+Root, `/models`, and `/dto` remain browser-safe; provider/network/filesystem code
+belongs under `/server`. Generated REST/CLI/MCP model mutation is disabled.
+
+See [README.md](README.md) for host integration and recovery/deletion operation.
+See [EXTRACTION.md](EXTRACTION.md) for extraction contracts, provider isolation
+requirements and corpus validation.
+
+All records require tenant and confidential scope. Hosts create service scopes
+from authenticated sessions; source metadata is never authority. Every read and
+write rechecks the host scope predicate. Analysis input revisions, attempts,
+proposals, decisions, executions and feedback are append-only except documented
+state projections and retention redaction. Terminal actions outlive job cleanup.
+
+Only SQLite and PostgreSQL are supported. Deploy manifest migrations before
+starting services; runtime performs no application DDL. Storage planning and
+preservation use AssetRuntime/AssetStore public methods. Blob I/O and queue
+submission never run inside ingestion database transactions.
+
+Validation: `pnpm test`, `pnpm test:postgres`, `pnpm test:providers`, `pnpm typecheck`, `pnpm build`.
+The PostgreSQL command provisions a real database and fails if unavailable.
+Build includes browser import validation. The provider lane exercises local SDK
+adapters and process isolation, without claiming live remote recognition quality.
+The component/e2e/evaluation
+commands are reserved by ADR 0004 and fail explicitly until their owning children
+supply real suites.
+See ../../docs/test-matrix/3670-ingestion-foundation.md for behavior coverage.

package/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

package/EXTRACTION.md

@@ -0,0 +1,120 @@
+# Extraction adapters
+
+`@happyvertical/smrt-ingestion/server` exports `createSDKExtractionAdapter`,
+`extractAnalysis`, extraction types, and `proposeDocumentSplits`. Root and DTO
+exports remain browser-safe. No extraction result authorizes a business action.
+
+Hosts construct the SDK adapter with explicit provider/model/version identities
+(`unknown` for unavailable model/version), credential-bearing SDK options, and
+`nativeMemoryIsolation: 'host-enforced'`. This last setting is an assertion by
+trusted deployment configuration, never a source/body/client option. Deploy the
+provider child under an OS/container native-memory ceiling. Node's heap limit
+cannot bound native allocations, and PDF's renderer has no pre-render pixel
+limit. Without that host guarantee, adapter construction refuses to run. A
+configured provider must be pinned, not `auto`; OCR fallback is disabled so
+capability provenance cannot silently change. Identity objects are projected onto
+exactly `provider`, `model`, and `version`, each a required nonblank string. Extra
+host fields never enter capability or segment identities, including through
+trusted custom adapter results at the publication boundary.
+
+The adapter forks one disposable Node process per retained part. Deadline or
+cancellation kills it (and its process group on POSIX); provider stdout/stderr
+are discarded. Calls are sequential and bounded by input bytes, page count,
+image decode pixels, serialized output, vision tokens, elapsed time and Node
+heap. Native memory remains the host boundary described above. Remote compute
+already accepted by a provider may continue/bill after the local process dies.
+Do not interpret a local cancellation as remote billing cancellation.
+
+Persist `{extraction: {configurationRevision, limits}}` using `service.analyze`,
+then claim its lease. Call `extractAnalysis(service, lease, adapter, options)`
+with exactly that revision and limits. The service reloads the immutable input
+snapshot and rechecks current processing authority before each provider and
+before publication. The production child requests a parent-side authorization
+handshake before each PDF/OCR/vision/speech operation, including calls on later
+pages. Denial kills the child; already transmitted material cannot be recalled.
+Custom trusted adapters must await `request.beforeProviderCall` before every
+provider invocation. Bytes are read through the service's integrity-checked
+reader; revoked, expired, foreign-scope, forged and stale leases fail closed.
+A retry keeps the same analysis input revision; changed provider/model/options
+must use a new configuration revision and `analyze` request key. Credentials are
+never placed in persisted configuration or provider identity.
+
+Results retain the original evidence ID/hash/part/parent references. Ordered
+segments form whole-document context; extraction does not split originals into
+business records. Valid UTF-8 JSON, text and HTML remain literal source text (render HTML
+only through an application sanitizer). JSON syntax is validated without rewriting
+its source text; malformed JSON or UTF-8 fails extraction. PDF text is extracted one page at a
+time, falling back to actual rendered-page OCR for empty pages. Embedded text
+is not evidence that every visual element was understood. TIFF is actually
+decoded page-by-page with sharp to PNG; each result cites its original TIFF page.
+Animated non-TIFF images are unsupported. Ordinary image OCR and scene vision
+are separately configured. Speech uses source-level text unless actual segment
+timestamps are returned; absent speech confidence calibration is unknown.
+
+Confidence and boxes are explicitly null unless an OCR provider advertises and
+returns them. Unlimited-OCR's synthetic common confidence is always discarded,
+even if an injected capability claims otherwise. Measured OCR values retain an
+unknown calibration label; they are not routing certainty or approval. Boxes
+are in decoded image pixels, attached only to that actual page/source. No tables
+are invented: the current PDF/OCR text APIs do not supply a table contract.
+
+The persisted receipt's `maxOutputBytes`, exposed by `getAnalysisInput`, bounds
+the complete serialized analysis envelope across all parts, including provenance,
+capabilities, usage, and failure metadata. Extraction reserves room for a `limit`
+outcome and retains fitting earlier parts/segments when the aggregate is too
+large. Top-level `truncated` and `omittedEvidenceCount` identify publication
+omissions; per-result omissions retain every discarded segment's exact page or
+time location. Existing errors and omitted ranges survive trimming unchanged.
+If those diagnostic records cannot fit, the entire result is omitted rather than
+publishing an incomplete account of its known failures. If the ceiling
+cannot fit the minimum result envelope, no provider runs: `failAnalysis` records
+only a fixed safe category and terminal lifecycle state, with no provider output
+or output digest. That service-owned failure accounting is outside the provider
+payload budget; `completeAnalysis` never accepts an oversized result.
+
+Partial page results survive later failures or process timeout. Warnings and
+omitted page/source references remain inspectable. `unsupported_type` covers
+video, unconfigured media and encrypted PDF; corrupt decoder failures without a
+safe typed SDK category are `unavailable`, never raw exception strings. A host
+can repair/reconfigure and create a fresh analysis without rewriting originals.
+`proposeDocumentSplits` produces a hashed, versioned, non-executable grouping of
+observed pages; overlap, unordered groups and nonexistent pages are rejected.
+
+## Evidence and limitations
+
+`pnpm --filter @happyvertical/smrt-ingestion test:providers` runs actual local PDF
+SDK extraction, actual sharp multipage TIFF decoding, and packaged process/HTTP
+adapter lifecycle tests. The local speech server tests adapter transport and
+termination, not transcription quality. The tiny TIFF fixture proves IFD/page
+mapping, not fax OCR accuracy. `extraction.test.ts` injects deterministic SDK
+boundaries for mixed/scanned PDF, Unlimited-OCR, camera observations, message
+attachments, audio timestamps and malformed/partial output. These fixtures are
+not real remote-provider integrations and are not the ADR evaluation corpus.
+
+No live Unlimited-OCR, speech recognition or vision quality run is claimed.
+No recorded-human-audio recognition accuracy, real camera-scene quality,
+calibrated confidence, table extraction, or automatic-action acceptance is
+established by these tests. Deployment owners must run the intended provider,
+model and representative retained corpus before making those claims. The
+[behavior matrix](../../docs/test-matrix/3672-ingestion-extraction.md) records the
+release commands and authority obligations.
+
+The versioned on-disk corpus manifest records exact hashes for valid embedded,
+raster and mixed PDFs, two-page TIFF, a synthetic camera-format JPEG, a valid
+PCM WAV recording of a generated tone, email text, corrupt PDF and unsupported
+video. The tone is not human speech; the illustrated scene is not a photograph.
+Real local HTTP fixture runs execute Unlimited-OCR's installed SDK and speech
+transport against deterministic responses/timeouts. They verify adapters, not
+remote models. Unlimited-OCR 0.61.6 rejects valid encoded images smaller than
+100 bytes; ingestion reports `unavailable` without altering them. This owning
+SDK defect is tracked in [happyvertical/ocr#143](https://github.com/happyvertical/ocr/issues/143).
+
+Each capability record explicitly reports `truncation` and `usage` as `unknown`
+or `reported`. Unlimited-OCR's current SDK discards upstream `finish_reason`
+and token usage, so both are `unknown` even for nonempty output. Downstream
+rules must not interpret `status: 'complete'` or `truncated: false` as proof of
+untruncated OCR: they describe completed input processing and *detected*
+truncation only. The adapter never bypasses the SDK to recover hidden fields.
+The missing SDK metadata is tracked in [happyvertical/ocr#144](https://github.com/happyvertical/ocr/issues/144).
+AI vision retains reported truncation when supplied by its SDK. Missing provider
+cost is unknown, not zero; no cost estimate is invented.

package/LICENSE

@@ -0,0 +1,7 @@
+Copyright <2025> <Happy Vertical Corporation>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/README.md

@@ -1,3 +1,148 @@
-# Temporary Holding Version
+# @happyvertical/smrt-ingestion
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Durable receipt, evidence, analysis attempts and action-journal foundations for
+[ADR 0004](../../docs/adr/0004-application-aware-ingestion.md).
+
+The root, `/models` and `/dto` entry points are browser-safe. Import
+`IngestionService` from `/server` in trusted host code. Generated REST, CLI and
+MCP access to the persistence models is disabled; hosts expose authorized DTO
+methods through their authenticated application transport.
+
+Before starting a service, deploy the package manifest through the public s-m-r-t
+migration APIs or `smrt db:migrate`. `assertReady()` only checks existing tables.
+SQLite and PostgreSQL are supported; other engines fail construction.
+
+A host supplies a database, AssetRuntime, authenticated tenant/actor/confidential
+scope, a live authorization callback, a registered jobs target, and an idempotent
+`purgeDerived` callback for host-owned previews, retrieval indexes and caches.
+The authorization callback receives the captured ceiling and the current
+transaction executor; it must intersect current grants with that ceiling. Never
+construct the scope, retention policy or budgets from incoming document labels.
+
+`receive` accepts bounded original byte buffers, with stable source part IDs,
+parent relationships and optional upstream identity/version. Source adapters own
+streaming and upstream snapshot retrieval. Receipt identity is tenant + source +
+delivery key; changing bytes or authenticated scope conflicts. Content equality
+in a different conversation does not deduplicate an arrival. The returned item
+ID and version are safe acknowledgment data, never a storage URL.
+
+Storage intents contain a stable asset ID and a public AssetStore-planned URI
+before byte I/O. Preservation verifies existing bytes instead of replacing them.
+A receipt becomes ready only after every part is durable and its dispatch intent
+commits. A source retries the same receive call after failure or lost
+acknowledgment. Hosts schedule `repairDispatches` after restart and periodically;
+it repairs missing/failed queue submissions through SmrtJobCollection. The host
+job restores the captured scope and calls the analysis lifecycle methods. Run
+TaskRunner with the `ingestion` queue enabled.
+
+`analyze` appends immutable input/configuration revisions. `claimAnalysis` uses
+leases, monotonic fences and attempt budgets; `completeAnalysis` rejects expired
+or superseded workers. Terminal attempt output/provenance is preserved. Queue
+submission budgets also stop repeated handler failures in visible
+`needs_attention`. Reprocessing does not mint actions. `createAction`,
+`publishProposal` and `appendPlan` persist explicit intentions and exact attempt
+bindings for the later review/execution layer. They do not approve or apply
+business operations. Decision/execution/feedback models are persistence
+foundations; policy, handler expansion, review and business apply belong to the
+later epic children.
+
+`expire` revokes access and pending work before removing bytes and calling
+`purgeDerived`. Interrupted deletion is retried by `repairDeletions` and
+`sweepRetention`. Minimal receipt/action identities remain, including successful
+action result digests; all content-bearing journal payloads are redacted. Private
+cleanup locators remain in deletion tombstones so repeated sweeps can remove a
+late stale writer's orphan bytes even after the asset owner row is gone. Source
+adapters must enforce their configured delivery-age cutoff; expired replay IDs
+never become fresh work. Continue running the cleanup sweep for the retained
+storage namespace. Upstream source mutation/deletion cannot change copied
+originals; explicit privacy deletion revokes every ingestion-derived payload and
+requires host cache/index purge.
+
+See [EXTRACTION.md](EXTRACTION.md) for bounded extraction, lease-bound input
+snapshots, provider isolation requirements and the extraction corpus.
+
+Validation commands are `pnpm test`, `pnpm test:postgres`, `pnpm test:providers`, `pnpm typecheck` and
+`pnpm build` (including built browser import inspection). The PostgreSQL lane
+requires the repository's configured database service and never silently skips.
+The provider lane exercises local SDK adapters and process isolation; it does not
+claim live remote recognition quality. Component, e2e and evaluation command names are reserved by the ADR and
+explicitly unavailable until their owning children deliver the corresponding
+capability. See the [behavior matrix](../../docs/test-matrix/3670-ingestion-foundation.md).
+
+## Authoritative review and execution
+
+Configure `IngestionOptions.execution` for policy, immutable previews, authenticated
+decisions and idempotent domain execution. See [execution integration](agents/execution.md)
+for the public handler catalog, required host authorization/target locks, deployment
+migrations and jobs continuation integration. The maintained application reference
+in `reference/handlers.ts` creates draft ContentDocuments and attaches retained
+evidence through public domain APIs; it is not a published runtime adapter.
+
+## Configured source adapters
+
+The server entry exports `createSourceDeliveryHandler`, `EmailSourceAdapter`,
+`WatchFolderSourceAdapter`, and `createVendorWebhookHandler`. Construct a
+`SourceBinding` from authenticated host configuration: enabled flag, scoped
+`IngestionService`, stable source ID/version, captured authority ceiling,
+retention policy, allowed media types and receipt limits. Re-resolve this binding
+when grants/configuration change. Source payloads never supply authority.
+
+The reference POST handler authenticates before reading bytes and requires
+`Idempotency-Key`. JSON accepts exactly `capturedAt` and one of `text`,
+`structured`, or `reference: { owner: 'asset'|'message', id, version }`.
+The host's `snapshotReference` must authorize current ownership and return copied
+bytes; an absent snapshot fails rather than acknowledging a mutable locator.
+Multipart accepts one `file` plus `captureId` (equal to the idempotency key),
+`capturedAt`, `captureSource` (`camera`, `native_picker`, `audio`, `document`), and optional
+lowercase SHA-256 `sha256`. This reuses mobile's durable UUID Idempotency-Key,
+MIME-bearing file part and explicit capture metadata convention. Captures retain
+original bytes and a parent metadata evidence record. Configure both the wire
+limit `maxRequestBytes` and receipt `maxBytes`/`maxParts`; include
+`application/json` in the media allowlist for capture/email metadata. Signature
+checks cover PDF, TIFF, PNG/JPEG/WebP and WAV/Ogg/WebM/MP3/MP4 audio; signature
+acceptance promises preservation, not extraction support or codec validity.
+
+Email is explicitly opt-in per configured account. `EmailSourceAdapter` takes a
+`readMessage` callback compatible with the messages owner's
+`EmailAccount.readIntakeMessage(providerLookup, maxBytes, imapIdentity?)` snapshot.
+The provider lookup is Gmail message ID, POP3 UIDL, or IMAP RFC Message-ID; it
+is separate from the delivery dedupe key. IMAP requires `{ folder, uidValidity,
+uid }`: the method selects the mailbox and verifies UIDVALIDITY and returned UID.
+Ambiguous sender IDs resolving to another UID fail explicitly. That method connects,
+fetches the full message including actual attachment bytes, copies the bytes,
+and disconnects. Metadata-only attachments fail explicitly. It never reads an
+attachment path supplied by a provider. A host lists provider deliveries using a
+stable account-specific locator (for IMAP include folder, UIDVALIDITY and UID),
+an explicit revision, and an opaque resumable checkpoint. Same locator/revision
+with changed bytes conflicts; an explicit new revision creates a new receipt.
+Do not use sender-controlled RFC Message-ID as the sole transport identity.
+`receiveBatch` processes in order and stops at the first failure. The host owns
+durable checkpoint storage and commits its `acknowledge` callback only after the
+adapter has received accepted/duplicate. If checkpoint persistence fails, retry
+returns the original receipt. Message/thread/attachment relationships are copied;
+credentials and arbitrary headers are excluded. The ordinary `syncFrom` metadata
+sync remains separate; `hasAttachments` is never proof of byte preservation.
+
+Watch-folder intake polls only an explicitly configured existing private local
+directory. Producers should close their files, preferably publishing with atomic
+rename. Two unchanged size/inode/mtime/ctime observations separated by
+`stabilityMs` are required, and the interval runs again after claim/restart.
+A private `.ingestion-claims/<UUID>/claim.json` intent is synced before atomically
+renaming the original into that directory. Keep the directory on one filesystem,
+under the host's ownership; symlink inputs are refused. A file descriptor still
+held by an uncooperative producer can write after any stability window, so the
+producer close/publish protocol is required for immutable input. Accepted or
+duplicate receipts move the entire claim to `.ingestion-processed`; do not delete
+claims to retry. Polling recovers persisted UUIDs after process death, so receipt
+replay deduplicates ready-without-ack. `failure.json` and poll results expose safe
+categories while originals remain available. Intent directories without an
+original mean death before claim or a competing poll won; the producer original
+has not been removed. Operators may remove these empty intents after confirming
+no active importer. Keep processed originals according to the host's retention
+policy. This adapter does not watch arbitrary paths or start background loops.
+
+Vendor webhooks use a host `verify(request, originalBytes)` callback after bounded
+raw-body reading. The callback must verify signature, delivery time/replay window
+and account ownership, then return trusted binding and durable delivery identity.
+No vendor implementation or hardware driver is selected. All adapters invoke
+only receipt preservation, never domain writes, AI tools or external actions.

package/dist/__smrt-register__.d.ts

@@ -0,0 +1,2 @@
+export {};
+//# sourceMappingURL=__smrt-register__.d.ts.map

package/dist/__smrt-register__.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"__smrt-register__.d.ts","sourceRoot":"","sources":["../src/__smrt-register__.ts"],"names":[],"mappings":""}