@shipfox/api-logs 2.0.0

package/.turbo/turbo-build.log

@@ -0,0 +1,2 @@
+$ shipfox-swc
+Successfully compiled: 59 files with swc (132.09ms)

package/.turbo/turbo-type$colon$emit.log

@@ -0,0 +1 @@
+$ shipfox-tsc-emit

package/.turbo/turbo-type.log

@@ -0,0 +1 @@
+$ shipfox-tsc-check

package/CHANGELOG.md

@@ -0,0 +1,179 @@
+# @shipfox/api-logs
+
+## 2.0.0
+
+### Minor Changes
+
+- 1b0d344: Publishes the complete API runtime closure with packed-consumer-safe internal imports and records its exact package set in application releases.
+
+### Patch Changes
+
+- Updated dependencies [0cd6dd4]
+- Updated dependencies [a68458a]
+- Updated dependencies [6eba800]
+- Updated dependencies [1b0d344]
+- Updated dependencies [521e006]
+  - @shipfox/node-module@0.2.0
+  - @shipfox/node-temporal@0.2.0
+  - @shipfox/api-agent-dto@2.0.0
+  - @shipfox/api-auth-context@2.0.0
+  - @shipfox/api-logs-dto@2.0.0
+  - @shipfox/api-workflows@2.0.0
+  - @shipfox/api-workflows-dto@2.0.0
+  - @shipfox/config@1.2.1
+  - @shipfox/node-drizzle@0.2.1
+  - @shipfox/node-fastify@0.2.1
+  - @shipfox/node-opentelemetry@0.5.0
+  - @shipfox/node-outbox@0.2.1
+  - @shipfox/node-postgres@0.4.1
+
+## 0.1.2
+
+### Patch Changes
+
+- Updated dependencies [705dd43]
+  - @shipfox/node-outbox@0.2.0
+  - @shipfox/api-workflows@0.1.2
+  - @shipfox/node-module@0.1.2
+
+## 0.1.1
+
+### Patch Changes
+
+- Updated dependencies [ec75cd5]
+- Updated dependencies [6a1fb54]
+  - @shipfox/node-drizzle@0.2.0
+  - @shipfox/node-postgres@0.4.0
+  - @shipfox/api-workflows@0.1.1
+  - @shipfox/node-module@0.1.1
+  - @shipfox/node-outbox@0.1.1
+
+## 0.1.0
+
+### Minor Changes
+
+- a56748d: Adds ingestion-time agent session parsing with a stored canonical SessionView read endpoint and workflow harness lookup.
+- f92122b: Adds the logs module foundation: a stateless monolith module with its own schema, the runner-facing offset-CAS append endpoint (job-lease authenticated, idempotent, multi-instance safe), a per-job accrual budget with a cap tombstone, and an S3-compatible client targeting Garage at startup. The NDJSON v1 record contract lives in the new `@shipfox/api-logs-dto` package, and `@shipfox/node-fastify` gains a `createRawBodyPlugin({contentType, bodyLimit})` factory for byte-exact request bodies.
+- 4207772: Add the session-authenticated log read endpoint: `GET /steps/:stepId/attempts/:attempt/logs?cursor=N`. One cursor endpoint serves both the live tail and the full history of a step attempt, workspace-scoped through the stream row's denormalized `workspaceId` (a 404 covers both a missing stream and a cross-workspace step, so existence never leaks).
+  - Open or closed-but-uncompacted streams return inline NDJSON read from the hot Postgres chunks, walked by chunk `seq` so server-injected control tombstones (`capped`, `runner_lost`) interleave with runner bytes exactly as compaction concatenates them. The inline bytes are therefore byte-identical to the decompressed compacted object. Pages are bounded by `LOG_READ_INLINE_MAX_BYTES` (default 1 MiB), with a `has_more` flag and a `next_cursor` the client drains before it tails.
+  - Compacted streams (`object_key` set) return a presigned GET URL (`LOG_READ_URL_TTL_SECONDS`, default 3600) plus `total_bytes`, `expires_at`, and `truncated`, so the browser fetches the object directly and API egress is bypassed.
+  - `@shipfox/api-logs-dto` gains `readLogsQuerySchema` and the `readLogsResponseSchema` discriminated union (`inline` or `presigned`) so the backend, client, and E2E helpers share one contract. `@shipfox/api-logs` adds the `@aws-sdk/s3-request-presigner` dependency.
+
+### Patch Changes
+
+- 68e4022: Collapse the log stream `kind` into one stream per `(job, step, attempt)`. `kind` was modeled at the wrong layer: the runner already funnels every record through one spool and one offset axis, so the per-kind split (separate stream rows, identity, and ingest dispatch) was unnecessary server-side modeling. A reader now pulls every record for a step attempt and filters by the record `type` at read time, without passing a kind.
+  - The append request drops the `kind` query param and `LogStreamClosedEvent` drops `kind`. `streamKind`/`StreamKind` and the `agent_session` line validator (`parseSessionLine`) are removed.
+  - `attempt_streams` identity becomes `(job, step, attempt)`; the `kind` and now-dead `capped` columns are dropped (budget cap is signaled in-band by a `capped` tombstone and tracked in `job_accounting`). The close path injects `capped`/`runner_lost` tombstones uniformly.
+  - `LOG_MAX_SESSION_LINE_BYTES` is removed and `LOG_APPEND_BODY_LIMIT_BYTES` is right-sized to 1 MiB. The runner's flush-window comment is corrected to match (the server body limit is now 1 MiB, no longer 8 MiB).
+
+  Agent-session capture (its own record `type`(s) and runner producer) is deferred to the agent-sessions feature, which will emit those records into this same stream.
+
+- ae7a63c: Adds daily dispatched outbox row retention with bounded cleanup batches and retention indexes on module outbox tables.
+- d49ee4c: Forward agent step session entries into the logs module as opaque `agent_session` records: the runner tails the pi session file and forwards each verbatim entry over a shared log-stream sink, and the write path stores them with a configurable per-line size cap sized for inline base64 content.
+- 8cab4e7: Close a log stream so it becomes eligible for compaction. A committed `end` record declared-closes the stream inside the append transaction, and a `WORKFLOWS_JOB_TERMINATED` subscriber arms a grace-then-close Temporal workflow that force-closes any stream the runner never ended (appending a `runner_lost` tombstone and marking it truncated). Both paths route through one guarded close that writes a single `logs.stream.closed` outbox event, and a closed-stream guard drops later appends. Adds `closed_at` and three partial indexes to the stream table.
+- b4103b5: Delete expired logs from both object storage and Postgres on an hourly cron, with a configurable horizon (`LOG_RETENTION_DAYS`, default 90), enforced by our own worker rather than bucket lifecycle rules so behavior is identical across object stores. A retention sweep drains closed streams past the horizon in batches, bounded by a self-imposed time budget so a timed-out run never overlaps the next; per stream it first hard-deletes the row (chunks cascading), guarded on the observed `object_key` so a concurrent compaction publish is left intact, then deletes the whole attempt object prefix (reclaiming orphan leaves left behind by a losing compaction attempt). Failed or raced rows are skipped for the rest of the run so a poison row cannot starve the streams behind it, and a `job_accounting` row is pruned only when its job has no remaining streams and no recent activity, so a live job's budget is never reset.
+- b0a0e1a: Reap log streams that leak open after the one-shot job-terminated close. That sweep snapshots a job's open streams once, so a stream whose first append lands after it ran was never closed and leaked forever: open streams are invisible to compaction and retention (both keyed on `state = 'closed'`). A new `reapStaleOpenStreamsCron` (every 10 minutes, on the logs lifecycle queue, staggered off the retention sweep) force-closes any stream left open past the job-lease window, marking it truncated so it re-enters the compaction and retention lifecycle. Adds the `LOG_STREAM_REAP_AFTER_SECONDS` config (default 7200s; startup validates it exceeds the configured `AUTH_JOB_LEASE_TOKEN_EXPIRES_IN`, read from auth's `config`) and the `logs_attempt_streams_open_age_idx` partial index.
+- 2883ab4: Add a stream `kind` to the logs contract and harden the write path. Streams are now identified by `(job, step, attempt, kind)` with two kinds: `log_stream` (process output) and `agent_session` (verbatim, format-agnostic agent JSONL).
+  - The record envelope drops `src` to `{v, ts}`; the stdout/stderr pipe moves to `stream` on output, and control records become a flat `type` discriminator with multi-level named groups (`group_id`/`parent_group_id`, capped depth with overflow flattening).
+  - The append validator is split into an appendable union and a read union, so a forged server-only `capped`/`runner_lost` tombstone is rejected. `agent_session` lines are stored verbatim after a whole-line, fatal-UTF-8, JSON-well-formedness check within a configurable line cap; the budget cap never injects an in-band tombstone into a session, and close derives its `capped`/`truncated` flags out of band.
+  - The append request carries `kind`; the shared body limit and session line cap are configurable with a startup invariant.
+
+- 4c39cb5: Makes log retention delete object prefixes before stream rows so transient object-storage delete failures leave rows discoverable for a later sweep instead of permanently orphaning billed objects.
+- 150f378: Compact a closed log stream's hot Postgres chunks into one gzip-compressed NDJSON object in object storage, record the object key on the stream row, and delete the chunk rows. Compaction is a Temporal workflow started from the `logs.stream.closed` event (deduped per stream) on a dedicated task queue, with a reconcile cron backstop that re-drives any closed stream whose compaction never started or permanently failed. Crash-safe and idempotent: each attempt uploads to its own per-attempt object key and a single-winner publish (atomic object-key set and chunk delete, guarded by `object_key IS NULL`) records exactly one, so a slow or retried attempt can never overwrite a published object; a streamed-vs-table integrity check over chunk count, last seq, and byte total guards the irreversible chunk delete. Keeps Postgres bounded by in-flight work rather than retention.
+- 362b3eb: Scope runner log append authorization to the dispatched step attempt carried by the job lease token.
+- 3bea87f: Adds a typed `subscriberFactory` that binds each outbox event name to its payload type at construction, so subscriber handlers receive a typed `(payload, event)` and the per-handler `event.payload as X` casts are gone; a private brand makes the factory the only way to build a module subscriber.
+- 8ac88f3: Runs the API logs test suite without per-file Vitest module isolation by replacing cached module mocks with explicit test dependencies.
+- 2933c33: Adds drain-boundary Zod validation for current outbox publisher event payloads.
+- Updated dependencies [eb40964]
+- Updated dependencies [5c18360]
+- Updated dependencies [2c156d2]
+- Updated dependencies [067a260]
+- Updated dependencies [34ba284]
+- Updated dependencies [a56748d]
+- Updated dependencies [5707d6d]
+- Updated dependencies [7a9943d]
+- Updated dependencies [f788565]
+- Updated dependencies [b9c3f32]
+- Updated dependencies [c17dd6e]
+- Updated dependencies [a81b68c]
+- Updated dependencies [c0a883c]
+- Updated dependencies [72ce351]
+- Updated dependencies [e47f8da]
+- Updated dependencies [736249b]
+- Updated dependencies [2bc5595]
+- Updated dependencies [b1f57d1]
+- Updated dependencies [de54da2]
+- Updated dependencies [7b175f5]
+- Updated dependencies [7ca4c65]
+- Updated dependencies [f104ff2]
+- Updated dependencies [68e4022]
+- Updated dependencies [5bcdbf4]
+- Updated dependencies [97162dd]
+- Updated dependencies [b694b09]
+- Updated dependencies [c47be09]
+- Updated dependencies [f9f059e]
+- Updated dependencies [940696a]
+- Updated dependencies [f98c2be]
+- Updated dependencies [e9396c9]
+- Updated dependencies [ae7a63c]
+- Updated dependencies [5729548]
+- Updated dependencies [f92122b]
+- Updated dependencies [4207772]
+- Updated dependencies [e250c4c]
+- Updated dependencies [b525dcd]
+- Updated dependencies [d49ee4c]
+- Updated dependencies [2883ab4]
+- Updated dependencies [857fd73]
+- Updated dependencies [a982f20]
+- Updated dependencies [aca162b]
+- Updated dependencies [998eba3]
+- Updated dependencies [5327934]
+- Updated dependencies [314e84e]
+- Updated dependencies [3afb7e3]
+- Updated dependencies [139e3be]
+- Updated dependencies [247cbd6]
+- Updated dependencies [c652a68]
+- Updated dependencies [121b42e]
+- Updated dependencies [75520ff]
+- Updated dependencies [bf8319f]
+- Updated dependencies [795f440]
+- Updated dependencies [3dcd751]
+- Updated dependencies [c0a883c]
+- Updated dependencies [362b3eb]
+- Updated dependencies [d6d4862]
+- Updated dependencies [c0a883c]
+- Updated dependencies [6077301]
+- Updated dependencies [9c1c947]
+- Updated dependencies [d635979]
+- Updated dependencies [3bea87f]
+- Updated dependencies [82d22e4]
+- Updated dependencies [d69b164]
+- Updated dependencies [2fb3e87]
+- Updated dependencies [b74f635]
+- Updated dependencies [ef1e917]
+- Updated dependencies [c6eb2ee]
+- Updated dependencies [f9153e8]
+- Updated dependencies [2933c33]
+- Updated dependencies [6181819]
+- Updated dependencies [a8905ed]
+- Updated dependencies [e699508]
+- Updated dependencies [282e66a]
+- Updated dependencies [0dd23a7]
+- Updated dependencies [9c149d1]
+- Updated dependencies [e1d4972]
+- Updated dependencies [f9bf446]
+- Updated dependencies [8ecc121]
+- Updated dependencies [d7b9596]
+  - @shipfox/api-workflows@0.1.0
+  - @shipfox/api-workflows-dto@0.1.0
+  - @shipfox/api-agent-dto@0.1.0
+  - @shipfox/node-fastify@0.2.0
+  - @shipfox/api-logs-dto@0.1.0
+  - @shipfox/node-drizzle@0.1.0
+  - @shipfox/api-auth-context@0.1.0
+  - @shipfox/node-opentelemetry@0.4.2
+  - @shipfox/node-postgres@0.3.2
+  - @shipfox/node-temporal@0.1.1
+  - @shipfox/node-module@0.1.0
+  - @shipfox/node-outbox@0.1.0
+  - @shipfox/config@1.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shipfox
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,151 @@
+# Shipfox API Logs
+
+Shipfox API Logs is the server-side store for step logs. It accepts an append-only
+byte stream from the runner under an offset protocol, enforces a per-job storage
+budget, keeps the hot bytes in PostgreSQL, and closes streams when a step ends or its
+runner is lost. It serves those logs back on a session-authed read endpoint, and owns its
+tables and routes.
+
+## Example
+
+Register the module with the API module runner:
+
+```ts
+import {logsModule} from '@shipfox/api-logs';
+import {createApp, listen} from '@shipfox/node-fastify';
+import {initializeModules} from '@shipfox/node-module';
+
+const {auth, routes} = await initializeModules({
+  modules: [logsModule],
+});
+
+await createApp({auth, routes});
+await listen();
+```
+
+This adds:
+
+- log database migrations from `libs/api/logs/drizzle`
+- the append route under `/runs/jobs/current/steps/:stepId/logs` (lease-authed)
+- the read route under `/steps/:stepId/attempts/:attempt/logs` (session-authed): inline NDJSON
+  while the stream is hot, a presigned object URL once it is compacted, plus the canonical
+  `SessionView` parsed from stored agent-session rows
+- the `logs.stream.closed` publisher and two close subscribers: the step-attempt-terminated
+  subscriber that closes each attempt's stream when the runner did not end it in-band, and the
+  job-terminated subscriber that force-closes any of a job's still-open streams
+- the Temporal lifecycle workers that carry a closed stream through to deletion: compaction to a
+  gzip object, plus the compaction-reconcile, stale-open-stream reaper, and retention-sweep crons
+
+## Streams
+
+One stream per `(job, step, attempt)`, scoped by `job_id` (from the lease), so a lease can
+only ever reach its own job's streams. There is no per-producer `kind` on the stream: every
+record a step emits lives in this one stream, and a reader filters by the record's `type`
+(see below) at read time without needing to know anything about the producer.
+
+## The log record contract
+
+One JSON object per line. The shared envelope is `{v, ts}`. The pipe rides on `stream` for
+output; control records are flat by `type`:
+
+```ts
+{v: 1, ts, type: 'output',        stream: 'stdout' | 'stderr', data}   // <= 16 KiB data
+{v: 1, ts, type: 'group_start',   group_id, parent_group_id, name}     // name <= 1 KiB
+{v: 1, ts, type: 'group_end',     group_id}
+{v: 1, ts, type: 'end',           total_bytes}
+{v: 1, ts, type: 'gap',           dropped_bytes}
+{v: 1, ts, type: 'agent_session', row}                                // normalized SessionView row
+{v: 1, ts, type: 'capped'}        // server-only
+{v: 1, ts, type: 'runner_lost'}   // server-only
+```
+
+### Write-path protection
+
+Two layers stop a runner from writing what it should not:
+
+1. **Lease scope** — the lease binds writes to the job's own `(step, attempt)`, so cross-job
+   injection is structurally impossible.
+2. **Distinct raw/write and stored/read unions** — every line is validated against the **raw**
+   record union. The server-only `capped`/`runner_lost` tombstones are members of the read union
+   only, not the raw write union, so a forged tombstone append is rejected (400). A forged
+   tombstone that is otherwise a valid record is logged as a narrowed audit warning (no payload,
+   no token).
+
+### Multi-level named groups
+
+`::group::<name>` / `::endgroup::` markers form a tree. The runner keeps a nesting stack: each
+`group_start` gets a monotonic id (`g1`, `g2`, …) and a `parent_group_id` (the enclosing group,
+or `null` at the root). Nesting is capped at depth 32; past the cap a group is flattened to plain
+output and counted so its matching `::endgroup::` never pops a real parent. The tree is
+recoverable from the ids, parent links, and the tombstone position at truncation.
+
+## Read path
+
+A session-authed `GET /steps/:stepId/attempts/:attempt/logs?cursor=N` serves one endpoint for both
+the live tail and the full history. It is workspace-scoped through the stream row's denormalized
+`workspace_id`; a 404 covers both a missing stream and a cross-workspace step, so existence never
+leaks.
+
+- **Hot (open, or closed but not yet compacted)** — inline NDJSON read from the Postgres chunks,
+  walked by chunk `seq` so server control tombstones interleave with normalized runner records
+  exactly as compaction concatenates them, making the inline bytes byte-identical to the
+  decompressed object.
+  Pages are bounded by `LOG_READ_INLINE_MAX_BYTES`; the client follows `has_more`/`next_cursor` to
+  drain the backlog, then tails from the last cursor.
+- **Cold (compacted, `object_key` set)** — a presigned GET URL (`LOG_READ_URL_TTL_SECONDS`) so the
+  browser fetches the object directly and API egress is bypassed.
+
+## Agent-session capture
+
+An agent step forwards each agent session entry as one `agent_session` record on this **same**
+append request, distinguished by its `type`. That append-side record carries the raw entry in
+`data`; it is a write contract only. On ingestion, the API parses each accepted entry into one or
+more canonical `SessionView` rows and writes those rows back as normal `agent_session` log records
+in the same chunk stream (`{v, ts, type: 'agent_session', row}`). The parser is selected from the
+step's `config.harness` (`pi` by default).
+
+The transformed records use the same offset-CAS append protocol, chunk table, compaction object,
+and `/logs` read endpoint as output records. Retries, gaps, closed-stream drops, and post-cap drops
+therefore behave exactly like any other log append: if the chunk is not stored, no normalized
+session records appear. A malformed or unsupported harness entry degrades to one `raw` row instead
+of failing ingestion or read.
+
+The `/logs` read endpoint returns one stream of log records. `state` and `truncated` come from the
+stream row so clients know when polling can stop. The endpoint never parses session entries on
+read; it serves the normalized records stored by ingestion.
+
+The runner forwards entries opaquely and never splits one across records, so a record always
+carries a whole entry. Per-entry size is bounded by `LOG_MAX_SESSION_LINE_BYTES` (a larger line is
+rejected with 400, and the runner drops it with a `gap`); the request body limit
+(`LOG_APPEND_BODY_LIMIT_BYTES`) must hold a full line plus framing, enforced by a startup invariant.
+
+## Budget and terminal state
+
+One shared, job-wide, generous accrual budget covers a job's logs. When the job's stored bytes
+cross the budget, the job is capped and further appends are dropped. The append that crosses the
+budget is stored in full, then an in-band `capped` tombstone record is injected once; later appends
+are accepted-and-dropped. The cap is a **job-level** signal — a stream can be byte-complete and
+still sit under a capped job if a sibling step exhausted the shared budget.
+
+A stream closes on one of four triggers, all converging on the same guarded, exactly-once close:
+the runner declares its end in-band (an `end` record in an append), which closes the stream
+`declared` the moment those bytes commit; the step-attempt-terminated subscriber closes an attempt
+the runner never ended in-band, `declared` when the runner drained its spool or `timeout` when it
+abandoned it (carried on the event's `logOutcome`); the job-terminated sweep force-closes any of a
+job's still-open streams once the job goes terminal, after a grace period
+(`LOG_STREAM_CLOSE_GRACE_SECONDS`); and the reaper cron force-closes any stream still open past the
+lease window (`LOG_STREAM_REAP_AFTER_SECONDS`), the backstop for a stream the one-shot sweeps missed
+(one whose first append landed after they ran). Every timeout close sets `truncated` and injects a
+`runner_lost` tombstone; each close writes one `logs.stream.closed` event, which drives compaction.
+
+## Setup
+
+Install the package from the registry:
+
+```sh
+pnpm add @shipfox/api-logs
+```
+
+Configuration lives in `src/config.ts` (object storage for compacted logs, the per-job budget,
+the close grace period, the append body limit, and the read path's presigned-URL TTL and inline
+page cap). See each variable's `desc` for what to set.

package/dist/api/object-storage.d.ts

@@ -0,0 +1,69 @@
+import type { Readable } from 'node:stream';
+import { S3Client } from '@aws-sdk/client-s3';
+import { type LogObjectKeyParams } from '#core/entities/log-object.js';
+/** Lazily-built S3 client targeting the configured object store (Garage in dev). */
+export declare function s3Client(): S3Client;
+export declare function closeS3Client(): void;
+/** Readiness probe: true when the configured bucket answers a HEAD. Used by the app readiness check, not at process boot. */
+export declare function checkBucketReachable(): Promise<boolean>;
+/**
+ * S3 client for streaming compaction uploads. `s3Client()` is tuned fail-fast for the
+ * readiness HEAD, but each multipart part upload can take as long as the link needs, so
+ * the per-request timeout is disabled here. Temporal owns the retry budget, so the SDK
+ * keeps only a small attempt count for transient connection blips.
+ */
+export declare function uploadS3Client(): S3Client;
+export declare function closeUploadS3Client(): void;
+/**
+ * Per-attempt object key for a compacted stream: the stream's stable prefix
+ * (`logObjectKey` -> `{prefix}/{workspaceId}/{jobId}/{stepId}/{attempt}`, shared with retention
+ * and workspace prefix deletes) plus a unique `uploadToken` leaf. Each compaction attempt
+ * uploads to its own key and the winner records it atomically, so a slow or zombie attempt
+ * can never overwrite a published object. Losing and crashed attempts leave a leaf under the
+ * same prefix, reclaimed by the bucket's incomplete-multipart-abort rule and prefix-scoped
+ * retention.
+ */
+export declare function compactedObjectKey(identity: LogObjectKeyParams, uploadToken: string): string;
+export interface PutCompactedObjectParams {
+    key: string;
+    body: Readable;
+    metadata: Record<string, string>;
+    signal?: AbortSignal;
+    onProgress?: () => void;
+}
+/**
+ * Streams `body` (gzip-compressed NDJSON) to the compacted object via a multipart upload.
+ * `Content-Encoding: gzip` so a browser-direct presigned GET auto-decompresses. Aborts the
+ * multipart upload when `signal` fires, so a cancelled activity leaves no dangling parts.
+ *
+ * A hard crash (no graceful abort) can still strand incomplete multipart parts, which on
+ * pay-per-storage providers (S3, R2) bill until aborted. The worker never sets bucket
+ * policy, so the deployed bucket must carry an `AbortIncompleteMultipartUpload` lifecycle
+ * rule. `dev/garage/bootstrap.sh` provisions it for local Garage; self-hosters set the same
+ * rule on their bucket.
+ */
+export declare function putCompactedObject(params: PutCompactedObjectParams): Promise<void>;
+/**
+ * Deletes a compacted object. S3 `DeleteObject` is idempotent (a missing key is a success),
+ * so retention and the publish orphan guard can call it freely without a pre-check.
+ */
+export declare function deleteObject(key: string): Promise<void>;
+/**
+ * Presigns a GET for a compacted log object so the browser can fetch it directly,
+ * bypassing API egress. `getSignedUrl` only computes a signature locally (no network
+ * call), so the fail-fast `s3Client()` is fine here. `expiresAt` mirrors the URL's
+ * `LOG_READ_URL_TTL_SECONDS` lifetime so the caller can hand the client an absolute
+ * deadline without re-deriving it.
+ */
+export declare function presignedGetUrl(objectKey: string): Promise<{
+    url: string;
+    expiresAt: Date;
+}>;
+/**
+ * Deletes all objects under a per-attempt prefix. Retention deletes by prefix, not only the
+ * recorded `object_key`, to reclaim orphan leaves from failed compaction attempts.
+ *
+ * Pass a trailing slash so attempt `1` never matches attempt `10`.
+ */
+export declare function deleteObjectsByPrefix(prefix: string): Promise<void>;
+//# sourceMappingURL=object-storage.d.ts.map

package/dist/api/object-storage.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"object-storage.d.ts","sourceRoot":"","sources":["../../src/api/object-storage.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAC,QAAQ,EAAC,MAAM,aAAa,CAAC;AAC1C,OAAO,EAML,QAAQ,EACT,MAAM,oBAAoB,CAAC;AAI5B,OAAO,EAAC,KAAK,kBAAkB,EAAe,MAAM,8BAA8B,CAAC;AAYnF,oFAAoF;AACpF,wBAAgB,QAAQ,IAAI,QAAQ,CAcnC;AAED,wBAAgB,aAAa,IAAI,IAAI,CAGpC;AAED,6HAA6H;AAC7H,wBAAsB,oBAAoB,IAAI,OAAO,CAAC,OAAO,CAAC,CAO7D;AAID;;;;;GAKG;AACH,wBAAgB,cAAc,IAAI,QAAQ,CAYzC;AAED,wBAAgB,mBAAmB,IAAI,IAAI,CAG1C;AAED;;;;;;;;GAQG;AACH,wBAAgB,kBAAkB,CAAC,QAAQ,EAAE,kBAAkB,EAAE,WAAW,EAAE,MAAM,GAAG,MAAM,CAE5F;AAOD,MAAM,WAAW,wBAAwB;IACvC,GAAG,EAAE,MAAM,CAAC;IACZ,IAAI,EAAE,QAAQ,CAAC;IACf,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACjC,MAAM,CAAC,EAAE,WAAW,CAAC;IACrB,UAAU,CAAC,EAAE,MAAM,IAAI,CAAC;CACzB;AAED;;;;;;;;;;GAUG;AACH,wBAAsB,kBAAkB,CAAC,MAAM,EAAE,wBAAwB,GAAG,OAAO,CAAC,IAAI,CAAC,CAiCxF;AAED;;;GAGG;AACH,wBAAsB,YAAY,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAE7D;AAED;;;;;;GAMG;AACH,wBAAsB,eAAe,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC;IAAC,GAAG,EAAE,MAAM,CAAC;IAAC,SAAS,EAAE,IAAI,CAAA;CAAC,CAAC,CAYhG;AAED;;;;;GAKG;AACH,wBAAsB,qBAAqB,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAmCzE"}

package/dist/api/object-storage.js

@@ -0,0 +1,199 @@
+import { DeleteObjectCommand, DeleteObjectsCommand, GetObjectCommand, HeadBucketCommand, ListObjectsV2Command, S3Client } from '@aws-sdk/client-s3';
+import { Upload } from '@aws-sdk/lib-storage';
+import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
+import { config } from '#config.js';
+import { logObjectKey } from '#core/entities/log-object.js';
+const explicitS3Credentials = config.LOG_STORAGE_S3_ACCESS_KEY_ID && config.LOG_STORAGE_S3_SECRET_ACCESS_KEY ? {
+    accessKeyId: config.LOG_STORAGE_S3_ACCESS_KEY_ID,
+    secretAccessKey: config.LOG_STORAGE_S3_SECRET_ACCESS_KEY
+} : undefined;
+let _client;
+/** Lazily-built S3 client targeting the configured object store (Garage in dev). */ export function s3Client() {
+    if (!_client) {
+        _client = new S3Client({
+            endpoint: config.LOG_STORAGE_S3_ENDPOINT,
+            region: config.LOG_STORAGE_S3_REGION,
+            forcePathStyle: config.LOG_STORAGE_S3_FORCE_PATH_STYLE,
+            ...explicitS3Credentials ? {
+                credentials: explicitS3Credentials
+            } : {},
+            // Fail fast: a slow or black-holed endpoint must not hang callers behind
+            // SDK backoff. One retry, short connect/request timeouts.
+            maxAttempts: 2,
+            requestHandler: {
+                connectionTimeout: 1_000,
+                requestTimeout: 3_000
+            }
+        });
+    }
+    return _client;
+}
+export function closeS3Client() {
+    _client?.destroy();
+    _client = undefined;
+}
+/** Readiness probe: true when the configured bucket answers a HEAD. Used by the app readiness check, not at process boot. */ export async function checkBucketReachable() {
+    try {
+        await s3Client().send(new HeadBucketCommand({
+            Bucket: config.LOG_STORAGE_S3_BUCKET
+        }));
+        return true;
+    } catch  {
+        return false;
+    }
+}
+let _uploadClient;
+/**
+ * S3 client for streaming compaction uploads. `s3Client()` is tuned fail-fast for the
+ * readiness HEAD, but each multipart part upload can take as long as the link needs, so
+ * the per-request timeout is disabled here. Temporal owns the retry budget, so the SDK
+ * keeps only a small attempt count for transient connection blips.
+ */ export function uploadS3Client() {
+    if (!_uploadClient) {
+        _uploadClient = new S3Client({
+            endpoint: config.LOG_STORAGE_S3_ENDPOINT,
+            region: config.LOG_STORAGE_S3_REGION,
+            forcePathStyle: config.LOG_STORAGE_S3_FORCE_PATH_STYLE,
+            ...explicitS3Credentials ? {
+                credentials: explicitS3Credentials
+            } : {},
+            maxAttempts: 3,
+            requestHandler: {
+                connectionTimeout: 5_000,
+                requestTimeout: 0
+            }
+        });
+    }
+    return _uploadClient;
+}
+export function closeUploadS3Client() {
+    _uploadClient?.destroy();
+    _uploadClient = undefined;
+}
+/**
+ * Per-attempt object key for a compacted stream: the stream's stable prefix
+ * (`logObjectKey` -> `{prefix}/{workspaceId}/{jobId}/{stepId}/{attempt}`, shared with retention
+ * and workspace prefix deletes) plus a unique `uploadToken` leaf. Each compaction attempt
+ * uploads to its own key and the winner records it atomically, so a slow or zombie attempt
+ * can never overwrite a published object. Losing and crashed attempts leave a leaf under the
+ * same prefix, reclaimed by the bucket's incomplete-multipart-abort rule and prefix-scoped
+ * retention.
+ */ export function compactedObjectKey(identity, uploadToken) {
+    return `${logObjectKey(config.LOG_STORAGE_S3_PREFIX, identity)}/${uploadToken}`;
+}
+// lib-storage defaults (5MB parts, queue of 4); we pin a small queue so peak buffer memory
+// stays bounded under concurrent compactions (queueSize x partSize per active upload).
+const UPLOAD_PART_SIZE = 5 * 1024 * 1024;
+const UPLOAD_QUEUE_SIZE = 2;
+/**
+ * Streams `body` (gzip-compressed NDJSON) to the compacted object via a multipart upload.
+ * `Content-Encoding: gzip` so a browser-direct presigned GET auto-decompresses. Aborts the
+ * multipart upload when `signal` fires, so a cancelled activity leaves no dangling parts.
+ *
+ * A hard crash (no graceful abort) can still strand incomplete multipart parts, which on
+ * pay-per-storage providers (S3, R2) bill until aborted. The worker never sets bucket
+ * policy, so the deployed bucket must carry an `AbortIncompleteMultipartUpload` lifecycle
+ * rule. `dev/garage/bootstrap.sh` provisions it for local Garage; self-hosters set the same
+ * rule on their bucket.
+ */ export async function putCompactedObject(params) {
+    const upload = new Upload({
+        client: uploadS3Client(),
+        partSize: UPLOAD_PART_SIZE,
+        queueSize: UPLOAD_QUEUE_SIZE,
+        params: {
+            Bucket: config.LOG_STORAGE_S3_BUCKET,
+            Key: params.key,
+            Body: params.body,
+            ContentType: 'application/x-ndjson',
+            ContentEncoding: 'gzip',
+            Metadata: params.metadata
+        }
+    });
+    if (params.onProgress) upload.on('httpUploadProgress', params.onProgress);
+    if (params.signal) {
+        // Subscribe before the aborted-check so a signal firing between them cannot be missed,
+        // then handle the already-aborted case eagerly (an already-aborted signal never fires 'abort').
+        params.signal.addEventListener('abort', ()=>{
+            void upload.abort().catch(()=>undefined);
+        }, {
+            once: true
+        });
+        if (params.signal.aborted) {
+            await upload.abort().catch(()=>undefined);
+            throw new Error('Compaction upload aborted before it started');
+        }
+    }
+    await upload.done();
+}
+/**
+ * Deletes a compacted object. S3 `DeleteObject` is idempotent (a missing key is a success),
+ * so retention and the publish orphan guard can call it freely without a pre-check.
+ */ export async function deleteObject(key) {
+    await s3Client().send(new DeleteObjectCommand({
+        Bucket: config.LOG_STORAGE_S3_BUCKET,
+        Key: key
+    }));
+}
+/**
+ * Presigns a GET for a compacted log object so the browser can fetch it directly,
+ * bypassing API egress. `getSignedUrl` only computes a signature locally (no network
+ * call), so the fail-fast `s3Client()` is fine here. `expiresAt` mirrors the URL's
+ * `LOG_READ_URL_TTL_SECONDS` lifetime so the caller can hand the client an absolute
+ * deadline without re-deriving it.
+ */ export async function presignedGetUrl(objectKey) {
+    const ttlSeconds = config.LOG_READ_URL_TTL_SECONDS;
+    // Stamp the deadline from before signing: getSignedUrl bakes its expiry from the clock at
+    // sign time, so reading Date.now() after the await could report a deadline slightly past the
+    // real one. Computing it first keeps expiresAt a conservative (never-late) bound.
+    const expiresAt = new Date(Date.now() + ttlSeconds * 1000);
+    const url = await getSignedUrl(s3Client(), new GetObjectCommand({
+        Bucket: config.LOG_STORAGE_S3_BUCKET,
+        Key: objectKey
+    }), {
+        expiresIn: ttlSeconds
+    });
+    return {
+        url,
+        expiresAt
+    };
+}
+/**
+ * Deletes all objects under a per-attempt prefix. Retention deletes by prefix, not only the
+ * recorded `object_key`, to reclaim orphan leaves from failed compaction attempts.
+ *
+ * Pass a trailing slash so attempt `1` never matches attempt `10`.
+ */ export async function deleteObjectsByPrefix(prefix) {
+    if (prefix === '') {
+        throw new Error('deleteObjectsByPrefix refuses an empty prefix (it would target the whole bucket)');
+    }
+    const client = s3Client();
+    let continuationToken;
+    do {
+        const listed = await client.send(new ListObjectsV2Command({
+            Bucket: config.LOG_STORAGE_S3_BUCKET,
+            Prefix: prefix,
+            ContinuationToken: continuationToken
+        }));
+        const objects = (listed.Contents ?? []).flatMap((object)=>object.Key ? [
+                {
+                    Key: object.Key
+                }
+            ] : []);
+        if (objects.length > 0) {
+            const deleted = await client.send(new DeleteObjectsCommand({
+                Bucket: config.LOG_STORAGE_S3_BUCKET,
+                Delete: {
+                    Objects: objects,
+                    Quiet: true
+                }
+            }));
+            if (deleted.Errors && deleted.Errors.length > 0) {
+                const [first] = deleted.Errors;
+                throw new Error(`Failed to delete ${deleted.Errors.length} object(s) under ${prefix}: ${first?.Key} ${first?.Message}`);
+            }
+        }
+        continuationToken = listed.IsTruncated ? listed.NextContinuationToken : undefined;
+    }while (continuationToken)
+}
+
+//# sourceMappingURL=object-storage.js.map