@ultimat3/scraping 23.0.0 → 25.0.0

package/README.md

@@ -84,6 +84,15 @@ Both legs replay from **one** fixture directory (`fixtureBrowser(dir)`), so a hy
 login, session handoff, HTTP bulk fetch — is tested end to end. Both legs apply the same robots
 gate, offline included.
 
+The robots gate follows RFC 9309 §2.3.1: a `4xx` robots.txt is **unavailable** (no rules); a `5xx`,
+a network error or the deadline is **unreachable**, which refuses every path on the origin
+(`X_SCRAPE_ROBOTS_DISALLOWED`) and is not cached. A `429` is a `4xx` the RFC calls unavailable; this
+gate treats it as unreachable anyway — our stricter policy, since a server that is rate-limiting
+the crawler has not said "no rules". A hand-written `fetchText` answers the text,
+`undefined` (no file) or `{ unreachable: '<why>' }` — `RobotsAnswer`. An offline scrape declares
+`robots: { ignore: 'offline fixture' }`: under the sealed test network its robots read is
+unreachable.
+
 ## A session a service is built on
 
 One connection, one run at a time, on its own exit, in a browser rented for that run — with a
@@ -210,7 +219,7 @@ export const answer = (input: { runId: string; index: number; answer: string })
 | `onSettled` | the job's own hook (`@ultimat3/jobs`), with what a scrape adds: `completed` hands over the `ScrapeReport`; `dead-lettered` / `dropped` / `refused` carry `code` and the last attempt's `usage` (`undefined` for `refused` — the body never ran). After the row is settled, under the job's tenant, **at most once** across a crash, and a hook that throws changes nothing |
 | `run({ progress, finalAttempt })` | the job's own, passed through |
 | `prompt` | `eventPrompt({ timeout, pollMs?, bus?, keySource?, env? })` polls the event bus in process — never `step.waitForEvent`, which would close the browser. An in-memory bus (`stored: false`) is refused outside development and test (`X_DRIVER_UNAVAILABLE`): the answer comes from another process. Event `promptEventName(runId, index)`; answered by `answerPrompt({ runId, index, answer })`, sealed for that one name. Timeout is `X_SCRAPE_PROMPT_UNANSWERED` |
-| the bus | the **stored** one (`createPgEventBus`) wherever the worker and the answering process differ. Every `x` boot installs it |
+| the bus | the **stored** one (`postgresEventBus`) wherever the worker and the answering process differ. Every `x` boot installs it |
 
 `PromptRequest` carries the run's `input` and `runId` — what ties a prompt to the app's own row
 for the run — and `index`, `clock`, `signal` and `keepAlive()` beside `label`, `scrape` and `url`:
@@ -240,7 +249,7 @@ resetScrapeClock(); // in the afterEach
 |---|---|---|
 | `systemScrapeClock` | waits | production |
 | `noWaitClock` | one turn of the event loop, whatever was asked; `now()` and deadlines are real | a test whose run waits on something that really happens in the same process — an answer the test publishes |
-| `testClock()` | advances virtual time and yields a microtask | a test of a timeout: thirty seconds pass in none |
+| `testScrapeClock()` | advances virtual time and yields a microtask | a test of a timeout: thirty seconds pass in none |
 
 A third-party driver takes part by reading three `SessionInit` fields: dial `proxy` or throw
 `egressUnsupported()`, pass `usage` to `pageOverTarget` and its HTTP transport, and answer
@@ -261,6 +270,14 @@ the port's. A caller that had to compute its own wrote
 `display !== 'none' && visibility !== 'hidden' && opacity !== '0'` a second time, which is the
 copy axiom 1 forbids.
 
+**An act scrolls its element into view; a read does not.** The `actionable` wait (every `click`,
+`type`, `fill`, `select`, `focus`, and `waitFor` at its default state) calls
+`ScrapeTarget.query(selector, { reveal: true })`, which scrolls the first match to the viewport's
+centre before measuring — `elementFromPoint` answers nothing off-screen, so an element below the
+fold read as covered. `text()`, `count()`, `values()` and `query()` never move the page. A target
+with no layout engine ignores `QueryOptions` (`cdp-target-reveal.test.ts`,
+`cdp-snapshot.e2e.test.ts`).
+
 **A frame verb reaches the FRAME.** `fill`, `type`, `select`, `clear`, `click` and `query` through
 a `ScrapeFrame` handle all address that frame's own document — never the parent's, even when the
 two carry the same ids, which is what an iframe'd SSO login looks like. `driver-parity-frames.test.ts`
@@ -313,8 +330,9 @@ bound threw away, so a count taken from one is never quietly a floor.
 All three are **redacted by value** on the way out, the same pass `page.html()` makes: a login
 endpoint fetched with the password in its query string, or a site that logs the credential it
 rejected, would otherwise put the value in `page.network()`/`page.console()` verbatim and from
-there into the stored failure artifact. The HTTP leg's `X_SCRAPE_HTTP_FAILED` cause is redacted
-too, at its throw site. Values shorter than `MIN_REDACTABLE_LENGTH` are not — a 3-character PIN is
+there into the stored failure artifact — percent-encoded, form-encoded or HTML-escaped as much as
+raw, since a password in a query string is never the raw string. The HTTP leg's
+`X_SCRAPE_HTTP_FAILED` cause is redacted too, at its throw site. Values shorter than `MIN_REDACTABLE_LENGTH` are not — a 3-character PIN is
 a substring of ordinary prose — and pixels never can be, which is why a typed secret TAINTS the
 page and `screenshot()`/`pdf()` are refused outright.
 
@@ -362,7 +380,7 @@ on a machine with no browser — the case CI is.
 | `driver-cdp.ts` / `cdp-*.ts` | the real browser, over a structural CDP port |
 | `driver-fake.ts` / `driver-recorded.ts` / `html-*.ts` | the offline drivers, on Bun's `HTMLRewriter` |
 | `http.ts` / `http-recorded.ts` | the second transport, live and replayed |
-| `auth.ts` / `session-state.ts` | acquire → persist → reuse → validate → burn. The session key encodes each part (`<sanitised>.<digest>`), so two account names that differ only outside `[a-zA-Z0-9._-]` are two sessions. A stored session is sealed |
+| `auth.ts` / `session-state.ts` | acquire → persist → reuse → validate → burn. The session key encodes each part (`<sanitised>.<digest>`), so two account names that differ only outside `[a-zA-Z0-9._-]` are two sessions. A stored session is sealed. `burnSession(plan, seen)` takes `recordVersion(record)` of the record the run used, so a record another run saved since is not burned |
 | `event-prompt.ts` | `eventPrompt()` and `answerPrompt()`: a prompt answered from another process, over the job event bus, with the browser open |
 | `cdp-resolver.ts` | a browser rented per session: acquired once, released once |
 | `usage.ts` | what one run used, counted where the work happens |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/scraping",
-  "version": "23.0.0",
+  "version": "25.0.0",
   "description": "Browser automation as a job: scrape() returns a JobHandle",
   "license": "MIT",
   "type": "module",
@@ -24,16 +24,16 @@
     "LICENSE"
   ],
   "engines": {
-    "bun": ">=1.4.0"
+    "bun": ">=1.4.2"
   },
   "scripts": {
     "typecheck": "tsc --noEmit -p tsconfig.json",
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "23.0.0",
-    "@ultimat3/jobs": "23.0.0",
-    "@ultimat3/schema": "23.0.0",
-    "@ultimat3/storage": "23.0.0"
+    "@ultimat3/core": "25.0.0",
+    "@ultimat3/jobs": "25.0.0",
+    "@ultimat3/schema": "25.0.0",
+    "@ultimat3/storage": "25.0.0"
   }
 }

package/src/artifacts.ts

@@ -5,7 +5,7 @@
 // Written through `@ultimat3/storage`'s driver, so the same call lands on a local disk in
 // development and on S3 in production, and this package owns no upload path of its own.
 
-import type { StorageDriver } from '@ultimat3/storage';
+import { DEFAULT_CONTENT_TYPE, type StorageDriver } from '@ultimat3/storage';
 
 export interface ArtifactRef {
   readonly key: string;
@@ -31,8 +31,6 @@ export interface ArtifactWriterInit {
 
 export const DEFAULT_ARTIFACT_PREFIX = 'scrape';
 
-export const DEFAULT_CONTENT_TYPE = 'application/octet-stream';
-
 /**
  * A `Map`, not an object literal — the same choice `failures.ts` makes, for the same reason. The
  * extension comes off a caller-supplied filename, and on a download that filename came off the
@@ -57,7 +55,7 @@ export const contentTypeFor = (name: string): string =>
  * that has not configured storage should still be able to run a scrape; losing the artifact is a
  * cost, and refusing the run over it is a bigger one.
  */
-export function createArtifactWriter(init: ArtifactWriterInit): ArtifactWriter {
+export function artifactWriter(init: ArtifactWriterInit): ArtifactWriter {
   const saved: ArtifactRef[] = [];
   const prefix = `${init.prefix ?? DEFAULT_ARTIFACT_PREFIX}/${init.scrape}/${init.runId}`;
   return {

package/src/auth.ts

@@ -20,10 +20,15 @@
 import type { Logger } from '@ultimat3/core';
 import { finiteCount } from '@ultimat3/core';
 import type { ScrapeClock } from './clock';
-import { authFailed, promptUnanswered, sessionExpired } from './error-throws';
+import { authFailed, promptUnanswered, scrapeSessionExpired } from './error-throws';
 import type { ScrapePage } from './page';
 import type { ScrapeSecrets } from './secrets';
-import { type ScrapeSessionStore, type SessionState, sessionDigest } from './session-state';
+import {
+  recordVersion,
+  type ScrapeSessionStore,
+  type SessionState,
+  sessionDigest,
+} from './session-state';
 
 export interface PromptRequest<I = unknown> {
   /**
@@ -119,7 +124,7 @@ export interface PromptInit<I = unknown> {
  */
 const KEEPALIVE_SELECTOR = 'ultimate-keepalive';
 
-export function createPrompt<I>(init: PromptInit<I>): (label: string) => Promise<string> {
+export function scrapePrompt<I>(init: PromptInit<I>): (label: string) => Promise<string> {
   let asked = 0;
   return async (label: string): Promise<string> => {
     const { scrape, handler, page } = init;
@@ -167,10 +172,30 @@ export interface AuthPlanInput<I> {
 export async function restorableSession<I>(
   plan: AuthPlanInput<I>,
 ): Promise<SessionState | undefined> {
+  return (await observeSession(plan)).restorable;
+}
+
+/** What `observeSession` answers: the session to restore, and the version of what was found. */
+export interface ObservedSession {
+  readonly restorable: SessionState | undefined;
+  /**
+   * `recordVersion` of the record found — restorable or NOT. A `reuse: false` run, or one whose
+   * record is past `maxAge`, still saw that record: a refused login must tombstone over it, and a
+   * run that compared against "nothing" let the rejected credential through again.
+   */
+  readonly seen: string | undefined;
+}
+
+/** `restorableSession`, plus the version of the record found, for the run's later writes. */
+export async function observeSession<I>(plan: AuthPlanInput<I>): Promise<ObservedSession> {
   const store = plan.auth?.store;
-  if (store === undefined) return undefined;
+  if (store === undefined) return { restorable: undefined, seen: undefined };
   const found = await store.load(plan.key);
-  if (found === undefined) return undefined;
+  if (found === undefined) return { restorable: undefined, seen: undefined };
+  return { restorable: restorableFrom(plan, found), seen: recordVersion(found) };
+}
+
+function restorableFrom<I>(plan: AuthPlanInput<I>, found: SessionState): SessionState | undefined {
   // The tombstone is read BEFORE `reuse` is honoured. `reuse: false` says "do not restore this
   // session"; it does not say "present the rejected credential again", and reading it first meant
   // a `reuse: false` scrape walked a refused password back to the login form on every requeue.
@@ -194,13 +219,42 @@ export async function restorableSession<I>(
   return found;
 }
 
-/** Written down so the next attempt cannot reach the login form with the same credentials. */
-export async function markRefused<I>(plan: AuthPlanInput<I>): Promise<void> {
+/**
+ * True when the stored record is still the one this run last saw — `seen` is the `recordVersion`
+ * it found or persisted, `undefined` when it found none. The store has no compare-and-set (nor has
+ * `StorageDriver` a conditional write to build one from), so a
+ * burn or a tombstone is a blind write: two runs on one key, and the one that failed destroyed the
+ * session the other had just persisted. Compared before the write, which narrows the race to the
+ * gap between this read and that write rather than the whole run. A record moved on is left alone.
+ */
+async function stillSeen<I>(
+  plan: AuthPlanInput<I>,
+  store: ScrapeSessionStore,
+  seen: string | undefined,
+  step: 'session.burn' | 'session.refuse',
+): Promise<boolean> {
+  const current = await store.load(plan.key);
+  if (current === undefined || recordVersion(current) === seen) return true;
+  plan.logger.warn('scrape.session.superseded', { step });
+  return false;
+}
+
+/**
+ * Written down so the next attempt cannot reach the login form with the same credentials — unless
+ * another run saved a session after this one read the record: that login SUCCEEDED, so this
+ * refusal is older news than the record it would overwrite.
+ */
+export async function markRefused<I>(
+  plan: AuthPlanInput<I>,
+  seen: string | undefined,
+): Promise<void> {
   const store = plan.auth?.store;
   if (store === undefined) return;
+  if (!(await stillSeen(plan, store, seen, 'session.refuse'))) return;
   await store.save({
     key: plan.key,
     savedAt: plan.clock.now().toISOString(),
+    version: crypto.randomUUID(),
     refusedAt: plan.clock.now().toISOString(),
     cookies: [],
     headers: {},
@@ -210,10 +264,19 @@ export async function markRefused<I>(plan: AuthPlanInput<I>): Promise<void> {
   });
 }
 
-/** New identity, from scratch. One call, because a flagged profile is unusable and must go. */
-export async function burnSession<I>(plan: AuthPlanInput<I>): Promise<void> {
-  if (plan.auth?.store === undefined) return;
-  await plan.auth.store.burn(plan.key);
+/**
+ * New identity, from scratch. One call, because a flagged profile is unusable and must go — the
+ * profile THIS run used: `seen` is that record's `recordVersion` (`observeSession`'s, or the one
+ * `persistSession` answered), and a record another run saved since is not burned.
+ */
+export async function burnSession<I>(
+  plan: AuthPlanInput<I>,
+  seen: string | undefined,
+): Promise<void> {
+  const store = plan.auth?.store;
+  if (store === undefined) return;
+  if (!(await stillSeen(plan, store, seen, 'session.burn'))) return;
+  await store.burn(plan.key);
   plan.logger.warn('scrape.session.burned', { burned: true });
 }
 
@@ -251,18 +314,31 @@ export async function ensureAuthenticated<I>(args: EnsureAuthInput<I>): Promise<
       return false;
     }
     args.logger.info('scrape.session.expired', { reused: false });
-    await burnSession(args);
+    await burnSession(args, recordVersion(args.restored));
   }
-  if (auth.login === undefined) throw sessionExpired(args.scrape, args.key);
+  if (auth.login === undefined) throw scrapeSessionExpired(args.scrape, args.key);
   await auth.login(context);
   return true;
 }
 
-/** After a login: keep what it produced. A 2FA code somebody typed is worth exactly this. */
-export async function persistSession<I>(plan: AuthPlanInput<I>, page: ScrapePage): Promise<void> {
+/**
+ * After a login: keep what it produced. A 2FA code somebody typed is worth exactly this. Answers
+ * the `version` it wrote — the record this run now owns, for a later burn to compare against.
+ */
+export async function persistSession<I>(
+  plan: AuthPlanInput<I>,
+  page: ScrapePage,
+): Promise<string | undefined> {
   const store = plan.auth?.store;
-  if (store === undefined) return;
+  if (store === undefined) return undefined;
   const snapshot = await page.session();
-  await store.save({ ...snapshot, key: plan.key, savedAt: plan.clock.now().toISOString() });
+  const version = crypto.randomUUID();
+  await store.save({
+    ...snapshot,
+    key: plan.key,
+    savedAt: plan.clock.now().toISOString(),
+    version,
+  });
   plan.logger.info('scrape.session.saved', sessionDigest(snapshot));
+  return version;
 }

package/src/cdp-arm.ts

@@ -4,7 +4,7 @@
 // three verbs `press`/`focus`/`accessibility` did not fit under it. The extraction is VERBATIM —
 // the request, console, `pageerror` and `error` handlers are the same code, one file over.
 
-import type { CdpPageLike, CdpRequestLike } from './cdp-port';
+import type { CdpBrowserLike, CdpPageLike, CdpRequestLike } from './cdp-port';
 import type { ScrapeClock } from './clock';
 import type { InterceptRules } from './intercept';
 import { interceptVerdict, refusalEntry } from './intercept';
@@ -174,3 +174,67 @@ export async function arm(init: CdpArmInit, sinks: CdpSinks): Promise<void> {
     crashed.value = readStringFrom(payload, 'message') ?? 'renderer crashed';
   });
 }
+
+/** A `Fetch.requestPaused` payload, read structurally — the fields the verdict needs, or nothing. */
+interface PausedRequest {
+  readonly requestId: string;
+  readonly resourceType: string;
+  readonly url: string;
+  readonly method: string;
+}
+
+const asPaused = (payload: unknown): PausedRequest | undefined => {
+  if (typeof payload !== 'object' || payload === null) return undefined;
+  const event = payload as Record<string, unknown>;
+  const request = event['request'];
+  if (typeof request !== 'object' || request === null) return undefined;
+  const { url, method } = request as Record<string, unknown>;
+  const { requestId, resourceType } = event;
+  if (typeof requestId !== 'string' || typeof url !== 'string') return undefined;
+  return {
+    requestId,
+    url,
+    resourceType: typeof resourceType === 'string' ? resourceType : 'Other',
+    method: typeof method === 'string' ? method : 'GET',
+  };
+};
+
+/**
+ * `allowHosts` and `block` on EVERY target the browser opens. Page-level interception (`arm`)
+ * covers the one page this package created; a popup — `window.open`, a `target=_blank` link — is
+ * a new target it never sees, and on Chrome 150 a scraped page's `window.open` to an off-list host
+ * reached that host. Browser-level `Fetch` pauses those requests too, and the same
+ * `interceptVerdict` decides them. The page's own requests are decided at the page first, so an
+ * allowed one is continued here as well and a refused one never arrives: one ring entry each.
+ *
+ * Enabled BEFORE the first navigation; a browser that refuses `Fetch.enable` rejects here, and
+ * `driver-cdp.ts`'s `opened()` closes it — a session whose rule would hold on one tab is refused.
+ */
+export async function armBrowser(
+  init: {
+    readonly browser: CdpBrowserLike;
+    readonly rules: InterceptRules;
+    readonly clock: ScrapeClock;
+  },
+  network: NetworkRing,
+): Promise<void> {
+  const session = await init.browser.target().createCDPSession();
+  session.on('Fetch.requestPaused', (payload) => {
+    const paused = asPaused(payload);
+    if (paused === undefined) return;
+    const type = asResourceType(paused.resourceType.toLowerCase());
+    const verdict = interceptVerdict(paused.url, type, init.rules);
+    const requestId = { requestId: paused.requestId };
+    // Caught, never floated, for `arm`'s reason: the target may be gone by the time this answers.
+    if (verdict === 'allow') {
+      session.send('Fetch.continueRequest', requestId).catch(() => undefined);
+      return;
+    }
+    const at = init.clock.now().getTime();
+    network.push(refusalEntry(paused.url, type, verdict, at, paused.method));
+    session
+      .send('Fetch.failRequest', { ...requestId, errorReason: 'BlockedByClient' })
+      .catch(() => undefined);
+  });
+  await session.send('Fetch.enable', { patterns: [{ urlPattern: '*' }] });
+}

package/src/cdp-port.ts

@@ -167,8 +167,29 @@ export interface CdpFrameLike {
   select(selector: string, ...values: string[]): Promise<string[]>;
 }
 
+/**
+ * A raw session on the BROWSER target — `browser.target().createCDPSession()` under puppeteer —
+ * with the event half `CdpSessionLike` does not need: `Fetch.requestPaused` arrives as an event.
+ * `event` stays a bare `string` for `CdpPageLike.on`'s reason.
+ */
+export interface CdpBrowserSessionLike extends CdpSessionLike {
+  on(event: string, handler: (payload: unknown) => void): unknown;
+}
+
+/** The browser's own target — `browser.target()` under puppeteer. */
+export interface CdpBrowserTargetLike {
+  createCDPSession(): Promise<CdpBrowserSessionLike>;
+}
+
 export interface CdpBrowserLike {
   newPage(): Promise<CdpPageLike>;
+  /**
+   * REQUIRED, unlike the page's optional verbs: it is how `allowHosts` reaches every target the
+   * browser opens — a popup, a `target=_blank` link — and not only the one page this package
+   * created. Page-level interception never sees those, and a rule that holds on one tab of a
+   * browser a scraped site controls is advisory. A launcher without it does not compile.
+   */
+  target(): CdpBrowserTargetLike;
   /** Present on a browser that owns a cookie jar. Absent ones answer `X_NOT_IMPLEMENTED`. */
   cookies?(): Promise<unknown>;
   setCookie?(...cookies: readonly unknown[]): Promise<void>;

package/src/cdp-snapshot.ts

@@ -8,16 +8,24 @@
 import type { StandardSchemaV1 } from '@ultimat3/schema';
 import { parse, t } from '@ultimat3/schema';
 import { browserRecord } from './browser-record';
-import type { ElementSnapshot } from './target';
+import type { ElementSnapshot, QueryOptions } from './target';
 
 /**
  * Returns JSON TEXT, not an object: a CDP round trip serialises the result anyway, and a string
  * has one deserialiser here — `parse` against the schema below — instead of an implicit one in
  * the library plus a cast.
+ *
+ * `reveal` scrolls BEFORE the loop, so every box below is measured where the page now is — the
+ * centre a click must land on, and the point `elementFromPoint` is asked about (`QueryOptions`).
  */
-export const snapshotExpression = (selector: string): string => `(() => {
+export const snapshotExpression = (
+  selector: string,
+  options: QueryOptions = {},
+): string => `(() => {
   const out = [];
-  for (const el of document.querySelectorAll(${JSON.stringify(selector)})) {
+  const matches = document.querySelectorAll(${JSON.stringify(selector)});
+  ${options.reveal === true ? "matches[0]?.scrollIntoView({ block: 'center', inline: 'center', behavior: 'instant' });" : ''}
+  for (const el of matches) {
     const style = getComputedStyle(el);
     const rect = el.getBoundingClientRect();
     const attrs = {};