@analitikisler/ai-tag-debugger 0.7.0 → 0.7.1

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 0.7.1 (2026-10-13)
+
+- A page whose script stops responding after a step can no longer stall a run. Reading the page now has a 1 minute limit too, and when it runs out the journey ends as incomplete with a note. Before, only the model call and the browser action itself had a limit.
+- When a saved site's recording can't be cleaned, a warning now says so before the file is deleted.
+
+Replays of saved sites are closer to the saved run, after the first save of 18 real shops:
+
+- Scripted runs, and so replays, wait 3 seconds after the last step, so tags that send a moment after the last click (GA4 batches events) are captured as in an AI run.
+- A step that failed in the saved run is tried again on replay without ending the journey there.
+- A journey the AI could not finish (no cart, a bot check) stays unfinished on replay, so its missing events are not reported as broken.
+- Cookies the server sets for the page's scripts, such as a country, are kept in the recording; HttpOnly cookies, where sessions and logins live, are still left out. Some sites don't render without them, for example a cookie banner that never appears.
+- Images, fonts and video left out of the recording get an empty answer on replay instead of failing, since a failed image can stop a script.
+- Requests that had no answer yet when the browser closed are left out of the recording. On replay they made the page that asked for them again never finish loading.
+- A site whose recording is missing (the browser closed early) is not saved, and `replay` reports a saved site it can't replay and goes on with the next one instead of stopping.
+
 ## 0.7.0 (2026-10-12)
 
 Fixes from the first audit run across 18 real shops.

package/README.md

@@ -180,7 +180,7 @@ npx @analitikisler/ai-tag-debugger replay saved/*
     - warning duplicate purchase desktop ga4 page_view
 ```
 
-The first run's findings become the expected ones. When a change is right, for example a false alarm that is fixed, `replay --update` makes the new findings the expected ones. Use it to check a new version of the tool against sites you know, or a site change against its last good run. A saved site is usually a few MB. It leaves out tracking requests, images and video, cookies, authorization headers, form data, and requests with an email or other user data in their address. It still holds the site's pages and scripts, so keep sites you don't own out of public repositories.
+The first run's findings become the expected ones. When a change is right, for example a false alarm that is fixed, `replay --update` makes the new findings the expected ones. Use it to check a new version of the tool against sites you know, or a site change against its last good run. A saved site is usually a few MB. It leaves out tracking requests, images and video, session cookies and the cookies the browser sent, authorization headers, form data, and requests with an email or other user data in their address. It still holds the site's pages and scripts, so keep sites you don't own out of public repositories.
 
 ## Configure your site
 

package/README.tr.md

@@ -182,7 +182,7 @@ npx @analitikisler/ai-tag-debugger replay saved/*
     - warning duplicate purchase desktop ga4 page_view
 ```
 
-İlk çalıştırmanın bulguları beklenen bulgular olur. Bir değişiklik doğruysa (örneğin düzeltilmiş bir yanlış alarm), `replay --update` yeni bulguları beklenen bulgular yapar. Aracın yeni bir sürümünü bildiğiniz sitelerle ya da bir site değişikliğini son sağlıklı çalıştırmayla karşılaştırmak için kullanın. Kaydedilmiş bir site genellikle birkaç MB'tır. İzleme isteklerini, görselleri ve videoları, çerezleri, yetkilendirme başlıklarını, form verilerini ve adresinde e-posta ya da başka kullanıcı verisi olan istekleri içermez. Yine de sitenin sayfalarını ve betiklerini içerdiği için size ait olmayan siteleri herkese açık depolara koymayın.
+İlk çalıştırmanın bulguları beklenen bulgular olur. Bir değişiklik doğruysa (örneğin düzeltilmiş bir yanlış alarm), `replay --update` yeni bulguları beklenen bulgular yapar. Aracın yeni bir sürümünü bildiğiniz sitelerle ya da bir site değişikliğini son sağlıklı çalıştırmayla karşılaştırmak için kullanın. Kaydedilmiş bir site genellikle birkaç MB'tır. İzleme isteklerini, görselleri ve videoları, oturum çerezlerini ve tarayıcının gönderdiği çerezleri, yetkilendirme başlıklarını, form verilerini ve adresinde e-posta ya da başka kullanıcı verisi olan istekleri içermez. Yine de sitenin sayfalarını ve betiklerini içerdiği için size ait olmayan siteleri herkese açık depolara koymayın.
 
 ## Sitenizi ayarlayın
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@analitikisler/ai-tag-debugger",
-  "version": "0.7.0",
+  "version": "0.7.1",
   "description": "Open-source AI agent that QA-tests your website's analytics, ad and consent tracking (GA4, GTM, Google Ads, Meta, TikTok). Bring your own model (Claude, OpenAI, Gemini) or use it from your chat app over MCP.",
   "type": "module",
   "bin": {

package/src/agent.js

@@ -88,9 +88,11 @@ export function validateAction(input) {
   return null;
 }
 
-// One hung page or model request must not stall a whole run.
-const MODEL_TIMEOUT_MS = 180_000;
-const ACTION_TIMEOUT_MS = 120_000;
+// One hung page or model request must not stall a whole run. A page whose script never
+// stops (an endless loop after a click, for example) can't be read, so reading it has a limit too.
+const TIMEOUTS = { model: 180_000, action: 120_000, page: 60_000 };
+
+const LAST_STEP_WAIT_MS = 3000;
 
 class StepTimeout extends Error {}
 
@@ -105,14 +107,18 @@ function withTimeout(promise, ms, message) {
  * Lets the AI model drive the browser through one journey.
  * @param {AuditSession} session
  * @param {{ id: string, goal: string }} journey
- * @param {{ provider?: any, providerName?: string, client?: any, model?: string, maxSteps?: number, lang?: string, log?: (msg: string) => void }} opts
+ * @param {{ provider?: any, providerName?: string, client?: any, model?: string, maxSteps?: number, lang?: string, log?: (msg: string) => void, timeouts?: { model?: number, action?: number, page?: number } }} opts
+ *   timeouts: milliseconds for one model answer, one browser action and one read of the page.
  */
 export async function runAgentJourney(session, journey, opts = {}) {
   const provider = opts.provider ?? (await createProvider({ name: opts.providerName, model: opts.model, client: opts.client }));
   const maxSteps = opts.maxSteps ?? 30;
   const log = opts.log ?? (() => {});
+  const limit = { ...TIMEOUTS, ...opts.timeouts };
+  const seconds = (ms) => (ms % 60_000 ? `${Math.round(ms / 1000)} seconds` : ms === 60_000 ? "a minute" : `${ms / 60_000} minutes`);
+  const readPage = () => withTimeout(session.snapshot(), limit.page, `The page stopped responding: it could not be read for ${seconds(limit.page)}.`);
 
-  const first = await session.snapshot();
+  const first = await readPage();
   const chat = provider.conversation({
     system: systemPrompt(opts.lang ?? "en"),
     tools: TOOLS,
@@ -120,7 +126,7 @@ export async function runAgentJourney(session, journey, opts = {}) {
   });
 
   for (let turn = 0; turn < maxSteps; turn++) {
-    const { toolCalls, stop } = await withTimeout(chat.next(), MODEL_TIMEOUT_MS, "The model did not answer within 3 minutes.");
+    const { toolCalls, stop } = await withTimeout(chat.next(), limit.model, `The model did not answer within ${seconds(limit.model)}.`);
     if (stop === "refusal") return { completed: false, note: "The model declined to continue this journey." };
     if (toolCalls.length === 0) {
       if (stop === "max_tokens") return { completed: false, note: "The model response was cut off." };
@@ -146,8 +152,8 @@ export async function runAgentJourney(session, journey, opts = {}) {
         continue;
       }
       log(`  ${call.input.reason}`);
-      const error = await withTimeout(session.act(call.input), ACTION_TIMEOUT_MS, "The browser action did not finish within 2 minutes.");
-      results.push({ id: call.id, content: pageView(await session.snapshot(), error) });
+      const error = await withTimeout(session.act(call.input), limit.action, `The browser action did not finish within ${seconds(limit.action)}.`);
+      results.push({ id: call.id, content: pageView(await readPage(), error) });
     }
     if (finished) return finished;
     chat.addToolResults(results);
@@ -164,6 +170,10 @@ export async function runScriptedJourney(session, journey, opts = {}) {
     const error = await session.act(step);
     if (error && !step.optional) return { completed: false, note: `Step "${step.reason ?? step.action}" failed: ${error}` };
   }
+  // Tags often send a moment after the last click (GA4 batches events). An AI run gives them that time
+  // while the model answers; a scripted run would otherwise close the browser first.
+  await session.page.waitForTimeout(LAST_STEP_WAIT_MS);
+  if (journey.incomplete) return { completed: false, note: journey.incomplete };
   return { completed: true, note: "All scripted steps completed." };
 }
 

package/src/audit.js

@@ -92,8 +92,9 @@ export async function runAudit(opts) {
 
   const findings = filterFindings(runs.flatMap((run) => checkRun(plan, run, { lang })), events);
   if (opts.saveSite) {
-    await writeSavedSite(opts.saveSite, { config, plan, runs, findings });
-    log(`\nSaved the site for offline replay in ${opts.saveSite}`);
+    await writeSavedSite(opts.saveSite, { config, plan, runs, findings })
+      .then(() => log(`\nSaved the site for offline replay in ${opts.saveSite}`))
+      .catch((err) => log(`\nCould not save the site in ${opts.saveSite}: ${err.message}`));
   }
   let explained = null;
   if (provider) {

package/src/browser.js

@@ -227,18 +227,32 @@ async function replayTarget(a, locator) {
 // Images, fonts and video don't change what tracking sends, and are most of a recording's size.
 const MEDIA = /^(image|font|video|audio)\//;
 
+const MEDIA_FILE = /\.(png|jpe?g|gif|webp|avif|svg|ico|bmp|woff2?|ttf|otf|eot|mp4|webm|mov|mp3|m4a|ogg|wav)$/i;
+const isDroppedMedia = (request) =>
+  ["image", "font", "media"].includes(request.resourceType()) || MEDIA_FILE.test(new URL(request.url()).pathname);
+
+// Cookies the server sets for the page's scripts (a country or language, for example) are kept, since
+// sites often need them to render; HttpOnly ones, where sessions and logins live, are dropped. Replay
+// applies only one Set-Cookie header, so the kept ones are joined into one, one cookie per line.
+function keptHeaders(headers) {
+  const isSetCookie = (h) => /^set-cookie$/i.test(h.name);
+  const cookies = headers.filter(isSetCookie).flatMap((h) => h.value.split("\n")).filter((c) => !/;\s*httponly\s*(;|$)/i.test(c));
+  return [...headers.filter((h) => !isSetCookie(h)), ...(cookies.length ? [{ name: "Set-Cookie", value: cookies.join("\n") }] : [])];
+}
+
 const SECRET_REQUEST_HEADER = /^(cookie|authorization|proxy-authorization)$/i;
 
 /**
  * Drops images, fonts, video and tracking requests from a recording (they are answered locally on replay, and their
- * URLs can hold the user data redacted everywhere else), requests with user data in their address, form bodies,
- * cookies both ways and authorization headers.
+ * URLs can hold the user data redacted everywhere else), unanswered requests, requests with user data in their address, form bodies,
+ * the cookies the browser sent, HttpOnly cookies the server set, and authorization headers.
  */
-
 async function cleanRecording(file) {
   const har = JSON.parse(await readFile(file, "utf8"));
   har.log.entries = har.log.entries
     .filter((e) => !isCollectionEndpoint(e.request.url) && !MEDIA.test(e.response?.content?.mimeType ?? ""))
+    // A request that had no answer yet when the browser closed would never finish on replay either.
+    .filter((e) => e.response?.status > 0)
     // A request whose address holds user data (an email in a form GET, for example) is left out whole.
     .filter((e) => redactUrl(e.request.url).url === e.request.url)
     .map((e) => {
@@ -247,7 +261,7 @@ async function cleanRecording(file) {
       return {
         ...e,
         request: { ...request, cookies: [], headers: request.headers.filter((h) => !SECRET_REQUEST_HEADER.test(h.name)) },
-        response: { ...e.response, cookies: [], headers: e.response.headers.filter((h) => !/^set-cookie$/i.test(h.name)) },
+        response: { ...e.response, cookies: [], headers: keptHeaders(e.response.headers) },
       };
     });
   await writeFile(file, JSON.stringify(har));
@@ -376,7 +390,14 @@ export class AuditSession {
     // cancelled before its route handler runs, but its request event has already fired.
     this.context.on("request", onRequest);
     // Registered first, so the routes below run before it and tracking requests are still answered locally.
-    if (this.opts.replayHar) await this.context.routeFromHAR(this.opts.replayHar, { notFound: "abort" });
+    if (this.opts.replayHar) {
+      // Requests the recording lacks fail, except the media left out of it: those get an empty answer,
+      // since a failed image or icon can stop a script (a consent banner waiting for its logo, for example).
+      await this.context.route("**/*", (route) =>
+        isDroppedMedia(route.request()) ? route.fulfill({ status: 200, body: "" }) : route.abort(),
+      );
+      await this.context.routeFromHAR(this.opts.replayHar, { notFound: "fallback" });
+    }
     if (this.opts.blockHits) {
       await this.context.route((url) => isCollectionEndpoint(url.href), async (route) => {
         await route.fulfill({ status: 204, body: "" });
@@ -658,6 +679,13 @@ export class AuditSession {
    * @param {{ action: string, ref?: number, selector?: string, text?: string, value?: string, url?: string, reason?: string, accepts_consent?: boolean }} a
    */
   async act(a) {
+    const error = await this.perform(a);
+    // A scripted replay of this step may fail the same way, which must not end the journey there.
+    if (error) this.steps[this.currentStep].error = error;
+    return error;
+  }
+
+  async perform(a) {
     const describe = a.reason || `${a.action} ${a.selector ?? (a.ref != null ? `#${a.ref}` : a.text != null ? `"${a.text}"` : a.url ?? "")}`.trim();
     this.beginStep(describe, { action: a.action, acceptsConsent: !!a.accepts_consent });
     const step = this.steps[this.currentStep];
@@ -725,7 +753,12 @@ export class AuditSession {
       // Writes the recording. If the browser already went away, there is none to clean.
       const written = await this.context.close().then(() => true, () => false);
       // A recording that could not be cleaned may hold cookies or form data, so it is deleted.
-      if (written) await cleanRecording(this.opts.recordHar).catch(() => rm(this.opts.recordHar, { force: true }));
+      if (written) {
+        await cleanRecording(this.opts.recordHar).catch(async (err) => {
+          console.warn(`Could not clean the recording ${this.opts.recordHar}, so it was deleted: ${err.message}`);
+          await rm(this.opts.recordHar, { force: true });
+        });
+      }
     }
     await this.browser?.close();
     await this.egress?.close();

package/src/cli.js

@@ -113,7 +113,12 @@ async function main() {
     if (!dirs.length) throw new Error("replay needs the folder of at least one site saved with --save-site.");
     let changed = false;
     for (const dir of dirs) {
-      const r = await replaySavedSite(dir, { update: values.update, outDir: dirs.length === 1 ? values.out : undefined });
+      const r = await replaySavedSite(dir, { update: values.update, outDir: dirs.length === 1 ? values.out : undefined }).catch((err) => {
+        console.log(`✗ ${dir}: could not replay: ${err.message.split("\n")[0]}`);
+        changed = true;
+        return null;
+      });
+      if (!r) continue;
       const same = !r.missing.length && !r.extra.length;
       changed ||= !same;
       console.log(`${same ? "✓" : values.update ? "↻" : "✗"} ${r.site} (${dir}): ${same ? "same findings" : `${r.extra.length} new, ${r.missing.length} gone${values.update ? ", saved as expected" : ""}`}`);

package/src/replay.js

@@ -1,8 +1,9 @@
 // Saved sites: the responses a site sent during an audit and the steps taken, so a later
 // version of the tool can audit the same pages offline and show what it now reports
 // differently. Replays need no network and no AI model, so they cost nothing.
-import { mkdir, readFile, writeFile } from "node:fs/promises";
+import { mkdir, readFile, stat, writeFile } from "node:fs/promises";
 import path from "node:path";
+import { harName } from "./agent.js";
 import { runAudit } from "./audit.js";
 
 export const SAVED_SITE_FILE = "site.json";
@@ -15,9 +16,11 @@ export function replayableJourney(run) {
     id: run.journeyId,
     viewport: run.viewport,
     ...(start ? { start } : {}),
+    // A journey the AI could not finish (no cart, a bot check) is still incomplete when its steps replay.
+    ...(run.completed ? {} : { incomplete: run.note }),
     steps: (start ? steps.slice(1) : steps)
       .filter((s) => s.replay)
-      .map((s) => ({ ...s.replay, reason: s.label, ...(s.acceptsConsent ? { accepts_consent: true } : {}) })),
+      .map((s) => ({ ...s.replay, reason: s.label, ...(s.acceptsConsent ? { accepts_consent: true } : {}), ...(s.error ? { optional: true } : {}) })),
   };
 }
 
@@ -41,6 +44,12 @@ export function compareKeys(expected, actual) {
 /** Writes site.json next to the recorded responses. The findings of this run become the expected ones. */
 export async function writeSavedSite(dir, { config, plan, runs, findings }) {
   await mkdir(dir, { recursive: true });
+  for (const run of runs) {
+    const har = path.join(dir, harName(run.journeyId, run.viewport));
+    if (!(await stat(har).catch(() => null))) {
+      throw new Error(`The recording of ${run.journeyId} (${run.viewport}) is missing, probably because the browser closed early. Save the site again.`);
+    }
+  }
   const { journeys: _journeys, plan: _plan, gtm_container: _gtm, ...site } = config;
   const saved = {
     version: 1,
@@ -86,7 +95,9 @@ export async function replaySavedSite(dir, { update = false, outDir, log = () =>
     });
     findings.push(...result.findings);
     reportPaths.push(result.reportPath);
-    incomplete.push(...result.runs.filter((r) => !r.completed).map((r) => `${r.journeyId} (${r.viewport}): ${r.note}`));
+    // Journeys that were already unfinished when saved are compared through their findings only.
+    const wasIncomplete = (r) => journeys.find((j) => j.id === r.journeyId)?.incomplete === r.note;
+    incomplete.push(...result.runs.filter((r) => !r.completed && !wasIncomplete(r)).map((r) => `${r.journeyId} (${r.viewport}): ${r.note}`));
   }
   const actual = findingKeys(findings);
   const { missing, extra } = compareKeys(saved.expected ?? [], actual);