ccqa-tools 1.37.0

package/README.md

@@ -0,0 +1,177 @@
+# ccqa-tools
+
+What `ccqa` needs installed **inside the application under test**, as one
+dependency with a directory per feature. Today that is coverage; the shape is
+here so the next one does not cost the consuming repository another dependency
+review.
+
+Everything in here runs in somebody else's application, which sets the rule for
+what may be added: **near-zero dependencies**. Coverage's only runtime
+dependency is `acorn`, and the Temporal integration is an optional peer. A
+feature that needs more than that belongs in its own package, or every consumer
+of one feature carries the others'.
+
+## Coverage
+
+`ccqa run --coverage` says which files each spec actually reached — including
+the ones a Temporal activity ran in another process. The browser half needs
+nothing installed: `ccqa` reads V8's own counters. This is the server half.
+
+### What it does
+
+`ccqa` sets a cookie on the browser at the start of each spec. Every request
+that browser makes carries it — and nothing else does, which is what lets an
+instrumented server tell one spec's traffic from the rest of a shared
+environment's. A load hook rewrites the project's own modules so that entering
+one records the file into whichever spec's context the current request belongs
+to, and a collector pushes what it has once a second.
+
+While no spec is running, an instrumented call is one global read and one
+truthiness test. That is the whole reason this needs no sampling — and the
+reason to leave the switch off in production rather than "just leaving it on".
+
+### Install
+
+```sh
+pnpm add -D ccqa-tools
+```
+
+### Turn it on
+
+Three things, and the process is measured:
+
+```sh
+CCQA_COVERAGE=1 \
+CCQA_COVERAGE_ENDPOINT=http://<host running ccqa run>:4757 \
+node --import ccqa-tools/coverage/register server.js
+```
+
+With `CCQA_COVERAGE` unset the register hook is never loaded and the
+application pays nothing at all.
+
+| Variable | Meaning |
+| --- | --- |
+| `CCQA_COVERAGE` | `1` to enable. Any other value is a `<runId>.<specId>` and becomes the ambient spec for a process with no inbound request to read — a worker started per spec. |
+| `CCQA_COVERAGE_ENDPOINT` | Where to push. Unset collects in memory and reports nothing. |
+| `CCQA_COVERAGE_TOKEN` | Sent as a bearer token, but the current `ccqa` sink does not check it. For a relay in front of it, or a future endpoint that does. |
+| `CCQA_COVERAGE_ROOT` | Root that file ids are relative to. Defaults to `process.cwd()`. In a workspace, point it at a directory containing the sibling packages too, and give `ccqa` the same one through `coverage.projectRoot`. |
+| `CCQA_COVERAGE_INCLUDE` | Comma-separated directories to instrument, relative to the root. Defaults to `src`. |
+| `CCQA_COVERAGE_DEBUG` | `1` for diagnostics on stderr. |
+
+### If the server code is bundled
+
+A load hook only ever sees what the runtime loads, so a bundled server is
+invisible to it — the bundler already swallowed the sources. Instrument at
+build time instead, and keep the register hook: it is what opens the per-request
+context the instrumentation records into.
+
+```ts
+// next.config.ts
+import { withCoverage } from "ccqa-tools/coverage/next";
+
+export default withCoverage(nextConfig, { root: import.meta.dirname });
+```
+
+Only server bundles are touched. The browser is measured through V8's counters
+and needs nothing injected.
+
+### If a chat platform drives the flow
+
+A Slack webhook is sent by Slack, not by the browser under test, so none of the
+carriers above is present and everything the flow runs would be unattributed.
+What the payload does say is which user acted, and recording that is enough for
+`ccqa` to work out the rest at its end.
+
+```ts
+import { slackActor } from "ccqa-tools/coverage/slack";
+
+app.use(slackActor());   // after whatever parses the body
+```
+
+It takes no arguments on purpose. It does not know which users are being
+measured, when a test is running, or what a spec is — the run holds all of
+that and nothing is ever sent back here. Which users matter is declared in the
+consuming project's `.ccqa/config.yaml`, under `coverage.actors.slack`.
+
+Extraction is a fixed list of the places Slack puts a user id (Events API,
+interactivity payloads, slash commands). A shape not on the list records
+nothing rather than guessing: a wrong identity is a wrong answer, a missing one
+is a counted gap.
+
+### If work continues in Temporal
+
+Three interceptors carry the spec across the three hops. The workflow one lives
+at its own specifier because Temporal evaluates it inside a deterministic
+sandbox with no Node built-ins. The Temporal SDK is not declared as a
+dependency here — these subpaths resolve it from the application's own tree,
+and declaring it would put a protobuf runtime in the lockfile of every consumer
+that has no Temporal at all.
+
+```ts
+import {
+  createClientInterceptor,
+  createActivityInterceptor,
+} from "ccqa-tools/coverage/temporal";
+
+new Client({ interceptors: { workflow: [createClientInterceptor()] } });
+
+Worker.create({
+  interceptors: {
+    activity: [() => ({ inbound: createActivityInterceptor() })],
+    workflowModules: ["ccqa-tools/coverage/temporal/workflow"],
+  },
+});
+```
+
+A pre-built workflow bundle takes the same specifier through
+`bundleWorkflowCode({ workflowInterceptorModules })`.
+
+The workflow body itself is not measured. It runs in the sandbox, which is one
+of the places nothing can reach; the activities it schedules are ordinary Node
+and are measured normally.
+
+### If the server is not Node's `http`
+
+`ccqa-tools/coverage/register` wraps `node:http`, which covers every framework that
+receives its requests from it. Anything else — an edge runtime, a fetch-style
+handler — opens the context itself:
+
+```ts
+import { coverageMiddleware, withCoverage } from "ccqa-tools/coverage/middleware";
+```
+
+### Things that will waste your afternoon
+
+- **A task runner that filters the environment turns this off silently.** If
+  the process gets `NODE_OPTIONS` but not `CCQA_COVERAGE` — which is exactly
+  what a runner with a declared-variable allowlist does — the preload loads
+  and then does nothing. Check that `CCQA_COVERAGE_DEBUG=1` prints `armed in
+  pid …` from the process actually serving requests, not just its launcher.
+- **Prefer an absolute path in `--import`.** `NODE_OPTIONS` is inherited by
+  every child process, and in a monorepo those include packages that cannot
+  resolve `ccqa-tools` at all. `--import file:///abs/path/to/register.js`
+  has no such failure mode.
+- **Clear the build cache after adding the plugin.** A bundler that cached its
+  modules will not re-run a loader you just added.
+- **Never write to stdout from a preload.** It is shared with whatever the host
+  process was parsing there. This package writes to stderr only. Most of that
+  is behind `CCQA_COVERAGE_DEBUG`, but a handful of warnings that mean nothing
+  is being reported at all — no endpoint set, a push failing, the load hooks
+  never installing — print regardless, since those are not debugging detail.
+
+### What it cannot see
+
+Declared up front, because a silent gap reads as "never reached" and that is
+the answer this exists to produce:
+
+- code that runs outside any request — schedulers, queue consumers, timers
+  started before the spec;
+- worker threads and other separate isolates, including a Temporal workflow
+  body;
+- module top level, which runs once and would otherwise belong to whichever
+  spec happened to import it first. Recorded separately;
+- a process killed outright (`SIGKILL`, an OOM kill) rather than exited — the
+  collector flushes on `beforeExit`, which a hard kill never fires.
+
+Executions that run during a spec but outside its context are counted, and
+`ccqa` reports the count next to the result.

package/dist/coverage/collector.cjs

@@ -0,0 +1,229 @@
+Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
+//#region src/coverage/core.ts
+const RUNTIME_KEY = Symbol.for("ccqa.coverage.runtime");
+function globals() {
+	return globalThis;
+}
+function getRuntime() {
+	return globals()[RUNTIME_KEY];
+}
+/**
+* The key both halves of the collector agree on. A space separates them safely:
+* a spec id can never contain one, so identity keys and spec ids stay disjoint
+* where the collector tracks both in one map.
+*/
+function actorBucketKey(tag, at) {
+	return `${tag} ${at}`;
+}
+/** Drops a spec's bucket once it has been handed to the collector. */
+function closeBucket(runtime, specId) {
+	if (runtime.buckets.delete(specId)) armGate(runtime);
+}
+/** Drops an identity's bucket once it has been handed to the collector. */
+function closeActorBucket(runtime, key) {
+	if (runtime.actors.delete(key)) armGate(runtime);
+}
+function armGate(runtime) {
+	runtime.active = runtime.buckets.size + runtime.actors.size;
+}
+//#endregion
+//#region src/coverage/runtime-env.ts
+/**
+* Diagnostics go to stderr and nowhere else. A `--import` preload is inherited
+* by every child node process, and writing to stdout corrupts whatever the host
+* was parsing there — enough to make a framework's own toolchain fail to start.
+*/
+function debugLog(config, message) {
+	if (!config.debug) return;
+	process.stderr.write(`[ccqa-tools] ${message}\n`);
+}
+//#endregion
+//#region src/coverage/collector.ts
+/**
+* Ships what this process reached to a sink, on a timer.
+*
+* Push, not pull: behind a load balancer nothing can address one replica of N,
+* so an endpoint the runner scrapes would silently report a fraction of the
+* truth. Every replica pushes instead, and the sink unions — file sets make
+* that commutative, associative and idempotent, so the sink never has to know
+* how many replicas there were.
+*/
+const DEFAULT_INTERVAL_MS = 1e3;
+const DEFAULT_IDLE_TTL_MS = 12e4;
+/**
+* How many times the exit-time flush may run. More than one because a push the
+* timer started may still be in flight when the first one fires, and that one
+* returns without sending anything.
+*/
+const MAX_EXIT_FLUSHES = 3;
+/**
+* How often a process that instrumented nothing re-announces itself when it has
+* nothing else to send.
+*
+* It has no files and no attributions to report, so the delta it pushes is
+* empty forever after the first success — and the run that heard it is over.
+* Without this, every later run against the same process is told nothing
+* reported at all, rather than that one process is blind.
+*/
+const BLIND_HEARTBEAT_MS = 3e4;
+function createCollectorState() {
+	return {
+		sent: /* @__PURE__ */ new Map(),
+		sentBoot: /* @__PURE__ */ new Set(),
+		sentActors: /* @__PURE__ */ new Map(),
+		lastChange: /* @__PURE__ */ new Map(),
+		lastSentUnattributed: 0,
+		lastSentUninstrumentedFiles: 0,
+		lastSentUninstrumentedProcess: false,
+		lastSentAt: 0,
+		droppedPushes: 0
+	};
+}
+function startCollector(options, config) {
+	const runtime = getRuntime();
+	if (runtime === void 0) return () => {};
+	const idleTtlMs = options.idleTtlMs ?? DEFAULT_IDLE_TTL_MS;
+	const state = createCollectorState();
+	let inFlight = false;
+	let consecutiveFailures = 0;
+	const flush = () => {
+		if (inFlight) return Promise.resolve();
+		evict(runtime, state, idleTtlMs);
+		const payload = diff(runtime, state);
+		if (payload === void 0) return Promise.resolve();
+		inFlight = true;
+		return post(options, payload).then(() => {
+			for (const [specId, files] of Object.entries(payload.specs)) {
+				const acked = state.sent.get(specId) ?? /* @__PURE__ */ new Set();
+				for (const file of files) acked.add(file);
+				state.sent.set(specId, acked);
+			}
+			for (const file of payload.boot) state.sentBoot.add(file);
+			for (const bucket of payload.actors) {
+				const key = actorBucketKey(bucket.tag, bucket.at);
+				const acked = state.sentActors.get(key) ?? /* @__PURE__ */ new Set();
+				for (const file of bucket.files) acked.add(file);
+				state.sentActors.set(key, acked);
+			}
+			state.lastSentUnattributed = payload.unattributed;
+			state.lastSentUninstrumentedFiles = payload.uninstrumentedFiles;
+			state.lastSentUninstrumentedProcess = payload.uninstrumentedProcess;
+			consecutiveFailures = 0;
+		}).catch((error) => {
+			state.droppedPushes++;
+			consecutiveFailures++;
+			if (config) debugLog(config, `push failed: ${String(error)}`);
+			if (consecutiveFailures === 1 || consecutiveFailures % 10 === 0) process.stderr.write(`[ccqa-tools] push to ${options.endpoint} failed ${consecutiveFailures} times in a row: ${String(error)}\n`);
+		}).finally(() => {
+			inFlight = false;
+		});
+	};
+	const timer = setInterval(() => void flush(), options.intervalMs ?? DEFAULT_INTERVAL_MS);
+	timer.unref?.();
+	let exitFlushes = 0;
+	const onBeforeExit = async () => {
+		if (consecutiveFailures > 0 || exitFlushes >= MAX_EXIT_FLUSHES) return;
+		exitFlushes++;
+		await flush();
+	};
+	process.on("beforeExit", onBeforeExit);
+	return () => {
+		clearInterval(timer);
+		process.off("beforeExit", onBeforeExit);
+	};
+}
+/**
+* `acked` is always a subset of `reached`, and neither ever shrinks, so equal
+* sizes mean nothing new — skip the copy-then-filter and check that first.
+*/
+function freshOf(reached, acked) {
+	if (acked !== void 0 && acked.size === reached.size) return [];
+	const fresh = [];
+	for (const file of reached) if (acked === void 0 || !acked.has(file)) fresh.push(file);
+	return fresh;
+}
+/** Exported for tests: the sink's HTTP round trip is the only real-I/O part. */
+function diff(runtime, state) {
+	const specs = {};
+	let any = false;
+	const now = Date.now();
+	for (const [specId, files] of runtime.buckets) {
+		const fresh = freshOf(files, state.sent.get(specId));
+		if (fresh.length === 0) continue;
+		specs[specId] = fresh;
+		state.lastChange.set(specId, now);
+		any = true;
+	}
+	const actors = [];
+	for (const [key, bucket] of runtime.actors) {
+		const fresh = freshOf(bucket.files, state.sentActors.get(key));
+		if (fresh.length === 0) continue;
+		actors.push({
+			tag: bucket.tag,
+			at: bucket.at,
+			files: fresh
+		});
+		state.lastChange.set(key, now);
+	}
+	const boot = freshOf(runtime.boot, state.sentBoot);
+	const unattributedChanged = runtime.unattributed !== state.lastSentUnattributed;
+	const healthChanged = runtime.uninstrumentedFiles !== state.lastSentUninstrumentedFiles || runtime.uninstrumentedProcess !== state.lastSentUninstrumentedProcess;
+	const reannounce = runtime.uninstrumentedProcess && now - state.lastSentAt >= BLIND_HEARTBEAT_MS;
+	if (!any && boot.length === 0 && actors.length === 0 && !unattributedChanged && !healthChanged && !reannounce) return;
+	state.lastSentAt = now;
+	return {
+		protocol: 1,
+		pid: runtime.pid,
+		startedAt: runtime.startedAt,
+		unattributed: runtime.unattributed,
+		uninstrumentedFiles: runtime.uninstrumentedFiles,
+		uninstrumentedProcess: runtime.uninstrumentedProcess,
+		specs,
+		boot,
+		actors,
+		droppedPushes: state.droppedPushes
+	};
+}
+/**
+* Forgets specs that stopped changing and have nothing outstanding, so a long
+* lived server does not accumulate every spec it has ever served — and so the
+* hot-path gate falls back to zero between runs.
+*/
+function evict(runtime, state, idleTtlMs) {
+	const now = Date.now();
+	const cutoff = now - idleTtlMs;
+	const actors = [...runtime.actors].map(([key, bucket]) => [key, bucket.files]);
+	dropQuiet([...runtime.buckets], state.sent, state, cutoff, now, (key) => closeBucket(runtime, key));
+	dropQuiet(actors, state.sentActors, state, cutoff, now, (key) => closeActorBucket(runtime, key));
+}
+function dropQuiet(entries, sent, state, cutoff, now, close) {
+	for (const [key, files] of entries) {
+		const seen = state.lastChange.get(key);
+		if (seen === void 0) {
+			state.lastChange.set(key, now);
+			continue;
+		}
+		if (seen > cutoff) continue;
+		if (freshOf(files, sent.get(key)).length > 0) continue;
+		close(key);
+		sent.delete(key);
+		state.lastChange.delete(key);
+	}
+}
+async function post(options, payload) {
+	const headers = { "content-type": "application/json" };
+	if (options.token) headers.authorization = `Bearer ${options.token}`;
+	const intervalMs = options.intervalMs ?? DEFAULT_INTERVAL_MS;
+	const response = await fetch(options.endpoint, {
+		method: "POST",
+		headers,
+		body: JSON.stringify(payload),
+		signal: AbortSignal.timeout(intervalMs * 5)
+	});
+	if (!response.ok) throw new Error(`sink returned ${response.status}`);
+}
+//#endregion
+exports.createCollectorState = createCollectorState;
+exports.diff = diff;
+exports.evict = evict;
+exports.startCollector = startCollector;

package/dist/coverage/collector.d.cts

@@ -0,0 +1,183 @@
+//#region src/coverage/core.d.ts
+/**
+ * The hot path, and the only module instrumented application code imports.
+ *
+ * It has **no imports at all**, on purpose. Bundlers pull this file into every
+ * layer they build — a Next.js RSC bundle, an SSR bundle, a route-handler
+ * bundle, a Temporal workflow sandbox — and each of those rejects a different
+ * subset of Node built-ins. `node:http` alone is enough to fail a Next.js
+ * webpack build outright. Everything platform-specific lives behind
+ * `ccqa-tools/coverage/register`, which only ever loads in real Node.
+ *
+ * For the same reason the process-wide state hangs off `globalThis` rather than
+ * module scope: a bundler produces several copies of this file per process, and
+ * a per-copy AsyncLocalStorage would shard attribution across them.
+ */
+/** The slice of AsyncLocalStorage this package needs, minus the import. */
+interface ContextStorage<T> {
+  getStore(): T | undefined;
+  run<R>(store: T, fn: () => R): R;
+}
+/**
+ * What one in-flight execution carries.
+ *
+ * Exactly one of the two says where its reach goes. `specId` is set when the
+ * request itself named a spec. `actor` is set when nothing did, and all that is
+ * known is who caused the work and when it was first asked for — enough for the
+ * run to decide later, and nothing this process could decide on its own.
+ */
+interface CoverageStore {
+  specId?: string;
+  actor?: ActorMark;
+  files: Set<string>;
+}
+/** Who caused a request, and the instant it arrived. Never interpreted here. */
+interface ActorMark {
+  tag: string;
+  at: number;
+}
+/** One identity's reach from one instant, as the collector ships it. */
+interface ActorBucket {
+  tag: string;
+  at: number;
+  files: Set<string>;
+}
+interface CoverageRuntime {
+  /**
+   * Shape version of this object. Two builds of `ccqa-tools` can end up in
+   * one process (an app dependency and a hoisted transitive one); the first to
+   * install wins and the rest defer to it, so the shape has to be recognisable.
+   */
+  readonly protocol: 1;
+  readonly als: ContextStorage<CoverageStore>;
+  /** specId -> reached file ids. Union is idempotent, so merging is free. */
+  readonly buckets: Map<string, Set<string>>;
+  /**
+   * `<tag> <at>` -> what that identity reached from that instant.
+   *
+   * Kept apart from `buckets` because nothing here knows which spec they belong
+   * to — or whether they belong to one at all. Most of them are other people
+   * using the same environment, and the run discards those.
+   */
+  readonly actors: Map<string, ActorBucket>;
+  /**
+   * Files reached at module top level. Kept out of the spec buckets because the
+   * first spec to import a module would otherwise own it, making the result
+   * depend on spec execution order.
+   */
+  readonly boot: Set<string>;
+  /**
+   * Number of open buckets, spec and identity alike. `record()` returns on zero
+   * before touching anything else — the reason instrumentation costs nothing
+   * while nothing is being measured, and the reason this package does not need
+   * sampling. A field rather than a `.size` sum because this check is the hot
+   * path itself.
+   *
+   * An identity bucket arms it the same way a spec does, so an application with
+   * the actor preset installed pays while any identity has acted recently, not
+   * only while a spec is running. That is the price of the application not
+   * being told which identities matter.
+   */
+  active: number;
+  /**
+   * Executions that ran while a spec was open but outside its async context.
+   * A silent gap here would read as "never reached", so it is always counted.
+   */
+  unattributed: number;
+  /**
+   * Files a load hook saw but could not turn into recorded coverage — an
+   * undecodable source, or a parse error. Uncounted, this would render
+   * identically to "reached by no spec", which is a lie.
+   */
+  uninstrumentedFiles: number;
+  /**
+   * Set when nothing in this process can be instrumented at all, so every file
+   * it runs is missing rather than some of them.
+   *
+   * A flag and not a count, because the failure is the process: counted as one
+   * file it reads as a rounding error next to the thousands it actually hides.
+   */
+  uninstrumentedProcess: boolean;
+  readonly pid: number;
+  readonly startedAt: number;
+}
+//#endregion
+//#region src/coverage/runtime-env.d.ts
+interface CoverageConfig {
+  enabled: boolean;
+  /** Set when the process is dedicated to one spec and has no request to read. */
+  ambientSpecId: string | undefined;
+  root: string;
+  /** Path prefixes, relative to `root`, whose files get instrumented. */
+  include: string[];
+  debug: boolean;
+}
+//#endregion
+//#region src/coverage/collector.d.ts
+interface CollectorOptions {
+  endpoint: string;
+  token?: string | undefined;
+  intervalMs?: number;
+  /** Drops a spec's bucket this long after its last change and last flush. */
+  idleTtlMs?: number;
+}
+interface CoveragePush {
+  protocol: 1;
+  pid: number;
+  startedAt: number;
+  unattributed: number;
+  uninstrumentedFiles: number;
+  uninstrumentedProcess: boolean;
+  /** Only ids not yet accepted by the sink, per spec. */
+  specs: Record<string, string[]>;
+  boot: string[];
+  /**
+   * What each identity reached, and when the work was first asked for. Which
+   * spec that is — if any — is the run's to decide; this side only reports.
+   */
+  actors: Array<{
+    tag: string;
+    at: number;
+    files: string[];
+  }>;
+  /**
+   * Failed push attempts over this process's whole life, not since the last
+   * ack. An application outlives the runs measuring it, and every second
+   * before the first of them stood up a sink is a failure — reported as a
+   * delta since the last ack, all of that arrives inside the first run and
+   * reads as that run having lost 25 minutes of reports. The sink subtracts
+   * what a process had already dropped when it first heard from it.
+   */
+  droppedPushes: number;
+}
+/** What survives between ticks: ack state plus the counters it gates on. */
+interface CollectorState {
+  /** Per spec, ids the sink has acknowledged. */
+  sent: Map<string, Set<string>>;
+  sentBoot: Set<string>;
+  /** Per identity bucket, ids the sink has acknowledged. */
+  sentActors: Map<string, Set<string>>;
+  /** Per bucket key (spec id or identity bucket), when it last gained a fresh id. */
+  lastChange: Map<string, number>;
+  /** `unattributed` as of the last push the sink acknowledged. */
+  lastSentUnattributed: number;
+  /** The two instrumentation-health figures as of that same push. */
+  lastSentUninstrumentedFiles: number;
+  lastSentUninstrumentedProcess: boolean;
+  /** When the last payload was built, for the blind-process heartbeat. */
+  lastSentAt: number;
+  /** Lifetime failed attempts. Never reset; the sink baselines it instead. */
+  droppedPushes: number;
+}
+declare function createCollectorState(): CollectorState;
+declare function startCollector(options: CollectorOptions, config?: CoverageConfig): () => void;
+/** Exported for tests: the sink's HTTP round trip is the only real-I/O part. */
+declare function diff(runtime: CoverageRuntime, state: CollectorState): CoveragePush | undefined;
+/**
+ * Forgets specs that stopped changing and have nothing outstanding, so a long
+ * lived server does not accumulate every spec it has ever served — and so the
+ * hot-path gate falls back to zero between runs.
+ */
+declare function evict(runtime: CoverageRuntime, state: CollectorState, idleTtlMs: number): void;
+//#endregion
+export { CollectorOptions, CollectorState, CoveragePush, createCollectorState, diff, evict, startCollector };