@milaboratories/pl-flight-recorder 0.2.0

package/README.md

@@ -0,0 +1,119 @@
+# @milaboratories/pl-flight-recorder
+
+Explains an out-of-memory death in the block model layer after the fact, when
+the process is already gone and the data that caused it cannot be obtained.
+
+## Why it is built this way
+
+The failure this exists for kills the process in seconds and runs no shutdown
+path: V8 fatal out-of-memory, a failed native allocation, the OS out-of-memory
+killer. That rules out anything that collects on demand from a live process —
+`pprofDump`, cache metrics, a log that flushes on exit — because by the time
+anyone asks, there is nothing to ask.
+
+Three consequences shape the design:
+
+- **Records are appended synchronously.** A buffered stream loses exactly the
+  tail that explains the death. The absence of a terminating `session-end`
+  record is how a crash is detected.
+- **Memory is sampled from another thread.** While the middle layer sits inside
+  a synchronous pframes call its own timers do not fire, so its memory series
+  goes dark precisely when memory is growing fastest.
+- **The parent records the cause.** A thread that runs out of heap cannot
+  describe its own death; its last reading predates the blow-up. Node reports
+  `ERR_WORKER_OUT_OF_MEMORY` to the parent, which turns a guess into a fact.
+
+## Enabling it
+
+Inert unless `MI_FLIGHT_RECORDER_DIR` points at a directory. Recording appends
+on every recorded operation and that cost has not yet been measured against a
+real project, so it is opt-in rather than on by default.
+
+```
+MI_FLIGHT_RECORDER_DIR=~/platforma-flight <start the app>
+```
+
+## Reading a log
+
+```ts
+import { analyzeLatest, renderReport } from "@milaboratories/pl-flight-recorder";
+
+const analysis = analyzeLatest(dir); // prefers the session that crashed
+console.log(renderReport(analysis));
+```
+
+The verdict combines three independent lines of evidence, none sufficient alone:
+
+1. **What was in flight.** Every operation writes a begin and an end record, so
+   an unmatched begin names the exact call that was running when memory ran out.
+2. **Which memory ran out.** JS heap (confirmed by the parent), off-heap
+   ArrayBuffers handed out by the engine, native allocation, or the machine.
+   This decides whether `--max-old-space-size` is even relevant.
+3. **Why the data was that big.** Structural faults readable from the recorded
+   shape before any data is touched — a cartesian join, or axes that agree on
+   name and type but differ in domain — plus the observed row count against the
+   input the definition declared.
+
+## What is recorded, and how
+
+A definition is recorded by **shape**, not by a hand-written digest per
+definition type. `redact` walks whatever the seam was handed and rebuilds it:
+keys, numbers and the small set of strings that are schema survive verbatim,
+every other string becomes a hash and a length, and a column payload is replaced
+by counts. The default for an unrecognised string is to hash it, so a field this
+code has never seen can make a report less informative but never make it leak.
+
+- Kept: definition shape, column and axis names, value types, axis domains
+  (they carry join identity), filter operators, row/byte/partition counts.
+- Hashed: filter reference values, annotation values (`pl7.app/label` carries
+  user-entered sample names), column ids, every other string. The hash is
+  stable, so two labels can be compared without either being revealed.
+- Dropped: cell values, inline column payloads, partition keys.
+
+Two things bound one record: arrays keep their head and carry an `$omitted`
+marker, and a node budget stops a pathological definition from filling the log.
+A session's log is bounded the same way, at two segments of 32 MiB. Rotation
+rewrites into the new segment the three things a report cannot be produced
+without — the session header, the earliest memory reading, and the begin record
+of every operation still open — so repeated rotation costs only operations that
+already completed. The report states how many rotations happened.
+Class instances are named (`$opaque`) rather than walked, which matters because
+a definition can carry live accessors that reference each other.
+
+On a realistic definition (two columns, 72 partitions, a 1200-value filter) the
+record is ~1 KB against ~23 KB for the definition as it stands — but size is not
+why this exists. The definition as it stands cannot be sent to us at all.
+
+Only one module knows a definition's types: `data_summary`, which reads row
+counts and byte sizes out of `DataInfo` because those numbers live at
+type-specific positions and they are what predicts a join's cost.
+
+## Rules run in the analyzer
+
+Structural rules are not evaluated while recording. Nothing is computed on the
+hot path, the rules can be revised against logs that already exist, and one
+implementation covers every definition shape: join nodes are recognised by their
+discriminator and their children by position, so the original tree API and the
+V2 query API are read by the same walk.
+
+## Where it hooks in
+
+- `pl-middle-layer/src/middle_layer/driver_kit.ts` wraps the pFrame driver once.
+  Every join a model builds and every row it reads passes through it, so both
+  the creation calls and the data calls are covered without touching call sites.
+- `pl-middle-layer/src/js_render/index.ts` opens a render span per block, and one
+  per resumption of a deferred render. Driver calls carry no block identity of
+  their own, so the enclosing span is what attributes a join to a block.
+
+`executeSingleLambda` is deliberately not instrumented: it evaluates `args()`
+style lambdas with no computable context, so it reaches no driver.
+
+## Known gaps
+
+- The synchronous append cost is unmeasured on a real project; the sampler
+  interval and event granularity should be revisited with that number in hand.
+- A table view can die in the renderer process rather than in the middle-layer
+  worker. That path needs `render-process-gone` in the desktop app and is not
+  covered here.
+- Domain values are kept because they carry join identity. If any producer puts
+  a sample name in a domain value, it needs hashing too.

package/dist/analyze.d.ts

@@ -0,0 +1,133 @@
+import { CrashMarker, SessionEnvironment } from "./events.js";
+import { FindingSeverity } from "./rules.js";
+//#region src/analyze.d.ts
+/**
+ * Turns a flight log into an attributed cause.
+ *
+ * Three independent lines of evidence are combined: which operation was still
+ * running when the process died (an unmatched begin record), where memory
+ * actually went (resident growth across each completed operation, and which
+ * region grew — JS heap, off-heap buffers, or native), and what the join tree
+ * looked like before any data was touched. Any one alone is suggestive;
+ * together they name a specific call in a specific block.
+ */
+export declare const THRESHOLDS: {
+  /** Fraction of the heap ceiling above which the heap counts as exhausted. */
+  readonly heapPressure: 0.85;
+  readonly nativeGrowthBytes: number;
+  readonly amplification: 10;
+  readonly unboundedRows: 1000000;
+  readonly returnedBytes: number;
+  readonly inlineEntries: 1000000;
+  readonly stallMs: 2000;
+  /** Backward tolerance when matching a marker by time, for parent/worker clock drift. */
+  readonly clockToleranceMs: 1000;
+  /** How many open sessions are considered as rivals for an unattributed marker. */
+  readonly maxRivalSessions: 8;
+  /** A machine-memory claim needs the process to actually be large. */
+  readonly machineRssShare: 0.25;
+};
+export type Finding = {
+  rule: string;
+  severity: FindingSeverity;
+  detail: string;
+  seq?: number;
+  path?: string;
+  join?: string;
+  source?: string;
+  block?: string;
+};
+export type MemoryAnalysis = {
+  samplerPresent: boolean;
+  sampleCount: number;
+  rssSeries: {
+    wall: number;
+    rss?: number;
+    freeMemory?: number;
+  }[];
+  peakRss: number;
+  rssAtDeath?: number;
+  rssGrowth: number;
+  heapUsedAtDeath?: number;
+  heapLimit?: number;
+  heapPressure?: number;
+  heapGrowth?: number;
+  externalGrowth?: number;
+  arrayBuffersGrowth?: number;
+  worstStallMs: number;
+  freeMemoryAtDeath?: number;
+  totalMemory?: number;
+};
+export type OperationSummary = {
+  op: string;
+  seq: number;
+  wall: number;
+  info: Record<string, unknown>;
+  end?: Record<string, unknown>;
+  ms?: number;
+  failed?: boolean;
+  rssDelta?: number;
+  heapDelta?: number;
+};
+export type RenderSummary = {
+  seq: number;
+  blockId?: string;
+  block?: string;
+  key?: string;
+  end?: boolean;
+  failed?: boolean;
+  ms?: number;
+  stats?: {
+    serOutBytes?: number;
+    serInBytes?: number;
+    [key: string]: unknown;
+  };
+};
+export type Verdict = {
+  outcome: string;
+  peakRss: number;
+  where: string;
+  memoryRegion?: string;
+  likelyCause?: string;
+  summary: string;
+};
+export type SessionAnalysis = {
+  file: string;
+  sessionId: string;
+  crashed: boolean;
+  truncatedTail: boolean;
+  endedReason?: string;
+  crashMarker?: CrashMarker;
+  env?: SessionEnvironment;
+  role?: string;
+  meta?: Record<string, unknown>;
+  recordCount: number;
+  /**
+   * How many times the log rotated. Each rotation re-emits the session header,
+   * so environment and metadata survive, but operations older than the retained
+   * segments do not — which is why the count is reported rather than implied.
+   */
+  rotations: number;
+  memory: MemoryAnalysis;
+  /** Completed operations ranked by resident growth. */
+  attribution: OperationSummary[];
+  inFlight: OperationSummary[];
+  /** The innermost operation that started and never returned. */
+  inFlightAtDeath?: OperationSummary;
+  renders: RenderSummary[];
+  findings: Finding[];
+  verdict: Verdict;
+  timeline: Record<string, unknown>[];
+};
+/** Analyzes the newest crashed session in a directory, else the newest session. */
+export declare function analyzeLatest(dir: string, options?: {
+  preferCrashed?: boolean;
+}): SessionAnalysis | undefined;
+/** Analyzes one flight log, merging the sibling sampler series when present. */
+export declare function analyzeSession(file: string, dir?: string): SessionAnalysis;
+/** Thousands separators, or `unknown` when the count was never observed. */
+export declare function formatCount(value: number | undefined): string;
+/** Binary byte units, or `unknown`. */
+export declare function formatBytes(value: number | undefined): string;
+//#endregion
+//# sourceMappingURL=analyze.d.ts.map

package/dist/analyze.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"analyze.d.ts","names":[],"sources":["../src/analyze.ts"],"mappings":";;;;;;;;;;;;;qBAwBa;;WAEX;WACA;WACA;WACA;WACA;WACA;WACA;;WAEA;;WAEA;;WAEA;;YAGU;EACV;EACA,UAAU;EACV;EACA;EACA;EACA;EACA;EACA;;YAGU;EACV;EACA;EACA;IAAa;IAAc;IAAc;;EACzC;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;EACA;;YAGU;EACV;EACA;EACA;EACA,MAAM;EACN,MAAM;EACN;EACA;EACA;EACA;;YAGU;EACV;EACA;EACA;EACA;EACA;EACA;EACA;EACA;IAAU;IAAsB;KAAsB;;;YAG5C;EACV;EACA;EACA;EACA;EACA;EACA;;YAGU;EACV;EACA;EACA;EACA;EACA;EACA,cAAc;EACd,MAAM;EACN;EACA,OAAO;EACP;;;;;;EAMA;EACA,QAAQ;;EAER,aAAa;EACb,UAAU;;EAEV,kBAAkB;EAClB,SAAS;EACT,UAAU;EACV,SAAS;EACT,UAAU;;;wBAII,cACd,aACA;EAAW;IACV;;wBASa,eAAe,cAAc,eAAmC;;wBA+DhE,YAAY;;wBAKZ,YAAY"}