@wafertools/testdata-parser 0.12.0 → 0.14.0

package/README.md

@@ -4,7 +4,7 @@
 
 Rust/WASM parsers for semiconductor test data formats: **STDF**, **ATDF**, **CSV**, **JSON**, and **Parquet**. Compiled to a single WASM module via `wasm-bindgen`; the same Rust source also builds natively (used by [tsmap](https://github.com/wafertools/tsmap)'s Tauri backend).
 
-All formats parse to one shared shape (`ParsedStdf` / `ScanResult`) — there is no format-specific output type on the JS side.
+All formats parse to one shared shape (`ParsedStdf` / `ScanResult`) — there is no format-specific output type on the JS side. A full parse returns it as a compact columnar buffer, which `decodeParsed` (rows) or `decodeColumns` (columns) from `@wafertools/testdata-parser/columnar.js` turns into that shape.
 
 ## Install
 
@@ -18,10 +18,11 @@ The module must be initialized once before calling any parse function — it loa
 
 ```js
 import init, { parse_stdf } from '@wafertools/testdata-parser';
+import { decodeParsed } from '@wafertools/testdata-parser/columnar.js';
 
 await init(); // fetches testdata_parser_bg.wasm relative to the module URL
 const bytes = new Uint8Array(await file.arrayBuffer());
-const parsed = parse_stdf(bytes); // ParsedStdf, or throws a ParserError
+const parsed = decodeParsed(parse_stdf(bytes)); // ParsedStdf; parse_stdf throws a ParserError
 ```
 
 **Import `init` as the default export, not by name.** There is also a named `init`
@@ -34,7 +35,7 @@ code, display the message — messages are prose and may be reworded.
 
 ```js
 try {
-  const parsed = parse_stdf(bytes);
+  const parsed = decodeParsed(parse_stdf(bytes));
 } catch (err) {
   if (err.code === 'not-stdf') {  // the file is not the format its name claims
     // ...offer to try another parser
@@ -69,23 +70,44 @@ and this README to each other.
 
 ## API
 
-Every parse function takes raw file bytes (`Uint8Array`) and returns a plain JS object (via `serde-wasm-bindgen`), or throws a `ParserError` (see above). Gzip-compressed input (`.gz`) is transparently decompressed for every format.
+Every parse function takes raw file bytes (`Uint8Array`), or throws a `ParserError` (see above). The full parses (`parse_*`) return a columnar buffer (`Uint8Array`) to decode as below; the scans and header reads return a plain JS object (via `serde-wasm-bindgen`). Gzip-compressed input (`.gz`) is transparently decompressed for every format.
 
 | Function | Signature | Returns |
 | --- | --- | --- |
-| `parse_stdf` | `(bytes: Uint8Array) => ParsedStdf` | Full parse of an STDF file |
-| `parse_atdf` | `(bytes: Uint8Array) => ParsedStdf` | Full parse of an ATDF file |
-| `parse_csv` | `(bytes: Uint8Array, mapping: CsvMapping) => ParsedStdf` | Full parse of a CSV, using an explicit column mapping |
-| `parse_json` | `(bytes: Uint8Array, mapping: CsvMapping) => ParsedStdf` | Full parse of a JSON array-of-records file, using the same mapping shape as CSV |
+| `parse_stdf` | `(bytes: Uint8Array) => Uint8Array` | Full parse of an STDF file, as a columnar buffer |
+| `parse_atdf` | `(bytes: Uint8Array) => Uint8Array` | Full parse of an ATDF file, as a columnar buffer |
+| `parse_csv` | `(bytes: Uint8Array, mapping: CsvMapping) => Uint8Array` | Full parse of a CSV, using an explicit column mapping |
+| `parse_json` | `(bytes: Uint8Array, mapping: CsvMapping) => Uint8Array` | Full parse of a JSON array-of-records file, using the same mapping shape as CSV |
 | `parquet_headers` | `(bytes: Uint8Array) => ParquetHeadersResult` | Schema + a sample of rows, for a column-mapping UI (see below — unlike CSV/JSON, this one *is* a WASM export) |
-| `parse_parquet` | `(bytes: Uint8Array, mapping: CsvMapping) => ParsedStdf` | Full parse of a Parquet file, using the same mapping shape as CSV/JSON |
+| `parse_parquet` | `(bytes: Uint8Array, mapping: CsvMapping) => Uint8Array` | Full parse of a Parquet file, using the same mapping shape as CSV/JSON |
 | `stdf_test_names` | `(bytes: Uint8Array) => ScanResult` | Fast first-pass scan: test definitions + die count, no die accumulation |
 | `atdf_test_names` | `(bytes: Uint8Array) => ScanResult` | Same first-pass scan for ATDF |
 | `stdf_file_meta` | `(bytes: Uint8Array) => FileMeta` | Lot metadata, wafer count, first/last timestamps and site count from an MIR/SDR/WIR/WRR-only scan — no PTR/FTR/PIR/PRR walk, so it stays cheap across a batch of files |
 | `atdf_file_meta` | `(bytes: Uint8Array) => FileMeta` | Same metadata-only scan for ATDF |
 | `parquet_distinct_count` | `(bytes: Uint8Array, columns: string[]) => number` | How many distinct combinations of those columns the file holds — a wafer count from `['lot','wafer']` without a full parse, read as a column projection. A column missing from the schema is an error, not a count of zero. Parquet only: CSV/JSON have no equivalent shortcut |
-| `parse_stdf_filtered` | `(bytes: Uint8Array, selected: number[]) => ParsedStdf` | Full parse, skipping per-site accumulation for test numbers not in `selected` |
-| `parse_atdf_filtered` | `(bytes: Uint8Array, selected: number[]) => ParsedStdf` | Same filtered parse for ATDF |
+| `parse_stdf_filtered` | `(bytes: Uint8Array, selected: number[]) => Uint8Array` | Full parse, skipping per-site accumulation for test numbers not in `selected` |
+| `parse_atdf_filtered` | `(bytes: Uint8Array, selected: number[]) => Uint8Array` | Same filtered parse for ATDF |
+
+### Decoding a parse: `decodeParsed` and `decodeColumns`
+
+A full parse returns one `Uint8Array`: typed columns, one per field and one per test, followed by
+a JSON header (everything in `ParsedStdf` except the dies) and its length. It is small, and it transfers from a
+Worker without a copy (`postMessage(buf, [buf.buffer])`). Decode it with one of two functions
+from `@wafertools/testdata-parser/columnar.js`, which ships in the package:
+
+- **`decodeParsed(buf)`** gives the `ParsedStdf` described below, with one `DieResult` object
+  per die.
+- **`decodeColumns(buf)`** gives the same `ParsedStdf`, except that each wafer's `results`
+  stays columns: positions, bins and site as typed arrays (−32768 = no position, 65535 = no
+  bin or site), and test values and verdicts as `{ indices, values }` per test number (only the
+  records that have one). That is the shape `@wafertools/wafermap`'s `buildWaferMap` accepts as
+  `results` (`DieColumns`), so a large lot is built without an object per die: pass
+  `wafer.results` as it is.
+  Within a wafer, tests with values (or verdicts) on the same records share one `indices`
+  array, so treat the arrays as read-only.
+
+Test values are sent as 32-bit floats when every value in the column is exactly representable
+as one (STDF readings are), and as 64-bit otherwise, so no value changes.
 
 ### Two-pass parsing (STDF/ATDF)
 
@@ -116,8 +138,10 @@ interface CsvMapping {
   testnameCol?: string;       // for "tall" CSVs: column holding the test name per row
   testnumberCol?: string;     // for "tall" CSVs: column holding the test's real number per row
   testvalueCol?: string;      // for "tall" CSVs: column holding the test value per row
-  loLimitCol?: string;
+  loLimitCol?: string;        // for "tall" CSVs: test limits (STDF LO_LIMIT/HI_LIMIT) per row
   hiLimitCol?: string;
+  loSpecCol?: string;         // for "tall" CSVs: spec limits (STDF LO_SPEC/HI_SPEC) per row — a separate pair
+  hiSpecCol?: string;
   unitsCol?: string;
   passBins: number[];         // hbin/sbin values treated as a pass for pass/fail summary
 }
@@ -220,11 +244,13 @@ type ParserWarningCode =
   | 'unpositioned-dies'         // dies with no X/Y — real data, not placeable
   | 'bin-invalid'               // a bin outside STDF's 0–32767, or a missing hard bin
   | 'coordinate-invalid'        // an X/Y outside STDF's -32767..32767
+  | 'site-invalid'              // a site number outside STDF's 0–255 (flat formats)
   | 'result-unusable'           // results the tester flagged unusable (value left out, verdict kept)
   | 'records-not-read'          // records this parser does not read yet (MPR)
   | 'wafer-end-missing'         // a wafer had no WRR; closed at the next wafer or end of file
   | 'file-truncated'            // the file ends part-way through a record
   | 'record-malformed'          // a PRR too short to hold its required fields
+  | 'test-number-invalid'       // an ATDF test number that is not a u32 (record left out)
   | 'values-not-numeric'        // a mapped column held values that would not coerce
   | 'retests-assumed'           // repeated positions read as retests
   | 'wafer-split-by-column'     // one wafer per value of a mapped column
@@ -262,7 +288,7 @@ an empty array, which asserts that nothing passes.
 **`warnings` carries a stable `code`, prose, and a severity** — branch on the code, display
 the message, and never match on the prose. `severity: 'error'` means a number or a plot
 built from this result can mislead, because data was dropped or a value was substituted
-(`unpositioned-dies`, `bin-invalid`, `coordinate-invalid`, `record-malformed`, `records-not-read`, `values-not-numeric`); `'warning'` means the
+(`unpositioned-dies`, `bin-invalid`, `coordinate-invalid`, `site-invalid`, `record-malformed`, `records-not-read`, `test-number-invalid`, `values-not-numeric`); `'warning'` means the
 parse applied a documented rule or interpretation — one you may want to change, or, like
 `result-unusable`, the spec's own rule for leaving out values the tester flagged — and the
 result means what the file says. Nothing here is fatal — the parse succeeded. Surface them: a silently discarded
@@ -375,6 +401,15 @@ pub struct CsvHeadersResult {
 - **Byte readers are panic-free.** STDF/ATDF field readers are bounds-checked and return `Option`/`Result` rather than panicking on truncated input — a panic inside WASM aborts the whole module with no recovery, so this is a hard requirement, not a style preference.
 - **Big-endian and little-endian STDF** are both supported (detected from the FAR record's `CPU_TYPE`).
 - **Gzip is transparent** — every entry point decompresses `.gz` input automatically by sniffing the magic bytes; callers don't need to branch on compression.
+- **MPR records are not read yet.** A multiple-result parametric record (STDF `MPR`, ATDF `MPR:`)
+  carries several results for one test; both parsers skip them — in the full parse and the
+  test-name scan alike — and count them in a `records-not-read` warning, so the missing tests are
+  never silent. PTR and FTR records are read in full.
+- **Every format applies STDF V4's value ranges.** A bin, coordinate or site number STDF cannot
+  store, or a test value that is not finite, is missing in CSV, JSON and Parquet exactly as in
+  STDF and ATDF, and is reported under the same codes (`bin-invalid`, `coordinate-invalid`,
+  `site-invalid`, `result-unusable`). The rule lives in one place (`SpecCheck` in `types.rs`),
+  which the flat formats reach through `flat_wafers::into_parsed`.
 - **CSV/JSON/Parquet test numbers fall back to a deterministic hash only when the file itself carries no real one.** Neither format has a *mandatory* STDF-style test number the way STDF/ATDF do, but a real one is used whenever the source data has it — see "Test identity — real number vs. synthesized one" above for the wide/tall rules. Hashing is the fallback, not the default: it fires per test only when no real number was mapped or the mapped column's value didn't parse (`test_identity::stable_test_number`, FNV-1a with a fixed seed and a reserved floor, collision-probed so two tests in one file can never collide, and never colliding with a genuine numeric-header/`testnumberCol` value either). Deliberately not sequential/encounter-order: a hash means the number for a given test doesn't change if the file is reordered or a column is added — a *hashed* number is otherwise meaningless and callers should never rely on its value, only on it being stable and unique within one parse. `order` (see `TestDef` above) carries the file's own display order instead.
 - **Parquet reads through a row-oriented API, not Arrow.** `parquet::record::Row`/`Field` rather than the `arrow` feature — a closer fit for this crate's row-based `DieResult` model, and a smaller WASM bundle (no Arrow array machinery pulled in). A typed Parquet cell is coerced to `f64` for numeric roles and to a plain string otherwise; a value that fails to coerce (e.g. a numeric role mapped to a genuinely string-typed column) is skipped and surfaced as one summarised entry in `warnings`, not a panic or a silent zero.
 - **Parquet's `zstd` codec is native-only.** `snappy`, `gzip`, `lz4`, and `brotli` build for `wasm32-unknown-unknown` with no extra toolchain; `zstd`'s C library needs a real C cross-compiler targeting wasm32, which a plain `wasm-pack build` doesn't assume is available. A `zstd`-compressed Parquet file parses natively but fails clearly on the WASM build.

package/columnar.d.ts

@@ -0,0 +1,42 @@
+import type { ParsedStdf } from './testdata_parser.js';
+
+/**
+ * Turn the buffer a parse function returns (`parse_stdf`, `parse_csv`, …) into a
+ * `ParsedStdf`. Accepts the `Uint8Array` the function returned, or the
+ * `ArrayBuffer` it was transferred or delivered as.
+ */
+export function decodeParsed(input: Uint8Array | ArrayBuffer): ParsedStdf;
+
+/** A wafer's records as columns: the shape `@wafertools/wafermap` accepts as `results` (`DieColumns`). */
+export interface ParsedColumns {
+  /** Number of records. */
+  count: number;
+  /** Per record; −32768 = no position. */
+  x?: Int32Array;
+  y?: Int32Array;
+  /** Per record; 65535 = none. */
+  hbin?: Uint32Array;
+  sbin?: Uint32Array;
+  siteNum?: Uint32Array;
+  partId?: Array<number | string | undefined>;
+  supersedes?: Array<'partId' | 'position' | undefined>;
+  /** Per test number: the records that have a value, and their values. */
+  testValues?: Record<number, { indices: Int32Array; values: Float32Array | Float64Array }>;
+  /** Per test number: the records that have a recorded verdict; 1 = pass, 0 = fail. */
+  testPass?: Record<number, { indices: Int32Array; values: Int8Array }>;
+}
+
+/** A `ParsedStdf` whose wafers carry their records as columns. */
+export type ParsedStdfColumns = Omit<ParsedStdf, 'wafers'> & {
+  wafers: Array<Omit<ParsedStdf['wafers'][number], 'results'> & { results: ParsedColumns }>;
+};
+
+/**
+ * Like `decodeParsed`, but each wafer's `results` stays columns, with no object
+ * per die: positions, bins and site as typed arrays with STDF's missing values,
+ * test values and verdicts sparse. Pass a wafer's `results` straight to
+ * `buildWaferMap`. Nothing refers to the input buffer afterwards. Within a
+ * wafer, columns with the same records present share one `indices` array:
+ * treat them as read-only.
+ */
+export function decodeColumns(input: Uint8Array | ArrayBuffer): ParsedStdfColumns;

package/columnar.js

@@ -0,0 +1,256 @@
+// Decodes the columnar buffer the parse functions return into a `ParsedStdf`.
+// The encoder, and the full layout, is `src/columnar.rs`; the two ship together
+// in this package so they cannot drift apart.
+
+/** The format this decoder reads — `FORMAT` in `src/columnar.rs`. */
+const FORMAT = 2;
+
+const MISSING_I32 = -2147483648;
+const MISSING_U32 = 4294967295;
+const SUPERSEDES = [undefined, 'partId', 'position'];
+
+/**
+ * The buffer's header, and each wafer's columns as typed-array views over the
+ * buffer (no copy). Shared by `decodeParsed` and `decodeColumns`.
+ */
+function readColumns(input) {
+  let bytes = input instanceof ArrayBuffer ? new Uint8Array(input) : input;
+  // Typed-array views need aligned offsets; a buffer that is itself a view at an
+  // odd offset is copied once so every column below can be viewed in place.
+  if (bytes.byteOffset % 8 !== 0) bytes = bytes.slice();
+  const { buffer, byteOffset } = bytes;
+
+  // Columns first, then the JSON header, then its length: see `src/columnar.rs`.
+  const headerLen = new DataView(buffer, byteOffset + bytes.length - 4, 4).getUint32(0, true);
+  const header = JSON.parse(new TextDecoder().decode(bytes.subarray(bytes.length - 4 - headerLen, bytes.length - 4)));
+  if (header.columnarFormat !== FORMAT) {
+    throw new Error(`testdata-parser: columnar format ${header.columnarFormat} is not the format ${FORMAT} this decoder reads — the parser and its decoder come from different versions`);
+  }
+  delete header.columnarFormat;
+  const body = byteOffset;
+  checkLayout(header.wafers, bytes.length - 4 - headerLen);
+
+  const wafers = header.wafers.map((wafer) => {
+    const n = wafer.dieCount;
+    const view = (c) => {
+      const at = body + c.offset;
+      switch (c.kind) {
+        case 'i32': return new Int32Array(buffer, at, n);
+        case 'u32': return new Uint32Array(buffer, at, n);
+        case 'u8':  return new Uint8Array(buffer, at, n);
+        case 'i8':  return new Int8Array(buffer, at, n);
+        case 'f32': return new Float32Array(buffer, at, n);
+        case 'f64': return new Float64Array(buffer, at, n);
+        default: throw new Error(`testdata-parser: unknown column kind ${c.kind}`);
+      }
+    };
+    const col = {};
+    const values = [];
+    const verdicts = [];
+    for (const c of wafer.columns) {
+      if (c.field === 'testValues') values.push([c.test, view(c)]);
+      else if (c.field === 'testPass') verdicts.push([c.test, view(c)]);
+      else col[c.field] = view(c);
+    }
+    const partIds = wafer.partIds;
+    delete wafer.dieCount;
+    delete wafer.columns;
+    delete wafer.partIds;
+    return { wafer, n, col, values, verdicts, partIds };
+  });
+  return { header, wafers };
+}
+
+const WIDTH = { i32: 4, u32: 4, f32: 4, f64: 8, u8: 1, i8: 1 };
+
+/**
+ * Every column must lie inside the body (before the header) and no two may
+ * share bytes. A typed array only refuses a column that runs off the buffer; one
+ * at a wrong offset inside it would decode as plausible, wrong data.
+ */
+function checkLayout(wafers, bodyLen) {
+  if (!(bodyLen >= 0)) throw new Error('testdata-parser: columnar header is longer than the buffer');
+  const spans = [];
+  for (const wafer of wafers) {
+    for (const c of wafer.columns) {
+      const width = WIDTH[c.kind];
+      if (width === undefined) throw new Error(`testdata-parser: unknown column kind ${c.kind}`);
+      const end = c.offset + wafer.dieCount * width;
+      if (!Number.isInteger(c.offset) || c.offset < 0 || end > bodyLen) {
+        throw new Error(`testdata-parser: column ${c.field}${c.test === undefined ? '' : ` ${c.test}`} lies outside the column data`);
+      }
+      spans.push([c.offset, end]);
+    }
+  }
+  spans.sort((a, b) => a[0] - b[0]);
+  for (let i = 1; i < spans.length; i++) {
+    if (spans[i][0] < spans[i - 1][1]) throw new Error('testdata-parser: two columns share bytes — the buffer is corrupt');
+  }
+}
+
+/**
+ * @param {Uint8Array | ArrayBuffer} input
+ * @returns {import('./testdata_parser.js').ParsedStdf}
+ */
+export function decodeParsed(input) {
+  const { header, wafers } = readColumns(input);
+  for (const { wafer, n, col, values, verdicts, partIds } of wafers) {
+    const { x, y, dieIndex, hbin, sbin, siteNum, supersedes } = col;
+
+    // Fields assigned in one fixed order, so every die shares a shape.
+    const results = new Array(n);
+    for (let i = 0; i < n; i++) {
+      const d = {};
+      if (x && x[i] !== MISSING_I32) d.x = x[i];
+      if (y && y[i] !== MISSING_I32) d.y = y[i];
+      if (dieIndex && dieIndex[i] !== MISSING_U32) d.dieIndex = dieIndex[i];
+      if (hbin && hbin[i] !== MISSING_U32) d.hbin = hbin[i];
+      if (sbin && sbin[i] !== MISSING_U32) d.sbin = sbin[i];
+      if (siteNum && siteNum[i] !== MISSING_U32) d.siteNum = siteNum[i];
+      if (partIds && partIds[i] != null) d.partId = partIds[i];
+      if (supersedes && supersedes[i] !== 0) d.supersedes = SUPERSEDES[supersedes[i]];
+      results[i] = d;
+    }
+    // Highest test number first. V8 picks an object's element storage from the
+    // first integer-like key it receives: a small one (1001 is below its
+    // 1024-slot gap limit) gets a holey array sized to the largest key, ~6 KB
+    // for 50 readings at 1001–1050; a large one gets a dictionary, ~1.5 KB.
+    // Enumeration order is unaffected — integer-like keys always enumerate
+    // ascending — so this changes memory only.
+    const byKeyDesc = (a, b) => Number(b[0]) - Number(a[0]);
+    values.sort(byKeyDesc);
+    verdicts.sort(byKeyDesc);
+    attachValues(results, values);
+    attachVerdicts(results, verdicts);
+    wafer.results = results;
+  }
+  return header;
+}
+
+/**
+ * Like `decodeParsed`, but each wafer's `results` stays columns: the shape
+ * `@wafertools/wafermap`'s `buildWaferMap` accepts as `results` (`DieColumns`),
+ * with no object per die. Positions, bins and site are per-record typed
+ * arrays with STDF's missing values (−32768 for coordinates, 65535 for bins and
+ * site); test values and verdicts are sparse (`indices` of the records that
+ * have one, and the `values`), values at the precision they were sent.
+ *
+ * Nothing refers to the input buffer afterwards, so it can be freed.
+ *
+ * @param {Uint8Array | ArrayBuffer} input
+ */
+export function decodeColumns(input) {
+  const { header, wafers } = readColumns(input);
+  for (const { wafer, n, col, values, verdicts, partIds } of wafers) {
+    const results = { count: n };
+    if (col.x) results.x = stdfInts(col.x, MISSING_I32, COORD_MISSING, Int32Array);
+    if (col.y) results.y = stdfInts(col.y, MISSING_I32, COORD_MISSING, Int32Array);
+    if (col.hbin) results.hbin = stdfInts(col.hbin, MISSING_U32, U16_MISSING, Uint32Array);
+    if (col.sbin) results.sbin = stdfInts(col.sbin, MISSING_U32, U16_MISSING, Uint32Array);
+    if (col.siteNum) results.siteNum = stdfInts(col.siteNum, MISSING_U32, U16_MISSING, Uint32Array);
+    if (partIds) results.partId = partIds.map(v => (v == null ? undefined : v));
+    if (col.supersedes) results.supersedes = Array.from(col.supersedes, c => SUPERSEDES[c]);
+    const shared = sharedIndices(n);
+    if (values.length) {
+      results.testValues = {};
+      for (const [key, dense] of values) results.testValues[key] = sparse(dense, v => v === v, shared);  // NaN is missing
+    }
+    if (verdicts.length) {
+      results.testPass = {};
+      for (const [key, dense] of verdicts) results.testPass[key] = sparse(dense, v => v !== -1, shared);
+    }
+    wafer.results = results;
+  }
+  return header;
+}
+
+const COORD_MISSING = -32768;
+const U16_MISSING = 65535;
+
+/** A copy of an integer column with the wire's missing value replaced by STDF's. */
+function stdfInts(column, wireMissing, stdfMissing, Type) {
+  const out = new Type(column.length);
+  for (let i = 0; i < column.length; i++) out[i] = column[i] === wireMissing ? stdfMissing : column[i];
+  return out;
+}
+
+/**
+ * A dense column as `{ indices, values }` of the entries `present` accepts,
+ * values in the column's own type. Columns with the same entries present share
+ * one `indices` array (from `shared`): in most lots every die has every test,
+ * so a wafer's columns all hold the same indices, and one copy of them about halves
+ * the memory the columns take. Nothing writes to an `indices` array.
+ */
+function sparse(dense, present, shared) {
+  const scratch = shared.scratch;
+  let k = 0;
+  let hash = 0;
+  for (let i = 0; i < dense.length; i++) {
+    if (present(dense[i])) { scratch[k++] = i; hash = (Math.imul(hash, 31) + i) | 0; }
+  }
+  const values = new dense.constructor(k);
+  for (let j = 0; j < k; j++) values[j] = dense[scratch[j]];
+  return { indices: shared.get(k, hash), values };
+}
+
+/**
+ * The index arrays of one wafer's columns: `get` returns an earlier array
+ * holding the same indices as `scratch[0..count)`, or a copy of them.
+ */
+function sharedIndices(n) {
+  const scratch = new Int32Array(n);
+  const seen = new Map();  // `${count}:${hash}` → arrays with that count and hash
+  return {
+    scratch,
+    get(count, hash) {
+      const key = `${count}:${hash}`;
+      const candidates = seen.get(key);
+      if (candidates) {
+        for (const c of candidates) {
+          let same = true;
+          for (let j = 0; j < count; j++) if (c[j] !== scratch[j]) { same = false; break; }
+          if (same) return c;
+        }
+      }
+      const indices = scratch.slice(0, count);
+      if (candidates) candidates.push(indices); else seen.set(key, [indices]);
+      return indices;
+    },
+  };
+}
+
+// Each die's map is built in one go, not a column at a time reaching back into
+// every die once per test, and in functions of their own: as one long loop
+// inside `decodeParsed`, WebKit's engine (the desktop app on Linux and macOS)
+// left the work in its slowest tier and it took ~7 s on a 266k-die lot.
+
+/** A test key as the property key to write: its number when it is one, so no string is parsed per die. */
+const propKey = (key) => (String(Number(key)) === key ? Number(key) : key);
+
+function attachValues(results, columns) {
+  const keys = columns.map(([key]) => propKey(key));
+  const cols = columns.map(([, v]) => v);
+  const t = cols.length;
+  for (let i = 0; i < results.length; i++) {
+    let map;
+    for (let j = 0; j < t; j++) {
+      const value = cols[j][i];
+      if (value === value) (map ??= {})[keys[j]] = value;  // NaN is missing
+    }
+    if (map) results[i].testValues = map;
+  }
+}
+
+function attachVerdicts(results, columns) {
+  const keys = columns.map(([key]) => propKey(key));
+  const cols = columns.map(([, v]) => v);
+  const t = cols.length;
+  for (let i = 0; i < results.length; i++) {
+    let map;
+    for (let j = 0; j < t; j++) {
+      const verdict = cols[j][i];
+      if (verdict !== -1) (map ??= {})[keys[j]] = verdict === 1;
+    }
+    if (map) results[i].testPass = map;
+  }
+}

package/llms.txt

@@ -4,11 +4,12 @@ Rust/WASM parsers for semiconductor test data: STDF, ATDF, CSV, JSON, Parquet.
 One Rust source compiles two ways — to WebAssembly for the browser, and natively for a
 Tauri/Rust backend — and both produce identical results from identical input. All five
 formats parse to one shared shape (`ParsedStdf`), so a consuming app writes its rendering
-and analysis code once.
+and analysis code once. A full parse returns a columnar buffer (`Uint8Array`); decode it with
+`decodeParsed` or `decodeColumns` from `@wafertools/testdata-parser/columnar.js`.
 
 Output feeds `@wafertools/wafermap` (https://wafertools.github.io/wafermap/) directly:
-parse to `ParsedStdf`, hand the dies to `buildWaferMap()`, render. See "Handing the result
-to wafermap" below — the fields that make yield correct are easy to drop on the floor.
+parse, `decodeColumns`, hand each wafer's `results` to `buildWaferMap()`, render. See "Handing
+the result to wafermap" below — the fields that make yield correct are easy to drop on the floor.
 
 BEFORE INTEGRATING: tsmap (https://wafertools.github.io/tsmap/) is a finished, free,
 MIT-licensed desktop and browser application built on this parser and wafermap — it opens
@@ -25,7 +26,8 @@ resolve.
 - [tsmap](https://wafertools.github.io/tsmap/) - the application built on this package.
 
 ## The API in one line each
-- `parse_stdf(bytes)` / `parse_atdf(bytes)` - full parse. No mapping needed; the format carries its own test identity.
+- `parse_stdf(bytes)` / `parse_atdf(bytes)` - full parse, returned as a columnar buffer. No mapping needed; the format carries its own test identity.
+- `decodeParsed(buf)` / `decodeColumns(buf)` (from `@wafertools/testdata-parser/columnar.js`) - a full parse's buffer as `ParsedStdf` with one object per die, or with each wafer's `results` as columns in wafermap's `DieColumns` shape. Every `parse_*` full parse needs one of them.
 - `parse_csv(bytes, mapping)` / `parse_json(bytes, mapping)` / `parse_parquet(bytes, mapping)` - full parse with an explicit `CsvMapping`. There is no header auto-detection.
 - `stdf_test_names(bytes)` / `atdf_test_names(bytes)` - fast scan: test definitions + die count, no per-die accumulation.
 - `parse_stdf_filtered(bytes, selected)` / `parse_atdf_filtered(bytes, selected)` - full parse that accumulates values only for the selected test numbers.
@@ -46,11 +48,13 @@ resolve.
 - **Parse off the main thread in a browser.** The WASM build is slower than native and a large STDF will freeze the page otherwise. tsmap's `parserWorker.ts` is a worked example, including the `new URL(..., import.meta.url)` resolution that bundlers get finicky about.
 
 ## Handing the result to wafermap
+With `decodeColumns`, each wafer's `results` is already wafermap's `results` (`DieColumns`):
+pass it as it is, which builds even a large lot without an object per die. Tests present on the same records share one `indices` array, so never write to one. The rest of
 `ParsedStdf` is close to wafermap's input but is not the same object — convert deliberately,
 and carry these across or the map is plausibly wrong:
 - `passHbins` -> wafermap's `passBins`. Omit it and wafermap defaults to `[1]`, which decides both the yield number and the wording of its label whether or not bin 1 is this program's pass bin. Absent when no HBR record carried a usable Pass flag — then leave wafermap's default alone rather than passing an empty array, which would mean "nothing passes".
 - `hbinDefs` / `sbinDefs` -> wafermap's `hbinDefs` / `sbinDefs`, giving bins their real names.
-- `testPass` -> wafermap's `die.testPass`: recorded per-test verdicts, `true` = pass. Functional (FTR) tests live here only — they have no measured value. Read them through wafermap's `getTestPassStatus()`; a missing verdict is no-data, never a fail.
+- `testPass` (in `results`, rows or columns) -> wafermap's `die.testPass`: recorded per-test verdicts, pass = `true` (`1` in columns). Functional (FTR) tests live here only — they have no measured value. Read them through wafermap's `getTestPassStatus()`; a missing verdict is no-data, never a fail.
 - `testDefs` is a `Record<string, TestDef>` keyed by test number as a string, and `TestDef` here has no `testNumber` field. wafermap wants an array of `TestDef` each carrying a numeric `testNumber`, so the key becomes the field during conversion.
 - `hbin` / `sbin` are already numbers — never `?? 0` them. A missing bin is not bin 0.
 - A parse `warning` with `severity: 'error'` is worth showing next to the map, not just logging: `unpositioned-dies` means the map holds fewer dies than the file, and `soft-bin-mirrored` means a soft-bin map shows numbers the file never stated.

package/package.json

@@ -2,7 +2,7 @@
   "name": "@wafertools/testdata-parser",
   "type": "module",
   "description": "Rust/WASM parsers for semiconductor test data formats (STDF, ATDF, CSV, JSON, Parquet)",
-  "version": "0.12.0",
+  "version": "0.14.0",
   "license": "MIT",
   "repository": {
     "type": "git",
@@ -12,7 +12,9 @@
     "testdata_parser_bg.wasm",
     "testdata_parser.js",
     "testdata_parser.d.ts",
-    "llms.txt"
+    "llms.txt",
+    "columnar.js",
+    "columnar.d.ts"
   ],
   "main": "testdata_parser.js",
   "types": "testdata_parser.d.ts",

package/testdata_parser.d.ts

@@ -87,11 +87,13 @@ export type ParserWarningCode =
 | "unpositioned-dies"
 | "bin-invalid"
 | "coordinate-invalid"
+| "site-invalid"
 | "result-unusable"
 | "records-not-read"
 | "wafer-end-missing"
 | "file-truncated"
 | "record-malformed"
+| "test-number-invalid"
 | "values-not-numeric"
 | "retests-assumed"
 | "wafer-split-by-column"
@@ -193,8 +195,13 @@ export interface CsvMapping {
     testnumberCol?: string | null;
     /** Tall format: the column holding each row's measured value. */
     testvalueCol?: string | null;
+    /** Tall format: the columns holding each test's test limits (STDF LO_LIMIT/HI_LIMIT). */
     loLimitCol?: string | null;
     hiLimitCol?: string | null;
+    /** Tall format: the columns holding each test's spec limits (STDF LO_SPEC/HI_SPEC) —
+     *  a separate pair from the test limits, never mixed with them. */
+    loSpecCol?: string | null;
+    hiSpecCol?: string | null;
     unitsCol?: string | null;
     /** Bins counted as a pass in this file's own pass/fail summary. */
     passBins: number[];
@@ -237,19 +244,19 @@ export function parquet_distinct_count(bytes: Uint8Array, columns: string[]): nu
 
 export function parquet_headers(bytes: Uint8Array): ParquetHeadersResult;
 
-export function parse_atdf(bytes: Uint8Array): ParsedStdf;
+export function parse_atdf(bytes: Uint8Array): Uint8Array;
 
-export function parse_atdf_filtered(bytes: Uint8Array, selected: number[]): ParsedStdf;
+export function parse_atdf_filtered(bytes: Uint8Array, selected: number[]): Uint8Array;
 
-export function parse_csv(bytes: Uint8Array, mapping: CsvMapping): ParsedStdf;
+export function parse_csv(bytes: Uint8Array, mapping: CsvMapping): Uint8Array;
 
-export function parse_json(bytes: Uint8Array, mapping: CsvMapping): ParsedStdf;
+export function parse_json(bytes: Uint8Array, mapping: CsvMapping): Uint8Array;
 
-export function parse_parquet(bytes: Uint8Array, mapping: CsvMapping): ParsedStdf;
+export function parse_parquet(bytes: Uint8Array, mapping: CsvMapping): Uint8Array;
 
-export function parse_stdf(bytes: Uint8Array): ParsedStdf;
+export function parse_stdf(bytes: Uint8Array): Uint8Array;
 
-export function parse_stdf_filtered(bytes: Uint8Array, selected: number[]): ParsedStdf;
+export function parse_stdf_filtered(bytes: Uint8Array, selected: number[]): Uint8Array;
 
 export function stdf_file_meta(bytes: Uint8Array): FileMeta;
 
@@ -264,13 +271,13 @@ export interface InitOutput {
     readonly init: () => void;
     readonly parquet_distinct_count: (a: number, b: number, c: any) => [number, number, number];
     readonly parquet_headers: (a: number, b: number) => [number, number, number];
-    readonly parse_atdf: (a: number, b: number) => [number, number, number];
-    readonly parse_atdf_filtered: (a: number, b: number, c: any) => [number, number, number];
-    readonly parse_csv: (a: number, b: number, c: any) => [number, number, number];
-    readonly parse_json: (a: number, b: number, c: any) => [number, number, number];
-    readonly parse_parquet: (a: number, b: number, c: any) => [number, number, number];
-    readonly parse_stdf: (a: number, b: number) => [number, number, number];
-    readonly parse_stdf_filtered: (a: number, b: number, c: any) => [number, number, number];
+    readonly parse_atdf: (a: number, b: number) => [number, number, number, number];
+    readonly parse_atdf_filtered: (a: number, b: number, c: any) => [number, number, number, number];
+    readonly parse_csv: (a: number, b: number, c: any) => [number, number, number, number];
+    readonly parse_json: (a: number, b: number, c: any) => [number, number, number, number];
+    readonly parse_parquet: (a: number, b: number, c: any) => [number, number, number, number];
+    readonly parse_stdf: (a: number, b: number) => [number, number, number, number];
+    readonly parse_stdf_filtered: (a: number, b: number, c: any) => [number, number, number, number];
     readonly stdf_file_meta: (a: number, b: number) => [number, number, number];
     readonly stdf_test_names: (a: number, b: number) => [number, number, number];
     readonly __wbindgen_malloc: (a: number, b: number) => number;

package/testdata_parser.js

@@ -64,105 +64,119 @@ export function parquet_headers(bytes) {
 
 /**
  * @param {Uint8Array} bytes
- * @returns {ParsedStdf}
+ * @returns {Uint8Array}
  */
 export function parse_atdf(bytes) {
     const ptr0 = passArray8ToWasm0(bytes, wasm.__wbindgen_malloc);
     const len0 = WASM_VECTOR_LEN;
     const ret = wasm.parse_atdf(ptr0, len0);
-    if (ret[2]) {
-        throw takeFromExternrefTable0(ret[1]);
+    if (ret[3]) {
+        throw takeFromExternrefTable0(ret[2]);
     }
-    return takeFromExternrefTable0(ret[0]);
+    var v2 = getArrayU8FromWasm0(ret[0], ret[1]).slice();
+    wasm.__wbindgen_free(ret[0], ret[1] * 1, 1);
+    return v2;
 }
 
 /**
  * @param {Uint8Array} bytes
  * @param {number[]} selected
- * @returns {ParsedStdf}
+ * @returns {Uint8Array}
  */
 export function parse_atdf_filtered(bytes, selected) {
     const ptr0 = passArray8ToWasm0(bytes, wasm.__wbindgen_malloc);
     const len0 = WASM_VECTOR_LEN;
     const ret = wasm.parse_atdf_filtered(ptr0, len0, selected);
-    if (ret[2]) {
-        throw takeFromExternrefTable0(ret[1]);
+    if (ret[3]) {
+        throw takeFromExternrefTable0(ret[2]);
     }
-    return takeFromExternrefTable0(ret[0]);
+    var v2 = getArrayU8FromWasm0(ret[0], ret[1]).slice();
+    wasm.__wbindgen_free(ret[0], ret[1] * 1, 1);
+    return v2;
 }
 
 /**
  * @param {Uint8Array} bytes
  * @param {CsvMapping} mapping
- * @returns {ParsedStdf}
+ * @returns {Uint8Array}
  */
 export function parse_csv(bytes, mapping) {
     const ptr0 = passArray8ToWasm0(bytes, wasm.__wbindgen_malloc);
     const len0 = WASM_VECTOR_LEN;
     const ret = wasm.parse_csv(ptr0, len0, mapping);
-    if (ret[2]) {
-        throw takeFromExternrefTable0(ret[1]);
+    if (ret[3]) {
+        throw takeFromExternrefTable0(ret[2]);
     }
-    return takeFromExternrefTable0(ret[0]);
+    var v2 = getArrayU8FromWasm0(ret[0], ret[1]).slice();
+    wasm.__wbindgen_free(ret[0], ret[1] * 1, 1);
+    return v2;
 }
 
 /**
  * @param {Uint8Array} bytes
  * @param {CsvMapping} mapping
- * @returns {ParsedStdf}
+ * @returns {Uint8Array}
  */
 export function parse_json(bytes, mapping) {
     const ptr0 = passArray8ToWasm0(bytes, wasm.__wbindgen_malloc);
     const len0 = WASM_VECTOR_LEN;
     const ret = wasm.parse_json(ptr0, len0, mapping);
-    if (ret[2]) {
-        throw takeFromExternrefTable0(ret[1]);
+    if (ret[3]) {
+        throw takeFromExternrefTable0(ret[2]);
     }
-    return takeFromExternrefTable0(ret[0]);
+    var v2 = getArrayU8FromWasm0(ret[0], ret[1]).slice();
+    wasm.__wbindgen_free(ret[0], ret[1] * 1, 1);
+    return v2;
 }
 
 /**
  * @param {Uint8Array} bytes
  * @param {CsvMapping} mapping
- * @returns {ParsedStdf}
+ * @returns {Uint8Array}
  */
 export function parse_parquet(bytes, mapping) {
     const ptr0 = passArray8ToWasm0(bytes, wasm.__wbindgen_malloc);
     const len0 = WASM_VECTOR_LEN;
     const ret = wasm.parse_parquet(ptr0, len0, mapping);
-    if (ret[2]) {
-        throw takeFromExternrefTable0(ret[1]);
+    if (ret[3]) {
+        throw takeFromExternrefTable0(ret[2]);
     }
-    return takeFromExternrefTable0(ret[0]);
+    var v2 = getArrayU8FromWasm0(ret[0], ret[1]).slice();
+    wasm.__wbindgen_free(ret[0], ret[1] * 1, 1);
+    return v2;
 }
 
 /**
  * @param {Uint8Array} bytes
- * @returns {ParsedStdf}
+ * @returns {Uint8Array}
  */
 export function parse_stdf(bytes) {
     const ptr0 = passArray8ToWasm0(bytes, wasm.__wbindgen_malloc);
     const len0 = WASM_VECTOR_LEN;
     const ret = wasm.parse_stdf(ptr0, len0);
-    if (ret[2]) {
-        throw takeFromExternrefTable0(ret[1]);
+    if (ret[3]) {
+        throw takeFromExternrefTable0(ret[2]);
     }
-    return takeFromExternrefTable0(ret[0]);
+    var v2 = getArrayU8FromWasm0(ret[0], ret[1]).slice();
+    wasm.__wbindgen_free(ret[0], ret[1] * 1, 1);
+    return v2;
 }
 
 /**
  * @param {Uint8Array} bytes
  * @param {number[]} selected
- * @returns {ParsedStdf}
+ * @returns {Uint8Array}
  */
 export function parse_stdf_filtered(bytes, selected) {
     const ptr0 = passArray8ToWasm0(bytes, wasm.__wbindgen_malloc);
     const len0 = WASM_VECTOR_LEN;
     const ret = wasm.parse_stdf_filtered(ptr0, len0, selected);
-    if (ret[2]) {
-        throw takeFromExternrefTable0(ret[1]);
+    if (ret[3]) {
+        throw takeFromExternrefTable0(ret[2]);
     }
-    return takeFromExternrefTable0(ret[0]);
+    var v2 = getArrayU8FromWasm0(ret[0], ret[1]).slice();
+    wasm.__wbindgen_free(ret[0], ret[1] * 1, 1);
+    return v2;
 }
 
 /**