@crewhaus/tool-data 0.7.0

package/README.md

@@ -0,0 +1,193 @@
+# @crewhaus/tool-data
+
+Deterministic structured-data tools. Every one is pure: no filesystem, no
+network, no clock, no randomness. The same input always produces the same
+bytes.
+
+That property is the point. A harness spends a model call when it needs
+judgement; it should not spend one to pull two fields out of an API response,
+join two result sets, or turn a CSV into records. These twenty-three tools
+cover the mechanical half so the model is left with the part that actually
+needs it.
+
+```yaml
+tools:
+  - all-data          # every tool below
+  - -DataConvert      # ...except this one
+```
+
+| Tool | What it does |
+|---|---|
+| `ColumnsToRecords` | Column arrays back to records, short columns padded and named |
+| `CsvParse` | RFC 4180 CSV to records or rows, with type inference |
+| `CsvWrite` | Records or rows to RFC 4180 CSV, quoting whatever needs it |
+| `DataConvert` | JSON ⇄ YAML ⇄ TOML ⇄ CSV ⇄ JSONL |
+| `DataDiff` | Structural diff of two documents: added, removed, changed, by path |
+| `DataShape` | What is actually in this document — fields, types, fill rates, examples |
+| `DedupeRecords` | Drop duplicates by key fields or by the whole value, key order ignored |
+| `FlattenObject` | Nested value to dotted keys |
+| `JsonFormat` | Re-indent, minify, key-sort, address a subtree, hash the result |
+| `JsonMergePatch` | Apply an RFC 7386 merge patch, or derive one between two documents |
+| `JsonPatch` | Apply RFC 6902 operations, all-or-nothing |
+| `JsonQuery` | Select values with a JSONPath-like expression |
+| `JsonSortKeys` | Key-sorted JSON, for diff-stable output |
+| `JsonlParse` | Line-delimited JSON, with the line number of any bad record |
+| `JsonlWrite` | Values to line-delimited JSON |
+| `RecordsToColumns` | Records to column arrays |
+| `SampleRecords` | Head, tail, every nth, or evenly spread — never random |
+| `SortRecords` | Stable, total ordering by one or more dotted paths |
+| `TableAggregate` | Group by, with count/sum/min/max/avg/first/last/distinct |
+| `TableJoin` | Inner, left, right or full join of two record arrays |
+| `TableQuery` | Filter, project, sort and page records with a small predicate |
+| `UnflattenObject` | Dotted keys back to a nested value |
+| `XmlParse` | XML or an HTML fragment to a JSON tree |
+
+## Two input conventions
+
+A whole **document** — JSON, YAML, TOML, CSV, XML, JSONL — arrives as text,
+because that is how it comes off disk or off the wire. A **record array**
+arrives as a real array of objects, because that is how it comes out of the
+previous tool in the chain. Field references inside the record tools are
+dotted paths (`user.team.id`), so nested records work without flattening
+them first.
+
+## The parsers are hand-written, so here is exactly what they support
+
+There is no YAML, TOML, XML or CSV dependency behind these tools. That keeps
+the package free of a supply chain, and it means the supported subset is a
+contract rather than a hopeful summary. Each module's header comment carries
+the authoritative version; this is the short form.
+
+### JsonQuery's grammar
+
+```
+$.a.b            child
+$["odd key"]     quoted child, for keys with dots or spaces
+$.a[0]  $.a[-1]  index, negative counts from the end
+$.a[1:5]  [::2]  slice, Python semantics, negative step included
+$.a[*]   $.*     every element / every value
+$..name  $..*    recursive descent
+$.u[?(@.age >= 30)]   filter: == != < <= > >= =~ , or a bare existence test
+```
+
+Not supported, each raising a parse error naming the position rather than
+returning an empty result: unions (`['a','b']`), functions, parent axes, and
+`&&` / `||` inside a filter — chain two filter steps instead. Comparing a
+number against a string is simply false; there is no coercion, because a
+quiet `"10" < 9` is how a filter silently drops rows.
+
+### YAML
+
+**Read:** block mappings and sequences nested by space indentation, a compact
+map on the dash line (`- name: a`), plain/single-quoted/double-quoted scalars
+with the usual escapes, core-schema typing (`null` `~` `true` `false`,
+integers, decimals, exponents, `0x`, `0o`), flow collections
+(`[1, {a: b}]`), block scalars `|` and `>` with the `-` and `+` chomping
+indicators, `#` comments, and `---` / `...` markers.
+
+`.inf` and `.nan` read and write correctly, but JSON has no spelling for
+either, so converting a document that uses them **to JSON** renders them
+`null`. The same is true of TOML's `inf` and `nan`.
+
+**Refused explicitly, never guessed at:** anchors and aliases (`&a`, `*a`),
+merge keys (`<<:`), tags (`!!str`), multiple documents in one stream, complex
+keys (`? `), and tabs in indentation. Dates stay strings.
+
+**Write:** block style, two-space indent, keys in their existing order, with
+a string quoted whenever leaving it bare would change how it reads back.
+
+### TOML
+
+**Read:** bare, quoted and dotted keys; tables and arrays of tables at any
+depth; basic, literal and both multi-line string forms, including the
+backslash line continuation; integers with `_` separators and `0x`/`0o`/`0b`
+literals; floats, `inf`, `nan`; arrays, nested and multi-line; inline tables;
+`#` comments. Redefining a table or key is an error, as the spec requires.
+
+**The one deliberate departure:** dates and times stay strings. TOML has four
+first-class date-time types and JSON has none, so `2026-01-01` reads back as
+`"2026-01-01"`. Converting to a `Date` would make the round trip lossy.
+Integers outside the JavaScript safe range likewise stay strings rather than
+being silently rounded.
+
+Redefinition is enforced in both directions the spec names: a `[header]`
+cannot reopen a table a dotted key already created, and a dotted key cannot
+reach into a table a `[header]` already defined.
+
+**Write:** scalars first, then sub-tables, then arrays of tables — the order
+that reads back identically. `null` has no TOML representation, so it is
+dropped and the dropped keys are named in the output.
+
+### CSV
+
+RFC 4180 in full: quoted fields, delimiters and newlines inside quotes,
+doubled quotes, a final row with no trailing newline, and a row that is a
+single quoted empty field (`""` is a record, not a blank line). Plus the two
+departures real files need — a configurable delimiter, and tolerance of bare
+LF as well as CRLF. A byte-order mark is stripped.
+
+Duplicate and blank column names are made unique before records are built,
+and the suffix is bumped until the name is genuinely unused, so `a,a,a_2`
+becomes `a, a_2, a_2_2` rather than losing a column to a collision.
+
+Type inference is opt-in and conservative: `007`, `+5` and anything outside
+the safe integer range stay strings, because turning a zip code into a number
+loses data. Ragged rows are reported, not silently padded.
+
+### XML
+
+A well-formedness-checking parser for the XML that shows up in feeds, SVG and
+config files. It is **not** a conformant XML processor and does not claim to
+be. Elements, attributes, text, CDATA, self-closing tags, the five predefined
+entities and numeric character references; comments, processing instructions
+and doctypes skipped; namespace prefixes kept verbatim, not resolved.
+
+DTD-declared and external entities are refused outright — there is no
+resolver, so this parser cannot be made to fetch anything. A numeric
+character reference outside the Unicode range, or naming a surrogate, is a
+reported error rather than a crash. Validation, `xml:space` and raw-text
+elements (`<script>`, `<style>`) are not supported.
+`html` mode adds void elements, unquoted and bare attributes, case-insensitive
+tag names, and browser-style recovery from a mismatched close tag.
+
+## What is deliberately not here
+
+Anything whose core is judgement rather than mechanism: inferring what a
+column *means*, reconciling two schemas by intent, deciding which of two
+conflicting records is right, summarizing a dataset in prose. `DataShape`
+reports types and fill rates, not semantics.
+
+Nor is there a random sampler. `SampleRecords` offers head, tail, every-nth
+and evenly-spread, because a tool that returned a different subset on each
+call could not be cached, retried, or compared between runs.
+
+Where a deterministic first pass exists, it belongs first and the model is the
+escalation path.
+
+## A caller's mistake is a result, not an exception
+
+Malformed JSON, an unparseable path, an unterminated CSV quote, a mismatched
+XML tag: each comes back as a readable string naming the format and, where
+the format has lines, the line number. Tools throw only on a bug in the tool.
+`JsonPatch` goes further and is transactional — if operation 4 of 6 fails, it
+says so and the document is unchanged. That includes a pointer that is not a
+pointer: `"a"` instead of `"/a"` names the operation it broke on, rather than
+raising out of the tool.
+
+`src/index.test.ts` drives every tool with malformed input and asserts a
+readable result, and `src/integration.test.ts` does the same through
+`executeTool`.
+
+## Layout
+
+`src/lib/` holds the pure functions and is where the behaviour is tested;
+`src/index.ts` wraps them as tools. A bug in the CSV quote handling reads
+better as a failing unit than as a failing tool call.
+
+## Safety flags
+
+All twenty-three are `readOnly`, non-destructive, `scope: "internal"`, and
+declare no io capability, because none of them crosses a process or network
+boundary. `packages/tool-data/src/index.test.ts` asserts that for every tool,
+so a future addition that reaches outside has to change the assertion
+deliberately.

package/dist/index.d.ts

@@ -0,0 +1,35 @@
+import type { RegisteredTool } from "@crewhaus/tool-catalog";
+export declare const jsonQuery: RegisteredTool;
+export declare const jsonPatch: RegisteredTool;
+export declare const jsonMergePatch: RegisteredTool;
+export declare const jsonFormat: RegisteredTool;
+export declare const dataDiff: RegisteredTool;
+export declare const dataConvert: RegisteredTool;
+export declare const csvParse: RegisteredTool;
+export declare const csvWrite: RegisteredTool;
+export declare const tableQuery: RegisteredTool;
+export declare const tableAggregate: RegisteredTool;
+export declare const tableJoin: RegisteredTool;
+export declare const recordsToColumns: RegisteredTool;
+export declare const columnsToRecords: RegisteredTool;
+export declare const flattenObject: RegisteredTool;
+export declare const unflattenObject: RegisteredTool;
+export declare const jsonlParse: RegisteredTool;
+export declare const jsonlWrite: RegisteredTool;
+export declare const xmlParse: RegisteredTool;
+export declare const sortRecords: RegisteredTool;
+export declare const dedupeRecords: RegisteredTool;
+export declare const sampleRecords: RegisteredTool;
+export declare const dataShape: RegisteredTool;
+export declare const jsonSortKeys: RegisteredTool;
+/** Every tool this package registers, in the order a catalog should list them. */
+export declare const DATA_TOOLS: ReadonlyArray<RegisteredTool>;
+/**
+ * The CSV and record helpers, re-exported for other tool packages.
+ *
+ * `@crewhaus/tool-table` profiles and reconciles the same files this package
+ * parses, and two CSV readers in one repository would disagree about
+ * quoting, embedded newlines and what counts as a null token — on the same
+ * file, in different tools, which is the worst way to disagree.
+ */
+export { type CsvParseOptions, type CsvParseResult, inferScalar, normalizeHeader, parseCsv, rowsToRecords, unionKeys, } from "./lib/csv";