@forzalabs/remora 2.5.0 → 2.6.0

package/CHANGELOG.md

@@ -6,6 +6,26 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 
 ## Unreleased
 
+## V 2.6.0 - 2026-10-26
+
+**Upgrading from 2.5.** Nothing in an existing project changes, and nothing that compiled under 2.5 is refused. One thing moves, and it is not opt-out: a `CSV` or `TXT` export now **quotes a field only where the quotes carry meaning**, where it used to quote every field. The values and the nulls are the same values and the same nulls — `"USR-2","free"` is now `USR-2,free` — but the bytes are not, so a destination that matches the file against a fixture, or parses it by splitting on `","` rather than with a CSV reader, needs checking before this version runs. This is the second consecutive release to move those bytes, after 2.4.0's null marker; a proper CSV reader has been unaffected by both.
+
+### Changed
+
+- **A positional field is quoted only when it needs to be.** Every field of a `CSV`/`TXT` export and of the internal staged dataset used to be wrapped in quotes unconditionally, which cost two bytes per field per row — written by each worker, concatenated by the merge, re-read by every post-processing pass, and shipped to the destination. A value is now written bare unless it holds the delimiter, a `"`, a line break, or a space at either end, and an **empty string stays quoted** because that is what keeps `""` (the empty string) apart from nothing at all (a null). The header line follows the same rule, so it still parses with the reader the data does.
+
+  ```
+  id,note,amount
+  USR-1,"",12.5        <- note is the empty string
+  USR-2,,              <- note and amount are null
+  USR-3,"a, b",12.5    <- note holds the delimiter, so it is quoted
+  USR-4,"  padded  ",0 <- note has edge spaces, so it is quoted
+  ```
+
+  **Measured -14.7% on the bytes** of a real 192MB export (201,795,101 → 172,095,071 over 990,000 records of 15 columns), which comes off every stage that reads or writes a staged file and off what the destination stores. The encode itself is 1.17x faster per row, and end to end the wall clock moves inside the harness's own ±6% noise floor — the bytes are the result here, not the throughput. A staged dataset written by an older version, where everything is quoted, still reads back identically, so nothing in a half-finished run is stranded.
+
+- **Edge spaces are quoted deliberately**, which is a rule the original specification did not have. Most CSV readers trim an unquoted field — Remora's own does — so a value of `  padded  ` written bare would come back shorter than it went out. Quoting it is what keeps the round trip exact, and it costs nothing measurable: the check rides in the same single pass that looks for the delimiter and the quote.
+
 ## V 2.5.0 - 2026-10-26
 
 **Upgrading from 2.4.** Nothing in an existing project changes. `FIXED` is a new output format and every rule this release adds fires only on a consumer that declares one, so no configuration needs touching, no exported bytes move, and nothing that compiled under 2.4 is refused. The one thing to know before writing a FIXED output is what the receiver has to do with the file: it leaves Remora as UTF-8 text with `lf` or `crlf` line terminators, so a z/OS target still needs the ASCII→EBCDIC and `RECFM=FB` conversion on its own side (see Added).

package/docs/Remora - Consumers.md

@@ -1042,11 +1042,11 @@ Records then legitimately differ in shape from one another, which is why it appl
 
 ### **The Header Line**
 
-A `CSV` or `TXT` file is positional: it carries values and nothing that names them. So both formats write a **header line** first — the exported field names, in column order, quoted and delimited exactly as the rows under them, so the header parses with the same reader the data does:
+A `CSV` or `TXT` file is positional: it carries values and nothing that names them. So both formats write a **header line** first — the exported field names, in column order, quoted and delimited on exactly the rule the rows under it follow, so the header parses with the same reader the data does:
 
 ```
-"id","primary_email","plan","country"
-"USR-00001","alice.rossi@example.com","free","Italy"
+id,primary_email,plan,country
+USR-00001,alice.rossi@example.com,free,Italy
 ```
 
 The names are the exported ones: a field's `alias` where it has one, its `key` otherwise. A `hidden` field is not written, so it is not in the header either. With a `split`, **every** file of the output carries the header, not just the first one — each file has to stand on its own.
@@ -1070,20 +1070,28 @@ It only applies to `CSV` and `TXT`. On a `JSON` or `API` output each record alre
 
 ### **How a value is written**
 
-A `CSV`/`TXT` file writes every value quoted, and that leaves the unquoted empty field free to mean
-something: it is a **null**. An empty string is `""`. The two are different values, and on a
-positional line this is the only way to tell them apart:
+A `CSV`/`TXT` file **quotes a value only where the quotes carry meaning**. A value is written bare
+unless it holds the delimiter, a `"`, a line break, or a space at either end — and an empty string
+is quoted too, because that is what leaves the unquoted empty field free to mean something else: it
+is a **null**. The two are different values, and on a positional line this is the only way to tell
+them apart:
 
 ```
-"id","note","amount"
-"USR-1","","12.5"       <- note is the empty string
-"USR-2",,               <- note and amount are null
+id,note,amount
+USR-1,"",12.5        <- note is the empty string
+USR-2,,              <- note and amount are null
+USR-3,"a, b",12.5    <- note holds the delimiter, so it is quoted
+USR-4,"  padded  ",0 <- note has edge spaces, so it is quoted
 ```
 
 A reader that treats an unquoted empty field as "no value" — which most CSV readers do, and which is
 what `COPY`, `LOAD DATA` and a pandas `read_csv` all assume — gets a null back. A reader that does
 not sees an empty string, exactly as before.
 
+Edge spaces are quoted for the same reason the empty string is: most CSV readers, Remora's own
+included, trim an unquoted field, so `  padded  ` written bare would come back shorter than it went
+out. Quoting it is what makes the round trip exact.
+
 A value holding a **newline** or a carriage return is written as the two characters `\n` or `\r`.
 A line of a remora file is one record, everywhere in the pipeline, so a value carrying a real
 newline would otherwise arrive as two broken rows. The escape is one-way: nothing turns it back into
@@ -1307,7 +1315,7 @@ Apache Parquet: a typed, SNAPPY-compressed **columnar** file, which is what a da
 }
 ```
 
-Two things it gives a reader that a `CSV` export cannot. The file **declares its types**, so nothing downstream has to guess them back out of the text. And it is much smaller: a four-column, 2-million-row dataset is about 28 MB as parquet against 115 MB as quoted CSV.
+Two things it gives a reader that a `CSV` export cannot. The file **declares its types**, so nothing downstream has to guess them back out of the text. And it is much smaller: a four-column, 2-million-row dataset is about 28 MB as parquet against roughly 100 MB as CSV.
 
 The schema comes from what the consumer already declares — each field's dimension type:
 

package/index.js

@@ -11761,7 +11761,7 @@ var CONSTANTS, Constants_default;
 var init_Constants = __esm({
   "../../packages/constants/src/Constants.ts"() {
     CONSTANTS = {
-      cliVersion: "2.5.0",
+      cliVersion: "2.6.0",
       backendVersion: 1,
       backendPort: 5088,
       workerVersion: 2,
@@ -33962,21 +33962,29 @@ var init_CSVParser = __esm({
 });
 
 // ../../packages/parsing/src/DatasetCodec.ts
-var NULL_FIELD, NEEDS_ESCAPE, DatasetCodecClass, DatasetCodec, DatasetCodec_default;
+var NULL_FIELD, EMPTY_FIELD, NEEDS_ESCAPE, QUOTE_CODE, QUOTE_TESTS, DatasetCodecClass, DatasetCodec, DatasetCodec_default;
 var init_DatasetCodec = __esm({
   "../../packages/parsing/src/DatasetCodec.ts"() {
     init_src();
     NULL_FIELD = "";
+    EMPTY_FIELD = '""';
     NEEDS_ESCAPE = /["\n\r]/;
+    QUOTE_CODE = 34;
+    QUOTE_TESTS = /* @__PURE__ */ new Map([[",", /^\s|\s$|[",\n\r]/]]);
     DatasetCodecClass = class {
       constructor() {
         /**
-         * One value as an internal positional field: quoted, with `"`, `\n` and `\r` escaped - or the
-         * unquoted null marker when there is no value.
+         * One value as an internal positional field. Bare where it can be, quoted where it must be -
+         * around the delimiter, a quote, a line break or an edge space - or the unquoted null marker
+         * when there is no value. The delimiter is a parameter rather than a comma because `redelimit`
+         * re-encodes a staged line with the export's own: a tab is safe bare in the staged file and is
+         * not once that file is re-emitted tab-delimited.
          */
-        this.encodeField = (value) => {
+        this.encodeField = (value, delimiter = ",") => {
           if (value === null || value === void 0) return NULL_FIELD;
           const text = typeof value === "string" ? value : String(value);
+          if (text === "") return EMPTY_FIELD;
+          if (!this._quoteTest(delimiter).test(text)) return text;
           if (!NEEDS_ESCAPE.test(text)) return `"${text}"`;
           return `"${text.replaceAll('"', '""').replaceAll("\r", "\\r").replaceAll("\n", "\\n")}"`;
         };
@@ -33988,7 +33996,7 @@ var init_DatasetCodec = __esm({
           for (let index = 0; index < values.length; index++) {
             if (index > 0)
               row += delimiter;
-            row += this.encodeField(values[index]);
+            row += this.encodeField(values[index], delimiter);
           }
           return row;
         };
@@ -34008,26 +34016,26 @@ var init_DatasetCodec = __esm({
           let index = 0;
           while (index < length) {
             const char = line.charCodeAt(index);
-            if (char === 34) {
+            if (char === QUOTE_CODE) {
               if (!inQuotes) {
                 inQuotes = true;
                 hasQuotes = true;
                 fieldStart = index + 1;
-              } else if (line.charCodeAt(index + 1) === 34) {
+              } else if (line.charCodeAt(index + 1) === QUOTE_CODE) {
                 index++;
               } else {
                 inQuotes = false;
                 fieldEnd = index;
               }
             } else if (char === delimiterCode && !inQuotes) {
-              fields.push(this._field(line, fieldStart, fieldEnd, hasQuotes));
+              fields.push(this._field(line, fieldStart, hasQuotes ? fieldEnd : index, hasQuotes));
               fieldStart = index + 1;
               fieldEnd = 0;
               hasQuotes = false;
             }
             index++;
           }
-          fields.push(this._field(line, fieldStart, fieldEnd, hasQuotes));
+          fields.push(this._field(line, fieldStart, hasQuotes ? fieldEnd : length, hasQuotes));
           return fields;
         };
         /**
@@ -34038,9 +34046,24 @@ var init_DatasetCodec = __esm({
           Affirm_default(delimiter, "Invalid delimiter");
           return this.encodeRow(this.decodeRow(line), delimiter);
         };
-        /** One field's text, or null for the unquoted marker. Only a quoted token carries a value. */
+        /**
+         * The quoting test for one delimiter, built on first use. The four characters escaped into the
+         * class are the ones a character class reads as syntax; anything else stands for itself there.
+         */
+        this._quoteTest = (delimiter) => {
+          let test = QUOTE_TESTS.get(delimiter);
+          if (!test) {
+            test = new RegExp(`^\\s|\\s$|["\\n\\r${delimiter.replace(/[\\\]^-]/g, "\\$&")}]`);
+            QUOTE_TESTS.set(delimiter, test);
+          }
+          return test;
+        };
+        /**
+         * One field's text, or null for the unquoted empty marker. A bare token carries its value as it
+         * stands - nothing bare can hold a quote, so there is nothing to unescape in it.
+         */
         this._field = (line, start, end, hasQuotes) => {
-          if (!hasQuotes) return null;
+          if (!hasQuotes) return start === end ? null : line.slice(start, end);
           const raw = line.slice(start, end);
           return raw.indexOf('"') === -1 ? raw : raw.replaceAll('""', '"');
         };
@@ -56695,7 +56718,7 @@ var init_OutputExecutor = __esm({
           for (let index = 0; index < visibleKeys.length; index++) {
             if (index > 0)
               row += delimiter;
-            row += DatasetCodec_default.encodeField(record[visibleKeys[index]]);
+            row += DatasetCodec_default.encodeField(record[visibleKeys[index]], delimiter);
           }
           return row;
         };
@@ -56880,9 +56903,10 @@ var init_OutputExecutor = __esm({
           return resultList[0];
         };
         /**
-         * The header line of a positional export: the visible field names, in column order, quoted and
-         * delimited exactly as the rows under them — so the header parses with the same reader the data
-         * does. Null for a JSON/API output, where a record already carries its own field names, for a
+         * The header line of a positional export: the visible field names, in column order, written
+         * through the SAME codec as the rows under them — so a name is quoted on the same rule a value
+         * is, and the header parses with the reader the data does. Null for a JSON/API output, where a
+         * record already carries its own field names, for a
          * FIXED one, where a name does not fit the column it describes and the receiver reads by
          * position anyway, and for an output that turns the header off.
          *
@@ -56891,7 +56915,7 @@ var init_OutputExecutor = __esm({
         this._composeHeader = (output, dsSchema) => {
           if (output.header === false) return null;
           if (output.format !== "CSV" && output.format !== "TXT") return null;
-          return CSVParser_default.stringifyRow(DatasetSchema_default.names(dsSchema), this._getDelimiter(output));
+          return DatasetCodec_default.encodeRow(DatasetSchema_default.names(dsSchema), this._getDelimiter(output));
         };
         /** The delimiter the exported file is written with. Only TXT configures it; CSV is a comma. */
         this._getDelimiter = (output) => {

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@forzalabs/remora",
-    "version": "2.5.0",
+    "version": "2.6.0",
     "description": "A powerful CLI tool for seamless data translation.",
     "main": "index.js",
     "private": false,

package/workers/ExecutorWorker.js

@@ -11809,7 +11809,7 @@ var CONSTANTS, Constants_default;
 var init_Constants = __esm({
   "../../packages/constants/src/Constants.ts"() {
     CONSTANTS = {
-      cliVersion: "2.5.0",
+      cliVersion: "2.6.0",
       backendVersion: 1,
       backendPort: 5088,
       workerVersion: 2,
@@ -33745,21 +33745,29 @@ var init_CSVParser = __esm({
 });
 
 // ../../packages/parsing/src/DatasetCodec.ts
-var NULL_FIELD, NEEDS_ESCAPE, DatasetCodecClass, DatasetCodec, DatasetCodec_default;
+var NULL_FIELD, EMPTY_FIELD, NEEDS_ESCAPE, QUOTE_CODE, QUOTE_TESTS, DatasetCodecClass, DatasetCodec, DatasetCodec_default;
 var init_DatasetCodec = __esm({
   "../../packages/parsing/src/DatasetCodec.ts"() {
     init_src();
     NULL_FIELD = "";
+    EMPTY_FIELD = '""';
     NEEDS_ESCAPE = /["\n\r]/;
+    QUOTE_CODE = 34;
+    QUOTE_TESTS = /* @__PURE__ */ new Map([[",", /^\s|\s$|[",\n\r]/]]);
     DatasetCodecClass = class {
       constructor() {
         /**
-         * One value as an internal positional field: quoted, with `"`, `\n` and `\r` escaped - or the
-         * unquoted null marker when there is no value.
+         * One value as an internal positional field. Bare where it can be, quoted where it must be -
+         * around the delimiter, a quote, a line break or an edge space - or the unquoted null marker
+         * when there is no value. The delimiter is a parameter rather than a comma because `redelimit`
+         * re-encodes a staged line with the export's own: a tab is safe bare in the staged file and is
+         * not once that file is re-emitted tab-delimited.
          */
-        this.encodeField = (value) => {
+        this.encodeField = (value, delimiter = ",") => {
           if (value === null || value === void 0) return NULL_FIELD;
           const text = typeof value === "string" ? value : String(value);
+          if (text === "") return EMPTY_FIELD;
+          if (!this._quoteTest(delimiter).test(text)) return text;
           if (!NEEDS_ESCAPE.test(text)) return `"${text}"`;
           return `"${text.replaceAll('"', '""').replaceAll("\r", "\\r").replaceAll("\n", "\\n")}"`;
         };
@@ -33771,7 +33779,7 @@ var init_DatasetCodec = __esm({
           for (let index = 0; index < values.length; index++) {
             if (index > 0)
               row += delimiter;
-            row += this.encodeField(values[index]);
+            row += this.encodeField(values[index], delimiter);
           }
           return row;
         };
@@ -33791,26 +33799,26 @@ var init_DatasetCodec = __esm({
           let index = 0;
           while (index < length) {
             const char = line.charCodeAt(index);
-            if (char === 34) {
+            if (char === QUOTE_CODE) {
               if (!inQuotes) {
                 inQuotes = true;
                 hasQuotes = true;
                 fieldStart = index + 1;
-              } else if (line.charCodeAt(index + 1) === 34) {
+              } else if (line.charCodeAt(index + 1) === QUOTE_CODE) {
                 index++;
               } else {
                 inQuotes = false;
                 fieldEnd = index;
               }
             } else if (char === delimiterCode && !inQuotes) {
-              fields.push(this._field(line, fieldStart, fieldEnd, hasQuotes));
+              fields.push(this._field(line, fieldStart, hasQuotes ? fieldEnd : index, hasQuotes));
               fieldStart = index + 1;
               fieldEnd = 0;
               hasQuotes = false;
             }
             index++;
           }
-          fields.push(this._field(line, fieldStart, fieldEnd, hasQuotes));
+          fields.push(this._field(line, fieldStart, hasQuotes ? fieldEnd : length, hasQuotes));
           return fields;
         };
         /**
@@ -33821,9 +33829,24 @@ var init_DatasetCodec = __esm({
           Affirm_default(delimiter, "Invalid delimiter");
           return this.encodeRow(this.decodeRow(line), delimiter);
         };
-        /** One field's text, or null for the unquoted marker. Only a quoted token carries a value. */
+        /**
+         * The quoting test for one delimiter, built on first use. The four characters escaped into the
+         * class are the ones a character class reads as syntax; anything else stands for itself there.
+         */
+        this._quoteTest = (delimiter) => {
+          let test = QUOTE_TESTS.get(delimiter);
+          if (!test) {
+            test = new RegExp(`^\\s|\\s$|["\\n\\r${delimiter.replace(/[\\\]^-]/g, "\\$&")}]`);
+            QUOTE_TESTS.set(delimiter, test);
+          }
+          return test;
+        };
+        /**
+         * One field's text, or null for the unquoted empty marker. A bare token carries its value as it
+         * stands - nothing bare can hold a quote, so there is nothing to unescape in it.
+         */
         this._field = (line, start, end, hasQuotes) => {
-          if (!hasQuotes) return null;
+          if (!hasQuotes) return start === end ? null : line.slice(start, end);
           const raw = line.slice(start, end);
           return raw.indexOf('"') === -1 ? raw : raw.replaceAll('""', '"');
         };
@@ -56687,7 +56710,7 @@ var init_OutputExecutor = __esm({
           for (let index = 0; index < visibleKeys.length; index++) {
             if (index > 0)
               row += delimiter;
-            row += DatasetCodec_default.encodeField(record[visibleKeys[index]]);
+            row += DatasetCodec_default.encodeField(record[visibleKeys[index]], delimiter);
           }
           return row;
         };
@@ -56872,9 +56895,10 @@ var init_OutputExecutor = __esm({
           return resultList[0];
         };
         /**
-         * The header line of a positional export: the visible field names, in column order, quoted and
-         * delimited exactly as the rows under them — so the header parses with the same reader the data
-         * does. Null for a JSON/API output, where a record already carries its own field names, for a
+         * The header line of a positional export: the visible field names, in column order, written
+         * through the SAME codec as the rows under them — so a name is quoted on the same rule a value
+         * is, and the header parses with the reader the data does. Null for a JSON/API output, where a
+         * record already carries its own field names, for a
          * FIXED one, where a name does not fit the column it describes and the receiver reads by
          * position anyway, and for an output that turns the header off.
          *
@@ -56883,7 +56907,7 @@ var init_OutputExecutor = __esm({
         this._composeHeader = (output, dsSchema) => {
           if (output.header === false) return null;
           if (output.format !== "CSV" && output.format !== "TXT") return null;
-          return CSVParser_default.stringifyRow(DatasetSchema_default.names(dsSchema), this._getDelimiter(output));
+          return DatasetCodec_default.encodeRow(DatasetSchema_default.names(dsSchema), this._getDelimiter(output));
         };
         /** The delimiter the exported file is written with. Only TXT configures it; CSV is a comma. */
         this._getDelimiter = (output) => {