@eltjs/elt 0.3.2 → 0.3.4

package/README.md

@@ -158,7 +158,7 @@ The shape of a stream is its fields and their types, and its declarations. The f
   - `CheckpointStore.reset(id)` keeps the stored rows and loads again from no checkpoint. This is the refresh of Airbyte that keeps records.
   - `pipeline.clear()` drops the rows. This is the Clear of Airbyte.
 - **A cursor field that the stream does not describe now** fails the copy before it reads. The copy must select a different cursor. That changes its binding, so the host also resets it.
-- **Stored nullability.** A stored table is `NOT NULL` only on `loaded_at` and on the explicit primary key columns, because a field that is added later has no value on earlier rows. The Postgres destination of Airbyte also adds columns as nullable, because it does not "know what constitutes a reasonable default value for preexisting records" ([source](https://github.com/airbytehq/airbyte/blob/f847ad015360face1837f5a77c64325f104157f0/airbyte-integrations/connectors/destination-postgres/src/main/kotlin/io/airbyte/integrations/destination/postgres/sql/PostgresDirectLoadSqlGenerator.kt#L608-L615)). The destination still checks each record that it writes for its required fields, and a deduplicating load refuses a NULL key or cursor.
+- **Stored nullability.** A stored table is `NOT NULL` only on `loaded_at`, on `committed_in` and on the explicit primary key columns, because a field that is added later has no value on earlier rows. The Postgres destination of Airbyte also adds columns as nullable, because it does not "know what constitutes a reasonable default value for preexisting records" ([source](https://github.com/airbytehq/airbyte/blob/f847ad015360face1837f5a77c64325f104157f0/airbyte-integrations/connectors/destination-postgres/src/main/kotlin/io/airbyte/integrations/destination/postgres/sql/PostgresDirectLoadSqlGenerator.kt#L608-L615)). The destination still checks each record that it writes for its required fields, and a deduplicating load refuses a NULL key or cursor.
 
 ### Target ownership
 
@@ -248,7 +248,7 @@ A SQL destination gives each column, table and reader view a name that it can ho
 6. A name that starts with a digit gets a `_` in front (`1st` becomes `_1st`).
 
 - Columns get their names in the order of the fields. When an earlier column has the name, the field gets `_1`, then `_2` and so on, after it (`a b` becomes `a_b`, and a later `a_b` becomes `a_b_1`).
-- The names that the destination writes come first. `loaded_at` is one in the two destinations. Postgres also has the system column names `xmin`, `xmax`, `cmin`, `cmax`, `ctid` and `tableoid`. Thus a field `loaded_at` becomes `loaded_at_1`.
+- The names that the destination writes come first. In the two destinations they are `loaded_at`, `committed_in`, `_elt_row_hash` and `replaced_in`. SQLite also has `_rowid_`, and Postgres has the system column names `xmin`, `xmax`, `cmin`, `cmax`, `ctid` and `tableoid`. Thus a field `loaded_at` becomes `loaded_at_1`.
 - Postgres compares names with their case. SQLite compares names without ASCII case, so `Foo` and `foo` become `Foo` and `foo_1`.
 - A table or view does not see the names of other tables. When the rule changes its name, the name gets `_` and the first 8 hex digits of the SHA-256 of the requested name (`dbo.a b` becomes `dbo_a_b_<hash>`). Thus two requested names never share a table.
 - Postgres keeps only 63 bytes of a name. A longer name keeps its first 54 bytes, then `_` and the first 8 hex digits of the SHA-256 of the field or the requested name. SQLite has no length limit.
@@ -810,6 +810,7 @@ try {
 ```
 
 - `PipelineError` is an `AggregateError`. `errors` holds each failure. `results` holds the committed `count` and `deleted` of each copy beside its `failures`, in the order of the connections, then of the copies. Each failure has its `partition`, `error` and [`failureType`](#failure-types).
+- The `message` of `PipelineError`, and the error that `passError` gives the sync history, names each distinct failure message once, with each copy and partition that failed with it. Thus a store that refuses five streams gives its remedy once: `The following copies did not load completely: calls, contacts: Turn on … Full Disk Access.; notes: The store is corrupt.`
 - A failure after rows committed, such as a checkpoint that the store could not save or a cleanup error after the publication, shows in the same way, with the committed counts. It does not mean a rollback.
 - A count of zero also covers an empty input that committed.
 - A `PipelineError` holds only failures that the sync history recorded. A pass that could not run, such as a pass whose sync history the pipeline could not write, is recorded nowhere. Thus `run()` rejects with that error itself, after each other pass ends.
@@ -973,7 +974,7 @@ Each source tells its answer in its README. Most sources answer `null`. These an
 Four clocks have different meanings:
 
 - The modification fields of a source record tell of changes in the upstream.
-- `loaded_at` tells of the load that last wrote a row.
+- `loaded_at` tells of the load that last wrote a row. It is the one column of its own that a destination shows to readers: a reader view shows the columns of the stream and `loaded_at`, now and as of a position. The other columns of the destination are internal ([SQLite](../destinations/sqlite/README.md#columns), [Postgres](../destinations/postgresql/README.md#reader-views)).
 - `last_successful_sync_at` tells when a recorded pass ended.
 - `current_as_of` tells when a stream last agreed with its upstream: the start of its last successful pass, or a later watch check that found it unchanged. It is never later than when its source tells that the upstream itself was last current. Use it to judge freshness.
 

package/dist/core/copy.d.ts

@@ -14,7 +14,7 @@ export type CopyOutcome<Target extends DestinationTarget> = CopyResult<Target> &
     readonly failures: readonly LoadFailure[];
     readonly cancelled: boolean;
 };
-export declare function describeFailures<Target extends DestinationTarget>({ copy, failures, }: CopyOutcome<Target>): string;
+export declare function describeFailures<Target extends DestinationTarget>(outcomes: readonly CopyOutcome<Target>[]): string;
 export declare class Copy<Target extends DestinationTarget> {
     readonly configuration: CopyConfiguration;
     readonly to: Target;

package/dist/core/copy.js

@@ -2,10 +2,18 @@ import { errorMessage } from '../state/sync-history.js';
 import { CopyConfiguration } from './copy-configuration.js';
 import { FileTransfer } from './file-transfer.js';
 import { Target as DestinationTarget } from './target.js';
-export function describeFailures({ copy, failures, }) {
-    return failures
-        .map(({ partition, error }) => `${copy.id ?? copy.from.name}${partition === null ? '' : ` ${JSON.stringify(partition)}`}: ${errorMessage(error)}`)
-        .join('; ');
+// What did not load, naming each distinct failure message once with every
+// copy and partition it failed, in order of first appearance, as a log
+// aggregator folds repeated lines: a store that refuses five streams gives
+// its remedy once.
+export function describeFailures(outcomes) {
+    const failed = new Map();
+    for (const { copy, failures } of outcomes)
+        for (const { partition, error } of failures)
+            failed
+                .getOrInsertComputed(errorMessage(error), () => [])
+                .push(`${copy.id ?? copy.from.name}${partition === null ? '' : ` ${JSON.stringify(partition)}`}`);
+    return `The following copies did not load completely: ${Array.from(failed, ([message, copies]) => `${copies.join(', ')}: ${message}`).join('; ')}`;
 }
 // One configured transfer from a source stream to a destination declaration.
 export class Copy {

package/dist/core/pipeline.js

@@ -33,9 +33,8 @@ export class PipelineError extends AggregateError {
     name = 'PipelineError';
     results;
     constructor(results) {
-        const incomplete = results.filter(({ failures }) => failures.length > 0);
-        const errors = incomplete.flatMap(({ failures }) => failures.map(({ error }) => error));
-        super(errors, `The following copies did not load completely: ${incomplete.map(describeFailures).join('; ')}`, { cause: errors[0] });
+        const errors = results.flatMap(({ failures }) => failures.map(({ error }) => error));
+        super(errors, describeFailures(results), { cause: errors[0] });
         this.results = Object.freeze([...results]);
     }
 }

package/dist/state/sync-history.js

@@ -73,8 +73,7 @@ export function passFailureType(outcomes) {
 }
 // What did not load in a pass, or null when every copy loaded completely.
 export function passError(outcomes) {
-    const incomplete = outcomes.filter(({ failures }) => failures.length > 0);
-    if (incomplete.length === 0)
-        return null;
-    return `The following copies did not load completely: ${incomplete.map(describeFailures).join('; ')}`;
+    return outcomes.some(({ failures }) => failures.length > 0)
+        ? describeFailures(outcomes)
+        : null;
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@eltjs/elt",
-  "version": "0.3.2",
+  "version": "0.3.4",
   "description": "A pipeline that copies the data of apps and services into stores that SQL can read: sources, streams, checkpoints, watches and derived datasets.",
   "license": "Apache-2.0",
   "repository": {