@eltjs/elt-postgresql 0.3.14 → 0.3.16

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -160,7 +160,7 @@ How a reader knows which numbers are done:
160
160
 
161
161
  The history of a table:
162
162
 
163
- - `<schema>._elt_history_<hash>` keeps each row that a commit replaced or removed, with `replaced_in`, the number of that commit, as a SQL Server temporal table keeps its history beside the current table. The writes that keep rows are a deduplicating upsert that changes the values (behind the same cursor guard), a staged `DELETE`, a reset of a partition, an overwrite, a reload, and a change of shape. A write into a hidden table keeps nothing, because no reader saw its rows.
163
+ - `<schema>._elt_history_<hash>` keeps each row that a commit replaced or removed, with `replaced_in`, the number of that commit, as a SQL Server temporal table keeps its history beside the current table. The writes that keep rows are a deduplicating upsert that changes the values (behind the same cursor guard), a staged `DELETE`, the end of the reload of a reset partition, an overwrite, a reload, and a change of shape. `<schema>._elt_reloads` names the partitions of each open reload of a reset partition. A write into a hidden table keeps nothing, because no reader saw its rows.
164
164
  - `clear()` forgets the rows that it removes. Its commit C sets `min_valid` to C, if it is higher, with `expired_by` `clear`, and the history of the cleared table deletes each of its rows, with the stored files that only they refer to. A reading or `positions(after)` before C gives `{ expired }`, with the reason that a clear forgot the rows of its table. The history of another table keeps its rows with `replaced_in <= C` until its next commit or the retention deletes them; a reading skips them, as for a change of type.
165
165
  - Each column of the history has the type of the table's column of the same name. It takes each new column of the table. A column that the table drops stays in the history with its last type.
166
166
  - When a change of shape, or the swap of a reload, gives a column of the table another type, the history changes the type of its column in the same commit. This includes a column that the table dropped and a later change adds back with another type. Delta Lake, Iceberg, BigQuery and Snowflake change a type in place only when the change is a widening, and they read the past with its own schema, which one Postgres view cannot do. The destination follows the same rule:
@@ -291,7 +291,7 @@ A refresh is a load in `_warehouse._elt_loads`, under the name of its dataset, s
291
291
 
292
292
  - It records only connections with a Postgres destination. It refuses other connections when the pipeline validates.
293
293
  - It records into `_warehouse.sync_attempts`, `_warehouse.extraction_coverage` and `_warehouse.stream_checks`. The last one holds the latest check of each stream by a watch that found it unchanged.
294
- - `install()` makes those tables and publishes the `marts` views `sync_attempts`, `extraction_coverage`, `sync_status`, `stream_status` and `catalog`, in one transaction. The pipeline calls it before each run and watch, and it is safe to repeat. See [ADR 0005](./docs/adr/0005-the-history-install-publishes-the-catalog.md).
294
+ - `install()` makes those tables and `_warehouse.derived_status`, and publishes the `marts` views `sync_attempts`, `extraction_coverage`, `sync_status`, `stream_status`, `derived_status` and `catalog`, in one transaction. The pipeline calls it before each run and watch, and it is safe to repeat. See [ADR 0005](./docs/adr/0005-the-history-install-publishes-the-catalog.md).
295
295
  - `new PostgresSyncHistory({ url, readers: ['agent_reader'] })` names the roles that read the database as of a position with their own session: `new PostgresDestination({ url: readerUrl, schema })`, then `positions()` and `reading()`. Each install grants each reader what a reading needs, which is the numbering in `_warehouse` and nothing else of it, as dbt applies the grants that a model declares after each build. Thus no host names the objects of the numbering, and a reader keeps its grants when the numbering is made again. The install grants nothing else: who may read `marts` is a grant of the database, such as its default privileges. A role that does not exist fails the install.
296
296
  - Before a pass reads, the history inserts one `sync_attempts` row, with the connection name as `connector` and the source identity as `source`. It inserts one `extraction_coverage` row for each selected stream, with the declared `coverage(stream)` of the source and the schema of the destination as `target_schema`. The outcomes close them.
297
297
  - A run declares each stream that a connection selected. A watch pass reads only the streams whose change signals changed, so an attempt vouches only for its own `extraction_coverage` rows.
@@ -300,13 +300,14 @@ A refresh is a load in `_warehouse._elt_loads`, under the name of its dataset, s
300
300
 
301
301
  The relations in `marts`:
302
302
 
303
- | Relation | Contents |
304
- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
305
- | `catalog` | Each view and column in `marts`, with its description. The start point of a reader. A table is not for readers, also when it is in `marts`, such as the raw table of a computed dataset. |
306
- | `sync_status` | For each connection: the latest attempt, the end of the last successful pass, and `current_as_of` (the oldest of its streams). |
307
- | `stream_status` | For each connection and stream: the latest attempt that declared the stream with the status of its copy, and the last attempt in which that copy succeeded. A watch pass reads only changed streams, so judge a stream here, by `current_as_of`. |
308
- | `sync_attempts` | The kept history of attempts, with start and end times, status and errors. |
309
- | `extraction_coverage` | For each attempt and stream: the configured selection, the explanation of the scope, the target, the outcome of the copy, the committed counts and the failed partitions. |
303
+ | Relation | Contents |
304
+ | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
305
+ | `catalog` | Each view and column in `marts`, with its description. The start point of a reader. A table is not for readers, also when it is in `marts`, such as the raw table of a computed dataset. |
306
+ | `sync_status` | For each connection: the latest attempt, the end of the last successful pass, and `current_as_of` (the oldest of its streams). |
307
+ | `stream_status` | For each connection and stream: the latest attempt that declared the stream with the status of its copy, and the last attempt in which that copy succeeded. A watch pass reads only changed streams, so judge a stream here, by `current_as_of`. |
308
+ | `derived_status` | For each view and computed dataset, named with its schema: whether it waits for its inputs, refreshes, is current or failed, its `as_of`, and for a computed dataset how many inputs it computed, found empty, failed on or has still to compute. See [derived status](#derived-status). |
309
+ | `sync_attempts` | The kept history of attempts, with start and end times, status and errors. |
310
+ | `extraction_coverage` | For each attempt and stream: the configured selection, the explanation of the scope, the target, the outcome of the copy, the committed counts and the failed partitions. |
310
311
 
311
312
  What the relations tell:
312
313
 
@@ -319,6 +320,17 @@ What the relations tell:
319
320
  - A first failure has no time of success.
320
321
  - Coverage is the [configured scope](../../elt/README.md#coverage). Judge freshness by `current_as_of` ([the four clocks](../../elt/README.md#the-four-clocks)).
321
322
 
323
+ ## Derived status
324
+
325
+ Each [view](#declare-a-view) and [computed dataset](#declare-a-computed-dataset) keeps one row in `_warehouse.derived_status`, which readers see as `marts.derived_status`. A reader needs it because `sync_status` says that a pass succeeded while a computed dataset still computes, or failed on each of its inputs.
326
+
327
+ - A refresh writes `waiting` when a relation that the dataset reads is not made yet, and `refreshing` when it begins, with `refreshing_since`. A computed dataset also writes how many inputs it read and how many it has still to compute (`pending`). Each commit of the refresh lowers `pending` in its own transaction, so a reader sees it fall while the rows land.
328
+ - A refresh that ends with no failure of the dataset writes `current`, with `as_of`, when it began to read, and `refreshed_at`. A refresh that fails writes `failed`, with `error`. A rule that fails on one input is not a failure of the dataset: `failed` counts its rows.
329
+ - `computed`, `empty` and `failed` count the output rows when the refresh began and when it ended.
330
+ - A dataset is caught up when its `state` is `current`, its `pending` is 0, and its `as_of` is at or after the start of the last pass of the connections that it reads. This is the frontier of Materialize and the `DATA_TIMESTAMP` of a Snowflake dynamic table, with the pending items of a pgai vectorizer.
331
+ - A refresh that a stopped process left stays `refreshing` until the next refresh. The status keeps no row versions, so a reading as of a position fails the view, as it fails the other views of the sync history.
332
+ - A status write takes no commit number: it is not data of a position.
333
+
322
334
  ## Decisions
323
335
 
324
336
  - [ADR 0001](./docs/adr/0001-checkpoints-are-kept-beside-the-data.md): checkpoints are kept beside the data.
@@ -354,6 +366,7 @@ src/
354
366
  postgres-views.ts the publication of views, for View and the sync history
355
367
  postgres-dependents.ts the views over a table, made again over its change
356
368
  postgres-privileges.ts the privileges of a relation, kept across its change
369
+ postgres-derived-status.ts the status that each derived dataset keeps of itself
357
370
  postgres-checkpoint-store.ts PostgresCheckpointStore
358
371
  postgres-sync-history.ts PostgresSyncHistory and the catalog
359
372
  postgres-sync-history-schema.ts the tables and views of the history
@@ -1,9 +1,10 @@
1
1
  import { hash } from 'node:crypto';
2
2
  import { Lock } from '@zukhruf/async';
3
- import { DerivedDataset, } from '@eltjs/elt';
3
+ import { DerivedDataset, errorMessage, } from '@eltjs/elt';
4
4
  import { identifier, quote } from './identifier.js';
5
5
  import { TargetComments } from './postgres-comments.js';
6
6
  import { rebindDependents } from './postgres-dependents.js';
7
+ import { PostgresDerivedStatus } from './postgres-derived-status.js';
7
8
  import { PostgresDestination } from './postgres-destination.js';
8
9
  import { storedColumns } from './postgres-fit.js';
9
10
  import { TargetHistory, rowHash } from './postgres-history.js';
@@ -122,9 +123,25 @@ export class ComputedDataset extends DerivedDataset {
122
123
  // Open from the start, so it closes a refresh that a process ended; with
123
124
  // no commit it is no span, and leaves the positions as they are.
124
125
  await using load = await PostgresLoad.open(this.into.url, { dataset: this.name }, { clock, retention: this.into.retention });
126
+ const status = new PostgresDerivedStatus(load.sql, this.qualifiedName, 'computed', clock);
125
127
  const inputs = await this.#inputs(load.sql, connections);
126
- if (inputs === null)
128
+ if (inputs === null) {
129
+ await status.waiting();
127
130
  return;
131
+ }
132
+ try {
133
+ await this.#update(load, status, inputs, trigger, signal);
134
+ }
135
+ catch (error) {
136
+ // A stop is not the dataset's failure.
137
+ if (signal?.aborted !== true)
138
+ await status.failed(errorMessage(error));
139
+ throw error;
140
+ }
141
+ }
142
+ // Brings the output rows in step with inputs, and tells status how many
143
+ // are due and how many settled.
144
+ async #update(load, status, inputs, trigger, signal) {
128
145
  const key = quote(this.#rule.key.name);
129
146
  const table = this.#qualified(this.#table);
130
147
  // The number of a transaction's commit, taken at its first change, so a
@@ -214,6 +231,10 @@ export class ComputedDataset extends DerivedDataset {
214
231
  trigger.tried.add(attempt);
215
232
  due.push({ id, input, inputHash, rule });
216
233
  }
234
+ await status.refreshing({
235
+ ...(await this.#counts(load.sql, inputs)),
236
+ pending: due.length,
237
+ });
217
238
  // A pool of calls, each taking the next batch once its own committed, so
218
239
  // a slow call holds only its place, as a worker pool keeps its other
219
240
  // workers busy. A call that fails stops the pool taking more.
@@ -247,11 +268,28 @@ export class ComputedDataset extends DerivedDataset {
247
268
  else
248
269
  await store(sql, commit, id, inputHash, rule, 'empty', null, {});
249
270
  }
271
+ await status.settled(sql, batch.length);
250
272
  });
251
273
  }
252
274
  }));
253
275
  if (failures.length > 0)
254
276
  throw failures[0];
277
+ if (signal?.aborted === true)
278
+ return;
279
+ await status.current({
280
+ ...(await this.#counts(load.sql, inputs)),
281
+ pending: 0,
282
+ });
283
+ }
284
+ // How many inputs there are, and how many output rows have each status.
285
+ async #counts(sql, inputs) {
286
+ const counted = new Map((await sql.unsafe(`SELECT "status", count(*)::int AS "n" FROM ${this.#qualified(this.#table)} GROUP BY "status"`)).map(({ status, n }) => [status, n]));
287
+ return {
288
+ inputs: inputs.size,
289
+ computed: counted.get('computed') ?? 0,
290
+ empty: counted.get('empty') ?? 0,
291
+ failed: counted.get('failed') ?? 0,
292
+ };
255
293
  }
256
294
  // Makes the table, its history and its view, or makes the table again when
257
295
  // its declaration changed: the history keeps each row of the old table, in
@@ -0,0 +1,20 @@
1
+ import type postgres from 'postgres';
2
+ import type { PipelineClock } from '@eltjs/elt';
3
+ import { type Queries } from './postgres-session.ts';
4
+ export declare const derivedStatusTable = "CREATE TABLE IF NOT EXISTS _warehouse.derived_status (\n dataset text PRIMARY KEY,\n kind text NOT NULL CHECK (kind IN ('view', 'computed')),\n state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),\n as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,\n inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint\n )";
5
+ export type DerivedCounts = {
6
+ readonly inputs: number;
7
+ readonly computed: number;
8
+ readonly empty: number;
9
+ readonly failed: number;
10
+ readonly pending: number;
11
+ };
12
+ export declare class PostgresDerivedStatus {
13
+ #private;
14
+ constructor(sql: postgres.Sql, dataset: string, kind: 'view' | 'computed', clock: PipelineClock);
15
+ waiting(): Promise<void>;
16
+ refreshing(counts: DerivedCounts): Promise<void>;
17
+ settled(sql: Queries, count: number): Promise<void>;
18
+ current(counts?: DerivedCounts): Promise<void>;
19
+ failed(error: string): Promise<void>;
20
+ }
@@ -0,0 +1,77 @@
1
+ import { lockSchema } from './postgres-session.js';
2
+ // One row for each derived dataset of the database: what its refreshes left,
3
+ // which readers see through marts.derived_status. The sync history makes it
4
+ // with its tables, and a dataset makes it too, so it never depends on the
5
+ // order of the two.
6
+ export const derivedStatusTable = `CREATE TABLE IF NOT EXISTS _warehouse.derived_status (
7
+ dataset text PRIMARY KEY,
8
+ kind text NOT NULL CHECK (kind IN ('view', 'computed')),
9
+ state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),
10
+ as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,
11
+ inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint
12
+ )`;
13
+ // What one refresh of a derived dataset tells readers, as Snowflake keeps the
14
+ // state and DATA_TIMESTAMP of each dynamic table, and pgai the pending items
15
+ // of each vectorizer. A refresh that begins keeps as_of, the data of the
16
+ // last refresh that ended, until it ends too. dataset is the name with its
17
+ // schema.
18
+ export class PostgresDerivedStatus {
19
+ #sql;
20
+ #dataset;
21
+ #kind;
22
+ #clock;
23
+ // When this refresh began to read, which its end gives as as_of.
24
+ #since;
25
+ // sql is a session outside any transaction, between the dataset's own.
26
+ constructor(sql, dataset, kind, clock) {
27
+ this.#sql = sql;
28
+ this.#dataset = dataset;
29
+ this.#kind = kind;
30
+ this.#clock = clock;
31
+ this.#since = this.#now();
32
+ }
33
+ waiting() {
34
+ return this.#write({ state: 'waiting', refreshing_since: null });
35
+ }
36
+ refreshing(counts) {
37
+ return this.#write({
38
+ state: 'refreshing',
39
+ refreshing_since: this.#since,
40
+ ...counts,
41
+ });
42
+ }
43
+ // Inside the transaction of a commit that settled count inputs.
44
+ async settled(sql, count) {
45
+ await sql.unsafe('UPDATE _warehouse.derived_status SET pending = GREATEST(pending - $1, 0) WHERE dataset = $2', [count, this.#dataset]);
46
+ }
47
+ current(counts) {
48
+ return this.#write({
49
+ state: 'current',
50
+ as_of: this.#since,
51
+ refreshed_at: this.#now(),
52
+ refreshing_since: null,
53
+ error: null,
54
+ ...counts,
55
+ });
56
+ }
57
+ failed(error) {
58
+ return this.#write({ state: 'failed', refreshing_since: null, error });
59
+ }
60
+ // In a transaction of its own, which takes no commit number: the status is
61
+ // no data of a position.
62
+ async #write(values) {
63
+ const columns = ['dataset', 'kind', ...Object.keys(values)];
64
+ const placeholders = columns.map((_, index) => `$${index + 1}`);
65
+ const assignments = columns
66
+ .slice(1)
67
+ .map((column) => `${column} = excluded.${column}`);
68
+ await this.#sql.begin(async (sql) => {
69
+ await lockSchema(sql, '_warehouse');
70
+ await sql.unsafe(derivedStatusTable);
71
+ await sql.unsafe(`INSERT INTO _warehouse.derived_status (${columns.join(', ')}) VALUES (${placeholders.join(', ')}) ON CONFLICT (dataset) DO UPDATE SET ${assignments.join(', ')}`, [this.#dataset, this.#kind, ...Object.values(values)]);
72
+ });
73
+ }
74
+ #now() {
75
+ return new Date(this.#clock.Date.now()).toISOString();
76
+ }
77
+ }
@@ -28,4 +28,4 @@ export declare const syncHistoryTables: readonly ['CREATE SCHEMA IF NOT EXISTS _
28
28
  )`, `CREATE TABLE IF NOT EXISTS _warehouse.stream_checks (
29
29
  connector text NOT NULL, stream text NOT NULL, checked_at timestamptz NOT NULL,
30
30
  PRIMARY KEY (connector, stream)
31
- )`];
31
+ )`, "CREATE TABLE IF NOT EXISTS _warehouse.derived_status (\n dataset text PRIMARY KEY,\n kind text NOT NULL CHECK (kind IN ('view', 'computed')),\n state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),\n as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,\n inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint\n )"];
@@ -1,4 +1,5 @@
1
1
  import { syncHistoryRelations } from '@eltjs/elt';
2
+ import { derivedStatusTable } from './postgres-derived-status.js';
2
3
  // When connector's stream last matched its upstream: the start of its last
3
4
  // successful pass, or a later check that found it unchanged, each no later
4
5
  // than when the source said the upstream itself was last current. NULL if it
@@ -52,6 +53,7 @@ const queries = {
52
53
  ORDER BY s.completed_at DESC, s.id DESC LIMIT 1
53
54
  ) success ON true
54
55
  ORDER BY a.connector, c.stream, a.id DESC`,
56
+ derived_status: `SELECT dataset, kind, state, as_of, refreshed_at, refreshing_since, error, inputs, computed, empty, failed, pending FROM _warehouse.derived_status ORDER BY dataset`,
55
57
  };
56
58
  // How readers see the sync history: what each pass declared and loaded.
57
59
  // The sync history keeps no row versions, so a reading as of a position
@@ -97,4 +99,5 @@ export const syncHistoryTables = [
97
99
  connector text NOT NULL, stream text NOT NULL, checked_at timestamptz NOT NULL,
98
100
  PRIMARY KEY (connector, stream)
99
101
  )`,
102
+ derivedStatusTable,
100
103
  ];
@@ -1,6 +1,6 @@
1
1
  import { hash } from 'node:crypto';
2
2
  import { isDeepStrictEqual } from 'node:util';
3
- import { FileContent, TargetOwnedError, Writer, describeTarget, errorMessage, reloadMode, } from '@eltjs/elt';
3
+ import { FileContent, TargetOwnedError, Writer, describeTarget, errorMessage, readPartitions, reloadMode, } from '@eltjs/elt';
4
4
  import { quote } from './identifier.js';
5
5
  import { TargetComments } from './postgres-comments.js';
6
6
  import { rebindDependents } from './postgres-dependents.js';
@@ -9,7 +9,7 @@ import { storedColumns, storedFit } from './postgres-fit.js';
9
9
  import { TargetHistory, rowHash, version } from './postgres-history.js';
10
10
  import { commitOnce, expireBefore, installNumbering, } from './postgres-numbering.js';
11
11
  import { TargetReaderView } from './postgres-reader-view.js';
12
- import { PostgresSession, lockSchema, makeSchema } from './postgres-session.js';
12
+ import { PostgresSession, lockSchema, makeSchema, } from './postgres-session.js';
13
13
  const batchSize = 1000;
14
14
  // A batch also flushes once its JSON reaches this size, so rows of large text
15
15
  // or binary values are sent a few at a time rather than a thousand at once.
@@ -224,6 +224,7 @@ export class PostgresWriter extends Writer {
224
224
  }
225
225
  // Clearing a copy also abandons the reload it left open.
226
226
  await transaction.unsafe(`DROP TABLE IF EXISTS ${this.#hidden}`);
227
+ await this.#reloaded(transaction);
227
228
  for (const column of this.table.columns.filter((column) => column.storesFile)) {
228
229
  const store = new PostgresFileStore(this.schema, this.table, column);
229
230
  const [chunks] = await transaction.unsafe('SELECT to_regclass($1) IS NOT NULL AS "exists"', [store.qualifiedName]);
@@ -326,13 +327,28 @@ export class PostgresWriter extends Writer {
326
327
  // keeps its indexes, its triggers and what depends on it. Each row of the
327
328
  // hidden target that no row matches enters with this commit, and the
328
329
  // loaded_at of the load that wrote it.
329
- async #reloadInPlace(sql, committedIn) {
330
- const target = { table: this.qualifiedName, id: 'ctid' };
330
+ async #reloadInPlace(sql, committedIn, target = { table: this.qualifiedName, id: 'ctid' }) {
331
331
  const reloaded = { table: this.#hidden, id: 'ctid' };
332
332
  await this.#history.remove(sql, committedIn, this.qualifiedName, ` WHERE "_elt_old".ctid IN (${this.#unmatched(target, reloaded)})`);
333
333
  await sql.unsafe(`INSERT INTO ${this.qualifiedName} (${this.written.join(', ')}) SELECT ${this.table.columns.map((column) => column.quotedName).join(', ')}, "loaded_at", $1::integer, ${version} FROM ${this.#hidden} WHERE ctid IN (${this.#unmatched(reloaded, target)}) ORDER BY ctid`, [committedIn]);
334
334
  await sql.unsafe(`DROP TABLE ${this.#hidden}`);
335
335
  }
336
+ // Gives a completed reload of some partitions their rows of the hidden
337
+ // target, in place and by difference: the rows of the target in those
338
+ // partitions are replaced as #reloadInPlace replaces a whole target, and
339
+ // the rows of every other partition stay as they are.
340
+ async #reloadPartitionsInPlace(sql, partitions, committedIn) {
341
+ const scoped = quote(`_elt_reloaded_${this.#hash}`);
342
+ const [within, values] = this.#within(partitions);
343
+ await sql.unsafe(`DROP TABLE IF EXISTS pg_temp.${scoped}`);
344
+ await sql.unsafe(`CREATE TEMP TABLE ${scoped} AS SELECT ctid AS "row" FROM ${this.qualifiedName} WHERE ${within}`, values);
345
+ await this.#reloadInPlace(sql, committedIn, {
346
+ table: this.qualifiedName,
347
+ id: 'ctid',
348
+ rows: `ctid IN (SELECT "row" FROM pg_temp.${scoped})`,
349
+ });
350
+ await sql.unsafe(`DROP TABLE pg_temp.${scoped}`);
351
+ }
336
352
  // Refuses a cast of column from its stored type from that would change a
337
353
  // stored value, such as double precision 1.5 to bigint 2, before the ALTER,
338
354
  // as a STRICT SQLite table refuses it: each value that is not null must
@@ -437,6 +453,23 @@ export class PostgresWriter extends Writer {
437
453
  #scope(partition) {
438
454
  if (partition === null)
439
455
  return ['', []];
456
+ const [condition, values] = this.#within([partition]);
457
+ return [` WHERE ${condition}`, values];
458
+ }
459
+ // The rows of any of the partitions, as a condition over the target's or
460
+ // the stage's columns, with one JSON parameter for each partition: NULL for
461
+ // a row that does not name its partition, such as a staged DELETE whose
462
+ // key leaves the partition out.
463
+ #within(partitions) {
464
+ const each = partitions.map((partition, index) => this.#partition(partition, `$${index + 1}::text::json`));
465
+ return [
466
+ each.length === 0
467
+ ? 'FALSE'
468
+ : each.map(([condition]) => `(${condition})`).join(' OR '),
469
+ each.map(([, value]) => value),
470
+ ];
471
+ }
472
+ #partition(partition, parameter) {
440
473
  const columns = Object.keys(partition).map((field) => {
441
474
  const column = this.table.columns.find((column) => column.field === field);
442
475
  if (column === undefined)
@@ -444,10 +477,45 @@ export class PostgresWriter extends Writer {
444
477
  return column;
445
478
  });
446
479
  return [
447
- ` WHERE ${columns.map((column, index) => `${column.quotedName} = ${column.valueFrom('$1::text::json', index)}`).join(' AND ')}`,
448
- [JSON.stringify(columns.map((column) => column.encode(partition)))],
480
+ columns
481
+ .map((column, index) => `${column.quotedName} = ${column.valueFrom(parameter, index)}`)
482
+ .join(' AND '),
483
+ JSON.stringify(columns.map((column) => column.encode(partition))),
449
484
  ];
450
485
  }
486
+ get #reloads() {
487
+ return `${quote(this.schema)}."_elt_reloads"`;
488
+ }
489
+ // The partitions that an open reload of the target reloads, in the
490
+ // writer's own table beside it, written in the commit that opens or
491
+ // continues the reload, so a run that resumes the reload continues it. A
492
+ // reload of the whole stream has no row.
493
+ async #reloading(sql) {
494
+ const [made] = await sql.unsafe('SELECT to_regclass($1) IS NOT NULL AS "exists"', [this.#reloads]);
495
+ if (made?.exists !== true)
496
+ return wholeStream;
497
+ const [row] = await sql.unsafe(`SELECT "partitions" FROM ${this.#reloads} WHERE "target" = $1`, [this.table.name]);
498
+ return row === undefined
499
+ ? wholeStream
500
+ : new SomePartitions(readPartitions(this.stream, JSON.parse(row.partitions)));
501
+ }
502
+ async #reload(sql, reloads) {
503
+ if (!(reloads instanceof SomePartitions)) {
504
+ await this.#reloaded(sql);
505
+ return;
506
+ }
507
+ // The first reloads of the schema's tables make the table together.
508
+ await lockSchema(sql, this.schema);
509
+ await sql.unsafe(`CREATE TABLE IF NOT EXISTS ${this.#reloads} ("target" TEXT PRIMARY KEY, "partitions" TEXT NOT NULL)`);
510
+ await sql.unsafe(`INSERT INTO ${this.#reloads} ("target", "partitions") VALUES ($1, $2) ON CONFLICT ("target") DO UPDATE SET "partitions" = excluded."partitions"`, [this.table.name, JSON.stringify(reloads.partitions)]);
511
+ }
512
+ async #reloaded(sql) {
513
+ const [made] = await sql.unsafe('SELECT to_regclass($1) IS NOT NULL AS "exists"', [this.#reloads]);
514
+ if (made?.exists === true)
515
+ await sql.unsafe(`DELETE FROM ${this.#reloads} WHERE "target" = $1`, [
516
+ this.table.name,
517
+ ]);
518
+ }
451
519
  // One stream's load. Operations wait in a session-private TEMP stage, so no
452
520
  // other session sees them and a crash leaves nothing behind. prepare only
453
521
  // refuses, evolves a stored table the stream no longer fits unless the load
@@ -461,13 +529,21 @@ export class PostgresWriter extends Writer {
461
529
  async prepare(load, { writer, reloading }) {
462
530
  const { sql } = load;
463
531
  const stage = quote(`_elt_stage_${this.#hash}`);
532
+ // The rows of a commit that belong to the partitions a reload reloads.
533
+ const reloadStage = quote(`_elt_reload_stage_${this.#hash}`);
534
+ const staging = (table) => `CREATE TEMP TABLE ${table} (${seq} BIGINT PRIMARY KEY, ${op} TEXT NOT NULL, ${this.table.columns.map((column) => `${column.quotedName} ${column.storageType}`).join(', ')}, ${version} TEXT)`;
535
+ // What an open reload reloads; the whole stream until a RESET of a
536
+ // partition opens one.
537
+ let reloads = wholeStream;
464
538
  const stores = this.table.columns
465
539
  .filter((column) => column.storesFile)
466
540
  .map((column) => new PostgresFileStore(this.schema, this.table, column));
467
541
  let mode = await load.transaction(async (sql) => {
468
542
  await this.#refuse(sql, writer);
469
543
  await sql.unsafe(`DROP TABLE IF EXISTS pg_temp.${stage}`);
470
- await sql.unsafe(`CREATE TEMP TABLE ${stage} (${seq} BIGINT PRIMARY KEY, ${op} TEXT NOT NULL, ${this.table.columns.map((column) => `${column.quotedName} ${column.storageType}`).join(', ')}, ${version} TEXT)`);
544
+ await sql.unsafe(staging(stage));
545
+ await sql.unsafe(`DROP TABLE IF EXISTS pg_temp.${reloadStage}`);
546
+ await sql.unsafe(staging(reloadStage));
471
547
  for (const store of stores)
472
548
  await store.stage(sql);
473
549
  const target = await storedFit(sql, this.table, this.qualifiedName, stage);
@@ -479,8 +555,12 @@ export class PostgresWriter extends Writer {
479
555
  hidden,
480
556
  });
481
557
  // A hidden target no reload continues is a leftover readers never saw.
482
- if (decided !== 'continue')
558
+ if (decided === 'continue')
559
+ reloads = await this.#reloading(sql);
560
+ else {
483
561
  await sql.unsafe(`DROP TABLE IF EXISTS ${this.#hidden}`);
562
+ await this.#reloaded(sql);
563
+ }
484
564
  await this.#view?.refuse(sql, target === 'stale');
485
565
  if (decided === 'evolve') {
486
566
  await this.#evolve(sql, stage, load);
@@ -491,6 +571,11 @@ export class PostgresWriter extends Writer {
491
571
  return decided;
492
572
  });
493
573
  const fresh = mode === 'create' || mode === 'reload';
574
+ // The reload as the last commit left it, which a discard goes back to.
575
+ let committed = {
576
+ mode,
577
+ reloads,
578
+ };
494
579
  const open = () => mode === 'reload' || mode === 'continue';
495
580
  const into = () => (open() ? this.#hidden : this.qualifiedName);
496
581
  const tables = () => [
@@ -565,11 +650,23 @@ export class PostgresWriter extends Writer {
565
650
  // The whole stream starts over in a new hidden target.
566
651
  await drop();
567
652
  mode = 'reload';
653
+ reloads = wholeStream;
568
654
  return;
569
655
  }
570
656
  const [rows, values] = this.#scope(partition);
571
657
  await sql.unsafe(`DELETE FROM ${stage}${rows}`, values);
572
658
  resets.push(partition);
659
+ // The partition starts over in a hidden target, and readers see its
660
+ // rows as they were until the stream ends. An overwrite replaces
661
+ // the whole target anyway.
662
+ if (this.replaces)
663
+ return;
664
+ if (mode === 'load') {
665
+ mode = 'reload';
666
+ reloads = new SomePartitions([partition]);
667
+ }
668
+ else if (open() && reloads instanceof SomePartitions)
669
+ reloads = reloads.with(partition);
573
670
  return;
574
671
  }
575
672
  // Each file is read here, before the source advances past it. The
@@ -609,6 +706,8 @@ export class PostgresWriter extends Writer {
609
706
  await sql.unsafe(`CREATE TABLE ${this.#tableSQL(this.#hidden)}`);
610
707
  await this.build(sql, this.#hidden);
611
708
  }
709
+ if (open())
710
+ await this.#reload(sql, reloads);
612
711
  const replacing = await claiming.claim(sql, committedIn);
613
712
  // The rows of a hidden target were never shown, so only the target
614
713
  // keeps the rows a commit replaces.
@@ -625,6 +724,18 @@ export class PostgresWriter extends Writer {
625
724
  const written = { loadedAt: load.loadedAt, committedIn };
626
725
  if (replacing)
627
726
  await this.#replace(sql, stage, written);
727
+ else if (open() && reloads instanceof SomePartitions) {
728
+ // The rows of the reloaded partitions go to the hidden target,
729
+ // the others to the target. A DELETE whose key does not name
730
+ // its partition goes to both, so the reload cannot bring back a
731
+ // row that the source deleted.
732
+ const [within, values] = this.#within(reloads.partitions);
733
+ await sql.unsafe(`INSERT INTO ${reloadStage} SELECT * FROM ${stage} WHERE (${within}) IS TRUE OR (${op} = 'D' AND (${within}) IS NULL)`, values);
734
+ await sql.unsafe(`DELETE FROM ${stage} WHERE (${within}) IS TRUE`, values);
735
+ await this.merge(sql, stage, written, this.qualifiedName, this.#history);
736
+ await this.merge(sql, reloadStage, written, this.#hidden, null);
737
+ await sql.unsafe(`TRUNCATE ${reloadStage}`);
738
+ }
628
739
  else
629
740
  await this.merge(sql, stage, written, into(), history);
630
741
  await sql.unsafe(`TRUNCATE ${stage}`);
@@ -636,6 +747,7 @@ export class PostgresWriter extends Writer {
636
747
  // Its first commit made the hidden target, so the reload continues.
637
748
  if (mode === 'reload')
638
749
  mode = 'continue';
750
+ committed = { mode, reloads };
639
751
  resets = [];
640
752
  },
641
753
  complete: async () => {
@@ -643,10 +755,17 @@ export class PostgresWriter extends Writer {
643
755
  return;
644
756
  await load.commit(async (sql, committedIn) => {
645
757
  const target = await storedFit(sql, this.table, this.qualifiedName, stage);
646
- if (target === 'fits')
758
+ // A reload of partitions keeps the target's shape: prepare evolves
759
+ // a stored target before it loads.
760
+ if (reloads instanceof SomePartitions && target === 'fits')
761
+ await this.#reloadPartitionsInPlace(sql, reloads.partitions, committedIn);
762
+ else if (reloads instanceof SomePartitions)
763
+ throw new TypeError(`${this.qualifiedName} changed its shape while some of its partitions reloaded; clear the copy to load it again`);
764
+ else if (target === 'fits')
647
765
  await this.#reloadInPlace(sql, committedIn);
648
766
  else
649
767
  await this.#swap(sql, committedIn);
768
+ await this.#reloaded(sql);
650
769
  await this.#history.expire(sql);
651
770
  for (const store of stores)
652
771
  await store.prune(sql, [
@@ -655,14 +774,40 @@ export class PostgresWriter extends Writer {
655
774
  ]);
656
775
  });
657
776
  mode = 'load';
777
+ reloads = wholeStream;
778
+ committed = { mode, reloads };
779
+ },
780
+ // A RESET that no commit took is undone with the operations after it:
781
+ // the source's checkpoint does not know it.
782
+ discard: async () => {
783
+ await drop();
784
+ ({ mode, reloads } = committed);
658
785
  },
659
- discard: drop,
660
786
  [Symbol.asyncDispose]: async () => {
661
787
  await drop();
662
788
  await sql.unsafe(`DROP TABLE pg_temp.${stage}`);
789
+ await sql.unsafe(`DROP TABLE pg_temp.${reloadStage}`);
663
790
  for (const store of stores)
664
791
  await store.unstage(sql);
665
792
  },
666
793
  };
667
794
  }
668
795
  }
796
+ // What an open reload reloads: the whole stream, or some partitions of it,
797
+ // while the others load into the target as before, as Delta Lake's
798
+ // replaceWhere replaces only the rows that its condition selects.
799
+ class WholeStream {
800
+ }
801
+ class SomePartitions {
802
+ partitions;
803
+ constructor(partitions) {
804
+ this.partitions = Object.freeze([...partitions]);
805
+ }
806
+ // These partitions and partition, which a later RESET reloads too.
807
+ with(partition) {
808
+ return this.partitions.some((other) => isDeepStrictEqual(other, partition))
809
+ ? this
810
+ : new SomePartitions([...this.partitions, partition]);
811
+ }
812
+ }
813
+ const wholeStream = new WholeStream();
package/dist/view.d.ts CHANGED
@@ -1,4 +1,4 @@
1
- import { type Connection, DerivedDataset } from '@eltjs/elt';
1
+ import { type Connection, DerivedDataset, type PipelineClock } from '@eltjs/elt';
2
2
  import { PostgresDestination } from './postgres-destination.ts';
3
3
  import type { PostgresTable } from './postgres-table.ts';
4
4
  export declare class View extends DerivedDataset<PostgresTable> {
@@ -14,9 +14,10 @@ export declare class View extends DerivedDataset<PostgresTable> {
14
14
  });
15
15
  get qualifiedName(): string;
16
16
  validate(connections: readonly Connection<PostgresTable>[]): void;
17
- refresh({ connections, earlier, }: {
17
+ refresh({ connections, earlier, clock, }: {
18
18
  readonly connections: readonly Connection<PostgresTable>[];
19
19
  readonly earlier: readonly DerivedDataset<PostgresTable>[];
20
+ readonly clock: PipelineClock;
20
21
  }): Promise<void>;
21
22
  }
22
23
  export declare function made(connections: readonly Connection<PostgresTable>[], into: PostgresDestination): Set<string>;
package/dist/view.js CHANGED
@@ -1,5 +1,6 @@
1
1
  import postgres from 'postgres';
2
- import { DerivedDataset } from '@eltjs/elt';
2
+ import { DerivedDataset, errorMessage, } from '@eltjs/elt';
3
+ import { PostgresDerivedStatus } from './postgres-derived-status.js';
3
4
  import { PostgresDestination } from './postgres-destination.js';
4
5
  import { PostgresSession } from './postgres-session.js';
5
6
  import { checkView, publishPostgresViews, } from './postgres-views.js';
@@ -33,8 +34,9 @@ export class View extends DerivedDataset {
33
34
  if (made(connections, this.into).has(this.qualifiedName))
34
35
  throw new TypeError(`View ${this.qualifiedName} has the name of a table or reader view a copy makes`);
35
36
  }
36
- async refresh({ connections, earlier, }) {
37
+ async refresh({ connections, earlier, clock, }) {
37
38
  await using session = new PostgresSession(this.into.url, 'elt-derived');
39
+ const status = new PostgresDerivedStatus(session.sql, this.qualifiedName, 'view', clock);
38
40
  try {
39
41
  await session.sql.begin((transaction) => publishPostgresViews(transaction, {
40
42
  schema: this.into.schema,
@@ -45,10 +47,14 @@ export class View extends DerivedDataset {
45
47
  const missing = missingRelation(error, this.into.schema);
46
48
  if (missing !== undefined &&
47
49
  (made(connections, this.into).has(missing) ||
48
- earlier.some((dataset) => derivedRelation(dataset) === missing)))
50
+ earlier.some((dataset) => derivedRelation(dataset) === missing))) {
51
+ await status.waiting();
49
52
  return;
53
+ }
54
+ await status.failed(errorMessage(error));
50
55
  throw error;
51
56
  }
57
+ await status.current();
52
58
  }
53
59
  }
54
60
  // The schema-qualified relation that Postgres found missing, with a bare name
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@eltjs/elt-postgresql",
3
- "version": "0.3.14",
3
+ "version": "0.3.16",
4
4
  "description": "The Postgres destination of elt: each stream is a typed table in one schema, with a reader view in another, a history of each row for readings as of a position, and the views and computed datasets that a pipeline derives.",
5
5
  "license": "Apache-2.0",
6
6
  "repository": {
@@ -37,7 +37,7 @@
37
37
  "access": "public"
38
38
  },
39
39
  "dependencies": {
40
- "@eltjs/elt": "^0.3.14",
40
+ "@eltjs/elt": "^0.3.16",
41
41
  "@zukhruf/async": "^0.8.27",
42
42
  "postgres": "^3.4.9"
43
43
  },