@eltjs/elt-postgresql 0.3.14 → 0.3.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +22 -9
- package/dist/computed-dataset.js +40 -2
- package/dist/postgres-derived-status.d.ts +20 -0
- package/dist/postgres-derived-status.js +77 -0
- package/dist/postgres-sync-history-schema.d.ts +1 -1
- package/dist/postgres-sync-history-schema.js +3 -0
- package/dist/postgres-writer.js +155 -10
- package/dist/view.d.ts +3 -2
- package/dist/view.js +9 -3
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -160,7 +160,7 @@ How a reader knows which numbers are done:
|
|
|
160
160
|
|
|
161
161
|
The history of a table:
|
|
162
162
|
|
|
163
|
-
- `<schema>._elt_history_<hash>` keeps each row that a commit replaced or removed, with `replaced_in`, the number of that commit, as a SQL Server temporal table keeps its history beside the current table. The writes that keep rows are a deduplicating upsert that changes the values (behind the same cursor guard), a staged `DELETE`,
|
|
163
|
+
- `<schema>._elt_history_<hash>` keeps each row that a commit replaced or removed, with `replaced_in`, the number of that commit, as a SQL Server temporal table keeps its history beside the current table. The writes that keep rows are a deduplicating upsert that changes the values (behind the same cursor guard), a staged `DELETE`, the end of the reload of a reset partition, an overwrite, a reload, and a change of shape. `<schema>._elt_reloads` names the partitions of each open reload of a reset partition. A write into a hidden table keeps nothing, because no reader saw its rows.
|
|
164
164
|
- `clear()` forgets the rows that it removes. Its commit C sets `min_valid` to C, if it is higher, with `expired_by` `clear`, and the history of the cleared table deletes each of its rows, with the stored files that only they refer to. A reading or `positions(after)` before C gives `{ expired }`, with the reason that a clear forgot the rows of its table. The history of another table keeps its rows with `replaced_in <= C` until its next commit or the retention deletes them; a reading skips them, as for a change of type.
|
|
165
165
|
- Each column of the history has the type of the table's column of the same name. It takes each new column of the table. A column that the table drops stays in the history with its last type.
|
|
166
166
|
- When a change of shape, or the swap of a reload, gives a column of the table another type, the history changes the type of its column in the same commit. This includes a column that the table dropped and a later change adds back with another type. Delta Lake, Iceberg, BigQuery and Snowflake change a type in place only when the change is a widening, and they read the past with its own schema, which one Postgres view cannot do. The destination follows the same rule:
|
|
@@ -291,7 +291,7 @@ A refresh is a load in `_warehouse._elt_loads`, under the name of its dataset, s
|
|
|
291
291
|
|
|
292
292
|
- It records only connections with a Postgres destination. It refuses other connections when the pipeline validates.
|
|
293
293
|
- It records into `_warehouse.sync_attempts`, `_warehouse.extraction_coverage` and `_warehouse.stream_checks`. The last one holds the latest check of each stream by a watch that found it unchanged.
|
|
294
|
-
- `install()` makes those tables and publishes the `marts` views `sync_attempts`, `extraction_coverage`, `sync_status`, `stream_status` and `catalog`, in one transaction. The pipeline calls it before each run and watch, and it is safe to repeat. See [ADR 0005](./docs/adr/0005-the-history-install-publishes-the-catalog.md).
|
|
294
|
+
- `install()` makes those tables and `_warehouse.derived_status`, and publishes the `marts` views `sync_attempts`, `extraction_coverage`, `sync_status`, `stream_status`, `derived_status` and `catalog`, in one transaction. The pipeline calls it before each run and watch, and it is safe to repeat. See [ADR 0005](./docs/adr/0005-the-history-install-publishes-the-catalog.md).
|
|
295
295
|
- `new PostgresSyncHistory({ url, readers: ['agent_reader'] })` names the roles that read the database as of a position with their own session: `new PostgresDestination({ url: readerUrl, schema })`, then `positions()` and `reading()`. Each install grants each reader what a reading needs, which is the numbering in `_warehouse` and nothing else of it, as dbt applies the grants that a model declares after each build. Thus no host names the objects of the numbering, and a reader keeps its grants when the numbering is made again. The install grants nothing else: who may read `marts` is a grant of the database, such as its default privileges. A role that does not exist fails the install.
|
|
296
296
|
- Before a pass reads, the history inserts one `sync_attempts` row, with the connection name as `connector` and the source identity as `source`. It inserts one `extraction_coverage` row for each selected stream, with the declared `coverage(stream)` of the source and the schema of the destination as `target_schema`. The outcomes close them.
|
|
297
297
|
- A run declares each stream that a connection selected. A watch pass reads only the streams whose change signals changed, so an attempt vouches only for its own `extraction_coverage` rows.
|
|
@@ -300,13 +300,14 @@ A refresh is a load in `_warehouse._elt_loads`, under the name of its dataset, s
|
|
|
300
300
|
|
|
301
301
|
The relations in `marts`:
|
|
302
302
|
|
|
303
|
-
| Relation | Contents
|
|
304
|
-
| --------------------- |
|
|
305
|
-
| `catalog` | Each view and column in `marts`, with its description. The start point of a reader. A table is not for readers, also when it is in `marts`, such as the raw table of a computed dataset.
|
|
306
|
-
| `sync_status` | For each connection: the latest attempt, the end of the last successful pass, and `current_as_of` (the oldest of its streams).
|
|
307
|
-
| `stream_status` | For each connection and stream: the latest attempt that declared the stream with the status of its copy, and the last attempt in which that copy succeeded. A watch pass reads only changed streams, so judge a stream here, by `current_as_of`.
|
|
308
|
-
| `
|
|
309
|
-
| `
|
|
303
|
+
| Relation | Contents |
|
|
304
|
+
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
305
|
+
| `catalog` | Each view and column in `marts`, with its description. The start point of a reader. A table is not for readers, also when it is in `marts`, such as the raw table of a computed dataset. |
|
|
306
|
+
| `sync_status` | For each connection: the latest attempt, the end of the last successful pass, and `current_as_of` (the oldest of its streams). |
|
|
307
|
+
| `stream_status` | For each connection and stream: the latest attempt that declared the stream with the status of its copy, and the last attempt in which that copy succeeded. A watch pass reads only changed streams, so judge a stream here, by `current_as_of`. |
|
|
308
|
+
| `derived_status` | For each view and computed dataset, named with its schema: whether it waits for its inputs, refreshes, is current or failed, its `as_of`, and for a computed dataset how many inputs it computed, found empty, failed on or has still to compute. See [derived status](#derived-status). |
|
|
309
|
+
| `sync_attempts` | The kept history of attempts, with start and end times, status and errors. |
|
|
310
|
+
| `extraction_coverage` | For each attempt and stream: the configured selection, the explanation of the scope, the target, the outcome of the copy, the committed counts and the failed partitions. |
|
|
310
311
|
|
|
311
312
|
What the relations tell:
|
|
312
313
|
|
|
@@ -319,6 +320,17 @@ What the relations tell:
|
|
|
319
320
|
- A first failure has no time of success.
|
|
320
321
|
- Coverage is the [configured scope](../../elt/README.md#coverage). Judge freshness by `current_as_of` ([the four clocks](../../elt/README.md#the-four-clocks)).
|
|
321
322
|
|
|
323
|
+
## Derived status
|
|
324
|
+
|
|
325
|
+
Each [view](#declare-a-view) and [computed dataset](#declare-a-computed-dataset) keeps one row in `_warehouse.derived_status`, which readers see as `marts.derived_status`. A reader needs it because `sync_status` says that a pass succeeded while a computed dataset still computes, or failed on each of its inputs.
|
|
326
|
+
|
|
327
|
+
- A refresh writes `waiting` when a relation that the dataset reads is not made yet, and `refreshing` when it begins, with `refreshing_since`. A computed dataset also writes how many inputs it read and how many it has still to compute (`pending`). Each commit of the refresh lowers `pending` in its own transaction, so a reader sees it fall while the rows land.
|
|
328
|
+
- A refresh that ends with no failure of the dataset writes `current`, with `as_of`, when it began to read, and `refreshed_at`. A refresh that fails writes `failed`, with `error`. A rule that fails on one input is not a failure of the dataset: `failed` counts its rows.
|
|
329
|
+
- `computed`, `empty` and `failed` count the output rows when the refresh began and when it ended.
|
|
330
|
+
- A dataset is caught up when its `state` is `current`, its `pending` is 0, and its `as_of` is at or after the start of the last pass of the connections that it reads. This is the frontier of Materialize and the `DATA_TIMESTAMP` of a Snowflake dynamic table, with the pending items of a pgai vectorizer.
|
|
331
|
+
- A refresh that a stopped process left stays `refreshing` until the next refresh. The status keeps no row versions, so a reading as of a position fails the view, as it fails the other views of the sync history.
|
|
332
|
+
- A status write takes no commit number: it is not data of a position.
|
|
333
|
+
|
|
322
334
|
## Decisions
|
|
323
335
|
|
|
324
336
|
- [ADR 0001](./docs/adr/0001-checkpoints-are-kept-beside-the-data.md): checkpoints are kept beside the data.
|
|
@@ -354,6 +366,7 @@ src/
|
|
|
354
366
|
postgres-views.ts the publication of views, for View and the sync history
|
|
355
367
|
postgres-dependents.ts the views over a table, made again over its change
|
|
356
368
|
postgres-privileges.ts the privileges of a relation, kept across its change
|
|
369
|
+
postgres-derived-status.ts the status that each derived dataset keeps of itself
|
|
357
370
|
postgres-checkpoint-store.ts PostgresCheckpointStore
|
|
358
371
|
postgres-sync-history.ts PostgresSyncHistory and the catalog
|
|
359
372
|
postgres-sync-history-schema.ts the tables and views of the history
|
package/dist/computed-dataset.js
CHANGED
|
@@ -1,9 +1,10 @@
|
|
|
1
1
|
import { hash } from 'node:crypto';
|
|
2
2
|
import { Lock } from '@zukhruf/async';
|
|
3
|
-
import { DerivedDataset, } from '@eltjs/elt';
|
|
3
|
+
import { DerivedDataset, errorMessage, } from '@eltjs/elt';
|
|
4
4
|
import { identifier, quote } from './identifier.js';
|
|
5
5
|
import { TargetComments } from './postgres-comments.js';
|
|
6
6
|
import { rebindDependents } from './postgres-dependents.js';
|
|
7
|
+
import { PostgresDerivedStatus } from './postgres-derived-status.js';
|
|
7
8
|
import { PostgresDestination } from './postgres-destination.js';
|
|
8
9
|
import { storedColumns } from './postgres-fit.js';
|
|
9
10
|
import { TargetHistory, rowHash } from './postgres-history.js';
|
|
@@ -122,9 +123,25 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
122
123
|
// Open from the start, so it closes a refresh that a process ended; with
|
|
123
124
|
// no commit it is no span, and leaves the positions as they are.
|
|
124
125
|
await using load = await PostgresLoad.open(this.into.url, { dataset: this.name }, { clock, retention: this.into.retention });
|
|
126
|
+
const status = new PostgresDerivedStatus(load.sql, this.qualifiedName, 'computed', clock);
|
|
125
127
|
const inputs = await this.#inputs(load.sql, connections);
|
|
126
|
-
if (inputs === null)
|
|
128
|
+
if (inputs === null) {
|
|
129
|
+
await status.waiting();
|
|
127
130
|
return;
|
|
131
|
+
}
|
|
132
|
+
try {
|
|
133
|
+
await this.#update(load, status, inputs, trigger, signal);
|
|
134
|
+
}
|
|
135
|
+
catch (error) {
|
|
136
|
+
// A stop is not the dataset's failure.
|
|
137
|
+
if (signal?.aborted !== true)
|
|
138
|
+
await status.failed(errorMessage(error));
|
|
139
|
+
throw error;
|
|
140
|
+
}
|
|
141
|
+
}
|
|
142
|
+
// Brings the output rows in step with inputs, and tells status how many
|
|
143
|
+
// are due and how many settled.
|
|
144
|
+
async #update(load, status, inputs, trigger, signal) {
|
|
128
145
|
const key = quote(this.#rule.key.name);
|
|
129
146
|
const table = this.#qualified(this.#table);
|
|
130
147
|
// The number of a transaction's commit, taken at its first change, so a
|
|
@@ -214,6 +231,10 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
214
231
|
trigger.tried.add(attempt);
|
|
215
232
|
due.push({ id, input, inputHash, rule });
|
|
216
233
|
}
|
|
234
|
+
await status.refreshing({
|
|
235
|
+
...(await this.#counts(load.sql, inputs)),
|
|
236
|
+
pending: due.length,
|
|
237
|
+
});
|
|
217
238
|
// A pool of calls, each taking the next batch once its own committed, so
|
|
218
239
|
// a slow call holds only its place, as a worker pool keeps its other
|
|
219
240
|
// workers busy. A call that fails stops the pool taking more.
|
|
@@ -247,11 +268,28 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
247
268
|
else
|
|
248
269
|
await store(sql, commit, id, inputHash, rule, 'empty', null, {});
|
|
249
270
|
}
|
|
271
|
+
await status.settled(sql, batch.length);
|
|
250
272
|
});
|
|
251
273
|
}
|
|
252
274
|
}));
|
|
253
275
|
if (failures.length > 0)
|
|
254
276
|
throw failures[0];
|
|
277
|
+
if (signal?.aborted === true)
|
|
278
|
+
return;
|
|
279
|
+
await status.current({
|
|
280
|
+
...(await this.#counts(load.sql, inputs)),
|
|
281
|
+
pending: 0,
|
|
282
|
+
});
|
|
283
|
+
}
|
|
284
|
+
// How many inputs there are, and how many output rows have each status.
|
|
285
|
+
async #counts(sql, inputs) {
|
|
286
|
+
const counted = new Map((await sql.unsafe(`SELECT "status", count(*)::int AS "n" FROM ${this.#qualified(this.#table)} GROUP BY "status"`)).map(({ status, n }) => [status, n]));
|
|
287
|
+
return {
|
|
288
|
+
inputs: inputs.size,
|
|
289
|
+
computed: counted.get('computed') ?? 0,
|
|
290
|
+
empty: counted.get('empty') ?? 0,
|
|
291
|
+
failed: counted.get('failed') ?? 0,
|
|
292
|
+
};
|
|
255
293
|
}
|
|
256
294
|
// Makes the table, its history and its view, or makes the table again when
|
|
257
295
|
// its declaration changed: the history keeps each row of the old table, in
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
import type postgres from 'postgres';
|
|
2
|
+
import type { PipelineClock } from '@eltjs/elt';
|
|
3
|
+
import { type Queries } from './postgres-session.ts';
|
|
4
|
+
export declare const derivedStatusTable = "CREATE TABLE IF NOT EXISTS _warehouse.derived_status (\n dataset text PRIMARY KEY,\n kind text NOT NULL CHECK (kind IN ('view', 'computed')),\n state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),\n as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,\n inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint\n )";
|
|
5
|
+
export type DerivedCounts = {
|
|
6
|
+
readonly inputs: number;
|
|
7
|
+
readonly computed: number;
|
|
8
|
+
readonly empty: number;
|
|
9
|
+
readonly failed: number;
|
|
10
|
+
readonly pending: number;
|
|
11
|
+
};
|
|
12
|
+
export declare class PostgresDerivedStatus {
|
|
13
|
+
#private;
|
|
14
|
+
constructor(sql: postgres.Sql, dataset: string, kind: 'view' | 'computed', clock: PipelineClock);
|
|
15
|
+
waiting(): Promise<void>;
|
|
16
|
+
refreshing(counts: DerivedCounts): Promise<void>;
|
|
17
|
+
settled(sql: Queries, count: number): Promise<void>;
|
|
18
|
+
current(counts?: DerivedCounts): Promise<void>;
|
|
19
|
+
failed(error: string): Promise<void>;
|
|
20
|
+
}
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
import { lockSchema } from './postgres-session.js';
|
|
2
|
+
// One row for each derived dataset of the database: what its refreshes left,
|
|
3
|
+
// which readers see through marts.derived_status. The sync history makes it
|
|
4
|
+
// with its tables, and a dataset makes it too, so it never depends on the
|
|
5
|
+
// order of the two.
|
|
6
|
+
export const derivedStatusTable = `CREATE TABLE IF NOT EXISTS _warehouse.derived_status (
|
|
7
|
+
dataset text PRIMARY KEY,
|
|
8
|
+
kind text NOT NULL CHECK (kind IN ('view', 'computed')),
|
|
9
|
+
state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),
|
|
10
|
+
as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,
|
|
11
|
+
inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint
|
|
12
|
+
)`;
|
|
13
|
+
// What one refresh of a derived dataset tells readers, as Snowflake keeps the
|
|
14
|
+
// state and DATA_TIMESTAMP of each dynamic table, and pgai the pending items
|
|
15
|
+
// of each vectorizer. A refresh that begins keeps as_of, the data of the
|
|
16
|
+
// last refresh that ended, until it ends too. dataset is the name with its
|
|
17
|
+
// schema.
|
|
18
|
+
export class PostgresDerivedStatus {
|
|
19
|
+
#sql;
|
|
20
|
+
#dataset;
|
|
21
|
+
#kind;
|
|
22
|
+
#clock;
|
|
23
|
+
// When this refresh began to read, which its end gives as as_of.
|
|
24
|
+
#since;
|
|
25
|
+
// sql is a session outside any transaction, between the dataset's own.
|
|
26
|
+
constructor(sql, dataset, kind, clock) {
|
|
27
|
+
this.#sql = sql;
|
|
28
|
+
this.#dataset = dataset;
|
|
29
|
+
this.#kind = kind;
|
|
30
|
+
this.#clock = clock;
|
|
31
|
+
this.#since = this.#now();
|
|
32
|
+
}
|
|
33
|
+
waiting() {
|
|
34
|
+
return this.#write({ state: 'waiting', refreshing_since: null });
|
|
35
|
+
}
|
|
36
|
+
refreshing(counts) {
|
|
37
|
+
return this.#write({
|
|
38
|
+
state: 'refreshing',
|
|
39
|
+
refreshing_since: this.#since,
|
|
40
|
+
...counts,
|
|
41
|
+
});
|
|
42
|
+
}
|
|
43
|
+
// Inside the transaction of a commit that settled count inputs.
|
|
44
|
+
async settled(sql, count) {
|
|
45
|
+
await sql.unsafe('UPDATE _warehouse.derived_status SET pending = GREATEST(pending - $1, 0) WHERE dataset = $2', [count, this.#dataset]);
|
|
46
|
+
}
|
|
47
|
+
current(counts) {
|
|
48
|
+
return this.#write({
|
|
49
|
+
state: 'current',
|
|
50
|
+
as_of: this.#since,
|
|
51
|
+
refreshed_at: this.#now(),
|
|
52
|
+
refreshing_since: null,
|
|
53
|
+
error: null,
|
|
54
|
+
...counts,
|
|
55
|
+
});
|
|
56
|
+
}
|
|
57
|
+
failed(error) {
|
|
58
|
+
return this.#write({ state: 'failed', refreshing_since: null, error });
|
|
59
|
+
}
|
|
60
|
+
// In a transaction of its own, which takes no commit number: the status is
|
|
61
|
+
// no data of a position.
|
|
62
|
+
async #write(values) {
|
|
63
|
+
const columns = ['dataset', 'kind', ...Object.keys(values)];
|
|
64
|
+
const placeholders = columns.map((_, index) => `$${index + 1}`);
|
|
65
|
+
const assignments = columns
|
|
66
|
+
.slice(1)
|
|
67
|
+
.map((column) => `${column} = excluded.${column}`);
|
|
68
|
+
await this.#sql.begin(async (sql) => {
|
|
69
|
+
await lockSchema(sql, '_warehouse');
|
|
70
|
+
await sql.unsafe(derivedStatusTable);
|
|
71
|
+
await sql.unsafe(`INSERT INTO _warehouse.derived_status (${columns.join(', ')}) VALUES (${placeholders.join(', ')}) ON CONFLICT (dataset) DO UPDATE SET ${assignments.join(', ')}`, [this.#dataset, this.#kind, ...Object.values(values)]);
|
|
72
|
+
});
|
|
73
|
+
}
|
|
74
|
+
#now() {
|
|
75
|
+
return new Date(this.#clock.Date.now()).toISOString();
|
|
76
|
+
}
|
|
77
|
+
}
|
|
@@ -28,4 +28,4 @@ export declare const syncHistoryTables: readonly ['CREATE SCHEMA IF NOT EXISTS _
|
|
|
28
28
|
)`, `CREATE TABLE IF NOT EXISTS _warehouse.stream_checks (
|
|
29
29
|
connector text NOT NULL, stream text NOT NULL, checked_at timestamptz NOT NULL,
|
|
30
30
|
PRIMARY KEY (connector, stream)
|
|
31
|
-
)
|
|
31
|
+
)`, "CREATE TABLE IF NOT EXISTS _warehouse.derived_status (\n dataset text PRIMARY KEY,\n kind text NOT NULL CHECK (kind IN ('view', 'computed')),\n state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),\n as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,\n inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint\n )"];
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { syncHistoryRelations } from '@eltjs/elt';
|
|
2
|
+
import { derivedStatusTable } from './postgres-derived-status.js';
|
|
2
3
|
// When connector's stream last matched its upstream: the start of its last
|
|
3
4
|
// successful pass, or a later check that found it unchanged, each no later
|
|
4
5
|
// than when the source said the upstream itself was last current. NULL if it
|
|
@@ -52,6 +53,7 @@ const queries = {
|
|
|
52
53
|
ORDER BY s.completed_at DESC, s.id DESC LIMIT 1
|
|
53
54
|
) success ON true
|
|
54
55
|
ORDER BY a.connector, c.stream, a.id DESC`,
|
|
56
|
+
derived_status: `SELECT dataset, kind, state, as_of, refreshed_at, refreshing_since, error, inputs, computed, empty, failed, pending FROM _warehouse.derived_status ORDER BY dataset`,
|
|
55
57
|
};
|
|
56
58
|
// How readers see the sync history: what each pass declared and loaded.
|
|
57
59
|
// The sync history keeps no row versions, so a reading as of a position
|
|
@@ -97,4 +99,5 @@ export const syncHistoryTables = [
|
|
|
97
99
|
connector text NOT NULL, stream text NOT NULL, checked_at timestamptz NOT NULL,
|
|
98
100
|
PRIMARY KEY (connector, stream)
|
|
99
101
|
)`,
|
|
102
|
+
derivedStatusTable,
|
|
100
103
|
];
|
package/dist/postgres-writer.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { hash } from 'node:crypto';
|
|
2
2
|
import { isDeepStrictEqual } from 'node:util';
|
|
3
|
-
import { FileContent, TargetOwnedError, Writer, describeTarget, errorMessage, reloadMode, } from '@eltjs/elt';
|
|
3
|
+
import { FileContent, TargetOwnedError, Writer, describeTarget, errorMessage, readPartitions, reloadMode, } from '@eltjs/elt';
|
|
4
4
|
import { quote } from './identifier.js';
|
|
5
5
|
import { TargetComments } from './postgres-comments.js';
|
|
6
6
|
import { rebindDependents } from './postgres-dependents.js';
|
|
@@ -9,7 +9,7 @@ import { storedColumns, storedFit } from './postgres-fit.js';
|
|
|
9
9
|
import { TargetHistory, rowHash, version } from './postgres-history.js';
|
|
10
10
|
import { commitOnce, expireBefore, installNumbering, } from './postgres-numbering.js';
|
|
11
11
|
import { TargetReaderView } from './postgres-reader-view.js';
|
|
12
|
-
import { PostgresSession, lockSchema, makeSchema } from './postgres-session.js';
|
|
12
|
+
import { PostgresSession, lockSchema, makeSchema, } from './postgres-session.js';
|
|
13
13
|
const batchSize = 1000;
|
|
14
14
|
// A batch also flushes once its JSON reaches this size, so rows of large text
|
|
15
15
|
// or binary values are sent a few at a time rather than a thousand at once.
|
|
@@ -224,6 +224,7 @@ export class PostgresWriter extends Writer {
|
|
|
224
224
|
}
|
|
225
225
|
// Clearing a copy also abandons the reload it left open.
|
|
226
226
|
await transaction.unsafe(`DROP TABLE IF EXISTS ${this.#hidden}`);
|
|
227
|
+
await this.#reloaded(transaction);
|
|
227
228
|
for (const column of this.table.columns.filter((column) => column.storesFile)) {
|
|
228
229
|
const store = new PostgresFileStore(this.schema, this.table, column);
|
|
229
230
|
const [chunks] = await transaction.unsafe('SELECT to_regclass($1) IS NOT NULL AS "exists"', [store.qualifiedName]);
|
|
@@ -326,13 +327,28 @@ export class PostgresWriter extends Writer {
|
|
|
326
327
|
// keeps its indexes, its triggers and what depends on it. Each row of the
|
|
327
328
|
// hidden target that no row matches enters with this commit, and the
|
|
328
329
|
// loaded_at of the load that wrote it.
|
|
329
|
-
async #reloadInPlace(sql, committedIn) {
|
|
330
|
-
const target = { table: this.qualifiedName, id: 'ctid' };
|
|
330
|
+
async #reloadInPlace(sql, committedIn, target = { table: this.qualifiedName, id: 'ctid' }) {
|
|
331
331
|
const reloaded = { table: this.#hidden, id: 'ctid' };
|
|
332
332
|
await this.#history.remove(sql, committedIn, this.qualifiedName, ` WHERE "_elt_old".ctid IN (${this.#unmatched(target, reloaded)})`);
|
|
333
333
|
await sql.unsafe(`INSERT INTO ${this.qualifiedName} (${this.written.join(', ')}) SELECT ${this.table.columns.map((column) => column.quotedName).join(', ')}, "loaded_at", $1::integer, ${version} FROM ${this.#hidden} WHERE ctid IN (${this.#unmatched(reloaded, target)}) ORDER BY ctid`, [committedIn]);
|
|
334
334
|
await sql.unsafe(`DROP TABLE ${this.#hidden}`);
|
|
335
335
|
}
|
|
336
|
+
// Gives a completed reload of some partitions their rows of the hidden
|
|
337
|
+
// target, in place and by difference: the rows of the target in those
|
|
338
|
+
// partitions are replaced as #reloadInPlace replaces a whole target, and
|
|
339
|
+
// the rows of every other partition stay as they are.
|
|
340
|
+
async #reloadPartitionsInPlace(sql, partitions, committedIn) {
|
|
341
|
+
const scoped = quote(`_elt_reloaded_${this.#hash}`);
|
|
342
|
+
const [within, values] = this.#within(partitions);
|
|
343
|
+
await sql.unsafe(`DROP TABLE IF EXISTS pg_temp.${scoped}`);
|
|
344
|
+
await sql.unsafe(`CREATE TEMP TABLE ${scoped} AS SELECT ctid AS "row" FROM ${this.qualifiedName} WHERE ${within}`, values);
|
|
345
|
+
await this.#reloadInPlace(sql, committedIn, {
|
|
346
|
+
table: this.qualifiedName,
|
|
347
|
+
id: 'ctid',
|
|
348
|
+
rows: `ctid IN (SELECT "row" FROM pg_temp.${scoped})`,
|
|
349
|
+
});
|
|
350
|
+
await sql.unsafe(`DROP TABLE pg_temp.${scoped}`);
|
|
351
|
+
}
|
|
336
352
|
// Refuses a cast of column from its stored type from that would change a
|
|
337
353
|
// stored value, such as double precision 1.5 to bigint 2, before the ALTER,
|
|
338
354
|
// as a STRICT SQLite table refuses it: each value that is not null must
|
|
@@ -437,6 +453,23 @@ export class PostgresWriter extends Writer {
|
|
|
437
453
|
#scope(partition) {
|
|
438
454
|
if (partition === null)
|
|
439
455
|
return ['', []];
|
|
456
|
+
const [condition, values] = this.#within([partition]);
|
|
457
|
+
return [` WHERE ${condition}`, values];
|
|
458
|
+
}
|
|
459
|
+
// The rows of any of the partitions, as a condition over the target's or
|
|
460
|
+
// the stage's columns, with one JSON parameter for each partition: NULL for
|
|
461
|
+
// a row that does not name its partition, such as a staged DELETE whose
|
|
462
|
+
// key leaves the partition out.
|
|
463
|
+
#within(partitions) {
|
|
464
|
+
const each = partitions.map((partition, index) => this.#partition(partition, `$${index + 1}::text::json`));
|
|
465
|
+
return [
|
|
466
|
+
each.length === 0
|
|
467
|
+
? 'FALSE'
|
|
468
|
+
: each.map(([condition]) => `(${condition})`).join(' OR '),
|
|
469
|
+
each.map(([, value]) => value),
|
|
470
|
+
];
|
|
471
|
+
}
|
|
472
|
+
#partition(partition, parameter) {
|
|
440
473
|
const columns = Object.keys(partition).map((field) => {
|
|
441
474
|
const column = this.table.columns.find((column) => column.field === field);
|
|
442
475
|
if (column === undefined)
|
|
@@ -444,10 +477,45 @@ export class PostgresWriter extends Writer {
|
|
|
444
477
|
return column;
|
|
445
478
|
});
|
|
446
479
|
return [
|
|
447
|
-
|
|
448
|
-
|
|
480
|
+
columns
|
|
481
|
+
.map((column, index) => `${column.quotedName} = ${column.valueFrom(parameter, index)}`)
|
|
482
|
+
.join(' AND '),
|
|
483
|
+
JSON.stringify(columns.map((column) => column.encode(partition))),
|
|
449
484
|
];
|
|
450
485
|
}
|
|
486
|
+
get #reloads() {
|
|
487
|
+
return `${quote(this.schema)}."_elt_reloads"`;
|
|
488
|
+
}
|
|
489
|
+
// The partitions that an open reload of the target reloads, in the
|
|
490
|
+
// writer's own table beside it, written in the commit that opens or
|
|
491
|
+
// continues the reload, so a run that resumes the reload continues it. A
|
|
492
|
+
// reload of the whole stream has no row.
|
|
493
|
+
async #reloading(sql) {
|
|
494
|
+
const [made] = await sql.unsafe('SELECT to_regclass($1) IS NOT NULL AS "exists"', [this.#reloads]);
|
|
495
|
+
if (made?.exists !== true)
|
|
496
|
+
return wholeStream;
|
|
497
|
+
const [row] = await sql.unsafe(`SELECT "partitions" FROM ${this.#reloads} WHERE "target" = $1`, [this.table.name]);
|
|
498
|
+
return row === undefined
|
|
499
|
+
? wholeStream
|
|
500
|
+
: new SomePartitions(readPartitions(this.stream, JSON.parse(row.partitions)));
|
|
501
|
+
}
|
|
502
|
+
async #reload(sql, reloads) {
|
|
503
|
+
if (!(reloads instanceof SomePartitions)) {
|
|
504
|
+
await this.#reloaded(sql);
|
|
505
|
+
return;
|
|
506
|
+
}
|
|
507
|
+
// The first reloads of the schema's tables make the table together.
|
|
508
|
+
await lockSchema(sql, this.schema);
|
|
509
|
+
await sql.unsafe(`CREATE TABLE IF NOT EXISTS ${this.#reloads} ("target" TEXT PRIMARY KEY, "partitions" TEXT NOT NULL)`);
|
|
510
|
+
await sql.unsafe(`INSERT INTO ${this.#reloads} ("target", "partitions") VALUES ($1, $2) ON CONFLICT ("target") DO UPDATE SET "partitions" = excluded."partitions"`, [this.table.name, JSON.stringify(reloads.partitions)]);
|
|
511
|
+
}
|
|
512
|
+
async #reloaded(sql) {
|
|
513
|
+
const [made] = await sql.unsafe('SELECT to_regclass($1) IS NOT NULL AS "exists"', [this.#reloads]);
|
|
514
|
+
if (made?.exists === true)
|
|
515
|
+
await sql.unsafe(`DELETE FROM ${this.#reloads} WHERE "target" = $1`, [
|
|
516
|
+
this.table.name,
|
|
517
|
+
]);
|
|
518
|
+
}
|
|
451
519
|
// One stream's load. Operations wait in a session-private TEMP stage, so no
|
|
452
520
|
// other session sees them and a crash leaves nothing behind. prepare only
|
|
453
521
|
// refuses, evolves a stored table the stream no longer fits unless the load
|
|
@@ -461,13 +529,21 @@ export class PostgresWriter extends Writer {
|
|
|
461
529
|
async prepare(load, { writer, reloading }) {
|
|
462
530
|
const { sql } = load;
|
|
463
531
|
const stage = quote(`_elt_stage_${this.#hash}`);
|
|
532
|
+
// The rows of a commit that belong to the partitions a reload reloads.
|
|
533
|
+
const reloadStage = quote(`_elt_reload_stage_${this.#hash}`);
|
|
534
|
+
const staging = (table) => `CREATE TEMP TABLE ${table} (${seq} BIGINT PRIMARY KEY, ${op} TEXT NOT NULL, ${this.table.columns.map((column) => `${column.quotedName} ${column.storageType}`).join(', ')}, ${version} TEXT)`;
|
|
535
|
+
// What an open reload reloads; the whole stream until a RESET of a
|
|
536
|
+
// partition opens one.
|
|
537
|
+
let reloads = wholeStream;
|
|
464
538
|
const stores = this.table.columns
|
|
465
539
|
.filter((column) => column.storesFile)
|
|
466
540
|
.map((column) => new PostgresFileStore(this.schema, this.table, column));
|
|
467
541
|
let mode = await load.transaction(async (sql) => {
|
|
468
542
|
await this.#refuse(sql, writer);
|
|
469
543
|
await sql.unsafe(`DROP TABLE IF EXISTS pg_temp.${stage}`);
|
|
470
|
-
await sql.unsafe(
|
|
544
|
+
await sql.unsafe(staging(stage));
|
|
545
|
+
await sql.unsafe(`DROP TABLE IF EXISTS pg_temp.${reloadStage}`);
|
|
546
|
+
await sql.unsafe(staging(reloadStage));
|
|
471
547
|
for (const store of stores)
|
|
472
548
|
await store.stage(sql);
|
|
473
549
|
const target = await storedFit(sql, this.table, this.qualifiedName, stage);
|
|
@@ -479,8 +555,12 @@ export class PostgresWriter extends Writer {
|
|
|
479
555
|
hidden,
|
|
480
556
|
});
|
|
481
557
|
// A hidden target no reload continues is a leftover readers never saw.
|
|
482
|
-
if (decided
|
|
558
|
+
if (decided === 'continue')
|
|
559
|
+
reloads = await this.#reloading(sql);
|
|
560
|
+
else {
|
|
483
561
|
await sql.unsafe(`DROP TABLE IF EXISTS ${this.#hidden}`);
|
|
562
|
+
await this.#reloaded(sql);
|
|
563
|
+
}
|
|
484
564
|
await this.#view?.refuse(sql, target === 'stale');
|
|
485
565
|
if (decided === 'evolve') {
|
|
486
566
|
await this.#evolve(sql, stage, load);
|
|
@@ -491,6 +571,11 @@ export class PostgresWriter extends Writer {
|
|
|
491
571
|
return decided;
|
|
492
572
|
});
|
|
493
573
|
const fresh = mode === 'create' || mode === 'reload';
|
|
574
|
+
// The reload as the last commit left it, which a discard goes back to.
|
|
575
|
+
let committed = {
|
|
576
|
+
mode,
|
|
577
|
+
reloads,
|
|
578
|
+
};
|
|
494
579
|
const open = () => mode === 'reload' || mode === 'continue';
|
|
495
580
|
const into = () => (open() ? this.#hidden : this.qualifiedName);
|
|
496
581
|
const tables = () => [
|
|
@@ -565,11 +650,23 @@ export class PostgresWriter extends Writer {
|
|
|
565
650
|
// The whole stream starts over in a new hidden target.
|
|
566
651
|
await drop();
|
|
567
652
|
mode = 'reload';
|
|
653
|
+
reloads = wholeStream;
|
|
568
654
|
return;
|
|
569
655
|
}
|
|
570
656
|
const [rows, values] = this.#scope(partition);
|
|
571
657
|
await sql.unsafe(`DELETE FROM ${stage}${rows}`, values);
|
|
572
658
|
resets.push(partition);
|
|
659
|
+
// The partition starts over in a hidden target, and readers see its
|
|
660
|
+
// rows as they were until the stream ends. An overwrite replaces
|
|
661
|
+
// the whole target anyway.
|
|
662
|
+
if (this.replaces)
|
|
663
|
+
return;
|
|
664
|
+
if (mode === 'load') {
|
|
665
|
+
mode = 'reload';
|
|
666
|
+
reloads = new SomePartitions([partition]);
|
|
667
|
+
}
|
|
668
|
+
else if (open() && reloads instanceof SomePartitions)
|
|
669
|
+
reloads = reloads.with(partition);
|
|
573
670
|
return;
|
|
574
671
|
}
|
|
575
672
|
// Each file is read here, before the source advances past it. The
|
|
@@ -609,6 +706,8 @@ export class PostgresWriter extends Writer {
|
|
|
609
706
|
await sql.unsafe(`CREATE TABLE ${this.#tableSQL(this.#hidden)}`);
|
|
610
707
|
await this.build(sql, this.#hidden);
|
|
611
708
|
}
|
|
709
|
+
if (open())
|
|
710
|
+
await this.#reload(sql, reloads);
|
|
612
711
|
const replacing = await claiming.claim(sql, committedIn);
|
|
613
712
|
// The rows of a hidden target were never shown, so only the target
|
|
614
713
|
// keeps the rows a commit replaces.
|
|
@@ -625,6 +724,18 @@ export class PostgresWriter extends Writer {
|
|
|
625
724
|
const written = { loadedAt: load.loadedAt, committedIn };
|
|
626
725
|
if (replacing)
|
|
627
726
|
await this.#replace(sql, stage, written);
|
|
727
|
+
else if (open() && reloads instanceof SomePartitions) {
|
|
728
|
+
// The rows of the reloaded partitions go to the hidden target,
|
|
729
|
+
// the others to the target. A DELETE whose key does not name
|
|
730
|
+
// its partition goes to both, so the reload cannot bring back a
|
|
731
|
+
// row that the source deleted.
|
|
732
|
+
const [within, values] = this.#within(reloads.partitions);
|
|
733
|
+
await sql.unsafe(`INSERT INTO ${reloadStage} SELECT * FROM ${stage} WHERE (${within}) IS TRUE OR (${op} = 'D' AND (${within}) IS NULL)`, values);
|
|
734
|
+
await sql.unsafe(`DELETE FROM ${stage} WHERE (${within}) IS TRUE`, values);
|
|
735
|
+
await this.merge(sql, stage, written, this.qualifiedName, this.#history);
|
|
736
|
+
await this.merge(sql, reloadStage, written, this.#hidden, null);
|
|
737
|
+
await sql.unsafe(`TRUNCATE ${reloadStage}`);
|
|
738
|
+
}
|
|
628
739
|
else
|
|
629
740
|
await this.merge(sql, stage, written, into(), history);
|
|
630
741
|
await sql.unsafe(`TRUNCATE ${stage}`);
|
|
@@ -636,6 +747,7 @@ export class PostgresWriter extends Writer {
|
|
|
636
747
|
// Its first commit made the hidden target, so the reload continues.
|
|
637
748
|
if (mode === 'reload')
|
|
638
749
|
mode = 'continue';
|
|
750
|
+
committed = { mode, reloads };
|
|
639
751
|
resets = [];
|
|
640
752
|
},
|
|
641
753
|
complete: async () => {
|
|
@@ -643,10 +755,17 @@ export class PostgresWriter extends Writer {
|
|
|
643
755
|
return;
|
|
644
756
|
await load.commit(async (sql, committedIn) => {
|
|
645
757
|
const target = await storedFit(sql, this.table, this.qualifiedName, stage);
|
|
646
|
-
|
|
758
|
+
// A reload of partitions keeps the target's shape: prepare evolves
|
|
759
|
+
// a stored target before it loads.
|
|
760
|
+
if (reloads instanceof SomePartitions && target === 'fits')
|
|
761
|
+
await this.#reloadPartitionsInPlace(sql, reloads.partitions, committedIn);
|
|
762
|
+
else if (reloads instanceof SomePartitions)
|
|
763
|
+
throw new TypeError(`${this.qualifiedName} changed its shape while some of its partitions reloaded; clear the copy to load it again`);
|
|
764
|
+
else if (target === 'fits')
|
|
647
765
|
await this.#reloadInPlace(sql, committedIn);
|
|
648
766
|
else
|
|
649
767
|
await this.#swap(sql, committedIn);
|
|
768
|
+
await this.#reloaded(sql);
|
|
650
769
|
await this.#history.expire(sql);
|
|
651
770
|
for (const store of stores)
|
|
652
771
|
await store.prune(sql, [
|
|
@@ -655,14 +774,40 @@ export class PostgresWriter extends Writer {
|
|
|
655
774
|
]);
|
|
656
775
|
});
|
|
657
776
|
mode = 'load';
|
|
777
|
+
reloads = wholeStream;
|
|
778
|
+
committed = { mode, reloads };
|
|
779
|
+
},
|
|
780
|
+
// A RESET that no commit took is undone with the operations after it:
|
|
781
|
+
// the source's checkpoint does not know it.
|
|
782
|
+
discard: async () => {
|
|
783
|
+
await drop();
|
|
784
|
+
({ mode, reloads } = committed);
|
|
658
785
|
},
|
|
659
|
-
discard: drop,
|
|
660
786
|
[Symbol.asyncDispose]: async () => {
|
|
661
787
|
await drop();
|
|
662
788
|
await sql.unsafe(`DROP TABLE pg_temp.${stage}`);
|
|
789
|
+
await sql.unsafe(`DROP TABLE pg_temp.${reloadStage}`);
|
|
663
790
|
for (const store of stores)
|
|
664
791
|
await store.unstage(sql);
|
|
665
792
|
},
|
|
666
793
|
};
|
|
667
794
|
}
|
|
668
795
|
}
|
|
796
|
+
// What an open reload reloads: the whole stream, or some partitions of it,
|
|
797
|
+
// while the others load into the target as before, as Delta Lake's
|
|
798
|
+
// replaceWhere replaces only the rows that its condition selects.
|
|
799
|
+
class WholeStream {
|
|
800
|
+
}
|
|
801
|
+
class SomePartitions {
|
|
802
|
+
partitions;
|
|
803
|
+
constructor(partitions) {
|
|
804
|
+
this.partitions = Object.freeze([...partitions]);
|
|
805
|
+
}
|
|
806
|
+
// These partitions and partition, which a later RESET reloads too.
|
|
807
|
+
with(partition) {
|
|
808
|
+
return this.partitions.some((other) => isDeepStrictEqual(other, partition))
|
|
809
|
+
? this
|
|
810
|
+
: new SomePartitions([...this.partitions, partition]);
|
|
811
|
+
}
|
|
812
|
+
}
|
|
813
|
+
const wholeStream = new WholeStream();
|
package/dist/view.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { type Connection, DerivedDataset } from '@eltjs/elt';
|
|
1
|
+
import { type Connection, DerivedDataset, type PipelineClock } from '@eltjs/elt';
|
|
2
2
|
import { PostgresDestination } from './postgres-destination.ts';
|
|
3
3
|
import type { PostgresTable } from './postgres-table.ts';
|
|
4
4
|
export declare class View extends DerivedDataset<PostgresTable> {
|
|
@@ -14,9 +14,10 @@ export declare class View extends DerivedDataset<PostgresTable> {
|
|
|
14
14
|
});
|
|
15
15
|
get qualifiedName(): string;
|
|
16
16
|
validate(connections: readonly Connection<PostgresTable>[]): void;
|
|
17
|
-
refresh({ connections, earlier, }: {
|
|
17
|
+
refresh({ connections, earlier, clock, }: {
|
|
18
18
|
readonly connections: readonly Connection<PostgresTable>[];
|
|
19
19
|
readonly earlier: readonly DerivedDataset<PostgresTable>[];
|
|
20
|
+
readonly clock: PipelineClock;
|
|
20
21
|
}): Promise<void>;
|
|
21
22
|
}
|
|
22
23
|
export declare function made(connections: readonly Connection<PostgresTable>[], into: PostgresDestination): Set<string>;
|
package/dist/view.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import postgres from 'postgres';
|
|
2
|
-
import { DerivedDataset } from '@eltjs/elt';
|
|
2
|
+
import { DerivedDataset, errorMessage, } from '@eltjs/elt';
|
|
3
|
+
import { PostgresDerivedStatus } from './postgres-derived-status.js';
|
|
3
4
|
import { PostgresDestination } from './postgres-destination.js';
|
|
4
5
|
import { PostgresSession } from './postgres-session.js';
|
|
5
6
|
import { checkView, publishPostgresViews, } from './postgres-views.js';
|
|
@@ -33,8 +34,9 @@ export class View extends DerivedDataset {
|
|
|
33
34
|
if (made(connections, this.into).has(this.qualifiedName))
|
|
34
35
|
throw new TypeError(`View ${this.qualifiedName} has the name of a table or reader view a copy makes`);
|
|
35
36
|
}
|
|
36
|
-
async refresh({ connections, earlier, }) {
|
|
37
|
+
async refresh({ connections, earlier, clock, }) {
|
|
37
38
|
await using session = new PostgresSession(this.into.url, 'elt-derived');
|
|
39
|
+
const status = new PostgresDerivedStatus(session.sql, this.qualifiedName, 'view', clock);
|
|
38
40
|
try {
|
|
39
41
|
await session.sql.begin((transaction) => publishPostgresViews(transaction, {
|
|
40
42
|
schema: this.into.schema,
|
|
@@ -45,10 +47,14 @@ export class View extends DerivedDataset {
|
|
|
45
47
|
const missing = missingRelation(error, this.into.schema);
|
|
46
48
|
if (missing !== undefined &&
|
|
47
49
|
(made(connections, this.into).has(missing) ||
|
|
48
|
-
earlier.some((dataset) => derivedRelation(dataset) === missing)))
|
|
50
|
+
earlier.some((dataset) => derivedRelation(dataset) === missing))) {
|
|
51
|
+
await status.waiting();
|
|
49
52
|
return;
|
|
53
|
+
}
|
|
54
|
+
await status.failed(errorMessage(error));
|
|
50
55
|
throw error;
|
|
51
56
|
}
|
|
57
|
+
await status.current();
|
|
52
58
|
}
|
|
53
59
|
}
|
|
54
60
|
// The schema-qualified relation that Postgres found missing, with a bare name
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@eltjs/elt-postgresql",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.16",
|
|
4
4
|
"description": "The Postgres destination of elt: each stream is a typed table in one schema, with a reader view in another, a history of each row for readings as of a position, and the views and computed datasets that a pipeline derives.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"repository": {
|
|
@@ -37,7 +37,7 @@
|
|
|
37
37
|
"access": "public"
|
|
38
38
|
},
|
|
39
39
|
"dependencies": {
|
|
40
|
-
"@eltjs/elt": "^0.3.
|
|
40
|
+
"@eltjs/elt": "^0.3.16",
|
|
41
41
|
"@zukhruf/async": "^0.8.27",
|
|
42
42
|
"postgres": "^3.4.9"
|
|
43
43
|
},
|