@eltjs/elt-postgresql 0.3.15 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -9
- package/dist/computed-dataset.js +40 -2
- package/dist/index.d.ts +1 -0
- package/dist/index.js +1 -0
- package/dist/postgres-columns.d.ts +2 -0
- package/dist/postgres-columns.js +11 -1
- package/dist/postgres-derived-status.d.ts +20 -0
- package/dist/postgres-derived-status.js +77 -0
- package/dist/postgres-passes.d.ts +44 -0
- package/dist/postgres-passes.js +191 -0
- package/dist/postgres-sync-history-schema.d.ts +1 -1
- package/dist/postgres-sync-history-schema.js +3 -0
- package/dist/postgres-sync-history.js +93 -52
- package/dist/view.d.ts +3 -2
- package/dist/view.js +9 -3
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -73,7 +73,7 @@ Inferred columns follow the stream schema. The [string formats](../../elt/README
|
|
|
73
73
|
- A `TIMESTAMPTZ` keeps the instant. It does not keep the offset that the value had.
|
|
74
74
|
- Arrays keep the order of their elements, and load a JSON `null` as SQL `NULL`. Readers use `= ANY(...)`, `unnest()` and `cardinality()`. `marts.catalog` shows the type of the element.
|
|
75
75
|
- ISO dates count years astronomically, and Postgres does not. Thus the year `0000` loads as `0001 BC`, which is the same day. Each other year is written as it is given.
|
|
76
|
-
- Explicit columns are `columns.text`, `integer`, `real`, `boolean`, `date`, `timestamp` and `blob(field)`, with `.notNull()` and `.primaryKey()`.
|
|
76
|
+
- Explicit columns are `columns.text`, `integer`, `real`, `boolean`, `date`, `timestamp` and `blob(field)`, with `.notNull()` and `.primaryKey()`. `columns.int64(field)` is a `BIGINT` that loads decimal text exactly, and `columns.localTimestamp(field, precision)` is a `TIMESTAMP` for SQL Server's `datetime2`, by the rows of the table above: a precision past 6 loads as text.
|
|
77
77
|
- A plain `blob()` stores `BYTEA`. `blob().from(file)` keeps the original file in chunk tables, and `text().from(file).parse(parser)` stores the parsed text as `TEXT`. See [file bytes in the database](#file-bytes-in-the-database).
|
|
78
78
|
- Table, view and column names follow the [naming rule](../../elt/README.md#names). Postgres compares names with their case, keeps 63 bytes of a name, and reserves `loaded_at`, `committed_in`, `replaced_in` and the system column names `xmin`, `xmax`, `cmin`, `cmax`, `ctid` and `tableoid`. The naming rule turns a field `_elt_row_hash` into `elt_row_hash`, so no field takes that name.
|
|
79
79
|
|
|
@@ -291,22 +291,25 @@ A refresh is a load in `_warehouse._elt_loads`, under the name of its dataset, s
|
|
|
291
291
|
|
|
292
292
|
- It records only connections with a Postgres destination. It refuses other connections when the pipeline validates.
|
|
293
293
|
- It records into `_warehouse.sync_attempts`, `_warehouse.extraction_coverage` and `_warehouse.stream_checks`. The last one holds the latest check of each stream by a watch that found it unchanged.
|
|
294
|
-
- `install()` makes those tables and publishes the `marts` views `sync_attempts`, `extraction_coverage`, `sync_status`, `stream_status` and `catalog`, in one transaction. The pipeline calls it before each run and watch, and it is safe to repeat. See [ADR 0005](./docs/adr/0005-the-history-install-publishes-the-catalog.md).
|
|
294
|
+
- `install()` makes those tables and `_warehouse.derived_status`, and publishes the `marts` views `sync_attempts`, `extraction_coverage`, `sync_status`, `stream_status`, `derived_status` and `catalog`, in one transaction. The pipeline calls it before each run and watch, and it is safe to repeat. See [ADR 0005](./docs/adr/0005-the-history-install-publishes-the-catalog.md).
|
|
295
295
|
- `new PostgresSyncHistory({ url, readers: ['agent_reader'] })` names the roles that read the database as of a position with their own session: `new PostgresDestination({ url: readerUrl, schema })`, then `positions()` and `reading()`. Each install grants each reader what a reading needs, which is the numbering in `_warehouse` and nothing else of it, as dbt applies the grants that a model declares after each build. Thus no host names the objects of the numbering, and a reader keeps its grants when the numbering is made again. The install grants nothing else: who may read `marts` is a grant of the database, such as its default privileges. A role that does not exist fails the install.
|
|
296
296
|
- Before a pass reads, the history inserts one `sync_attempts` row, with the connection name as `connector` and the source identity as `source`. It inserts one `extraction_coverage` row for each selected stream, with the declared `coverage(stream)` of the source and the schema of the destination as `target_schema`. The outcomes close them.
|
|
297
297
|
- A run declares each stream that a connection selected. A watch pass reads only the streams whose change signals changed, so an attempt vouches only for its own `extraction_coverage` rows.
|
|
298
298
|
- A connection that is not valid or that stopped, a failed change signal and a failed upkeep each record a failed attempt with no outcomes.
|
|
299
|
+
- Each attempt has a session of its own until its pass ends, and that session holds a lock on the attempt. A pass whose process stopped, for example a container restart, stays `running` until the next attempt of its connection begins. That attempt records it as `failed`, because no session holds its lock. The pipeline records a failed change signal while a pass of the same connection runs, and the running pass stays `running`. No host option is necessary, because Postgres knows which sessions are alive. See [ADR 0009](./docs/adr/0009-a-running-attempt-holds-a-lock-of-its-session.md).
|
|
299
300
|
- `_warehouse` tables are made only when they are missing.
|
|
301
|
+
- `readPassStatus(url)` gives the latest pass of the database and the latest outcome of each stream, as `readPassStatus(path)` of elt-sqlite gives them for a file. The latest passes of the connections show as one pass. It is `running` when one connection runs. Else it is `cancelled` when one pass stopped, `succeeded` when all passes succeeded, `failed` when all passes failed, and `partial` in all other cases. It starts with the first pass and completes with the last. Its error has the error of each pass that did not succeed, one on each line. Its failure type is `config` when one pass has a `config` failure. Its last success and its `current_as_of` are the earliest of the passes, or null when one pass has no value. Each stream has the error of its latest attempt when its copy failed or was partial in that attempt: the error of each failure, one on each line. Times are ISO text. A database with no sync history gives no pass and no streams. It reads `marts`, so a reader role that may read `marts` can call it with its own URL.
|
|
300
302
|
|
|
301
303
|
The relations in `marts`:
|
|
302
304
|
|
|
303
|
-
| Relation | Contents
|
|
304
|
-
| --------------------- |
|
|
305
|
-
| `catalog` | Each view and column in `marts`, with its description. The start point of a reader. A table is not for readers, also when it is in `marts`, such as the raw table of a computed dataset.
|
|
306
|
-
| `sync_status` | For each connection: the latest attempt, the end of the last successful pass, and `current_as_of` (the oldest of its streams).
|
|
307
|
-
| `stream_status` | For each connection and stream: the latest attempt that declared the stream with the status of its copy, and the last attempt in which that copy succeeded. A watch pass reads only changed streams, so judge a stream here, by `current_as_of`.
|
|
308
|
-
| `
|
|
309
|
-
| `
|
|
305
|
+
| Relation | Contents |
|
|
306
|
+
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
307
|
+
| `catalog` | Each view and column in `marts`, with its description. The start point of a reader. A table is not for readers, also when it is in `marts`, such as the raw table of a computed dataset. |
|
|
308
|
+
| `sync_status` | For each connection: the latest attempt, the end of the last successful pass, and `current_as_of` (the oldest of its streams). |
|
|
309
|
+
| `stream_status` | For each connection and stream: the latest attempt that declared the stream with the status of its copy, and the last attempt in which that copy succeeded. A watch pass reads only changed streams, so judge a stream here, by `current_as_of`. |
|
|
310
|
+
| `derived_status` | For each view and computed dataset, named with its schema: whether it waits for its inputs, refreshes, is current or failed, its `as_of`, and for a computed dataset how many inputs it computed, found empty, failed on or has still to compute. See [derived status](#derived-status). |
|
|
311
|
+
| `sync_attempts` | The kept history of attempts, with start and end times, status and errors. |
|
|
312
|
+
| `extraction_coverage` | For each attempt and stream: the configured selection, the explanation of the scope, the target, the outcome of the copy, the committed counts and the failed partitions. |
|
|
310
313
|
|
|
311
314
|
What the relations tell:
|
|
312
315
|
|
|
@@ -319,6 +322,17 @@ What the relations tell:
|
|
|
319
322
|
- A first failure has no time of success.
|
|
320
323
|
- Coverage is the [configured scope](../../elt/README.md#coverage). Judge freshness by `current_as_of` ([the four clocks](../../elt/README.md#the-four-clocks)).
|
|
321
324
|
|
|
325
|
+
## Derived status
|
|
326
|
+
|
|
327
|
+
Each [view](#declare-a-view) and [computed dataset](#declare-a-computed-dataset) keeps one row in `_warehouse.derived_status`, which readers see as `marts.derived_status`. A reader needs it because `sync_status` says that a pass succeeded while a computed dataset still computes, or failed on each of its inputs.
|
|
328
|
+
|
|
329
|
+
- A refresh writes `waiting` when a relation that the dataset reads is not made yet, and `refreshing` when it begins, with `refreshing_since`. A computed dataset also writes how many inputs it read and how many it has still to compute (`pending`). Each commit of the refresh lowers `pending` in its own transaction, so a reader sees it fall while the rows land.
|
|
330
|
+
- A refresh that ends with no failure of the dataset writes `current`, with `as_of`, when it began to read, and `refreshed_at`. A refresh that fails writes `failed`, with `error`. A rule that fails on one input is not a failure of the dataset: `failed` counts its rows.
|
|
331
|
+
- `computed`, `empty` and `failed` count the output rows when the refresh began and when it ended.
|
|
332
|
+
- A dataset is caught up when its `state` is `current`, its `pending` is 0, and its `as_of` is at or after the start of the last pass of the connections that it reads. This is the frontier of Materialize and the `DATA_TIMESTAMP` of a Snowflake dynamic table, with the pending items of a pgai vectorizer.
|
|
333
|
+
- A refresh that a stopped process left stays `refreshing` until the next refresh. The status keeps no row versions, so a reading as of a position fails the view, as it fails the other views of the sync history.
|
|
334
|
+
- A status write takes no commit number: it is not data of a position.
|
|
335
|
+
|
|
322
336
|
## Decisions
|
|
323
337
|
|
|
324
338
|
- [ADR 0001](./docs/adr/0001-checkpoints-are-kept-beside-the-data.md): checkpoints are kept beside the data.
|
|
@@ -329,6 +343,7 @@ What the relations tell:
|
|
|
329
343
|
- [ADR 0006](./docs/adr/0006-a-change-of-a-table-makes-the-views-over-it-again.md): a change of a table makes the views over it again.
|
|
330
344
|
- [ADR 0007](./docs/adr/0007-readers-are-named-on-the-sync-history.md): readers are named on the sync history.
|
|
331
345
|
- [ADR 0008](./docs/adr/0008-derived-datasets-follow-elt-sqlite.md): derived datasets follow elt-sqlite.
|
|
346
|
+
- [ADR 0009](./docs/adr/0009-a-running-attempt-holds-a-lock-of-its-session.md): a running attempt holds a lock of its session.
|
|
332
347
|
|
|
333
348
|
## Development
|
|
334
349
|
|
|
@@ -354,6 +369,7 @@ src/
|
|
|
354
369
|
postgres-views.ts the publication of views, for View and the sync history
|
|
355
370
|
postgres-dependents.ts the views over a table, made again over its change
|
|
356
371
|
postgres-privileges.ts the privileges of a relation, kept across its change
|
|
372
|
+
postgres-derived-status.ts the status that each derived dataset keeps of itself
|
|
357
373
|
postgres-checkpoint-store.ts PostgresCheckpointStore
|
|
358
374
|
postgres-sync-history.ts PostgresSyncHistory and the catalog
|
|
359
375
|
postgres-sync-history-schema.ts the tables and views of the history
|
package/dist/computed-dataset.js
CHANGED
|
@@ -1,9 +1,10 @@
|
|
|
1
1
|
import { hash } from 'node:crypto';
|
|
2
2
|
import { Lock } from '@zukhruf/async';
|
|
3
|
-
import { DerivedDataset, } from '@eltjs/elt';
|
|
3
|
+
import { DerivedDataset, errorMessage, } from '@eltjs/elt';
|
|
4
4
|
import { identifier, quote } from './identifier.js';
|
|
5
5
|
import { TargetComments } from './postgres-comments.js';
|
|
6
6
|
import { rebindDependents } from './postgres-dependents.js';
|
|
7
|
+
import { PostgresDerivedStatus } from './postgres-derived-status.js';
|
|
7
8
|
import { PostgresDestination } from './postgres-destination.js';
|
|
8
9
|
import { storedColumns } from './postgres-fit.js';
|
|
9
10
|
import { TargetHistory, rowHash } from './postgres-history.js';
|
|
@@ -122,9 +123,25 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
122
123
|
// Open from the start, so it closes a refresh that a process ended; with
|
|
123
124
|
// no commit it is no span, and leaves the positions as they are.
|
|
124
125
|
await using load = await PostgresLoad.open(this.into.url, { dataset: this.name }, { clock, retention: this.into.retention });
|
|
126
|
+
const status = new PostgresDerivedStatus(load.sql, this.qualifiedName, 'computed', clock);
|
|
125
127
|
const inputs = await this.#inputs(load.sql, connections);
|
|
126
|
-
if (inputs === null)
|
|
128
|
+
if (inputs === null) {
|
|
129
|
+
await status.waiting();
|
|
127
130
|
return;
|
|
131
|
+
}
|
|
132
|
+
try {
|
|
133
|
+
await this.#update(load, status, inputs, trigger, signal);
|
|
134
|
+
}
|
|
135
|
+
catch (error) {
|
|
136
|
+
// A stop is not the dataset's failure.
|
|
137
|
+
if (signal?.aborted !== true)
|
|
138
|
+
await status.failed(errorMessage(error));
|
|
139
|
+
throw error;
|
|
140
|
+
}
|
|
141
|
+
}
|
|
142
|
+
// Brings the output rows in step with inputs, and tells status how many
|
|
143
|
+
// are due and how many settled.
|
|
144
|
+
async #update(load, status, inputs, trigger, signal) {
|
|
128
145
|
const key = quote(this.#rule.key.name);
|
|
129
146
|
const table = this.#qualified(this.#table);
|
|
130
147
|
// The number of a transaction's commit, taken at its first change, so a
|
|
@@ -214,6 +231,10 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
214
231
|
trigger.tried.add(attempt);
|
|
215
232
|
due.push({ id, input, inputHash, rule });
|
|
216
233
|
}
|
|
234
|
+
await status.refreshing({
|
|
235
|
+
...(await this.#counts(load.sql, inputs)),
|
|
236
|
+
pending: due.length,
|
|
237
|
+
});
|
|
217
238
|
// A pool of calls, each taking the next batch once its own committed, so
|
|
218
239
|
// a slow call holds only its place, as a worker pool keeps its other
|
|
219
240
|
// workers busy. A call that fails stops the pool taking more.
|
|
@@ -247,11 +268,28 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
247
268
|
else
|
|
248
269
|
await store(sql, commit, id, inputHash, rule, 'empty', null, {});
|
|
249
270
|
}
|
|
271
|
+
await status.settled(sql, batch.length);
|
|
250
272
|
});
|
|
251
273
|
}
|
|
252
274
|
}));
|
|
253
275
|
if (failures.length > 0)
|
|
254
276
|
throw failures[0];
|
|
277
|
+
if (signal?.aborted === true)
|
|
278
|
+
return;
|
|
279
|
+
await status.current({
|
|
280
|
+
...(await this.#counts(load.sql, inputs)),
|
|
281
|
+
pending: 0,
|
|
282
|
+
});
|
|
283
|
+
}
|
|
284
|
+
// How many inputs there are, and how many output rows have each status.
|
|
285
|
+
async #counts(sql, inputs) {
|
|
286
|
+
const counted = new Map((await sql.unsafe(`SELECT "status", count(*)::int AS "n" FROM ${this.#qualified(this.#table)} GROUP BY "status"`)).map(({ status, n }) => [status, n]));
|
|
287
|
+
return {
|
|
288
|
+
inputs: inputs.size,
|
|
289
|
+
computed: counted.get('computed') ?? 0,
|
|
290
|
+
empty: counted.get('empty') ?? 0,
|
|
291
|
+
failed: counted.get('failed') ?? 0,
|
|
292
|
+
};
|
|
255
293
|
}
|
|
256
294
|
// Makes the table, its history and its view, or makes the table again when
|
|
257
295
|
// its declaration changed: the history keeps each row of the old table, in
|
package/dist/index.d.ts
CHANGED
|
@@ -4,6 +4,7 @@ export { PostgresCheckpointStore } from './postgres-checkpoint-store.ts';
|
|
|
4
4
|
export { PostgresColumn } from './postgres-column.ts';
|
|
5
5
|
export { PostgresColumns } from './postgres-columns.ts';
|
|
6
6
|
export { PostgresDestination } from './postgres-destination.ts';
|
|
7
|
+
export { type PassState, type PassStatus, readPassStatus, type StreamPassStatus, } from './postgres-passes.ts';
|
|
7
8
|
export { PostgresSyncHistory } from './postgres-sync-history.ts';
|
|
8
9
|
export { PostgresTable } from './postgres-table.ts';
|
|
9
10
|
export { View } from './view.ts';
|
package/dist/index.js
CHANGED
|
@@ -3,6 +3,7 @@ export { PostgresCheckpointStore } from './postgres-checkpoint-store.js';
|
|
|
3
3
|
export { PostgresColumn } from './postgres-column.js';
|
|
4
4
|
export { PostgresColumns } from './postgres-columns.js';
|
|
5
5
|
export { PostgresDestination } from './postgres-destination.js';
|
|
6
|
+
export { readPassStatus, } from './postgres-passes.js';
|
|
6
7
|
export { PostgresSyncHistory } from './postgres-sync-history.js';
|
|
7
8
|
export { PostgresTable } from './postgres-table.js';
|
|
8
9
|
export { View } from './view.js';
|
|
@@ -5,9 +5,11 @@ export declare class PostgresColumns {
|
|
|
5
5
|
static fromSchema({ properties, required, }: Stream['jsonSchema']): readonly PostgresColumn[];
|
|
6
6
|
text(field: string): PostgresColumn;
|
|
7
7
|
integer(field: string): PostgresColumn;
|
|
8
|
+
int64(field: string): PostgresColumn;
|
|
8
9
|
real(field: string): PostgresColumn;
|
|
9
10
|
boolean(field: string): PostgresColumn;
|
|
10
11
|
date(field: string): PostgresColumn;
|
|
11
12
|
timestamp(field: string): PostgresColumn;
|
|
13
|
+
localTimestamp(field: string, precision: number): PostgresColumn;
|
|
12
14
|
blob(field: string): PostgresColumn;
|
|
13
15
|
}
|
package/dist/postgres-columns.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { declaredFormat } from '@eltjs/elt';
|
|
1
|
+
import { Int64Format, LocalTimestampFormat, declaredFormat, } from '@eltjs/elt';
|
|
2
2
|
import { PostgresColumn } from './postgres-column.js';
|
|
3
3
|
function scalarType(name, type, format) {
|
|
4
4
|
switch (type) {
|
|
@@ -55,6 +55,11 @@ export class PostgresColumns {
|
|
|
55
55
|
integer(field) {
|
|
56
56
|
return this.#column(field, 'integer');
|
|
57
57
|
}
|
|
58
|
+
// A signed 64-bit integer, which a stream spells as decimal text (format
|
|
59
|
+
// int64), so a value past Number.MAX_SAFE_INTEGER keeps every digit.
|
|
60
|
+
int64(field) {
|
|
61
|
+
return this.#column(field, new Int64Format());
|
|
62
|
+
}
|
|
58
63
|
real(field) {
|
|
59
64
|
return this.#column(field, 'real');
|
|
60
65
|
}
|
|
@@ -67,6 +72,11 @@ export class PostgresColumns {
|
|
|
67
72
|
timestamp(field) {
|
|
68
73
|
return this.#column(field, 'timestamp');
|
|
69
74
|
}
|
|
75
|
+
// A wall-clock date and time with no zone and precision digits of a second
|
|
76
|
+
// (format date-time-local), as SQL Server's datetime2 holds it.
|
|
77
|
+
localTimestamp(field, precision) {
|
|
78
|
+
return this.#column(field, new LocalTimestampFormat(precision));
|
|
79
|
+
}
|
|
70
80
|
blob(field) {
|
|
71
81
|
return this.#column(field, 'blob');
|
|
72
82
|
}
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
import type postgres from 'postgres';
|
|
2
|
+
import type { PipelineClock } from '@eltjs/elt';
|
|
3
|
+
import { type Queries } from './postgres-session.ts';
|
|
4
|
+
export declare const derivedStatusTable = "CREATE TABLE IF NOT EXISTS _warehouse.derived_status (\n dataset text PRIMARY KEY,\n kind text NOT NULL CHECK (kind IN ('view', 'computed')),\n state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),\n as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,\n inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint\n )";
|
|
5
|
+
export type DerivedCounts = {
|
|
6
|
+
readonly inputs: number;
|
|
7
|
+
readonly computed: number;
|
|
8
|
+
readonly empty: number;
|
|
9
|
+
readonly failed: number;
|
|
10
|
+
readonly pending: number;
|
|
11
|
+
};
|
|
12
|
+
export declare class PostgresDerivedStatus {
|
|
13
|
+
#private;
|
|
14
|
+
constructor(sql: postgres.Sql, dataset: string, kind: 'view' | 'computed', clock: PipelineClock);
|
|
15
|
+
waiting(): Promise<void>;
|
|
16
|
+
refreshing(counts: DerivedCounts): Promise<void>;
|
|
17
|
+
settled(sql: Queries, count: number): Promise<void>;
|
|
18
|
+
current(counts?: DerivedCounts): Promise<void>;
|
|
19
|
+
failed(error: string): Promise<void>;
|
|
20
|
+
}
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
import { lockSchema } from './postgres-session.js';
|
|
2
|
+
// One row for each derived dataset of the database: what its refreshes left,
|
|
3
|
+
// which readers see through marts.derived_status. The sync history makes it
|
|
4
|
+
// with its tables, and a dataset makes it too, so it never depends on the
|
|
5
|
+
// order of the two.
|
|
6
|
+
export const derivedStatusTable = `CREATE TABLE IF NOT EXISTS _warehouse.derived_status (
|
|
7
|
+
dataset text PRIMARY KEY,
|
|
8
|
+
kind text NOT NULL CHECK (kind IN ('view', 'computed')),
|
|
9
|
+
state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),
|
|
10
|
+
as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,
|
|
11
|
+
inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint
|
|
12
|
+
)`;
|
|
13
|
+
// What one refresh of a derived dataset tells readers, as Snowflake keeps the
|
|
14
|
+
// state and DATA_TIMESTAMP of each dynamic table, and pgai the pending items
|
|
15
|
+
// of each vectorizer. A refresh that begins keeps as_of, the data of the
|
|
16
|
+
// last refresh that ended, until it ends too. dataset is the name with its
|
|
17
|
+
// schema.
|
|
18
|
+
export class PostgresDerivedStatus {
|
|
19
|
+
#sql;
|
|
20
|
+
#dataset;
|
|
21
|
+
#kind;
|
|
22
|
+
#clock;
|
|
23
|
+
// When this refresh began to read, which its end gives as as_of.
|
|
24
|
+
#since;
|
|
25
|
+
// sql is a session outside any transaction, between the dataset's own.
|
|
26
|
+
constructor(sql, dataset, kind, clock) {
|
|
27
|
+
this.#sql = sql;
|
|
28
|
+
this.#dataset = dataset;
|
|
29
|
+
this.#kind = kind;
|
|
30
|
+
this.#clock = clock;
|
|
31
|
+
this.#since = this.#now();
|
|
32
|
+
}
|
|
33
|
+
waiting() {
|
|
34
|
+
return this.#write({ state: 'waiting', refreshing_since: null });
|
|
35
|
+
}
|
|
36
|
+
refreshing(counts) {
|
|
37
|
+
return this.#write({
|
|
38
|
+
state: 'refreshing',
|
|
39
|
+
refreshing_since: this.#since,
|
|
40
|
+
...counts,
|
|
41
|
+
});
|
|
42
|
+
}
|
|
43
|
+
// Inside the transaction of a commit that settled count inputs.
|
|
44
|
+
async settled(sql, count) {
|
|
45
|
+
await sql.unsafe('UPDATE _warehouse.derived_status SET pending = GREATEST(pending - $1, 0) WHERE dataset = $2', [count, this.#dataset]);
|
|
46
|
+
}
|
|
47
|
+
current(counts) {
|
|
48
|
+
return this.#write({
|
|
49
|
+
state: 'current',
|
|
50
|
+
as_of: this.#since,
|
|
51
|
+
refreshed_at: this.#now(),
|
|
52
|
+
refreshing_since: null,
|
|
53
|
+
error: null,
|
|
54
|
+
...counts,
|
|
55
|
+
});
|
|
56
|
+
}
|
|
57
|
+
failed(error) {
|
|
58
|
+
return this.#write({ state: 'failed', refreshing_since: null, error });
|
|
59
|
+
}
|
|
60
|
+
// In a transaction of its own, which takes no commit number: the status is
|
|
61
|
+
// no data of a position.
|
|
62
|
+
async #write(values) {
|
|
63
|
+
const columns = ['dataset', 'kind', ...Object.keys(values)];
|
|
64
|
+
const placeholders = columns.map((_, index) => `$${index + 1}`);
|
|
65
|
+
const assignments = columns
|
|
66
|
+
.slice(1)
|
|
67
|
+
.map((column) => `${column} = excluded.${column}`);
|
|
68
|
+
await this.#sql.begin(async (sql) => {
|
|
69
|
+
await lockSchema(sql, '_warehouse');
|
|
70
|
+
await sql.unsafe(derivedStatusTable);
|
|
71
|
+
await sql.unsafe(`INSERT INTO _warehouse.derived_status (${columns.join(', ')}) VALUES (${placeholders.join(', ')}) ON CONFLICT (dataset) DO UPDATE SET ${assignments.join(', ')}`, [this.#dataset, this.#kind, ...Object.values(values)]);
|
|
72
|
+
});
|
|
73
|
+
}
|
|
74
|
+
#now() {
|
|
75
|
+
return new Date(this.#clock.Date.now()).toISOString();
|
|
76
|
+
}
|
|
77
|
+
}
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
import type { FailureType, SyncStatus } from '@eltjs/elt';
|
|
2
|
+
export type PassState = SyncStatus | 'running';
|
|
3
|
+
type Started = {
|
|
4
|
+
readonly startedAt: string;
|
|
5
|
+
readonly lastSucceededAt: string | null;
|
|
6
|
+
readonly currentAsOf: string | null;
|
|
7
|
+
};
|
|
8
|
+
export type PassStatus = (Started & {
|
|
9
|
+
readonly state: 'running';
|
|
10
|
+
readonly completedAt: null;
|
|
11
|
+
readonly error: null;
|
|
12
|
+
readonly failureType: null;
|
|
13
|
+
}) | (Started & {
|
|
14
|
+
readonly state: 'succeeded';
|
|
15
|
+
readonly completedAt: string;
|
|
16
|
+
readonly error: null;
|
|
17
|
+
readonly failureType: null;
|
|
18
|
+
}) | (Started & {
|
|
19
|
+
readonly state: 'cancelled';
|
|
20
|
+
readonly completedAt: string;
|
|
21
|
+
readonly error: string;
|
|
22
|
+
readonly failureType: null;
|
|
23
|
+
}) | (Started & {
|
|
24
|
+
readonly state: 'partial' | 'failed';
|
|
25
|
+
readonly completedAt: string;
|
|
26
|
+
readonly error: string;
|
|
27
|
+
readonly failureType: FailureType;
|
|
28
|
+
});
|
|
29
|
+
export type StreamPassStatus = {
|
|
30
|
+
readonly stream: string;
|
|
31
|
+
readonly lastSucceededAt: string | null;
|
|
32
|
+
readonly currentAsOf: string | null;
|
|
33
|
+
} & ({
|
|
34
|
+
readonly state: 'running' | 'succeeded' | 'cancelled';
|
|
35
|
+
readonly error: null;
|
|
36
|
+
} | {
|
|
37
|
+
readonly state: 'partial' | 'failed';
|
|
38
|
+
readonly error: string;
|
|
39
|
+
});
|
|
40
|
+
export declare function readPassStatus(url: string): Promise<{
|
|
41
|
+
readonly pass: PassStatus | null;
|
|
42
|
+
readonly streams: readonly StreamPassStatus[];
|
|
43
|
+
}>;
|
|
44
|
+
export {};
|
|
@@ -0,0 +1,191 @@
|
|
|
1
|
+
import { PostgresSession } from './postgres-session.js';
|
|
2
|
+
// The latest pass of the connections that the sync history of a database
|
|
3
|
+
// records, and each stream's latest outcome, with the errors of its failures
|
|
4
|
+
// in that attempt, one on each line; null and empty until the history
|
|
5
|
+
// recorded a pass, also while no history is installed. The latest passes of
|
|
6
|
+
// several connections read as one pass, as the passes of a SQLite file that
|
|
7
|
+
// several connections load do. One snapshot reads both.
|
|
8
|
+
export async function readPassStatus(url) {
|
|
9
|
+
await using session = new PostgresSession(url, 'elt-pass-status');
|
|
10
|
+
try {
|
|
11
|
+
return await session.sql.begin('ISOLATION LEVEL REPEATABLE READ READ ONLY', async (sql) => {
|
|
12
|
+
const [first, ...others] = (await sql `
|
|
13
|
+
SELECT status, started_at, completed_at, error, failure_type,
|
|
14
|
+
last_successful_sync_at, current_as_of
|
|
15
|
+
FROM marts.sync_status`).map((row) => recorded(row, passState(row.status)));
|
|
16
|
+
const streams = await streamOutcomes(sql);
|
|
17
|
+
return {
|
|
18
|
+
pass: first === undefined ? null : combined([first, ...others]),
|
|
19
|
+
streams,
|
|
20
|
+
};
|
|
21
|
+
});
|
|
22
|
+
}
|
|
23
|
+
catch (error) {
|
|
24
|
+
if (Object(error).code === '42P01')
|
|
25
|
+
return { pass: null, streams: [] };
|
|
26
|
+
throw error;
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
async function streamOutcomes(sql) {
|
|
30
|
+
const rows = await sql `
|
|
31
|
+
SELECT s.stream, s.status, s.last_successful_sync_at, s.current_as_of,
|
|
32
|
+
(SELECT string_agg(f.value->>'error', E'\n' ORDER BY f.ordinality)
|
|
33
|
+
FROM marts.extraction_coverage c,
|
|
34
|
+
jsonb_array_elements(c.failures) WITH ORDINALITY f
|
|
35
|
+
WHERE c.attempt_id = s.latest_attempt_id AND c.stream = s.stream) AS error
|
|
36
|
+
FROM marts.stream_status s ORDER BY s.stream COLLATE "C"`;
|
|
37
|
+
return rows.map((row) => streamRecorded(row, passState(row.status)));
|
|
38
|
+
}
|
|
39
|
+
// A stream_status row as the stream's latest outcome: only a copy that
|
|
40
|
+
// failed has an error.
|
|
41
|
+
function streamRecorded(row, state) {
|
|
42
|
+
const outcome = {
|
|
43
|
+
stream: String(row.stream),
|
|
44
|
+
lastSucceededAt: optionalInstant(row.last_successful_sync_at),
|
|
45
|
+
currentAsOf: optionalInstant(row.current_as_of),
|
|
46
|
+
};
|
|
47
|
+
switch (state) {
|
|
48
|
+
case 'partial':
|
|
49
|
+
case 'failed':
|
|
50
|
+
return { ...outcome, state, error: String(row.error) };
|
|
51
|
+
default:
|
|
52
|
+
return { ...outcome, state, error: null };
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
// A sync_status row as the pass it records: the history completes every pass
|
|
56
|
+
// that ended, and gives an error to each that did not load completely.
|
|
57
|
+
function recorded(row, state) {
|
|
58
|
+
const started = {
|
|
59
|
+
startedAt: instant(row.started_at),
|
|
60
|
+
lastSucceededAt: optionalInstant(row.last_successful_sync_at),
|
|
61
|
+
currentAsOf: optionalInstant(row.current_as_of),
|
|
62
|
+
};
|
|
63
|
+
switch (state) {
|
|
64
|
+
case 'running':
|
|
65
|
+
return {
|
|
66
|
+
...started,
|
|
67
|
+
state,
|
|
68
|
+
completedAt: null,
|
|
69
|
+
error: null,
|
|
70
|
+
failureType: null,
|
|
71
|
+
};
|
|
72
|
+
case 'succeeded':
|
|
73
|
+
return {
|
|
74
|
+
...started,
|
|
75
|
+
state,
|
|
76
|
+
completedAt: instant(row.completed_at),
|
|
77
|
+
error: null,
|
|
78
|
+
failureType: null,
|
|
79
|
+
};
|
|
80
|
+
case 'cancelled':
|
|
81
|
+
return {
|
|
82
|
+
...started,
|
|
83
|
+
state,
|
|
84
|
+
completedAt: instant(row.completed_at),
|
|
85
|
+
error: String(row.error),
|
|
86
|
+
failureType: null,
|
|
87
|
+
};
|
|
88
|
+
default:
|
|
89
|
+
return {
|
|
90
|
+
...started,
|
|
91
|
+
state,
|
|
92
|
+
completedAt: instant(row.completed_at),
|
|
93
|
+
error: String(row.error),
|
|
94
|
+
failureType: failureType(row.failure_type),
|
|
95
|
+
};
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
// The latest passes of the connections of one database as one pass, which
|
|
99
|
+
// is only as good as its worst connection: it runs while any of them runs,
|
|
100
|
+
// began when the first began, ended when the last ended, and is current only
|
|
101
|
+
// as of the least current. A failure of one connection makes the pass
|
|
102
|
+
// partial, not succeeded, and its error keeps the error of each connection
|
|
103
|
+
// that did not succeed, as Go's errors.Join does, one on each line.
|
|
104
|
+
function combined(passes) {
|
|
105
|
+
const started = {
|
|
106
|
+
startedAt: earliest(passes.map(({ startedAt }) => startedAt)),
|
|
107
|
+
lastSucceededAt: earliestOfAll(passes.map(({ lastSucceededAt }) => lastSucceededAt)),
|
|
108
|
+
currentAsOf: earliestOfAll(passes.map(({ currentAsOf }) => currentAsOf)),
|
|
109
|
+
};
|
|
110
|
+
const ended = passes.filter((pass) => pass.state !== 'running');
|
|
111
|
+
if (ended.length < passes.length)
|
|
112
|
+
return {
|
|
113
|
+
...started,
|
|
114
|
+
state: 'running',
|
|
115
|
+
completedAt: null,
|
|
116
|
+
error: null,
|
|
117
|
+
failureType: null,
|
|
118
|
+
};
|
|
119
|
+
const completedAt = latest(ended.map(({ completedAt }) => completedAt));
|
|
120
|
+
const error = ended
|
|
121
|
+
.flatMap((pass) => (pass.error === null ? [] : [pass.error]))
|
|
122
|
+
.join('\n');
|
|
123
|
+
if (ended.some(({ state }) => state === 'cancelled'))
|
|
124
|
+
return {
|
|
125
|
+
...started,
|
|
126
|
+
state: 'cancelled',
|
|
127
|
+
completedAt,
|
|
128
|
+
error,
|
|
129
|
+
failureType: null,
|
|
130
|
+
};
|
|
131
|
+
if (ended.every(({ state }) => state === 'succeeded'))
|
|
132
|
+
return {
|
|
133
|
+
...started,
|
|
134
|
+
state: 'succeeded',
|
|
135
|
+
completedAt,
|
|
136
|
+
error: null,
|
|
137
|
+
failureType: null,
|
|
138
|
+
};
|
|
139
|
+
return {
|
|
140
|
+
...started,
|
|
141
|
+
state: ended.every(({ state }) => state === 'failed')
|
|
142
|
+
? 'failed'
|
|
143
|
+
: 'partial',
|
|
144
|
+
completedAt,
|
|
145
|
+
error,
|
|
146
|
+
failureType: ended.some(({ failureType }) => failureType === 'config')
|
|
147
|
+
? 'config'
|
|
148
|
+
: 'system',
|
|
149
|
+
};
|
|
150
|
+
}
|
|
151
|
+
function earliest(instants) {
|
|
152
|
+
return instants.reduce((first, next) => Date.parse(next) < Date.parse(first) ? next : first);
|
|
153
|
+
}
|
|
154
|
+
function latest(instants) {
|
|
155
|
+
return instants.reduce((last, next) => Date.parse(next) > Date.parse(last) ? next : last);
|
|
156
|
+
}
|
|
157
|
+
// The earliest instant, or null when any of them is null.
|
|
158
|
+
function earliestOfAll(instants) {
|
|
159
|
+
const known = instants.filter((instant) => instant !== null);
|
|
160
|
+
return known.length < instants.length ? null : earliest(known);
|
|
161
|
+
}
|
|
162
|
+
// A timestamptz as the ISO text that elt-sqlite stores and gives.
|
|
163
|
+
function instant(value) {
|
|
164
|
+
if (value instanceof Date)
|
|
165
|
+
return value.toISOString();
|
|
166
|
+
throw new TypeError(`A sync history view gave ${String(value)} for a time`);
|
|
167
|
+
}
|
|
168
|
+
function optionalInstant(value) {
|
|
169
|
+
return value === null ? null : instant(value);
|
|
170
|
+
}
|
|
171
|
+
function failureType(value) {
|
|
172
|
+
switch (value) {
|
|
173
|
+
case 'config':
|
|
174
|
+
case 'system':
|
|
175
|
+
return value;
|
|
176
|
+
default:
|
|
177
|
+
throw new TypeError(`Unknown failure type ${String(value)}`);
|
|
178
|
+
}
|
|
179
|
+
}
|
|
180
|
+
function passState(value) {
|
|
181
|
+
switch (value) {
|
|
182
|
+
case 'running':
|
|
183
|
+
case 'succeeded':
|
|
184
|
+
case 'partial':
|
|
185
|
+
case 'failed':
|
|
186
|
+
case 'cancelled':
|
|
187
|
+
return value;
|
|
188
|
+
default:
|
|
189
|
+
throw new TypeError(`Unknown pass status ${String(value)}`);
|
|
190
|
+
}
|
|
191
|
+
}
|
|
@@ -28,4 +28,4 @@ export declare const syncHistoryTables: readonly ['CREATE SCHEMA IF NOT EXISTS _
|
|
|
28
28
|
)`, `CREATE TABLE IF NOT EXISTS _warehouse.stream_checks (
|
|
29
29
|
connector text NOT NULL, stream text NOT NULL, checked_at timestamptz NOT NULL,
|
|
30
30
|
PRIMARY KEY (connector, stream)
|
|
31
|
-
)
|
|
31
|
+
)`, "CREATE TABLE IF NOT EXISTS _warehouse.derived_status (\n dataset text PRIMARY KEY,\n kind text NOT NULL CHECK (kind IN ('view', 'computed')),\n state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),\n as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,\n inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint\n )"];
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { syncHistoryRelations } from '@eltjs/elt';
|
|
2
|
+
import { derivedStatusTable } from './postgres-derived-status.js';
|
|
2
3
|
// When connector's stream last matched its upstream: the start of its last
|
|
3
4
|
// successful pass, or a later check that found it unchanged, each no later
|
|
4
5
|
// than when the source said the upstream itself was last current. NULL if it
|
|
@@ -52,6 +53,7 @@ const queries = {
|
|
|
52
53
|
ORDER BY s.completed_at DESC, s.id DESC LIMIT 1
|
|
53
54
|
) success ON true
|
|
54
55
|
ORDER BY a.connector, c.stream, a.id DESC`,
|
|
56
|
+
derived_status: `SELECT dataset, kind, state, as_of, refreshed_at, refreshing_since, error, inputs, computed, empty, failed, pending FROM _warehouse.derived_status ORDER BY dataset`,
|
|
55
57
|
};
|
|
56
58
|
// How readers see the sync history: what each pass declared and loaded.
|
|
57
59
|
// The sync history keeps no row versions, so a reading as of a position
|
|
@@ -97,4 +99,5 @@ export const syncHistoryTables = [
|
|
|
97
99
|
connector text NOT NULL, stream text NOT NULL, checked_at timestamptz NOT NULL,
|
|
98
100
|
PRIMARY KEY (connector, stream)
|
|
99
101
|
)`,
|
|
102
|
+
derivedStatusTable,
|
|
100
103
|
];
|
|
@@ -3,7 +3,7 @@ import { SyncHistory, copyStatus, errorMessage, passError, passFailureType, pass
|
|
|
3
3
|
import { identifier } from './identifier.js';
|
|
4
4
|
import { PostgresDestination } from './postgres-destination.js';
|
|
5
5
|
import { grantReading, installNumbering } from './postgres-numbering.js';
|
|
6
|
-
import { lockSchema } from './postgres-session.js';
|
|
6
|
+
import { PostgresSession, lockSchema } from './postgres-session.js';
|
|
7
7
|
import { syncHistoryTables, syncHistoryViews, } from './postgres-sync-history-schema.js';
|
|
8
8
|
import { publishPostgresViews } from './postgres-views.js';
|
|
9
9
|
// Lists what readers of marts query, its views, from the comments every load
|
|
@@ -52,40 +52,53 @@ export class PostgresSyncHistory extends SyncHistory {
|
|
|
52
52
|
});
|
|
53
53
|
}
|
|
54
54
|
// Every selected stream is declared before reading, even if it produces no
|
|
55
|
-
// rows or the process dies
|
|
55
|
+
// rows or the process dies. The pass's own session holds a lock on its
|
|
56
|
+
// attempt until the pipeline disposes the pass, and Postgres lets go of
|
|
57
|
+
// the locks of a session that ends, also of a killed process. So an attempt
|
|
58
|
+
// of the connection that runs while no session holds its lock will get no
|
|
59
|
+
// outcome, and the next attempt records it as failed, as Que and GoodJob
|
|
60
|
+
// tell a running job from an abandoned one by its advisory lock.
|
|
56
61
|
async begin(connection, copies, at) {
|
|
57
62
|
const schema = this.#schema(connection);
|
|
58
|
-
const
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
${copy.to.name}, ${copy.configuration.syncMode},
|
|
71
|
-
${copy.configuration.destinationSyncMode}, ${coverage.description},
|
|
72
|
-
${transaction.json(coverage.selection)})`;
|
|
73
|
-
return attempt.id;
|
|
74
|
-
});
|
|
75
|
-
return {
|
|
76
|
-
finish: (outcomes, upstream, ended) => this.#finish(id, outcomes, upstream, ended),
|
|
77
|
-
fail: (error, failureType, ended) => this.#write(async (transaction) => {
|
|
78
|
-
await transaction `
|
|
79
|
-
UPDATE _warehouse.extraction_coverage SET status = 'failed',
|
|
80
|
-
failures = ${transaction.json([recordedFailure({ partition: null, error, failureType })])}
|
|
81
|
-
WHERE attempt_id = ${id}`;
|
|
63
|
+
const session = new PostgresSession(this.#url, 'elt-sync-history');
|
|
64
|
+
try {
|
|
65
|
+
const id = await session.sql.begin(async (transaction) => {
|
|
66
|
+
await failEnded(transaction, connection.name, at);
|
|
67
|
+
const [attempt] = await transaction `
|
|
68
|
+
INSERT INTO _warehouse.sync_attempts (connector, source, started_at)
|
|
69
|
+
VALUES (${connection.name}, ${connection.source.identity}, ${at}) RETURNING id`;
|
|
70
|
+
if (attempt === undefined)
|
|
71
|
+
throw new Error('Sync attempt was not recorded');
|
|
72
|
+
// A session lock outlives the transaction, so no reader sees the
|
|
73
|
+
// attempt before its lock is held. A try never waits: when the hash
|
|
74
|
+
// of another running attempt takes the key, that key is held anyway.
|
|
82
75
|
await transaction `
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
76
|
+
SELECT pg_try_advisory_lock(${transaction.unsafe(attemptLock)}, hashtext(${attempt.id}))`;
|
|
77
|
+
for (const { copy, coverage } of copies)
|
|
78
|
+
await transaction `
|
|
79
|
+
INSERT INTO _warehouse.extraction_coverage
|
|
80
|
+
(attempt_id, stream, target_schema, target_table, sync_mode,
|
|
81
|
+
destination_sync_mode, description, selection)
|
|
82
|
+
VALUES (${attempt.id}, ${copy.from.name}, ${schema},
|
|
83
|
+
${copy.to.name}, ${copy.configuration.syncMode},
|
|
84
|
+
${copy.configuration.destinationSyncMode}, ${coverage.description},
|
|
85
|
+
${transaction.json(coverage.selection)})`;
|
|
86
|
+
return attempt.id;
|
|
87
|
+
});
|
|
88
|
+
return {
|
|
89
|
+
finish: async (outcomes, upstream, ended) => {
|
|
90
|
+
await session.sql.begin((transaction) => finish(transaction, id, outcomes, upstream, ended));
|
|
91
|
+
},
|
|
92
|
+
fail: async (error, failureType, ended) => {
|
|
93
|
+
await session.sql.begin((transaction) => fail(transaction, [id], error, failureType, ended));
|
|
94
|
+
},
|
|
95
|
+
[Symbol.asyncDispose]: () => session[Symbol.asyncDispose](),
|
|
96
|
+
};
|
|
97
|
+
}
|
|
98
|
+
catch (error) {
|
|
99
|
+
await session[Symbol.asyncDispose]();
|
|
100
|
+
throw error;
|
|
101
|
+
}
|
|
89
102
|
}
|
|
90
103
|
// Creates the history's tables, publishes its views and the catalog, and
|
|
91
104
|
// grants each reader the numbering; safe to repeat.
|
|
@@ -113,26 +126,6 @@ export class PostgresSyncHistory extends SyncHistory {
|
|
|
113
126
|
throw new TypeError(`Connection ${name}: Postgres sync history records Postgres destinations only`);
|
|
114
127
|
return destination.schema;
|
|
115
128
|
}
|
|
116
|
-
#finish(id, outcomes, upstream, ended) {
|
|
117
|
-
return this.#write(async (transaction) => {
|
|
118
|
-
for (const outcome of outcomes)
|
|
119
|
-
await transaction `
|
|
120
|
-
UPDATE _warehouse.extraction_coverage SET
|
|
121
|
-
status = ${copyStatus(outcome)},
|
|
122
|
-
written_count = ${outcome.count}, deleted_count = ${outcome.deleted},
|
|
123
|
-
failures = ${transaction.json(outcome.failures.map(recordedFailure))}
|
|
124
|
-
WHERE attempt_id = ${id} AND stream = ${outcome.copy.from.name}`;
|
|
125
|
-
for (const [stream, at] of upstream)
|
|
126
|
-
await transaction `
|
|
127
|
-
UPDATE _warehouse.extraction_coverage SET upstream_current_as_of = ${at}
|
|
128
|
-
WHERE attempt_id = ${id} AND stream = ${stream}`;
|
|
129
|
-
await transaction `
|
|
130
|
-
UPDATE _warehouse.sync_attempts SET completed_at = ${ended},
|
|
131
|
-
status = ${passStatus(outcomes)}, error = ${passError(outcomes)},
|
|
132
|
-
failure_type = ${passFailureType(outcomes)}
|
|
133
|
-
WHERE id = ${id}`;
|
|
134
|
-
});
|
|
135
|
-
}
|
|
136
129
|
#write(work) {
|
|
137
130
|
return this.#connect((sql) => sql.begin(work));
|
|
138
131
|
}
|
|
@@ -150,3 +143,51 @@ export class PostgresSyncHistory extends SyncHistory {
|
|
|
150
143
|
}
|
|
151
144
|
}
|
|
152
145
|
}
|
|
146
|
+
// The key space of the lock that the session of each running attempt holds,
|
|
147
|
+
// apart from the two-key locks of the checkpoints and the commit numbers.
|
|
148
|
+
const attemptLock = `hashtext('elt:sync-attempt')`;
|
|
149
|
+
// Records as failed each attempt of connector that runs while no session
|
|
150
|
+
// holds its lock: the session that ran it ended first, so no outcome will
|
|
151
|
+
// come. Only Postgres knows which sessions live, so a try of each lock is the
|
|
152
|
+
// test, and the lock it takes ends with the transaction.
|
|
153
|
+
async function failEnded(transaction, connector, at) {
|
|
154
|
+
const ended = await transaction `
|
|
155
|
+
WITH running AS MATERIALIZED (
|
|
156
|
+
SELECT id FROM _warehouse.sync_attempts
|
|
157
|
+
WHERE connector = ${connector} AND status = 'running'
|
|
158
|
+
)
|
|
159
|
+
SELECT id FROM running
|
|
160
|
+
WHERE pg_try_advisory_xact_lock(${transaction.unsafe(attemptLock)}, hashtext(id::text))`;
|
|
161
|
+
if (ended.length > 0)
|
|
162
|
+
await fail(transaction, ended.map(({ id }) => id), new Error('The session running this pass ended before the pass did: its process stopped, or lost the database'), 'system', at);
|
|
163
|
+
}
|
|
164
|
+
// Attempts that produced no outcomes, so what they committed is unknown.
|
|
165
|
+
async function fail(transaction, ids, error, failureType, at) {
|
|
166
|
+
await transaction `
|
|
167
|
+
UPDATE _warehouse.extraction_coverage SET status = 'failed',
|
|
168
|
+
failures = ${transaction.json([recordedFailure({ partition: null, error, failureType })])}
|
|
169
|
+
WHERE attempt_id IN ${transaction(ids)}`;
|
|
170
|
+
await transaction `
|
|
171
|
+
UPDATE _warehouse.sync_attempts SET status = 'failed',
|
|
172
|
+
completed_at = ${at}, error = ${errorMessage(error)},
|
|
173
|
+
failure_type = ${failureType}
|
|
174
|
+
WHERE id IN ${transaction(ids)}`;
|
|
175
|
+
}
|
|
176
|
+
async function finish(transaction, id, outcomes, upstream, ended) {
|
|
177
|
+
for (const outcome of outcomes)
|
|
178
|
+
await transaction `
|
|
179
|
+
UPDATE _warehouse.extraction_coverage SET
|
|
180
|
+
status = ${copyStatus(outcome)},
|
|
181
|
+
written_count = ${outcome.count}, deleted_count = ${outcome.deleted},
|
|
182
|
+
failures = ${transaction.json(outcome.failures.map(recordedFailure))}
|
|
183
|
+
WHERE attempt_id = ${id} AND stream = ${outcome.copy.from.name}`;
|
|
184
|
+
for (const [stream, at] of upstream)
|
|
185
|
+
await transaction `
|
|
186
|
+
UPDATE _warehouse.extraction_coverage SET upstream_current_as_of = ${at}
|
|
187
|
+
WHERE attempt_id = ${id} AND stream = ${stream}`;
|
|
188
|
+
await transaction `
|
|
189
|
+
UPDATE _warehouse.sync_attempts SET completed_at = ${ended},
|
|
190
|
+
status = ${passStatus(outcomes)}, error = ${passError(outcomes)},
|
|
191
|
+
failure_type = ${passFailureType(outcomes)}
|
|
192
|
+
WHERE id = ${id}`;
|
|
193
|
+
}
|
package/dist/view.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { type Connection, DerivedDataset } from '@eltjs/elt';
|
|
1
|
+
import { type Connection, DerivedDataset, type PipelineClock } from '@eltjs/elt';
|
|
2
2
|
import { PostgresDestination } from './postgres-destination.ts';
|
|
3
3
|
import type { PostgresTable } from './postgres-table.ts';
|
|
4
4
|
export declare class View extends DerivedDataset<PostgresTable> {
|
|
@@ -14,9 +14,10 @@ export declare class View extends DerivedDataset<PostgresTable> {
|
|
|
14
14
|
});
|
|
15
15
|
get qualifiedName(): string;
|
|
16
16
|
validate(connections: readonly Connection<PostgresTable>[]): void;
|
|
17
|
-
refresh({ connections, earlier, }: {
|
|
17
|
+
refresh({ connections, earlier, clock, }: {
|
|
18
18
|
readonly connections: readonly Connection<PostgresTable>[];
|
|
19
19
|
readonly earlier: readonly DerivedDataset<PostgresTable>[];
|
|
20
|
+
readonly clock: PipelineClock;
|
|
20
21
|
}): Promise<void>;
|
|
21
22
|
}
|
|
22
23
|
export declare function made(connections: readonly Connection<PostgresTable>[], into: PostgresDestination): Set<string>;
|
package/dist/view.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import postgres from 'postgres';
|
|
2
|
-
import { DerivedDataset } from '@eltjs/elt';
|
|
2
|
+
import { DerivedDataset, errorMessage, } from '@eltjs/elt';
|
|
3
|
+
import { PostgresDerivedStatus } from './postgres-derived-status.js';
|
|
3
4
|
import { PostgresDestination } from './postgres-destination.js';
|
|
4
5
|
import { PostgresSession } from './postgres-session.js';
|
|
5
6
|
import { checkView, publishPostgresViews, } from './postgres-views.js';
|
|
@@ -33,8 +34,9 @@ export class View extends DerivedDataset {
|
|
|
33
34
|
if (made(connections, this.into).has(this.qualifiedName))
|
|
34
35
|
throw new TypeError(`View ${this.qualifiedName} has the name of a table or reader view a copy makes`);
|
|
35
36
|
}
|
|
36
|
-
async refresh({ connections, earlier, }) {
|
|
37
|
+
async refresh({ connections, earlier, clock, }) {
|
|
37
38
|
await using session = new PostgresSession(this.into.url, 'elt-derived');
|
|
39
|
+
const status = new PostgresDerivedStatus(session.sql, this.qualifiedName, 'view', clock);
|
|
38
40
|
try {
|
|
39
41
|
await session.sql.begin((transaction) => publishPostgresViews(transaction, {
|
|
40
42
|
schema: this.into.schema,
|
|
@@ -45,10 +47,14 @@ export class View extends DerivedDataset {
|
|
|
45
47
|
const missing = missingRelation(error, this.into.schema);
|
|
46
48
|
if (missing !== undefined &&
|
|
47
49
|
(made(connections, this.into).has(missing) ||
|
|
48
|
-
earlier.some((dataset) => derivedRelation(dataset) === missing)))
|
|
50
|
+
earlier.some((dataset) => derivedRelation(dataset) === missing))) {
|
|
51
|
+
await status.waiting();
|
|
49
52
|
return;
|
|
53
|
+
}
|
|
54
|
+
await status.failed(errorMessage(error));
|
|
50
55
|
throw error;
|
|
51
56
|
}
|
|
57
|
+
await status.current();
|
|
52
58
|
}
|
|
53
59
|
}
|
|
54
60
|
// The schema-qualified relation that Postgres found missing, with a bare name
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@eltjs/elt-postgresql",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "The Postgres destination of elt: each stream is a typed table in one schema, with a reader view in another, a history of each row for readings as of a position, and the views and computed datasets that a pipeline derives.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"repository": {
|
|
@@ -37,7 +37,7 @@
|
|
|
37
37
|
"access": "public"
|
|
38
38
|
},
|
|
39
39
|
"dependencies": {
|
|
40
|
-
"@eltjs/elt": "^0.
|
|
40
|
+
"@eltjs/elt": "^0.4.0",
|
|
41
41
|
"@zukhruf/async": "^0.8.27",
|
|
42
42
|
"postgres": "^3.4.9"
|
|
43
43
|
},
|