@eltjs/elt-postgresql 0.3.16 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -1
- package/dist/index.d.ts +1 -0
- package/dist/index.js +1 -0
- package/dist/postgres-columns.d.ts +2 -0
- package/dist/postgres-columns.js +11 -1
- package/dist/postgres-passes.d.ts +44 -0
- package/dist/postgres-passes.js +191 -0
- package/dist/postgres-sync-history.js +93 -52
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -73,7 +73,7 @@ Inferred columns follow the stream schema. The [string formats](../../elt/README
|
|
|
73
73
|
- A `TIMESTAMPTZ` keeps the instant. It does not keep the offset that the value had.
|
|
74
74
|
- Arrays keep the order of their elements, and load a JSON `null` as SQL `NULL`. Readers use `= ANY(...)`, `unnest()` and `cardinality()`. `marts.catalog` shows the type of the element.
|
|
75
75
|
- ISO dates count years astronomically, and Postgres does not. Thus the year `0000` loads as `0001 BC`, which is the same day. Each other year is written as it is given.
|
|
76
|
-
- Explicit columns are `columns.text`, `integer`, `real`, `boolean`, `date`, `timestamp` and `blob(field)`, with `.notNull()` and `.primaryKey()`.
|
|
76
|
+
- Explicit columns are `columns.text`, `integer`, `real`, `boolean`, `date`, `timestamp` and `blob(field)`, with `.notNull()` and `.primaryKey()`. `columns.int64(field)` is a `BIGINT` that loads decimal text exactly, and `columns.localTimestamp(field, precision)` is a `TIMESTAMP` for SQL Server's `datetime2`, by the rows of the table above: a precision past 6 loads as text.
|
|
77
77
|
- A plain `blob()` stores `BYTEA`. `blob().from(file)` keeps the original file in chunk tables, and `text().from(file).parse(parser)` stores the parsed text as `TEXT`. See [file bytes in the database](#file-bytes-in-the-database).
|
|
78
78
|
- Table, view and column names follow the [naming rule](../../elt/README.md#names). Postgres compares names with their case, keeps 63 bytes of a name, and reserves `loaded_at`, `committed_in`, `replaced_in` and the system column names `xmin`, `xmax`, `cmin`, `cmax`, `ctid` and `tableoid`. The naming rule turns a field `_elt_row_hash` into `elt_row_hash`, so no field takes that name.
|
|
79
79
|
|
|
@@ -296,7 +296,9 @@ A refresh is a load in `_warehouse._elt_loads`, under the name of its dataset, s
|
|
|
296
296
|
- Before a pass reads, the history inserts one `sync_attempts` row, with the connection name as `connector` and the source identity as `source`. It inserts one `extraction_coverage` row for each selected stream, with the declared `coverage(stream)` of the source and the schema of the destination as `target_schema`. The outcomes close them.
|
|
297
297
|
- A run declares each stream that a connection selected. A watch pass reads only the streams whose change signals changed, so an attempt vouches only for its own `extraction_coverage` rows.
|
|
298
298
|
- A connection that is not valid or that stopped, a failed change signal and a failed upkeep each record a failed attempt with no outcomes.
|
|
299
|
+
- Each attempt has a session of its own until its pass ends, and that session holds a lock on the attempt. A pass whose process stopped, for example a container restart, stays `running` until the next attempt of its connection begins. That attempt records it as `failed`, because no session holds its lock. The pipeline records a failed change signal while a pass of the same connection runs, and the running pass stays `running`. No host option is necessary, because Postgres knows which sessions are alive. See [ADR 0009](./docs/adr/0009-a-running-attempt-holds-a-lock-of-its-session.md).
|
|
299
300
|
- `_warehouse` tables are made only when they are missing.
|
|
301
|
+
- `readPassStatus(url)` gives the latest pass of the database and the latest outcome of each stream, as `readPassStatus(path)` of elt-sqlite gives them for a file. The latest passes of the connections show as one pass. It is `running` when one connection runs. Else it is `cancelled` when one pass stopped, `succeeded` when all passes succeeded, `failed` when all passes failed, and `partial` in all other cases. It starts with the first pass and completes with the last. Its error has the error of each pass that did not succeed, one on each line. Its failure type is `config` when one pass has a `config` failure. Its last success and its `current_as_of` are the earliest of the passes, or null when one pass has no value. Each stream has the error of its latest attempt when its copy failed or was partial in that attempt: the error of each failure, one on each line. Times are ISO text. A database with no sync history gives no pass and no streams. It reads `marts`, so a reader role that may read `marts` can call it with its own URL.
|
|
300
302
|
|
|
301
303
|
The relations in `marts`:
|
|
302
304
|
|
|
@@ -341,6 +343,7 @@ Each [view](#declare-a-view) and [computed dataset](#declare-a-computed-dataset)
|
|
|
341
343
|
- [ADR 0006](./docs/adr/0006-a-change-of-a-table-makes-the-views-over-it-again.md): a change of a table makes the views over it again.
|
|
342
344
|
- [ADR 0007](./docs/adr/0007-readers-are-named-on-the-sync-history.md): readers are named on the sync history.
|
|
343
345
|
- [ADR 0008](./docs/adr/0008-derived-datasets-follow-elt-sqlite.md): derived datasets follow elt-sqlite.
|
|
346
|
+
- [ADR 0009](./docs/adr/0009-a-running-attempt-holds-a-lock-of-its-session.md): a running attempt holds a lock of its session.
|
|
344
347
|
|
|
345
348
|
## Development
|
|
346
349
|
|
package/dist/index.d.ts
CHANGED
|
@@ -4,6 +4,7 @@ export { PostgresCheckpointStore } from './postgres-checkpoint-store.ts';
|
|
|
4
4
|
export { PostgresColumn } from './postgres-column.ts';
|
|
5
5
|
export { PostgresColumns } from './postgres-columns.ts';
|
|
6
6
|
export { PostgresDestination } from './postgres-destination.ts';
|
|
7
|
+
export { type PassState, type PassStatus, readPassStatus, type StreamPassStatus, } from './postgres-passes.ts';
|
|
7
8
|
export { PostgresSyncHistory } from './postgres-sync-history.ts';
|
|
8
9
|
export { PostgresTable } from './postgres-table.ts';
|
|
9
10
|
export { View } from './view.ts';
|
package/dist/index.js
CHANGED
|
@@ -3,6 +3,7 @@ export { PostgresCheckpointStore } from './postgres-checkpoint-store.js';
|
|
|
3
3
|
export { PostgresColumn } from './postgres-column.js';
|
|
4
4
|
export { PostgresColumns } from './postgres-columns.js';
|
|
5
5
|
export { PostgresDestination } from './postgres-destination.js';
|
|
6
|
+
export { readPassStatus, } from './postgres-passes.js';
|
|
6
7
|
export { PostgresSyncHistory } from './postgres-sync-history.js';
|
|
7
8
|
export { PostgresTable } from './postgres-table.js';
|
|
8
9
|
export { View } from './view.js';
|
|
@@ -5,9 +5,11 @@ export declare class PostgresColumns {
|
|
|
5
5
|
static fromSchema({ properties, required, }: Stream['jsonSchema']): readonly PostgresColumn[];
|
|
6
6
|
text(field: string): PostgresColumn;
|
|
7
7
|
integer(field: string): PostgresColumn;
|
|
8
|
+
int64(field: string): PostgresColumn;
|
|
8
9
|
real(field: string): PostgresColumn;
|
|
9
10
|
boolean(field: string): PostgresColumn;
|
|
10
11
|
date(field: string): PostgresColumn;
|
|
11
12
|
timestamp(field: string): PostgresColumn;
|
|
13
|
+
localTimestamp(field: string, precision: number): PostgresColumn;
|
|
12
14
|
blob(field: string): PostgresColumn;
|
|
13
15
|
}
|
package/dist/postgres-columns.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { declaredFormat } from '@eltjs/elt';
|
|
1
|
+
import { Int64Format, LocalTimestampFormat, declaredFormat, } from '@eltjs/elt';
|
|
2
2
|
import { PostgresColumn } from './postgres-column.js';
|
|
3
3
|
function scalarType(name, type, format) {
|
|
4
4
|
switch (type) {
|
|
@@ -55,6 +55,11 @@ export class PostgresColumns {
|
|
|
55
55
|
integer(field) {
|
|
56
56
|
return this.#column(field, 'integer');
|
|
57
57
|
}
|
|
58
|
+
// A signed 64-bit integer, which a stream spells as decimal text (format
|
|
59
|
+
// int64), so a value past Number.MAX_SAFE_INTEGER keeps every digit.
|
|
60
|
+
int64(field) {
|
|
61
|
+
return this.#column(field, new Int64Format());
|
|
62
|
+
}
|
|
58
63
|
real(field) {
|
|
59
64
|
return this.#column(field, 'real');
|
|
60
65
|
}
|
|
@@ -67,6 +72,11 @@ export class PostgresColumns {
|
|
|
67
72
|
timestamp(field) {
|
|
68
73
|
return this.#column(field, 'timestamp');
|
|
69
74
|
}
|
|
75
|
+
// A wall-clock date and time with no zone and precision digits of a second
|
|
76
|
+
// (format date-time-local), as SQL Server's datetime2 holds it.
|
|
77
|
+
localTimestamp(field, precision) {
|
|
78
|
+
return this.#column(field, new LocalTimestampFormat(precision));
|
|
79
|
+
}
|
|
70
80
|
blob(field) {
|
|
71
81
|
return this.#column(field, 'blob');
|
|
72
82
|
}
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
import type { FailureType, SyncStatus } from '@eltjs/elt';
|
|
2
|
+
export type PassState = SyncStatus | 'running';
|
|
3
|
+
type Started = {
|
|
4
|
+
readonly startedAt: string;
|
|
5
|
+
readonly lastSucceededAt: string | null;
|
|
6
|
+
readonly currentAsOf: string | null;
|
|
7
|
+
};
|
|
8
|
+
export type PassStatus = (Started & {
|
|
9
|
+
readonly state: 'running';
|
|
10
|
+
readonly completedAt: null;
|
|
11
|
+
readonly error: null;
|
|
12
|
+
readonly failureType: null;
|
|
13
|
+
}) | (Started & {
|
|
14
|
+
readonly state: 'succeeded';
|
|
15
|
+
readonly completedAt: string;
|
|
16
|
+
readonly error: null;
|
|
17
|
+
readonly failureType: null;
|
|
18
|
+
}) | (Started & {
|
|
19
|
+
readonly state: 'cancelled';
|
|
20
|
+
readonly completedAt: string;
|
|
21
|
+
readonly error: string;
|
|
22
|
+
readonly failureType: null;
|
|
23
|
+
}) | (Started & {
|
|
24
|
+
readonly state: 'partial' | 'failed';
|
|
25
|
+
readonly completedAt: string;
|
|
26
|
+
readonly error: string;
|
|
27
|
+
readonly failureType: FailureType;
|
|
28
|
+
});
|
|
29
|
+
export type StreamPassStatus = {
|
|
30
|
+
readonly stream: string;
|
|
31
|
+
readonly lastSucceededAt: string | null;
|
|
32
|
+
readonly currentAsOf: string | null;
|
|
33
|
+
} & ({
|
|
34
|
+
readonly state: 'running' | 'succeeded' | 'cancelled';
|
|
35
|
+
readonly error: null;
|
|
36
|
+
} | {
|
|
37
|
+
readonly state: 'partial' | 'failed';
|
|
38
|
+
readonly error: string;
|
|
39
|
+
});
|
|
40
|
+
export declare function readPassStatus(url: string): Promise<{
|
|
41
|
+
readonly pass: PassStatus | null;
|
|
42
|
+
readonly streams: readonly StreamPassStatus[];
|
|
43
|
+
}>;
|
|
44
|
+
export {};
|
|
@@ -0,0 +1,191 @@
|
|
|
1
|
+
import { PostgresSession } from './postgres-session.js';
|
|
2
|
+
// The latest pass of the connections that the sync history of a database
|
|
3
|
+
// records, and each stream's latest outcome, with the errors of its failures
|
|
4
|
+
// in that attempt, one on each line; null and empty until the history
|
|
5
|
+
// recorded a pass, also while no history is installed. The latest passes of
|
|
6
|
+
// several connections read as one pass, as the passes of a SQLite file that
|
|
7
|
+
// several connections load do. One snapshot reads both.
|
|
8
|
+
export async function readPassStatus(url) {
|
|
9
|
+
await using session = new PostgresSession(url, 'elt-pass-status');
|
|
10
|
+
try {
|
|
11
|
+
return await session.sql.begin('ISOLATION LEVEL REPEATABLE READ READ ONLY', async (sql) => {
|
|
12
|
+
const [first, ...others] = (await sql `
|
|
13
|
+
SELECT status, started_at, completed_at, error, failure_type,
|
|
14
|
+
last_successful_sync_at, current_as_of
|
|
15
|
+
FROM marts.sync_status`).map((row) => recorded(row, passState(row.status)));
|
|
16
|
+
const streams = await streamOutcomes(sql);
|
|
17
|
+
return {
|
|
18
|
+
pass: first === undefined ? null : combined([first, ...others]),
|
|
19
|
+
streams,
|
|
20
|
+
};
|
|
21
|
+
});
|
|
22
|
+
}
|
|
23
|
+
catch (error) {
|
|
24
|
+
if (Object(error).code === '42P01')
|
|
25
|
+
return { pass: null, streams: [] };
|
|
26
|
+
throw error;
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
async function streamOutcomes(sql) {
|
|
30
|
+
const rows = await sql `
|
|
31
|
+
SELECT s.stream, s.status, s.last_successful_sync_at, s.current_as_of,
|
|
32
|
+
(SELECT string_agg(f.value->>'error', E'\n' ORDER BY f.ordinality)
|
|
33
|
+
FROM marts.extraction_coverage c,
|
|
34
|
+
jsonb_array_elements(c.failures) WITH ORDINALITY f
|
|
35
|
+
WHERE c.attempt_id = s.latest_attempt_id AND c.stream = s.stream) AS error
|
|
36
|
+
FROM marts.stream_status s ORDER BY s.stream COLLATE "C"`;
|
|
37
|
+
return rows.map((row) => streamRecorded(row, passState(row.status)));
|
|
38
|
+
}
|
|
39
|
+
// A stream_status row as the stream's latest outcome: only a copy that
|
|
40
|
+
// failed has an error.
|
|
41
|
+
function streamRecorded(row, state) {
|
|
42
|
+
const outcome = {
|
|
43
|
+
stream: String(row.stream),
|
|
44
|
+
lastSucceededAt: optionalInstant(row.last_successful_sync_at),
|
|
45
|
+
currentAsOf: optionalInstant(row.current_as_of),
|
|
46
|
+
};
|
|
47
|
+
switch (state) {
|
|
48
|
+
case 'partial':
|
|
49
|
+
case 'failed':
|
|
50
|
+
return { ...outcome, state, error: String(row.error) };
|
|
51
|
+
default:
|
|
52
|
+
return { ...outcome, state, error: null };
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
// A sync_status row as the pass it records: the history completes every pass
|
|
56
|
+
// that ended, and gives an error to each that did not load completely.
|
|
57
|
+
function recorded(row, state) {
|
|
58
|
+
const started = {
|
|
59
|
+
startedAt: instant(row.started_at),
|
|
60
|
+
lastSucceededAt: optionalInstant(row.last_successful_sync_at),
|
|
61
|
+
currentAsOf: optionalInstant(row.current_as_of),
|
|
62
|
+
};
|
|
63
|
+
switch (state) {
|
|
64
|
+
case 'running':
|
|
65
|
+
return {
|
|
66
|
+
...started,
|
|
67
|
+
state,
|
|
68
|
+
completedAt: null,
|
|
69
|
+
error: null,
|
|
70
|
+
failureType: null,
|
|
71
|
+
};
|
|
72
|
+
case 'succeeded':
|
|
73
|
+
return {
|
|
74
|
+
...started,
|
|
75
|
+
state,
|
|
76
|
+
completedAt: instant(row.completed_at),
|
|
77
|
+
error: null,
|
|
78
|
+
failureType: null,
|
|
79
|
+
};
|
|
80
|
+
case 'cancelled':
|
|
81
|
+
return {
|
|
82
|
+
...started,
|
|
83
|
+
state,
|
|
84
|
+
completedAt: instant(row.completed_at),
|
|
85
|
+
error: String(row.error),
|
|
86
|
+
failureType: null,
|
|
87
|
+
};
|
|
88
|
+
default:
|
|
89
|
+
return {
|
|
90
|
+
...started,
|
|
91
|
+
state,
|
|
92
|
+
completedAt: instant(row.completed_at),
|
|
93
|
+
error: String(row.error),
|
|
94
|
+
failureType: failureType(row.failure_type),
|
|
95
|
+
};
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
// The latest passes of the connections of one database as one pass, which
|
|
99
|
+
// is only as good as its worst connection: it runs while any of them runs,
|
|
100
|
+
// began when the first began, ended when the last ended, and is current only
|
|
101
|
+
// as of the least current. A failure of one connection makes the pass
|
|
102
|
+
// partial, not succeeded, and its error keeps the error of each connection
|
|
103
|
+
// that did not succeed, as Go's errors.Join does, one on each line.
|
|
104
|
+
function combined(passes) {
|
|
105
|
+
const started = {
|
|
106
|
+
startedAt: earliest(passes.map(({ startedAt }) => startedAt)),
|
|
107
|
+
lastSucceededAt: earliestOfAll(passes.map(({ lastSucceededAt }) => lastSucceededAt)),
|
|
108
|
+
currentAsOf: earliestOfAll(passes.map(({ currentAsOf }) => currentAsOf)),
|
|
109
|
+
};
|
|
110
|
+
const ended = passes.filter((pass) => pass.state !== 'running');
|
|
111
|
+
if (ended.length < passes.length)
|
|
112
|
+
return {
|
|
113
|
+
...started,
|
|
114
|
+
state: 'running',
|
|
115
|
+
completedAt: null,
|
|
116
|
+
error: null,
|
|
117
|
+
failureType: null,
|
|
118
|
+
};
|
|
119
|
+
const completedAt = latest(ended.map(({ completedAt }) => completedAt));
|
|
120
|
+
const error = ended
|
|
121
|
+
.flatMap((pass) => (pass.error === null ? [] : [pass.error]))
|
|
122
|
+
.join('\n');
|
|
123
|
+
if (ended.some(({ state }) => state === 'cancelled'))
|
|
124
|
+
return {
|
|
125
|
+
...started,
|
|
126
|
+
state: 'cancelled',
|
|
127
|
+
completedAt,
|
|
128
|
+
error,
|
|
129
|
+
failureType: null,
|
|
130
|
+
};
|
|
131
|
+
if (ended.every(({ state }) => state === 'succeeded'))
|
|
132
|
+
return {
|
|
133
|
+
...started,
|
|
134
|
+
state: 'succeeded',
|
|
135
|
+
completedAt,
|
|
136
|
+
error: null,
|
|
137
|
+
failureType: null,
|
|
138
|
+
};
|
|
139
|
+
return {
|
|
140
|
+
...started,
|
|
141
|
+
state: ended.every(({ state }) => state === 'failed')
|
|
142
|
+
? 'failed'
|
|
143
|
+
: 'partial',
|
|
144
|
+
completedAt,
|
|
145
|
+
error,
|
|
146
|
+
failureType: ended.some(({ failureType }) => failureType === 'config')
|
|
147
|
+
? 'config'
|
|
148
|
+
: 'system',
|
|
149
|
+
};
|
|
150
|
+
}
|
|
151
|
+
function earliest(instants) {
|
|
152
|
+
return instants.reduce((first, next) => Date.parse(next) < Date.parse(first) ? next : first);
|
|
153
|
+
}
|
|
154
|
+
function latest(instants) {
|
|
155
|
+
return instants.reduce((last, next) => Date.parse(next) > Date.parse(last) ? next : last);
|
|
156
|
+
}
|
|
157
|
+
// The earliest instant, or null when any of them is null.
|
|
158
|
+
function earliestOfAll(instants) {
|
|
159
|
+
const known = instants.filter((instant) => instant !== null);
|
|
160
|
+
return known.length < instants.length ? null : earliest(known);
|
|
161
|
+
}
|
|
162
|
+
// A timestamptz as the ISO text that elt-sqlite stores and gives.
|
|
163
|
+
function instant(value) {
|
|
164
|
+
if (value instanceof Date)
|
|
165
|
+
return value.toISOString();
|
|
166
|
+
throw new TypeError(`A sync history view gave ${String(value)} for a time`);
|
|
167
|
+
}
|
|
168
|
+
function optionalInstant(value) {
|
|
169
|
+
return value === null ? null : instant(value);
|
|
170
|
+
}
|
|
171
|
+
function failureType(value) {
|
|
172
|
+
switch (value) {
|
|
173
|
+
case 'config':
|
|
174
|
+
case 'system':
|
|
175
|
+
return value;
|
|
176
|
+
default:
|
|
177
|
+
throw new TypeError(`Unknown failure type ${String(value)}`);
|
|
178
|
+
}
|
|
179
|
+
}
|
|
180
|
+
function passState(value) {
|
|
181
|
+
switch (value) {
|
|
182
|
+
case 'running':
|
|
183
|
+
case 'succeeded':
|
|
184
|
+
case 'partial':
|
|
185
|
+
case 'failed':
|
|
186
|
+
case 'cancelled':
|
|
187
|
+
return value;
|
|
188
|
+
default:
|
|
189
|
+
throw new TypeError(`Unknown pass status ${String(value)}`);
|
|
190
|
+
}
|
|
191
|
+
}
|
|
@@ -3,7 +3,7 @@ import { SyncHistory, copyStatus, errorMessage, passError, passFailureType, pass
|
|
|
3
3
|
import { identifier } from './identifier.js';
|
|
4
4
|
import { PostgresDestination } from './postgres-destination.js';
|
|
5
5
|
import { grantReading, installNumbering } from './postgres-numbering.js';
|
|
6
|
-
import { lockSchema } from './postgres-session.js';
|
|
6
|
+
import { PostgresSession, lockSchema } from './postgres-session.js';
|
|
7
7
|
import { syncHistoryTables, syncHistoryViews, } from './postgres-sync-history-schema.js';
|
|
8
8
|
import { publishPostgresViews } from './postgres-views.js';
|
|
9
9
|
// Lists what readers of marts query, its views, from the comments every load
|
|
@@ -52,40 +52,53 @@ export class PostgresSyncHistory extends SyncHistory {
|
|
|
52
52
|
});
|
|
53
53
|
}
|
|
54
54
|
// Every selected stream is declared before reading, even if it produces no
|
|
55
|
-
// rows or the process dies
|
|
55
|
+
// rows or the process dies. The pass's own session holds a lock on its
|
|
56
|
+
// attempt until the pipeline disposes the pass, and Postgres lets go of
|
|
57
|
+
// the locks of a session that ends, also of a killed process. So an attempt
|
|
58
|
+
// of the connection that runs while no session holds its lock will get no
|
|
59
|
+
// outcome, and the next attempt records it as failed, as Que and GoodJob
|
|
60
|
+
// tell a running job from an abandoned one by its advisory lock.
|
|
56
61
|
async begin(connection, copies, at) {
|
|
57
62
|
const schema = this.#schema(connection);
|
|
58
|
-
const
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
${copy.to.name}, ${copy.configuration.syncMode},
|
|
71
|
-
${copy.configuration.destinationSyncMode}, ${coverage.description},
|
|
72
|
-
${transaction.json(coverage.selection)})`;
|
|
73
|
-
return attempt.id;
|
|
74
|
-
});
|
|
75
|
-
return {
|
|
76
|
-
finish: (outcomes, upstream, ended) => this.#finish(id, outcomes, upstream, ended),
|
|
77
|
-
fail: (error, failureType, ended) => this.#write(async (transaction) => {
|
|
78
|
-
await transaction `
|
|
79
|
-
UPDATE _warehouse.extraction_coverage SET status = 'failed',
|
|
80
|
-
failures = ${transaction.json([recordedFailure({ partition: null, error, failureType })])}
|
|
81
|
-
WHERE attempt_id = ${id}`;
|
|
63
|
+
const session = new PostgresSession(this.#url, 'elt-sync-history');
|
|
64
|
+
try {
|
|
65
|
+
const id = await session.sql.begin(async (transaction) => {
|
|
66
|
+
await failEnded(transaction, connection.name, at);
|
|
67
|
+
const [attempt] = await transaction `
|
|
68
|
+
INSERT INTO _warehouse.sync_attempts (connector, source, started_at)
|
|
69
|
+
VALUES (${connection.name}, ${connection.source.identity}, ${at}) RETURNING id`;
|
|
70
|
+
if (attempt === undefined)
|
|
71
|
+
throw new Error('Sync attempt was not recorded');
|
|
72
|
+
// A session lock outlives the transaction, so no reader sees the
|
|
73
|
+
// attempt before its lock is held. A try never waits: when the hash
|
|
74
|
+
// of another running attempt takes the key, that key is held anyway.
|
|
82
75
|
await transaction `
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
76
|
+
SELECT pg_try_advisory_lock(${transaction.unsafe(attemptLock)}, hashtext(${attempt.id}))`;
|
|
77
|
+
for (const { copy, coverage } of copies)
|
|
78
|
+
await transaction `
|
|
79
|
+
INSERT INTO _warehouse.extraction_coverage
|
|
80
|
+
(attempt_id, stream, target_schema, target_table, sync_mode,
|
|
81
|
+
destination_sync_mode, description, selection)
|
|
82
|
+
VALUES (${attempt.id}, ${copy.from.name}, ${schema},
|
|
83
|
+
${copy.to.name}, ${copy.configuration.syncMode},
|
|
84
|
+
${copy.configuration.destinationSyncMode}, ${coverage.description},
|
|
85
|
+
${transaction.json(coverage.selection)})`;
|
|
86
|
+
return attempt.id;
|
|
87
|
+
});
|
|
88
|
+
return {
|
|
89
|
+
finish: async (outcomes, upstream, ended) => {
|
|
90
|
+
await session.sql.begin((transaction) => finish(transaction, id, outcomes, upstream, ended));
|
|
91
|
+
},
|
|
92
|
+
fail: async (error, failureType, ended) => {
|
|
93
|
+
await session.sql.begin((transaction) => fail(transaction, [id], error, failureType, ended));
|
|
94
|
+
},
|
|
95
|
+
[Symbol.asyncDispose]: () => session[Symbol.asyncDispose](),
|
|
96
|
+
};
|
|
97
|
+
}
|
|
98
|
+
catch (error) {
|
|
99
|
+
await session[Symbol.asyncDispose]();
|
|
100
|
+
throw error;
|
|
101
|
+
}
|
|
89
102
|
}
|
|
90
103
|
// Creates the history's tables, publishes its views and the catalog, and
|
|
91
104
|
// grants each reader the numbering; safe to repeat.
|
|
@@ -113,26 +126,6 @@ export class PostgresSyncHistory extends SyncHistory {
|
|
|
113
126
|
throw new TypeError(`Connection ${name}: Postgres sync history records Postgres destinations only`);
|
|
114
127
|
return destination.schema;
|
|
115
128
|
}
|
|
116
|
-
#finish(id, outcomes, upstream, ended) {
|
|
117
|
-
return this.#write(async (transaction) => {
|
|
118
|
-
for (const outcome of outcomes)
|
|
119
|
-
await transaction `
|
|
120
|
-
UPDATE _warehouse.extraction_coverage SET
|
|
121
|
-
status = ${copyStatus(outcome)},
|
|
122
|
-
written_count = ${outcome.count}, deleted_count = ${outcome.deleted},
|
|
123
|
-
failures = ${transaction.json(outcome.failures.map(recordedFailure))}
|
|
124
|
-
WHERE attempt_id = ${id} AND stream = ${outcome.copy.from.name}`;
|
|
125
|
-
for (const [stream, at] of upstream)
|
|
126
|
-
await transaction `
|
|
127
|
-
UPDATE _warehouse.extraction_coverage SET upstream_current_as_of = ${at}
|
|
128
|
-
WHERE attempt_id = ${id} AND stream = ${stream}`;
|
|
129
|
-
await transaction `
|
|
130
|
-
UPDATE _warehouse.sync_attempts SET completed_at = ${ended},
|
|
131
|
-
status = ${passStatus(outcomes)}, error = ${passError(outcomes)},
|
|
132
|
-
failure_type = ${passFailureType(outcomes)}
|
|
133
|
-
WHERE id = ${id}`;
|
|
134
|
-
});
|
|
135
|
-
}
|
|
136
129
|
#write(work) {
|
|
137
130
|
return this.#connect((sql) => sql.begin(work));
|
|
138
131
|
}
|
|
@@ -150,3 +143,51 @@ export class PostgresSyncHistory extends SyncHistory {
|
|
|
150
143
|
}
|
|
151
144
|
}
|
|
152
145
|
}
|
|
146
|
+
// The key space of the lock that the session of each running attempt holds,
|
|
147
|
+
// apart from the two-key locks of the checkpoints and the commit numbers.
|
|
148
|
+
const attemptLock = `hashtext('elt:sync-attempt')`;
|
|
149
|
+
// Records as failed each attempt of connector that runs while no session
|
|
150
|
+
// holds its lock: the session that ran it ended first, so no outcome will
|
|
151
|
+
// come. Only Postgres knows which sessions live, so a try of each lock is the
|
|
152
|
+
// test, and the lock it takes ends with the transaction.
|
|
153
|
+
async function failEnded(transaction, connector, at) {
|
|
154
|
+
const ended = await transaction `
|
|
155
|
+
WITH running AS MATERIALIZED (
|
|
156
|
+
SELECT id FROM _warehouse.sync_attempts
|
|
157
|
+
WHERE connector = ${connector} AND status = 'running'
|
|
158
|
+
)
|
|
159
|
+
SELECT id FROM running
|
|
160
|
+
WHERE pg_try_advisory_xact_lock(${transaction.unsafe(attemptLock)}, hashtext(id::text))`;
|
|
161
|
+
if (ended.length > 0)
|
|
162
|
+
await fail(transaction, ended.map(({ id }) => id), new Error('The session running this pass ended before the pass did: its process stopped, or lost the database'), 'system', at);
|
|
163
|
+
}
|
|
164
|
+
// Attempts that produced no outcomes, so what they committed is unknown.
|
|
165
|
+
async function fail(transaction, ids, error, failureType, at) {
|
|
166
|
+
await transaction `
|
|
167
|
+
UPDATE _warehouse.extraction_coverage SET status = 'failed',
|
|
168
|
+
failures = ${transaction.json([recordedFailure({ partition: null, error, failureType })])}
|
|
169
|
+
WHERE attempt_id IN ${transaction(ids)}`;
|
|
170
|
+
await transaction `
|
|
171
|
+
UPDATE _warehouse.sync_attempts SET status = 'failed',
|
|
172
|
+
completed_at = ${at}, error = ${errorMessage(error)},
|
|
173
|
+
failure_type = ${failureType}
|
|
174
|
+
WHERE id IN ${transaction(ids)}`;
|
|
175
|
+
}
|
|
176
|
+
async function finish(transaction, id, outcomes, upstream, ended) {
|
|
177
|
+
for (const outcome of outcomes)
|
|
178
|
+
await transaction `
|
|
179
|
+
UPDATE _warehouse.extraction_coverage SET
|
|
180
|
+
status = ${copyStatus(outcome)},
|
|
181
|
+
written_count = ${outcome.count}, deleted_count = ${outcome.deleted},
|
|
182
|
+
failures = ${transaction.json(outcome.failures.map(recordedFailure))}
|
|
183
|
+
WHERE attempt_id = ${id} AND stream = ${outcome.copy.from.name}`;
|
|
184
|
+
for (const [stream, at] of upstream)
|
|
185
|
+
await transaction `
|
|
186
|
+
UPDATE _warehouse.extraction_coverage SET upstream_current_as_of = ${at}
|
|
187
|
+
WHERE attempt_id = ${id} AND stream = ${stream}`;
|
|
188
|
+
await transaction `
|
|
189
|
+
UPDATE _warehouse.sync_attempts SET completed_at = ${ended},
|
|
190
|
+
status = ${passStatus(outcomes)}, error = ${passError(outcomes)},
|
|
191
|
+
failure_type = ${passFailureType(outcomes)}
|
|
192
|
+
WHERE id = ${id}`;
|
|
193
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@eltjs/elt-postgresql",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "The Postgres destination of elt: each stream is a typed table in one schema, with a reader view in another, a history of each row for readings as of a position, and the views and computed datasets that a pipeline derives.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"repository": {
|
|
@@ -37,7 +37,7 @@
|
|
|
37
37
|
"access": "public"
|
|
38
38
|
},
|
|
39
39
|
"dependencies": {
|
|
40
|
-
"@eltjs/elt": "^0.
|
|
40
|
+
"@eltjs/elt": "^0.4.0",
|
|
41
41
|
"@zukhruf/async": "^0.8.27",
|
|
42
42
|
"postgres": "^3.4.9"
|
|
43
43
|
},
|