@eltjs/elt-postgresql 0.3.15 → 0.3.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -8
- package/dist/computed-dataset.js +40 -2
- package/dist/postgres-derived-status.d.ts +20 -0
- package/dist/postgres-derived-status.js +77 -0
- package/dist/postgres-sync-history-schema.d.ts +1 -1
- package/dist/postgres-sync-history-schema.js +3 -0
- package/dist/view.d.ts +3 -2
- package/dist/view.js +9 -3
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -291,7 +291,7 @@ A refresh is a load in `_warehouse._elt_loads`, under the name of its dataset, s
|
|
|
291
291
|
|
|
292
292
|
- It records only connections with a Postgres destination. It refuses other connections when the pipeline validates.
|
|
293
293
|
- It records into `_warehouse.sync_attempts`, `_warehouse.extraction_coverage` and `_warehouse.stream_checks`. The last one holds the latest check of each stream by a watch that found it unchanged.
|
|
294
|
-
- `install()` makes those tables and publishes the `marts` views `sync_attempts`, `extraction_coverage`, `sync_status`, `stream_status` and `catalog`, in one transaction. The pipeline calls it before each run and watch, and it is safe to repeat. See [ADR 0005](./docs/adr/0005-the-history-install-publishes-the-catalog.md).
|
|
294
|
+
- `install()` makes those tables and `_warehouse.derived_status`, and publishes the `marts` views `sync_attempts`, `extraction_coverage`, `sync_status`, `stream_status`, `derived_status` and `catalog`, in one transaction. The pipeline calls it before each run and watch, and it is safe to repeat. See [ADR 0005](./docs/adr/0005-the-history-install-publishes-the-catalog.md).
|
|
295
295
|
- `new PostgresSyncHistory({ url, readers: ['agent_reader'] })` names the roles that read the database as of a position with their own session: `new PostgresDestination({ url: readerUrl, schema })`, then `positions()` and `reading()`. Each install grants each reader what a reading needs, which is the numbering in `_warehouse` and nothing else of it, as dbt applies the grants that a model declares after each build. Thus no host names the objects of the numbering, and a reader keeps its grants when the numbering is made again. The install grants nothing else: who may read `marts` is a grant of the database, such as its default privileges. A role that does not exist fails the install.
|
|
296
296
|
- Before a pass reads, the history inserts one `sync_attempts` row, with the connection name as `connector` and the source identity as `source`. It inserts one `extraction_coverage` row for each selected stream, with the declared `coverage(stream)` of the source and the schema of the destination as `target_schema`. The outcomes close them.
|
|
297
297
|
- A run declares each stream that a connection selected. A watch pass reads only the streams whose change signals changed, so an attempt vouches only for its own `extraction_coverage` rows.
|
|
@@ -300,13 +300,14 @@ A refresh is a load in `_warehouse._elt_loads`, under the name of its dataset, s
|
|
|
300
300
|
|
|
301
301
|
The relations in `marts`:
|
|
302
302
|
|
|
303
|
-
| Relation | Contents
|
|
304
|
-
| --------------------- |
|
|
305
|
-
| `catalog` | Each view and column in `marts`, with its description. The start point of a reader. A table is not for readers, also when it is in `marts`, such as the raw table of a computed dataset.
|
|
306
|
-
| `sync_status` | For each connection: the latest attempt, the end of the last successful pass, and `current_as_of` (the oldest of its streams).
|
|
307
|
-
| `stream_status` | For each connection and stream: the latest attempt that declared the stream with the status of its copy, and the last attempt in which that copy succeeded. A watch pass reads only changed streams, so judge a stream here, by `current_as_of`.
|
|
308
|
-
| `
|
|
309
|
-
| `
|
|
303
|
+
| Relation | Contents |
|
|
304
|
+
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
305
|
+
| `catalog` | Each view and column in `marts`, with its description. The start point of a reader. A table is not for readers, also when it is in `marts`, such as the raw table of a computed dataset. |
|
|
306
|
+
| `sync_status` | For each connection: the latest attempt, the end of the last successful pass, and `current_as_of` (the oldest of its streams). |
|
|
307
|
+
| `stream_status` | For each connection and stream: the latest attempt that declared the stream with the status of its copy, and the last attempt in which that copy succeeded. A watch pass reads only changed streams, so judge a stream here, by `current_as_of`. |
|
|
308
|
+
| `derived_status` | For each view and computed dataset, named with its schema: whether it waits for its inputs, refreshes, is current or failed, its `as_of`, and for a computed dataset how many inputs it computed, found empty, failed on or has still to compute. See [derived status](#derived-status). |
|
|
309
|
+
| `sync_attempts` | The kept history of attempts, with start and end times, status and errors. |
|
|
310
|
+
| `extraction_coverage` | For each attempt and stream: the configured selection, the explanation of the scope, the target, the outcome of the copy, the committed counts and the failed partitions. |
|
|
310
311
|
|
|
311
312
|
What the relations tell:
|
|
312
313
|
|
|
@@ -319,6 +320,17 @@ What the relations tell:
|
|
|
319
320
|
- A first failure has no time of success.
|
|
320
321
|
- Coverage is the [configured scope](../../elt/README.md#coverage). Judge freshness by `current_as_of` ([the four clocks](../../elt/README.md#the-four-clocks)).
|
|
321
322
|
|
|
323
|
+
## Derived status
|
|
324
|
+
|
|
325
|
+
Each [view](#declare-a-view) and [computed dataset](#declare-a-computed-dataset) keeps one row in `_warehouse.derived_status`, which readers see as `marts.derived_status`. A reader needs it because `sync_status` says that a pass succeeded while a computed dataset still computes, or failed on each of its inputs.
|
|
326
|
+
|
|
327
|
+
- A refresh writes `waiting` when a relation that the dataset reads is not made yet, and `refreshing` when it begins, with `refreshing_since`. A computed dataset also writes how many inputs it read and how many it has still to compute (`pending`). Each commit of the refresh lowers `pending` in its own transaction, so a reader sees it fall while the rows land.
|
|
328
|
+
- A refresh that ends with no failure of the dataset writes `current`, with `as_of`, when it began to read, and `refreshed_at`. A refresh that fails writes `failed`, with `error`. A rule that fails on one input is not a failure of the dataset: `failed` counts its rows.
|
|
329
|
+
- `computed`, `empty` and `failed` count the output rows when the refresh began and when it ended.
|
|
330
|
+
- A dataset is caught up when its `state` is `current`, its `pending` is 0, and its `as_of` is at or after the start of the last pass of the connections that it reads. This is the frontier of Materialize and the `DATA_TIMESTAMP` of a Snowflake dynamic table, with the pending items of a pgai vectorizer.
|
|
331
|
+
- A refresh that a stopped process left stays `refreshing` until the next refresh. The status keeps no row versions, so a reading as of a position fails the view, as it fails the other views of the sync history.
|
|
332
|
+
- A status write takes no commit number: it is not data of a position.
|
|
333
|
+
|
|
322
334
|
## Decisions
|
|
323
335
|
|
|
324
336
|
- [ADR 0001](./docs/adr/0001-checkpoints-are-kept-beside-the-data.md): checkpoints are kept beside the data.
|
|
@@ -354,6 +366,7 @@ src/
|
|
|
354
366
|
postgres-views.ts the publication of views, for View and the sync history
|
|
355
367
|
postgres-dependents.ts the views over a table, made again over its change
|
|
356
368
|
postgres-privileges.ts the privileges of a relation, kept across its change
|
|
369
|
+
postgres-derived-status.ts the status that each derived dataset keeps of itself
|
|
357
370
|
postgres-checkpoint-store.ts PostgresCheckpointStore
|
|
358
371
|
postgres-sync-history.ts PostgresSyncHistory and the catalog
|
|
359
372
|
postgres-sync-history-schema.ts the tables and views of the history
|
package/dist/computed-dataset.js
CHANGED
|
@@ -1,9 +1,10 @@
|
|
|
1
1
|
import { hash } from 'node:crypto';
|
|
2
2
|
import { Lock } from '@zukhruf/async';
|
|
3
|
-
import { DerivedDataset, } from '@eltjs/elt';
|
|
3
|
+
import { DerivedDataset, errorMessage, } from '@eltjs/elt';
|
|
4
4
|
import { identifier, quote } from './identifier.js';
|
|
5
5
|
import { TargetComments } from './postgres-comments.js';
|
|
6
6
|
import { rebindDependents } from './postgres-dependents.js';
|
|
7
|
+
import { PostgresDerivedStatus } from './postgres-derived-status.js';
|
|
7
8
|
import { PostgresDestination } from './postgres-destination.js';
|
|
8
9
|
import { storedColumns } from './postgres-fit.js';
|
|
9
10
|
import { TargetHistory, rowHash } from './postgres-history.js';
|
|
@@ -122,9 +123,25 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
122
123
|
// Open from the start, so it closes a refresh that a process ended; with
|
|
123
124
|
// no commit it is no span, and leaves the positions as they are.
|
|
124
125
|
await using load = await PostgresLoad.open(this.into.url, { dataset: this.name }, { clock, retention: this.into.retention });
|
|
126
|
+
const status = new PostgresDerivedStatus(load.sql, this.qualifiedName, 'computed', clock);
|
|
125
127
|
const inputs = await this.#inputs(load.sql, connections);
|
|
126
|
-
if (inputs === null)
|
|
128
|
+
if (inputs === null) {
|
|
129
|
+
await status.waiting();
|
|
127
130
|
return;
|
|
131
|
+
}
|
|
132
|
+
try {
|
|
133
|
+
await this.#update(load, status, inputs, trigger, signal);
|
|
134
|
+
}
|
|
135
|
+
catch (error) {
|
|
136
|
+
// A stop is not the dataset's failure.
|
|
137
|
+
if (signal?.aborted !== true)
|
|
138
|
+
await status.failed(errorMessage(error));
|
|
139
|
+
throw error;
|
|
140
|
+
}
|
|
141
|
+
}
|
|
142
|
+
// Brings the output rows in step with inputs, and tells status how many
|
|
143
|
+
// are due and how many settled.
|
|
144
|
+
async #update(load, status, inputs, trigger, signal) {
|
|
128
145
|
const key = quote(this.#rule.key.name);
|
|
129
146
|
const table = this.#qualified(this.#table);
|
|
130
147
|
// The number of a transaction's commit, taken at its first change, so a
|
|
@@ -214,6 +231,10 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
214
231
|
trigger.tried.add(attempt);
|
|
215
232
|
due.push({ id, input, inputHash, rule });
|
|
216
233
|
}
|
|
234
|
+
await status.refreshing({
|
|
235
|
+
...(await this.#counts(load.sql, inputs)),
|
|
236
|
+
pending: due.length,
|
|
237
|
+
});
|
|
217
238
|
// A pool of calls, each taking the next batch once its own committed, so
|
|
218
239
|
// a slow call holds only its place, as a worker pool keeps its other
|
|
219
240
|
// workers busy. A call that fails stops the pool taking more.
|
|
@@ -247,11 +268,28 @@ export class ComputedDataset extends DerivedDataset {
|
|
|
247
268
|
else
|
|
248
269
|
await store(sql, commit, id, inputHash, rule, 'empty', null, {});
|
|
249
270
|
}
|
|
271
|
+
await status.settled(sql, batch.length);
|
|
250
272
|
});
|
|
251
273
|
}
|
|
252
274
|
}));
|
|
253
275
|
if (failures.length > 0)
|
|
254
276
|
throw failures[0];
|
|
277
|
+
if (signal?.aborted === true)
|
|
278
|
+
return;
|
|
279
|
+
await status.current({
|
|
280
|
+
...(await this.#counts(load.sql, inputs)),
|
|
281
|
+
pending: 0,
|
|
282
|
+
});
|
|
283
|
+
}
|
|
284
|
+
// How many inputs there are, and how many output rows have each status.
|
|
285
|
+
async #counts(sql, inputs) {
|
|
286
|
+
const counted = new Map((await sql.unsafe(`SELECT "status", count(*)::int AS "n" FROM ${this.#qualified(this.#table)} GROUP BY "status"`)).map(({ status, n }) => [status, n]));
|
|
287
|
+
return {
|
|
288
|
+
inputs: inputs.size,
|
|
289
|
+
computed: counted.get('computed') ?? 0,
|
|
290
|
+
empty: counted.get('empty') ?? 0,
|
|
291
|
+
failed: counted.get('failed') ?? 0,
|
|
292
|
+
};
|
|
255
293
|
}
|
|
256
294
|
// Makes the table, its history and its view, or makes the table again when
|
|
257
295
|
// its declaration changed: the history keeps each row of the old table, in
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
import type postgres from 'postgres';
|
|
2
|
+
import type { PipelineClock } from '@eltjs/elt';
|
|
3
|
+
import { type Queries } from './postgres-session.ts';
|
|
4
|
+
export declare const derivedStatusTable = "CREATE TABLE IF NOT EXISTS _warehouse.derived_status (\n dataset text PRIMARY KEY,\n kind text NOT NULL CHECK (kind IN ('view', 'computed')),\n state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),\n as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,\n inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint\n )";
|
|
5
|
+
export type DerivedCounts = {
|
|
6
|
+
readonly inputs: number;
|
|
7
|
+
readonly computed: number;
|
|
8
|
+
readonly empty: number;
|
|
9
|
+
readonly failed: number;
|
|
10
|
+
readonly pending: number;
|
|
11
|
+
};
|
|
12
|
+
export declare class PostgresDerivedStatus {
|
|
13
|
+
#private;
|
|
14
|
+
constructor(sql: postgres.Sql, dataset: string, kind: 'view' | 'computed', clock: PipelineClock);
|
|
15
|
+
waiting(): Promise<void>;
|
|
16
|
+
refreshing(counts: DerivedCounts): Promise<void>;
|
|
17
|
+
settled(sql: Queries, count: number): Promise<void>;
|
|
18
|
+
current(counts?: DerivedCounts): Promise<void>;
|
|
19
|
+
failed(error: string): Promise<void>;
|
|
20
|
+
}
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
import { lockSchema } from './postgres-session.js';
|
|
2
|
+
// One row for each derived dataset of the database: what its refreshes left,
|
|
3
|
+
// which readers see through marts.derived_status. The sync history makes it
|
|
4
|
+
// with its tables, and a dataset makes it too, so it never depends on the
|
|
5
|
+
// order of the two.
|
|
6
|
+
export const derivedStatusTable = `CREATE TABLE IF NOT EXISTS _warehouse.derived_status (
|
|
7
|
+
dataset text PRIMARY KEY,
|
|
8
|
+
kind text NOT NULL CHECK (kind IN ('view', 'computed')),
|
|
9
|
+
state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),
|
|
10
|
+
as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,
|
|
11
|
+
inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint
|
|
12
|
+
)`;
|
|
13
|
+
// What one refresh of a derived dataset tells readers, as Snowflake keeps the
|
|
14
|
+
// state and DATA_TIMESTAMP of each dynamic table, and pgai the pending items
|
|
15
|
+
// of each vectorizer. A refresh that begins keeps as_of, the data of the
|
|
16
|
+
// last refresh that ended, until it ends too. dataset is the name with its
|
|
17
|
+
// schema.
|
|
18
|
+
export class PostgresDerivedStatus {
|
|
19
|
+
#sql;
|
|
20
|
+
#dataset;
|
|
21
|
+
#kind;
|
|
22
|
+
#clock;
|
|
23
|
+
// When this refresh began to read, which its end gives as as_of.
|
|
24
|
+
#since;
|
|
25
|
+
// sql is a session outside any transaction, between the dataset's own.
|
|
26
|
+
constructor(sql, dataset, kind, clock) {
|
|
27
|
+
this.#sql = sql;
|
|
28
|
+
this.#dataset = dataset;
|
|
29
|
+
this.#kind = kind;
|
|
30
|
+
this.#clock = clock;
|
|
31
|
+
this.#since = this.#now();
|
|
32
|
+
}
|
|
33
|
+
waiting() {
|
|
34
|
+
return this.#write({ state: 'waiting', refreshing_since: null });
|
|
35
|
+
}
|
|
36
|
+
refreshing(counts) {
|
|
37
|
+
return this.#write({
|
|
38
|
+
state: 'refreshing',
|
|
39
|
+
refreshing_since: this.#since,
|
|
40
|
+
...counts,
|
|
41
|
+
});
|
|
42
|
+
}
|
|
43
|
+
// Inside the transaction of a commit that settled count inputs.
|
|
44
|
+
async settled(sql, count) {
|
|
45
|
+
await sql.unsafe('UPDATE _warehouse.derived_status SET pending = GREATEST(pending - $1, 0) WHERE dataset = $2', [count, this.#dataset]);
|
|
46
|
+
}
|
|
47
|
+
current(counts) {
|
|
48
|
+
return this.#write({
|
|
49
|
+
state: 'current',
|
|
50
|
+
as_of: this.#since,
|
|
51
|
+
refreshed_at: this.#now(),
|
|
52
|
+
refreshing_since: null,
|
|
53
|
+
error: null,
|
|
54
|
+
...counts,
|
|
55
|
+
});
|
|
56
|
+
}
|
|
57
|
+
failed(error) {
|
|
58
|
+
return this.#write({ state: 'failed', refreshing_since: null, error });
|
|
59
|
+
}
|
|
60
|
+
// In a transaction of its own, which takes no commit number: the status is
|
|
61
|
+
// no data of a position.
|
|
62
|
+
async #write(values) {
|
|
63
|
+
const columns = ['dataset', 'kind', ...Object.keys(values)];
|
|
64
|
+
const placeholders = columns.map((_, index) => `$${index + 1}`);
|
|
65
|
+
const assignments = columns
|
|
66
|
+
.slice(1)
|
|
67
|
+
.map((column) => `${column} = excluded.${column}`);
|
|
68
|
+
await this.#sql.begin(async (sql) => {
|
|
69
|
+
await lockSchema(sql, '_warehouse');
|
|
70
|
+
await sql.unsafe(derivedStatusTable);
|
|
71
|
+
await sql.unsafe(`INSERT INTO _warehouse.derived_status (${columns.join(', ')}) VALUES (${placeholders.join(', ')}) ON CONFLICT (dataset) DO UPDATE SET ${assignments.join(', ')}`, [this.#dataset, this.#kind, ...Object.values(values)]);
|
|
72
|
+
});
|
|
73
|
+
}
|
|
74
|
+
#now() {
|
|
75
|
+
return new Date(this.#clock.Date.now()).toISOString();
|
|
76
|
+
}
|
|
77
|
+
}
|
|
@@ -28,4 +28,4 @@ export declare const syncHistoryTables: readonly ['CREATE SCHEMA IF NOT EXISTS _
|
|
|
28
28
|
)`, `CREATE TABLE IF NOT EXISTS _warehouse.stream_checks (
|
|
29
29
|
connector text NOT NULL, stream text NOT NULL, checked_at timestamptz NOT NULL,
|
|
30
30
|
PRIMARY KEY (connector, stream)
|
|
31
|
-
)
|
|
31
|
+
)`, "CREATE TABLE IF NOT EXISTS _warehouse.derived_status (\n dataset text PRIMARY KEY,\n kind text NOT NULL CHECK (kind IN ('view', 'computed')),\n state text NOT NULL CHECK (state IN ('waiting', 'refreshing', 'current', 'failed')),\n as_of timestamptz, refreshed_at timestamptz, refreshing_since timestamptz, error text,\n inputs bigint, computed bigint, empty bigint, failed bigint, pending bigint\n )"];
|
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { syncHistoryRelations } from '@eltjs/elt';
|
|
2
|
+
import { derivedStatusTable } from './postgres-derived-status.js';
|
|
2
3
|
// When connector's stream last matched its upstream: the start of its last
|
|
3
4
|
// successful pass, or a later check that found it unchanged, each no later
|
|
4
5
|
// than when the source said the upstream itself was last current. NULL if it
|
|
@@ -52,6 +53,7 @@ const queries = {
|
|
|
52
53
|
ORDER BY s.completed_at DESC, s.id DESC LIMIT 1
|
|
53
54
|
) success ON true
|
|
54
55
|
ORDER BY a.connector, c.stream, a.id DESC`,
|
|
56
|
+
derived_status: `SELECT dataset, kind, state, as_of, refreshed_at, refreshing_since, error, inputs, computed, empty, failed, pending FROM _warehouse.derived_status ORDER BY dataset`,
|
|
55
57
|
};
|
|
56
58
|
// How readers see the sync history: what each pass declared and loaded.
|
|
57
59
|
// The sync history keeps no row versions, so a reading as of a position
|
|
@@ -97,4 +99,5 @@ export const syncHistoryTables = [
|
|
|
97
99
|
connector text NOT NULL, stream text NOT NULL, checked_at timestamptz NOT NULL,
|
|
98
100
|
PRIMARY KEY (connector, stream)
|
|
99
101
|
)`,
|
|
102
|
+
derivedStatusTable,
|
|
100
103
|
];
|
package/dist/view.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { type Connection, DerivedDataset } from '@eltjs/elt';
|
|
1
|
+
import { type Connection, DerivedDataset, type PipelineClock } from '@eltjs/elt';
|
|
2
2
|
import { PostgresDestination } from './postgres-destination.ts';
|
|
3
3
|
import type { PostgresTable } from './postgres-table.ts';
|
|
4
4
|
export declare class View extends DerivedDataset<PostgresTable> {
|
|
@@ -14,9 +14,10 @@ export declare class View extends DerivedDataset<PostgresTable> {
|
|
|
14
14
|
});
|
|
15
15
|
get qualifiedName(): string;
|
|
16
16
|
validate(connections: readonly Connection<PostgresTable>[]): void;
|
|
17
|
-
refresh({ connections, earlier, }: {
|
|
17
|
+
refresh({ connections, earlier, clock, }: {
|
|
18
18
|
readonly connections: readonly Connection<PostgresTable>[];
|
|
19
19
|
readonly earlier: readonly DerivedDataset<PostgresTable>[];
|
|
20
|
+
readonly clock: PipelineClock;
|
|
20
21
|
}): Promise<void>;
|
|
21
22
|
}
|
|
22
23
|
export declare function made(connections: readonly Connection<PostgresTable>[], into: PostgresDestination): Set<string>;
|
package/dist/view.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import postgres from 'postgres';
|
|
2
|
-
import { DerivedDataset } from '@eltjs/elt';
|
|
2
|
+
import { DerivedDataset, errorMessage, } from '@eltjs/elt';
|
|
3
|
+
import { PostgresDerivedStatus } from './postgres-derived-status.js';
|
|
3
4
|
import { PostgresDestination } from './postgres-destination.js';
|
|
4
5
|
import { PostgresSession } from './postgres-session.js';
|
|
5
6
|
import { checkView, publishPostgresViews, } from './postgres-views.js';
|
|
@@ -33,8 +34,9 @@ export class View extends DerivedDataset {
|
|
|
33
34
|
if (made(connections, this.into).has(this.qualifiedName))
|
|
34
35
|
throw new TypeError(`View ${this.qualifiedName} has the name of a table or reader view a copy makes`);
|
|
35
36
|
}
|
|
36
|
-
async refresh({ connections, earlier, }) {
|
|
37
|
+
async refresh({ connections, earlier, clock, }) {
|
|
37
38
|
await using session = new PostgresSession(this.into.url, 'elt-derived');
|
|
39
|
+
const status = new PostgresDerivedStatus(session.sql, this.qualifiedName, 'view', clock);
|
|
38
40
|
try {
|
|
39
41
|
await session.sql.begin((transaction) => publishPostgresViews(transaction, {
|
|
40
42
|
schema: this.into.schema,
|
|
@@ -45,10 +47,14 @@ export class View extends DerivedDataset {
|
|
|
45
47
|
const missing = missingRelation(error, this.into.schema);
|
|
46
48
|
if (missing !== undefined &&
|
|
47
49
|
(made(connections, this.into).has(missing) ||
|
|
48
|
-
earlier.some((dataset) => derivedRelation(dataset) === missing)))
|
|
50
|
+
earlier.some((dataset) => derivedRelation(dataset) === missing))) {
|
|
51
|
+
await status.waiting();
|
|
49
52
|
return;
|
|
53
|
+
}
|
|
54
|
+
await status.failed(errorMessage(error));
|
|
50
55
|
throw error;
|
|
51
56
|
}
|
|
57
|
+
await status.current();
|
|
52
58
|
}
|
|
53
59
|
}
|
|
54
60
|
// The schema-qualified relation that Postgres found missing, with a bare name
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@eltjs/elt-postgresql",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.16",
|
|
4
4
|
"description": "The Postgres destination of elt: each stream is a typed table in one schema, with a reader view in another, a history of each row for readings as of a position, and the views and computed datasets that a pipeline derives.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"repository": {
|
|
@@ -37,7 +37,7 @@
|
|
|
37
37
|
"access": "public"
|
|
38
38
|
},
|
|
39
39
|
"dependencies": {
|
|
40
|
-
"@eltjs/elt": "^0.3.
|
|
40
|
+
"@eltjs/elt": "^0.3.16",
|
|
41
41
|
"@zukhruf/async": "^0.8.27",
|
|
42
42
|
"postgres": "^3.4.9"
|
|
43
43
|
},
|