@firebase-function-kits/bigquery-firestore-export 0.0.2-rc.0 → 0.0.2-rc.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +1 -1
- package/README.md +120 -0
- package/lib/config.d.ts.map +1 -1
- package/lib/config.js +113 -7
- package/lib/config.js.map +1 -1
- package/lib/dts.d.ts +9 -2
- package/lib/dts.d.ts.map +1 -1
- package/lib/dts.js +10 -6
- package/lib/dts.js.map +1 -1
- package/lib/export-config.d.ts +0 -3
- package/lib/export-config.d.ts.map +1 -1
- package/lib/export-config.js +0 -1
- package/lib/export-config.js.map +1 -1
- package/lib/index.js +1 -1
- package/lib/index.js.map +1 -1
- package/npm-shrinkwrap.json +5049 -0
- package/package.json +1 -1
- package/src/config.ts +124 -7
- package/src/dts.ts +10 -7
- package/src/export-config.ts +0 -4
- package/src/index.ts +1 -1
- package/tests/dts.test.ts +1 -5
- package/tests/export-config.test.ts +0 -4
package/CHANGELOG.md
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
- Initial release
|
|
1
|
+
- Initial release of kit, see README for differences between the legacy extension and this kit
|
package/README.md
CHANGED
|
@@ -173,6 +173,126 @@ The run document stores DTS metadata and row counts. Its `output` subcollection
|
|
|
173
173
|
contains converted query rows. The `latest` document is updated transactionally
|
|
174
174
|
so an older completion message cannot replace a newer run.
|
|
175
175
|
|
|
176
|
+
## Differences from the Export BigQuery to Firestore extension
|
|
177
|
+
|
|
178
|
+
This kit is version 0.2.2 of the extension repackaged as an npm package. It is a
|
|
179
|
+
close port: the same two functions, the same BigQuery Data Transfer scheduled
|
|
180
|
+
query, the same `transferConfigs/{configId}/runs/{runId}` and `runs/latest`
|
|
181
|
+
documents, the same `WRITE_TRUNCATE` destination table naming and the same
|
|
182
|
+
row-by-row copy into Firestore. Every setting keeps its extension environment
|
|
183
|
+
variable name and default, so a `.env` copied from your installed instance needs
|
|
184
|
+
no value changes. What changes is the instance id, the Pub/Sub topic, the identity
|
|
185
|
+
the scheduled query runs as, and how repeated BigQuery columns land in Firestore.
|
|
186
|
+
|
|
187
|
+
### You set `INSTANCE_ID` yourself, and the Pub/Sub topic is renamed
|
|
188
|
+
|
|
189
|
+
The extension derived an instance id at install and used it to name its
|
|
190
|
+
notification topic (`ext-<instance id>-processMessages`) and to tag its transfer
|
|
191
|
+
config document with `extInstanceId`. Here `INSTANCE_ID` is a setting you
|
|
192
|
+
provide, and it must match this instance's key in the `instances` map in
|
|
193
|
+
`firebase.json`.
|
|
194
|
+
|
|
195
|
+
The topic becomes `kit-<INSTANCE_ID>-processMessages`, and the kit creates it on
|
|
196
|
+
first run if it does not already exist. Set `INSTANCE_ID` to your installed
|
|
197
|
+
instance's id if you want the kit to adopt the scheduled query that instance
|
|
198
|
+
created, because the lookup is by `extInstanceId` on the documents in
|
|
199
|
+
`COLLECTION_PATH`. With a different id the kit finds nothing, creates a second
|
|
200
|
+
scheduled query, and you end up with two writing into the same collection.
|
|
201
|
+
|
|
202
|
+
The existing transfer config still points its notifications at the old `ext-`
|
|
203
|
+
topic; the kit's update path rewrites `notification_pubsub_topic` to the new one
|
|
204
|
+
on the first deploy, so the old topic can be deleted afterwards.
|
|
205
|
+
|
|
206
|
+
### Repeated BigQuery columns are now written as arrays
|
|
207
|
+
|
|
208
|
+
A repeated (`ARRAY`) column used to arrive in Firestore as a map keyed by
|
|
209
|
+
position, `{ "0": ..., "1": ... }`, because the conversion treated every
|
|
210
|
+
non-scalar value as an object. The kit writes a real Firestore array instead.
|
|
211
|
+
Anything reading those fields by numeric string key needs updating, and rows
|
|
212
|
+
written before and after the change are not the same shape. Scalars, timestamps,
|
|
213
|
+
dates, times, datetimes, bytes and geography values convert exactly as before.
|
|
214
|
+
|
|
215
|
+
### The scheduled query runs as a different service account
|
|
216
|
+
|
|
217
|
+
The extension created the transfer config with
|
|
218
|
+
`serviceAccountName: ext-<instance id>@<project>.iam.gserviceaccount.com`, so the
|
|
219
|
+
query ran as the extension's own service account. The kit creates it without a
|
|
220
|
+
service account name, so BigQuery Data Transfer runs it as the identity that
|
|
221
|
+
created it, which is your function's runtime service account (the default compute
|
|
222
|
+
service account unless you have set one).
|
|
223
|
+
|
|
224
|
+
That account needs to be able to read whatever `QUERY_STRING` touches and write
|
|
225
|
+
to `DATASET_ID`. `roles/bigquery.admin` on the function covers this for datasets
|
|
226
|
+
in the same project; a cross-project query needs the grant made explicitly. This
|
|
227
|
+
was not exercised against a live deploy.
|
|
228
|
+
|
|
229
|
+
Note also that a service account cannot be changed on an existing transfer config,
|
|
230
|
+
so a scheduled query originally created by an installed extension instance keeps
|
|
231
|
+
running as the extension's service account even after the kit adopts it. Delete
|
|
232
|
+
and recreate the scheduled query if you want it moved.
|
|
233
|
+
|
|
234
|
+
### The setup step runs on every deploy
|
|
235
|
+
|
|
236
|
+
The install, update and configure hooks are replaced by an `upsertTransferConfig`
|
|
237
|
+
task that the CLI runs after your first deploy and after every redeploy. It does
|
|
238
|
+
the same work: create the scheduled query if this instance has none, otherwise
|
|
239
|
+
reconcile the existing one against your current `QUERY_STRING`, `DATASET_ID`,
|
|
240
|
+
`TABLE_NAME`, `PARTITIONING_FIELD` and `SCHEDULE`.
|
|
241
|
+
|
|
242
|
+
Two consequences of it now being an ordinary task rather than a lifecycle event.
|
|
243
|
+
There is no install UI to report progress into, so failures show up in the task's
|
|
244
|
+
function logs, and the task retries up to five times with a 30 second minimum
|
|
245
|
+
backoff. And `DISPLAY_NAME` is no longer immutable, but changing it does not
|
|
246
|
+
rename an existing scheduled query, because display name is not part of the
|
|
247
|
+
update; it only applies to a config the kit creates.
|
|
248
|
+
|
|
249
|
+
Removing `PARTITIONING_FIELD` once it has been set still fails, with the same
|
|
250
|
+
explanation, because the BigQuery Data Transfer API cannot clear it.
|
|
251
|
+
|
|
252
|
+
### You can link an existing scheduled query
|
|
253
|
+
|
|
254
|
+
`TRANSFER_CONFIG_NAME` is a new setting. Point it at the full resource name of a
|
|
255
|
+
scheduled query you already have
|
|
256
|
+
(`projects/<project>/locations/<location>/transferConfigs/<id>`) and the kit
|
|
257
|
+
records that config in Firestore and consumes its notifications instead of
|
|
258
|
+
creating one of its own. The extension carried the code for this but no setting to
|
|
259
|
+
reach it. Leave it empty for the create-or-reconcile behaviour described above.
|
|
260
|
+
|
|
261
|
+
### Region, and no location setting
|
|
262
|
+
|
|
263
|
+
`LOCATION` is gone. Both functions deploy to your codebase's default region
|
|
264
|
+
(`us-central1` unless you have changed it) rather than the immutable location you
|
|
265
|
+
picked at install. `BIGQUERY_DATASET_LOCATION` is unchanged and still tells the
|
|
266
|
+
result query where your dataset lives.
|
|
267
|
+
|
|
268
|
+
### Failed notifications are retried
|
|
269
|
+
|
|
270
|
+
`processMessages` is a 2nd gen Pub/Sub function with retries enabled, where the
|
|
271
|
+
extension's 1st gen trigger did not retry. A run whose results fail to copy, for
|
|
272
|
+
example because BigQuery or Firestore is briefly unavailable, is now retried
|
|
273
|
+
rather than dropped. A notification that keeps failing, such as one for a transfer
|
|
274
|
+
config not tagged with this `INSTANCE_ID`, is also retried until Pub/Sub gives up.
|
|
275
|
+
|
|
276
|
+
Both functions' service accounts need `roles/eventarc.eventReceiver` and
|
|
277
|
+
`roles/run.invoker` on top of the three roles the extension asked for, and the
|
|
278
|
+
Pub/Sub API is now requested explicitly. The Firebase CLI handles all of this.
|
|
279
|
+
|
|
280
|
+
### Unchanged
|
|
281
|
+
|
|
282
|
+
- `COLLECTION_PATH` still defaults to `transferConfigs`, and the document layout
|
|
283
|
+
under it is identical: the transfer config document keyed by config id, a `runs`
|
|
284
|
+
subcollection keyed by run id holding `runMetadata`, `totalRowCount` and
|
|
285
|
+
`failedRowCount`, a `latest` document, and an `output` collection of rows per
|
|
286
|
+
run.
|
|
287
|
+
- The destination table is still `TABLE_NAME_{run_time|"%H%M%S"}` with
|
|
288
|
+
`WRITE_TRUNCATE`, and results are still read with `SELECT *` in
|
|
289
|
+
`BIGQUERY_DATASET_LOCATION`.
|
|
290
|
+
- Runs that do not succeed still write a run document with zeroed counts and still
|
|
291
|
+
update `latest`, and `latest` is still only moved forward by a newer run.
|
|
292
|
+
- Rows are still written one document at a time in chunks of 10,000, with
|
|
293
|
+
per-row failures logged and counted rather than aborting the run.
|
|
294
|
+
- `LOG_LEVEL` still accepts `debug`, `info`, `warn`, `error` and `silent`.
|
|
295
|
+
|
|
176
296
|
## API surface
|
|
177
297
|
|
|
178
298
|
- **Main entry** (`@firebase/bigquery-firestore-export`): exports
|
package/lib/config.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAsBA,OAAO,KAAK,EACV,6BAA6B,EAC7B,iBAAiB,EAElB,MAAM,iBAAiB,CAAC;
|
|
1
|
+
{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAsBA,OAAO,KAAK,EACV,6BAA6B,EAC7B,iBAAiB,EAElB,MAAM,iBAAiB,CAAC;AA+IzB,eAAO,MAAM,kBAAkB,EAAE,iBAEhC,CAAC;AAcF,iEAAiE;AACjE,wBAAgB,aAAa,IAAI,6BAA6B,CAkB7D"}
|
package/lib/config.js
CHANGED
|
@@ -23,21 +23,127 @@ const instanceId = (0, params_1.defineString)("INSTANCE_ID");
|
|
|
23
23
|
const params = {
|
|
24
24
|
instanceId,
|
|
25
25
|
bigqueryDatasetLocation: (0, params_1.defineString)("BIGQUERY_DATASET_LOCATION", {
|
|
26
|
+
label: "BigQuery Dataset Location",
|
|
27
|
+
description: "What is the location of the BigQuery dataset referenced in the query?",
|
|
26
28
|
default: "US",
|
|
29
|
+
input: (0, params_1.select)({
|
|
30
|
+
"Columbus, Ohio (us-east5)": "us-east5",
|
|
31
|
+
"Iowa (us-central1)": "us-central1",
|
|
32
|
+
"Las Vegas (us-west4)": "us-west4",
|
|
33
|
+
"Los Angeles (us-west2)": "us-west2",
|
|
34
|
+
"Montréal (northamerica-northeast1)": "northamerica-northeast1",
|
|
35
|
+
"Northern Virginia (us-east4)": "us-east4",
|
|
36
|
+
"Oregon (us-west1)": "us-west1",
|
|
37
|
+
"Salt Lake City (us-west3)": "us-west3",
|
|
38
|
+
"São Paulo (southamerica-east1)": "southamerica-east1",
|
|
39
|
+
"Santiago (southamerica-west1)": "southamerica-west1",
|
|
40
|
+
"South Carolina (us-east1)": "us-east1",
|
|
41
|
+
"Toronto (northamerica-northeast2)": "northamerica-northeast2",
|
|
42
|
+
"Delhi (asia-south2)": "asia-south2",
|
|
43
|
+
"Hong Kong (asia-east2)": "asia-east2",
|
|
44
|
+
"Jakarta (asia-southeast2)": "asia-southeast2",
|
|
45
|
+
"Melbourne (australia-southeast2)": "australia-southeast2",
|
|
46
|
+
"Mumbai (asia-south1)": "asia-south1",
|
|
47
|
+
"Osaka (asia-northeast2)": "asia-northeast2",
|
|
48
|
+
"Seoul (asia-northeast3)": "asia-northeast3",
|
|
49
|
+
"Singapore (asia-southeast1)": "asia-southeast1",
|
|
50
|
+
"Sydney (australia-southeast1)": "australia-southeast1",
|
|
51
|
+
"Taiwan (asia-east1)": "asia-east1",
|
|
52
|
+
"Tokyo (asia-northeast1)": "asia-northeast1",
|
|
53
|
+
"Belgium (europe-west1)": "europe-west1",
|
|
54
|
+
"Finland (europe-north1)": "europe-north1",
|
|
55
|
+
"Frankfurt (europe-west3)": "europe-west3",
|
|
56
|
+
"London (europe-west2)": "europe-west2",
|
|
57
|
+
"Madrid (europe-southwest1)": "europe-southwest1",
|
|
58
|
+
"Milan (europe-west8)": "europe-west8",
|
|
59
|
+
"Netherlands (europe-west4)": "europe-west4",
|
|
60
|
+
"Paris (europe-west9)": "europe-west9",
|
|
61
|
+
"Warsaw (europe-central2)": "europe-central2",
|
|
62
|
+
"Zürich (europe-west6)": "europe-west6",
|
|
63
|
+
"US Multi-Region (US)": "US",
|
|
64
|
+
"EU Mutli-Region (EU)": "EU",
|
|
65
|
+
}),
|
|
27
66
|
}),
|
|
28
67
|
transferConfigName: (0, params_1.defineString)("TRANSFER_CONFIG_NAME", { default: "" }),
|
|
29
|
-
datasetId: (0, params_1.defineString)("DATASET_ID"
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
68
|
+
datasetId: (0, params_1.defineString)("DATASET_ID", {
|
|
69
|
+
label: "Dataset ID",
|
|
70
|
+
description: "What's the BigQuery destination dataset you'd like to use? Each transfer run will write to a table in this destination dataset.",
|
|
71
|
+
input: {
|
|
72
|
+
text: {
|
|
73
|
+
nonEmpty: true,
|
|
74
|
+
example: "customer_data",
|
|
75
|
+
},
|
|
76
|
+
},
|
|
77
|
+
}),
|
|
78
|
+
tableName: (0, params_1.defineString)("TABLE_NAME", {
|
|
79
|
+
label: "Destination Table Name",
|
|
80
|
+
description: "What's the destination table name prefix you'd like to use? Each transfer run will write to the table with this name, postfixed with the runtime.",
|
|
81
|
+
input: {
|
|
82
|
+
text: {
|
|
83
|
+
nonEmpty: true,
|
|
84
|
+
example: "transactions",
|
|
85
|
+
},
|
|
86
|
+
},
|
|
87
|
+
}),
|
|
88
|
+
queryString: (0, params_1.defineString)("QUERY_STRING", {
|
|
89
|
+
label: "Query String",
|
|
90
|
+
description: "What's the BQ query you'd like to execute?",
|
|
91
|
+
input: {
|
|
92
|
+
text: {
|
|
93
|
+
nonEmpty: true,
|
|
94
|
+
example: "SELECT * from <PROJECT_ID>.customer_data.transactions",
|
|
95
|
+
},
|
|
96
|
+
},
|
|
97
|
+
}),
|
|
98
|
+
displayName: (0, params_1.defineString)("DISPLAY_NAME", {
|
|
99
|
+
label: "Display Name",
|
|
100
|
+
description: "What display name would you like to use?",
|
|
101
|
+
input: {
|
|
102
|
+
text: {
|
|
103
|
+
nonEmpty: true,
|
|
104
|
+
example: "Daily Rollup - Customer Transactions",
|
|
105
|
+
},
|
|
106
|
+
},
|
|
107
|
+
}),
|
|
108
|
+
partitioningField: (0, params_1.defineString)("PARTITIONING_FIELD", {
|
|
109
|
+
label: "Partitioning Field",
|
|
110
|
+
description: "What's the partitioning field on the destination table ID? Leave empty if not using a partitioning field.",
|
|
111
|
+
default: "",
|
|
112
|
+
input: { text: { example: "timestamp" } },
|
|
113
|
+
}),
|
|
114
|
+
schedule: (0, params_1.defineString)("SCHEDULE", {
|
|
115
|
+
label: "Schedule",
|
|
116
|
+
description: "What's the execution schedule you'd like to use for this query?",
|
|
117
|
+
input: {
|
|
118
|
+
text: {
|
|
119
|
+
nonEmpty: true,
|
|
120
|
+
example: "every 15 minutes",
|
|
121
|
+
},
|
|
122
|
+
},
|
|
123
|
+
}),
|
|
35
124
|
firestoreCollection: (0, params_1.defineString)("COLLECTION_PATH", {
|
|
125
|
+
label: "Firestore Collection",
|
|
126
|
+
description: "What's the top-level Firestore Collection to store transfer configs, run metadata, and query output?",
|
|
36
127
|
default: "transferConfigs",
|
|
128
|
+
input: {
|
|
129
|
+
text: {
|
|
130
|
+
example: "transferConfigs",
|
|
131
|
+
validationRegex: /^[^\/]+(\/[^\/]+\/[^\/]+)*$/,
|
|
132
|
+
validationErrorMessage: "Must be a valid Cloud Firestore Collection",
|
|
133
|
+
},
|
|
134
|
+
},
|
|
37
135
|
}),
|
|
38
136
|
logLevel: (0, params_1.defineString)("LOG_LEVEL", {
|
|
137
|
+
label: "Log Level",
|
|
138
|
+
description: "What's the log level you'd like to use for this extension?",
|
|
39
139
|
default: "info",
|
|
40
|
-
input: (0, params_1.select)(
|
|
140
|
+
input: (0, params_1.select)({
|
|
141
|
+
DEBUG: "debug",
|
|
142
|
+
INFO: "info",
|
|
143
|
+
WARN: "warn",
|
|
144
|
+
ERROR: "error",
|
|
145
|
+
SILENT: "silent",
|
|
146
|
+
}),
|
|
41
147
|
}),
|
|
42
148
|
};
|
|
43
149
|
exports.CONFIG_EXPRESSIONS = {
|
package/lib/config.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;GAcG;;;;AAEH,sDAKmC;AAOnC,MAAM,iBAAiB,GAAG,CAAC,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,QAAQ,CAAU,CAAC;AAChF,MAAM,UAAU,GAAG,IAAA,qBAAY,EAAC,aAAa,CAAC,CAAC;AAE/C,MAAM,MAAM,GAAG;IACb,UAAU;IACV,uBAAuB,EAAE,IAAA,qBAAY,EAAC,2BAA2B,EAAE;QACjE,OAAO,EAAE,IAAI;
|
|
1
|
+
{"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;GAcG;;;;AAEH,sDAKmC;AAOnC,MAAM,iBAAiB,GAAG,CAAC,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,QAAQ,CAAU,CAAC;AAChF,MAAM,UAAU,GAAG,IAAA,qBAAY,EAAC,aAAa,CAAC,CAAC;AAE/C,MAAM,MAAM,GAAG;IACb,UAAU;IACV,uBAAuB,EAAE,IAAA,qBAAY,EAAC,2BAA2B,EAAE;QACjE,KAAK,EAAE,2BAA2B;QAClC,WAAW,EACT,uEAAuE;QAEzE,OAAO,EAAE,IAAI;QACb,KAAK,EAAE,IAAA,eAAM,EAAC;YACZ,2BAA2B,EAAE,UAAU;YACvC,oBAAoB,EAAE,aAAa;YACnC,sBAAsB,EAAE,UAAU;YAClC,wBAAwB,EAAE,UAAU;YACpC,oCAAoC,EAAE,yBAAyB;YAC/D,8BAA8B,EAAE,UAAU;YAC1C,mBAAmB,EAAE,UAAU;YAC/B,2BAA2B,EAAE,UAAU;YACvC,gCAAgC,EAAE,oBAAoB;YACtD,+BAA+B,EAAE,oBAAoB;YACrD,2BAA2B,EAAE,UAAU;YACvC,mCAAmC,EAAE,yBAAyB;YAC9D,qBAAqB,EAAE,aAAa;YACpC,wBAAwB,EAAE,YAAY;YACtC,2BAA2B,EAAE,iBAAiB;YAC9C,kCAAkC,EAAE,sBAAsB;YAC1D,sBAAsB,EAAE,aAAa;YACrC,yBAAyB,EAAE,iBAAiB;YAC5C,yBAAyB,EAAE,iBAAiB;YAC5C,6BAA6B,EAAE,iBAAiB;YAChD,+BAA+B,EAAE,sBAAsB;YACvD,qBAAqB,EAAE,YAAY;YACnC,yBAAyB,EAAE,iBAAiB;YAC5C,wBAAwB,EAAE,cAAc;YACxC,yBAAyB,EAAE,eAAe;YAC1C,0BAA0B,EAAE,cAAc;YAC1C,uBAAuB,EAAE,cAAc;YACvC,4BAA4B,EAAE,mBAAmB;YACjD,sBAAsB,EAAE,cAAc;YACtC,4BAA4B,EAAE,cAAc;YAC5C,sBAAsB,EAAE,cAAc;YACtC,0BAA0B,EAAE,iBAAiB;YAC7C,uBAAuB,EAAE,cAAc;YACvC,sBAAsB,EAAE,IAAI;YAC5B,sBAAsB,EAAE,IAAI;SAC7B,CAAC;KACH,CAAC;IACF,kBAAkB,EAAE,IAAA,qBAAY,EAAC,sBAAsB,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;IACzE,SAAS,EAAE,IAAA,qBAAY,EAAC,YAAY,EAAE;QACpC,KAAK,EAAE,YAAY;QACnB,WAAW,EACT,iIAAiI;QACnI,KAAK,EAAE;YACL,IAAI,EAAE;gBACJ,QAAQ,EAAE,IAAI;gBACd,OAAO,EAAE,eAAe;aACzB;SACF;KACF,CAAC;IACF,SAAS,EAAE,IAAA,qBAAY,EAAC,YAAY,EAAE;QACpC,KAAK,EAAE,wBAAwB;QAC/B,WAAW,EACT,mJAAmJ;QACrJ,KAAK,EAAE;YACL,IAAI,EAAE;gBACJ,QAAQ,EAAE,IAAI;gBACd,OAAO,EAAE,cAAc;aACxB;SACF;KACF,CAAC;IACF,WAAW,EAAE,IAAA,qBAAY,EAAC,cAAc,EAAE;QACxC,KAAK,EAAE,cAAc;QACrB,WAAW,EAAE,4CAA4C;QACzD,KAAK,EAAE;YACL,IAAI,EAAE;gBACJ,QAAQ,EAAE,IAAI;gBAEd,OAAO,EAAE,uDAAuD;aACjE;SACF;KACF,CAAC;IACF,WAAW,EAAE,IAAA,qBAAY,EAAC,cAAc,EAAE;QACxC,KAAK,EAAE,cAAc;QACrB,WAAW,EAAE,0CAA0C;QACvD,KAAK,EAAE;YACL,IAAI,EAAE;gBACJ,QAAQ,EAAE,IAAI;gBACd,OAAO,EAAE,sCAAsC;aAChD;SACF;KACF,CAAC;IACF,iBAAiB,EAAE,IAAA,qBAAY,EAAC,oBAAoB,EAAE;QACpD,KAAK,EAAE,oBAAoB;QAC3B,WAAW,EACT,2GAA2G;QAC7G,OAAO,EAAE,EAAE;QACX,KAAK,EAAE,EAAE,IAAI,EAAE,EAAE,OAAO,EAAE,WAAW,EAAE,EAAE;KAC1C,CAAC;IACF,QAAQ,EAAE,IAAA,qBAAY,EAAC,UAAU,EAAE;QACjC,KAAK,EAAE,UAAU;QACjB,WAAW,EACT,iEAAiE;QACnE,KAAK,EAAE;YACL,IAAI,EAAE;gBACJ,QAAQ,EAAE,IAAI;gBACd,OAAO,EAAE,kBAAkB;aAC5B;SACF;KACF,CAAC;IACF,mBAAmB,EAAE,IAAA,qBAAY,EAAC,iBAAiB,EAAE;QACnD,KAAK,EAAE,sBAAsB;QAC7B,WAAW,EACT,sGAAsG;QAExG,OAAO,EAAE,iBAAiB;QAC1B,KAAK,EAAE;YACL,IAAI,EAAE;gBACJ,OAAO,EAAE,iBAAiB;gBAE1B,eAAe,EAAE,6BAA6B;gBAC9C,sBAAsB,EAAE,4CAA4C;aACrE;SACF;KACF,CAAC;IACF,QAAQ,EAAE,IAAA,qBAAY,EAAC,WAAW,EAAE;QAClC,KAAK,EAAE,WAAW;QAClB,WAAW,EAAE,4DAA4D;QAEzE,OAAO,EAAE,MAAM;QACf,KAAK,EAAE,IAAA,eAAM,EAAC;YACZ,KAAK,EAAE,OAAO;YACd,IAAI,EAAE,MAAM;YACZ,IAAI,EAAE,MAAM;YACZ,KAAK,EAAE,OAAO;YACd,MAAM,EAAE,QAAQ;SACjB,CAAC;KACH,CAAC;CACH,CAAC;AAEW,QAAA,kBAAkB,GAAsB;IACnD,WAAW,EAAE,IAAA,aAAI,EAAA,OAAO,UAAU,kBAAkB;CACrD,CAAC;AAEF,SAAS,QAAQ,CAAC,KAAa;IAC7B,MAAM,UAAU,GAAG,KAAK,CAAC,IAAI,EAAE,CAAC;IAChC,OAAO,UAAU,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,SAAS,CAAC;AACxD,CAAC;AAED,SAAS,iBAAiB,CAAC,KAAa;IACtC,MAAM,UAAU,GAAG,KAAK,CAAC,WAAW,EAAE,CAAC;IACvC,OAAO,iBAAiB,CAAC,QAAQ,CAAC,UAAsB,CAAC;QACvD,CAAC,CAAE,UAAuB;QAC1B,CAAC,CAAC,MAAM,CAAC;AACb,CAAC;AAED,iEAAiE;AACjE;IACE,MAAM,kBAAkB,GAAG,MAAM,CAAC,UAAU,CAAC,KAAK,EAAE,CAAC;IAErD,OAAO;QACL,uBAAuB,EAAE,MAAM,CAAC,uBAAuB,CAAC,KAAK,EAAE;QAC/D,SAAS,EAAE,kBAAS,CAAC,KAAK,EAAE;QAC5B,UAAU,EAAE,kBAAkB;QAC9B,kBAAkB,EAAE,QAAQ,CAAC,MAAM,CAAC,kBAAkB,CAAC,KAAK,EAAE,CAAC;QAC/D,SAAS,EAAE,MAAM,CAAC,SAAS,CAAC,KAAK,EAAE;QACnC,SAAS,EAAE,MAAM,CAAC,SAAS,CAAC,KAAK,EAAE;QACnC,WAAW,EAAE,MAAM,CAAC,WAAW,CAAC,KAAK,EAAE;QACvC,WAAW,EAAE,MAAM,CAAC,WAAW,CAAC,KAAK,EAAE;QACvC,iBAAiB,EAAE,QAAQ,CAAC,MAAM,CAAC,iBAAiB,CAAC,KAAK,EAAE,CAAC;QAC7D,QAAQ,EAAE,MAAM,CAAC,QAAQ,CAAC,KAAK,EAAE;QACjC,WAAW,EAAE,OAAO,kBAAkB,kBAAkB;QACxD,mBAAmB,EAAE,MAAM,CAAC,mBAAmB,CAAC,KAAK,EAAE;QACvD,QAAQ,EAAE,iBAAiB,CAAC,MAAM,CAAC,QAAQ,CAAC,KAAK,EAAE,CAAC;KACrD,CAAC;AACJ,CAAC"}
|
package/lib/dts.d.ts
CHANGED
|
@@ -4,8 +4,15 @@ export type DataTransferClient = bigqueryDataTransfer.v1.DataTransferServiceClie
|
|
|
4
4
|
export type TransferConfig = bigqueryDataTransfer.protos.google.cloud.bigquery.datatransfer.v1.ITransferConfig;
|
|
5
5
|
export declare const PARTITIONING_FIELD_REMOVAL_ERROR_PREFIX = "Cannot remove partitioning_field from an existing transfer config";
|
|
6
6
|
export declare const PARTITIONING_FIELD_REMOVAL_ERROR = "Cannot remove partitioning_field from an existing transfer config. The BigQuery Data Transfer API does not support clearing this parameter once it has been set. To change partitioning, create a new transfer config with the desired setting.";
|
|
7
|
-
/**
|
|
8
|
-
|
|
7
|
+
/**
|
|
8
|
+
* Creates the protobuf-shaped request used for a scheduled query.
|
|
9
|
+
*
|
|
10
|
+
* The query runs as whichever identity creates it, which is this function. To
|
|
11
|
+
* name a different account, `serviceAccountName` goes on the request rather
|
|
12
|
+
* than on the transfer config, and the caller needs `actAs` on the account it
|
|
13
|
+
* names, including its own.
|
|
14
|
+
*/
|
|
15
|
+
export declare function createTransferConfigRequest(config: ResolvedBigqueryFirestoreExportConfig): bigqueryDataTransfer.protos.google.cloud.bigquery.datatransfer.v1.ICreateTransferConfigRequest;
|
|
9
16
|
export declare function getTransferConfig(client: DataTransferClient, transferConfigName: string): Promise<TransferConfig | null>;
|
|
10
17
|
export declare function createTransferConfig(client: DataTransferClient, config: ResolvedBigqueryFirestoreExportConfig): Promise<TransferConfig>;
|
|
11
18
|
/** Builds a minimal update mask while retaining unsupported immutable fields. */
|
package/lib/dts.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"dts.d.ts","sourceRoot":"","sources":["../src/dts.ts"],"names":[],"mappings":"AAgBA,OAAO,KAAK,oBAAoB,MAAM,sCAAsC,CAAC;AAC7E,OAAO,KAAK,EAAE,qCAAqC,EAAE,MAAM,iBAAiB,CAAC;AAG7E,MAAM,MAAM,kBAAkB,GAC5B,oBAAoB,CAAC,EAAE,CAAC,yBAAyB,CAAC;AACpD,MAAM,MAAM,cAAc,GACxB,oBAAoB,CAAC,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,YAAY,CAAC,EAAE,CAAC,eAAe,CAAC;AAMpF,eAAO,MAAM,uCAAuC,sEACxB,CAAC;AAC7B,eAAO,MAAM,gCAAgC,oPAAgN,CAAC;AAoC9P
|
|
1
|
+
{"version":3,"file":"dts.d.ts","sourceRoot":"","sources":["../src/dts.ts"],"names":[],"mappings":"AAgBA,OAAO,KAAK,oBAAoB,MAAM,sCAAsC,CAAC;AAC7E,OAAO,KAAK,EAAE,qCAAqC,EAAE,MAAM,iBAAiB,CAAC;AAG7E,MAAM,MAAM,kBAAkB,GAC5B,oBAAoB,CAAC,EAAE,CAAC,yBAAyB,CAAC;AACpD,MAAM,MAAM,cAAc,GACxB,oBAAoB,CAAC,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,YAAY,CAAC,EAAE,CAAC,eAAe,CAAC;AAMpF,eAAO,MAAM,uCAAuC,sEACxB,CAAC;AAC7B,eAAO,MAAM,gCAAgC,oPAAgN,CAAC;AAoC9P;;;;;;;GAOG;AACH,wBAAgB,2BAA2B,CACzC,MAAM,EAAE,qCAAqC,GAC5C,oBAAoB,CAAC,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,YAAY,CAAC,EAAE,CAAC,4BAA4B,CAqBhG;AAED,wBAAsB,iBAAiB,CACrC,MAAM,EAAE,kBAAkB,EAC1B,kBAAkB,EAAE,MAAM,GACzB,OAAO,CAAC,cAAc,GAAG,IAAI,CAAC,CAchC;AAED,wBAAsB,oBAAoB,CACxC,MAAM,EAAE,kBAAkB,EAC1B,MAAM,EAAE,qCAAqC,GAC5C,OAAO,CAAC,cAAc,CAAC,CAUzB;AAED,iFAAiF;AACjF,wBAAsB,oCAAoC,CACxD,MAAM,EAAE,kBAAkB,EAC1B,kBAAkB,EAAE,MAAM,EAC1B,MAAM,EAAE,qCAAqC,GAC5C,OAAO,CAAC,oBAAoB,CAAC,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,YAAY,CAAC,EAAE,CAAC,4BAA4B,CAAC,CAyDzG;AAED,wBAAsB,oBAAoB,CACxC,MAAM,EAAE,kBAAkB,EAC1B,kBAAkB,EAAE,MAAM,EAC1B,MAAM,EAAE,qCAAqC,GAC5C,OAAO,CAAC,cAAc,CAAC,CAczB"}
|
package/lib/dts.js
CHANGED
|
@@ -82,8 +82,15 @@ function transferConfigFields(config) {
|
|
|
82
82
|
function stringField(value) {
|
|
83
83
|
return { stringValue: value ?? "" };
|
|
84
84
|
}
|
|
85
|
-
/**
|
|
86
|
-
|
|
85
|
+
/**
|
|
86
|
+
* Creates the protobuf-shaped request used for a scheduled query.
|
|
87
|
+
*
|
|
88
|
+
* The query runs as whichever identity creates it, which is this function. To
|
|
89
|
+
* name a different account, `serviceAccountName` goes on the request rather
|
|
90
|
+
* than on the transfer config, and the caller needs `actAs` on the account it
|
|
91
|
+
* names, including its own.
|
|
92
|
+
*/
|
|
93
|
+
function createTransferConfigRequest(config) {
|
|
87
94
|
return {
|
|
88
95
|
parent: `projects/${config.projectId}`,
|
|
89
96
|
transferConfig: {
|
|
@@ -100,9 +107,6 @@ function createTransferConfigRequest(config, serviceAccountEmail) {
|
|
|
100
107
|
},
|
|
101
108
|
schedule: config.schedule,
|
|
102
109
|
notificationPubsubTopic: `projects/${config.projectId}/topics/${config.pubSubTopic}`,
|
|
103
|
-
...(serviceAccountEmail
|
|
104
|
-
? { serviceAccountName: serviceAccountEmail }
|
|
105
|
-
: {}),
|
|
106
110
|
},
|
|
107
111
|
};
|
|
108
112
|
}
|
|
@@ -124,7 +128,7 @@ async function getTransferConfig(client, transferConfigName) {
|
|
|
124
128
|
}
|
|
125
129
|
async function createTransferConfig(client, config) {
|
|
126
130
|
logs.createTransferConfig();
|
|
127
|
-
const [created] = await client.createTransferConfig(createTransferConfigRequest(config
|
|
131
|
+
const [created] = await client.createTransferConfig(createTransferConfigRequest(config));
|
|
128
132
|
if (!created.name) {
|
|
129
133
|
throw new Error("BigQuery API returned a transfer config without a name");
|
|
130
134
|
}
|
package/lib/dts.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"dts.js","sourceRoot":"","sources":["../src/dts.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;GAcG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAEH,MAAY,oBAAoB,iEAA6C;AAE7E,MAAY,IAAI,mCAAe;AAO/B,MAAM,cAAc,GAAG,CAAC,CAAC;AACzB,MAAM,0BAA0B,GAC9B,mEAAmE,CAAC;AAEzD,QAAA,uCAAuC,GAClD,0BAA0B,CAAC;AAChB,QAAA,gCAAgC,GAAG,GAAG,0BAA0B,gLAAgL,CAAC;AAE9P,SAAS,eAAe,CAAC,GAAY;IACnC,OAAO,CACL,OAAO,GAAG,KAAK,QAAQ;QACvB,GAAG,KAAK,IAAI;QACZ,MAAM,IAAI,GAAG;QACb,GAAG,CAAC,IAAI,KAAK,cAAc,CAC5B,CAAC;AACJ,CAAC;AAED,SAAS,oBAAoB,CAAC,MAAsB;IAClD,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC;IACrC,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,MAAM,IAAI,KAAK,CACb,8DAA8D,CAC/D,CAAC;IACJ,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC;QAClB,MAAM,IAAI,KAAK,CACb,oEAAoE,CACrE,CAAC;IACJ,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,+BAA+B,EAAE,CAAC;QAC5C,MAAM,IAAI,KAAK,CACb,8FAA8F,CAC/F,CAAC;IACJ,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,SAAS,WAAW,CAAC,KAAyB;IAC5C,OAAO,EAAE,WAAW,EAAE,KAAK,IAAI,EAAE,EAAE,CAAC;AACtC,CAAC;AAED
|
|
1
|
+
{"version":3,"file":"dts.js","sourceRoot":"","sources":["../src/dts.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;GAcG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAEH,MAAY,oBAAoB,iEAA6C;AAE7E,MAAY,IAAI,mCAAe;AAO/B,MAAM,cAAc,GAAG,CAAC,CAAC;AACzB,MAAM,0BAA0B,GAC9B,mEAAmE,CAAC;AAEzD,QAAA,uCAAuC,GAClD,0BAA0B,CAAC;AAChB,QAAA,gCAAgC,GAAG,GAAG,0BAA0B,gLAAgL,CAAC;AAE9P,SAAS,eAAe,CAAC,GAAY;IACnC,OAAO,CACL,OAAO,GAAG,KAAK,QAAQ;QACvB,GAAG,KAAK,IAAI;QACZ,MAAM,IAAI,GAAG;QACb,GAAG,CAAC,IAAI,KAAK,cAAc,CAC5B,CAAC;AACJ,CAAC;AAED,SAAS,oBAAoB,CAAC,MAAsB;IAClD,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC;IACrC,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,MAAM,IAAI,KAAK,CACb,8DAA8D,CAC/D,CAAC;IACJ,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC;QAClB,MAAM,IAAI,KAAK,CACb,oEAAoE,CACrE,CAAC;IACJ,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,+BAA+B,EAAE,CAAC;QAC5C,MAAM,IAAI,KAAK,CACb,8FAA8F,CAC/F,CAAC;IACJ,CAAC;IAED,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,SAAS,WAAW,CAAC,KAAyB;IAC5C,OAAO,EAAE,WAAW,EAAE,KAAK,IAAI,EAAE,EAAE,CAAC;AACtC,CAAC;AAED;;;;;;;GAOG;AACH,qCACE,MAA6C;IAE7C,OAAO;QACL,MAAM,EAAE,YAAY,MAAM,CAAC,SAAS,EAAE;QACtC,cAAc,EAAE;YACd,oBAAoB,EAAE,MAAM,CAAC,SAAS;YACtC,WAAW,EAAE,MAAM,CAAC,WAAW;YAC/B,YAAY,EAAE,iBAAiB;YAC/B,MAAM,EAAE;gBACN,MAAM,EAAE;oBACN,KAAK,EAAE,WAAW,CAAC,MAAM,CAAC,WAAW,CAAC;oBACtC,+BAA+B,EAAE,WAAW,CAC1C,GAAG,MAAM,CAAC,SAAS,sBAAsB,CAC1C;oBACD,iBAAiB,EAAE,WAAW,CAAC,gBAAgB,CAAC;oBAChD,kBAAkB,EAAE,WAAW,CAAC,MAAM,CAAC,iBAAiB,CAAC;iBAC1D;aACF;YACD,QAAQ,EAAE,MAAM,CAAC,QAAQ;YACzB,uBAAuB,EAAE,YAAY,MAAM,CAAC,SAAS,WAAW,MAAM,CAAC,WAAW,EAAE;SACrF;KACF,CAAC;AACJ,CAAC;AAEM,KAAK,4BACV,MAA0B,EAC1B,kBAA0B;IAE1B,IAAI,CAAC;QACH,MAAM,CAAC,MAAM,CAAC,GAAG,MAAM,MAAM,CAAC,iBAAiB,CAAC;YAC9C,IAAI,EAAE,kBAAkB;SACzB,CAAC,CAAC;QACH,OAAO,MAAM,CAAC;IAChB,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,IAAI,eAAe,CAAC,GAAG,CAAC,EAAE,CAAC;YACzB,IAAI,CAAC,sBAAsB,CAAC,kBAAkB,CAAC,CAAC;YAChD,OAAO,IAAI,CAAC;QACd,CAAC;QACD,IAAI,CAAC,uBAAuB,CAAC,kBAAkB,EAAE,GAAG,CAAC,CAAC;QACtD,MAAM,GAAG,CAAC;IACZ,CAAC;AACH,CAAC;AAEM,KAAK,+BACV,MAA0B,EAC1B,MAA6C;IAE7C,IAAI,CAAC,oBAAoB,EAAE,CAAC;IAC5B,MAAM,CAAC,OAAO,CAAC,GAAG,MAAM,MAAM,CAAC,oBAAoB,CACjD,2BAA2B,CAAC,MAAM,CAAC,CACpC,CAAC;IACF,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,CAAC;QAClB,MAAM,IAAI,KAAK,CAAC,wDAAwD,CAAC,CAAC;IAC5E,CAAC;IACD,IAAI,CAAC,qBAAqB,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;IACzC,OAAO,OAAO,CAAC;AACjB,CAAC;AAED,iFAAiF;AAC1E,KAAK,+CACV,MAA0B,EAC1B,kBAA0B,EAC1B,MAA6C;IAE7C,MAAM,cAAc,GAAG,MAAM,iBAAiB,CAAC,MAAM,EAAE,kBAAkB,CAAC,CAAC;IAC3E,IAAI,CAAC,cAAc;QAAE,MAAM,IAAI,KAAK,CAAC,2BAA2B,CAAC,CAAC;IAElE,MAAM,MAAM,GAAG,oBAAoB,CAAC,cAAc,CAAC,CAAC;IACpD,MAAM,aAAa,GAAG,IAAI,CAAC,KAAK,CAC9B,IAAI,CAAC,SAAS,CAAC,cAAc,CAAC,CACb,CAAC;IACpB,MAAM,aAAa,GAAG,oBAAoB,CAAC,aAAa,CAAC,CAAC;IAC1D,MAAM,UAAU,GAAa,EAAE,CAAC;IAEhC,IAAI,MAAM,CAAC,SAAS,KAAK,cAAc,CAAC,oBAAoB,EAAE,CAAC;QAC7D,UAAU,CAAC,IAAI,CAAC,wBAAwB,CAAC,CAAC;QAC1C,aAAa,CAAC,oBAAoB,GAAG,MAAM,CAAC,SAAS,CAAC;IACxD,CAAC;IACD,IAAI,MAAM,CAAC,WAAW,KAAK,MAAM,CAAC,KAAK,CAAC,WAAW,EAAE,CAAC;QACpD,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QAC1B,aAAa,CAAC,KAAK,CAAC,WAAW,GAAG,MAAM,CAAC,WAAW,CAAC;IACvD,CAAC;IAED,MAAM,aAAa,GAAG,GAAG,MAAM,CAAC,SAAS,sBAAsB,CAAC;IAChE,IAAI,aAAa,KAAK,MAAM,CAAC,+BAA+B,CAAC,WAAW,EAAE,CAAC;QACzE,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QAC1B,aAAa,CAAC,+BAA+B,CAAC,WAAW,GAAG,aAAa,CAAC;IAC5E,CAAC;IAED,MAAM,yBAAyB,GAC7B,MAAM,CAAC,kBAAkB,EAAE,WAAW,IAAI,EAAE,CAAC;IAC/C,MAAM,oBAAoB,GAAG,MAAM,CAAC,iBAAiB,IAAI,EAAE,CAAC;IAC5D,IAAI,oBAAoB,KAAK,yBAAyB,EAAE,CAAC;QACvD,IAAI,CAAC,oBAAoB,EAAE,CAAC;YAC1B,IAAI,CAAC,iCAAiC,CACpC,kBAAkB,EAClB,yBAAyB,CAC1B,CAAC;YACF,MAAM,IAAI,KAAK,CAAC,QAAA,gCAAgC,CAAC,CAAC;QACpD,CAAC;QACD,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QAC1B,aAAa,CAAC,kBAAkB,KAAK,EAAE,CAAC;QACxC,aAAa,CAAC,kBAAkB,CAAC,WAAW,GAAG,oBAAoB,CAAC;IACtE,CAAC;IAED,IAAI,MAAM,CAAC,QAAQ,KAAK,cAAc,CAAC,QAAQ,EAAE,CAAC;QAChD,UAAU,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;QAC5B,aAAa,CAAC,QAAQ,GAAG,MAAM,CAAC,QAAQ,CAAC;IAC3C,CAAC;IAED,MAAM,aAAa,GAAG,YAAY,MAAM,CAAC,SAAS,WAAW,MAAM,CAAC,WAAW,EAAE,CAAC;IAClF,IAAI,aAAa,KAAK,cAAc,CAAC,uBAAuB,EAAE,CAAC;QAC7D,UAAU,CAAC,IAAI,CAAC,2BAA2B,CAAC,CAAC;QAC7C,aAAa,CAAC,uBAAuB,GAAG,aAAa,CAAC;IACxD,CAAC;IAED,OAAO;QACL,cAAc,EAAE,aAAa;QAC7B,UAAU,EAAE,EAAE,KAAK,EAAE,CAAC,GAAG,IAAI,GAAG,CAAC,UAAU,CAAC,CAAC,EAAE;KAChD,CAAC;AACJ,CAAC;AAEM,KAAK,+BACV,MAA0B,EAC1B,kBAA0B,EAC1B,MAA6C;IAE7C,MAAM,OAAO,GAAG,MAAM,oCAAoC,CACxD,MAAM,EACN,kBAAkB,EAClB,MAAM,CACP,CAAC;IACF,IAAI,CAAC,oBAAoB,CAAC,kBAAkB,CAAC,CAAC;IAC9C,MAAM,SAAS,GACb,oBAAoB,CAAC,MAAM,CAAC,MAAM,CAAC,KAAK,CAAC,QAAQ,CAAC,YAAY,CAAC,EAAE,CAAC,2BAA2B,CAAC,UAAU,CACtG,OAAO,CACR,CAAC;IACJ,MAAM,CAAC,OAAO,CAAC,GAAG,MAAM,MAAM,CAAC,oBAAoB,CAAC,SAAS,CAAC,CAAC;IAC/D,IAAI,CAAC,qBAAqB,CAAC,kBAAkB,CAAC,CAAC;IAC/C,OAAO,OAAO,CAAC;AACjB,CAAC"}
|
package/lib/export-config.d.ts
CHANGED
|
@@ -27,8 +27,6 @@ export interface BigqueryFirestoreExportConfig {
|
|
|
27
27
|
pubSubTopic?: string;
|
|
28
28
|
/** Root Firestore collection for configs and run output. */
|
|
29
29
|
firestoreCollection?: string;
|
|
30
|
-
/** Runtime identity used when creating a DTS config. */
|
|
31
|
-
serviceAccount?: string;
|
|
32
30
|
/** Log verbosity. Defaults to `info`. */
|
|
33
31
|
logLevel?: LogLevel;
|
|
34
32
|
}
|
|
@@ -46,7 +44,6 @@ export interface ResolvedBigqueryFirestoreExportConfig {
|
|
|
46
44
|
schedule: string;
|
|
47
45
|
pubSubTopic: string;
|
|
48
46
|
firestoreCollection: string;
|
|
49
|
-
serviceAccount?: string;
|
|
50
47
|
logLevel: LogLevel;
|
|
51
48
|
}
|
|
52
49
|
/** Deploy-time values used to construct the v2 triggers. */
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"export-config.d.ts","sourceRoot":"","sources":["../src/export-config.ts"],"names":[],"mappings":"AAgBA,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,2BAA2B,CAAC;AAE5D,sDAAsD;AACtD,MAAM,MAAM,QAAQ,GAAG,OAAO,GAAG,MAAM,GAAG,MAAM,GAAG,OAAO,GAAG,QAAQ,CAAC;AAEtE,oEAAoE;AACpE,MAAM,WAAW,6BAA6B;IAC5C,sEAAsE;IACtE,uBAAuB,EAAE,MAAM,CAAC;IAChC,+BAA+B;IAC/B,SAAS,EAAE,MAAM,CAAC;IAClB,qEAAqE;IACrE,UAAU,EAAE,MAAM,CAAC;IACnB,6DAA6D;IAC7D,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B,uCAAuC;IACvC,SAAS,EAAE,MAAM,CAAC;IAClB,6CAA6C;IAC7C,SAAS,EAAE,MAAM,CAAC;IAClB,4BAA4B;IAC5B,WAAW,EAAE,MAAM,CAAC;IACpB,+CAA+C;IAC/C,WAAW,EAAE,MAAM,CAAC;IACpB,qDAAqD;IACrD,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,uEAAuE;IACvE,QAAQ,EAAE,MAAM,CAAC;IACjB,4DAA4D;IAC5D,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,4DAA4D;IAC5D,mBAAmB,CAAC,EAAE,MAAM,CAAC;IAC7B,
|
|
1
|
+
{"version":3,"file":"export-config.d.ts","sourceRoot":"","sources":["../src/export-config.ts"],"names":[],"mappings":"AAgBA,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,2BAA2B,CAAC;AAE5D,sDAAsD;AACtD,MAAM,MAAM,QAAQ,GAAG,OAAO,GAAG,MAAM,GAAG,MAAM,GAAG,OAAO,GAAG,QAAQ,CAAC;AAEtE,oEAAoE;AACpE,MAAM,WAAW,6BAA6B;IAC5C,sEAAsE;IACtE,uBAAuB,EAAE,MAAM,CAAC;IAChC,+BAA+B;IAC/B,SAAS,EAAE,MAAM,CAAC;IAClB,qEAAqE;IACrE,UAAU,EAAE,MAAM,CAAC;IACnB,6DAA6D;IAC7D,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B,uCAAuC;IACvC,SAAS,EAAE,MAAM,CAAC;IAClB,6CAA6C;IAC7C,SAAS,EAAE,MAAM,CAAC;IAClB,4BAA4B;IAC5B,WAAW,EAAE,MAAM,CAAC;IACpB,+CAA+C;IAC/C,WAAW,EAAE,MAAM,CAAC;IACpB,qDAAqD;IACrD,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,uEAAuE;IACvE,QAAQ,EAAE,MAAM,CAAC;IACjB,4DAA4D;IAC5D,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,4DAA4D;IAC5D,mBAAmB,CAAC,EAAE,MAAM,CAAC;IAC7B,yCAAyC;IACzC,QAAQ,CAAC,EAAE,QAAQ,CAAC;CACrB;AAED,qEAAqE;AACrE,MAAM,WAAW,qCAAqC;IACpD,uBAAuB,EAAE,MAAM,CAAC;IAChC,SAAS,EAAE,MAAM,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B,SAAS,EAAE,MAAM,CAAC;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,WAAW,EAAE,MAAM,CAAC;IACpB,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,QAAQ,EAAE,MAAM,CAAC;IACjB,WAAW,EAAE,MAAM,CAAC;IACpB,mBAAmB,EAAE,MAAM,CAAC;IAC5B,QAAQ,EAAE,QAAQ,CAAC;CACpB;AAED,4DAA4D;AAC5D,MAAM,WAAW,iBAAiB;IAChC,WAAW,EAAE,MAAM,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;CAC1C;AAgBD,4EAA4E;AAC5E,wBAAgB,aAAa,CAC3B,MAAM,EAAE,6BAA6B,GACpC,qCAAqC,CA4BvC"}
|
package/lib/export-config.js
CHANGED
|
@@ -49,7 +49,6 @@ function resolveConfig(config) {
|
|
|
49
49
|
schedule: required(config.schedule, "schedule"),
|
|
50
50
|
pubSubTopic: optional(config.pubSubTopic) ?? `kit-${instanceId}-processMessages`,
|
|
51
51
|
firestoreCollection: optional(config.firestoreCollection) ?? "transferConfigs",
|
|
52
|
-
serviceAccount: optional(config.serviceAccount),
|
|
53
52
|
logLevel,
|
|
54
53
|
};
|
|
55
54
|
}
|
package/lib/export-config.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"export-config.js","sourceRoot":"","sources":["../src/export-config.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;GAcG;;;
|
|
1
|
+
{"version":3,"file":"export-config.js","sourceRoot":"","sources":["../src/export-config.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;GAcG;;;AA2DH,SAAS,QAAQ,CAAC,KAAa,EAAE,KAAa;IAC5C,MAAM,UAAU,GAAG,KAAK,CAAC,IAAI,EAAE,CAAC;IAChC,IAAI,CAAC,UAAU,EAAE,CAAC;QAChB,MAAM,IAAI,KAAK,CAAC,GAAG,KAAK,8BAA8B,CAAC,CAAC;IAC1D,CAAC;IACD,OAAO,UAAU,CAAC;AACpB,CAAC;AAED,SAAS,QAAQ,CAAC,KAAyB;IACzC,IAAI,KAAK,KAAK,SAAS;QAAE,OAAO,SAAS,CAAC;IAC1C,MAAM,UAAU,GAAG,KAAK,CAAC,IAAI,EAAE,CAAC;IAChC,OAAO,UAAU,IAAI,SAAS,CAAC;AACjC,CAAC;AAED,4EAA4E;AAC5E,uBACE,MAAqC;IAErC,MAAM,UAAU,GAAG,QAAQ,CAAC,MAAM,CAAC,UAAU,EAAE,YAAY,CAAC,CAAC;IAC7D,MAAM,QAAQ,GAAG,MAAM,CAAC,QAAQ,IAAI,MAAM,CAAC;IAE3C,IAAI,CAAC,CAAC,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,QAAQ,CAAC,CAAC,QAAQ,CAAC,QAAQ,CAAC,EAAE,CAAC;QACrE,MAAM,IAAI,KAAK,CAAC,yBAAyB,QAAQ,EAAE,CAAC,CAAC;IACvD,CAAC;IAED,OAAO;QACL,uBAAuB,EAAE,QAAQ,CAC/B,MAAM,CAAC,uBAAuB,EAC9B,yBAAyB,CAC1B;QACD,SAAS,EAAE,QAAQ,CAAC,MAAM,CAAC,SAAS,EAAE,WAAW,CAAC;QAClD,UAAU;QACV,kBAAkB,EAAE,QAAQ,CAAC,MAAM,CAAC,kBAAkB,CAAC;QACvD,SAAS,EAAE,QAAQ,CAAC,MAAM,CAAC,SAAS,EAAE,WAAW,CAAC;QAClD,SAAS,EAAE,QAAQ,CAAC,MAAM,CAAC,SAAS,EAAE,WAAW,CAAC;QAClD,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,aAAa,CAAC;QACxD,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,WAAW,EAAE,aAAa,CAAC;QACxD,iBAAiB,EAAE,QAAQ,CAAC,MAAM,CAAC,iBAAiB,CAAC;QACrD,QAAQ,EAAE,QAAQ,CAAC,MAAM,CAAC,QAAQ,EAAE,UAAU,CAAC;QAC/C,WAAW,EACT,QAAQ,CAAC,MAAM,CAAC,WAAW,CAAC,IAAI,OAAO,UAAU,kBAAkB;QACrE,mBAAmB,EACjB,QAAQ,CAAC,MAAM,CAAC,mBAAmB,CAAC,IAAI,iBAAiB;QAC3D,QAAQ;KACT,CAAC;AACJ,CAAC"}
|
package/lib/index.js
CHANGED
|
@@ -127,7 +127,7 @@ function getContext() {
|
|
|
127
127
|
/** Consumes BigQuery Data Transfer completion notifications. */
|
|
128
128
|
exports.processMessages = (0, pubsub_2.onMessagePublished)({
|
|
129
129
|
topic: config_1.CONFIG_EXPRESSIONS.pubSubTopic,
|
|
130
|
-
retry:
|
|
130
|
+
retry: false,
|
|
131
131
|
}, (event) => (0, handlers_1.handleMessagePublished)(event, getContext()));
|
|
132
132
|
/** Creates, links, or reconciles this deployment's scheduled query. */
|
|
133
133
|
exports.upsertTransferConfig = (0, tasks_1.onTaskDispatched)({
|
package/lib/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;GAcG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAEH;;;;GAIG;AAEH,qDAAkD;AAClD,iFAAkF;AAClF,iDAA8C;AAC9C,4CAA4D;AAC5D,wDAAwD;AACxD,yDAAkE;AAClE,uDAA+D;AAE/D,8CAAkE;AAClE,+DAGyC;AACzC,qCAA6D;AAE7D,mDAAgD;AAChD,yCAIoB;AACpB,MAAY,IAAI,mCAAe;AAG/B,wCAAsB;AAEtB,MAAM,eAAe,GAAG,sBAAsB,CAAC;AAC/C,MAAM,cAAc,GAAwB;IAC1C,sBAAsB;IACtB,sBAAsB;IACtB,oBAAoB;IACpB,8BAA8B;IAC9B,mBAAmB;CACpB,CAAC;AACF,MAAM,aAAa,GAAG;IACpB;QACE,GAAG,EAAE,yBAAyB;QAC9B,MAAM,EAAE,4DAA4D;KACrE;IACD;QACE,GAAG,EAAE,qCAAqC;QAC1C,MAAM,EAAE,6DAA6D;KACtE;IACD;QACE,GAAG,EAAE,uBAAuB;QAC5B,MAAM,EAAE,sDAAsD;KAC/D;CACO,CAAC;AAEX,KAAK,MAAM,IAAI,IAAI,cAAc,EAAE,CAAC;IAClC,IAAA,iBAAY,EAAC,IAAI,CAAC,CAAC;AACrB,CAAC;AAED,KAAK,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,aAAa,EAAE,CAAC;IAC5C,IAAA,gBAAW,EAAC,GAAG,EAAE,MAAM,CAAC,CAAC;AAC3B,CAAC;AAED,IAAA,4BAAgB,EAAC;IACf,IAAI,EAAE,EAAE,QAAQ,EAAE,eAAe,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,EAAE,EAAE,EAAE;CACxD,CAAC,CAAC;AACH,IAAA,yBAAa,EAAC;IACZ,IAAI,EAAE,EAAE,QAAQ,EAAE,eAAe,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,EAAE,EAAE,EAAE;CACxD,CAAC,CAAC;AAEH,IAAI,GAA+B,CAAC;AAEpC,SAAS,UAAU;IACjB,IAAI,GAAG;QAAE,OAAO,GAAG,CAAC;IAEpB,MAAM,MAAM,GAA0C,IAAA,6BAAa,EACjE,IAAA,sBAAa,GAAE,CAChB,CAAC;IACF,IAAI,IAAA,aAAO,GAAE,CAAC,MAAM,KAAK,CAAC;QAAE,IAAA,mBAAa,EAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC;IAC3E,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IAElB,GAAG,GAAG;QACJ,EAAE,EAAE,IAAA,wBAAY,GAAE;QAClB,QAAQ,EAAE,IAAI,mBAAQ,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC;QACvD,YAAY,EAAE,IAAI,2BAAoB,CAAC,yBAAyB,CAAC;YAC/D,SAAS,EAAE,MAAM,CAAC,SAAS;SAC5B,CAAC;QACF,MAAM,EAAE,IAAI,eAAM,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC;QACnD,MAAM;KACP,CAAC;IACF,OAAO,GAAG,CAAC;AACb,CAAC;AAED,gEAAgE;AACnD,QAAA,eAAe,GAAG,IAAA,2BAAkB,EAC/C;IACE,KAAK,EAAE,2BAAkB,CAAC,WAAW;IACrC,KAAK,EAAE,
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;GAcG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAEH;;;;GAIG;AAEH,qDAAkD;AAClD,iFAAkF;AAClF,iDAA8C;AAC9C,4CAA4D;AAC5D,wDAAwD;AACxD,yDAAkE;AAClE,uDAA+D;AAE/D,8CAAkE;AAClE,+DAGyC;AACzC,qCAA6D;AAE7D,mDAAgD;AAChD,yCAIoB;AACpB,MAAY,IAAI,mCAAe;AAG/B,wCAAsB;AAEtB,MAAM,eAAe,GAAG,sBAAsB,CAAC;AAC/C,MAAM,cAAc,GAAwB;IAC1C,sBAAsB;IACtB,sBAAsB;IACtB,oBAAoB;IACpB,8BAA8B;IAC9B,mBAAmB;CACpB,CAAC;AACF,MAAM,aAAa,GAAG;IACpB;QACE,GAAG,EAAE,yBAAyB;QAC9B,MAAM,EAAE,4DAA4D;KACrE;IACD;QACE,GAAG,EAAE,qCAAqC;QAC1C,MAAM,EAAE,6DAA6D;KACtE;IACD;QACE,GAAG,EAAE,uBAAuB;QAC5B,MAAM,EAAE,sDAAsD;KAC/D;CACO,CAAC;AAEX,KAAK,MAAM,IAAI,IAAI,cAAc,EAAE,CAAC;IAClC,IAAA,iBAAY,EAAC,IAAI,CAAC,CAAC;AACrB,CAAC;AAED,KAAK,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,IAAI,aAAa,EAAE,CAAC;IAC5C,IAAA,gBAAW,EAAC,GAAG,EAAE,MAAM,CAAC,CAAC;AAC3B,CAAC;AAED,IAAA,4BAAgB,EAAC;IACf,IAAI,EAAE,EAAE,QAAQ,EAAE,eAAe,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,EAAE,EAAE,EAAE;CACxD,CAAC,CAAC;AACH,IAAA,yBAAa,EAAC;IACZ,IAAI,EAAE,EAAE,QAAQ,EAAE,eAAe,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,EAAE,EAAE,EAAE;CACxD,CAAC,CAAC;AAEH,IAAI,GAA+B,CAAC;AAEpC,SAAS,UAAU;IACjB,IAAI,GAAG;QAAE,OAAO,GAAG,CAAC;IAEpB,MAAM,MAAM,GAA0C,IAAA,6BAAa,EACjE,IAAA,sBAAa,GAAE,CAChB,CAAC;IACF,IAAI,IAAA,aAAO,GAAE,CAAC,MAAM,KAAK,CAAC;QAAE,IAAA,mBAAa,EAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC;IAC3E,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IAElB,GAAG,GAAG;QACJ,EAAE,EAAE,IAAA,wBAAY,GAAE;QAClB,QAAQ,EAAE,IAAI,mBAAQ,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC;QACvD,YAAY,EAAE,IAAI,2BAAoB,CAAC,yBAAyB,CAAC;YAC/D,SAAS,EAAE,MAAM,CAAC,SAAS;SAC5B,CAAC;QACF,MAAM,EAAE,IAAI,eAAM,CAAC,EAAE,SAAS,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC;QACnD,MAAM;KACP,CAAC;IACF,OAAO,GAAG,CAAC;AACb,CAAC;AAED,gEAAgE;AACnD,QAAA,eAAe,GAAG,IAAA,2BAAkB,EAC/C;IACE,KAAK,EAAE,2BAAkB,CAAC,WAAW;IACrC,KAAK,EAAE,KAAK;CACb,EACD,CAAC,KAAK,EAAE,EAAE,CAAC,IAAA,iCAAsB,EAAC,KAAK,EAAE,UAAU,EAAE,CAAC,CACvD,CAAC;AAEF,uEAAuE;AAC1D,QAAA,oBAAoB,GAAG,IAAA,wBAAgB,EAClD;IACE,MAAM,EAAE,MAAM;IACd,WAAW,EAAE,EAAE,WAAW,EAAE,CAAC,EAAE,iBAAiB,EAAE,EAAE,EAAE;CACvD,EACD,GAAG,EAAE,CAAC,IAAA,qCAA0B,EAAC,UAAU,EAAE,CAAC,CAC/C,CAAC"}
|