@firebase-function-kits/firestore-bigquery-export 0.0.1 → 0.0.2-rc.0

package/CHANGELOG.md

@@ -0,0 +1 @@
+- Initial release

package/README.md

@@ -0,0 +1,234 @@
+# @firebase/firestore-bigquery-export
+
+Stream a Cloud Firestore collection to BigQuery. This is the Stream Firestore to
+BigQuery Firebase Extension as an npm package you add to your own Firebase
+Functions codebase and deploy.
+
+It listens for document writes on a collection, serializes each change, and
+writes it to a BigQuery changelog table. Failed writes are retried through a
+Firebase Functions runtime retry policy. The functions run in your own Firebase
+project; there is no hosted version, so you deploy them yourself.
+
+## Install
+
+```sh
+npm install @firebase/firestore-bigquery-export
+```
+
+## Required IAM
+
+Deploy needs these Google Cloud roles and APIs for the function's service
+account. Firebase CLI 15.23.0 or later creates that account, grants the roles
+below, enables the listed APIs, and attaches the account to every function in
+this kit. Do not set a custom runtime service account for this codebase — it
+conflicts with that automatic setup.
+
+| Role / API | Why |
+|---|---|
+| `roles/bigquery.dataEditor` | create dataset/table/views; insert rows |
+| `roles/bigquery.user` | run BigQuery jobs and materialized views |
+| `roles/datastore.user` | write failed-row records back to Firestore (only if you configure a backup collection) |
+| `roles/eventarc.eventReceiver` | receive Gen2 Firestore trigger events |
+| `roles/run.invoker` | allow Eventarc to invoke the Gen2 Cloud Run service |
+| `bigquery.googleapis.com` | mirror Firestore collection changes in BigQuery |
+
+If the dataset lives in a different project (`BIGQUERY_PROJECT_ID`), grant the
+managed runtime service account the `bigquery.*` roles on that project. For a
+CMEK dataset, also grant the BigQuery service account access to your KMS key.
+
+## Usage
+
+Export the three functions from your functions codebase entry:
+
+```ts
+// functions/src/index.ts
+export {
+  fsexportbigquery,
+  initBigQuerySync,
+  setupBigQuerySync,
+} from "@firebase/firestore-bigquery-export";
+```
+
+and configure them with a `.env` (or `.env.<projectId>`):
+
+```sh
+COLLECTION_PATH=users
+DATASET_ID=analytics
+TABLE_ID=users
+DATABASE_REGION=europe-west2
+```
+
+- `fsexportbigquery` is the Firestore trigger.
+- `initBigQuerySync` is the first-deploy provisioning lifecycle task.
+- `setupBigQuerySync` is the reconfigure provisioning lifecycle task.
+
+Importing the package without exporting its functions deploys nothing — the CLI
+only deploys what your entry file exports.
+
+## Deploy
+
+The package's `firebase.json` declares a `kit` stanza (Firebase CLI 15.25.1 or
+later, behind the `kits` experiment):
+
+```json
+{
+  "functions": [
+    {
+      "source": ".",
+      "kit": "firestore-bigquery-export",
+      "instances": {
+        "default": "."
+      }
+    }
+  ]
+}
+```
+
+`instances` maps each instance id to the directory (relative to
+`firebase.json`) holding that instance's `.env`. The CLI prefixes every
+function and task queue name with `kit-<instance id>-`, so the functions above
+deploy as `kit-default-fsexportbigquery`, `kit-default-initBigQuerySync`, and
+`kit-default-setupBigQuerySync`.
+
+```sh
+firebase experiments:enable kits
+firebase deploy --only functions
+```
+
+Deploy a single instance with `firebase deploy --only functions:<instance id>`.
+
+## Configuration
+
+Set these values in a `.env` (or `.env.<projectId>`) file. The Firebase CLI
+loads them at deploy time and prompts for any required values that are missing.
+`PROJECT_ID` is supplied by the Firebase CLI.
+
+| Field | Env var | Required | Default | Description |
+|---|---|---|---|---|
+| `collectionPath` | `COLLECTION_PATH` | no | `posts` | Collection or collection-group path |
+| `datasetId` | `DATASET_ID` | no | `firestore_export` | BigQuery dataset |
+| `tableId` | `TABLE_ID` | no | `posts` | BigQuery changelog table |
+| `databaseRegion` | `DATABASE_REGION` | yes | — | Region for the trigger and queues |
+| `datasetLocation` | `DATASET_LOCATION` | no | `us` | BigQuery dataset location |
+| `database` | `DATABASE` | no | `(default)` | Firestore database id |
+| `bigqueryProjectId` | `BIGQUERY_PROJECT_ID` | no | project id | Dataset project, if different |
+| `backupCollection` | `BACKUP_COLLECTION` | no | (empty) | Firestore collection for failed rows |
+| `transformFunction` | `TRANSFORM_FUNCTION` | no | (empty) | Optional transform Cloud Function |
+| `tablePartitioning` | `TABLE_PARTITIONING` | no | `NONE` | Table partitioning strategy |
+| `timePartitioningField` | `TIME_PARTITIONING_FIELD` | no | (empty) | Time-partitioning column name |
+| `timePartitioningFieldType` | `TIME_PARTITIONING_FIELD_TYPE` | no | `omit` | Time-partitioning field type |
+| `timePartitioningFirestoreField` | `TIME_PARTITIONING_FIRESTORE_FIELD` | no | (empty) | Firestore field for partitioning |
+| `clustering` | `CLUSTERING` | no | (empty) | Clustering columns (max 4) |
+| `wildcardIds` | `WILDCARD_IDS` | no | `false` | Store path-param values as columns |
+| `useNewSnapshotQuerySyntax` | `USE_NEW_SNAPSHOT_QUERY_SYNTAX` | no | `false` | Use newer snapshot query syntax |
+| `excludeOldData` | `EXCLUDE_OLD_DATA` | no | `false` | Skip previous document state on updates |
+| `viewType` | `VIEW_TYPE` | no | `view` | `view`, `materialized_incremental`, `materialized_non_incremental` |
+| `maxStaleness` | `MAX_STALENESS` | no | (empty) | Materialized view max staleness |
+| `refreshIntervalMinutes` | `REFRESH_INTERVAL_MINUTES` | no | (empty) | Materialized view refresh interval |
+| `kmsKeyName` | `KMS_KEY_NAME` | no | (empty) | CMEK key for the dataset |
+| `logLevel` | `LOG_LEVEL` | no | `info` | `debug`, `info`, `warn`, `error`, `silent` |
+
+## Multiple instances
+
+To export several collections, add one entry per instance to the `instances`
+map, each pointing at its own config directory with its own `.env`:
+
+```json
+{
+  "functions": [
+    {
+      "source": ".",
+      "kit": "firestore-bigquery-export",
+      "instances": {
+        "users": "instances/users",
+        "orders": "instances/orders"
+      }
+    }
+  ]
+}
+```
+
+Instance ids must be unique across all kit stanzas in the project, and every
+instance's function names are namespaced by its `kit-<instance id>-` prefix, so
+the instances cannot collide.
+
+## Events
+
+When `EVENTARC_CHANNEL` is configured, the function publishes lifecycle events
+such as `onStart`, `onError`, `onSuccess`, and `onCompletion` under
+`firebase.extensions.firestore-bigquery-export.v1.*`.
+
+## Provisioning
+
+The BigQuery dataset, table, and views are created by `tracker.initialize()`
+through the shared provisioning path used by both task functions. Both tasks
+are idempotent and retry on transient BigQuery failures (up to 15 attempts,
+60s minimum backoff).
+
+Deploy wiring (declared in the package):
+
+- First deploy runs `initBigQuerySync` automatically (`afterFirstDeploy`).
+- Later deploys run `setupBigQuerySync` automatically (`afterRedeploy`).
+
+`initBigQuerySync` and `setupBigQuerySync` call the same handler; they exist as
+separate task functions so first-deploy and redeploy can target different
+queues, matching the extension's install vs update/configure split.
+
+If automatic post-deploy enqueue did not run, enqueue a task yourself. The
+snippets below use the `default` instance; substitute your instance id in the
+`kit-<instance id>-` prefix if you named yours differently. Prefer
+`initBigQuerySync` after a first deploy and `setupBigQuerySync` after a
+redeploy or schema-related config change (`TABLE_PARTITIONING`, `CLUSTERING`,
+`WILDCARD_IDS`, `VIEW_TYPE`, and related fields).
+
+```sh
+node -e '
+const { initializeApp } = require("firebase-admin/app");
+const { getFunctions } = require("firebase-admin/functions");
+initializeApp();
+getFunctions()
+  .taskQueue("locations/'"$DATABASE_REGION"'/functions/kit-default-initBigQuerySync")
+  .enqueue({})
+  .then(() => console.log("init task enqueued"));
+'
+```
+
+Run it from your functions directory (it uses the installed `firebase-admin`)
+with application-default credentials and `GOOGLE_CLOUD_PROJECT` set. The caller
+needs `roles/cloudtasks.enqueuer`.
+
+Under the hood the task queue is an authenticated HTTP endpoint, so for a quick
+manual run you can also POST to it directly — note this skips the queue, so a
+failure is not retried:
+
+```sh
+URL=$(gcloud functions describe kit-default-initBigQuerySync \
+  --region "$DATABASE_REGION" --gen2 --format='value(url)')
+
+curl -fsS -X POST -H "Content-Type: application/json" -d '{"data":{}}' \
+  -H "Authorization: Bearer $(gcloud auth print-identity-token --audiences="$URL")" "$URL"
+```
+
+The Firestore write path never provisions on the hot path. If resources are
+missing when a write arrives, the inline write fails, the handler calls
+`ensureInitialized()` once as a self-heal and retries the write, and a remaining
+failure is surfaced to the function runtime retry policy (`retry: true` on
+`fsexportbigquery`).
+
+## API surface
+
+- **Main entry** (`@firebase/firestore-bigquery-export`): exports
+  `fsexportbigquery`, `initBigQuerySync`, and `setupBigQuerySync`, and
+  registers the first-deploy / redeploy provisioning hooks. Runtime config is
+  resolved lazily on first invocation. Use this entry from Firebase
+  deploy/emulator/runtime. For your own triggers, import from `./lib` instead.
+- **Library entry** (`./lib`): `handleDocumentWrite`, the raw handler for owning
+  trigger registration yourself, plus the config types and helpers
+  (`ExportConfig`, `resolveExportConfig`, `toTrackerConfig`) for building its
+  injected `HandlerContext`. Safe to import anywhere.
+
+The change-tracker engine is an internal dependency and is not exported.
+
+## License
+
+Apache-2.0

package/lib/config.d.ts

@@ -0,0 +1,33 @@
+import type { ChangeTrackerConfig, TimePartitioningGranularity } from "@firebaseextensions/firestore-bigquery-change-tracker";
+import type { Expression } from "firebase-functions/params";
+import type { ExportConfig } from "./export-config";
+type ConfigExpression<T extends string | number | boolean> = Expression<T>;
+export interface ConfigExpressions {
+    collectionPath: ConfigExpression<string>;
+    datasetId: ConfigExpression<string>;
+    tableId: ConfigExpression<string>;
+    location: ConfigExpression<string>;
+    database: ConfigExpression<string>;
+}
+export declare const CONFIG_EXPRESSIONS: ConfigExpressions;
+export declare function clustering(clusters: string | undefined): string[];
+export declare function buildPartitioningConfig(params: {
+    timePartitioning: TimePartitioningGranularity | null;
+    timePartitioningField: string | undefined;
+    timePartitioningFieldType: string | undefined;
+    timePartitioningFirestoreField: string | undefined;
+}): ChangeTrackerConfig["partitioning"];
+/**
+ * Resolves all deploy-time params into an {@link ExportConfig}.
+ *
+ * Param values are read when this is called. During the Firebase deploy-time
+ * discovery pass params return their declared defaults, so the default
+ * `TABLE_PARTITIONING=NONE` keeps {@link buildPartitioningConfig} from throwing.
+ * This is the bridge for the env-driven path: env params in, typed config out,
+ * which the main entry point wires into the exported functions.
+ *
+ * @returns The export configuration assembled from environment params.
+ */
+export declare function configFromEnv(): ExportConfig;
+export {};
+//# sourceMappingURL=config.d.ts.map

package/lib/config.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAgBA,OAAO,KAAK,EACV,mBAAmB,EAEnB,2BAA2B,EAC5B,MAAM,uDAAuD,CAAC;AAE/D,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,2BAA2B,CAAC;AAO5D,OAAO,KAAK,EAAE,YAAY,EAAY,MAAM,iBAAiB,CAAC;AAG9D,KAAK,gBAAgB,CAAC,CAAC,SAAS,MAAM,GAAG,MAAM,GAAG,OAAO,IAAI,UAAU,CAAC,CAAC,CAAC,CAAC;AAkH3E,MAAM,WAAW,iBAAiB;IAChC,cAAc,EAAE,gBAAgB,CAAC,MAAM,CAAC,CAAC;IACzC,SAAS,EAAE,gBAAgB,CAAC,MAAM,CAAC,CAAC;IACpC,OAAO,EAAE,gBAAgB,CAAC,MAAM,CAAC,CAAC;IAClC,QAAQ,EAAE,gBAAgB,CAAC,MAAM,CAAC,CAAC;IACnC,QAAQ,EAAE,gBAAgB,CAAC,MAAM,CAAC,CAAC;CACpC;AAiED,eAAO,MAAM,kBAAkB,EAAE,iBAMhC,CAAC;AAiBF,wBAAgB,UAAU,CAAC,QAAQ,EAAE,MAAM,GAAG,SAAS,YAEtD;AA4BD,wBAAgB,uBAAuB,CAAC,MAAM,EAAE;IAC9C,gBAAgB,EAAE,2BAA2B,GAAG,IAAI,CAAC;IACrD,qBAAqB,EAAE,MAAM,GAAG,SAAS,CAAC;IAC1C,yBAAyB,EAAE,MAAM,GAAG,SAAS,CAAC;IAC9C,8BAA8B,EAAE,MAAM,GAAG,SAAS,CAAC;CACpD,GAAG,mBAAmB,CAAC,cAAc,CAAC,CAuEtC;AA6BD;;;;;;;;;;GAUG;AACH,wBAAgB,aAAa,IAAI,YAAY,CAiC5C"}

package/lib/config.js

@@ -0,0 +1,345 @@
+"use strict";
+/*
+ * Copyright 2019 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.CONFIG_EXPRESSIONS = void 0;
+exports.clustering = clustering;
+exports.buildPartitioningConfig = buildPartitioningConfig;
+exports.configFromEnv = configFromEnv;
+const firestore_bigquery_change_tracker_1 = require("@firebaseextensions/firestore-bigquery-change-tracker");
+const params_1 = require("firebase-functions/params");
+const DECIMAL_RADIX = 10;
+const DATASET_LOCATION_OPTIONS = [
+    "us-central1",
+    "us-west4",
+    "europe-central2",
+    "us-west2",
+    "northamerica-northeast1",
+    "us-east4",
+    "us-west1",
+    "us-west3",
+    "southamerica-east1",
+    "us-east1",
+    "europe-west1",
+    "europe-north1",
+    "europe-west3",
+    "europe-west2",
+    "europe-west4",
+    "europe-west6",
+    "asia-east1",
+    "asia-east2",
+    "asia-southeast2",
+    "asia-south1",
+    "asia-southeast1",
+    "asia-northeast2",
+    "asia-northeast3",
+    "australia-southeast1",
+    "asia-northeast1",
+    "us",
+    "eu",
+    "africa-south1",
+    "me-west1",
+    "me-central1",
+    "me-central2",
+    "europe-west12",
+    "europe-north2",
+    "europe-west9",
+    "europe-west8",
+    "europe-southwest1",
+    "europe-west10",
+    "australia-southeast2",
+    "asia-south2",
+    "northamerica-northeast2",
+    "southamerica-west1",
+    "northamerica-south1",
+    "us-south1",
+    "us-east5",
+];
+const DATABASE_REGION_OPTIONS = [
+    "eur3",
+    "nam5",
+    "nam7",
+    "us-central1",
+    "us-west1",
+    "us-west2",
+    "us-west3",
+    "us-west4",
+    "us-east1",
+    "us-east4",
+    "us-east5",
+    "us-south1",
+    "northamerica-northeast1",
+    "northamerica-northeast2",
+    "northamerica-south1",
+    "southamerica-east1",
+    "southamerica-west1",
+    "europe-west1",
+    "europe-west2",
+    "europe-west3",
+    "europe-west4",
+    "europe-west6",
+    "europe-west8",
+    "europe-west9",
+    "europe-west10",
+    "europe-west12",
+    "europe-southwest1",
+    "europe-north1",
+    "europe-north2",
+    "europe-central2",
+    "me-central1",
+    "me-central2",
+    "me-west1",
+    "asia-south1",
+    "asia-south2",
+    "asia-southeast1",
+    "asia-southeast2",
+    "asia-east1",
+    "asia-east2",
+    "asia-northeast1",
+    "asia-northeast2",
+    "asia-northeast3",
+    "australia-southeast1",
+    "australia-southeast2",
+    "africa-south1",
+];
+const TABLE_PARTITIONING_OPTIONS = [
+    "HOUR",
+    "DAY",
+    "MONTH",
+    "YEAR",
+    "NONE",
+];
+const TIME_PARTITIONING_FIELD_TYPE_OPTIONS = [
+    "TIMESTAMP",
+    "DATETIME",
+    "DATE",
+    "omit",
+];
+const VIEW_TYPE_OPTIONS = [
+    "view",
+    "materialized_incremental",
+    "materialized_non_incremental",
+];
+const LOG_LEVEL_OPTIONS = ["debug", "info", "warn", "error", "silent"];
+/**
+ * Deploy-time parameters. Set these via a `.env` / `.env.<project>` file or the
+ * interactive prompts shown by `firebase deploy`.
+ *
+ * @see https://firebase.google.com/docs/functions/config-env
+ */
+const params = {
+    bigqueryProjectId: (0, params_1.defineString)("BIGQUERY_PROJECT_ID", {
+        default: params_1.projectID,
+    }),
+    database: (0, params_1.defineString)("DATABASE", { default: "(default)" }),
+    databaseRegion: (0, params_1.defineString)("DATABASE_REGION", {
+        input: (0, params_1.select)([...DATABASE_REGION_OPTIONS]),
+    }),
+    collectionPath: (0, params_1.defineString)("COLLECTION_PATH", { default: "posts" }),
+    datasetId: (0, params_1.defineString)("DATASET_ID", { default: "firestore_export" }),
+    tableId: (0, params_1.defineString)("TABLE_ID", { default: "posts" }),
+    datasetLocation: (0, params_1.defineString)("DATASET_LOCATION", {
+        default: "us",
+        input: (0, params_1.select)([...DATASET_LOCATION_OPTIONS]),
+    }),
+    backupCollection: (0, params_1.defineString)("BACKUP_COLLECTION", { default: "" }),
+    transformFunction: (0, params_1.defineString)("TRANSFORM_FUNCTION", { default: "" }),
+    tablePartitioning: (0, params_1.defineString)("TABLE_PARTITIONING", {
+        default: "NONE",
+        input: (0, params_1.select)([...TABLE_PARTITIONING_OPTIONS]),
+    }),
+    timePartitioningField: (0, params_1.defineString)("TIME_PARTITIONING_FIELD", {
+        default: "",
+    }),
+    timePartitioningFieldType: (0, params_1.defineString)("TIME_PARTITIONING_FIELD_TYPE", {
+        default: "omit",
+        input: (0, params_1.select)([...TIME_PARTITIONING_FIELD_TYPE_OPTIONS]),
+    }),
+    timePartitioningFirestoreField: (0, params_1.defineString)("TIME_PARTITIONING_FIRESTORE_FIELD", { default: "" }),
+    clustering: (0, params_1.defineString)("CLUSTERING", { default: "" }),
+    wildcardIds: (0, params_1.defineBoolean)("WILDCARD_IDS", {
+        default: false,
+    }),
+    useNewSnapshotQuerySyntax: (0, params_1.defineBoolean)("USE_NEW_SNAPSHOT_QUERY_SYNTAX", {
+        default: false,
+    }),
+    excludeOldData: (0, params_1.defineBoolean)("EXCLUDE_OLD_DATA", {
+        default: false,
+    }),
+    viewType: (0, params_1.defineString)("VIEW_TYPE", {
+        default: "view",
+        input: (0, params_1.select)([...VIEW_TYPE_OPTIONS]),
+    }),
+    maxStaleness: (0, params_1.defineString)("MAX_STALENESS", { default: "" }),
+    refreshIntervalMinutes: (0, params_1.defineString)("REFRESH_INTERVAL_MINUTES", {
+        default: "",
+    }),
+    kmsKeyName: (0, params_1.defineString)("KMS_KEY_NAME", { default: "" }),
+    logLevel: (0, params_1.defineString)("LOG_LEVEL", {
+        default: "info",
+        input: (0, params_1.select)([...LOG_LEVEL_OPTIONS]),
+    }),
+};
+exports.CONFIG_EXPRESSIONS = {
+    collectionPath: params.collectionPath,
+    datasetId: params.datasetId,
+    tableId: params.tableId,
+    location: params.databaseRegion,
+    database: params.database,
+};
+function timePartitioning(type) {
+    if (type === "HOUR" ||
+        type === "DAY" ||
+        type === "MONTH" ||
+        type === "YEAR") {
+        return type;
+    }
+    return null;
+}
+function clustering(clusters) {
+    return clusters ? clusters.split(",").slice(0, 4) : null;
+}
+function normalizeOptionalPartitionValue(value) {
+    const normalized = value?.trim();
+    if (!normalized || normalized === "NONE" || normalized === "omit") {
+        return undefined;
+    }
+    return normalized;
+}
+function normalizePartitionFieldType(value) {
+    const normalized = normalizeOptionalPartitionValue(value);
+    if (normalized === "TIMESTAMP" ||
+        normalized === "DATE" ||
+        normalized === "DATETIME") {
+        return normalized;
+    }
+    return undefined;
+}
+function buildPartitioningConfig(params) {
+    const { timePartitioning } = params;
+    const rawFieldName = params.timePartitioningField?.trim();
+    const rawFieldType = params.timePartitioningFieldType?.trim();
+    const rawFirestoreField = params.timePartitioningFirestoreField?.trim();
+    const formatValue = (value) => value && value.length > 0 ? `"${value}"` : "(empty)";
+    const throwInvalidPartitioningConfig = (detail) => {
+        throw new Error([
+            "Invalid partitioning configuration for firestore-bigquery-export.",
+            detail,
+            `Received TABLE_PARTITIONING=${formatValue(timePartitioning ?? undefined)},`,
+            `TIME_PARTITIONING_FIELD=${formatValue(rawFieldName)},`,
+            `TIME_PARTITIONING_FIRESTORE_FIELD=${formatValue(rawFirestoreField)},`,
+            `TIME_PARTITIONING_FIELD_TYPE=${formatValue(rawFieldType)}.`,
+            "Valid combinations are:",
+            "1) Ingestion-time: TABLE_PARTITIONING set and all TIME_PARTITIONING_* values empty/NONE/omit.",
+            "2) Timestamp field: TABLE_PARTITIONING set, TIME_PARTITIONING_FIELD=timestamp, TIME_PARTITIONING_FIRESTORE_FIELD empty.",
+            "3) Custom field: TABLE_PARTITIONING set, and TIME_PARTITIONING_FIELD + TIME_PARTITIONING_FIRESTORE_FIELD + TIME_PARTITIONING_FIELD_TYPE all provided.",
+        ].join(" "));
+    };
+    const fieldName = normalizeOptionalPartitionValue(params.timePartitioningField);
+    const fieldType = normalizePartitionFieldType(params.timePartitioningFieldType);
+    const firestoreField = normalizeOptionalPartitionValue(params.timePartitioningFirestoreField);
+    if (!timePartitioning) {
+        if (fieldName || fieldType || firestoreField) {
+            return throwInvalidPartitioningConfig("Partition-specific fields cannot be provided when TABLE_PARTITIONING is NONE.");
+        }
+        return { granularity: "NONE" };
+    }
+    if (!fieldName && !firestoreField) {
+        return { granularity: timePartitioning };
+    }
+    if (fieldName === "timestamp" && !firestoreField) {
+        return {
+            granularity: timePartitioning,
+            bigqueryColumnName: "timestamp",
+            ...(fieldType ? { bigqueryColumnType: fieldType } : {}),
+        };
+    }
+    if (fieldName && firestoreField && fieldType) {
+        return {
+            granularity: timePartitioning,
+            bigqueryColumnName: fieldName,
+            bigqueryColumnType: fieldType,
+            firestoreFieldName: firestoreField,
+        };
+    }
+    return throwInvalidPartitioningConfig("When TABLE_PARTITIONING is set, partitioning fields are either incomplete or invalid.");
+}
+function normalizeLogLevel(level) {
+    switch ((level || "").toLowerCase()) {
+        case "debug":
+            return "debug";
+        case "info":
+            return "info";
+        case "warn":
+            return "warn";
+        case "error":
+            return "error";
+        case "silent":
+            return "silent";
+        default:
+            return firestore_bigquery_change_tracker_1.LogLevel.INFO;
+    }
+}
+function normalizePositiveInt(value) {
+    const normalized = Number.parseInt(value, DECIMAL_RADIX);
+    return normalized > 0 ? normalized : undefined;
+}
+/** Coerce an empty-string param value to `undefined`. */
+function optional(value) {
+    return value.length > 0 ? value : undefined;
+}
+/**
+ * Resolves all deploy-time params into an {@link ExportConfig}.
+ *
+ * Param values are read when this is called. During the Firebase deploy-time
+ * discovery pass params return their declared defaults, so the default
+ * `TABLE_PARTITIONING=NONE` keeps {@link buildPartitioningConfig} from throwing.
+ * This is the bridge for the env-driven path: env params in, typed config out,
+ * which the main entry point wires into the exported functions.
+ *
+ * @returns The export configuration assembled from environment params.
+ */
+function configFromEnv() {
+    const tablePartitioning = optional(params.tablePartitioning.value());
+    return {
+        collectionPath: params.collectionPath.value(),
+        datasetId: params.datasetId.value(),
+        tableId: params.tableId.value(),
+        location: params.databaseRegion.value(),
+        datasetLocation: optional(params.datasetLocation.value()),
+        bqProjectId: optional(params.bigqueryProjectId.value()),
+        projectId: params_1.projectID.value(),
+        databaseId: optional(params.database.value()) || "(default)",
+        wildcardIds: params.wildcardIds.value(),
+        excludeOldData: params.excludeOldData.value(),
+        useNewSnapshotQuerySyntax: params.useNewSnapshotQuerySyntax.value(),
+        viewType: (optional(params.viewType.value()) || "view"),
+        partitioning: buildPartitioningConfig({
+            timePartitioning: timePartitioning(tablePartitioning),
+            timePartitioningField: params.timePartitioningField.value(),
+            timePartitioningFieldType: params.timePartitioningFieldType.value(),
+            timePartitioningFirestoreField: params.timePartitioningFirestoreField.value(),
+        }),
+        clustering: clustering(optional(params.clustering.value())),
+        maxStaleness: optional(params.maxStaleness.value()),
+        refreshIntervalMinutes: normalizePositiveInt(params.refreshIntervalMinutes.value()),
+        backupCollectionId: optional(params.backupCollection.value()),
+        transformFunction: optional(params.transformFunction.value()),
+        kmsKeyName: optional(params.kmsKeyName.value()),
+        logLevel: normalizeLogLevel(params.logLevel.value()),
+    };
+}
+//# sourceMappingURL=config.js.map

package/lib/config.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;;;;;GAcG;;;;;;AAOH,6GAAiF;AAEjF,sDAKmC;AAKnC,MAAM,aAAa,GAAG,EAAE,CAAC;AACzB,MAAM,wBAAwB,GAAG;IAC/B,aAAa;IACb,UAAU;IACV,iBAAiB;IACjB,UAAU;IACV,yBAAyB;IACzB,UAAU;IACV,UAAU;IACV,UAAU;IACV,oBAAoB;IACpB,UAAU;IACV,cAAc;IACd,eAAe;IACf,cAAc;IACd,cAAc;IACd,cAAc;IACd,cAAc;IACd,YAAY;IACZ,YAAY;IACZ,iBAAiB;IACjB,aAAa;IACb,iBAAiB;IACjB,iBAAiB;IACjB,iBAAiB;IACjB,sBAAsB;IACtB,iBAAiB;IACjB,IAAI;IACJ,IAAI;IACJ,eAAe;IACf,UAAU;IACV,aAAa;IACb,aAAa;IACb,eAAe;IACf,eAAe;IACf,cAAc;IACd,cAAc;IACd,mBAAmB;IACnB,eAAe;IACf,sBAAsB;IACtB,aAAa;IACb,yBAAyB;IACzB,oBAAoB;IACpB,qBAAqB;IACrB,WAAW;IACX,UAAU;CACF,CAAC;AACX,MAAM,uBAAuB,GAAG;IAC9B,MAAM;IACN,MAAM;IACN,MAAM;IACN,aAAa;IACb,UAAU;IACV,UAAU;IACV,UAAU;IACV,UAAU;IACV,UAAU;IACV,UAAU;IACV,UAAU;IACV,WAAW;IACX,yBAAyB;IACzB,yBAAyB;IACzB,qBAAqB;IACrB,oBAAoB;IACpB,oBAAoB;IACpB,cAAc;IACd,cAAc;IACd,cAAc;IACd,cAAc;IACd,cAAc;IACd,cAAc;IACd,cAAc;IACd,eAAe;IACf,eAAe;IACf,mBAAmB;IACnB,eAAe;IACf,eAAe;IACf,iBAAiB;IACjB,aAAa;IACb,aAAa;IACb,UAAU;IACV,aAAa;IACb,aAAa;IACb,iBAAiB;IACjB,iBAAiB;IACjB,YAAY;IACZ,YAAY;IACZ,iBAAiB;IACjB,iBAAiB;IACjB,iBAAiB;IACjB,sBAAsB;IACtB,sBAAsB;IACtB,eAAe;CACP,CAAC;AACX,MAAM,0BAA0B,GAAG;IACjC,MAAM;IACN,KAAK;IACL,OAAO;IACP,MAAM;IACN,MAAM;CACE,CAAC;AACX,MAAM,oCAAoC,GAAG;IAC3C,WAAW;IACX,UAAU;IACV,MAAM;IACN,MAAM;CACE,CAAC;AACX,MAAM,iBAAiB,GAAG;IACxB,MAAM;IACN,0BAA0B;IAC1B,8BAA8B;CACtB,CAAC;AACX,MAAM,iBAAiB,GAAG,CAAC,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,QAAQ,CAAU,CAAC;AAShF;;;;;GAKG;AACH,MAAM,MAAM,GAAG;IACb,iBAAiB,EAAE,IAAA,qBAAY,EAAC,qBAAqB,EAAE;QACrD,OAAO,EAAE,kBAAS;KACnB,CAAC;IACF,QAAQ,EAAE,IAAA,qBAAY,EAAC,UAAU,EAAE,EAAE,OAAO,EAAE,WAAW,EAAE,CAAC;IAC5D,cAAc,EAAE,IAAA,qBAAY,EAAC,iBAAiB,EAAE;QAC9C,KAAK,EAAE,IAAA,eAAM,EAAC,CAAC,GAAG,uBAAuB,CAAC,CAAC;KAC5C,CAAC;IACF,cAAc,EAAE,IAAA,qBAAY,EAAC,iBAAiB,EAAE,EAAE,OAAO,EAAE,OAAO,EAAE,CAAC;IACrE,SAAS,EAAE,IAAA,qBAAY,EAAC,YAAY,EAAE,EAAE,OAAO,EAAE,kBAAkB,EAAE,CAAC;IACtE,OAAO,EAAE,IAAA,qBAAY,EAAC,UAAU,EAAE,EAAE,OAAO,EAAE,OAAO,EAAE,CAAC;IACvD,eAAe,EAAE,IAAA,qBAAY,EAAC,kBAAkB,EAAE;QAChD,OAAO,EAAE,IAAI;QACb,KAAK,EAAE,IAAA,eAAM,EAAC,CAAC,GAAG,wBAAwB,CAAC,CAAC;KAC7C,CAAC;IACF,gBAAgB,EAAE,IAAA,qBAAY,EAAC,mBAAmB,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;IACpE,iBAAiB,EAAE,IAAA,qBAAY,EAAC,oBAAoB,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;IACtE,iBAAiB,EAAE,IAAA,qBAAY,EAAC,oBAAoB,EAAE;QACpD,OAAO,EAAE,MAAM;QACf,KAAK,EAAE,IAAA,eAAM,EAAC,CAAC,GAAG,0BAA0B,CAAC,CAAC;KAC/C,CAAC;IACF,qBAAqB,EAAE,IAAA,qBAAY,EAAC,yBAAyB,EAAE;QAC7D,OAAO,EAAE,EAAE;KACZ,CAAC;IACF,yBAAyB,EAAE,IAAA,qBAAY,EAAC,8BAA8B,EAAE;QACtE,OAAO,EAAE,MAAM;QACf,KAAK,EAAE,IAAA,eAAM,EAAC,CAAC,GAAG,oCAAoC,CAAC,CAAC;KACzD,CAAC;IACF,8BAA8B,EAAE,IAAA,qBAAY,EAC1C,mCAAmC,EACnC,EAAE,OAAO,EAAE,EAAE,EAAE,CAChB;IACD,UAAU,EAAE,IAAA,qBAAY,EAAC,YAAY,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;IACvD,WAAW,EAAE,IAAA,sBAAa,EAAC,cAAc,EAAE;QACzC,OAAO,EAAE,KAAK;KACf,CAAC;IACF,yBAAyB,EAAE,IAAA,sBAAa,EAAC,+BAA+B,EAAE;QACxE,OAAO,EAAE,KAAK;KACf,CAAC;IACF,cAAc,EAAE,IAAA,sBAAa,EAAC,kBAAkB,EAAE;QAChD,OAAO,EAAE,KAAK;KACf,CAAC;IACF,QAAQ,EAAE,IAAA,qBAAY,EAAC,WAAW,EAAE;QAClC,OAAO,EAAE,MAAM;QACf,KAAK,EAAE,IAAA,eAAM,EAAC,CAAC,GAAG,iBAAiB,CAAC,CAAC;KACtC,CAAC;IACF,YAAY,EAAE,IAAA,qBAAY,EAAC,eAAe,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;IAC5D,sBAAsB,EAAE,IAAA,qBAAY,EAAC,0BAA0B,EAAE;QAC/D,OAAO,EAAE,EAAE;KACZ,CAAC;IACF,UAAU,EAAE,IAAA,qBAAY,EAAC,cAAc,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC;IACzD,QAAQ,EAAE,IAAA,qBAAY,EAAC,WAAW,EAAE;QAClC,OAAO,EAAE,MAAM;QACf,KAAK,EAAE,IAAA,eAAM,EAAC,CAAC,GAAG,iBAAiB,CAAC,CAAC;KACtC,CAAC;CACH,CAAC;AAEW,QAAA,kBAAkB,GAAsB;IACnD,cAAc,EAAE,MAAM,CAAC,cAAc;IACrC,SAAS,EAAE,MAAM,CAAC,SAAS;IAC3B,OAAO,EAAE,MAAM,CAAC,OAAO;IACvB,QAAQ,EAAE,MAAM,CAAC,cAAc;IAC/B,QAAQ,EAAE,MAAM,CAAC,QAAQ;CAC1B,CAAC;AAEF,SAAS,gBAAgB,CACvB,IAAwB;IAExB,IACE,IAAI,KAAK,MAAM;QACf,IAAI,KAAK,KAAK;QACd,IAAI,KAAK,OAAO;QAChB,IAAI,KAAK,MAAM,EACf,CAAC;QACD,OAAO,IAAI,CAAC;IACd,CAAC;IAED,OAAO,IAAI,CAAC;AACd,CAAC;AAED,oBAA2B,QAA4B;IACrD,OAAO,QAAQ,CAAC,CAAC,CAAC,QAAQ,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;AAC3D,CAAC;AAED,SAAS,+BAA+B,CACtC,KAAyB;IAEzB,MAAM,UAAU,GAAG,KAAK,EAAE,IAAI,EAAE,CAAC;IAEjC,IAAI,CAAC,UAAU,IAAI,UAAU,KAAK,MAAM,IAAI,UAAU,KAAK,MAAM,EAAE,CAAC;QAClE,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,OAAO,UAAU,CAAC;AACpB,CAAC;AAED,SAAS,2BAA2B,CAClC,KAAyB;IAEzB,MAAM,UAAU,GAAG,+BAA+B,CAAC,KAAK,CAAC,CAAC;IAC1D,IACE,UAAU,KAAK,WAAW;QAC1B,UAAU,KAAK,MAAM;QACrB,UAAU,KAAK,UAAU,EACzB,CAAC;QACD,OAAO,UAAU,CAAC;IACpB,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC;AAED,iCAAwC,MAKvC;IACC,MAAM,EAAE,gBAAgB,EAAE,GAAG,MAAM,CAAC;IACpC,MAAM,YAAY,GAAG,MAAM,CAAC,qBAAqB,EAAE,IAAI,EAAE,CAAC;IAC1D,MAAM,YAAY,GAAG,MAAM,CAAC,yBAAyB,EAAE,IAAI,EAAE,CAAC;IAC9D,MAAM,iBAAiB,GAAG,MAAM,CAAC,8BAA8B,EAAE,IAAI,EAAE,CAAC;IAExE,MAAM,WAAW,GAAG,CAAC,KAAyB,EAAU,EAAE,CACxD,KAAK,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,KAAK,GAAG,CAAC,CAAC,CAAC,SAAS,CAAC;IAEvD,MAAM,8BAA8B,GAAG,CAAC,MAAc,EAAS,EAAE;QAC/D,MAAM,IAAI,KAAK,CACb;YACE,mEAAmE;YACnE,MAAM;YACN,+BAA+B,WAAW,CACxC,gBAAgB,IAAI,SAAS,CAC9B,GAAG;YACJ,2BAA2B,WAAW,CAAC,YAAY,CAAC,GAAG;YACvD,qCAAqC,WAAW,CAAC,iBAAiB,CAAC,GAAG;YACtE,gCAAgC,WAAW,CAAC,YAAY,CAAC,GAAG;YAC5D,yBAAyB;YACzB,+FAA+F;YAC/F,yHAAyH;YACzH,uJAAuJ;SACxJ,CAAC,IAAI,CAAC,GAAG,CAAC,CACZ,CAAC;IACJ,CAAC,CAAC;IAEF,MAAM,SAAS,GAAG,+BAA+B,CAC/C,MAAM,CAAC,qBAAqB,CAC7B,CAAC;IACF,MAAM,SAAS,GAAG,2BAA2B,CAC3C,MAAM,CAAC,yBAAyB,CACjC,CAAC;IACF,MAAM,cAAc,GAAG,+BAA+B,CACpD,MAAM,CAAC,8BAA8B,CACtC,CAAC;IAEF,IAAI,CAAC,gBAAgB,EAAE,CAAC;QACtB,IAAI,SAAS,IAAI,SAAS,IAAI,cAAc,EAAE,CAAC;YAC7C,OAAO,8BAA8B,CACnC,+EAA+E,CAChF,CAAC;QACJ,CAAC;QACD,OAAO,EAAE,WAAW,EAAE,MAAM,EAAE,CAAC;IACjC,CAAC;IAED,IAAI,CAAC,SAAS,IAAI,CAAC,cAAc,EAAE,CAAC;QAClC,OAAO,EAAE,WAAW,EAAE,gBAAgB,EAAE,CAAC;IAC3C,CAAC;IAED,IAAI,SAAS,KAAK,WAAW,IAAI,CAAC,cAAc,EAAE,CAAC;QACjD,OAAO;YACL,WAAW,EAAE,gBAAgB;YAC7B,kBAAkB,EAAE,WAAW;YAC/B,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC,EAAE,kBAAkB,EAAE,SAAS,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;SACxD,CAAC;IACJ,CAAC;IAED,IAAI,SAAS,IAAI,cAAc,IAAI,SAAS,EAAE,CAAC;QAC7C,OAAO;YACL,WAAW,EAAE,gBAAgB;YAC7B,kBAAkB,EAAE,SAAS;YAC7B,kBAAkB,EAAE,SAAS;YAC7B,kBAAkB,EAAE,cAAc;SACnC,CAAC;IACJ,CAAC;IAED,OAAO,8BAA8B,CACnC,uFAAuF,CACxF,CAAC;AACJ,CAAC;AAED,SAAS,iBAAiB,CAAC,KAAyB;IAClD,QAAQ,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,WAAW,EAAE,EAAE,CAAC;QACpC,KAAK,OAAO;YACV,OAAO,OAAO,CAAC;QACjB,KAAK,MAAM;YACT,OAAO,MAAM,CAAC;QAChB,KAAK,MAAM;YACT,OAAO,MAAM,CAAC;QAChB,KAAK,OAAO;YACV,OAAO,OAAO,CAAC;QACjB,KAAK,QAAQ;YACX,OAAO,QAAQ,CAAC;QAClB;YACE,OAAO,4CAAQ,CAAC,IAAI,CAAC;IACzB,CAAC;AACH,CAAC;AAED,SAAS,oBAAoB,CAAC,KAAa;IACzC,MAAM,UAAU,GAAG,MAAM,CAAC,QAAQ,CAAC,KAAK,EAAE,aAAa,CAAC,CAAC;IACzD,OAAO,UAAU,GAAG,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,SAAS,CAAC;AACjD,CAAC;AAED,yDAAyD;AACzD,SAAS,QAAQ,CAAC,KAAa;IAC7B,OAAO,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS,CAAC;AAC9C,CAAC;AAED;;;;;;;;;;GAUG;AACH;IACE,MAAM,iBAAiB,GAAG,QAAQ,CAAC,MAAM,CAAC,iBAAiB,CAAC,KAAK,EAAE,CAAC,CAAC;IAErE,OAAO;QACL,cAAc,EAAE,MAAM,CAAC,cAAc,CAAC,KAAK,EAAE;QAC7C,SAAS,EAAE,MAAM,CAAC,SAAS,CAAC,KAAK,EAAE;QACnC,OAAO,EAAE,MAAM,CAAC,OAAO,CAAC,KAAK,EAAE;QAC/B,QAAQ,EAAE,MAAM,CAAC,cAAc,CAAC,KAAK,EAAE;QACvC,eAAe,EAAE,QAAQ,CAAC,MAAM,CAAC,eAAe,CAAC,KAAK,EAAE,CAAC;QACzD,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,iBAAiB,CAAC,KAAK,EAAE,CAAC;QACvD,SAAS,EAAE,kBAAS,CAAC,KAAK,EAAE;QAC5B,UAAU,EAAE,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,KAAK,EAAE,CAAC,IAAI,WAAW;QAC5D,WAAW,EAAE,MAAM,CAAC,WAAW,CAAC,KAAK,EAAE;QACvC,cAAc,EAAE,MAAM,CAAC,cAAc,CAAC,KAAK,EAAE;QAC7C,yBAAyB,EAAE,MAAM,CAAC,yBAAyB,CAAC,KAAK,EAAE;QACnE,QAAQ,EAAE,CAAC,QAAQ,CAAC,MAAM,CAAC,QAAQ,CAAC,KAAK,EAAE,CAAC,IAAI,MAAM,CAAa;QACnE,YAAY,EAAE,uBAAuB,CAAC;YACpC,gBAAgB,EAAE,gBAAgB,CAAC,iBAAiB,CAAC;YACrD,qBAAqB,EAAE,MAAM,CAAC,qBAAqB,CAAC,KAAK,EAAE;YAC3D,yBAAyB,EAAE,MAAM,CAAC,yBAAyB,CAAC,KAAK,EAAE;YACnE,8BAA8B,EAC5B,MAAM,CAAC,8BAA8B,CAAC,KAAK,EAAE;SAChD,CAAC;QACF,UAAU,EAAE,UAAU,CAAC,QAAQ,CAAC,MAAM,CAAC,UAAU,CAAC,KAAK,EAAE,CAAC,CAAC;QAC3D,YAAY,EAAE,QAAQ,CAAC,MAAM,CAAC,YAAY,CAAC,KAAK,EAAE,CAAC;QACnD,sBAAsB,EAAE,oBAAoB,CAC1C,MAAM,CAAC,sBAAsB,CAAC,KAAK,EAAE,CACtC;QACD,kBAAkB,EAAE,QAAQ,CAAC,MAAM,CAAC,gBAAgB,CAAC,KAAK,EAAE,CAAC;QAC7D,iBAAiB,EAAE,QAAQ,CAAC,MAAM,CAAC,iBAAiB,CAAC,KAAK,EAAE,CAAC;QAC7D,UAAU,EAAE,QAAQ,CAAC,MAAM,CAAC,UAAU,CAAC,KAAK,EAAE,CAAC;QAC/C,QAAQ,EAAE,iBAAiB,CAAC,MAAM,CAAC,QAAQ,CAAC,KAAK,EAAE,CAAC;KACrD,CAAC;AACJ,CAAC"}

package/lib/events.d.ts

@@ -0,0 +1,45 @@
+/**
+ * Sets up the Eventarc channel.
+ *
+ * This function retrieves the Eventarc channel based on the environment variables:
+ * - `EVENTARC_CHANNEL` specifies the channel to use for publishing events.
+ * - `EXT_SELECTED_EVENTS` defines the allowed event types.
+ *
+ * @function setupEventChannel
+ */
+export declare const setupEventChannel: () => void;
+/**
+ * Publishes a "start" event using both OLD and NEW event types.
+ *
+ * @param data The payload to send with the event. Can be a string or an object.
+ * @returns A Promise resolving when both events are published.
+ */
+export declare const recordStartEvent: (data: string | object) => Promise<void | void[]>;
+/**
+ * Publishes an "error" event using both OLD and NEW event types.
+ *
+ * @param err The Error object containing the error message.
+ * @param subject (Optional) Subject identifier related to the error event.
+ * @returns A Promise resolving when both events are published.
+ */
+export declare const recordErrorEvent: (err: Error, subject?: string) => Promise<void | void[]>;
+/**
+ * Publishes a "success" event using both OLD and NEW event types.
+ *
+ * @param params An object containing the subject and the event data.
+ * @param params.subject A string representing the subject of the event.
+ * @param params.data The payload to send with the event.
+ * @returns A Promise resolving when both events are published.
+ */
+export declare const recordSuccessEvent: ({ subject, data, }: {
+    subject: string;
+    data: string | object;
+}) => Promise<void | void[]>;
+/**
+ * Publishes a "completion" event using both OLD and NEW event types.
+ *
+ * @param data The payload to send with the event. Can be a string or an object.
+ * @returns A Promise resolving when both events are published.
+ */
+export declare const recordCompletionEvent: (data: string | object) => Promise<void | void[]>;
+//# sourceMappingURL=events.d.ts.map