ga4-export-fixer 0.9.0-dev.2 → 0.9.0-dev.4

package/README.md

@@ -314,7 +314,7 @@ All fields are optional except `sourceTable`. Default values are applied automat
 
 | Field                  | Type                    | Default/Required                   | Description                                                                                                                                                                                                                                                                                                                  |
 | ---------------------- | ----------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `sourceTable`          | Dataform ref() / string | **required**                       | Source GA4 export table. Use `ref()` in Dataform or a string in format ``project.dataset.table``                                                                                                                                                                                                                             |
+| `sourceTable`          | Dataform ref / object / string | **required**                       | Source GA4 export table. Inside an SQLX `js { }` block use `ref(...)`. From a `.js` definition file use a `{ schema, name }` ref object (resolved later via `ctx.ref()`) or a backtick-quoted ``` `project.dataset.events_*` ``` string for an external table. |
 | `self`                 | Dataform self()         | **required for .SQLX deployment**  | Reference to the table itself. Use `self()` in Dataform                                                                                                                                                                                                                                                                      |
 | `incremental`          | Dataform incremental()  | **required for .SQLX deployment**  | Switch between incremental and full refresh logic. Use `incremental()` in Dataform                                                                                                                                                                                                                                           |
 | `dataformTableConfig`  | object                  | **In JS deployment only.** [See default](#default-dataformtableconfig) | Override the default Dataform table configuration for JS deployment. See: [ITableConfig reference](https://docs.cloud.google.com/dataform/docs/reference/dataform-core-reference#itableconfig) |
@@ -535,35 +535,35 @@ For typical use cases this is the right tool; reach for `customSteps` only when
 | --- | --- | --- | --- |
 | `name` | string | Yes | Used in the generated `enrich_<name>` CTE name. Unique within `enrichments`. |
 | `level` | `'event'` | No, defaults to `'event'` | Join grain. Currently only `'event'` is supported (item-level enrichments will arrive in a later release). |
-| `source` | Dataform `ref()` / string | Yes | Source dim table. Use `ref()` in Dataform or a backtick-quoted ``` `project.dataset.table` ``` string. |
+| `source` | Dataform ref / object / string | Yes | Source dim table. Inside an SQLX `js { }` block use `ref(...)`. From a `.js` definition file use a `{ schema, name }` ref object (resolved later via `ctx.ref()`) or a backtick-quoted ``` `project.dataset.table` ``` string for an external table. |
 | `joinKey` | string / string[] | Yes | Column name(s) on `enhanced_events` to join on. Composite keys (array) compile to `USING(col1, col2, ...)`. |
 | `columns` | string[] | Yes | Source columns to add to the output (excluding `joinKey`). Names matching existing columns REPLACE them. |
 | `dedupe` | boolean | No, defaults to `false` | When `true`, wraps the source CTE in `qualify row_number() over (partition by <joinKey>) = 1` for non-unique-key dim sources. Non-deterministic which row wins; for strict needs, pre-aggregate in source SQL. |
 
 **Replace-or-add semantics.** If an enrichment column name matches an existing column on `enhanced_events` (a column promoted via `eventParamsToColumns`, a package-generated column, or a default GA4 column from the export), the enrichment value REPLACES it. If there is no overlap, the column is added.
 
-**Example** — attach user cohort labels by `user_pseudo_id`:
+**Example** — attach user cohort labels by `user_pseudo_id` (Dataform-declared table referenced by `{ schema, name }`):
 
 ```javascript
 enrichments: [
     {
         name: 'cohorts',
         level: 'event',
-        source: ctx.ref('user_cohorts'),
+        source: { schema: 'analytics', name: 'user_cohorts' },
         joinKey: 'user_pseudo_id',
         columns: ['cohort_label', 'lifecycle_stage'],
     },
 ],
 ```
 
-**Example** — composite key (date + user) for daily-varying dim data, with dedupe safety net:
+**Example** — composite key (date + user) for daily-varying dim data, with dedupe safety net (external table referenced by backtick-FQN):
 
 ```javascript
 enrichments: [
     {
         name: 'segments',
         level: 'event',
-        source: ctx.ref('daily_user_segments'),
+        source: '`my-project.analytics.daily_user_segments`',
         joinKey: ['event_date', 'user_pseudo_id'],
         columns: ['segment'],
         dedupe: true,
@@ -580,7 +580,7 @@ enrichments: [
         {
             name: 'titles',
             level: 'event',
-            source: ctx.ref('page_title_overrides'),
+            source: { schema: 'analytics', name: 'page_title_overrides' },
             joinKey: 'page_location',
             columns: ['page_title'],   // overlaps the promoted column → replaces it
         },

package/helpers/ga4Transforms.js

@@ -95,50 +95,50 @@ const isFinalData = (detectionMethod, dayThreshold) => {
 };
 
 /**
- * Checks whether a given column name is part of the standard/expected GA4 BigQuery export columns.
+ * The standard GA4 BigQuery export top-level column names, based on the official schema.
  *
- * The list of recognized GA4 export columns is based on the official schema as of 2026-02-18.
- * This function can be used to filter or validate column names when processing GA4 data exports.
+ * list updated 2026-02-18
+ */
+const ga4ExportColumns = [
+  "event_date",
+  "event_timestamp",
+  "event_name",
+  "event_params",
+  "event_previous_timestamp",
+  "event_value_in_usd",
+  "event_bundle_sequence_id",
+  "event_server_timestamp_offset",
+  "user_id",
+  "user_pseudo_id",
+  "privacy_info",
+  "user_properties",
+  "user_first_touch_timestamp",
+  "user_ltv",
+  "device",
+  "geo",
+  "app_info",
+  "traffic_source",
+  "stream_id",
+  "platform",
+  "event_dimensions",
+  "ecommerce",
+  "items",
+  "collected_traffic_source",
+  "is_active_user",
+  "batch_event_index",
+  "batch_page_id",
+  "batch_ordering_id",
+  "session_traffic_source_last_click",
+  "publisher"
+];
+
+/**
+ * Checks whether a given column name is part of the standard GA4 BigQuery export columns.
  *
  * @param {string} columnName - The name of the column to check.
  * @returns {boolean} True if the column name is a GA4 export column, otherwise false.
  */
-const isGa4ExportColumn = (columnName) => {
-  // list updated 2026-02-18
-  const ga4ExportColumns = [
-    "event_date",
-    "event_timestamp",
-    "event_name",
-    "event_params",
-    "event_previous_timestamp",
-    "event_value_in_usd",
-    "event_bundle_sequence_id",
-    "event_server_timestamp_offset",
-    "user_id",
-    "user_pseudo_id",
-    "privacy_info",
-    "user_properties",
-    "user_first_touch_timestamp",
-    "user_ltv",
-    "device",
-    "geo",
-    "app_info",
-    "traffic_source",
-    "stream_id",
-    "platform",
-    "event_dimensions",
-    "ecommerce",
-    "items",
-    "collected_traffic_source",
-    "is_active_user",
-    "batch_event_index",
-    "batch_page_id",
-    "batch_ordering_id",
-    "session_traffic_source_last_click",
-    "publisher"
-  ];
-  return ga4ExportColumns.includes(columnName);
-};
+const isGa4ExportColumn = (columnName) => ga4ExportColumns.includes(columnName);
 
 /**
  * Generates a SQL CASE expression that determines the GA4 export type from a table suffix.
@@ -255,6 +255,7 @@ module.exports = {
   sessionId,
   fixEcommerceStruct,
   isFinalData,
+  ga4ExportColumns,
   isGa4ExportColumn,
   getGa4ExportType,
   itemListAttributionExpr,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ga4-export-fixer",
-  "version": "0.9.0-dev.2",
+  "version": "0.9.0-dev.4",
   "description": "",
   "main": "index.js",
   "files": [
@@ -17,7 +17,7 @@
     "createTable.js"
   ],
   "scripts": {
-    "test": "node tests/ga4EventsEnhanced.test.js && node tests/assertions.test.js && node tests/mergeSQLConfigurations.test.js && node tests/preOperations.test.js && node tests/documentation.test.js && node tests/inputValidation.test.js && node tests/createTable.test.js && node tests/queryBuilder.test.js && node tests/customSteps.test.js && node tests/enrichments.test.js",
+    "test": "node tests/ga4EventsEnhanced.test.js && node tests/assertions.test.js && node tests/mergeSQLConfigurations.test.js && node tests/preOperations.test.js && node tests/documentation.test.js && node tests/inputValidation.test.js && node tests/createTable.test.js && node tests/queryBuilder.test.js && node tests/customSteps.test.js && node tests/enrichments.test.js && node tests/eventDataColumns.test.js && node tests/utils.test.js",
     "test:summary": "node tests/testRunner.js",
     "test:docs": "node tests/documentation.test.js",
     "test:preops": "node tests/preOperations.test.js",
@@ -29,6 +29,8 @@
     "test:queryBuilder": "node tests/queryBuilder.test.js",
     "test:customSteps": "node tests/customSteps.test.js",
     "test:enrichments": "node tests/enrichments.test.js",
+    "test:eventDataColumns": "node tests/eventDataColumns.test.js",
+    "test:utils": "node tests/utils.test.js",
     "test:integration": "node tests/integration/integration.test.js",
     "release:dev": "./scripts/release-dev.sh",
     "readme": "node scripts/updateReadme.js",

package/tables/ga4EventsEnhanced/index.js

@@ -197,51 +197,56 @@ const _generateEnhancedEventsSQL = (mergedConfig) => {
         return excludedColumns;
     };
 
-    // initial step: extract data from the export tables
+    // initial step: extract data from the export tables.
+    // Explicit columns first (transforms + package-promoted + user-excluded sentinels);
+    // then pass-through entries for every GA4 export column not already accounted for.
+    // After this, Object.keys(eventDataStep.select.columns) is the complete column set of event_data.
+    const eventDataExplicitColumns = {
+        // exclude default export columns that are not needed
+        // do this first so that the columns defined later are not excluded
+        ...getExcludedColumns(),
+        // date and time
+        event_date: helpers.eventDate,
+        event_datetime: `extract(datetime from timestamp_micros(${helpers.getEventTimestampMicros(mergedConfig.customTimestampParam)}) at time zone '${mergedConfig.timezone}')`,
+        event_timestamp: 'event_timestamp',
+        event_custom_timestamp: mergedConfig.customTimestampParam ? helpers.getEventTimestampMicros(mergedConfig.customTimestampParam) : undefined,
+        // event name
+        event_name: 'event_name',
+        // identifiers
+        session_id: helpers.sessionId,
+        user_pseudo_id: 'user_pseudo_id',
+        user_id: 'user_id',
+        // page
+        page_location: helpers.unnestEventParam('page_location', 'string'),
+        page: helpers.extractPageDetails(),
+        // event parameters and user properties
+        ...promotedEventParameters(),
+        event_params: helpers.filterEventParams(mergedConfig.excludedEventParams, 'exclude'),
+        user_properties: 'user_properties',
+        // traffic source
+        collected_traffic_source: 'collected_traffic_source',
+        session_traffic_source_last_click: 'session_traffic_source_last_click',
+        user_traffic_source: 'traffic_source',
+        // ecommerce
+        ecommerce: helpers.fixEcommerceStruct('ecommerce'),
+        items: 'items',
+        _item_row_id: itemListAttribution ? helpers.itemRowId(ecommerceEventsFilter) : undefined,
+        // flag if the data is "final" and is not expected to change anymore
+        data_is_final: helpers.isFinalData(mergedConfig.dataIsFinal.detectionMethod, mergedConfig.dataIsFinal.dayThreshold),
+        export_type: helpers.getGa4ExportType('_table_suffix'),
+        // prep columns for later steps
+        entrances: helpers.unnestEventParam('entrances', 'int'),
+        session_params_prep: mergedConfig.sessionParams.length > 0 ? helpers.filterEventParams(mergedConfig.sessionParams, 'include') : undefined,
+    };
+    // Pass through every GA4 export column not already covered by an explicit transform,
+    // promotion, exclusion sentinel, or value-side rename in eventDataExplicitColumns.
+    const eventDataPassThroughs = utils.buildPassThroughs(eventDataExplicitColumns, helpers.ga4ExportColumns);
     const eventDataStep = {
         name: 'event_data',
         select: {
             columns: {
-                // exclude default export columns that are not needed
-                // do this first so that the columns defined later are not excluded
-                ...getExcludedColumns(),
-                // date and time
-                event_date: helpers.eventDate,
-                event_datetime: `extract(datetime from timestamp_micros(${helpers.getEventTimestampMicros(mergedConfig.customTimestampParam)}) at time zone '${mergedConfig.timezone}')`,
-                event_timestamp: 'event_timestamp',
-                event_custom_timestamp: mergedConfig.customTimestampParam ? helpers.getEventTimestampMicros(mergedConfig.customTimestampParam) : undefined,
-                // event name
-                event_name: 'event_name',
-                // identifiers
-                session_id: helpers.sessionId,
-                user_pseudo_id: 'user_pseudo_id',
-                user_id: 'user_id',
-                // page
-                page_location: helpers.unnestEventParam('page_location', 'string'),
-                page: helpers.extractPageDetails(),
-                // event parameters and user properties
-                ...promotedEventParameters(),
-                event_params: helpers.filterEventParams(mergedConfig.excludedEventParams, 'exclude'),
-                user_properties: 'user_properties',
-                // traffic source
-                collected_traffic_source: 'collected_traffic_source',
-                session_traffic_source_last_click: 'session_traffic_source_last_click',
-                user_traffic_source: 'traffic_source',
-                // ecommerce
-                ecommerce: helpers.fixEcommerceStruct('ecommerce'),
-                items: 'items',
-                _item_row_id: itemListAttribution ? helpers.itemRowId(ecommerceEventsFilter) : undefined,
-                // flag if the data is "final" and is not expected to change anymore
-                data_is_final: helpers.isFinalData(mergedConfig.dataIsFinal.detectionMethod, mergedConfig.dataIsFinal.dayThreshold),
-                export_type: helpers.getGa4ExportType('_table_suffix'),
-                // prep columns for later steps
-                entrances: helpers.unnestEventParam('entrances', 'int'),
-                session_params_prep: mergedConfig.sessionParams.length > 0 ? helpers.filterEventParams(mergedConfig.sessionParams, 'include') : undefined,
-                // include all other columns from the export data
-                get '[sql]other_columns'() {
-                    const definedColumns = Object.keys(this);
-                    return `* except (${definedColumns.filter(column => helpers.isGa4ExportColumn(column)).join(', ')})`;
-                },
+                ...eventDataExplicitColumns,
+                ...eventDataPassThroughs,
             },
         },
         from: mergedConfig.sourceTable,
@@ -385,6 +390,15 @@ ${excludedEventsSQL}`,
     }
     const enrichmentExcludedColumns = [...enrichmentColumnNames];
 
+    // Only forward enrichment columns to each wildcard's EXCEPT input if they actually exist
+    // in that wildcard's source CTE. Otherwise BigQuery rejects with "Column X in SELECT *
+    // EXCEPT list does not exist". After M1, Object.keys(step.select.columns) is the complete
+    // column set of both event_data and session_data — so the same predicate works for both.
+    const eventDataExplicit = new Set(Object.keys(eventDataStep.select.columns));
+    const sessionDataExplicit = new Set(Object.keys(sessionDataStep.select.columns));
+    const eventDataEnrichmentExcept = enrichmentExcludedColumns.filter(c => eventDataExplicit.has(c));
+    const sessionDataEnrichmentExcept = enrichmentExcludedColumns.filter(c => sessionDataExplicit.has(c));
+
     // Join event_data and session_data, include additional logic
     // Named 'enhanced_events' so user-supplied customSteps can reference it as a stable handle.
     const enhancedEventsStep = {
@@ -407,14 +421,14 @@ ${excludedEventsSQL}`,
                         'data_is_final',
                         'export_type',
                         ...itemListExcludedColumns,
-                        ...enrichmentExcludedColumns,
+                        ...eventDataEnrichmentExcept,
                     ]
                 ),
                 // get the rest of the session_data columns
                 '[sql]session_data': utils.selectOtherColumns(
                     sessionDataStep,
                     Object.keys(finalColumnOrder),
-                    [...enrichmentExcludedColumns],
+                    sessionDataEnrichmentExcept,
                 ),
                 // include additional columns
                 row_inserted_timestamp: 'current_timestamp()',

package/utils.js

@@ -521,6 +521,45 @@ const selectOtherColumns = (step, alreadyDefinedColumns = [], excludedColumns =
 };
 
 
+/**
+ * Builds a queryBuilder `select.columns` fragment that passes through every source column
+ * not already covered by an explicit columns object.
+ *
+ * A source column is considered "covered" — and skipped from pass-throughs — when it appears as:
+ *   - a KEY in `explicitColumns` (a transform, package promotion, or undefined-valued exclusion
+ *     sentinel like `{ event_dimensions: undefined }`), OR
+ *   - a VALUE in `explicitColumns` (a bare source-column identifier referenced by a value-side
+ *     rename, e.g. `{ user_traffic_source: 'traffic_source' }` covers 'traffic_source').
+ *
+ * Values that are SQL expressions, function calls, or non-strings never count as coverage —
+ * they reference the source column internally but the column itself is still available as a
+ * pass-through. (`.includes()` compares by strict equality, so 'extract(datetime from ...)'
+ * never matches a bare column name.)
+ *
+ * @param {Object} explicitColumns - A queryBuilder step's explicit `select.columns` entries.
+ * @param {Iterable<string>} sourceColumns - Column names available on the source schema.
+ * @returns {Object} A map of `{ column: column }` entries for every source column not covered.
+ *
+ * @example
+ *   buildPassThroughs(
+ *     { event_name: 'event_name', user_traffic_source: 'traffic_source' },
+ *     ['event_name', 'traffic_source', 'device', 'geo']
+ *   );
+ *   // → { device: 'device', geo: 'geo' }
+ */
+const buildPassThroughs = (explicitColumns, sourceColumns) => {
+    const explicitKeys = Object.keys(explicitColumns);
+    const explicitValues = Object.values(explicitColumns);
+    const passThroughs = {};
+    for (const column of sourceColumns) {
+        if (!explicitKeys.includes(column) && !explicitValues.includes(column)) {
+            passThroughs[column] = column;
+        }
+    }
+    return passThroughs;
+};
+
+
 /**
  * Processes a date input string and returns a corresponding SQL date casting expression,
  * or passes through BigQuery SQL statements as-is.
@@ -596,6 +635,7 @@ module.exports = {
     isDataformTableReferenceObject,
     setDataformContext,
     selectOtherColumns,
+    buildPassThroughs,
     processDate,
     getDatasetName
 };