ga4-export-fixer 0.2.3-dev.3 → 0.2.4-dev.0

package/README.md

@@ -1,26 +1,54 @@
 # ga4-export-fixer
 
-**ga4-export-fixer** is a **Dataform NPM package** that transforms raw GA4 BigQuery export data into a cleaner, more queryable incremental table. It combines daily and intraday exports so the best available version of each event is always in use, adds session-level fields like `session_id` and `landing_page`, promotes key event parameters to columns, and fixes known GA4 export issues — handling the boilerplate transformations that are otherwise tedious to include in every GA4 query.
+[![npm version](https://img.shields.io/npm/v/ga4-export-fixer)](https://www.npmjs.com/package/ga4-export-fixer)
+
+**ga4-export-fixer** is a **Dataform NPM package** that transforms raw GA4 BigQuery export data into a cleaner, more queryable incremental table. It combines daily and intraday exports (360 fresh export support not yet available) so the best available version of each event is always in use, adds session-level fields like `session_id` and `landing_page`, promotes key event parameters to columns, and fixes known GA4 export issues — handling the boilerplate transformations that are otherwise tedious to include in every GA4 query.
 
 The goal of the package is to **speed up development** when building data models and pipelines on top of GA4 export data, allowing you to focus on your use case instead of wrestling with the raw export format.
 
+<img src="./docs/images/example_data_model.png" alt="Example Data Model" width="600">
+
+*Example data model built with ga4-export-fixer*
+
 ### Table of Contents
 <!-- TOC -->
+  - [Main Features](#main-features)
   - [Planned, Upcoming Features](#planned-upcoming-features)
 - [Installation](#installation)
   - [Bash](#bash)
   - [In Google Cloud Dataform](#in-google-cloud-dataform)
 - [Usage](#usage)
   - [Create GA4 Events Enhanced Table](#create-ga4-events-enhanced-table)
-  - [Building on top of the ga4_events_enhanced table](#building-on-top-of-the-ga4_events_enhanced-table)
   - [Configuration Object](#configuration-object)
+  - [Creating Incremental Downstream Tables from ga4_events_enhanced](#creating-incremental-downstream-tables-from-ga4_events_enhanced)
   - [Helpers](#helpers)
 - [License](#license)
 <!-- /TOC -->
 
+### Main Features
+
+The **ga4_events_enhanced** table comes with features such as these:
+
+- **Best available data at any time** – Combines daily (processed) and intraday exports so the most complete, accurate version of the data is always available
+- **Robust incremental updates** – Run on any schedule (daily, hourly, or custom)
+- **Flexible schema, better optimized for building data models** – Keeps the flexible structure of the original export while promoting key fields (e.g. `page_location`, `session_id`) to columns for better query performance; **partitioning and clustering** enabled
+- **Session-level identity resolution** – `user_id` resolved to the last authenticated value per session; `merged_user_id` coalesces it with `user_pseudo_id`
+- **Session traffic sources** – `session_first_traffic_source` and session-scoped `session_traffic_source_last_click` (adjusting for sessions that span midnight) computed automatically
+- **Landing page detection** – `landing_page` derived per session from the first page where `entrances > 0`
+- **Page URL parsing** – `page` struct with parsed `hostname`, `path`, `query`, and `query_params` from `page_location`
+- **Ecommerce data fixes** – Nullifies `transaction_id` placeholder values and corrects `purchase_revenue` NaN / missing-value bugs
+- **Event parameter handling** – Promote event params to columns; include or exclude by name
+- **Session parameters** – Promote selected event parameters as session-level parameters
+- **Custom timestamp support** – Optionally use a custom event parameter as the primary timestamp, with automatic fallback to `event_timestamp`
+- **Schema lock** – Lock the table schema to a specific GA4 export date to prevent schema drift
+- **Data freshness tracking** – `data_is_final` flag and `export_type` label on every row
+- **Timezone-aware datetime** – `event_datetime` converted to a configurable IANA timezone
+- **Column descriptions** – Full column-level documentation included in the Dataform table configuration, reflecting the specific configuration used to build the table
+
 ### Planned, Upcoming Features
 
-- Column descriptions
+Features under consideration for future releases:
+
 - Web and app specific default configurations
 - Ecommerce item list attribution
 - Custom channel grouping
@@ -48,7 +76,7 @@ Include the package in the package.json file in your Dataform repository.
 {
   "dependencies": {
     "@dataform/core": "3.0.42",
-    "ga4-export-fixer": "0.2.2"
+    "ga4-export-fixer": "0.2.3"
   }
 }
 ```
@@ -65,14 +93,6 @@ If your Dataform repository does not have a package.json file, see this guide: [
 
 Creates an **enhanced** version of the GA4 BigQuery export (daily & intraday).
 
-The main features include:
-
-- **Best available data at any time** – Combines daily (processed) and intraday exports so the most complete, accurate version of the data is always available
-- **Robust incremental updates** – Run on any schedule (daily, hourly, or custom)
-- **Flexible schema, better optimized for building data models** – Keeps the flexible structure of the original export while promoting key fields (e.g. `page_location`, `session_id`) to columns for better query performance; **partitioning and clustering** enabled
-- **Event parameter handling** – Promote event params to columns; include or exclude by name
-- **Session parameters** – Promote selected event parameters as session-level parameters
-
 #### JS Deployment (Recommended)
 
 Create a new **ga4_events_enhanced** table using a **.js** file in your repository's **definitions** folder.
@@ -103,7 +123,7 @@ const config = {
   // use dataformTableConfig to make changes to the default Dataform table configuration
   dataformTableConfig: {
       schema: 'ga4'
-  }
+  },
   // test configurations
   test: false,
   testConfig: {
@@ -187,76 +207,6 @@ pre_operations {
 }
 ```
 
-### Building on top of the ga4_events_enhanced table
-
-Setting up incremental updates is easy using the **setPreOperations()** function. Just ensure that your result table includes the **data_is_final** flag from the **ga4_enhanced_events** table.
-
-The **incrementalDateFilter()** function applies the same date filtering used by **ga4_events_enhanced**, based on the **config** options and the variables declared by **setPreOperations()**.
-
-Key fields such as session_id, user_id, and session_traffic_source_last_click are available as clean, sessionized versions that handle edge cases like sessions spanning midnight.
-
-**`definitions/ga4/ga4_sessions.sqlx`**
-
-```javascript
-config {
-  type: "incremental",
-  description: "GA4 sessions table",
-  schema: "ga4_export_fixer",
-  bigquery: {
-    partitionBy: "event_date",
-    clusterBy: ['session_id', 'data_is_final'],
-  },
-  tags: ['ga4_export_fixer']
-}
-
-js {
-  const { setPreOperations, helpers } = require('ga4-export-fixer');
-
-  const config = {
-    self: self(),
-    incremental: incremental(),
-    /*
-    Default options that can be overriden:
-    test: false,
-    testConfig: {
-        dateRangeStart: 'current_date()-1',
-        dateRangeEnd: 'current_date()',
-    },
-    preOperations: {
-        dateRangeStartFullRefresh: 'date(2000, 1, 1)',
-        dateRangeEnd: 'current_date()',
-        // incremental date range overrides allow re-processing only a subset of the data:
-        //incrementalStartOverride: undefined,
-        //incrementalEndOverride: undefined,
-    },
-    */
-  };
-}
-
-select
-  event_date,
-  session_id,
-  user_pseudo_id,
-  user_id,
-  any_value(session_traffic_source_last_click.cross_channel_campaign) as session_traffic_source,
-  any_value(landing_page) as landing_page,
-  current_datetime() as row_inserted_timestamp,
-  min(data_is_final) as data_is_final
-from
-  ${ref('ga4_events_enhanced_298233330')}
-where
-  ${helpers.incrementalDateFilter(config)}
-group by 
-  event_date,
-  session_id,
-  user_pseudo_id,
-  user_id
-
-pre_operations {
-  ${setPreOperations(config)}
-}
-```
-
 ### Configuration Object
 
 All fields are optional except `sourceTable`. Default values are applied automatically, so you only need to specify the fields you want to override.
@@ -286,7 +236,7 @@ All fields are optional except `sourceTable`. Default values are applied automat
 {
     "name": "ga4_events_enhanced_<dataset_id>",
     "type": "incremental",
-    "schema": "ga4_export_fixer",
+    "schema": "<source_dataset>",
     "description": "<default description>",
     "bigquery": {
         "partitionBy": "event_date",
@@ -312,7 +262,7 @@ The `onSchemaChange: "EXTEND"` setting updates the result table schema on increm
 </details>
 <br>
 
-`**includedExportTypes`** — which GA4 export types to include:
+`**includedExportTypes**` — which GA4 export types to include:
 
 
 | Field                          | Type    | Default | Description                      |
@@ -323,7 +273,7 @@ The `onSchemaChange: "EXTEND"` setting updates the result table schema on increm
 
 > **Intraday-only mode:** Set `daily` to `false` and `intraday` to `true` to use only intraday export tables. When using intraday-only mode, `dataIsFinal.detectionMethod` must be set to `'DAY_THRESHOLD'`.
 
-`**dataIsFinal`** — how to determine whether data is final (not expected to change):
+**`dataIsFinal`** — how to determine whether data is final (not expected to change):
 
 
 | Field                         | Type    | Default         | Description                                                                                                                                                                                          |
@@ -332,7 +282,7 @@ The `onSchemaChange: "EXTEND"` setting updates the result table schema on increm
 | `dataIsFinal.dayThreshold`    | integer | `4`             | Days after which data is considered final. Required when `detectionMethod` is `'DAY_THRESHOLD'`                                                                                                      |
 
 
-`**testConfig**` — date range used when `test` is `true`:
+**`testConfig`** — date range used when `test` is `true`:
 
 
 | Field                       | Type              | Default              | Description                 |
@@ -341,7 +291,7 @@ The `onSchemaChange: "EXTEND"` setting updates the result table schema on increm
 | `testConfig.dateRangeEnd`   | string (SQL date) | `'current_date()'`   | End date for test queries   |
 
 
-`**preOperations**` — date range and incremental refresh configuration:
+**`preOperations`** — date range and incremental refresh configuration:
 
 
 | Field                                      | Type              | Default              | Description                                                                                                                                                                                         |
@@ -353,7 +303,7 @@ The `onSchemaChange: "EXTEND"` setting updates the result table schema on increm
 | `preOperations.incrementalEndOverride`     | string (SQL date) | `undefined`          | Override the incremental end date to re-process a specific range                                                                                                                                    |
 
 
-`**eventParamsToColumns**` — each item in the array is an object:
+**`eventParamsToColumns`** — each item in the array is an object:
 
 
 | Field        | Type   | Required | Description                                                                                                                           |
@@ -365,6 +315,76 @@ The `onSchemaChange: "EXTEND"` setting updates the result table schema on increm
 
 Date fields (`dateRangeStart`, `dateRangeEnd`, etc.) accept string dates in `YYYYMMDD` or `YYYY-MM-DD` format, or BigQuery SQL expressions (e.g. `'current_date()'`, `'date(2026, 1, 1)'`).
 
+### Creating Incremental Downstream Tables from ga4_events_enhanced
+
+Setting up incremental updates is easy using the **setPreOperations()** function. Just ensure that your result table includes the **data_is_final** flag from the **ga4_events_enhanced** table.
+
+The **incrementalDateFilter()** function applies the same date filtering used by **ga4_events_enhanced**, based on the **config** options and the variables declared by **setPreOperations()**.
+
+Key fields such as session_id, user_id, and session_traffic_source_last_click are available as clean, sessionized versions that handle edge cases like sessions spanning midnight.
+
+**`definitions/ga4/ga4_sessions.sqlx`**
+
+```javascript
+config {
+  type: "incremental",
+  description: "GA4 sessions table",
+  schema: "ga4_export_fixer",
+  bigquery: {
+    partitionBy: "event_date",
+    clusterBy: ['session_id', 'data_is_final'],
+  },
+  tags: ['ga4_export_fixer']
+}
+
+js {
+  const { setPreOperations, helpers } = require('ga4-export-fixer');
+
+  const config = {
+    self: self(),
+    incremental: incremental(),
+    /*
+    Default options that can be overriden:
+    test: false,
+    testConfig: {
+        dateRangeStart: 'current_date()-1',
+        dateRangeEnd: 'current_date()',
+    },
+    preOperations: {
+        dateRangeStartFullRefresh: 'date(2000, 1, 1)',
+        dateRangeEnd: 'current_date()',
+        // incremental date range overrides allow re-processing only a subset of the data:
+        //incrementalStartOverride: undefined,
+        //incrementalEndOverride: undefined,
+    },
+    */
+  };
+}
+
+select
+  event_date,
+  session_id,
+  user_pseudo_id,
+  user_id,
+  any_value(session_traffic_source_last_click.cross_channel_campaign) as session_traffic_source,
+  any_value(landing_page) as landing_page,
+  current_datetime() as row_inserted_timestamp,
+  min(data_is_final) as data_is_final
+from
+  ${ref('ga4_events_enhanced_298233330')}
+where
+  ${helpers.incrementalDateFilter(config)}
+group by 
+  event_date,
+  session_id,
+  user_pseudo_id,
+  user_id
+
+pre_operations {
+  ${setPreOperations(config)}
+}
+```
+
 ### Helpers
 
 The helpers contain templates for common SQL expressions. The functions are referenced by **ga4EventsEnhanced** but can also be imported as utility functions for working with GA4 data.

package/columns/columnDescriptions.json

@@ -1,7 +1,7 @@
 {
     "event_date": "Date of the event, cast to DATE from the original YYYYMMDD string",
     "event_datetime": "Event datetime converted to the configured timezone from event_timestamp (or custom timestamp if configured)",
-    "event_timestamp": "Time in microseconds (UTC) when the event was logged by Google Analytics",
+    "event_timestamp": "Time in microseconds (UTC) when the event was received by Google Analytics",
     "event_custom_timestamp": "Event timestamp in microseconds derived from a custom event parameter (e.g. collected via Date.now()), falling back to event_timestamp when the custom parameter is null. Only present when customTimestampParam is configured",
     "event_name": "Name of the event (e.g. page_view, purchase, scroll)",
     "session_id": "Unique session identifier, constructed by concatenating user_pseudo_id with the ga_session_id event parameter",
@@ -48,7 +48,7 @@
                 "columns": {
                     "string_value": "String value of the event parameter",
                     "int_value": "Integer value of the event parameter",
-                    "float_value": "Float value of the event parameter",
+                    "float_value": "Float value of the event parameter (currently unused in GA4)",
                     "double_value": "Double value of the event parameter"
                 }
             }
@@ -63,14 +63,14 @@
                 "columns": {
                     "string_value": "String value of the session parameter",
                     "int_value": "Integer value of the session parameter",
-                    "float_value": "Float value of the session parameter",
+                    "float_value": "Float value of the session parameter (currently unused in GA4)",
                     "double_value": "Double value of the session parameter"
                 }
             }
         }
     },
     "user_properties": {
-        "description": "User properties set via the Google Analytics SDK or gtag API",
+        "description": "User properties",
         "columns": {
             "key": "Name of the user property",
             "value": {
@@ -79,7 +79,7 @@
                     "string_value": "String value of the user property",
                     "int_value": "Integer value of the user property",
                     "double_value": "Double value of the user property",
-                    "float_value": "Float value of the user property (currently unused by GA4)",
+                    "float_value": "Float value of the user property (currently unused in GA4)",
                     "set_timestamp_micros": "Time in microseconds at which the user property was last set"
                 }
             }
@@ -284,7 +284,7 @@
             "source": "Source network that first acquired the user"
         }
     },
-    "event_previous_timestamp": "Time in microseconds (UTC) when the previous event was logged",
+    "event_previous_timestamp": "Time in microseconds (UTC) when the previous event happened",
     "event_value_in_usd": "Currency-converted value (in USD) of the event's 'value' parameter",
     "event_bundle_sequence_id": "Sequential ID of the bundle in which the event was uploaded",
     "event_server_timestamp_offset": "Timestamp offset between collection time and upload time in microseconds",

package/documentation.js

@@ -10,6 +10,8 @@ const columnDescriptions = require('./columns/columnDescriptions.json');
 const getColumnDescriptions = (config) => {
     const descriptions = JSON.parse(JSON.stringify(columnDescriptions));
 
+    if (!config) return descriptions;
+
     const appendToDescription = (key, suffix) => {
         if (!descriptions[key]) return;
         if (typeof descriptions[key] === 'string') {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ga4-export-fixer",
-  "version": "0.2.3-dev.3",
+  "version": "0.2.4-dev.0",
   "description": "",
   "main": "index.js",
   "files": [

package/preOperations.js

@@ -128,6 +128,11 @@ const createSchemaLockTable = (config) => {
 
 // Set the pre operations for the query
 const setPreOperations = (config) => {
+  // if in test mode, avoid setting BigQuery variables to make query dry run estimation accurate
+  if (config.test) {
+    return '';
+  }
+
   // define the pre operations
   const preOperations = [
     {