ga4-export-fixer 0.8.0 → 0.9.0-dev.10

package/README.md

@@ -128,6 +128,12 @@ The goal of the package is to **speed up development** when building data models
   <b>🕐 Timezone-Aware Datetime</b><br>
   <code>event_datetime</code> converted to a configurable IANA timezone
 </td>
+<td valign="top">
+  <b>🧩 Custom Processing Steps</b><br>
+  Append user-defined CTEs via <code>customSteps</code> to derive new columns or join external tables
+</td>
+</tr>
+<tr>
 <td valign="top">
   <b>🛡️ Zero Dependencies</b><br>
   No additional external dependencies added to your Dataform repository
@@ -139,10 +145,10 @@ The goal of the package is to **speed up development** when building data models
 
 Features under consideration for future releases:
 
+- Data enrichment (item-level, session-level, event-level)
+- Aggregated tables (ga4_session, ga4_ecommerce...)
 - Web and app specific default configurations
 - Custom channel grouping
-- Data enrichment (item-level, session-level, event-level)
-- Custom processing steps (additional CTEs)
 - Custom traffic source attribution
 
 ## Installation
@@ -192,7 +198,8 @@ Create a new **ga4_events_enhanced** table using a **.js** file in your reposito
 const { ga4EventsEnhanced } = require('ga4-export-fixer');
 
 const config = {
-  sourceTable: constants.GA4_TABLES.MY_GA4_EXPORT
+  // using hard-coded GA4 export path
+  sourceTable: '`project.analytics_12345.events_*`'
 };
 
 ga4EventsEnhanced.createTable(publish, config);
@@ -206,6 +213,7 @@ ga4EventsEnhanced.createTable(publish, config);
 const { ga4EventsEnhanced } = require('ga4-export-fixer');
 
 const config = {
+  // GA4 export path declared, using the table reference object
   sourceTable: constants.GA4_TABLES.MY_GA4_EXPORT,
   // use dataformTableConfig to make changes to the default Dataform table configuration
   dataformTableConfig: {
@@ -284,7 +292,8 @@ js {
   const { ga4EventsEnhanced } = require('ga4-export-fixer');
 
   const config = {
-    sourceTable: ref(constants.GA4_TABLES.MY_GA4_EXPORT),
+    // using hard-coded GA4 export path
+    sourceTable: '`project.analytics_12345.events_*`',
     self: self(),
     incremental: incremental()
   };
@@ -308,7 +317,7 @@ All fields are optional except `sourceTable`. Default values are applied automat
 
 | Field                  | Type                    | Default/Required                   | Description                                                                                                                                                                                                                                                                                                                  |
 | ---------------------- | ----------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `sourceTable`          | Dataform ref() / string | **required**                       | Source GA4 export table. Use `ref()` in Dataform or a string in format ``project.dataset.table``                                                                                                                                                                                                                             |
+| `sourceTable`          | Dataform ref / object / string | **required**                       | Source GA4 export table. Inside an SQLX `js { }` block use `ref(...)`. From a `.js` definition file use a `{ schema, name }` ref object (resolved later via `ctx.ref()`) or a backtick-quoted ``` `project.dataset.events_*` ``` string for an external table. |
 | `self`                 | Dataform self()         | **required for .SQLX deployment**  | Reference to the table itself. Use `self()` in Dataform                                                                                                                                                                                                                                                                      |
 | `incremental`          | Dataform incremental()  | **required for .SQLX deployment**  | Switch between incremental and full refresh logic. Use `incremental()` in Dataform                                                                                                                                                                                                                                           |
 | `dataformTableConfig`  | object                  | **In JS deployment only.** [See default](#default-dataformtableconfig) | Override the default Dataform table configuration for JS deployment. See: [ITableConfig reference](https://docs.cloud.google.com/dataform/docs/reference/dataform-core-reference#itableconfig) |
@@ -328,6 +337,7 @@ All fields are optional except `sourceTable`. Default values are applied automat
 | `preOperations`        | object                  | [See details](#preOperations)       | Date range and incremental refresh configuration                                                                                                                                                                                                                                                                             |
 | `eventParamsToColumns` | object[]                | `[]`                                | Event parameters to promote to columns. [See item schema](#eventParamsToColumns)                                                                                                                                                                                                                                             |
 | `customSteps`          | object[]                | `[]`                                | User-defined CTEs appended to the pipeline after `enhanced_events`. [See Custom CTEs](#custom-ctes)                                                                                                                                                                                                                          |
+| `enrichments`          | object[]                | `[]`                                | Declarative external-data enrichments joined into `enhanced_events`. [See Data Enrichments](#data-enrichments)                                                                                                                                                                                                               |
 
 <a id="default-dataformtableconfig"></a>
 <details>
@@ -465,10 +475,12 @@ itemListAttribution: { lookbackType: 'TIME', lookbackTimeMs: 86400000 }
 | ------------------------ | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `event_data`             | yes                                   | Extracted and shaped events from `sourceTable`, with date filtering and column promotions applied. *Unfiltered for the buffer-days range.*           |
 | `session_data`           | yes                                   | Session-level aggregations (grouped by `session_id`).                                                                                                |
-| `item_list_attribution`  | only when `itemListAttribution` is on | Per-event item attribution rows.                                                                                                                    |
-| `item_list_data`         | only when `itemListAttribution` is on | Re-aggregated items with attributed list fields.                                                                                                    |
-| `enhanced_events`        | yes                                   | The package's standard output shape (joined event_data + session_data + item_list_data, columns ordered, incremental date filter applied). The natural starting point for most custom CTEs. |
+| `items_unnested`         | only when `itemListAttribution` is on | Per-event item rows (one row per item per ecommerce event), with attribution window function applied.                                                |
+| `items_rebuilt`          | only when `itemListAttribution` is on | Re-aggregated items with attributed list fields, joined back to events via `_item_row_id`.                                                           |
+| `enrich_<name>`          | only when configured via `enrichments` | One CTE per [enrichment](#data-enrichments) entry, providing dim data for joining into `enhanced_events`.                                              |
+| `enhanced_events`        | yes                                   | The package's standard output shape (joined event_data + session_data + items_rebuilt + enrich_*, columns ordered, incremental date filter applied). The natural starting point for most custom CTEs. |
 
+Example custom step using the raw SQL format:
 
 ```javascript
 // Add a content_group column derived from page.path
@@ -488,10 +500,119 @@ from enhanced_events`,
 ],
 ```
 
+The same example in the structured shape:
+
+```javascript
+customSteps: [
+    {
+        name: 'final',
+        select: {
+            columns: {
+                '[sql]passthrough': 'enhanced_events.*',
+                content_group: `case
+  when page.path like '/blog/%' then 'blog'
+  when page.path like '/products/%' then 'product'
+  when page.path = '/' then 'home'
+  else 'other'
+end`,
+            },
+        },
+        from: 'enhanced_events',
+    },
+],
+```
+
 > **Note:** Custom columns aren't auto-documented. Use `dataformTableConfig.columns` to add descriptions — it's deep-merged with the package's defaults, so your keys are added or override matching defaults, and untouched defaults stay.
 
 > **Note:** Built-in assertions assume the package's standard schema. If your custom CTEs rename, drop, or filter rows in ways that break those assumptions, disable the affected assertions explicitly via the `assertions` config option.
 
+<a id="data-enrichments"></a>
+
+**`enrichments`** — declaratively join external dimension data into `enhanced_events` (cohort labels, page metadata, marketing attribution, etc.). Each entry describes one dim source plus the join — the package generates the source CTE, the `LEFT JOIN`, and column descriptions automatically.
+
+For typical use cases this is the right tool; reach for `customSteps` only when you need a transformation that doesn't fit a flat dim join.
+
+**Per-enrichment shape:**
+
+| Field | Type | Required | Description |
+| --- | --- | --- | --- |
+| `name` | string | Yes | Used in the generated `enrich_<name>` CTE name. Unique within `enrichments`. |
+| `level` | `'event'` / `'item'` | No, defaults to `'event'` | Join grain. `'event'` joins external dim data onto each event row (any column on `enhanced_events` as the key). `'item'` joins external dim data onto each item inside the `items` array (any field on the items struct or any event_data column as the key). |
+| `source` | Dataform ref / object / string | Yes | Source dim table. Inside an SQLX `js { }` block use `ref(...)`. From a `.js` definition file use a `{ schema, name }` ref object (resolved later via `ctx.ref()`) or a backtick-quoted ``` `project.dataset.table` ``` string for an external table. |
+| `joinKey` | string / string[] | Yes | For `level: 'event'`: column name(s) on `enhanced_events`. For `level: 'item'`: field name(s) on the items struct (e.g. `'item_id'`) or column name(s) on `event_data` (e.g. `'user_pseudo_id'`). Composite keys (array) compile to `USING(col1, col2, ...)`. |
+| `columns` | string[] | Yes | Source columns to add to the output (excluding `joinKey`). Names matching existing columns are coalesced with the original (`coalesce(enrich.col, original)`) so missed JOINs fall back to the existing value. |
+| `dedupe` | boolean | No, defaults to `false` | When `true`, wraps the source CTE in `qualify row_number() over (partition by <joinKey>) = 1` for non-unique-key dim sources. Non-deterministic which row wins; for strict needs, pre-aggregate in source SQL. |
+
+**Coalesce-or-add semantics.** If an enrichment column name matches an existing column on `enhanced_events` (a column promoted via `eventParamsToColumns`, a package-generated column, or a default GA4 column from the export), the enrichment value is coalesced with the original: `coalesce(enrich_<name>.<col>, <original>) as <col>`. Rows where the JOIN matches get the enrichment value; rows where it misses fall back to the existing value rather than going NULL. If there is no overlap, the column is added as a plain `enrich_<name>.<col>`.
+
+**Example** — attach user cohort labels by `user_pseudo_id` (Dataform-declared table referenced by `{ schema, name }`):
+
+```javascript
+enrichments: [
+    {
+        name: 'cohorts',
+        level: 'event',
+        source: { schema: 'analytics', name: 'user_cohorts' },
+        joinKey: 'user_pseudo_id',
+        columns: ['cohort_label', 'lifecycle_stage'],
+    },
+],
+```
+
+**Example** — composite key (date + user) for daily-varying dim data, with dedupe safety net (external table referenced by backtick-FQN):
+
+```javascript
+enrichments: [
+    {
+        name: 'segments',
+        level: 'event',
+        source: '`my-project.analytics.daily_user_segments`',
+        joinKey: ['event_date', 'user_pseudo_id'],
+        columns: ['segment'],
+        dedupe: true,
+    },
+],
+```
+
+**Example** — fix a promoted event parameter via enrichment (coalesce case: enrichment value wins where the JOIN matches, original kept where it doesn't):
+
+```javascript
+{
+    eventParamsToColumns: [{ name: 'page_title', type: 'string' }],
+    enrichments: [
+        {
+            name: 'titles',
+            level: 'event',
+            source: { schema: 'analytics', name: 'page_title_overrides' },
+            joinKey: 'page_location',
+            columns: ['page_title'],   // overlaps the promoted column → coalesce(enrich.page_title, event_data.page_title)
+        },
+    ],
+}
+```
+
+**Example** — item-level enrichment: attach product master data to each item via `item_id`. The enrichment flows into the `items` array struct; `margin_bucket` is added as a new item-struct field, and `item_category` overlap-coalesces against the original. Item-level enrichment columns do NOT appear at the event grain — they live inside `items[].<col>`:
+
+```javascript
+enrichments: [
+    {
+        name: 'products',
+        level: 'item',
+        source: { schema: 'analytics', name: 'product_master' },
+        joinKey: 'item_id',                                  // joins on item.item_id
+        columns: ['margin_bucket', 'item_category'],         // margin_bucket is additive; item_category overlap-coalesces
+    },
+],
+```
+
+For `level: 'item'`, valid `joinKey` values are any field on the GA4 items struct (`item_id`, `item_category`, etc.) or any column on `event_data` (`user_pseudo_id`, `event_date`, etc.). An event-level and an item-level enrichment may share the same column name (e.g. both writing `cohort`) — the two columns target structurally distinct slots (`enhanced_events.cohort` at event grain vs `items[].cohort` inside the items array) and are not in collision.
+
+> **Note:** Each enrichment generates a CTE named `enrich_<name>` at the top of the pipeline. The `enrich_*` namespace is part of the reserved-names contract — `customSteps` cannot use these names. The active reserved set includes only the names of enrichments actually configured.
+
+> **Note:** Event-level enrichment columns get auto-generated descriptions (`Added by enrichment '<name>' (joined on <joinKey> from <source>).` for new columns; `Coalesced by enrichment '<name>' (...; falls back to original on missed JOIN). Original: <description>` for overlapping columns). User-supplied `dataformTableConfig.columns` overrides win — the auto-generated description is the default. Item-level enrichment columns do not receive auto-generated descriptions (BigQuery does not surface per-field descriptions on STRUCT-array fields cleanly through Dataform's column-description mechanism).
+
+> **Note:** `joinKey` and `columns` entries must be plain SQL identifiers — inline aliases like `'id as user_id'` are rejected at validation time. If your dim source uses a different column name, alias it in an upstream Dataform view and point `source` at that view.
+
 <br>
 
 ---