ga4-export-fixer 0.3.1 → 0.3.2-dev.1

package/README.md

@@ -228,6 +228,11 @@ All fields are optional except `sourceTable`. Default values are applied automat
 | `excludedEvents`       | string[]                | `['session_start', 'first_visit']` | Event names to exclude from the table. These events are excluded by default because they have no use for analysis purposes. Override this to include them if needed                                                                                                                                                          |
 | `excludedColumns`      | string[]                | `[]`                               | Default GA4 export columns to exclude from the final table, for example `'app_info'` or `'publisher'`                                                                                                                                                                                                                        |
 | `sessionParams`        | string[]                | `[]`                               | Event parameter names to aggregate as session-level parameters                                                                                                                                                                                                                                                               |
+| `includedExportTypes`  | object                  | [See details](#includedExportTypes) | Which GA4 export types to include (daily, fresh, intraday)                                                                                                                                                                                                                                                                  |
+| `dataIsFinal`          | object                  | [See details](#dataIsFinal)         | How to determine whether data is final (not expected to change)                                                                                                                                                                                                                                                              |
+| `testConfig`           | object                  | [See details](#testConfig)          | Date range used when `test` is `true`                                                                                                                                                                                                                                                                                        |
+| `preOperations`        | object                  | [See details](#preOperations)       | Date range and incremental refresh configuration                                                                                                                                                                                                                                                                             |
+| `eventParamsToColumns` | object[]                | `[]`                                | Event parameters to promote to columns. [See item schema](#eventParamsToColumns)                                                                                                                                                                                                                                             |
 
 <a id="default-dataformtableconfig"></a>
 <details>
@@ -263,7 +268,9 @@ The `onSchemaChange: "EXTEND"` setting updates the result table schema on increm
 </details>
 <br>
 
-`**includedExportTypes**` — which GA4 export types to include:
+<a id="includedExportTypes"></a>
+
+**`includedExportTypes`** — which GA4 export types to include:
 
 
 | Field                          | Type    | Default | Description                      |
@@ -284,6 +291,8 @@ The boundary between fresh and intraday is timestamp-based because the fresh exp
 
 > **Without daily export:** When `daily` is `false`, `dataIsFinal.detectionMethod` must be set to `'DAY_THRESHOLD'`, because `EXPORT_TYPE` detection relies on daily tables to mark data as final.
 
+<a id="dataIsFinal"></a>
+
 **`dataIsFinal`** — how to determine whether data is final (not expected to change):
 
 
@@ -293,6 +302,8 @@ The boundary between fresh and intraday is timestamp-based because the fresh exp
 | `dataIsFinal.dayThreshold`    | integer | `3`               | Days after which data is considered final. According to GA4 documentation, data up to 72 hours old is subject to possible changes. Required when `detectionMethod` is `'DAY_THRESHOLD'`            |
 
 
+<a id="testConfig"></a>
+
 **`testConfig`** — date range used when `test` is `true`:
 
 
@@ -302,6 +313,8 @@ The boundary between fresh and intraday is timestamp-based because the fresh exp
 | `testConfig.dateRangeEnd`   | string (SQL date) | `'current_date()'`   | End date for test queries   |
 
 
+<a id="preOperations"></a>
+
 **`preOperations`** — date range and incremental refresh configuration:
 
 
@@ -315,6 +328,8 @@ The boundary between fresh and intraday is timestamp-based because the fresh exp
 | `preOperations.numberOfDaysToProcess`      | integer           | `undefined`          | Limit each run to N days of data. When set, the end date becomes `start + N - 1` (capped at `current_date()`). When `undefined`, `dateRangeEnd` is used as-is. `incrementalEndOverride` takes priority  |
 
 
+<a id="eventParamsToColumns"></a>
+
 **`eventParamsToColumns`** — each item in the array is an object:
 
 

package/columns/columnLineage.json

@@ -0,0 +1,146 @@
+{
+    "event_date": {
+        "source": "ga4_export_modified",
+        "note": "Cast to DATE from the original YYYYMMDD string format"
+    },
+    "event_datetime": {
+        "source": "derived",
+        "note": "Computed from event_timestamp (or custom timestamp parameter) with timezone conversion"
+    },
+    "event_timestamp": {
+        "source": "ga4_export"
+    },
+    "event_custom_timestamp": {
+        "source": "derived",
+        "note": "Derived from a custom event parameter (milliseconds converted to microseconds), falling back to event_timestamp"
+    },
+    "event_name": {
+        "source": "ga4_export"
+    },
+    "session_id": {
+        "source": "derived",
+        "note": "Concatenation of user_pseudo_id and the ga_session_id event parameter"
+    },
+    "user_pseudo_id": {
+        "source": "ga4_export"
+    },
+    "user_id": {
+        "source": "ga4_export_modified",
+        "note": "Session-level aggregation: last non-null user_id within the session, ordered by timestamp"
+    },
+    "merged_user_id": {
+        "source": "derived",
+        "note": "Coalesces session-level user_id with user_pseudo_id"
+    },
+    "page_location": {
+        "source": "derived",
+        "note": "Unnested from the page_location event parameter"
+    },
+    "page": {
+        "source": "derived",
+        "note": "Parsed from the page_location event parameter into hostname, path, query, and query_params"
+    },
+    "landing_page": {
+        "source": "derived",
+        "note": "Session-level aggregation: first page struct where entrances > 0, ordered by timestamp"
+    },
+    "event_params": {
+        "source": "ga4_export_modified",
+        "note": "Excluded and promoted parameters are filtered out from the original array"
+    },
+    "session_params": {
+        "source": "derived",
+        "note": "Aggregated from event_params at session level: last non-null value per configured parameter"
+    },
+    "user_properties": {
+        "source": "ga4_export"
+    },
+    "ecommerce": {
+        "source": "ga4_export_modified",
+        "note": "Fixes applied: transaction_id '(not set)' nullified, purchase_revenue NaN and missing-value corrections"
+    },
+    "items": {
+        "source": "ga4_export"
+    },
+    "user_ltv": {
+        "source": "ga4_export"
+    },
+    "collected_traffic_source": {
+        "source": "ga4_export"
+    },
+    "session_first_traffic_source": {
+        "source": "derived",
+        "note": "Session-level aggregation: collected_traffic_source from the first event in the session"
+    },
+    "session_traffic_source_last_click": {
+        "source": "ga4_export_modified",
+        "note": "Session-level aggregation: first non-null value in the session (the field is session-scoped in the GA4 export)"
+    },
+    "user_traffic_source": {
+        "source": "ga4_export_modified",
+        "note": "Renamed from the GA4 export traffic_source field"
+    },
+    "event_previous_timestamp": {
+        "source": "ga4_export"
+    },
+    "event_value_in_usd": {
+        "source": "ga4_export"
+    },
+    "event_bundle_sequence_id": {
+        "source": "ga4_export"
+    },
+    "event_server_timestamp_offset": {
+        "source": "ga4_export"
+    },
+    "privacy_info": {
+        "source": "ga4_export"
+    },
+    "user_first_touch_timestamp": {
+        "source": "ga4_export"
+    },
+    "device": {
+        "source": "ga4_export"
+    },
+    "geo": {
+        "source": "ga4_export"
+    },
+    "app_info": {
+        "source": "ga4_export"
+    },
+    "stream_id": {
+        "source": "ga4_export"
+    },
+    "platform": {
+        "source": "ga4_export"
+    },
+    "event_dimensions": {
+        "source": "ga4_export"
+    },
+    "is_active_user": {
+        "source": "ga4_export"
+    },
+    "batch_event_index": {
+        "source": "ga4_export"
+    },
+    "batch_page_id": {
+        "source": "ga4_export"
+    },
+    "batch_ordering_id": {
+        "source": "ga4_export"
+    },
+    "publisher": {
+        "source": "ga4_export"
+    },
+    "row_inserted_timestamp": {
+        "source": "derived",
+        "note": "Set to current_timestamp() when the row is inserted or refreshed by the incremental pipeline"
+    },
+    "data_is_final": {
+        "source": "derived",
+        "note": "Computed from export type or day threshold since event_date"
+    },
+    "export_type": {
+        "source": "derived",
+        "note": "Determined from the GA4 export table suffix pattern"
+    }
+}

package/columns/columnTypicalUse.json

@@ -0,0 +1,23 @@
+{
+    "event_date": "Partition column. Always include in WHERE clauses to limit scanned data and reduce query cost",
+    "event_datetime": "Human-readable timestamp in the configured timezone. Use for time-of-day analysis and reporting",
+    "event_timestamp": "Microsecond-precision UTC timestamp. Use for precise event ordering and time difference calculations",
+    "event_name": "Primary event filter. Use in WHERE clauses to select specific event types (e.g. WHERE event_name = 'purchase')",
+    "session_id": "Use for counting unique sessions (COUNT(DISTINCT session_id)) and as a GROUP BY key for session-level aggregations",
+    "user_pseudo_id": "Device-level user identifier. Use for counting unique devices or as a fallback when merged_user_id is not needed",
+    "user_id": "Authenticated user identifier. NULL when the user is not logged in. Prefer merged_user_id for user-level analysis",
+    "merged_user_id": "Preferred user identifier for user-level aggregations. Resolves authenticated and anonymous users into a single ID per session",
+    "page_location": "Full URL string. Use for exact-match filtering. For path-level or hostname-level analysis, use the page struct instead",
+    "page": "Use page.hostname, page.path, and page.query_params for structured URL analysis without manual string parsing",
+    "landing_page": "Session entry page. Use for landing page performance reports and campaign attribution analysis",
+    "event_params": "Nested array of event parameters. Unnest with CROSS JOIN UNNEST(event_params) to access individual parameter values",
+    "session_params": "Session-scoped parameters. Unnest with CROSS JOIN UNNEST(session_params) to access session-level parameter values",
+    "ecommerce": "Transaction and revenue data. Filter to purchase or refund events (WHERE event_name = 'purchase') for ecommerce reporting",
+    "items": "Product-level data array. Unnest with CROSS JOIN UNNEST(items) for item-level analysis in ecommerce reports",
+    "collected_traffic_source": "Event-level UTM parameters and click identifiers. For session-level attribution, prefer session_first_traffic_source instead",
+    "session_first_traffic_source": "First-touch traffic source for the session. Use for session-level acquisition and campaign reporting",
+    "session_traffic_source_last_click": "Google-attributed session traffic source. Use for last-click attribution analysis across Google Ads and manual channels",
+    "data_is_final": "Data stability flag. Use WHERE data_is_final = true when you need only stable data that will not change in future refreshes",
+    "export_type": "Source export type (daily, intraday, fresh). Use for debugging data freshness or filtering by export source",
+    "row_inserted_timestamp": "Pipeline metadata. Use to identify when data was last refreshed or to debug incremental update issues"
+}

package/documentation.js

@@ -1,85 +1,177 @@
-const columnDescriptions = require('./columns/columnDescriptions.json');
-
-/**
- * Returns a deep copy of the default column descriptions, enriched with
- * configuration-specific context appended to the relevant descriptions.
- *
- * @param {Object} config - The merged configuration object from ga4EventsEnhanced.
- * @returns {Object} Column descriptions object in Dataform ITableConfig columns format.
- */
-const getColumnDescriptions = (config) => {
-    const descriptions = JSON.parse(JSON.stringify(columnDescriptions));
-
-    if (!config) return descriptions;
-
-    const appendToDescription = (key, suffix) => {
-        if (!descriptions[key]) return;
-        if (typeof descriptions[key] === 'string') {
-            descriptions[key] = `${descriptions[key]}. ${suffix}`;
-        } else if (typeof descriptions[key] === 'object' && descriptions[key].description) {
-            descriptions[key].description = `${descriptions[key].description}. ${suffix}`;
-        }
-    };
-
-    // timezone
-    if (config.timezone) {
-        appendToDescription('event_datetime', `Timezone: ${config.timezone}`);
-    }
-
-    // customTimestampParam
-    if (config.customTimestampParam) {
-        appendToDescription('event_datetime', `Custom timestamp parameter: '${config.customTimestampParam}'`);
-        appendToDescription('event_custom_timestamp', `Source parameter: '${config.customTimestampParam}'`);
-    } else {
-        delete descriptions.event_custom_timestamp;
-    }
-
-    // data_is_final
-    if (config.dataIsFinal) {
-        const method = config.dataIsFinal.detectionMethod;
-        if (method === 'DAY_THRESHOLD') {
-            appendToDescription('data_is_final', `Detection method: DAY_THRESHOLD (${config.dataIsFinal.dayThreshold} days)`);
-        } else {
-            appendToDescription('data_is_final', `Detection method: EXPORT_TYPE`);
-        }
-    }
-
-    // excludedEvents
-    if (config.excludedEvents && config.excludedEvents.length > 0) {
-        appendToDescription('event_name', `Excluded events: ${config.excludedEvents.join(', ')}`);
-    }
-
-    // excludedEventParams
-    if (config.excludedEventParams && config.excludedEventParams.length > 0) {
-        appendToDescription('event_params', `Excluded parameters: ${config.excludedEventParams.join(', ')}`);
-    }
-
-    // sessionParams
-    if (config.sessionParams && config.sessionParams.length > 0) {
-        appendToDescription('session_params', `Configured parameters: ${config.sessionParams.join(', ')}`);
-    }
-
-    // eventParamsToColumns — add descriptions for dynamically promoted columns
-    if (config.eventParamsToColumns && config.eventParamsToColumns.length > 0) {
-        config.eventParamsToColumns.forEach(p => {
-            const columnName = p.columnName || p.name;
-            const type = p.type ? ` (${p.type})` : ' (any data type)';
-            descriptions[columnName] = `Promoted from event parameter '${p.name}'${type}`;
-        });
-    }
-
-    // includedExportTypes
-    if (config.includedExportTypes) {
-        const types = Object.entries(config.includedExportTypes)
-            .filter(([, enabled]) => enabled)
-            .map(([type]) => type);
-        appendToDescription('export_type', `Included export types: ${types.join(', ')}`);
-    }
-
-    return descriptions;
-};
-
-module.exports = {
-    columnDescriptions,
-    getColumnDescriptions
-};
+const columnDescriptions = require('./columns/columnDescriptions.json');
+const columnLineage = require('./columns/columnLineage.json');
+const columnTypicalUse = require('./columns/columnTypicalUse.json');
+
+/**
+ * Composes a multi-section column description string from individual sections.
+ * Sections with null/undefined/empty values are omitted.
+ * Sections are separated by line breaks for readability.
+ *
+ * @param {Object} sections - { base, lineage, typicalUse, config }
+ * @returns {string} Composed description
+ */
+const composeDescription = (sections) => {
+    const parts = [];
+
+    if (sections.base) {
+        parts.push(sections.base);
+    }
+
+    if (sections.lineage) {
+        parts.push(`Lineage: ${sections.lineage}`);
+    }
+
+    if (sections.typicalUse) {
+        parts.push(`Typical use: ${sections.typicalUse}`);
+    }
+
+    if (sections.config) {
+        parts.push(`Config: ${sections.config}`);
+    }
+
+    return parts.join('\n\n');
+};
+
+/**
+ * Returns a formatted lineage text string for a column, or null if no lineage data exists.
+ *
+ * @param {string} columnName - The column name to look up.
+ * @returns {string|null} Formatted lineage string, e.g. "Derived -- Concatenation of ..."
+ */
+const getLineageText = (columnName) => {
+    const entry = columnLineage[columnName];
+    if (!entry) return null;
+
+    const sourceLabels = {
+        'ga4_export': 'Standard GA4 export field',
+        'ga4_export_modified': 'GA4 export field (modified)',
+        'derived': 'Derived',
+    };
+
+    const label = sourceLabels[entry.source] || entry.source;
+    return entry.note ? `${label} -- ${entry.note}` : label;
+};
+
+/**
+ * Builds a map of config-specific notes for columns based on the provided configuration.
+ * Extracts the configuration-dependent description suffixes into a { columnName: "note" } map.
+ *
+ * @param {Object} config - The merged configuration object.
+ * @returns {Object} Map of column names to config note strings.
+ */
+const buildConfigNotes = (config) => {
+    const notes = {};
+
+    if (!config) return notes;
+
+    const append = (key, text) => {
+        notes[key] = notes[key] ? `${notes[key]}. ${text}` : text;
+    };
+
+    // timezone
+    if (config.timezone) {
+        append('event_datetime', `Timezone: ${config.timezone}`);
+    }
+
+    // customTimestampParam
+    if (config.customTimestampParam) {
+        append('event_datetime', `Custom timestamp parameter: '${config.customTimestampParam}'`);
+        append('event_custom_timestamp', `Source parameter: '${config.customTimestampParam}'`);
+    }
+
+    // data_is_final
+    if (config.dataIsFinal) {
+        const method = config.dataIsFinal.detectionMethod;
+        if (method === 'DAY_THRESHOLD') {
+            append('data_is_final', `Detection method: DAY_THRESHOLD (${config.dataIsFinal.dayThreshold} days)`);
+        } else {
+            append('data_is_final', `Detection method: EXPORT_TYPE`);
+        }
+    }
+
+    // excludedEvents
+    if (config.excludedEvents && config.excludedEvents.length > 0) {
+        append('event_name', `Excluded events: ${config.excludedEvents.join(', ')}`);
+    }
+
+    // excludedEventParams
+    if (config.excludedEventParams && config.excludedEventParams.length > 0) {
+        append('event_params', `Excluded parameters: ${config.excludedEventParams.join(', ')}`);
+    }
+
+    // sessionParams
+    if (config.sessionParams && config.sessionParams.length > 0) {
+        append('session_params', `Configured parameters: ${config.sessionParams.join(', ')}`);
+    }
+
+    // includedExportTypes
+    if (config.includedExportTypes) {
+        const types = Object.entries(config.includedExportTypes)
+            .filter(([, enabled]) => enabled)
+            .map(([type]) => type);
+        if (types.length > 0) {
+            append('export_type', `Included export types: ${types.join(', ')}`);
+        }
+    }
+
+    return notes;
+};
+
+/**
+ * Returns a deep copy of the default column descriptions, enriched with
+ * lineage, typical use, and configuration-specific sections composed into
+ * multi-section descriptions.
+ *
+ * @param {Object} config - The merged configuration object from ga4EventsEnhanced.
+ * @returns {Object} Column descriptions object in Dataform ITableConfig columns format.
+ */
+const getColumnDescriptions = (config) => {
+    const descriptions = JSON.parse(JSON.stringify(columnDescriptions));
+
+    const configNotes = buildConfigNotes(config);
+
+    // Compose multi-section descriptions for each top-level column
+    for (const key of Object.keys(descriptions)) {
+        const isStruct = typeof descriptions[key] === 'object' && descriptions[key].description;
+        const baseDesc = isStruct ? descriptions[key].description : (typeof descriptions[key] === 'string' ? descriptions[key] : null);
+
+        if (!baseDesc) continue;
+
+        const composed = composeDescription({
+            base: baseDesc,
+            lineage: getLineageText(key),
+            typicalUse: columnTypicalUse[key] || null,
+            config: configNotes[key] || null,
+        });
+
+        if (isStruct) {
+            descriptions[key].description = composed;
+        } else {
+            descriptions[key] = composed;
+        }
+    }
+
+    // Add descriptions for dynamically promoted event parameter columns
+    if (config && config.eventParamsToColumns && config.eventParamsToColumns.length > 0) {
+        config.eventParamsToColumns.forEach(p => {
+            const columnName = p.columnName || p.name;
+            const type = p.type ? ` (${p.type})` : ' (any data type)';
+            descriptions[columnName] = composeDescription({
+                base: `Promoted from event parameter '${p.name}'${type}`,
+                lineage: `Derived -- Promoted from the event_params array`,
+                typicalUse: 'Promoted event parameter available as a top-level column for direct filtering and aggregation',
+                config: null,
+            });
+        });
+    }
+
+    return descriptions;
+};
+
+module.exports = {
+    columnDescriptions,
+    getColumnDescriptions,
+    composeDescription,
+    getLineageText,
+    buildConfigNotes,
+};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ga4-export-fixer",
-  "version": "0.3.1",
+  "version": "0.3.2-dev.1",
   "description": "",
   "main": "index.js",
   "files": [
@@ -17,7 +17,8 @@
     "documentation.js"
   ],
   "scripts": {
-    "test": "node tests/ga4EventsEnhanced.test.js && node tests/mergeSQLConfigurations.test.js && node tests/preOperations.test.js",
+    "test": "node tests/ga4EventsEnhanced.test.js && node tests/mergeSQLConfigurations.test.js && node tests/preOperations.test.js && node tests/documentation.test.js",
+    "test:docs": "node tests/documentation.test.js",
     "test:preops": "node tests/preOperations.test.js",
     "test:events": "node tests/ga4EventsEnhanced.test.js",
     "test:merge": "node tests/mergeSQLConfigurations.test.js",

package/tables/ga4EventsEnhanced.js

@@ -338,7 +338,8 @@ The last full table refresh was done using this configuration:
 ${JSON.stringify(
   Object.fromEntries(
     // don't display the default arrays here, their contents are included in the main arrays via the mergeSQLConfigurations function
-    Object.entries(mergedConfig).filter(([key]) => !key.startsWith('default'))
+    // dataformTAbleConfig is also excluded since it's not relevant for the SQL generation and is more of a deployment detail
+    Object.entries(mergedConfig).filter(([key]) => !key.startsWith('default') && key !== 'dataformTableConfig')
   ),
   null,
   2