ga4-export-fixer 0.4.1 → 0.4.2-dev.1

package/README.md

@@ -27,26 +27,98 @@ The goal of the package is to **speed up development** when building data models
 
 ### Main Features
 
-The **ga4_events_enhanced** table comes with features such as these:
-
-- **Best available data at any time** – Combines daily (processed), fresh (360), and intraday exports so the most complete, accurate version of the data is always available
-- **Robust incremental updates** – Run on any schedule (daily, hourly, or custom)
-- **Flexible schema, better optimized for building data models** – Keeps the flexible structure of the original export while promoting key fields (e.g. `page_location`, `session_id`) to columns for better query performance; **partitioning and clustering** enabled
-- **AI agent ready** – Extensive table description and detailed column descriptions (including lineage, configuration context, and typical use examples) make the data understandable for both AI agents (e.g. BigQuery data agents) and humans
-- **Session-level identity resolution** – `user_id` resolved to the last authenticated value per session; `merged_user_id` coalesces it with `user_pseudo_id`
-- **Session traffic sources** – `session_first_traffic_source` and session-scoped `session_traffic_source_last_click` (adjusting for sessions that span midnight) computed automatically
-- **Landing page detection** – `landing_page` derived per session from the first page where `entrances > 0`
-- **Page URL parsing** – `page` struct with parsed `hostname`, `path`, `query`, and `query_params` from `page_location`
-- **Ecommerce data fixes** – Nullifies `transaction_id` placeholder values and corrects `purchase_revenue` NaN / missing-value bugs
-- **Event parameter handling** – Promote event params to columns; include or exclude by name
-- **Session parameters** – Promote selected event parameters as session-level parameters
-- **Custom timestamp support** – Optionally use a custom event parameter as the primary timestamp, with automatic fallback to `event_timestamp`
-- **Schema lock** – Lock the table schema to a specific GA4 export date to prevent schema drift
-- **Data freshness tracking** – `data_is_final` flag and `export_type` label on every row
-- **Selective date range re-processing** – Re-process a subset of data without a full table rebuild, using `incrementalStartOverride` and `incrementalEndOverride`
-- **Batch processing** – Process large GA4 exports in smaller batches using the `numberOfDaysToProcess` configuration setting
-- **Timezone-aware datetime** – `event_datetime` converted to a configurable IANA timezone
-- **Zero dependencies** – The package has no external dependencies and will not add any additional packages to your Dataform repository
+<table>
+<tr>
+<td width="50%" valign="top">
+  <b>📦 Best Available Data</b><br>
+  Combines daily, fresh (360) &amp; intraday exports so the most complete version is always available
+</td>
+<td width="50%" valign="top">
+  <b>🔄 Incremental Updates</b><br>
+  Run on any schedule — daily, hourly, or custom
+</td>
+</tr>
+<tr>
+<td valign="top">
+  <b>📐 Flexible Schema</b><br>
+  Keeps the flexible structure of the original export with key fields promoted to columns for better query performance; partitioning &amp; clustering enabled
+</td>
+<td valign="top">
+  <b>🤖 AI Agent Ready</b><br>
+  Extensive table &amp; column descriptions for AI agents and humans
+</td>
+</tr>
+<tr>
+<td valign="top">
+  <b>🔑 Session Identity Resolution</b><br>
+  <code>user_id</code> resolved per session; <code>merged_user_id</code> coalesces with <code>user_pseudo_id</code>
+</td>
+<td valign="top">
+  <b>📡 Session Traffic Sources</b><br>
+  <code>session_first_traffic_source</code> and <code>session_traffic_source_last_click</code> computed automatically, adjusting for sessions that span midnight
+</td>
+</tr>
+<tr>
+<td valign="top">
+  <b>📍 Landing Page Detection</b><br>
+  Derived per session from the first page where <code>entrances > 0</code>
+</td>
+<td valign="top">
+  <b>🔗 Page URL Parsing</b><br>
+  Parsed <code>hostname</code>, <code>path</code>, <code>query</code>, and <code>query_params</code> from <code>page_location</code>
+</td>
+</tr>
+<tr>
+<td valign="top">
+  <b>🛒 Ecommerce Data Fixes</b><br>
+  Nullifies placeholder <code>transaction_id</code>; corrects <code>purchase_revenue</code> bugs
+</td>
+<td valign="top">
+  <b>⚙️ Event Parameter Handling</b><br>
+  Promote event params to columns; include or exclude by name
+</td>
+</tr>
+<tr>
+<td valign="top">
+  <b>📊 Session Parameters</b><br>
+  Promote selected event parameters as <code>session_params</code>
+</td>
+<td valign="top">
+  <b>⏱️ Custom Timestamp</b><br>
+  Use a custom event parameter as primary timestamp with automatic fallback
+</td>
+</tr>
+<tr>
+<td valign="top">
+  <b>🔒 Schema Lock</b><br>
+  Lock table schema to a specific GA4 export date to prevent schema drift
+</td>
+<td valign="top">
+  <b>✅ Data Freshness Tracking</b><br>
+  <code>data_is_final</code> flag and <code>export_type</code> label on every row
+</td>
+</tr>
+<tr>
+<td valign="top">
+  <b>🔃 Selective Re-processing</b><br>
+  Re-process a date range without full table rebuild using <code>incrementalStartOverride</code> and <code>incrementalEndOverride</code>
+</td>
+<td valign="top">
+  <b>📑 Batch Processing</b><br>
+  Process large exports in smaller batches via <code>numberOfDaysToProcess</code>
+</td>
+</tr>
+<tr>
+<td valign="top">
+  <b>🕐 Timezone-Aware Datetime</b><br>
+  <code>event_datetime</code> converted to a configurable IANA timezone
+</td>
+<td valign="top">
+  <b>🛡️ Zero Dependencies</b><br>
+  No additional external dependencies added to your Dataform repository
+</td>
+</tr>
+</table>
 
 ### Planned, Upcoming Features
 

package/helpers/aggregation.js

@@ -0,0 +1,226 @@
+/*
+Handling event and session parameters
+*/
+
+/**
+ * Generates a SQL expression that filters the `event_params` array by the given parameter names.
+ *
+ * When filterType is 'include', only parameters whose key matches one of the given names are kept.
+ * When filterType is 'exclude', parameters whose key matches are removed. If the params array is
+ * empty with 'exclude', the original `event_params` column is returned unfiltered.
+ *
+ * @param {string[]} params - Array of event parameter names to include or exclude.
+ * @param {'include'|'exclude'} filterType - Whether to include or exclude the listed parameters.
+ * @returns {string} SQL expression that produces a filtered event_params array.
+ * @throws {Error} If params is not an array of strings, or if filterType is not 'include' or 'exclude'.
+ */
+const filterEventParams = (params, filterType) => {
+  if (!Array.isArray(params) || !params.every(p => typeof p === 'string')) {
+    throw new Error("filterEventParams: 'params' must be an array of strings (empty array allowed).");
+  }
+
+  if (filterType !== 'include' && filterType !== 'exclude') {
+    throw new Error("filterEventParams: 'filterType' must be 'include' or 'exclude'.");
+  }
+
+  const filterParams = params.map(p => `'${p}'`).join(', ');
+
+  if (filterType === 'include') {
+    return `array(select as struct * from unnest(event_params) where key in (${filterParams}))`;
+  }
+
+  if (filterType === 'exclude') {
+    if (!params || params.length === 0) {
+      return 'event_params';
+    }
+
+    return `array(select as struct * from unnest(event_params) where key not in (${filterParams}))`;
+  }
+};
+
+/**
+ * Generates a BigQuery SQL expression that aggregates specified session parameters across events,
+ * returning, for each parameter, the most recent (last non-null) value by timestamp. If a parameter
+ * does not appear, a dummy struct with null values for all types is returned for that key.
+ *
+ * This is useful for building an array of session parameter structs for analytic purposes,
+ * ensuring proper presence of all expected keys and null placeholders where values are missing.
+ *
+ * The resulting SQL expression yields an ARRAY<STRUCT<key STRING, value STRUCT<string_value STRING, int_value INT64, float_value FLOAT64, double_value FLOAT64>>>.
+ *
+ * @param {string[]} paramNames - Array of parameter names (keys) to aggregate.
+ * @param {string} paramsArray - SQL expression or column reference representing the array of session parameters to aggregate.
+ * @param {string} timestampColumn - SQL expression or column indicating the timestamp associated with each parameter, used for ordering.
+ * @returns {string} SQL expression that produces an array of parameter structs with their last values or null if not present.
+ */
+const aggregateSessionParams = (paramNames, paramsArray, timestampColumn) => {
+  // Validate paramNames
+  if (!Array.isArray(paramNames) || !paramNames.every(p => typeof p === 'string')) {
+    throw new Error("aggregateSessionParams: 'paramNames' must be an array of strings (empty array allowed).");
+  }
+  // Validate paramsArray
+  if (typeof paramsArray !== 'string' || paramsArray.trim() === '') {
+    throw new Error("aggregateSessionParams: 'paramsArray' must be a non-empty string reference to a SQL field or expression.");
+  }
+  // Validate timestampColumn
+  if (typeof timestampColumn !== 'string' || timestampColumn.trim() === '') {
+    throw new Error("aggregateSessionParams: 'timestampColumn' must be a non-empty string reference to a SQL field or expression.");
+  }
+
+  if (paramNames.length > 0) {
+    const sessionParamStructs = paramNames.map(p => {
+      return `ifnull(
+      -- get the last non-null value for the parameter
+      array_agg(
+        (select as struct * from unnest(${paramsArray}) where key = '${p}') ignore nulls
+        order by ${timestampColumn} desc
+        limit 1
+      )[safe_offset(0)],
+      -- if no value is found, return a dummy value
+      (
+        select as struct 
+          '${p}' as key, 
+          (
+            select as struct
+              cast(null as string) as string_value,
+              cast(null as int64) as int_value,
+              cast(null as float64) as float_value,
+              cast(null as float64) as double_value
+          ) as value
+      )
+    )`;
+    });
+
+    return `[
+      ${sessionParamStructs.join(',\n    ')}
+    ]`;
+  } else {
+    // declare the session_params in the schema even if no session params are specified
+    return `cast([] as array<struct<key string, value struct<string_value string, int_value int64, float_value float64, double_value float64>>>)`;
+  }
+};
+
+/**
+ * Produces a SQL expression that returns an array of session parameter structs
+ * from the given paramsArray, excluding any where all value fields are null.
+ *
+ * This helper is useful for cleaning up session_params or event_params arrays
+ * by removing elements whose value is entirely null (i.e., string_value, int_value,
+ * float_value, and double_value are all null). The resulting array contains
+ * only parameter entries with at least one non-null value.
+ *
+ * @param {string} paramsArray - The name of the array field or SQL expression to unnest (e.g. 'session_params' or 'event_params').
+ * @returns {string} SQL expression that yields an array of non-null parameter structs.
+ *
+ * @example
+ *   excludeNullSessionParams('session_params')
+ *   // => "array(select as struct * from unnest(session_params) where value.string_value is not null or value.int_value is not null or value.float_value is not null or value.double_value is not null)"
+ */
+const excludeNullSessionParams = (paramsArray) => {
+  if (typeof paramsArray !== 'string' || paramsArray.trim() === '') {
+    throw new Error("excludeNullSessionParams: 'paramsArray' is required and must be a non-empty string.");
+  }
+
+  return `array(select as struct * from unnest(${paramsArray}) where value.string_value is not null or value.int_value is not null or value.float_value is not null or value.double_value is not null)`;
+};
+
+/*
+Aggregation
+*/
+
+/**
+ * Generates a SQL aggregation expression for a specified column and aggregation type,
+ * optionally using a timestamp column for ordering 'first' or 'last' values.
+ *
+ * Supported aggregation types:
+ * - 'max': Returns the maximum value of the column.
+ * - 'min': Returns the minimum value of the column.
+ * - 'first': Returns the first non-null value of the column, ordered by the timestampColumn ascending.
+ * - 'last': Returns the last non-null value of the column, ordered by the timestampColumn descending.
+ * - 'any': Returns any (typically arbitrary) value of the column (uses BigQuery's any_value).
+ *
+ * Throws an error if required parameters are missing or an unsupported aggregation type is requested.
+ *
+ * @param {string} column - The name of the column to aggregate.
+ * @param {'max'|'min'|'first'|'last'|'any'} aggregateType - Type of aggregation.
+ * @param {string} [timestampColumn] - Column to use for ordering. Required when aggregateType is 'first' or 'last'.
+ * @returns {string} A SQL expression for the requested aggregation.
+ * @throws {Error} If required parameters are missing or an unsupported aggregateType is provided.
+ *
+ * @example
+ *   aggregateValue('user_id', 'last', 'event_timestamp')
+ *   // => SQL expression for the last user_id by event_timestamp.
+ */
+const aggregateValue = (column, aggregateType, timestampColumn) => {
+  if (typeof column === 'undefined') {
+    throw new Error("aggregateValue: 'column' is a required parameter and must be defined.");
+  }
+  if (typeof aggregateType === 'undefined') {
+    throw new Error("aggregateValue: 'aggregateType' is a required parameter and must be defined.");
+  }
+  if ((aggregateType === 'first' || aggregateType === 'last') && typeof timestampColumn === 'undefined') {
+    throw new Error(`aggregateValue: 'timestampColumn' is required when aggregateType is '${aggregateType}'.`);
+  }
+
+  if (aggregateType === 'max') {
+    return `max(${column})`;
+  }
+
+  if (aggregateType === 'min') {
+    return `min(${column})`;
+  }
+
+  if (aggregateType === 'first') {
+    return `array_agg(
+    ${column} ignore nulls 
+    order by ${timestampColumn} 
+    limit 1
+  )[safe_offset(0)]`;
+  }
+
+  if (aggregateType === 'last') {
+    return `array_agg(
+    ${column} ignore nulls 
+    order by ${timestampColumn} desc 
+    limit 1
+  )[safe_offset(0)]`;
+  }
+
+  if (aggregateType === 'any') {
+    return `any_value(${column})`;
+  }
+
+  throw new Error(`aggregateValue: Unsupported aggregateType '${aggregateType}'. Supported values are 'max', 'min', 'first', 'last', and 'any'.`);
+};
+
+/**
+ * Generates SQL aggregation expressions for an array of column specifications.
+ *
+ * Each item in the array is passed to {@link aggregateValue} and optionally aliased.
+ * The results are joined with commas for use in a SELECT clause.
+ *
+ * @param {Object[]} values - Array of aggregation specifications.
+ * @param {string} values[].column - The column to aggregate.
+ * @param {'max'|'min'|'first'|'last'|'any'} values[].aggregateType - Type of aggregation.
+ * @param {string} [values[].timestampColumn] - Column for ordering. Required for 'first'/'last'.
+ * @param {string} [values[].alias] - Optional output alias (appended as `AS alias`).
+ * @returns {string} Comma-separated SQL aggregation expressions.
+ * @throws {Error} If values is not an array.
+ */
+const aggregateValues = (values) => {
+  if (Array.isArray(values)) {
+    return values.map(value => {
+      const sqlExpression = aggregateValue(value.column, value.aggregateType, value.timestampColumn)
+      return `${sqlExpression}${value.alias ? ` as ${value.alias}` : ''}`;
+    }).join(',\n ');
+  }
+  throw new Error("aggregateValues: 'values' must be an array of objects with 'column', 'aggregateType', and 'timestampColumn' properties.");
+};
+
+module.exports = {
+  filterEventParams,
+  aggregateSessionParams,
+  excludeNullSessionParams,
+  aggregateValue,
+  aggregateValues
+};

package/helpers/dateFilters.js

@@ -0,0 +1,206 @@
+const constants = require('../constants');
+const { baseConfig } = require('../defaultConfig');
+
+// Filter the export tables by date range
+/**
+ * Generates a SQL filter condition for selecting GA4 export tables based on the export type and a date range.
+ *
+ * This helper produces SQL snippets to be used in WHERE clauses, ensuring only tables within the provided date range and export type are included.
+ *
+ * - For 'daily' exports: Matches table suffixes formatted as YYYYMMDD (e.g., 20240101).
+ * - For 'fresh' exports: Matches table suffixes prefixed with 'fresh_' followed by the date (e.g., fresh_20240101).
+ * - For 'intraday' exports: Matches table suffixes prefixed with 'intraday_' followed by the date (e.g., intraday_20240101).
+ *
+ * @param {'daily'|'fresh'|'intraday'} exportType - The type of export table.
+ * @param {string} start - The start date value as a SQL date expression (e.g. 'current_date()-1').
+ * @param {string} end - The end date value as a SQL date expression (e.g. 'current_date()').
+ * @returns {string} SQL condition to restrict tables by _table_suffix to the appropriate date range and export type.
+ *
+ * @throws {Error} If exportType is not supported, or if start/end are not defined.
+ */
+const ga4ExportDateFilter = (exportType, start, end) => {
+  if (exportType !== 'intraday' && exportType !== 'daily' && exportType !== 'fresh') {
+    throw new Error(
+      `ga4ExportDateFilter: Unsupported exportType '${exportType}'. Supported values are 'daily', 'fresh', and 'intraday'.`
+    );
+  }
+  if (typeof start === 'undefined' || typeof end === 'undefined') {
+    throw new Error("ga4ExportDateFilter: 'start' and 'end' parameters must be defined.");
+  }
+
+  const prefix = exportType === 'daily' ? '' : `'${exportType}_' || `;
+  return `(_table_suffix >= ${prefix}cast(${start} as string format "YYYYMMDD") and _table_suffix <= ${prefix}cast(${end} as string format "YYYYMMDD"))`;
+};
+
+/**
+ * Builds a `_table_suffix` WHERE clause for GA4 BigQuery export tables (daily, fresh, and/or intraday).
+ *
+ * Date boundaries are resolved differently depending on the mode:
+ *   - **test** -- literal dates from `config.testConfig`
+ *   - **incremental** -- BigQuery variable placeholders set by pre-operations
+ *   - **full refresh** -- static dates from `config.preOperations`
+ *
+ * `bufferDays` is subtracted from the daily start date so sessions that span
+ * midnight are not partially excluded.
+ *
+ * Export priority: daily > fresh > intraday. Each lower-priority export only
+ * provides data not already covered by a higher-priority one.
+ *
+ * When fresh and daily are both enabled, the fresh start date comes from
+ * `FRESH_DATE_RANGE_START_VARIABLE` (first day with fresh but no daily table).
+ *
+ * When fresh and intraday are both enabled, intraday rows are filtered by
+ * `event_timestamp > fresh_max_event_timestamp` to avoid duplicating fresh data.
+ *
+ * When only daily and intraday are enabled (no fresh), the existing
+ * `INTRADAY_DATE_RANGE_START_VARIABLE` checkpoint logic is preserved.
+ *
+ * @param {Object} config
+ * @param {boolean}  config.test                      - Use literal test dates.
+ * @param {Object}   config.testConfig                - `{ dateRangeStart, dateRangeEnd }`.
+ * @param {boolean}  config.incremental               - Use BigQuery variable placeholders.
+ * @param {Object}   config.preOperations             - `{ dateRangeStartFullRefresh, dateRangeEnd }`.
+ * @param {Object}   config.includedExportTypes       - `{ daily: boolean, fresh: boolean, intraday: boolean }`.
+ * @param {number}  [config.bufferDays=0]             - Extra days subtracted from the start date.
+ * @returns {string} SQL fragment for a WHERE clause.
+ */
+const ga4ExportDateFilters = (config) => {
+  const bufferDays = config.bufferDays || 0;
+
+  const getStartDate = () => {
+    if (config.test) {
+      return config.testConfig.dateRangeStart;
+    }
+    if (config.incremental) {
+      return constants.DATE_RANGE_START_VARIABLE;
+    }
+    return config.preOperations.dateRangeStartFullRefresh;
+  };
+
+  const getEndDate = () => {
+    if (config.test) {
+      return config.testConfig.dateRangeEnd;
+    }
+    if (config.incremental) {
+      return constants.DATE_RANGE_END_VARIABLE;
+    }
+    if (config.preOperations.numberOfDaysToProcess !== undefined) {
+      return `least(${config.preOperations.dateRangeStartFullRefresh}+${config.preOperations.numberOfDaysToProcess}-1, current_date())`;
+    }
+    return config.preOperations.dateRangeEnd;
+  };
+
+  const getFreshStartDate = () => {
+    // Fresh tables persist alongside daily tables (unlike intraday which gets deleted),
+    // so the checkpoint variable is needed even in test mode to avoid duplicate data.
+    if (config.includedExportTypes.fresh && config.includedExportTypes.daily) {
+      return constants.FRESH_DATE_RANGE_START_VARIABLE;
+    }
+    if (config.includedExportTypes.fresh && !config.includedExportTypes.daily) {
+      return getStartDate();
+    }
+  };
+
+  const getIntradayStartDate = () => {
+    // When fresh is enabled: intraday starts from the same point as fresh.
+    // Fresh tables persist alongside intraday tables, so the checkpoint is
+    // needed even in test mode to avoid duplicate data.
+    if (config.includedExportTypes.fresh) {
+      return getFreshStartDate();
+    }
+    // For non-fresh paths, test mode skips pre-operation variables.
+    if (config.test) {
+      return config.testConfig.dateRangeStart;
+    }
+    // When daily+intraday without fresh: use the existing date-based checkpoint
+    if (config.includedExportTypes.intraday && config.includedExportTypes.daily) {
+      return constants.INTRADAY_DATE_RANGE_START_VARIABLE;
+    }
+    // Intraday-only: reuse the daily start-date logic with bufferDays
+    if (config.includedExportTypes.intraday && !config.includedExportTypes.daily) {
+      return `${getStartDate()}-${bufferDays}`;
+    }
+  };
+
+  const getIntradayFilter = () => {
+    const intradayStart = getIntradayStartDate();
+    const suffixFilter = ga4ExportDateFilter('intraday', intradayStart, end);
+
+    // When fresh is also enabled, add timestamp condition to avoid duplicating fresh data.
+    // Applied even in test mode because fresh and intraday tables coexist for the same days.
+    if (config.includedExportTypes.fresh) {
+      return `(${suffixFilter} and event_timestamp > coalesce(${constants.FRESH_MAX_EVENT_TIMESTAMP_VARIABLE}, 0))`;
+    }
+
+    return suffixFilter;
+  };
+
+  const dailyStart = `${getStartDate()}-${bufferDays}`;
+  const freshStart = getFreshStartDate();
+  const end = getEndDate();
+
+  const dateFilters = [
+    config.includedExportTypes.daily ? ga4ExportDateFilter('daily', dailyStart, end) : null,
+    config.includedExportTypes.fresh ? ga4ExportDateFilter('fresh', freshStart, end) : null,
+    config.includedExportTypes.intraday ? getIntradayFilter() : null,
+  ];
+
+  return `(
+    ${dateFilters.filter(filter => !!filter).join(' or ')}
+  )`;
+};
+
+/**
+ * Generates a SQL filter condition for restricting event data to a specific date range.
+ *
+ * This function is used to dynamically create a WHERE clause for filtering the `event_date`
+ * based on the provided configuration. It handles three primary scenarios:
+ *   1. **Test Mode (`config.test`)**: Uses explicit start and end dates from the test configuration.
+ *   2. **Incremental Refresh (`config.incremental`)**: Uses BigQuery variable placeholders
+ *      for efficient incremental queries (`constants.DATE_RANGE_START_VARIABLE` and
+ *      `constants.DATE_RANGE_END_VARIABLE`).
+ *   3. **Full Refresh (default)**: Uses static start and end dates from the standard config,
+ *      generally for full table rebuilds.
+ *
+ * This behavior ensures that query cost estimation in BigQuery remains accurate by avoiding
+ * variable use in non-incremental queries.
+ *
+ * @param {Object} config - Configuration object controlling the date filter logic.
+ *   @param {boolean} [config.test] - If true, uses explicit test dates.
+ *   @param {Object} [config.testConfig] - Contains `dateRangeStart` and `dateRangeEnd` for testing.
+ *   @param {boolean} [config.incremental] - If true, uses variable placeholders for incremental queries.
+ *   @param {Object} [config.preOperations] - Contains full refresh date range values.
+ * @returns {string} - SQL condition string to filter the query by date range.
+ */
+const incrementalDateFilter = (config) => {
+  const setDateRange = (start, end) => {
+    return `(event_date >= ${start} and event_date <= ${end})`;
+  };
+
+  // test mode
+  if (config.test) {
+    const testStart = config?.testConfig?.dateRangeStart || baseConfig.testConfig.dateRangeStart;
+    const testEnd = config?.testConfig?.dateRangeEnd || baseConfig.testConfig.dateRangeEnd;
+
+    return setDateRange(testStart, testEnd);
+  }
+
+  // incremental mode
+  if (config.incremental) {
+    return setDateRange(constants.DATE_RANGE_START_VARIABLE, constants.DATE_RANGE_END_VARIABLE);
+  }
+
+  // full refresh mode
+  const fullRefreshStart = config?.preOperations?.dateRangeStartFullRefresh || baseConfig.preOperations.dateRangeStartFullRefresh;
+  const fullRefreshEnd = config?.preOperations?.numberOfDaysToProcess !== undefined
+    ? `least(${fullRefreshStart}+${config.preOperations.numberOfDaysToProcess}-1, current_date())`
+    : (config?.preOperations?.dateRangeEnd || baseConfig.preOperations.dateRangeEnd);
+
+  return setDateRange(fullRefreshStart, fullRefreshEnd);
+};
+
+module.exports = {
+  ga4ExportDateFilter,
+  ga4ExportDateFilters,
+  incrementalDateFilter
+};

package/helpers/dateTime.js

@@ -0,0 +1,58 @@
+/*
+Date and time
+*/
+
+/**
+ * SQL expression that casts the GA4 `event_date` string column to a DATE using YYYYMMDD format.
+ */
+const eventDate = `cast(event_date as date format 'YYYYMMDD')`;
+
+/**
+ * Returns a SQL expression for the event timestamp in microseconds.
+ *
+ * If a custom event parameter is provided (e.g., a parameter collected as a JavaScript timestamp in milliseconds using Date.now()),
+ * this function will attempt to extract its value (via event_params) and convert it to microseconds by multiplying by 1000.
+ * If the custom parameter is not present or null, the function falls back to the default 'event_timestamp' field.
+ *
+ * Usage of customTimestampParameter is intended for event parameters that carry a JS timestamp in milliseconds (for example, set using Date.now()).
+ *
+ * @param {string} [customTimestampParameter] - Name of an event parameter containing a JS timestamp in milliseconds (e.g., collected via Date.now()).
+ * @returns {string} SQL expression for the event timestamp in microseconds.
+ */
+const getEventTimestampMicros = (customTimestampParameter) => {
+  if (typeof customTimestampParameter !== 'undefined' && (typeof customTimestampParameter !== 'string' || customTimestampParameter.trim() === '')) {
+    throw new Error("getEventTimestampMicros: customTimestampParameter must be undefined or a non-empty string.");
+  }
+
+  if (customTimestampParameter) {
+    return `coalesce((select value.int_value from unnest(event_params) where key = '${customTimestampParameter}')*1000, event_timestamp)`;
+  }
+  return 'event_timestamp';
+};
+
+/**
+ * Returns a SQL expression representing the event's local datetime (in the specified time zone),
+ * derived from the default event_timestamp field.
+ *
+ * - This function always uses the exported GA4 event_timestamp (in microseconds) for datetime calculation.
+ * - No custom timestamp parameter from event_params is used; the extraction is strictly from event_timestamp.
+ * - The returned expression converts event_timestamp to a TIMESTAMP, then extracts the DATETIME in the desired time zone.
+ *
+ * @param {Object} [config] - Optional configuration with a timezone property (defaults to 'Etc/UTC').
+ * @param {string} [config.timezone] - IANA time zone string (e.g., 'Europe/Helsinki'). Defaults to 'Etc/UTC'.
+ * @returns {string} SQL expression for the local datetime of the event.
+ *
+ * @example
+ *   getEventDateTime({ timezone: 'Europe/Helsinki' })
+ *   // => "extract(datetime from timestamp_micros(event_timestamp) at time zone 'Europe/Helsinki')"
+ */
+const getEventDateTime = (config) => {
+  const timezone = config?.timezone || 'Etc/UTC';
+  return `extract(datetime from timestamp_micros(${getEventTimestampMicros()}) at time zone '${timezone}')`;
+};
+
+module.exports = {
+  eventDate,
+  getEventTimestampMicros,
+  getEventDateTime
+};