@ken-e/dataform-youtube 0.0.10 → 0.0.12

package/CLAUDE.md

@@ -0,0 +1,228 @@
+# CLAUDE.md — dataform-youtube
+
+## Overview
+
+`@ken-e/dataform-youtube` is a Dataform package that processes YouTube analytics data from Google's BigQuery Data Transfer Service. It transforms raw partitioned transfer tables into clean, enriched output tables ready for downstream reporting and analysis.
+
+The package depends on `@ken-e/dataform-helpers` for incremental partition detection logic.
+
+## Architecture
+
+### Two-layer processing model
+
+1. **Staging layer** (`stg_ytc_*`): Reads raw BQ Data Transfer partitioned tables, normalizes columns, and handles the a2/a3 table version migration via UNION ALL. All staging tables are incremental, partitioned by `data_date`.
+
+2. **Output layer** (`ytc_*`): Reads from staging tables, joins lookup/dimension tables for human-readable names, computes derived metrics (e.g., `subscribers_net`, `row_max_duration_seconds`), deduplicates via `QUALIFY rank() OVER (PARTITION BY data_date, site_nm ORDER BY updated_at DESC) = 1`, and joins video/playlist titles.
+
+### Incremental processing
+
+Both layers use partition-based checkpointing provided by `@ken-e/dataform-helpers`:
+
+- **Staging preops** (`BQ_DATA_TRANSFER_PARTITION_STAGING_PREOPS`): Queries `INFORMATION_SCHEMA.PARTITIONS` on the raw source tables to find partitions modified since the last run. Populates a `partitions_to_update` array used in WHERE clauses.
+- **Output preops** (`BQ_DATA_TRANSFER_PARTITION_OUTPUT_PREOPS`): Queries `INFORMATION_SCHEMA.PARTITIONS` on the staging table to find updated partitions.
+
+On full refresh, the `self_checkpoint` is set to `config.startDate` and all partitions after that date are processed.
+
+## Directory structure
+
+```
+index.js                          Main entry point — config API, source table declarations, model wiring
+includes/
+  helpers.js                      Thin wrapper re-exporting preops from @ken-e/dataform-helpers
+  column_descriptions.js          Shared column descriptions applied to all output tables
+  constants.js                    Package-level constants (currently unused)
+  project_variables.js            Template file (currently unused)
+  definitions/
+    ytc_basic.js                  Output: channel basic metrics
+    ytc_combined.js               Output: channel combined dimensions
+    ytc_demographics.js           Output: age/gender breakdowns
+    ytc_demographics_views.js     Output: demographics with view counts
+    ytc_device_os.js              Output: device and OS breakdowns
+    ytc_playback.js               Output: playback location
+    ytc_province.js               Output: geographic (province/state)
+    ytc_traffic_source.js         Output: traffic source analysis
+    ytc_share_platform.js         Output: sharing service breakdowns
+    ytc_annotation.js             Output: annotation performance
+    ytc_cards.js                  Output: card performance
+    ytc_end_screens.js            Output: end screen element performance
+    ytc_subtitles.js              Output: subtitle/caption analytics
+    ytc_list_basic.js             Output: playlist basic metrics
+    ytc_list_combined.js          Output: playlist combined dimensions
+    ytc_list_device_os.js         Output: playlist device/OS
+    ytc_list_playback.js          Output: playlist playback location
+    ytc_list_province.js          Output: playlist geographic
+    ytc_list_traffic_source.js    Output: playlist traffic source
+    sources/
+      stg_ytc_*.js                Staging models (13 with dual-table UNION ALL, 5 single-table)
+      stg_ytc_lu_*.js             Lookup/dimension tables (8 static reference tables)
+```
+
+## Configuration API
+
+The package exports a single function that accepts a `config` object:
+
+```javascript
+const youtube = require("@ken-e/dataform-youtube");
+
+youtube({
+  startDate: "2025-01-01",
+  days_back: 10,
+
+  // Video and playlist title lookup tables
+  titlesProject: "my-project",
+  titlesDataset: "my_dataset",
+  titlesTable: "stg_yt_video_titles",
+  playlistTable: "stg_yt_playlist_titles",
+
+  // One or more YouTube data sources
+  sources: [{
+    database: "my-bq-project",
+    schema: "youtube_channel",
+    suffix: "_"
+  }],
+
+  // Target schemas for staging and output layers
+  target: {
+    database: dataform.projectConfig.defaultDatabase,
+    stagingSchema: "my_staging_dataset",
+    outputSchema: "my_output_dataset"
+  }
+});
+```
+
+### Required config fields
+
+| Field | Description |
+|---|---|
+| `sources` | Array of `{ database, schema, suffix }` objects pointing to raw BQ Data Transfer datasets |
+| `target.stagingSchema` | BigQuery dataset for staging tables |
+| `target.outputSchema` | BigQuery dataset for output tables |
+| `startDate` | Earliest date to process on full refresh (YYYY-MM-DD) |
+| `titlesProject` | Project containing the video titles lookup table |
+| `titlesDataset` | Dataset containing the video titles lookup table |
+| `titlesTable` | Table name for video titles |
+| `playlistTable` | Table name for playlist titles |
+
+## Source table versioning (a2 to a3 migration)
+
+### Background
+
+On September 22, 2025, Google's BigQuery Data Transfer Service deprecated several YouTube source table versions and introduced new ones:
+
+- **Channel reports**: `a2` tables replaced by `a3` tables
+- **Playlist reports**: `a1` tables replaced by `a2` tables
+
+The old tables stopped receiving new data around mid-November 2025. The new tables include one additional column (`engaged_views`) and use a different view-counting methodology that results in significantly higher view counts.
+
+### Official documentation
+
+- YouTube Channel Transfer overview: https://docs.cloud.google.com/bigquery/docs/youtube-channel-transfer
+- BQ Data Transfer change log: https://docs.cloud.google.com/bigquery/docs/transfer-changes
+
+### Implementation
+
+The 13 affected staging models use `.flatMap()` to produce two SELECT blocks per source, joined with UNION ALL:
+
+- **Old table block**: Filters to `date(_PARTITIONTIME) < date '2025-09-22'`, casts `engaged_views` as null
+- **New table block**: Filters to `date(_PARTITIONTIME) >= date '2025-09-22'`, selects `engaged_views` natively
+
+The cutover date `2025-09-22` is hardcoded in each staging model. On incremental runs, the date filter is also applied to prevent duplicate rows when both old and new tables have partitions for overlapping dates.
+
+The preops helper accepts an array of table names (e.g., `["p_channel_basic_a2_", "p_channel_basic_a3_"]`) to detect modified partitions across both table versions.
+
+### Data discontinuity warning
+
+The a3 source tables report significantly higher view counts than the a2 tables for the same dates. Our analysis of the `p_channel_basic` table for October 15, 2025 showed:
+
+- a2 table: 18,892 total views
+- a3 table: 119,044 total views (~6.3x higher)
+
+This is a Google-side methodology change, not a data processing error. View counts (and likely other metrics) are **not directly comparable** across the September 22, 2025 cutover boundary. Downstream reports should account for this discontinuity.
+
+### Tables affected by the migration
+
+| Old (deprecated) | New | Type |
+|---|---|---|
+| `p_channel_basic_a2_` | `p_channel_basic_a3_` | Channel |
+| `p_channel_combined_a2_` | `p_channel_combined_a3_` | Channel |
+| `p_channel_device_os_a2_` | `p_channel_device_os_a3_` | Channel |
+| `p_channel_playback_location_a2_` | `p_channel_playback_location_a3_` | Channel |
+| `p_channel_province_a2_` | `p_channel_province_a3_` | Channel |
+| `p_channel_subtitles_a2_` | `p_channel_subtitles_a3_` | Channel |
+| `p_channel_traffic_source_a2_` | `p_channel_traffic_source_a3_` | Channel |
+| `p_playlist_basic_a1_` | `p_playlist_basic_a2_` | Playlist |
+| `p_playlist_combined_a1_` | `p_playlist_combined_a2_` | Playlist |
+| `p_playlist_device_os_a1_` | `p_playlist_device_os_a2_` | Playlist |
+| `p_playlist_playback_location_a1_` | `p_playlist_playback_location_a2_` | Playlist |
+| `p_playlist_province_a1_` | `p_playlist_province_a2_` | Playlist |
+| `p_playlist_traffic_source_a1_` | `p_playlist_traffic_source_a2_` | Playlist |
+
+**Unchanged** (no version bump): `p_channel_demographics_a1_`, `p_channel_sharing_service_a1_`, `p_channel_annotations_a1_`, `p_channel_cards_a1_`, `p_channel_end_screens_a1_`
+
+## Output tables
+
+### Channel reports
+
+| Table | Description | Key dimensions |
+|---|---|---|
+| `ytc_basic` | Core video metrics (views, likes, comments, shares, watch time) | video_id, country_code |
+| `ytc_combined` | Metrics broken down by all major dimensions simultaneously | video_id, playback_location, traffic_source, device, OS |
+| `ytc_demographics` | Age group and gender breakdowns (percentage-based) | video_id, age_group, gender |
+| `ytc_demographics_views` | Demographics with absolute view counts (joined from basic) | video_id, age_group, gender |
+| `ytc_device_os` | Device type and operating system breakdowns | video_id, device_type, operating_system |
+| `ytc_playback` | Playback location breakdowns | video_id, playback_location_type |
+| `ytc_province` | Province/state-level geographic breakdowns | video_id, province_code |
+| `ytc_traffic_source` | Traffic source breakdowns | video_id, traffic_source_type |
+| `ytc_share_platform` | Social sharing service breakdowns | video_id, sharing_service |
+| `ytc_annotation` | Annotation performance metrics | video_id, annotation_type |
+| `ytc_cards` | Card performance metrics | video_id, card_type |
+| `ytc_end_screens` | End screen element performance | video_id, end_screen_element_type |
+| `ytc_subtitles` | Subtitle/caption analytics | video_id, subtitle_language |
+
+### Playlist reports
+
+| Table | Description | Key dimensions |
+|---|---|---|
+| `ytc_list_basic` | Core playlist metrics | playlist_id, video_id, country_code |
+| `ytc_list_combined` | Playlist metrics by all dimensions | playlist_id, video_id, playback_location, traffic_source, device, OS |
+| `ytc_list_device_os` | Playlist device/OS breakdowns | playlist_id, video_id, device_type, operating_system |
+| `ytc_list_playback` | Playlist playback location breakdowns | playlist_id, video_id, playback_location_type |
+| `ytc_list_province` | Playlist province-level breakdowns | playlist_id, video_id, province_code |
+| `ytc_list_traffic_source` | Playlist traffic source breakdowns | playlist_id, video_id, traffic_source_type |
+
+### Lookup tables
+
+| Table | Enriches | Provides |
+|---|---|---|
+| `stg_ytc_lu_annotation_type` | annotation_type (int) | annotation_type_name |
+| `stg_ytc_lu_card_type` | card_type (int) | card_type_name |
+| `stg_ytc_lu_device_types` | device_type (int) | device_name |
+| `stg_ytc_lu_end_screen_element_type` | end_screen_element_type (int) | end_screen_element_type_name |
+| `stg_ytc_lu_operating_systems` | operating_system (int) | operating_system_name |
+| `stg_ytc_lu_playback_location` | playback_location_type (int) | playback_location_name |
+| `stg_ytc_lu_sharing_services` | sharing_service (int) | sharing_service_name |
+| `stg_ytc_lu_traffic_sources` | traffic_source_type (int) | traffic_source_name |
+
+## Development notes
+
+### Adding a new column to staging models
+
+1. Add the column to the appropriate `columns` or `restColumns` template literal in the staging model
+2. If the column exists only in the new (a3) table version, add `cast(null as <type>) as <column>` to the old table block and the raw column name to the new table block
+3. Add a description in `includes/column_descriptions.js`
+
+### Adding a new output model
+
+1. Create a new file in `includes/definitions/` following the pattern of existing output models
+2. Wire it up in `index.js` by adding it to the model instantiation section
+3. Ensure it uses `helpers.output_preops()` for incremental processing
+4. Include `QUALIFY rank() OVER (...) = 1` for deduplication
+
+### The `.flatMap()` dual-table pattern
+
+Staging models that span the a2/a3 migration use `.flatMap()` instead of `.map()` to return two SELECT blocks per source. Each block targets a different table version with complementary date filters. The preops helper accepts an array of table names to detect modified partitions across both versions.
+
+### Known issues
+
+- `config.target.datasetStaging` is used in staging models but some consumer configs set `config.target.stagingSchema` — ensure your config uses the property name expected by the staging models
+- The `daysBack` / `days_back` parameter is declared in consumer configs but currently unused by the package

package/README.md

@@ -1,4 +1,85 @@
-# Propeller - Youtube Dataform Package
+# @ken-e/dataform-youtube
 
-## Overview
-Welcome to the **Youtube** Dataform project.
+A [Dataform](https://dataform.co/) package for processing YouTube analytics data from Google's [BigQuery Data Transfer Service](https://docs.cloud.google.com/bigquery/docs/youtube-channel-transfer).
+
+## Installation
+
+```bash
+npm install @ken-e/dataform-youtube
+```
+
+## Quick start
+
+```javascript
+const youtube = require("@ken-e/dataform-youtube");
+
+youtube({
+  startDate: "2025-01-01",
+
+  // Video and playlist title lookup tables
+  titlesProject: "my-project",
+  titlesDataset: "my_dataset",
+  titlesTable: "stg_yt_video_titles",
+  playlistTable: "stg_yt_playlist_titles",
+
+  // YouTube data sources (one or more)
+  sources: [{
+    database: "my-bq-project",
+    schema: "youtube_channel",
+    suffix: "_"
+  }],
+
+  // Target BigQuery datasets
+  target: {
+    database: dataform.projectConfig.defaultDatabase,
+    stagingSchema: "my_staging_dataset",
+    outputSchema: "my_output_dataset"
+  }
+});
+```
+
+## Output tables
+
+### Channel reports
+
+| Table | Description |
+|---|---|
+| `ytc_basic` | Core video metrics: views, likes, comments, shares, watch time, subscribers |
+| `ytc_combined` | Metrics by all major dimensions (location, traffic source, device, OS) |
+| `ytc_demographics` | Age group and gender breakdowns |
+| `ytc_demographics_views` | Demographics with absolute view counts |
+| `ytc_device_os` | Device type and operating system breakdowns |
+| `ytc_playback` | Playback location breakdowns |
+| `ytc_province` | Province/state-level geographic breakdowns |
+| `ytc_traffic_source` | Traffic source breakdowns |
+| `ytc_share_platform` | Social sharing service breakdowns |
+| `ytc_annotation` | Annotation performance metrics |
+| `ytc_cards` | Card performance metrics |
+| `ytc_end_screens` | End screen element performance |
+| `ytc_subtitles` | Subtitle/caption analytics |
+
+### Playlist reports
+
+| Table | Description |
+|---|---|
+| `ytc_list_basic` | Core playlist metrics |
+| `ytc_list_combined` | Playlist metrics by all dimensions |
+| `ytc_list_device_os` | Playlist device/OS breakdowns |
+| `ytc_list_playback` | Playlist playback location breakdowns |
+| `ytc_list_province` | Playlist province-level breakdowns |
+| `ytc_list_traffic_source` | Playlist traffic source breakdowns |
+
+## Data source migration (September 2025)
+
+On September 22, 2025, Google deprecated several YouTube BQ Data Transfer table versions (`a2` channel tables and `a1` playlist tables) and replaced them with new versions (`a3` and `a2` respectively). This package handles the migration transparently by reading from both old and new tables with a date-based cutover.
+
+**Important**: Google changed the view-counting methodology in the new tables. View counts after September 22, 2025 are significantly higher than before and are not directly comparable across the cutover boundary. See the [BQ Data Transfer change log](https://docs.cloud.google.com/bigquery/docs/transfer-changes) for details.
+
+## Documentation
+
+See [CLAUDE.md](./CLAUDE.md) for detailed development documentation including architecture, configuration reference, and the source table migration guide.
+
+## Google documentation
+
+- [YouTube Channel Reports in BigQuery](https://docs.cloud.google.com/bigquery/docs/youtube-channel-transfer)
+- [BigQuery Data Transfer change log](https://docs.cloud.google.com/bigquery/docs/transfer-changes)

package/includes/column_descriptions.js

@@ -5,6 +5,7 @@
 
 const column_descriptions = {
   ad_impressions: "The number of verified ad impressions served.",
+  data_date: "The date of the reported metrics, parsed from the raw date field.",
   ad_type:
     "The type of ad displayed (e.g., auction_display, auction_instream, reserved_instream).",
   age_group:
@@ -52,6 +53,8 @@ const column_descriptions = {
   device_type:
     "The type of device used by the viewer (e.g., DESKTOP, MOBILE, TABLET, TV, GAME_CONSOLE).",
   dislikes: "The number of times users disliked videos (negative ratings).",
+  engaged_views:
+    "The number of engaged views. Available only for data after September 22, 2025 (null for earlier data). Note: Google changed the view-counting methodology when migrating from a2 to a3 tables, so total view counts are significantly higher in post-cutover data and are not directly comparable to pre-cutover figures.",
   end_screen_element_id: "Unique identifier for the end screen element.",
   end_screen_element_type:
     "Integer index indicating the type of end screen element.",
@@ -81,10 +84,16 @@ const column_descriptions = {
   playback_based_cpm:
     "Estimated gross revenue per 1000 monetized playbacks (Playback-based CPM, USD).",
   playback_location: "Type of location where playback occured (text).",
+  playback_location_detail:
+    "Specific detail about the playback location (e.g., embedded page URL).",
   playback_location_type:
     "Integer index indicating type of location where playback occured.",
   playback_location_name: "String name of the playback location.",
   playlist_id: "Unique identifier for the YouTube playlist.",
+  playlist_saves_added:
+    "The number of times videos were saved (added) to playlists.",
+  playlist_saves_removed:
+    "The number of times videos were removed from playlists.",
   playlist_saves_net:
     "The number of times users saved videos to their playlists less the number of times they removed them.",
   playlist_starts: "Times the playlist was started.",
@@ -92,9 +101,15 @@ const column_descriptions = {
   province_code:
     "The ISO code for the province or state where the interactions occurred.",
   red_views: "The number of times YouTube Premium members viewed videos.",
+  red_watch_time_minutes:
+    "The number of minutes that YouTube Premium members watched videos.",
   row_max_duration_seconds:
     "Maximum duration in seconds for the row's if all videos were played to the end. Provided simplify downstream calculation of percentage of videos watched.",
   shares: "The number of times users shared videos via the 'Share' button.",
+  site_nm:
+    "Identifier for the data source, derived from the source schema name.",
+  source_partition_date:
+    "The partition date from the raw BigQuery Data Transfer source table.",
   sharing_service:
     "The service used to share the video (e.g., FACEBOOK, WHATSAPP, REDDIT, EMAIL).",
   sharing_service_name: "String name of the sharing service.",
@@ -112,6 +127,8 @@ const column_descriptions = {
   traffic_source_name: "String name of the traffic source type.",
   traffic_source_type:
     "The type of referrer through which viewers reached the video (e.g., YT_SEARCH, RELATED_VIDEO, EXT_URL, SUBSCRIBER).",
+  updated_at:
+    "Timestamp of when the staging row was created during processing.",
   uploader_type:
     "Indicates if metrics relate to content uploaded by the owner ('SELF') or claimed third-party content ('THIRD_PARTY').",
   video_id: "The ID of a YouTube video.",
@@ -121,7 +138,7 @@ const column_descriptions = {
   videos_removed_from_playlists:
     "The number of times videos were removed from any YouTube playlist (owner's or others').",
   views:
-    "The total number of times videos were viewed. In playlist reports, counts views within the playlist context.",
+    "The total number of times videos were viewed. In playlist reports, counts views within the playlist context. Note: Google changed the view-counting methodology in the a3 source tables effective September 22, 2025. View counts after this date are significantly higher than before and are not directly comparable across the cutover boundary.",
   views_percentage:
     "Percentage of total views for the time period that were logged-in.",
   views_with_demographics:

package/includes/definitions/sources/stg_ytc_basic.js

@@ -39,23 +39,23 @@ module.exports = (config) => {
   })
     .preOps((ctx) => {
       // Get pre_operations to find updated source partitions
-      return `${helpers.staging_preops(ctx, config, "p_channel_basic_a2_")}`;
+      return `${helpers.staging_preops(ctx, config, ["p_channel_basic_a2_", "p_channel_basic_a3_"])}`;
     })
     .query((ctx) =>
       config.sources
-        .map((t) => {
-          return `
-select 
+        .flatMap((t) => {
+          const columns = `
   _PARTITIONDATE as source_partition_date,
   parse_date('%Y%m%d', date) as data_date,
   "${t.schema}" as site_nm,
-  current_timestamp() as updated_at,  
+  current_timestamp() as updated_at,
   channel_id,
   video_id,
   live_or_on_demand,
   subscribed_status,
   country_code,
-  views,
+  views,`;
+          const restColumns = `
   comments,
   likes,
   dislikes,
@@ -81,14 +81,32 @@ select
   videos_added_to_playlists,
   videos_removed_from_playlists,
   red_views,
-  red_watch_time_minutes,
-from ${ctx.ref(t.database, t.schema, "p_channel_basic_a2_" + t.suffix)} 
+  red_watch_time_minutes,`;
+          const incrementalWhere = `where date(_PARTITIONTIME) in unnest((select partition_dates from unnest(partitions_to_update) where site_nm = "${t.schema}"))`;
+          return [
+            `
+-- Deprecated a2 table (data before cutover)
+select ${columns}
+  cast(null as int64) as engaged_views,${restColumns}
+from ${ctx.ref(t.database, t.schema, "p_channel_basic_a2_" + t.suffix)}
     ${ctx.when(
       ctx.incremental(),
-      `where date(_PARTITIONTIME) in unnest((select partition_dates from unnest(partitions_to_update) where site_nm = "${t.schema}"))`,
-      `where date(_PARTITIONTIME) > date(self_checkpoint)`,
+      `${incrementalWhere} and date(_PARTITIONTIME) < date '2025-09-22'`,
+      `where date(_PARTITIONTIME) > date(self_checkpoint) and date(_PARTITIONTIME) < date '2025-09-22'`,
     )}
-`;
+`,
+            `
+-- New a3 table (data from cutover onward)
+select ${columns}
+  engaged_views,${restColumns}
+from ${ctx.ref(t.database, t.schema, "p_channel_basic_a3_" + t.suffix)}
+    ${ctx.when(
+      ctx.incremental(),
+      `${incrementalWhere} and date(_PARTITIONTIME) >= date '2025-09-22'`,
+      `where date(_PARTITIONTIME) > date(self_checkpoint) and date(_PARTITIONTIME) >= date '2025-09-22'`,
+    )}
+`,
+          ];
         })
         .join(" union all "),
     );

package/includes/definitions/sources/stg_ytc_combined.js

@@ -43,40 +43,57 @@ module.exports = (config) => {
   })
     .preOps((ctx) => {
       // Get pre_operations to find updated source partitions
-      return `${helpers.staging_preops(ctx, config, "p_channel_combined_a2_")}`;
+      return `${helpers.staging_preops(ctx, config, ["p_channel_combined_a2_", "p_channel_combined_a3_"])}`;
     })
     .query((ctx) =>
       config.sources
-        .map((t) => {
-          return `
-
-select 
+        .flatMap((t) => {
+          const columns = `
   _PARTITIONDATE as source_partition_date,
   parse_date('%Y%m%d', date) as data_date,
   "${t.schema}" as site_nm,
-  current_timestamp() as updated_at,  
+  current_timestamp() as updated_at,
   channel_id,
   video_id,
   live_or_on_demand,
   subscribed_status,
   country_code,
-  playback_location_type, 
-  traffic_source_type, 
-  device_type, 
+  playback_location_type,
+  traffic_source_type,
+  device_type,
   operating_system,
-  views,
+  views,`;
+          const restColumns = `
   watch_time_minutes,
   average_view_duration_seconds,
   average_view_duration_percentage,
   red_views,
-  red_watch_time_minutes,
-from ${ctx.ref(t.database, t.schema, "p_channel_combined_a2_" + t.suffix)} 
+  red_watch_time_minutes,`;
+          const incrementalWhere = `where date(_PARTITIONTIME) in unnest((select partition_dates from unnest(partitions_to_update) where site_nm = "${t.schema}"))`;
+          return [
+            `
+-- Deprecated a2 table (data before cutover)
+select ${columns}
+  cast(null as int64) as engaged_views,${restColumns}
+from ${ctx.ref(t.database, t.schema, "p_channel_combined_a2_" + t.suffix)}
+    ${ctx.when(
+      ctx.incremental(),
+      `${incrementalWhere} and date(_PARTITIONTIME) < date '2025-09-22'`,
+      `where date(_PARTITIONTIME) > date(self_checkpoint) and date(_PARTITIONTIME) < date '2025-09-22'`,
+    )}
+`,
+            `
+-- New a3 table (data from cutover onward)
+select ${columns}
+  engaged_views,${restColumns}
+from ${ctx.ref(t.database, t.schema, "p_channel_combined_a3_" + t.suffix)}
     ${ctx.when(
       ctx.incremental(),
-      `where date(_PARTITIONTIME) in unnest((select partition_dates from unnest(partitions_to_update) where site_nm = "${t.schema}"))`,
-      `where date(_PARTITIONTIME) > date(self_checkpoint)`,
+      `${incrementalWhere} and date(_PARTITIONTIME) >= date '2025-09-22'`,
+      `where date(_PARTITIONTIME) > date(self_checkpoint) and date(_PARTITIONTIME) >= date '2025-09-22'`,
     )}
-`;
+`,
+          ];
         })
         .join(" union all "),
     );

package/includes/definitions/sources/stg_ytc_device_os.js

@@ -39,14 +39,12 @@ module.exports = (config) => {
   })
     .preOps((ctx) => {
       // Get pre_operations to find updated source partitions
-      return `${helpers.staging_preops(ctx, config, "p_channel_device_os_a2_")}`;
+      return `${helpers.staging_preops(ctx, config, ["p_channel_device_os_a2_", "p_channel_device_os_a3_"])}`;
     })
     .query((ctx) =>
       config.sources
-        .map((t) => {
-          return `
-
-select
+        .flatMap((t) => {
+          const columns = `
   _PARTITIONDATE as source_partition_date,
   parse_date('%Y%m%d', date) as data_date,
   "${t.schema}" as site_nm,
@@ -58,19 +56,38 @@ select
   country_code,
   device_type,
   operating_system,
-  views,
+  views,`;
+          const restColumns = `
   watch_time_minutes,
   average_view_duration_seconds,
   average_view_duration_percentage,
   red_views,
-  red_watch_time_minutes
+  red_watch_time_minutes`;
+          const incrementalWhere = `where date(_PARTITIONTIME) in unnest((select partition_dates from unnest(partitions_to_update) where site_nm = "${t.schema}"))`;
+          return [
+            `
+-- Deprecated a2 table (data before cutover)
+select ${columns}
+  cast(null as int64) as engaged_views,${restColumns}
 from ${ctx.ref(t.database, t.schema, "p_channel_device_os_a2_" + t.suffix)}
     ${ctx.when(
       ctx.incremental(),
-      `where date(_PARTITIONTIME) in unnest((select partition_dates from unnest(partitions_to_update) where site_nm = "${t.schema}"))`,
-      `where date(_PARTITIONTIME) > date(self_checkpoint)`,
+      `${incrementalWhere} and date(_PARTITIONTIME) < date '2025-09-22'`,
+      `where date(_PARTITIONTIME) > date(self_checkpoint) and date(_PARTITIONTIME) < date '2025-09-22'`,
+    )}
+`,
+            `
+-- New a3 table (data from cutover onward)
+select ${columns}
+  engaged_views,${restColumns}
+from ${ctx.ref(t.database, t.schema, "p_channel_device_os_a3_" + t.suffix)}
+    ${ctx.when(
+      ctx.incremental(),
+      `${incrementalWhere} and date(_PARTITIONTIME) >= date '2025-09-22'`,
+      `where date(_PARTITIONTIME) > date(self_checkpoint) and date(_PARTITIONTIME) >= date '2025-09-22'`,
     )}
-`;
+`,
+          ];
         })
         .join(" union all "),
     );

package/includes/definitions/sources/stg_ytc_list_basic.js

@@ -40,37 +40,54 @@ module.exports = (config) => {
   })
     .preOps((ctx) => {
       // Get pre_operations to find updated source partitions
-      return `${helpers.staging_preops(ctx, config, "p_playlist_basic_a1_")}`;
+      return `${helpers.staging_preops(ctx, config, ["p_playlist_basic_a1_", "p_playlist_basic_a2_"])}`;
     })
     .query((ctx) =>
       config.sources
-        .map((t) => {
-          return `
-
-select 
+        .flatMap((t) => {
+          const columns = `
   _PARTITIONDATE as source_partition_date,
   parse_date('%Y%m%d', date) as data_date,
   "${t.schema}" as site_nm,
-  current_timestamp() as updated_at,  
+  current_timestamp() as updated_at,
   channel_id,
   playlist_id,
   video_id,
   live_or_on_demand,
   subscribed_status,
   country_code,
-  views,
+  views,`;
+          const restColumns = `
   watch_time_minutes,
   average_view_duration_seconds,
   playlist_starts,
   playlist_saves_added,
-  playlist_saves_removed
-from ${ctx.ref(t.database, t.schema, "p_playlist_basic_a1_" + t.suffix)} 
+  playlist_saves_removed`;
+          const incrementalWhere = `where date(_PARTITIONTIME) in unnest((select partition_dates from unnest(partitions_to_update) where site_nm = "${t.schema}"))`;
+          return [
+            `
+-- Deprecated a1 table (data before cutover)
+select ${columns}
+  cast(null as int64) as engaged_views,${restColumns}
+from ${ctx.ref(t.database, t.schema, "p_playlist_basic_a1_" + t.suffix)}
+    ${ctx.when(
+      ctx.incremental(),
+      `${incrementalWhere} and date(_PARTITIONTIME) < date '2025-09-22'`,
+      `where date(_PARTITIONTIME) > date(self_checkpoint) and date(_PARTITIONTIME) < date '2025-09-22'`,
+    )}
+`,
+            `
+-- New a2 table (data from cutover onward)
+select ${columns}
+  engaged_views,${restColumns}
+from ${ctx.ref(t.database, t.schema, "p_playlist_basic_a2_" + t.suffix)}
     ${ctx.when(
       ctx.incremental(),
-      `where date(_PARTITIONTIME) in unnest((select partition_dates from unnest(partitions_to_update) where site_nm = "${t.schema}"))`,
-      `where date(_PARTITIONTIME) > date(self_checkpoint)`,
+      `${incrementalWhere} and date(_PARTITIONTIME) >= date '2025-09-22'`,
+      `where date(_PARTITIONTIME) > date(self_checkpoint) and date(_PARTITIONTIME) >= date '2025-09-22'`,
     )}
-`;
+`,
+          ];
         })
         .join(" union all "),
     );