@clickzetta/cz-cli-darwin-x64 0.3.95 → 0.3.96

package/bin/skills/lakehouse-doc-en/references/realtime-pipeline-selection-guide.md

@@ -0,0 +1,348 @@
+# Real-Time Data Pipeline Selection Guide
+
+Singdata Lakehouse provides three types of real-time data processing objects: **Pipe** (continuous ingestion), **Table Stream** (change data capture), and **Dynamic Table** (incremental computation). They solve different problems and are often used together, but choosing the wrong one will lead you down unproductive paths.
+
+This article answers two questions: which object suits which scenario, and how to choose for common business requirements.
+
+---
+
+## Positioning of the Three Object Types
+
+| Object | Problem It Solves | Data Flow Direction | Typical Usage |
+|--------|-------------------|---------------------|---------------|
+| **Pipe** | Continuous flow of external data into Lakehouse | External -> Table | Kafka message ingestion, OSS file loading |
+| **Table Stream** | Perceiving data changes within a table (CDC) | Table -> Downstream processing | Change sync, incremental ETL driver |
+| **Dynamic Table** | Automatically maintaining real-time processing results | Table -> Processed result table | ODS->DWD->DWS incremental processing chain |
+
+These three are not in competition; they are different stages of a pipeline:
+
+```
+External Data Source
+    ↓ Pipe (continuous ingestion)
+  ODS Table
+    ↓ Dynamic Table (incremental processing)
+  DWD / DWS Table
+    ↓ Table Stream (change perception, driving downstream sync)
+  Downstream System / Target Table
+```
+
+---
+
+## Selection Decision Tree
+
+**First question: Where does the data come from?**
+
+- From Kafka or object storage (OSS/S3/COS) → Use **Pipe**
+- Already in a Lakehouse table → See next question
+
+**Second question: What do you want to do?**
+
+- Process and transform table data, keeping results up-to-date in real time → Use **Dynamic Table**
+- Need to perceive row-level changes (INSERT/UPDATE/DELETE) in a table to drive downstream processing → Use **Table Stream**
+- Just a one-time or low-frequency import → Use **COPY INTO**, no Pipe needed
+
+**Third question (for Pipe scenarios): Trigger method?**
+
+- Real-time trigger when files are uploaded (minute-level latency) → **EVENT_NOTIFICATION mode** (requires message queue configuration)
+- No real-time requirement; periodic scanning with original file deletion is acceptable → **LIST_PURGE mode** (simple configuration)
+
+---
+
+## Pipe: Suitable for Continuous Ingestion, Not Batch Import
+
+### Scenarios Suitable for Pipe
+
+- Kafka Topic data continuously writing to Lakehouse tables
+- OSS/S3 files being continuously uploaded with automatic loading needed
+- Data sources produce data continuously, requiring the system to automatically maintain read positions
+
+### Scenarios NOT Suitable for Pipe
+
+- One-time historical data import → Use `COPY INTO` directly
+- Low-frequency (once daily) batch sync → Scheduling `COPY INTO` tasks is simpler
+- Strictly ordered data processing is required → **Pipe does not guarantee load order**
+
+### Pipe vs COPY INTO Comparison
+
+| | Pipe | COPY INTO (Scheduled) |
+|-|------|-----------------------|
+| Suitable Scenario | Data flows continuously; new files auto-processed | Low-frequency batch import, one-time migration |
+| Trigger Method | File event-driven or periodic scanning | Manual or external scheduling system trigger |
+| Read Position Management | System auto-maintains (load_history) | Caller must manage on their own |
+| File Deduplication | Auto dedup (same path + same filename = imported once) | No built-in dedup; repeated execution = repeated import |
+| Ops Complexity | Auto-runs after creation, no intervention needed | Requires schedule config, execution monitoring |
+| Compute Resources | Requires VCluster online | Requires VCluster online |
+
+**File Deduplication Behavior**: Pipe records imported files via `load_history` (retained for 7 days). Files with the same path and filename are imported only once. If you need to re-import a file, you must manually execute `COPY INTO`.
+
+**File Size Recommendations**:
+- gzip-compressed files: recommended within 50MB
+- CSV / Parquet uncompressed: recommended 128MB to 256MB
+
+Files that are too small increase scheduling overhead; files that are too large affect single-batch loading time.
+
+### Pipe: Two Mode Choices
+
+| | EVENT_NOTIFICATION | LIST_PURGE |
+|-|--------------------|------------|
+| Latency | Minute-level (event-triggered) | Depends on scan interval |
+| Configuration Complexity | Requires message queue config (Alibaba Cloud MNS / AWS SQS) | Simple, no extra config needed |
+| Original File Handling | Preserves original files | **Deletes original files after import** |
+| Suitable Scenario | Latency-sensitive, files need to be preserved | Low latency requirement, file deletion acceptable |
+| Cloud Provider Support | Alibaba Cloud OSS, AWS S3 | All object storage |
+
+> ⚠️ **Note**: LIST_PURGE mode deletes source files after successful import. Confirm that your scenario allows deletion of original files before using this mode.
+
+---
+
+## Table Stream: Suitable for Perceiving Changes, Not Direct Queries
+
+### Scenarios Suitable for Table Stream
+
+- Need to capture INSERT / UPDATE / DELETE changes on a source table and sync to a target system
+- Incremental ETL: only process data that is "new/changed since last processing," avoiding full scans
+- Driving SCD (Slowly Changing Dimension) maintenance
+- Listening for OSS file arrival events (Volume Stream)
+
+### Scenarios NOT Suitable for Table Stream
+
+- Only need to query current data → Query the table directly, no Stream needed
+- Only need to capture new data, not updates or deletes → Use **APPEND_ONLY mode**, better performance
+- Need real-time data processing with maintained result updates → Use **Dynamic Table**
+
+### STANDARD vs APPEND_ONLY?
+
+| | STANDARD | APPEND_ONLY |
+|-|---------|-------------|
+| Captured Operations | INSERT, UPDATE, DELETE | INSERT only |
+| UPDATE Representation | UPDATE_BEFORE + UPDATE_AFTER (two rows) | Not recorded (only original insert value) |
+| Suitable Scenario | Need to handle updates and deletes (e.g., MERGE sync) | Append-only log data, pursue performance |
+| Performance | Higher overhead (tracking all changes) | Lighter weight |
+
+### Offset Advancement Rules (Important)
+
+Table Stream offsets advance **only after a DML transaction containing the Stream commits successfully**.
+
+- `SELECT * FROM stream` → **Does not advance** offset; data remains available next query
+- `INSERT INTO target SELECT ... FROM stream` → Offset advances after transaction commit; data is consumed
+- Transaction rollback → No advancement; data remains
+
+This means: if you only SELECT to view Stream data, the offset does not change; you must consume data through DML operations that include the Stream to advance the offset.
+
+### Multi-Consumer Pattern
+
+**One Stream can only be fully consumed by one consumer**. When Task A consumes the Stream via DML, the offset advances, and Task B querying the same Stream will no longer see that batch of changes.
+
+If multiple downstream tasks all need to consume changes from the same table, **create a separate Stream for each consumer**:
+
+```sql
+-- Enable change tracking on the table
+ALTER TABLE orders SET PROPERTIES ('change_tracking' = 'true');
+
+-- Create a Stream for each consumer
+CREATE TABLE STREAM orders_stream_for_dw
+    ON TABLE orders
+    WITH PROPERTIES ('TABLE_STREAM_MODE' = 'STANDARD');
+
+CREATE TABLE STREAM orders_stream_for_notify
+    ON TABLE orders
+    WITH PROPERTIES ('TABLE_STREAM_MODE' = 'APPEND_ONLY');
+```
+
+Each Stream independently maintains its own offset, without interfering with others. Streams only store offsets, not a copy of the data, so the additional cost of creating multiple Streams is very low.
+
+### Data Retention Period and Stream Invalidation
+
+Table Stream relies on the source table's historical version data (Time Travel) to return change records. If a Stream is not consumed for a long time and the source table's historical versions are cleaned up (exceeding `DATA_RETENTION_DAYS`, default 7 days), the Stream will become unavailable.
+
+Daily recommendation: the Stream consumption frequency should be much shorter than the source table's `DATA_RETENTION_DAYS` to avoid Stream invalidation due to historical version loss.
+
+---
+
+## Dynamic Table: Suitable for Declarative Incremental Processing, Not High-Real-Time Requirements
+
+### Scenarios Suitable for Dynamic Table
+
+- Multi-layer ETL processing chains: ODS -> DWD -> DWS
+- Query performance optimization: materialize complex computation results, avoiding recalculation on each query
+- Near-real-time analysis with fixed dimensions, tolerating minute-level latency
+
+### Scenarios NOT Suitable for Dynamic Table
+
+- Sub-second real-time requirements → Current minimum refresh interval is 1 minute
+- SQL with extensive ORDER BY → Incremental refresh is limited; degrades to full refresh
+- Extensive Outer Joins with frequently changing right-side tables → Low incremental computation efficiency
+- Need to directly modify data (UPDATE/DELETE/TRUNCATE) → Dynamic Table does not support this
+
+### Refresh Mode Selection
+
+Singdata Dynamic Table supports three refresh scheduling methods:
+
+| Scheduling Method | Suitable Scenario | Characteristics |
+|-------------------|-------------------|-----------------|
+| DDL-defined refresh interval (`REFRESH INTERVAL`) | Simple scenarios, quick launch | No upstream/downstream dependency support; min interval 1 minute |
+| Lakehouse Studio scheduling | Multi-layer DT chains, dependency control needed | Supports task dependencies (A completes then triggers B); monitoring and alerting |
+| Third-party scheduling engine | Existing scheduling system, flexible control needed | Time interval not limited, but introduces external dependency |
+
+**Key Constraint for Multi-Layer Chains**: The upstream DT's refresh frequency determines the minimum latency achievable by the downstream DT. If upstream refreshes every 5 minutes, downstream latency will be at least 5 minutes even if set to 1 minute.
+
+### Freshness vs Cost Trade-off
+
+More frequent refreshing means fresher data but higher compute costs. Method for determining a reasonable refresh interval:
+
+1. Assess business tolerance for data latency ("Is 5-minute-old data acceptable?")
+2. Check single refresh duration (`SHOW DYNAMIC TABLE REFRESH HISTORY`)
+3. Refresh interval > single refresh duration to avoid task backlog
+4. If refresh duration approaches the interval, incremental computation may have degraded to full; optimize SQL or extend the interval
+
+### Incremental Refresh vs Full Refresh
+
+Dynamic Table automatically selects incremental or full mode:
+
+- **Incremental Refresh (INCREMENTAL)**: Only computes data changed since the last refresh; high efficiency
+- **Full Refresh (FULL)**: Recomputes all data; suitable for initial refresh or when incremental conditions are not met
+
+The following situations trigger full refresh or prevent incremental computation:
+- SQL contains operators that cannot be incrementally processed (e.g., window functions with ORDER BY)
+- First-time `REFRESH DYNAMIC TABLE`
+- Source table changes are too large (approaching full data volume)
+
+Use `SHOW DYNAMIC TABLE REFRESH HISTORY` to view the `refresh_mode` field for each refresh, confirming whether incremental mode was successfully used.
+
+---
+
+## Typical Architecture Patterns
+
+### Pattern 1: Kafka Real-Time Ingestion + Incremental Processing
+
+Suitable for: log analysis, real-time dashboards, user behavior analysis
+
+```
+Kafka Topic
+    ↓ Pipe (continuous ingestion)
+  ods.events table
+    ↓ Dynamic Table (1-minute incremental aggregation)
+  dwd.event_stats table
+    ↓ Dynamic Table (5-minute summarization)
+  dws.daily_summary table
+```
+
+**End-to-End SQL Example** (replace table names and connection names with actual values):
+
+```SQL
+-- Pattern 1 complete example: Kafka log real-time ingestion + dynamic table incremental aggregation
+-- Assumes existing Kafka Connection: kafka_conn, Topic: app_events
+
+-- Step 1: Create target table (ODS layer)
+CREATE TABLE IF NOT EXISTS ods.app_events (
+    event_id    STRING,
+    user_id     BIGINT,
+    event_type  STRING,
+    page        STRING,
+    ts          TIMESTAMP
+);
+
+-- Step 2: Create Pipe for continuous Kafka consumption
+CREATE PIPE ods.kafka_events_pipe AS
+    COPY INTO ods.app_events
+    FROM READ_KAFKA(
+        CONNECTION => 'kafka_conn',
+        TOPIC => 'app_events'
+    ) USING JSON;
+
+-- Step 3: Create Dynamic Table for minute-level aggregation (DWD layer)
+CREATE DYNAMIC TABLE dwd.event_stats
+    TARGET_LAG = '1 minute'
+    VCLUSTER = 'default'
+AS
+SELECT
+    DATE_TRUNC('minute', ts)  AS minute,
+    event_type,
+    COUNT(*)                  AS event_cnt,
+    COUNT(DISTINCT user_id)   AS uv
+FROM ods.app_events
+GROUP BY 1, 2;
+
+-- Query results (after refresh)
+SELECT * FROM dwd.event_stats ORDER BY minute DESC LIMIT 5;
+```
+
+The Pipe auto-runs after creation; no manual trigger is needed. The Dynamic Table auto-refreshes incrementally every minute, processing only events newly added since the last refresh.
+
+### Pattern 2: OSS File Auto-Loading + Incremental Processing
+
+Suitable for: IoT device data, log archiving, batch file processing
+
+```
+OSS Bucket (new files continuously uploaded)
+    ↓ Pipe EVENT_NOTIFICATION (minute-level trigger)
+  ods.raw_files table
+    ↓ Dynamic Table (incremental parsing/cleaning)
+  dwd.cleaned_data table
+```
+
+### Pattern 3: Table Stream-Driven Cross-System Sync
+
+Suitable for: syncing Lakehouse internal table changes to external systems, SCD maintenance
+
+```
+source_table (business table)
+    ↓ Table Stream (STANDARD mode)
+  MERGE INTO target (Studio scheduled task, consuming every minute)
+  target_table (sync target)
+```
+
+**End-to-End SQL Example** (replace table names with actual values):
+
+```SQL
+-- Pattern 3 complete example: Table Stream-driven incremental sync
+-- Scenario: sync changes from orders table to orders_replica (cross-schema sync)
+
+-- Step 1: Create STANDARD mode Stream on the source table
+CREATE TABLE STREAM orders_sync_stream
+    ON TABLE ods.orders
+    WITH PROPERTIES ('TABLE_STREAM_MODE' = 'STANDARD');
+
+-- Step 2: Create target replica table (same structure as source)
+CREATE TABLE IF NOT EXISTS dwd.orders_replica
+    AS SELECT * FROM ods.orders WHERE 1=0;  -- Copy structure only, not data
+
+-- Step 3: Studio scheduled task (executes every minute)
+MERGE INTO dwd.orders_replica t
+USING orders_sync_stream s ON t.order_id = s.order_id
+WHEN MATCHED AND s.__change_type = 'UPDATE_AFTER'
+    THEN UPDATE SET t.status = s.status, t.amount = s.amount
+WHEN MATCHED AND s.__change_type = 'DELETE'
+    THEN DELETE
+WHEN NOT MATCHED AND s.__change_type = 'INSERT'
+    THEN INSERT VALUES (s.order_id, s.customer_id, s.product,
+                        s.amount, s.status, s.order_date);
+-- UPDATE_BEFORE rows are automatically ignored
+```
+
+The Stream's offset advances automatically after MERGE commits successfully. If MERGE fails (transaction rollback), the offset does not advance, and the next execution will re-consume the same batch of changes, ensuring no data loss.
+
+### Pattern 4: Volume Stream Monitoring File Changes
+
+Suitable for: image/document AI processing, incremental file analysis
+
+```
+OSS Bucket (new images uploaded)
+    ↓ Volume Stream (based on Directory Table)
+  INSERT INTO (consume Stream, invoke AI functions for processing)
+  results table
+```
+
+---
+
+## Related Documents
+
+- [Pipe Continuous Ingestion Overview](pipe-summary.md)
+- [Using Pipe for Continuous Object Storage Data Ingestion](pipe-storage-object.md)
+- [Table Stream Introduction](tablestream_summary.md)
+- [Table Stream Best Practices](lakehouse-table-stream-best-practices.md)
+- [Dynamic Table Introduction](dynamic_table_summary.md)
+- [Dynamic Table Refresh Scheduling](dynamic-table-scheduling.md)
+- [Incremental Computing Overview](streaming_data_pipeline_overview.md)
+- [Developing Dynamic Tables for Near-Real-Time Incremental Processing](streaming_pipeline_with_dynamic_table.md)
+- [Implementing SCD with Streams and Tasks](slowly-changing-dimensions-with-streams-and-tasks.md)

package/bin/skills/lakehouse-doc-en/references/realtime-sales-dashboard-with-dynamic-table.md

@@ -0,0 +1,317 @@
+# Real-Time Sales Dashboard: Building a Multi-Layer Incremental Data Warehouse with Dynamic Tables
+
+## Business Context
+
+E-commerce operations teams face a common challenge every day: during a promotion, GMV and rankings across categories are changing in real time, but the data dashboard only refreshes the next day. Missing real-time data means missing the window to adjust campaign strategy — a category is already selling out, but the decisions to restock and boost spend have to wait until tomorrow.
+
+The traditional solution is to write a scheduled SQL job that recalculates everything from scratch every minute. This approach has two problems:
+
+- **Wasted computation**: Every run scans all historical orders. The larger the data, the slower it gets, and cost grows linearly with data volume.
+- **Operational burden**: You have to manage scheduling yourself, handle task failure retries, and ensure idempotency.
+
+![](/.topwrite/assets/29-realtime-sales-dashboard.svg)
+
+The value of Dynamic Tables is this: you write SQL with full-scan semantics, and the system automatically identifies new data and only computes the changed portion — it never recomputes history. If data volume grows 10x, the cost of incremental computation stays the same.
+
+## Use Cases
+
+| Scenario | Description |
+|----------|-------------|
+| Real-time sales leaderboard | Category/product/store GMV rankings, updated by the minute |
+| Promotion monitoring | Track SKU sales and inventory consumption in real time during campaigns |
+| Automated daily operations report | Daily GMV, order count, and average order value summarized automatically — no manual queries needed |
+| Multi-layer data warehouse maintenance | Full-chain incremental refresh across ODS → DWD → ADS, replacing scheduled full-scan jobs |
+
+## SQL Commands Involved
+
+| Command | Purpose |
+|---------|---------|
+| `CREATE DYNAMIC TABLE` | Define a table that refreshes automatically |
+| `REFRESH INTERVAL` | Set the data update frequency |
+| `SHOW DYNAMIC TABLE REFRESH HISTORY` | View refresh status and duration for each layer |
+| `DROP DYNAMIC TABLE` | Clean up resources |
+
+## Data Architecture
+
+A three-layer Dynamic Table pipeline, each layer refreshing every 1 minute:
+
+```
+External data source (MySQL / Kafka / object storage / ...)
+              │  real-time writes
+              ▼
+doc_orders          doc_products
+    └──────────────────┘
+              │  JOIN
+              ▼
+   doc_dwd_order_detail     ← DWD: enriched orders (adds category, brand)
+              │
+              ▼
+   doc_ads_category_gmv     ← ADS: category GMV summary + average price
+              │
+              ▼
+   doc_ads_category_rank    ← ADS: real-time category leaderboard
+```
+
+**How data is written to the base tables in real time**
+
+This guide uses `INSERT INTO` to simulate data writes so you can reproduce the example quickly in a test environment. In production, order data typically comes from a business database or message queue. Singdata Lakehouse provides several ways to continuously write to the base tables:
+
+| Data source | Recommended approach | Reference |
+|-------------|---------------------|-----------|
+| MySQL / PostgreSQL / Oracle and other relational databases | Studio real-time sync task (CDC) | [Real-time sync task](realtime_sync.md) |
+| Multiple business tables in one migration | Studio multi-table real-time sync | [Multi-table real-time sync task](multitable_realtime_sync.md) |
+| Kafka message queue | Pipe continuous ingestion | [Continuous ingestion via read_kafka function](pipe-kafka.md) |
+| Object storage (OSS / COS / S3) files | Pipe continuous ingestion | [Continuous ingestion from object storage with Pipe](pipe-storage-object.md) |
+| Offline batch sync (T+1 or hourly) | Studio offline sync task | [Offline sync task](batch_sync.md) |
+
+Once new data is written to the base tables, the Dynamic Table pipeline automatically detects it at the next refresh cycle and computes incrementally — no additional configuration needed.
+
+## Prerequisites
+
+### Create the source tables
+
+```sql
+CREATE TABLE IF NOT EXISTS doc_orders (
+    order_id   STRING,
+    user_id    STRING,
+    product_id STRING,
+    quantity   INT,
+    unit_price DECIMAL(10,2),
+    amount     DECIMAL(10,2),
+    order_time TIMESTAMP,
+    status     STRING
+);
+
+CREATE TABLE IF NOT EXISTS doc_products (
+    product_id   STRING,
+    product_name STRING,
+    category     STRING,
+    brand        STRING
+);
+```
+
+### Insert initial data
+
+```sql
+INSERT INTO doc_products VALUES
+  ('P001', 'Wireless Bluetooth Earbuds', 'Electronics', 'SoundMax'),
+  ('P002', 'Mechanical Keyboard',        'Computer Peripherals', 'KeyPro'),
+  ('P003', 'Running Shoes',              'Sports & Outdoors', 'SpeedRun'),
+  ('P004', 'Yoga Mat',                   'Sports & Outdoors', 'FlexFit'),
+  ('P005', 'Insulated Tumbler',          'Home & Living', 'ThermoKeep'),
+  ('P006', 'USB-C Charger',              'Electronics', 'ChargeFast'),
+  ('P007', 'Laptop Stand',               'Computer Peripherals', 'DeskPro'),
+  ('P008', 'Jump Rope',                  'Sports & Outdoors', 'JumpFit');
+
+INSERT INTO doc_orders VALUES
+  ('O001','U101','P001',2,299.00,598.00, CAST('2026-05-28 16:00:00' AS TIMESTAMP),'completed'),
+  ('O002','U102','P002',1,459.00,459.00, CAST('2026-05-28 16:05:00' AS TIMESTAMP),'completed'),
+  ('O003','U103','P003',1,389.00,389.00, CAST('2026-05-28 16:10:00' AS TIMESTAMP),'completed'),
+  ('O004','U104','P001',1,299.00,299.00, CAST('2026-05-28 16:15:00' AS TIMESTAMP),'completed'),
+  ('O005','U105','P004',2,128.00,256.00, CAST('2026-05-28 16:20:00' AS TIMESTAMP),'completed'),
+  ('O006','U106','P005',3, 89.00,267.00, CAST('2026-05-28 16:25:00' AS TIMESTAMP),'completed'),
+  ('O007','U107','P006',2,129.00,258.00, CAST('2026-05-28 16:30:00' AS TIMESTAMP),'completed'),
+  ('O008','U108','P003',2,389.00,778.00, CAST('2026-05-28 16:35:00' AS TIMESTAMP),'completed'),
+  ('O009','U109','P007',1,199.00,199.00, CAST('2026-05-28 16:40:00' AS TIMESTAMP),'completed'),
+  ('O010','U110','P008',3, 49.00,147.00, CAST('2026-05-28 16:45:00' AS TIMESTAMP),'completed');
+```
+
+> ⚠️ **Note**: TIMESTAMP columns must be inserted using `CAST('...' AS TIMESTAMP)`. String literals cannot be written directly.
+
+## Scenario 1: Create the Three-Layer Dynamic Tables
+
+### Layer 1 (DWD): Enrich Orders
+
+Join the orders table with the products table to add category and brand information, keeping only completed orders.
+
+```sql
+CREATE OR REPLACE DYNAMIC TABLE doc_dwd_order_detail
+REFRESH INTERVAL '1' MINUTE VCLUSTER default
+AS
+SELECT
+    o.order_id,
+    o.user_id,
+    o.product_id,
+    p.product_name,
+    p.category,
+    p.brand,
+    o.quantity,
+    o.unit_price,
+    o.amount,
+    o.order_time,
+    o.status
+FROM doc_orders o
+JOIN doc_products p ON o.product_id = p.product_id
+WHERE o.status = 'completed';
+```
+
+### Layer 2 (ADS): Category GMV Summary
+
+Aggregate by category to compute order count, total quantity, GMV, and average price.
+
+```sql
+CREATE OR REPLACE DYNAMIC TABLE doc_ads_category_gmv
+REFRESH INTERVAL '1' MINUTE VCLUSTER default
+AS
+SELECT
+    category,
+    COUNT(DISTINCT order_id)                              AS order_cnt,
+    SUM(quantity)                                         AS total_qty,
+    SUM(amount)                                           AS gmv,
+    CAST(SUM(amount) / SUM(quantity) AS DECIMAL(10,2))   AS avg_price
+FROM doc_dwd_order_detail
+GROUP BY category;
+```
+
+> ⚠️ **Note**: Average price must be computed with `CAST(SUM(amount) / SUM(quantity) AS DECIMAL(10,2))` to explicitly specify precision. Using `ROUND(SUM(amount)/SUM(quantity), 2)` directly will trigger an optimizer type inference error.
+
+### Layer 3 (ADS): Category Leaderboard
+
+Compute real-time rankings based on the GMV summary table.
+
+```sql
+CREATE OR REPLACE DYNAMIC TABLE doc_ads_category_rank
+REFRESH INTERVAL '1' MINUTE VCLUSTER default
+AS
+SELECT
+    RANK() OVER (ORDER BY gmv DESC) AS gmv_rank,
+    category,
+    order_cnt,
+    total_qty,
+    gmv,
+    avg_price
+FROM doc_ads_category_gmv;
+```
+
+> ⚠️ **Note**: A Dynamic Table will not have data immediately after creation. The first refresh is triggered within about 1 minute of creation. In practice, the DWD layer and the GMV layer refresh in the same batch, while the rank layer detects the change in the next cycle, resulting in an end-to-end latency of about 1–2 minutes.
+
+### Verify Initial Results
+
+Wait about 2 minutes, then query the leaderboard:
+
+```sql
+SELECT * FROM doc_ads_category_rank ORDER BY gmv_rank;
+```
+
+```
++--------+---------------------+-----------+-----------+---------+-----------+
+|gmv_rank|category             |order_cnt  |total_qty  |gmv      |avg_price  |
++--------+---------------------+-----------+-----------+---------+-----------+
+|       1|Sports & Outdoors    |          4|          8|  1570.00|     196.25|
+|       2|Electronics          |          3|          5|  1155.00|     231.00|
+|       3|Computer Peripherals |          2|          2|   658.00|     329.00|
+|       4|Home & Living        |          1|          3|   267.00|      89.00|
++--------+---------------------+-----------+-----------+---------+-----------+
+```
+
+Sports & Outdoors ranks first with 1570 in GMV, from 4 orders across running shoes (2 orders), yoga mat (1 order), and jump rope (1 order), with an average price of 196.25.
+
+## Scenario 2: New Orders Written, Rankings Update Automatically
+
+Simulate the next batch of orders coming in — Electronics and Computer Peripherals each have new orders:
+
+```sql
+INSERT INTO doc_orders VALUES
+  ('O011','U111','P005',2, 89.00,178.00, CAST('2026-05-28 17:00:00' AS TIMESTAMP),'completed'),
+  ('O012','U112','P006',3,129.00,387.00, CAST('2026-05-28 17:02:00' AS TIMESTAMP),'completed'),
+  ('O013','U113','P001',1,299.00,299.00, CAST('2026-05-28 17:05:00' AS TIMESTAMP),'completed'),
+  ('O014','U114','P004',1,128.00,128.00, CAST('2026-05-28 17:08:00' AS TIMESTAMP),'completed'),
+  ('O015','U115','P002',2,459.00,918.00, CAST('2026-05-28 17:10:00' AS TIMESTAMP),'completed');
+```
+
+Wait about 2 minutes, then query the leaderboard again:
+
+```sql
+SELECT * FROM doc_ads_category_rank ORDER BY gmv_rank;
+```
+
+```
++--------+---------------------+-----------+-----------+---------+-----------+
+|gmv_rank|category             |order_cnt  |total_qty  |gmv      |avg_price  |
++--------+---------------------+-----------+-----------+---------+-----------+
+|       1|Electronics          |          5|          9|  1841.00|     204.56|
+|       2|Sports & Outdoors    |          5|          9|  1698.00|     188.67|
+|       3|Computer Peripherals |          3|          4|  1576.00|     394.00|
+|       4|Home & Living        |          2|          5|   445.00|      89.00|
++--------+---------------------+-----------+-----------+---------+-----------+
+```
+
+Electronics added an insulated tumbler (178), USB-C charger (387), and wireless earbuds (299) for a total of 686 in new GMV, rising from 1155 to 1841 and overtaking Sports & Outdoors to claim first place. The ranking change is computed automatically by the system — no manual intervention required.
+
+## Scenario 3: Modify a Dynamic Table Definition
+
+When business requirements change, you can update the SQL logic with `CREATE OR REPLACE DYNAMIC TABLE`. Existing data will not be lost.
+
+**Changes suitable for OR REPLACE**: modifying WHERE filter conditions, adding pass-through columns (columns that come directly from upstream without any computation).
+
+**Changes that require DROP + recreate**: modifying column types, modifying aggregation logic (e.g., adding a computed column like `avg_price`).
+
+> ⚠️ **Note**: OR REPLACE does not allow changing the type of an existing column — it will raise `NotAllowed: cannot replace <col> from type X to Y`. To change a column type, you must DROP and recreate the table.
+
+After OR REPLACE, if no new data has been written upstream, the system will not recompute existing rows — the new column values will remain null until the next refresh is triggered by new data.
+
+## View Refresh Status
+
+```sql
+SHOW DYNAMIC TABLE REFRESH HISTORY WHERE name = 'doc_ads_category_rank' LIMIT 5;
+```
+
+Example output from a real run:
+
+| state | refresh_mode | start_time | rows_inserted | Meaning |
+|-------|-------------|------------|---------------|---------|
+| SUCCEED | INCREMENTAL | 18:11:48 | 4 | Normal incremental refresh, 4 rows written |
+| SUCCEED | NO_DATA | 18:12:48 | — | No new data upstream, computation skipped |
+| FAILED | FULL | 18:09:41 | — | Full refresh failed, see error_message |
+
+Key field descriptions:
+
+| Field | Meaning |
+|-------|---------|
+| `state` | `SUCCEED` success / `FAILED` failure |
+| `refresh_mode` | `INCREMENTAL` incremental / `FULL` full scan / `NO_DATA` no new data, skipped |
+| `refresh_trigger` | `SYSTEM_SCHEDULED` automatic / `MANUAL` manually triggered |
+| `start_time` / `end_time` | Refresh start and end time |
+| `stats` | `rows_inserted` / `rows_deleted` — rows written and deleted in this refresh |
+| `error_message` | Error details when state is FAILED |
+
+## Behavior When an Upstream Table Is Dropped
+
+When an upstream Dynamic Table is dropped, downstream Dynamic Tables will not error out or cascade-delete. Their refresh state changes to `NO_DATA` and the objects continue to exist.
+
+```sql
+-- Drop the DWD layer
+DROP DYNAMIC TABLE IF EXISTS doc_dwd_order_detail;
+
+-- The GMV layer still exists, but refreshes become NO_DATA
+SHOW DYNAMIC TABLE REFRESH HISTORY WHERE name = 'doc_ads_category_gmv' LIMIT 3;
+-- state: SUCCEED, refresh_mode: NO_DATA
+```
+
+This means: if you accidentally drop an upstream Dynamic Table, downstream data will not disappear. Once the upstream is recreated, the downstream will automatically resume incremental computation at the next refresh cycle.
+
+## Clean Up Resources
+
+```sql
+DROP DYNAMIC TABLE IF EXISTS doc_ads_category_rank;
+DROP DYNAMIC TABLE IF EXISTS doc_ads_category_gmv;
+DROP DYNAMIC TABLE IF EXISTS doc_dwd_order_detail;
+DROP TABLE IF EXISTS doc_orders;
+DROP TABLE IF EXISTS doc_products;
+```
+
+> 💡 **Tip**: The DROP order does not affect the outcome — any order will succeed.
+
+## Key Takeaways
+
+- **Write full-scan SQL, let the system handle incremental computation**: The SQL for all three Dynamic Table layers is plain SELECT — no incremental logic needed. The system automatically identifies new data and only computes the changed portion.
+- **Changes propagate through the pipeline automatically**: New data in the source table → DWD layer refreshes → ADS summary layer refreshes → leaderboard updates. The entire pipeline requires no manual triggering.
+- **End-to-end latency is about 1–2 minutes**: In practice, the DWD and GMV layers refresh in the same batch, and the rank layer detects the change in the next cycle, giving an end-to-end latency of about 1–2 minutes when the refresh interval is set to 1 minute.
+- **OR REPLACE has limitations**: Column types cannot be changed, and modifying aggregation logic requires DROP + recreate. After OR REPLACE, if no new data arrives upstream, new column values will be null until the next refresh.
+
+## Related Documentation
+
+- [Dynamic Table Overview](dynamic-table.md)
+- [CREATE DYNAMIC TABLE](create-dynamic-table.md)
+- [ALTER DYNAMIC TABLE](alter-dynamic-table.md)
+- [SHOW DYNAMIC TABLE REFRESH HISTORY](refresh-history.md)