@xdev-asia/xdev-knowledge-mcp 1.0.42 → 1.0.44

package/content/series/luyen-thi/luyen-thi-gcp-ml-engineer/chapters/02-phan-2-data-engineering/lessons/03-bai-3-data-pipeline.md

@@ -0,0 +1,174 @@
+---
+id: 019c9619-lt03-l03
+title: 'Bài 3: Data Pipeline — Dataflow, Pub/Sub, Dataproc'
+slug: bai-3-data-pipeline
+description: >-
+  Apache Beam trên Dataflow cho batch/streaming ETL.
+  Pub/Sub cho event-driven pipelines. Dataproc cho Spark.
+  Cloud Composer (Airflow) cho orchestration.
+duration_minutes: 60
+is_free: true
+video_url: null
+sort_order: 3
+section_title: "Phần 2: Data Engineering & Feature Engineering"
+course:
+  id: 019c9619-lt03-7003-c003-lt0300000003
+  title: 'Luyện thi Google Cloud Professional Machine Learning Engineer'
+  slug: luyen-thi-gcp-ml-engineer
+---
+
+<div style="text-align: center; margin: 2rem 0;">
+<img src="/storage/uploads/2026/04/gcp-mle-bai3-data-pipeline.png" alt="GCP Data Pipeline Architecture" style="max-width: 800px; width: 100%; border-radius: 12px;" />
+<p><em>GCP Data Pipeline: Pub/Sub, Dataflow, Dataproc, Cloud Composer và luồng dữ liệu cho ML</em></p>
+</div>
+
+<h2 id="gcp-data-pipeline"><strong>1. GCP Data Pipeline Services</strong></h2>
+
+<table>
+<thead><tr><th>Service</th><th>Type</th><th>When to Use</th></tr></thead>
+<tbody>
+<tr><td><strong>Pub/Sub</strong></td><td>Managed message queue</td><td>Event streaming, decouple producers/consumers</td></tr>
+<tr><td><strong>Dataflow</strong></td><td>Managed Apache Beam runner</td><td>Unified batch + streaming ETL</td></tr>
+<tr><td><strong>Dataproc</strong></td><td>Managed Spark / Hadoop</td><td>Existing Spark/Hadoop workloads, ML at scale</td></tr>
+<tr><td><strong>Cloud Composer</strong></td><td>Managed Apache Airflow</td><td>Orchestrate multi-step ML workflows</td></tr>
+<tr><td><strong>Cloud Storage</strong></td><td>Object store</td><td>Raw data landing zone, model artifacts</td></tr>
+<tr><td><strong>BigQuery</strong></td><td>Data warehouse</td><td>Structured analysis, BigQuery ML</td></tr>
+</tbody>
+</table>
+
+<h2 id="pubsub"><strong>2. Pub/Sub — Event Streaming</strong></h2>
+
+<pre><code class="language-text">Pub/Sub Architecture:
+
+Data Source → Publisher → [Topic] → Subscription → Subscriber
+(IoT devices,                         (Pull or            (Dataflow,
+web clicks,                            Push)              Cloud Functions,
+logs)                                                     BigQuery)
+
+Key concepts:
+- Topic: named resource where messages are sent
+- Subscription: named resource attached to topic
+- Publisher: sends messages to topic  
+- Subscriber: receives messages from subscription
+- At-least-once delivery (not exactly-once by default)
+</code></pre>
+
+<table>
+<thead><tr><th>Feature</th><th>Details</th></tr></thead>
+<tbody>
+<tr><td><strong>Message retention</strong></td><td>7 days default (configurable)</td></tr>
+<tr><td><strong>At-least-once delivery</strong></td><td>Idempotent subscribers needed</td></tr>
+<tr><td><strong>Exactly-once</strong></td><td>Available in Pub/Sub Lite (same region)</td></tr>
+<tr><td><strong>Ordering</strong></td><td>Enable message ordering with ordering key</td></tr>
+</tbody>
+</table>
+
+<blockquote>
+<p><strong>Exam tip:</strong> Pub/Sub → Dataflow → BigQuery là pipeline pattern cực phổ biến trong đề thi. Pub/Sub ingest, Dataflow transform, BigQuery store + analyze.</p>
+</blockquote>
+
+<h2 id="dataflow"><strong>3. Cloud Dataflow — Apache Beam</strong></h2>
+
+<p>Dataflow là managed runner cho <strong>Apache Beam</strong> — framework cho unified batch và streaming processing. Không cần quản lý servers.</p>
+
+<table>
+<thead><tr><th>Concept</th><th>Description</th></tr></thead>
+<tbody>
+<tr><td><strong>Pipeline</strong></td><td>Chuỗi transform operations</td></tr>
+<tr><td><strong>PCollection</strong></td><td>Distributed data collection (bounded or unbounded)</td></tr>
+<tr><td><strong>Transform</strong></td><td>ParDo, GroupByKey, Combine, Flatten, Partition</td></tr>
+<tr><td><strong>Windowing</strong></td><td>Fixed, Sliding, Session windows cho streaming</td></tr>
+<tr><td><strong>Watermarks</strong></td><td>Handle late-arriving data in streaming</td></tr>
+</tbody>
+</table>
+
+<pre><code class="language-text">Dataflow Windowing for Streaming ML:
+
+Event stream: ──●──●──●──────●──●──●──────●──●──
+
+Fixed Window (1 min):
+├─── [W1] ──┤├─── [W2] ──┤├─── [W3] ──┤
+
+Sliding Window (1 min, slide 30s):
+├── [W1] ────┤
+       ├── [W2] ────┤
+              ├── [W3] ────┤
+
+Session Window (2 min gap):
+├── [S1] ──────────┤          ├── [S2] ──┤
+     (user session)            (new session)
+</code></pre>
+
+<h2 id="dataproc"><strong>4. Cloud Dataproc — Managed Spark/Hadoop</strong></h2>
+
+<table>
+<thead><tr><th>Dataproc Feature</th><th>Details</th></tr></thead>
+<tbody>
+<tr><td><strong>Cluster lifecycle</strong></td><td>Create in 90 seconds, delete after job — cost efficient</td></tr>
+<tr><td><strong>Ephemeral clusters</strong></td><td>Spin up → run job → shut down (per-job pricing)</td></tr>
+<tr><td><strong>Preemptible VMs</strong></td><td>Use for worker nodes to reduce cost 60-80%</td></tr>
+<tr><td><strong>Component gateway</strong></td><td>Access Jupyter, Zeppelin, Spark UI via browser</td></tr>
+<tr><td><strong>ML libraries</strong></td><td>Spark MLlib, TensorFlow on Spark (TFoS)</td></tr>
+</tbody>
+</table>
+
+<h2 id="composer"><strong>5. Cloud Composer — Workflow Orchestration</strong></h2>
+
+<p>Cloud Composer là managed Apache Airflow. Dùng để orchestrate multi-step ML pipelines bao gồm data ingestion, preprocessing, training, và deployment.</p>
+
+<pre><code class="language-text">Cloud Composer ML Workflow:
+
+[DAG: daily_ml_pipeline]
+   Task 1: Extract data from BigQuery
+       ↓
+   Task 2: Run Dataflow preprocessing job
+       ↓
+   Task 3: Submit Vertex AI Training Job
+       ↓
+   Task 4: Evaluate model metrics
+       ↓ (if metrics pass threshold)
+   Task 5: Deploy to Vertex AI Endpoint
+</code></pre>
+
+<h2 id="decision-guide"><strong>6. Data Pipeline Service Selection</strong></h2>
+
+<table>
+<thead><tr><th>Scenario</th><th>Recommended Service</th></tr></thead>
+<tbody>
+<tr><td>Real-time event streaming ingestion</td><td>Pub/Sub</td></tr>
+<tr><td>Unified batch + streaming ETL (no infra mgmt)</td><td>Dataflow (Apache Beam)</td></tr>
+<tr><td>Migrate existing Spark jobs to GCP</td><td>Dataproc</td></tr>
+<tr><td>Complex ML DAG orchestration</td><td>Cloud Composer</td></tr>
+<tr><td>Stream data into BigQuery</td><td>Pub/Sub → Dataflow → BigQuery</td></tr>
+<tr><td>Serverless data processing (SQL)</td><td>BigQuery (ETL via SQL)</td></tr>
+</tbody>
+</table>
+
+<h2 id="practice"><strong>7. Practice Questions</strong></h2>
+
+<p><strong>Q1:</strong> A company receives millions of IoT sensor events per second from factory equipment. They need to process these events in real time, detect anomalies, and store results in BigQuery. Which pipeline architecture is MOST appropriate?</p>
+<ul>
+<li>A) Dataproc → Spark Streaming → BigQuery</li>
+<li>B) Pub/Sub → Dataflow → BigQuery ✓</li>
+<li>C) Cloud Functions → Cloud SQL</li>
+<li>D) Batch upload to Cloud Storage → BigQuery import</li>
+</ul>
+<p><em>Explanation: Pub/Sub ingests high-volume streaming events reliably. Dataflow processes the stream in real time using Apache Beam (windowing, transformations, anomaly detection). BigQuery stores the results for analysis. This is the canonical GCP streaming analytics pattern.</em></p>
+
+<p><strong>Q2:</strong> A data engineering team has an existing Apache Spark job that processes training data for ML models. They want to migrate it to GCP with minimal code changes. Which service should they use?</p>
+<ul>
+<li>A) Cloud Dataflow</li>
+<li>B) Cloud Dataproc ✓</li>
+<li>C) BigQuery ETL</li>
+<li>D) Cloud Composer</li>
+</ul>
+<p><em>Explanation: Cloud Dataproc supports Apache Spark natively, allowing teams to run existing Spark jobs on GCP with minimal changes. Dataflow uses Apache Beam (different programming model). Dataproc is the lift-and-shift option for Spark workloads.</em></p>
+
+<p><strong>Q3:</strong> A team needs to orchestrate a daily ML pipeline that includes data extraction from BigQuery, preprocessing, Vertex AI training, and deployment if accuracy exceeds 90%. Which service handles this workflow orchestration?</p>
+<ul>
+<li>A) Vertex AI Pipelines</li>
+<li>B) Cloud Dataflow</li>
+<li>C) Cloud Composer ✓</li>
+<li>D) Pub/Sub triggers</li>
+</ul>
+<p><em>Explanation: Cloud Composer (managed Apache Airflow) is designed for complex DAG orchestration across multiple services. It handles scheduling, conditional branching (deploy only if accuracy > 90%), retry logic, and monitoring across heterogeneous services like BigQuery, Dataflow, and Vertex AI.</em></p>

package/content/series/luyen-thi/luyen-thi-gcp-ml-engineer/chapters/02-phan-2-data-engineering/lessons/04-bai-4-feature-engineering.md

@@ -0,0 +1,156 @@
+---
+id: 019c9619-lt03-l04
+title: 'Bài 4: Feature Engineering & Vertex AI Feature Store'
+slug: bai-4-feature-engineering
+description: >-
+  Feature engineering techniques. BigQuery cho feature computation.
+  Vertex AI Feature Store: online/offline serving.
+  Feature monitoring, consistency giữa training/serving.
+duration_minutes: 60
+is_free: true
+video_url: null
+sort_order: 4
+section_title: "Phần 2: Data Engineering & Feature Engineering"
+course:
+  id: 019c9619-lt03-7003-c003-lt0300000003
+  title: 'Luyện thi Google Cloud Professional Machine Learning Engineer'
+  slug: luyen-thi-gcp-ml-engineer
+---
+
+<div style="text-align: center; margin: 2rem 0;">
+<img src="/storage/uploads/2026/04/gcp-mle-bai4-feature-store.png" alt="Vertex AI Feature Store" style="max-width: 800px; width: 100%; border-radius: 12px;" />
+<p><em>Feature Engineering & Vertex AI Feature Store: tạo, lưu trữ, và tái sử dụng features cho ML</em></p>
+</div>
+
+<h2 id="feature-engineering"><strong>1. Feature Engineering Techniques</strong></h2>
+
+<table>
+<thead><tr><th>Technique</th><th>When to Use</th><th>Example</th></tr></thead>
+<tbody>
+<tr><td><strong>Normalization (Min-Max)</strong></td><td>Bounded range required (0-1)</td><td>Image pixels, probabilities</td></tr>
+<tr><td><strong>Standardization (Z-score)</strong></td><td>Normal-ish distribution, no bounds</td><td>Customer age, transaction amount</td></tr>
+<tr><td><strong>Log Transform</strong></td><td>Skewed distributions (price, salary)</td><td>Log(price) for housing</td></tr>
+<tr><td><strong>One-Hot Encoding</strong></td><td>Nominal categorical (no order)</td><td>Country, brand, color</td></tr>
+<tr><td><strong>Label Encoding</strong></td><td>Ordinal categorical (has order)</td><td>Low/Medium/High → 0/1/2</td></tr>
+<tr><td><strong>Feature Crossing</strong></td><td>Capture interaction between features</td><td>city × day_of_week</td></tr>
+<tr><td><strong>Bucketizing</strong></td><td>Convert continuous to categorical</td><td>Age → age_group</td></tr>
+<tr><td><strong>Embeddings</strong></td><td>High-cardinality categorical</td><td>UserID, ProductID</td></tr>
+</tbody>
+</table>
+
+<h2 id="missing-values"><strong>2. Handling Missing Values</strong></h2>
+
+<table>
+<thead><tr><th>Strategy</th><th>When</th></tr></thead>
+<tbody>
+<tr><td><strong>Mean/Median imputation</strong></td><td>Numerical, low missingness rate</td></tr>
+<tr><td><strong>Mode imputation</strong></td><td>Categorical features</td></tr>
+<tr><td><strong>Model-based imputation</strong></td><td>High missingness, complex patterns</td></tr>
+<tr><td><strong>Indicator variable</strong></td><td>Missingness itself is informative (add is_missing flag)</td></tr>
+<tr><td><strong>Drop rows</strong></td><td>Missing target / very few rows affected</td></tr>
+<tr><td><strong>Drop column</strong></td><td>&gt;80% missing</td></tr>
+</tbody>
+</table>
+
+<h2 id="training-serving-skew"><strong>3. Training-Serving Skew</strong></h2>
+
+<p><strong>Training-serving skew</strong> là vấn đề nghiêm trọng: features được compute khác nhau giữa training và serving, khiến model hoạt động kém trong production dù test metrics tốt.</p>
+
+<pre><code class="language-text">Training-Serving Skew Example:
+
+TRAINING TIME:
+  avg_purchase_last_30d = mean(all purchases in batch)  ← computed over full period
+
+SERVING TIME:
+  avg_purchase_last_30d = mean(last 5 purchases)        ← computed differently!
+
+Result: Feature distribution mismatch → poor predictions
+
+SOLUTION: Vertex AI Feature Store
+  Same feature serve logic used at training AND serving time
+</code></pre>
+
+<h2 id="feature-store"><strong>4. Vertex AI Feature Store</strong></h2>
+
+<table>
+<thead><tr><th>Component</th><th>Description</th></tr></thead>
+<tbody>
+<tr><td><strong>Feature Store</strong></td><td>Centralized repository for ML features</td></tr>
+<tr><td><strong>Entity Type</strong></td><td>Category of things you track (User, Product)</td></tr>
+<tr><td><strong>Feature</strong></td><td>Named attribute of an entity (user.avg_spend)</td></tr>
+<tr><td><strong>Online Store</strong></td><td>Low-latency serving (ms) for real-time predictions</td></tr>
+<tr><td><strong>Offline Store</strong></td><td>BigQuery-backed, for batch training data retrieval</td></tr>
+</tbody>
+</table>
+
+<pre><code class="language-text">Vertex AI Feature Store Architecture:
+
+Feature Ingestion (Batch or Streaming)
+        ↓
+┌──── Feature Store ────────────────┐
+│  Offline Store (BigQuery)          │  ← Training data export
+│  Online Store (Bigtable-backed)    │  ← Serving (ms latency)
+└───────────────────────────────────┘
+        ↑ Same features ↑
+  Training      Inference
+  Pipeline      Endpoint
+</code></pre>
+
+<h2 id="bigquery-features"><strong>5. BigQuery for Feature Engineering</strong></h2>
+
+<p>BigQuery là công cụ tốt nhất trên GCP để compute aggregate features từ large datasets.</p>
+
+<table>
+<thead><tr><th>Feature Pattern</th><th>BigQuery Approach</th></tr></thead>
+<tbody>
+<tr><td>Rolling window aggregates</td><td>Window functions: AVG() OVER (PARTITION BY ... ORDER BY ... ROWS BETWEEN ...)</td></tr>
+<tr><td>User activity counts</td><td>COUNT() GROUP BY user_id</td></tr>
+<tr><td>Categorical encoding</td><td>CASE WHEN ... or ML.ONE_HOT_ENCODE()</td></tr>
+<tr><td>Hash embedding (high cardinality)</td><td>FARM_FINGERPRINT() mod N</td></tr>
+<tr><td>Feature normalization</td><td>ML.STANDARD_SCALER() in BigQuery ML</td></tr>
+</tbody>
+</table>
+
+<blockquote>
+<p><strong>Exam tip:</strong> Khi câu hỏi nhắc đến "training-serving consistency" hoặc "feature reuse across multiple models" → <strong>Vertex AI Feature Store</strong>. Khi nhắc đến "compute features from BigQuery data at scale" → BigQuery window functions + scheduled queries.</p>
+</blockquote>
+
+<h2 id="feature-monitoring"><strong>6. Feature Drift Monitoring</strong></h2>
+
+<table>
+<thead><tr><th>Type</th><th>What Changes</th><th>Detection Method</th></tr></thead>
+<tbody>
+<tr><td><strong>Feature Skew</strong></td><td>Training vs serving feature distribution differs</td><td>Compare training baseline vs serving stats</td></tr>
+<tr><td><strong>Feature Drift</strong></td><td>Serving features change over time</td><td>Monitor serving feature distributions daily</td></tr>
+<tr><td><strong>Label Drift</strong></td><td>Target variable distribution changes</td><td>Track prediction distribution shifts</td></tr>
+</tbody>
+</table>
+
+<h2 id="practice"><strong>7. Practice Questions</strong></h2>
+
+<p><strong>Q1:</strong> A team's ML model has excellent accuracy during testing but performs poorly in production. Investigations reveal that the average purchase feature is calculated differently in training (using historical batch data) vs. serving (using real-time lookups). What is this problem called and how should it be solved?</p>
+<ul>
+<li>A) Model drift — retrain the model more frequently</li>
+<li>B) Training-serving skew — use Vertex AI Feature Store ✓</li>
+<li>C) Data leakage — remove the purchase feature</li>
+<li>D) Overfitting — add dropout layers</li>
+</ul>
+<p><em>Explanation: Training-serving skew occurs when features are computed differently at training and serving time. Vertex AI Feature Store solves this by providing a single source of truth for feature computation, ensuring the same logic is used for both training data export and online serving.</em></p>
+
+<p><strong>Q2:</strong> A feature has values ranging from $10 to $10,000,000 with a heavily right-skewed distribution. Which transformation is MOST appropriate before using this feature in a linear model?</p>
+<ul>
+<li>A) One-Hot Encoding</li>
+<li>B) Min-Max Normalization</li>
+<li>C) Log transformation ✓</li>
+<li>D) Label Encoding</li>
+</ul>
+<p><em>Explanation: Log transformation compresses the scale of highly skewed distributions, making them more normal-like and suitable for linear models. Min-Max normalization would still preserve the skew. One-hot encoding is for categorical data.</em></p>
+
+<p><strong>Q3:</strong> Which Vertex AI Feature Store store type is optimized for serving features to real-time prediction endpoints with millisecond latency?</p>
+<ul>
+<li>A) Offline Store (BigQuery)</li>
+<li>B) Online Store (Bigtable-backed) ✓</li>
+<li>C) Feature Catalog</li>
+<li>D) Cloud Memorystore</li>
+</ul>
+<p><em>Explanation: The Online Store in Vertex AI Feature Store is backed by Bigtable and designed for sub-100ms latency lookups, serving fresh feature values to real-time prediction endpoints. The Offline Store uses BigQuery and is for batch training data retrieval.</em></p>

package/content/series/luyen-thi/luyen-thi-gcp-ml-engineer/chapters/03-phan-3-model-development/lessons/05-bai-5-vertex-ai-training.md

@@ -0,0 +1,155 @@
+---
+id: 019c9619-lt03-l05
+title: 'Bài 5: Vertex AI Training — Custom & AutoML'
+slug: bai-5-vertex-ai-training
+description: >-
+  Custom Training Jobs: pre-built containers, custom containers.
+  Distributed training trên GPU/TPU. AutoML: Tabular, Image, Text, Video.
+  Training pipeline setup. Hyperparameter tuning service.
+duration_minutes: 60
+is_free: true
+video_url: null
+sort_order: 5
+section_title: "Phần 3: Model Development trên Vertex AI"
+course:
+  id: 019c9619-lt03-7003-c003-lt0300000003
+  title: 'Luyện thi Google Cloud Professional Machine Learning Engineer'
+  slug: luyen-thi-gcp-ml-engineer
+---
+
+<div style="text-align: center; margin: 2rem 0;">
+<img src="/storage/uploads/2026/04/gcp-mle-bai5-vertex-training.png" alt="Vertex AI Custom Training" style="max-width: 800px; width: 100%; border-radius: 12px;" />
+<p><em>Vertex AI Custom Training: Training Jobs, AutoML, phân tán distributed training và tối ưu</em></p>
+</div>
+
+<h2 id="custom-training"><strong>1. Vertex AI Custom Training</strong></h2>
+
+<p>Custom Training cho phép bạn chạy training code của mình trên Google Cloud infrastructure. Có 2 cách đóng gói code:</p>
+
+<table>
+<thead><tr><th>Method</th><th>Description</th><th>When to Use</th></tr></thead>
+<tbody>
+<tr><td><strong>Pre-built containers</strong></td><td>GCP-provided containers: TF, PyTorch, Scikit-learn, XGBoost</td><td>Standard ML frameworks, fast setup</td></tr>
+<tr><td><strong>Custom containers</strong></td><td>Build your own Docker image</td><td>Custom dependencies, special environments</td></tr>
+</tbody>
+</table>
+
+<pre><code class="language-text">Custom Training Job Structure:
+
+training_package/ (Python package or Docker image)
+│
+├── trainer/
+│   ├── __init__.py
+│   ├── task.py        ← entry point (main training script)
+│   └── model.py       ← model definition
+│
+└── setup.py
+
+Arguments passed via:
+  TRAINING_DATA_URI: gs://bucket/data/
+  TRAINING_OUTPUT_URI: gs://bucket/model/
+  Hyperparameters: --learning-rate=0.001
+</code></pre>
+
+<h2 id="compute-options"><strong>2. Compute Options</strong></h2>
+
+<table>
+<thead><tr><th>Hardware</th><th>Best For</th><th>Notes</th></tr></thead>
+<tbody>
+<tr><td><strong>CPU</strong></td><td>Scikit-learn, small tabular</td><td>Cheapest, no GPU parallelism</td></tr>
+<tr><td><strong>GPU (T4, A100, V100)</strong></td><td>Deep learning, NLP, CV</td><td>10-100x faster than CPU for DL</td></tr>
+<tr><td><strong>TPU v3, v4</strong></td><td>TensorFlow large-scale training</td><td>Google-specific; very fast for TF/JAX</td></tr>
+</tbody>
+</table>
+
+<blockquote>
+<p><strong>Exam tip:</strong> TPU is Google-specific hardware optimized for TensorFlow and JAX. GPUs work with all frameworks. TPUs are most cost-effective for very large TF models; GPUs are more versatile. Exam may ask "most cost-effective for TensorFlow large-scale" → TPU.</p>
+</blockquote>
+
+<h2 id="distributed-training"><strong>3. Distributed Training on Vertex AI</strong></h2>
+
+<table>
+<thead><tr><th>Strategy</th><th>Description</th><th>Use Case</th></tr></thead>
+<tbody>
+<tr><td><strong>Data Parallelism</strong></td><td>Split data across workers, same model</td><td>Most DL training scenarios</td></tr>
+<tr><td><strong>Model Parallelism</strong></td><td>Split model layers across workers</td><td>Model too large for one GPU</td></tr>
+<tr><td><strong>MirroredStrategy (TF)</strong></td><td>Multi-GPU, single machine</td><td>Single node, multiple GPUs</td></tr>
+<tr><td><strong>MultiWorkerMirroredStrategy</strong></td><td>Multi-GPU, multi-machine</td><td>Cluster training</td></tr>
+<tr><td><strong>ParameterServerStrategy</strong></td><td>Async updates via parameter server</td><td>Very large models (legacy)</td></tr>
+</tbody>
+</table>
+
+<h2 id="automl"><strong>4. Vertex AI AutoML</strong></h2>
+
+<table>
+<thead><tr><th>AutoML Type</th><th>Input Data</th><th>Supported Tasks</th></tr></thead>
+<tbody>
+<tr><td><strong>AutoML Tabular</strong></td><td>CSV, BigQuery table</td><td>Classification, Regression, Forecasting</td></tr>
+<tr><td><strong>AutoML Image</strong></td><td>JPEG, PNG, BMP</td><td>Classification (single/multi), Object Detection, Segmentation</td></tr>
+<tr><td><strong>AutoML Text</strong></td><td>Text documents</td><td>Classification, Entity Extraction, Sentiment</td></tr>
+<tr><td><strong>AutoML Video</strong></td><td>MP4, AVI, MOV</td><td>Classification, Object Detection, Action Recognition</td></tr>
+</tbody>
+</table>
+
+<h2 id="hyperparameter-tuning"><strong>5. Vertex AI Hyperparameter Tuning</strong></h2>
+
+<p>Vertex AI Hyperparameter Tuning tự động tìm hyperparameter combinations tốt nhất.</p>
+
+<table>
+<thead><tr><th>Search Algorithm</th><th>Description</th></tr></thead>
+<tbody>
+<tr><td><strong>Grid Search</strong></td><td>Exhaustive, expensive; small search space</td></tr>
+<tr><td><strong>Random Search</strong></td><td>Random sampling; often better than grid</td></tr>
+<tr><td><strong>Bayesian Optimization</strong></td><td>Smart search using Gaussian Process; most efficient</td></tr>
+</tbody>
+</table>
+
+<pre><code class="language-text">HPT Job Setup:
+
+hyperparameters:
+  - parameter_id: learning_rate
+    type: DOUBLE
+    min_value: 0.0001
+    max_value: 0.1
+    scale: LOG  ← log scale for LR
+
+  - parameter_id: batch_size
+    type: INTEGER
+    values: [32, 64, 128, 256]
+
+metric:
+  metric_id: val_accuracy
+  goal: MAXIMIZE
+  
+max_trial_count: 50
+parallel_trial_count: 5
+</code></pre>
+
+<h2 id="practice"><strong>6. Practice Questions</strong></h2>
+
+<p><strong>Q1:</strong> A team wants to train a custom TensorFlow model across multiple machines with 8 GPUs each. They want gradients synchronized across all workers without a parameter server. Which TensorFlow distribution strategy should they use?</p>
+<ul>
+<li>A) MirroredStrategy</li>
+<li>B) MultiWorkerMirroredStrategy ✓</li>
+<li>C) ParameterServerStrategy</li>
+<li>D) TPUStrategy</li>
+</ul>
+<p><em>Explanation: MultiWorkerMirroredStrategy enables synchronous data-parallel training across multiple machines, each with multiple GPUs. MirroredStrategy is single-machine multi-GPU only. ParameterServerStrategy uses asynchronous updates. TPUStrategy is for TPU pods.</em></p>
+
+<p><strong>Q2:</strong> A company needs to train an image classification model but their team has no deep learning expertise. They have 5,000 labeled product images. Which Vertex AI option requires the LEAST ML expertise?</p>
+<ul>
+<li>A) Vertex AI Custom Training with TensorFlow CNN</li>
+<li>B) Vertex AI AutoML Image Classification ✓</li>
+<li>C) Dataproc Spark ML</li>
+<li>D) BigQuery ML</li>
+</ul>
+<p><em>Explanation: AutoML Image Classification handles architecture selection, hyperparameter tuning, and training automatically. A team just needs to upload labeled images and specify the task. No code or deep learning expertise is required.</em></p>
+
+<p><strong>Q3:</strong> Which hyperparameter search strategy is MOST efficient when evaluating expensive-to-train deep learning models with a large search space?</p>
+<ul>
+<li>A) Grid Search — tests all combinations</li>
+<li>B) Random Search — samples uniformly</li>
+<li>C) Bayesian Optimization — uses past trial results to guide search ✓</li>
+<li>D) Manual tuning — expert selects parameters</li>
+</ul>
+<p><em>Explanation: Bayesian Optimization builds a probabilistic model of the objective function using Gaussian Processes to intelligently select the next hyperparameter configuration to evaluate, based on past trial results. It finds good configurations with far fewer trials than grid or random search.</em></p>

package/content/series/luyen-thi/luyen-thi-gcp-ml-engineer/chapters/03-phan-3-model-development/lessons/06-bai-6-bigquery-ml-tensorflow.md

@@ -0,0 +1,141 @@
+---
+id: 019c9619-lt03-l06
+title: 'Bài 6: BigQuery ML & TensorFlow on GCP'
+slug: bai-6-bigquery-ml-tensorflow
+description: >-
+  BigQuery ML: CREATE MODEL syntax, supported models.
+  TensorFlow Extended (TFX) pipeline components.
+  TFServing, TFLite. Model optimization techniques.
+duration_minutes: 60
+is_free: true
+video_url: null
+sort_order: 6
+section_title: "Phần 3: Model Development trên Vertex AI"
+course:
+  id: 019c9619-lt03-7003-c003-lt0300000003
+  title: 'Luyện thi Google Cloud Professional Machine Learning Engineer'
+  slug: luyen-thi-gcp-ml-engineer
+---
+
+<div style="text-align: center; margin: 2rem 0;">
+<img src="/storage/uploads/2026/04/gcp-mle-bai6-bqml-tfx.png" alt="BigQuery ML & TFX Pipeline" style="max-width: 800px; width: 100%; border-radius: 12px;" />
+<p><em>BigQuery ML và TFX Pipeline: train models bằng SQL, optimize model, và production ML pipelines</em></p>
+</div>
+
+<h2 id="bigquery-ml"><strong>1. BigQuery ML (BQML)</strong></h2>
+
+<p>BigQuery ML cho phép data analysts train và serve ML models bằng SQL trong BigQuery — không cần export data, không cần biết framework ML.</p>
+
+<pre><code class="language-text">BigQuery ML Workflow:
+
+1. CREATE MODEL → train
+2. ML.EVALUATE() → evaluate metrics
+3. ML.PREDICT() → generate predictions
+4. ML.EXPLAIN_PREDICT() → SHAP-based explanations
+5. EXPORT MODEL → export to Cloud Storage (TF SavedModel format)
+</code></pre>
+
+<table>
+<thead><tr><th>Model Type</th><th>BQML Option</th><th>Task</th></tr></thead>
+<tbody>
+<tr><td>Linear Regression</td><td>LINEAR_REG</td><td>Regression</td></tr>
+<tr><td>Logistic Regression</td><td>LOGISTIC_REG</td><td>Binary/Multiclass classification</td></tr>
+<tr><td>K-Means</td><td>KMEANS</td><td>Clustering</td></tr>
+<tr><td>XGBoost</td><td>BOOSTED_TREE_CLASSIFIER / BOOSTED_TREE_REGRESSOR</td><td>Tabular classification/regression</td></tr>
+<tr><td>Random Forest</td><td>RANDOM_FOREST_CLASSIFIER / RANDOM_FOREST_REGRESSOR</td><td>Tabular classification/regression</td></tr>
+<tr><td>DNN</td><td>DNN_CLASSIFIER / DNN_REGRESSOR</td><td>Complex patterns</td></tr>
+<tr><td>Wide &amp; Deep</td><td>WIDE_AND_DEEP_CLASSIFIER</td><td>Recommendations (memorization + generalization)</td></tr>
+<tr><td>AutoML</td><td>AUTOML_CLASSIFIER / AUTOML_REGRESSOR</td><td>Automated model selection</td></tr>
+<tr><td>Time Series</td><td>ARIMA_PLUS</td><td>Forecasting</td></tr>
+<tr><td>Matrix Factorization</td><td>MATRIX_FACTORIZATION</td><td>Collaborative filtering</td></tr>
+</tbody>
+</table>
+
+<blockquote>
+<p><strong>Exam tip:</strong> BQML ARIMA_PLUS tự động xử lý seasonality, holiday effects, trend decomposition. Khi đề hỏi "forecast using BigQuery data" → ARIMA_PLUS. Khi hỏi "recommendation system in BigQuery" → MATRIX_FACTORIZATION.</p>
+</blockquote>
+
+<h2 id="tfx"><strong>2. TensorFlow Extended (TFX)</strong></h2>
+
+<p>TFX là production ML pipeline library dành cho TensorFlow. Cung cấp standard components cho mỗi bước trong ML lifecycle.</p>
+
+<table>
+<thead><tr><th>TFX Component</th><th>Purpose</th></tr></thead>
+<tbody>
+<tr><td><strong>ExampleGen</strong></td><td>Ingest data từ CSV, BigQuery, Avro, Parquet</td></tr>
+<tr><td><strong>StatisticsGen</strong></td><td>Compute statistics về training data</td></tr>
+<tr><td><strong>SchemaGen</strong></td><td>Infer schema từ statistics</td></tr>
+<tr><td><strong>ExampleValidator</strong></td><td>Detect anomalies: missing, distribution skew</td></tr>
+<tr><td><strong>Transform</strong></td><td>Feature engineering (Apache Beam-based)</td></tr>
+<tr><td><strong>Trainer</strong></td><td>Train TF model (EvalSpec + TrainSpec)</td></tr>
+<tr><td><strong>Tuner</strong></td><td>Hyperparameter tuning (KerasTuner)</td></tr>
+<tr><td><strong>Evaluator</strong></td><td>Evaluate model against baseline</td></tr>
+<tr><td><strong>ModelValidator</strong></td><td>Validate model meets quality thresholds</td></tr>
+<tr><td><strong>Pusher</strong></td><td>Push model to serving (TF Serving, Vertex AI)</td></tr>
+</tbody>
+</table>
+
+<pre><code class="language-text">TFX Pipeline (simplified):
+
+ExampleGen → StatisticsGen → SchemaGen → ExampleValidator
+                ↓
+            Transform (feature engineering)
+                ↓
+            Trainer (model training)
+                ↓
+            Evaluator (metrics vs baseline)
+                ↓ (if pass)
+            Pusher → TF Serving / Vertex AI Endpoint
+</code></pre>
+
+<h2 id="tf-serving"><strong>3. TF Serving & TFLite</strong></h2>
+
+<table>
+<thead><tr><th>Option</th><th>Use Case</th></tr></thead>
+<tbody>
+<tr><td><strong>TF Serving</strong></td><td>High-performance serving trên server/cloud (gRPC or REST)</td></tr>
+<tr><td><strong>TFLite</strong></td><td>Mobile devices, edge devices, microcontrollers</td></tr>
+<tr><td><strong>TF.js</strong></td><td>Browser-based inference</td></tr>
+</tbody>
+</table>
+
+<h2 id="model-optimization"><strong>4. Model Optimization Techniques</strong></h2>
+
+<table>
+<thead><tr><th>Technique</th><th>Description</th><th>Trade-off</th></tr></thead>
+<tbody>
+<tr><td><strong>Quantization</strong></td><td>Float32 → INT8 weights</td><td>4x smaller, ~2x faster, slight accuracy loss</td></tr>
+<tr><td><strong>Pruning</strong></td><td>Remove low-weight connections</td><td>Smaller model, preserve accuracy</td></tr>
+<tr><td><strong>Knowledge Distillation</strong></td><td>Train small "student" model from large "teacher"</td><td>Smaller + fast, slight accuracy loss</td></tr>
+<tr><td><strong>TensorRT</strong></td><td>NVIDIA GPU optimization (layer fusion)</td><td>3-5x inference speedup on NVIDIA GPUs</td></tr>
+</tbody>
+</table>
+
+<h2 id="practice"><strong>5. Practice Questions</strong></h2>
+
+<p><strong>Q1:</strong> A data analyst team needs to build a sales forecasting model on data already in BigQuery. They are comfortable with SQL but have no Python/ML framework experience. Which BigQuery ML model type should they use for time series forecasting?</p>
+<ul>
+<li>A) KMEANS</li>
+<li>B) LOGISTIC_REG</li>
+<li>C) ARIMA_PLUS ✓</li>
+<li>D) MATRIX_FACTORIZATION</li>
+</ul>
+<p><em>Explanation: BigQuery ML ARIMA_PLUS is designed for time series forecasting and automatically handles seasonality, trend, and holiday effects. It can be trained with a simple CREATE MODEL statement in SQL, requiring no Python expertise.</em></p>
+
+<p><strong>Q2:</strong> A TFX pipeline is detecting that the distribution of the "age" feature in new production data differs significantly from the training data distribution. Which TFX component is responsible for detecting this anomaly?</p>
+<ul>
+<li>A) StatisticsGen</li>
+<li>B) SchemaGen</li>
+<li>C) ExampleValidator ✓</li>
+<li>D) Transform</li>
+</ul>
+<p><em>Explanation: ExampleValidator compares data statistics against the expected schema and flags anomalies including distribution skew (significant difference between training and serving data distributions). StatisticsGen computes statistics; SchemaGen creates the schema; Transform does feature engineering.</em></p>
+
+<p><strong>Q3:</strong> A team needs to deploy a TensorFlow image classification model to mobile devices with limited compute resources. They need to reduce model size by 4x with minimal accuracy loss. Which technique should they apply?</p>
+<ul>
+<li>A) Knowledge Distillation</li>
+<li>B) Model Pruning</li>
+<li>C) Post-training quantization (INT8) ✓</li>
+<li>D) TensorRT optimization</li>
+</ul>
+<p><em>Explanation: Post-training quantization converts Float32 weights to INT8, reducing model size by approximately 4x and improving inference speed by 2x, with minimal accuracy loss for most models. TFLite supports INT8 quantization for mobile/edge deployment. TensorRT is for NVIDIA GPUs, not mobile.</em></p>