@groupby/ai-dev 0.5.7 → 0.5.9

package/teams/fhr-ai-team/skills/ml-tooling-dev/references/mongodb-config.md

@@ -0,0 +1,218 @@
+# MongoDB Configuration Reference
+
+## Connection
+
+| Parameter | Value |
+|-----------|-------|
+| Host | `mongodb://10.11.96.21:27017` |
+| Database | `earlybirds` |
+| Collection | `predictors` |
+| Tool | `mongosh` (install via `brew install mongosh` if missing) |
+
+---
+
+## Config Structure
+
+Training hyperparameters live inside each predictor document at:
+
+```
+config.batch.<strategy-id>
+```
+
+For semantic search learning, the path is:
+
+```
+config.batch.semantic-search-learning
+├── modelConfig
+│   ├── pretrainedModelNameOrPath    (e.g. "intfloat/multilingual-e5-large")
+│   ├── poolingType                  ("MEAN", "CLS", "LAST_TOKEN")
+│   ├── lossType                     ("CONTRASTIVE", "TRIPLET", "COSINE")
+│   ├── similarityScale              (float, e.g. 20.0)
+│   └── contrastiveLossConfig
+│       ├── similarityScale          (float, e.g. 20.0)
+│       └── temperature              (float, e.g. 1.0)
+├── pipelineConfig
+│   ├── maxSequenceLength            (int, e.g. 32, 64, 128)
+│   ├── pretrainedModelNameOrPath    (same as modelConfig)
+│   ├── queryPrefix                  (e.g. "query: ")
+│   └── itemPrefix                   (e.g. "passage: ")
+└── learningConfig
+    ├── batchSize                    (int, dataset loading batch size)
+    ├── useEvaluation                (bool)
+    ├── useEarlyStopping             (bool)
+    ├── earlyStoppingPatience        (int)
+    ├── earlyStoppingThreshold       (float)
+    ├── evaluationSplitRatio         (float, e.g. 0.1)
+    ├── onnxQuantization             { "quantize": true }
+    └── trainingArguments
+        ├── numTrainEpochs                (int)
+        ├── perDeviceTrainBatchSize       (int, in-batch negatives count)
+        ├── perDeviceEvalBatchSize        (int)
+        ├── gradientAccumulationSteps     (int)
+        ├── gradientCheckpointing         (bool, required for large batch on L4)
+        ├── evaluationStrategy            ("epoch" | "steps")
+        ├── saveStrategy                  ("epoch" | "steps")
+        ├── loggingStrategy               ("steps")
+        ├── loggingSteps                  (int, e.g. 50)
+        ├── warmupSteps                   (int, e.g. 400)
+        ├── weightDecay                   (float, e.g. 0.01)
+        ├── loadBestModelAtEnd            (bool)
+        ├── metricForBestModel            ("eval_loss")
+        ├── bf16                          (bool)
+        ├── fp16                          (bool)
+        ├── dataloaderNumWorkers          (int)
+        ├── dataloaderPinMemory           (bool)
+        ├── dataloaderPersistentWorkers   (bool)
+        ├── dataloaderPrefetchFactor      (int)
+        ├── removeUnusedColumns           (bool)
+        ├── torchCompile                  (bool)
+        └── torchCompileBackend           (string | null)
+```
+
+---
+
+## Common Operations
+
+All read/update/replace operations on `config.batch.<strategy>` go through `scripts/mongo_predictor.py` (see `SKILL.md` for full options). It validates the ObjectId, escapes input through env-bound JSON (no shell injection surface), and preserves `Int32` types on full-replace.
+
+### Read current config
+
+```bash
+python3 scripts/mongo_predictor.py read <PREDICTOR_ID>
+```
+
+### Update individual fields (preferred for targeted changes)
+
+```bash
+python3 scripts/mongo_predictor.py update <PREDICTOR_ID> \
+  --set pipelineConfig.maxSequenceLength=64 \
+  --set learningConfig.batchSize=128 \
+  --set learningConfig.trainingArguments.perDeviceTrainBatchSize=128 \
+  --set learningConfig.trainingArguments.perDeviceEvalBatchSize=128 \
+  --set learningConfig.trainingArguments.gradientAccumulationSteps=4 \
+  --set learningConfig.trainingArguments.gradientCheckpointing=true
+```
+
+Keys are dot-paths *under* the strategy. Values auto-coerce: `true`/`false`/`null`/int/float/JSON literal/string.
+
+### Replace entire strategy config (for full overrides)
+
+Stage the new config as JSON and apply it. Preview first with `diff`:
+
+```bash
+python3 scripts/mongo_predictor.py diff  <PREDICTOR_ID> --file experiment-A.json
+python3 scripts/mongo_predictor.py apply <PREDICTOR_ID> --file experiment-A.json
+```
+
+Example `experiment-A.json` (top-level matches `config.batch.<strategy>`):
+
+```json
+{
+  "modelConfig": {
+    "pretrainedModelNameOrPath": "intfloat/multilingual-e5-large",
+    "poolingType": "MEAN",
+    "similarityScale": 20.0,
+    "contrastiveLossConfig": { "similarityScale": 20.0, "temperature": 1.0 }
+  },
+  "pipelineConfig": {
+    "maxSequenceLength": 64,
+    "pretrainedModelNameOrPath": "intfloat/multilingual-e5-large",
+    "queryPrefix": "query: ",
+    "itemPrefix": "passage: "
+  },
+  "learningConfig": {
+    "useEvaluation": true,
+    "batchSize": 64,
+    "useEarlyStopping": true,
+    "earlyStoppingPatience": 3,
+    "earlyStoppingThreshold": 0.001,
+    "evaluationSplitRatio": 0.1,
+    "onnxQuantization": { "quantize": true },
+    "trainingArguments": {
+      "numTrainEpochs": 3,
+      "perDeviceTrainBatchSize": 64,
+      "perDeviceEvalBatchSize": 64,
+      "gradientAccumulationSteps": 4,
+      "evaluationStrategy": "epoch",
+      "saveStrategy": "epoch",
+      "loggingStrategy": "steps",
+      "loggingSteps": 50,
+      "warmupSteps": 400,
+      "weightDecay": 0.01,
+      "loadBestModelAtEnd": true,
+      "metricForBestModel": "eval_loss",
+      "bf16": true,
+      "fp16": false,
+      "dataloaderNumWorkers": 4,
+      "dataloaderPinMemory": false,
+      "dataloaderPersistentWorkers": false,
+      "dataloaderPrefetchFactor": 2,
+      "removeUnusedColumns": true,
+      "torchCompile": false,
+      "torchCompileBackend": null,
+      "gradientCheckpointing": false
+    }
+  }
+}
+```
+
+Bare integers in this JSON are stored as `Int32` (the script wraps them in `$numberInt` before sending to mongosh). Bare floats stay as Double.
+
+### Verify config was read by a running pod
+
+The training pod reads the MongoDB config at startup. Grep the pod logs to confirm. Use `kf_logs.py` to skip the manual pod-name lookup:
+
+```bash
+python3 scripts/kf_logs.py <run_id> --step get-container-base-task-2 --tail 500 \
+  | grep -E 'maxSequenceLength:|perDeviceTrainBatchSize:|gradientAccumulationSteps:|gradientCheckpointing:|numTrainEpochs:'
+```
+
+Pod naming on this cluster: `<workflow-name>-<numeric-id>` (e.g., `python-batch-pipeline-26dsn-2177690451`). See `references/kubectl-debug.md` for the full kubectl reference.
+
+### Raw mongosh fallback
+
+If `mongo_predictor.py` is unavailable or you need a one-off query that the script does not cover, use `mongosh` directly:
+
+```bash
+mongosh "mongodb://10.11.96.21:27017/earlybirds" --quiet --eval '
+  const doc = db.predictors.findOne({_id: ObjectId("<PREDICTOR_ID>")});
+  print(JSON.stringify(doc.config.batch["semantic-search-learning"], null, 2));
+'
+```
+
+Remember the `NumberInt()` rule for full-document `$set` (see Gotchas below).
+
+---
+
+## Sequential Launch Pattern (Multiple Experiments Sharing One Predictor)
+
+All runs for the same predictor read config from the **same MongoDB document**. The basic loop:
+
+1. `mongo_predictor.py apply` Config A
+2. Launch run A (`python -m run -c ...`)
+3. `kf_wait.py <run_id> --step get-container-base-task-2 --state RUNNING` (the training pod has now read the config)
+4. `mongo_predictor.py apply` Config B
+5. Launch run B
+6. Repeat
+
+```bash
+python3 scripts/mongo_predictor.py apply <PREDICTOR_ID> --file experiment-A.json
+# launch run A, capture <run_id_A> from output
+python3 scripts/kf_wait.py <run_id_A> --step get-container-base-task-2 --state RUNNING --timeout 600
+
+python3 scripts/mongo_predictor.py apply <PREDICTOR_ID> --file experiment-B.json
+# launch run B
+python3 scripts/kf_wait.py <run_id_B> --step get-container-base-task-2 --state RUNNING --timeout 600
+```
+
+**Critical:** if you apply the next config before the previous run's pod reaches `RUNNING`, that run gets the wrong config. Always wait, then verify with the `kf_logs.py` grep above.
+
+---
+
+## Key Gotchas
+
+- **Database is `earlybirds`**, not `ebap`. Early attempts used `ebap` which is wrong.
+- **Use `NumberInt()` for integer values** when doing full-document `$set` to avoid MongoDB storing them as doubles. Dot-notation `$set` with bare integers is fine.
+- **`batchSize` under `learningConfig`** is the dataset loading batch size; **`perDeviceTrainBatchSize` under `trainingArguments`** is what controls in-batch negatives for contrastive learning. Keep them in sync for consistency.
+- **`gradientCheckpointing: true`** is required when `perDeviceTrainBatchSize` is large (128+) on L4 GPUs to avoid OOM
+- **`pretrainedModelNameOrPath`** appears in BOTH `modelConfig` and `pipelineConfig`; update both when changing the base model

package/teams/fhr-ai-team/skills/ml-tooling-dev/references/pipeline-configs.md

@@ -0,0 +1,190 @@
+# Pipeline Configuration Reference
+
+## Endpoints
+- **Kubeflow UI**: http://10.11.96.10/
+- **MLflow UI**: http://10.11.96.16:5000/
+- **Config files location**: `attraqt-kubeflow-configs/configs/development/`
+- **Pipeline definitions**: `attraqt-kubeflow-pipelines/kubeflow_pipelines/pipelines/`
+
+---
+
+## Launching a Run from a Config File
+
+```bash
+# Must be run from the scripts/ directory of attraqt-kubeflow-configs
+cd attraqt-kubeflow-configs/scripts
+python -m run -c <absolute_path_to_config.json>
+```
+
+**Prerequisites:**
+- `attraqt-kubeflow-configs/.env` must exist (copy from `.env.dev` for dev)
+- `version_name` must match an existing pipeline version. Check with:
+  ```bash
+  python3 kf_query.py --pipeline-versions <pipeline_name>
+  ```
+
+**What `run.py` does:** Reads the JSON config, calls `KubeflowClient.create_run()` with `pipeline_name`, `version_name`, `experiment_name` (also used as `run_name`), and `params`.
+
+---
+
+## 1. `python_batch_pipeline`: Python GPU/CPU batch job
+
+**Template** (`configs/development/batch/python_batch_pipeline.json`):
+```json
+{
+    "pipeline_name": "python_batch_pipeline",
+    "version_name": "0.1.271",
+    "experiment_name": "<EXPERIMENT_NAME>",
+    "params": {
+        "predictor_id": "<PREDICTOR_ID>",
+        "strategy_id": "<STRATEGY_ID>",
+        "image_name": "<IMAGE_NAME>",
+        "cmd_script_path": "<SCRIPT_PATH>",
+        "disk_enabled": "False",
+        "disk_name": null,
+        "batch_config": {
+            "arguments": {},
+            "cpu": "1000m",
+            "memory": "2G",
+            "extra_options": "",
+            "gpu": "1",
+            "gpu_vendor": "nvidia.com/gpu",
+            "gpu_accelerator_name": "nvidia-l4",
+            "version": "<IMAGE_VERSION>"
+        }
+    }
+}
+```
+
+**`batch_config` fields:**
+| Field | Description |
+|-------|-------------|
+| `arguments` | Dict passed as job arguments (strategy-specific) |
+| `version` | Docker image version tag |
+| `cpu` | CPU request (e.g. `"1000m"`, `"11"`) |
+| `memory` | Memory request (e.g. `"2G"`, `"80G"`) |
+| `gpu` | GPU count (e.g. `"1"`) |
+| `gpu_vendor` | Always `"nvidia.com/gpu"` |
+| `gpu_accelerator_name` | GPU type: `"nvidia-l4"` or `"nvidia-tesla-t4"` |
+| `extra_options` | JVM/env extra flags string |
+
+---
+
+## 2. `scala_batch_pipeline`: Scala JVM batch job
+
+**Template** (`configs/development/batch/scala_batch_pipeline.json`):
+```json
+{
+    "pipeline_name": "scala_batch_pipeline",
+    "version_name": "0.1.271",
+    "experiment_name": "<EXPERIMENT_NAME>",
+    "params": {
+        "predictor_id": "<PREDICTOR_ID>",
+        "strategy_id": "<STRATEGY_ID>",
+        "image_name": "<IMAGE_NAME>",
+        "cmd_script_path": "/opt/start.sh",
+        "launcher_class": "<MAIN_CLASS>",
+        "disk_enabled": "False",
+        "disk_name": null,
+        "batch_config": {
+            "custom_params": {},
+            "version": "<IMAGE_VERSION>",
+            "cpu": "1000m",
+            "memory": "2G",
+            "java_memory": "1G",
+            "timeout_s": 3600,
+            "extra_options": "",
+            "gpu": "1",
+            "gpu_vendor": "nvidia.com/gpu"
+        }
+    }
+}
+```
+
+**`batch_config` fields:**
+| Field | Description |
+|-------|-------------|
+| `custom_params` | Dict of strategy-specific params (NOT `arguments`) |
+| `version` | Docker image version tag |
+| `java_memory` | JVM heap max (e.g. `"1G"`, `"36G"`) |
+| `timeout_s` | Job timeout in seconds (default `3600`) |
+
+---
+
+## 3. `semantic_search_item_encoding_pipeline`: Full encoding pipeline
+
+**Example** (`configs/development/ai/semantic_search_item_encoding_pipeline/`):
+```json
+{
+    "pipeline_name": "semantic_search_item_encoding_pipeline",
+    "version_name": "0.1.269",
+    "experiment_name": "<NAME> - Encoding workflow",
+    "params": {
+        "search_predictor_id": "<PREDICTOR_ID>",
+        "model_repository": "mlflow",
+        "model_id": "<MLFLOW_RUN_ID>",
+        "item_search_encoding_version": "0",
+        "item_seo_keyphrases_generator": {
+            "cpu": "4",
+            "extra_options": "-DmappedProductsVersion=v1",
+            "custom_params": {
+                "imageDatasetName": "xo",
+                "imageBasePath": "/mnt",
+                "promptId": "SEOKeyPhrasesFashionPrompt"
+            },
+            "java_memory": "2G",
+            "memory": "4G",
+            "timeout_s": 7200,
+            "version": "3.45.0"
+        },
+        "items_encoding": {
+            "cpu": "10",
+            "extra_options": "-DmappedProductsVersion=v1 -DproductCatalogPastValidityInDays=3",
+            "java_memory": "36G",
+            "memory": "40G",
+            "gpu_vendor": "nvidia.com/gpu",
+            "timeout_s": 7200,
+            "version": "3.64.0"
+        }
+    }
+}
+```
+
+---
+
+## Strategy IDs & Image Names Reference
+
+| Strategy | Image | Script | Launcher class |
+|----------|-------|--------|----------------|
+| `semantic-search-learning` | `semantic-search` | `/opt/start-semantic-search-batch.sh` | (n/a) |
+| `semantic-search-evaluation` | `semantic-search` | `/opt/start-semantic-search-batch.sh` | (n/a) |
+| `items-encoding` | `algo-search-batch` | `/opt/start.sh` | `earlybirds.algo.search.batch.SearchSimpleBatchLauncher` |
+| `query-dataset-generation` | `algo-search-batch` | `/opt/start.sh` | `earlybirds.algo.search.batch.SearchSimpleBatchLauncher` |
+| `seo-keyphrases-generator` | `item-utils` | `/opt/start.sh` | `earlybirds.item.dump.seo_keyphrases.SEOKeyPhrasesGeneratorLauncher` |
+| `search-item-data-dataset-preprocessing` | `algo-search-batch` | `/opt/start.sh` | (Spark/Dataproc) |
+
+---
+
+## Docker Image Version Defaults (from `versions.py`)
+
+| Image | Version constant | Current value |
+|-------|-----------------|---------------|
+| `semantic-search` | `SEMANTIC_SEARCH_ML_VERSION` | `0.0.15` (check for overrides) |
+| `algo-search-batch` | `SEARCH_VERSION` | `3.64.0` |
+| `item-utils` | `ITEM_UTILS_VERSION` | `2.23.0` |
+
+> Always check the current `kubeflow_pipelines/pipelines/utils/versions.py` for latest values; these drift.
+
+---
+
+## GCS Path Patterns (outputs from pipeline steps)
+
+| Step | Output key | GCS path pattern |
+|------|-----------|-----------------|
+| item_data_dataset_preprocessing | `item_data_dataset_directory_path` | `search/search_item_data_dataset_preprocessing/<version>/<predictor_id>/<timestamp>/item-data-dataset-preprocessing-dataframe` |
+| item_data_dataset_preprocessing | `item_data_dataset_meta_info_directory_path` | `search/search_item_data_dataset_preprocessing/<version>/<predictor_id>/<timestamp>/item-data-dataset-preprocessing-meta-info` |
+| query_dataset_generation | `query_training_dataset_directory_path` | `search/query-dataset-generation/<version>/<predictor_id>/<timestamp>/query-training-dataset-preprocessing-dataframe` |
+| query_dataset_generation | `query_evaluation_dataset_directory_path` | `search/query-dataset-generation/<version>/<predictor_id>/<timestamp>/query-evaluation-dataset-preprocessing-dataframe` |
+| learning | `run_id` | MLflow run ID (UUID hex string, e.g. `5046f02ec2c146b3b66abcd3b82f15d4`) |
+
+> **Find output paths**: In Kubeflow UI → run → step → "Output artifacts" tab, or via `scripts/kf_query.py`.

package/teams/fhr-ai-team/skills/ml-tooling-dev/references/pipeline-steps.md

@@ -0,0 +1,182 @@
+# Semantic Search Pipeline Steps Reference
+
+## `semantic_search_learning_with_generated_analytics_pipeline`
+
+Full pipeline DAG. Steps run in order (with conditional branches):
+
+```
+1. item_seo_keyphrases_generator          [Scala, item-utils image]
+2. item_data_dataset_preprocessing        [Spark/Dataproc, algo-search-batch]
+3. query_dataset_generation               [Scala, algo-search-batch]  → outputs: training + evaluation dataset paths
+4. learning                               [Python GPU, semantic-search image]  → outputs: mlflow run_id
+   ├── 5. evaluation (conditional)        [Python GPU, semantic-search image]  ; only if evaluation dataset != ""
+   ├── 6. items_encoding (conditional)    [Scala, algo-search-batch]  ; only if items_encoding_enabled=True
+   │   └── 7. set_model_alias (conditional) ; only if model_alias_transition_enabled=True
+   └── send_pubsub_message
+```
+
+---
+
+## Step-by-Step Configs for Manual Re-runs
+
+### Step 4: Learning (`python_batch_pipeline`)
+
+```json
+{
+    "pipeline_name": "python_batch_pipeline",
+    "version_name": "0.1.271",
+    "experiment_name": "<TENANT> - EBAP - Search - Learning workflow",
+    "params": {
+        "predictor_id": "<PREDICTOR_ID>",
+        "strategy_id": "semantic-search-learning",
+        "image_name": "semantic-search",
+        "cmd_script_path": "/opt/start-semantic-search-batch.sh",
+        "batch_config": {
+            "arguments": {
+                "itemDataDatasetPreprocessingDirectoryPath": "<FROM_STEP_2_item_data_dataset_directory_path>",
+                "pretrainedModelKey": {
+                    "modelId": "<PRETRAINED_RUN_ID_or_huggingface_model_id>",
+                    "modelRepository": "<mlflow_or_huggingface>"
+                },
+                "queryDatasetPreprocessingDirectoryPath": "<FROM_STEP_3_query_training_dataset_directory_path>",
+                "verticalRunPredictorIds": ["<PREDICTOR_ID>"]
+            },
+            "cpu": "11",
+            "gpu": "1",
+            "gpu_accelerator_name": "nvidia-l4",
+            "gpu_vendor": "nvidia.com/gpu",
+            "memory": "80G",
+            "version": "<SEMANTIC_SEARCH_ML_VERSION>"
+        }
+    }
+}
+```
+
+**Required inputs:**
+- `itemDataDatasetPreprocessingDirectoryPath` → from step 2 output
+- `queryDatasetPreprocessingDirectoryPath` → from step 3 `query_training_dataset_directory_path`
+- `pretrainedModelKey` → either a HuggingFace model (e.g. `sentence-transformers/all-MiniLM-L6-v2`) or an MLflow run_id from a previous training
+
+**Output:** MLflow `run_id`. Find via MLflow UI or `scripts/mlflow_query.py`.
+
+---
+
+### Step 5: Evaluation (`python_batch_pipeline`)
+
+```json
+{
+    "pipeline_name": "python_batch_pipeline",
+    "version_name": "0.1.271",
+    "experiment_name": "<TENANT> - EBAP - Search - Learning workflow",
+    "params": {
+        "predictor_id": "<PREDICTOR_ID>",
+        "strategy_id": "semantic-search-evaluation",
+        "image_name": "semantic-search",
+        "cmd_script_path": "/opt/start-semantic-search-batch.sh",
+        "batch_config": {
+            "arguments": {
+                "itemDataDatasetPreprocessingDirectoryPath": "<FROM_STEP_2_item_data_dataset_directory_path>",
+                "pretrainedModelKey": {
+                    "modelId": "<MLFLOW_RUN_ID_FROM_STEP_4>",
+                    "modelRepository": "mlflow"
+                },
+                "queryDatasetPreprocessingDirectoryPath": "<FROM_STEP_3_query_evaluation_dataset_directory_path>"
+            },
+            "cpu": "11",
+            "gpu": "1",
+            "gpu_accelerator_name": "nvidia-l4",
+            "gpu_vendor": "nvidia.com/gpu",
+            "memory": "80G",
+            "version": "<SEMANTIC_SEARCH_ML_VERSION>"
+        }
+    }
+}
+```
+
+**Key differences from learning:**
+- `strategy_id`: `semantic-search-evaluation`
+- `queryDatasetPreprocessingDirectoryPath` → uses the **evaluation** dataset path (step 3's `query_evaluation_dataset_directory_path`), not the training one
+- No `verticalRunPredictorIds`
+- `pretrainedModelKey.modelId` → MLflow `run_id` from step 4, `modelRepository` = `"mlflow"`
+
+**Skip if:** `query_evaluation_dataset_directory_path` was empty in step 3.
+
+---
+
+### Step 6: Items Encoding (`scala_batch_pipeline`)
+
+```json
+{
+    "pipeline_name": "scala_batch_pipeline",
+    "version_name": "0.1.271",
+    "experiment_name": "<TENANT> - EBAP - Search - Learning workflow",
+    "params": {
+        "predictor_id": "<PREDICTOR_ID>",
+        "strategy_id": "items-encoding",
+        "image_name": "algo-search-batch",
+        "cmd_script_path": "/opt/start.sh",
+        "launcher_class": "earlybirds.algo.search.batch.SearchSimpleBatchLauncher",
+        "batch_config": {
+            "custom_params": {
+                "itemEncodingInferenceKey": {
+                    "modelRepository": "mlflow",
+                    "modelId": "<MLFLOW_RUN_ID_FROM_STEP_4>",
+                    "version": ""
+                }
+            },
+            "version": "3.64.0",
+            "cpu": "10",
+            "memory": "40G",
+            "java_memory": "36G",
+            "timeout_s": 7200,
+            "gpu_vendor": "nvidia.com/gpu",
+            "extra_options": "-DmappedProductsVersion=v1 -DproductCatalogPastValidityInDays=3"
+        }
+    }
+}
+```
+
+**Key notes:**
+- Uses `custom_params` (not `arguments`); it's a Scala job
+- `version: ""` in `itemEncodingInferenceKey` → auto-generates new encoding version
+- `model_id` = MLflow `run_id` from step 4
+
+---
+
+### Step 7: Set Model Alias (MLflow CLI / UI)
+
+```bash
+# Via MLflow Python client
+python3 - <<'EOF'
+import mlflow
+mlflow.set_tracking_uri("http://10.11.96.16:5000/")
+client = mlflow.MlflowClient()
+client.set_registered_model_alias(
+    name="semantic-search-<predictor_id>",
+    alias="Production",
+    version="<MODEL_VERSION>"
+)
+EOF
+```
+
+Or do it directly in the MLflow UI at http://10.11.96.16:5000/.
+
+---
+
+## Other Common Pipeline Types
+
+### `semantic_search_item_encoding_pipeline` (standalone encoding)
+
+Use when re-running only the encoding step with a new model. Runs SEO keyphrases generator + items encoding. See `pipeline-configs.md` for full config.
+
+### `clip_search_rnn_learning_pipeline`
+
+Learning pipeline for CLIP+RNN search model. Config example in `configs/development/ai/clip_search_rnn_learning_pipeline/`.
+
+---
+
+## How to Find Output Paths from a Completed Step
+
+1. **Kubeflow UI**: http://10.11.96.10/ → Runs → click run → click step → "Output artifacts" tab
+2. **Via script**: `python3 scripts/kf_query.py <run_id>` → shows all step outputs
+3. **Pattern**: Outputs follow `<strategy>/<version>/<predictor_id>/<epoch_ms>/...` in GCS