@groupby/ai-dev 0.5.7 → 0.5.9

package/teams/fhr-ai-team/skills/naming-conventions-reviewer/SKILL.md

@@ -0,0 +1,230 @@
+---
+name: naming-conventions-reviewer
+description: "Expert reviewer for naming conventions across Crownpeak/Earlybirds ML repositories. Use when: (1) reviewing code for naming consistency, (2) writing new code in any algo.*, toolbox, or pipeline repo, (3) naming new datasets, pipeline steps, config keys, or strategy IDs, (4) checking variable names match cross-repo conventions, (5) creating Kubeflow pipeline configs. Covers: dataset names, parquet columns, GCS paths, LakeFS repos, Docker images, strategy IDs, algorithm names, config class patterns, label constants, and ID naming across Python/Scala/JSON."
+---
+
+# Naming Conventions Reviewer
+
+## Review Workflow
+
+1. Identify the repo and language context (Python, Scala, JSON/YAML)
+2. Check names against the conventions below and the reference files
+3. Flag violations with the correct canonical name
+4. For ambiguous cases, check [domain-vocabulary.md](references/domain-vocabulary.md) for the authoritative pattern
+
+## Core Rules
+
+### Rule 1: Language-Appropriate Casing
+| Context | Convention |
+|---|---|
+| Python variables/functions | `snake_case` |
+| Python constants | `SCREAMING_SNAKE_CASE` |
+| Python classes | `PascalCase` |
+| Scala variables | `camelCase` |
+| Scala classes/case classes | `PascalCase` |
+| JSON/YAML config keys | `camelCase` |
+| Parquet column names | `camelCase` |
+| GCS path segments | `kebab-case` |
+| Docker image names | `kebab-case` |
+| Strategy IDs | `kebab-case` |
+| Environment variables | `SCREAMING_SNAKE_CASE` |
+
+### Rule 2: Entity Naming - Use "Item", Not "Product"
+The canonical entity name is **item**, not product. In ML code:
+- `item_id` (Python), `itemId` (Scala/JSON), NOT `product_id`
+- `item_data`, `ItemDataDataset`, NOT `product_data`
+- Exception: LakeFS repo `rawproducts` uses "products" (legacy)
+
+### Rule 3: "Encodings" Over "Embeddings"
+This codebase prefers `encodings` for vector representations:
+- `item_encodings`, `query_encodings`, `image_encodings`
+- `_image_encodings_label = "imageEncodings"` (class-level on DataLoader)
+- `compute_item_encodings()`, `compute_query_encoding()`
+- `items-encoding` (strategy ID), `item-encoding-export`
+- Exception: some newer code uses `embeddings` - prefer `encodings` for consistency
+
+### Rule 4: Parquet Column Labels — Class-Level, Not Module-Level
+
+Parquet column name labels MUST be defined as **class-level attributes on the Dataset or DataLoader class** that reads the parquet data. Do NOT define them as standalone module-level SCREAMING_SNAKE constants.
+
+**Correct — class-level on Dataset (dominant pattern across 7+ repos):**
+```python
+class ItemTextDataDataset(TorchDataset):
+    _tenant_id_label = "tenantId"
+    _item_kind_label = "itemKind"
+    _item_id_label = "itemId"
+    _description_label = "description"
+    _item_seo_key_phrases_label = "itemSEOKeyPhrasesOpt"
+```
+
+**Wrong — standalone module-level constants:**
+```python
+# DO NOT do this in ML repos
+ITEM_ID_LABEL = "itemId"
+TENANT_ID_LABEL = "tenantId"
+```
+
+**Naming format:** `_<field_name>_label = "<camelCaseParquetColumn>"`
+- Prefix with `_` (class-private)
+- Attribute name in snake_case
+- Value in camelCase (matching parquet column produced by Scala)
+- Value must NEVER be snake_case (`"item_id"` is wrong, `"itemId"` is correct)
+
+**Inheritance:** Toolbox base classes define shared labels, ML repos inherit or extend:
+```python
+# item-toolbox: base class defines common labels
+class ItemDataDataLoader(SequenceExampleFeatureDataLoader, ABC):
+    _tenant_id_label = "tenantId"
+    _item_kind_label = "itemKind"
+    _item_id_label = "itemId"
+    _locale_label = "locale"
+    _image_encodings_label = "imageEncodings"
+
+# algo.tagging-ml: extends with domain-specific labels
+class TransformerTaggingDataLoader(SequenceExampleFeatureDataLoader):
+    _item_kind_label = "itemKind"
+    _targets_label = "targets"
+```
+
+**Cross-referencing:** When another class needs a label value owned by a Dataset class, reference the class attribute directly rather than duplicating:
+```python
+# Prefer: reference the owning class
+column = ItemTextDataDataset._item_id_label
+
+# Avoid: duplicating the string literal
+column = "itemId"
+```
+
+**Dataclass models:** When a dataclass needs labels for `to_dict()`, use `ClassVar` attributes with camelCase values matching the Dataset class:
+```python
+@dataclass
+class ItemTextData:
+    _tenant_id_label: ClassVar[str] = "tenantId"    # NOT "tenant_id"
+    _item_id_label: ClassVar[str] = "itemId"         # NOT "item_id"
+```
+
+### Rule 5: Config Class Suffixes
+| Suffix | Purpose |
+|---|---|
+| `*Config` | General configuration |
+| `*ModelConfig` | Model hyperparameters |
+| `*LearningAlgorithmConfig` | Training algorithm config |
+| `*EvaluatorConfig` | Evaluation settings |
+| `*BatchConfig` | Kubeflow batch job config |
+| `*PreProcessingPipelineConfig` | Data preprocessing config |
+| `*DatasetMetaInfo` | Dataset structure metadata |
+
+### Rule 6: Parameter Constants
+Pattern: `{NAME}_PARAM = *Param(...)`
+- Optional params: `{NAME}_OPT_PARAM`
+```python
+BATCH_SIZE_PARAM = IntParam(...)
+LEARNING_RATE_PARAM = FloatParam(...)
+NB_OBSERVATION_BY_TENANT_OPT_PARAM = OptionalIntParam(...)  # _OPT_ for optional
+```
+
+### Rule 7: Dataset Class Naming
+Pattern: `{Domain}{DataType}Dataset`
+- `QueryDataDataset`, NOT `QueryDataset`
+- `ItemTextDataDataset`, NOT `ItemTextDataset`
+- `ItemCutoutDataDataset`
+
+MetaInfo pattern: `{Domain}DatasetMetaInfo`
+Config pattern: `{Domain}DatasetConfig`
+
+### Rule 8: Algorithm Name Constants
+Pattern: `{ALGO}_ALGO_NAME = "{kebab-case-value}"`
+```python
+FM_ALGO_NAME = "fm"
+SEARCH_ALGO_NAME = "search"
+CLIP_ALGO_NAME = "clip"
+SHOP_THE_LOOK_ALGO_NAME = "shop-the-look"
+```
+
+### Rule 9: Service Model Names
+Pattern: `{TYPE}_SERVICE_MODEL_NAME = "{type}-service-model"`
+```python
+ITEM_ENCODING_SERVICE_MODEL_NAME = "item-encoding-service-model"
+QUERY_ENCODING_SERVICE_MODEL_NAME = "query-encoding-service-model"
+IMAGE_ENCODING_SERVICE_MODEL_NAME = "image-encoding-service-model"
+```
+
+### Rule 10: Strategy IDs
+Format: `kebab-case`, structured as `{domain}-{operation}`
+- Learning: `semantic-search-learning`, `clip-learning`, `transformer-tagging-learning`
+- Preprocessing: `query_dataset_preprocessing`, `item_data_dataset_preprocessing` (note: legacy uses underscores)
+- Encoding: `item-images-single-encoding`, `item-cutout-image-encoding`
+- Export: `item-encoding-export`, `ftp-exporter`
+
+### Rule 11: GCS Path Naming
+Segments use kebab-case:
+```
+gs://xo-{env}-ai-eu-eb-algo-models/{algo}/{step}/{version}/{predictor_id}/{timestamp}/{artifact-type}
+```
+Artifact types: `query-training-dataset-preprocessing-dataframe`, `item-data-dataset-preprocessing-dataframe`, `multi-image-item-encodings`
+
+### Rule 12: Pipeline Task Naming
+Pattern: `create_{step_name}_task`
+```python
+create_query_dataset_generation_task
+create_item_data_dataset_preprocessing_task
+create_items_encoding_task
+create_learning_task
+create_evaluation_task
+```
+
+### Rule 13: Kubeflow Config Arguments
+Always camelCase in YAML arguments block:
+```yaml
+arguments:
+  preprocessingRootPath: "gs://..."
+  modelRootPath: "gs://..."
+  itemDataPreprocessingRootPath: "gs://..."
+```
+
+### Rule 14: Docker Image Names
+| Pattern | Examples |
+|---|---|
+| `algo-{algo}-batch` | `algo-fm-batch`, `algo-stl-batch` |
+| `algo-{algo}-dataproc-batch` | `algo-nlp-dataproc-batch`, `algo-search-dataproc-batch` |
+| `{algo}` (Python ML) | `semantic-search`, `tagging`, `image-encoder` |
+| `ebap-{service}` | `ebap-ftp-exporter`, `ebap-fhr-exporter` |
+
+### Rule 15: LakeFS Repository Names
+Lowercase, no separators: `rawproducts`, `rawanalyticsincremental`, `mappedanalyticsincremental`
+
+### Rule 16: Batch Config File Location
+Always at `config/batch.py` or `config/batches.py` within an ML repo.
+
+### Rule 17: Image Type Column Names
+Parquet columns for image types: `stdImages`, `cropImages`, `cutoutImages`, `topTotalImages`, `otherImages`
+
+### Rule 18: KPI Template Names
+Pattern: `*_KPI_TEMPLATE_NAME = "batch-*-kpi"`
+```python
+TENANT_ATTRIBUTION_KPI_TEMPLATE_NAME = "batch-tenant-attribution-kpi"
+```
+
+## Reference Files
+
+- **[repo-dependency-graph.md](references/repo-dependency-graph.md)** - Read when reviewing cross-repo dependencies, Docker image references, or understanding the library hierarchy (earlybirds_commons -> toolboxes -> algo repos)
+- **[dataset-naming.md](references/dataset-naming.md)** - Read when reviewing dataset class names, parquet column names, GCS paths, LakeFS repos, or pipeline preprocessing step names
+- **[domain-vocabulary.md](references/domain-vocabulary.md)** - Read when reviewing specific variable names, checking ID patterns, or looking up the canonical constant name for a domain concept
+
+## Common Violations to Flag
+
+1. Using `product` instead of `item` in ML code
+2. Using `embeddings` where `encodings` is the convention
+3. snake_case in JSON config keys (should be camelCase)
+4. camelCase in Python variables (should be snake_case)
+5. **Module-level SCREAMING_SNAKE label constants** (`ITEM_ID_LABEL = "itemId"`) — should be class-level `_item_id_label = "itemId"` on the Dataset/DataLoader class
+6. **snake_case values in label constants** (`_item_id_label = "item_id"`) — parquet column values must be camelCase (`"itemId"`)
+7. **Duplicated label string literals** — reference the owning Dataset class attribute instead of repeating `"itemId"` in multiple places
+8. **Dataclass labels diverging from Dataset labels** — `models.py` ClassVar labels must use the same camelCase values as `datasets.py` class attributes
+9. Config class missing standard suffix (`Config`, `BatchConfig`, etc.)
+10. Parameter constant missing `_PARAM` suffix
+11. Optional parameter missing `_OPT_` in name
+12. Strategy ID using underscores instead of kebab-case (or vice versa for legacy preprocessing)
+13. GCS path segments using camelCase instead of kebab-case
+14. Algorithm name not matching the canonical registry
+15. Docker image name not following `algo-{name}-batch` or `{name}` pattern

package/teams/fhr-ai-team/skills/naming-conventions-reviewer/references/dataset-naming.md

@@ -0,0 +1,190 @@
+# Dataset Naming Conventions
+
+## Table of Contents
+- [Dataset Class Names](#dataset-class-names)
+- [Dataset MetaInfo Classes](#dataset-metainfo-classes)
+- [Parquet Column Names](#parquet-column-names)
+- [GCS Path Patterns](#gcs-path-patterns)
+- [LakeFS Repository Names](#lakefs-repository-names)
+- [Data Split Names](#data-split-names)
+- [Pipeline Dataset Step Names](#pipeline-dataset-step-names)
+
+## Dataset Class Names
+
+### Naming Pattern
+`{Domain}{DataType}Dataset` - always suffixed with `Dataset`
+
+### PyTorch Datasets (inherit from TorchDataset in pytorch-toolbox)
+| Class Name | Repo | Description |
+|---|---|---|
+| `QueryDataDataset` | algo.semantic-search-ml, algo.semantic-search-bge-m3-ml | Query text data |
+| `ItemTextDataDataset` | algo.semantic-search-ml, algo.semantic-search-bge-m3-ml | Item text/descriptions |
+| `QueryItemTextDataDataset` | algo.semantic-search-ml | Combined query+item text |
+| `ItemDataDataset` | algo.image-generative-tagging | General item data |
+| `ItemCutoutDataDataset` | algo.image-generative-tagging | Item cutout images |
+| `EnrichedItemDataDataset` | algo.image-generative-tagging | Enriched item data |
+| `TorchDataset` | pytorch-toolbox | Abstract base class |
+
+### TensorFlow Datasets (eb_tensorflow patterns)
+Dataset handling done through config-driven loaders, not explicit Dataset subclasses.
+
+### Config Classes
+Pattern: `{Domain}DatasetConfig`
+- `TorchDatasetConfig` (pytorch-toolbox)
+- `ItemDataDatasetConfig` (algo.image-generative-tagging)
+- `ItemCutoutDataDatasetConfig` (algo.image-generative-tagging)
+- `SemanticSearchSamplingConfig` (algo.semantic-search-ml)
+
+## Dataset MetaInfo Classes
+
+Pattern: `{Domain}DatasetMetaInfo` - metadata about dataset structure
+
+| Class Name | Repo |
+|---|---|
+| `UserIntentDatasetMetaInfo` | algo.user-intent-ml |
+| `ActivitiesDatasetMetaInfo` | algo.gpt-ml |
+| `ItemDataDatasetMetaInfo` | item-toolbox |
+| `ImageSegmentationDatasetMetaInfo` | algo.segmentation |
+| `ImageBoundingBoxesDatasetMetaInfo` | algo.object-detection |
+| `TextEncoderDatasetMetaInfo` | algo.text-encoder-ml |
+
+## Parquet Column Names
+
+**CRITICAL**: Column names in parquet files use **camelCase** (matching Scala/JSON conventions).
+
+### Query Data Columns
+| Column Name | Type | Description |
+|---|---|---|
+| `tenantId` | String | Tenant identifier |
+| `query` | String | Search query text |
+| `sortedTenantItemKeys` | List | Sorted item keys for the query |
+| `sortedTenantItemIdLocales` | List | Sorted item IDs with locales |
+| `sortedNbUniqueSearches` | List[Int] | Number of unique searches per item |
+
+### Item Data Columns
+| Column Name | Type | Description |
+|---|---|---|
+| `tenantId` | String | Tenant identifier |
+| `itemKind` | String | Item kind/type |
+| `itemId` | String | Item identifier |
+| `description` | String | Item description |
+| `shortDescription` | String | Short description |
+| `namedAttributes` | Map | Named attribute key-value pairs |
+| `itemSEOKeyPhrasesOpt` | Optional[String] | SEO keyphrases |
+| `locale` | String | Locale code |
+| `variantId` | String | Variant identifier |
+| `categories` | List[String] | Category hierarchy |
+| `attributes` | Map | Item attributes |
+
+### Image Data Columns
+| Column Name | Type | Description |
+|---|---|---|
+| `stdImages` | List | Standard product images |
+| `cropImages` | List | Cropped images |
+| `cutoutImages` | List | Cutout/transparent background images |
+| `topTotalImages` | List | Top/total images |
+| `otherImages` | List | Other image types |
+| `imageEncodings` | Tensor | Image embedding vectors |
+| `closeupImageEncodings` | Tensor | Closeup image embeddings |
+
+### Model Output Columns
+| Column Name | Type | Context |
+|---|---|---|
+| `item_id` / `itemId` | String | Python snake_case / Parquet camelCase |
+| `tenant_id` / `tenantId` | String | Python snake_case / Parquet camelCase |
+| `image_id` / `imageId` | String | Python snake_case / Parquet camelCase |
+| `is_outfit` | Boolean | Outfit detection flag |
+| `is_facing` | Boolean | Facing detection flag |
+
+## GCS Path Patterns
+
+### Bucket Naming
+`gs://xo-{environment}-ai-eu-eb-algo-models/` - Primary model/data bucket
+`gs://xo-{environment}-ai-eu-mlflow/` - MLflow artifacts
+`gs://xo-{environment}-ai-eu-eb-dumps/` - Data dumps
+`gs://xo-{environment}-ai-eu-eb-spark-tmp/` - Spark temp data
+
+### Dataset Path Structure
+```
+gs://xo-{env}-ai-eu-eb-algo-models/{algo}/{step}/{version}/{predictor_id}/{timestamp}/{artifact_type}
+```
+
+### Common Dataset Paths
+| Path Pattern | Dataset Type |
+|---|---|
+| `search/query_dataset_preprocessing/{v}/{pid}/{ts}/query-training-dataset-preprocessing-dataframe` | Query training data |
+| `search/search_item_data_dataset_preprocessing/{v}/{pid}/{ts}/item-data-dataset-preprocessing-dataframe` | Item data for search |
+| `clip/clip_item_data_dataset_preprocessing/...` | CLIP item data |
+| `clip/item-image-cutout-images-preprocessing/...` | Cutout images for CLIP |
+| `gpt/preprocessing/{v}/{pid}/{ts}` | GPT preprocessing |
+| `gpt/gpt_item_data_dataset_preprocessing/{v}/{pid}/{ts}` | GPT item data |
+| `dev/als/preprocessing/{pid}/{ts}` | ALS preprocessing |
+| `dev/image-encoder/image/{v}/{pid}/{ts}/multi-image-item-encodings` | Image encodings |
+| `dev/nlp/character-tokenizer/{v}/{pid}` | Character tokenizer |
+| `dev/nlp/word-tokenizer/{v}/{pid}/queries-tokenizer` | Word tokenizer |
+
+### Artifact Type Names (leaf directory)
+- `query-training-dataset-preprocessing-dataframe`
+- `item-data-dataset-preprocessing-dataframe`
+- `multi-image-item-encodings`
+- `queries-tokenizer`
+- `item-image-outfit-classification`
+- `fine-tuned-yolo-bounding-boxes`
+
+## LakeFS Repository Names
+
+| Constant | Value | Content |
+|---|---|---|
+| `RAW_PRODUCTS_REPO_NAME` | `"rawproducts"` | Raw product catalog data |
+| `RAW_ANALYTICS_INCREMENTAL_REPO_NAME` | `"rawanalyticsincremental"` | Raw analytics events |
+| `MAPPED_ANALYTICS_INCREMENTAL_REPO_NAME` | `"mappedanalyticsincremental"` | Mapped analytics data |
+| `MAPPED_PRODUCTS_REPO_NAME` | (from earlybirds_commons) | Mapped product data |
+
+### LakeFS Path Pattern
+```
+lakefs://{REPO_NAME}/{BRANCH_OR_VERSION}/{tenant_uid}/...
+lakefs://rawproducts/default/{tenant_uid}/products
+lakefs://rawanalyticsincremental/default/{tenant_uid}/...
+lakefs://mappedanalyticsincremental/{version}/{tenant_uid}/...
+```
+
+Default branch: `DEFAULT_BRANCH_NAME = "default"`
+
+## Data Split Names
+
+| Name | Usage Context |
+|---|---|
+| `training` / `train` | Training split |
+| `validation` / `val` | Validation split |
+| `test` | Test split |
+| `preprocessing` | Intermediate preprocessing outputs |
+
+Methods: `_split_dataset()`, `_retrieve_training_and_evaluation_datasets()`
+
+## Pipeline Dataset Step Names
+
+### Preprocessing Steps (strategy_id values)
+| Strategy ID | Description |
+|---|---|
+| `query_dataset_preprocessing` | Query dataset generation |
+| `item_data_dataset_preprocessing` | Item data preprocessing |
+| `search_item_data_dataset_preprocessing` | Search-specific item data |
+| `clip_item_data_dataset_preprocessing` | CLIP-specific item data |
+| `gpt_item_data_dataset_preprocessing` | GPT-specific item data |
+| `character_tokenizer_preprocessing` | NLP character tokenizer |
+| `word_tokenizer_preprocessing` | NLP word tokenizer |
+| `item-image-cutout-images-preprocessing` | Image cutout preprocessing |
+| `item-images-preprocessing` | General image preprocessing |
+| `item-images-single-encoding` | Single image encoding |
+| `cutout-item-images-preprocessing` | Cutout image processing |
+| `visual-search-preprocessing` | Visual search data prep |
+| `global_preprocessing` | FM global preprocessing |
+| `complementarity_preprocessing` | FM complementarity preprocessing |
+
+### Data Formats
+| Format | Usage |
+|---|---|
+| `.parquet` | Standard tabular data (preferred) |
+| `.ipc` | Apache Arrow IPC (cached parquet) |
+| `.tfrecord` | Legacy TensorFlow datasets |
+| `.csv` / `.tsv` | Some data exports (`csv_separator="\t\t"`) |