claude-autopm 2.12.3 → 3.0.0

package/packages/plugin-ml/commands/ml:feature-engineering.md

@@ -0,0 +1,668 @@
+---
+command: ml:feature-engineering
+plugin: ml
+category: ml-operations
+description: Automated feature generation, selection, and transformation
+tags:
+  - ml
+  - feature-engineering
+  - feature-selection
+  - data-transformation
+  - featuretools
+tools:
+  - @python-backend-expert
+  - Read
+  - Write
+  - Bash
+usage: |
+  /ml:feature-engineering --input data.csv --target sales --auto-generate --select-top 20
+examples:
+  - input: /ml:feature-engineering --input data.csv --encode categorical --scale numerical
+    description: Apply encoding and scaling
+  - input: /ml:feature-engineering --auto-generate --max-depth 2 --select mutual-info
+    description: Auto-generate features and select best
+---
+
+# /ml:feature-engineering
+
+Automated feature generation, selection, and transformation following industry best practices from Featuretools, scikit-learn, category-encoders, and feature-engine.
+
+## Required Documentation Access
+
+**MANDATORY:** Before feature engineering, query Context7 for best practices:
+
+**Documentation Queries:**
+- `mcp://context7/featuretools/automated-feature-engineering` - Featuretools patterns for Deep Feature Synthesis
+- `mcp://context7/scikit-learn/feature-selection` - RFE, LASSO, mutual information methods
+- `mcp://context7/pandas/transformations` - Data transformation and aggregation patterns
+- `mcp://context7/category-encoders/encoding` - Categorical encoding strategies (one-hot, target, WOE)
+- `mcp://context7/feature-engine/transformers` - Feature engineering transformers and pipelines
+- `mcp://context7/scikit-learn/preprocessing` - Scaling, binning, polynomial features
+- `mcp://context7/time-series/feature-extraction` - Lag, rolling, seasonal features
+- `mcp://context7/nlp/text-features` - TF-IDF, embeddings, text statistics
+
+**Why This is Required:**
+- Ensures use of proven feature engineering methodologies (Featuretools DFS, sklearn pipelines)
+- Prevents feature leakage and data snooping
+- Applies correct encoding strategies for categorical variables
+- Validates feature selection approaches against current research
+- Ensures proper handling of time-series and text data
+- Follows reproducibility best practices for feature definitions
+
+## Purpose
+
+Automate the complex and time-consuming process of feature engineering by:
+- **Auto-generating** features using Deep Feature Synthesis (Featuretools)
+- **Selecting** most important features using multiple methods (RFE, LASSO, mutual info)
+- **Encoding** categorical variables with optimal strategies
+- **Transforming** numerical features (scaling, binning, polynomial)
+- **Creating** time-series features (lag, rolling windows, seasonal)
+- **Extracting** text features (TF-IDF, embeddings)
+- **Analyzing** feature importance and interactions
+- **Handling** missing values with appropriate strategies
+
+## Command Structure
+
+```bash
+/ml:feature-engineering [OPTIONS]
+```
+
+### Options
+
+#### Input/Output
+- `--input <path>` - Input dataset (CSV, Parquet, JSON)
+- `--output <path>` - Output feature matrix (default: features.csv)
+- `--target <column>` - Target variable column name
+- `--export-definitions <path>` - Save feature definitions for reuse
+
+#### Automated Feature Generation
+- `--auto-generate` - Enable Deep Feature Synthesis (Featuretools)
+- `--entities <json>` - Entity definitions for multi-table data
+- `--relationships <json>` - Relationships between entities
+- `--max-depth <n>` - Maximum feature depth (default: 2)
+- `--primitives <list>` - Primitives to use (sum, mean, max, min, etc.)
+- `--custom-primitives <path>` - Custom primitive definitions
+
+#### Feature Selection
+- `--select <method>` - Selection method: rfe, rfecv, lasso, mutual_info, chi2
+- `--select-top <n>` - Select top N features
+- `--remove-correlated <threshold>` - Remove features with correlation > threshold
+- `--variance-threshold <value>` - Remove low-variance features
+
+#### Categorical Encoding
+- `--encode <method>` - Encoding method: onehot, target, ordinal, binary, woe, catboost
+- `--encode-columns <list>` - Columns to encode (default: auto-detect)
+- `--high-cardinality-threshold <n>` - Use binary encoding above this threshold
+
+#### Numerical Transformations
+- `--scale <method>` - Scaling: standard, minmax, robust
+- `--polynomial-degree <n>` - Create polynomial features up to degree N
+- `--interactions-only` - Create only interaction features, not powers
+- `--bin <column:bins>` - Bin numerical features (e.g., age:5, income:quantile:10)
+- `--log-transform <columns>` - Apply log transformation to skewed features
+
+#### Time-Series Features
+- `--lags <column:lags>` - Create lag features (e.g., value:1,2,3)
+- `--rolling <column:window:func>` - Rolling statistics (e.g., value:7:mean,std)
+- `--datetime-components` - Extract year, month, day, dayofweek, etc.
+- `--seasonal-period <n>` - Extract seasonal components with period N
+
+#### Text Features
+- `--text-column <column>` - Text column for feature extraction
+- `--tfidf` - Create TF-IDF features
+- `--max-features <n>` - Maximum text features (default: 100)
+- `--ngram-range <min:max>` - N-gram range (default: 1:1)
+
+#### Missing Value Handling
+- `--impute <strategy>` - Imputation strategy: mean, median, mode, knn, ffill, bfill
+- `--create-missing-indicators` - Create binary indicators for missing values
+- `--missing-threshold <value>` - Drop features with > threshold missing
+
+#### Analysis
+- `--analyze-importance` - Calculate feature importance
+- `--detect-interactions` - Detect feature interactions
+- `--visualize` - Generate visualizations
+- `--validate` - Validate feature matrix quality
+
+## Workflow
+
+### 1. Auto-Generate Features (Featuretools)
+
+```bash
+/ml:feature-engineering \
+  --input sales_data/ \
+  --entities entities.json \
+  --relationships relationships.json \
+  --auto-generate \
+  --max-depth 2 \
+  --primitives sum,mean,max,min,std
+```
+
+**entities.json:**
+```json
+{
+  "customers": {
+    "data": "customers.csv",
+    "index": "customer_id"
+  },
+  "transactions": {
+    "data": "transactions.csv",
+    "index": "transaction_id"
+  }
+}
+```
+
+**relationships.json:**
+```json
+[
+  {
+    "parent": "customers",
+    "parent_column": "customer_id",
+    "child": "transactions",
+    "child_column": "customer_id"
+  }
+]
+```
+
+### 2. Feature Selection
+
+```bash
+# RFE with cross-validation
+/ml:feature-engineering \
+  --input features.csv \
+  --target sales \
+  --select rfecv \
+  --select-top 20
+
+# LASSO regularization
+/ml:feature-engineering \
+  --input features.csv \
+  --target sales \
+  --select lasso \
+  --variance-threshold 0.01
+
+# Mutual information
+/ml:feature-engineering \
+  --input features.csv \
+  --target category \
+  --select mutual_info \
+  --remove-correlated 0.95
+```
+
+### 3. Categorical Encoding
+
+```bash
+# One-hot encoding
+/ml:feature-engineering \
+  --input data.csv \
+  --encode onehot \
+  --encode-columns category,region,type
+
+# Target encoding with validation
+/ml:feature-engineering \
+  --input data.csv \
+  --target sales \
+  --encode target \
+  --encode-columns category
+
+# Binary encoding for high cardinality
+/ml:feature-engineering \
+  --input data.csv \
+  --encode binary \
+  --high-cardinality-threshold 10
+```
+
+### 4. Numerical Transformations
+
+```bash
+# Scaling and polynomial features
+/ml:feature-engineering \
+  --input data.csv \
+  --scale standard \
+  --polynomial-degree 2 \
+  --interactions-only
+
+# Binning and log transform
+/ml:feature-engineering \
+  --input data.csv \
+  --bin age:5 income:quantile:10 \
+  --log-transform income,revenue
+```
+
+### 5. Time-Series Features
+
+```bash
+/ml:feature-engineering \
+  --input timeseries.csv \
+  --lags value:1,2,3,7 \
+  --rolling value:7:mean,std value:30:mean \
+  --datetime-components \
+  --seasonal-period 7
+```
+
+### 6. Text Features
+
+```bash
+/ml:feature-engineering \
+  --input reviews.csv \
+  --text-column review_text \
+  --tfidf \
+  --max-features 100 \
+  --ngram-range 1:2
+```
+
+### 7. Complete Pipeline
+
+```bash
+/ml:feature-engineering \
+  --input data.csv \
+  --target sales \
+  --impute mean \
+  --create-missing-indicators \
+  --encode onehot \
+  --scale standard \
+  --polynomial-degree 2 \
+  --select mutual_info \
+  --select-top 50 \
+  --analyze-importance \
+  --detect-interactions \
+  --export-definitions feature_pipeline.json \
+  --output engineered_features.csv
+```
+
+## Implementation Approach
+
+### @python-backend-expert Tasks
+
+1. **Setup Environment**
+   ```python
+   # Install dependencies
+   pip install featuretools scikit-learn category-encoders feature-engine pandas numpy
+   ```
+
+2. **Parse Command Arguments**
+   - Extract all options from command line
+   - Validate input files exist
+   - Check option compatibility
+
+3. **Load and Validate Data**
+   ```python
+   import pandas as pd
+   df = pd.read_csv(input_path)
+   validate_data_quality(df)
+   ```
+
+4. **Execute Feature Generation Pipeline**
+   - Auto-generate features (if enabled)
+   - Apply transformations in correct order
+   - Handle missing values
+   - Encode categorical variables
+   - Transform numerical features
+   - Create time-series features
+   - Extract text features
+   - Select features
+   - Analyze importance
+
+5. **Export Results**
+   - Save feature matrix
+   - Export feature definitions (JSON)
+   - Generate visualization reports
+   - Create summary statistics
+
+### Key Libraries Integration
+
+#### Featuretools (Deep Feature Synthesis)
+```python
+import featuretools as ft
+
+# Create entity set
+es = ft.EntitySet(id="sales")
+es = es.add_dataframe(dataframe_name="customers", dataframe=customers_df, index="customer_id")
+es = es.add_dataframe(dataframe_name="transactions", dataframe=transactions_df, index="transaction_id")
+
+# Define relationships
+es = es.add_relationship("customers", "customer_id", "transactions", "customer_id")
+
+# Generate features
+feature_matrix, feature_defs = ft.dfs(
+    entityset=es,
+    target_dataframe_name="customers",
+    max_depth=2,
+    trans_primitives=["sum", "mean", "std", "max", "min"]
+)
+```
+
+#### Scikit-learn (Feature Selection)
+```python
+from sklearn.feature_selection import RFECV, SelectFromModel, mutual_info_classif
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.linear_model import LassoCV
+
+# RFE with cross-validation
+selector = RFECV(estimator=RandomForestClassifier(), cv=5)
+X_selected = selector.fit_transform(X, y)
+
+# LASSO
+lasso = LassoCV(cv=5)
+lasso.fit(X, y)
+selected_features = X.columns[lasso.coef_ != 0]
+
+# Mutual information
+mi_scores = mutual_info_classif(X, y)
+top_features = X.columns[np.argsort(mi_scores)[-20:]]
+```
+
+#### Category-Encoders
+```python
+import category_encoders as ce
+
+# One-hot encoding
+encoder = ce.OneHotEncoder(cols=['category', 'region'])
+X_encoded = encoder.fit_transform(X)
+
+# Target encoding
+encoder = ce.TargetEncoder(cols=['category'])
+X_encoded = encoder.fit_transform(X, y)
+
+# Binary encoding for high cardinality
+encoder = ce.BinaryEncoder(cols=['user_id'])
+X_encoded = encoder.fit_transform(X)
+```
+
+#### Feature-engine
+```python
+from feature_engine.imputation import MeanMedianImputer
+from feature_engine.transformation import LogTransformer
+from feature_engine.discretisation import EqualFrequencyDiscretiser
+
+# Imputation
+imputer = MeanMedianImputer(imputation_method='mean', variables=['age', 'income'])
+X_imputed = imputer.fit_transform(X)
+
+# Log transformation
+log_tf = LogTransformer(variables=['income', 'revenue'])
+X_log = log_tf.fit_transform(X_imputed)
+
+# Binning
+discretiser = EqualFrequencyDiscretiser(q=5, variables=['age'])
+X_binned = discretiser.fit_transform(X_log)
+```
+
+#### Pandas (Time-Series & Transformations)
+```python
+# Lag features
+for lag in [1, 2, 3, 7]:
+    df[f'value_lag_{lag}'] = df['value'].shift(lag)
+
+# Rolling statistics
+df['value_rolling_mean_7'] = df['value'].rolling(window=7).mean()
+df['value_rolling_std_7'] = df['value'].rolling(window=7).std()
+
+# Datetime components
+df['year'] = df['date'].dt.year
+df['month'] = df['date'].dt.month
+df['dayofweek'] = df['date'].dt.dayofweek
+df['quarter'] = df['date'].dt.quarter
+
+# GroupBy aggregations
+customer_features = df.groupby('customer_id').agg({
+    'amount': ['sum', 'mean', 'std', 'max', 'min'],
+    'transaction_id': 'count'
+})
+```
+
+## Output
+
+### Feature Matrix
+```
+engineered_features.csv
+- All original features
+- Generated features
+- Transformed features
+- Selected features only (if selection applied)
+```
+
+### Feature Definitions
+```json
+{
+  "version": "1.0",
+  "timestamp": "2025-10-21T10:30:00Z",
+  "transformations": [
+    {
+      "type": "imputation",
+      "strategy": "mean",
+      "columns": ["age", "income"],
+      "params": {"age_mean": 35.5, "income_mean": 65000}
+    },
+    {
+      "type": "encoding",
+      "method": "onehot",
+      "columns": ["category"],
+      "mapping": {"A": [1,0,0], "B": [0,1,0], "C": [0,0,1]}
+    },
+    {
+      "type": "scaling",
+      "method": "standard",
+      "columns": ["age", "income"],
+      "params": {"age_mean": 35.5, "age_std": 7.2, "income_mean": 65000, "income_std": 15000}
+    },
+    {
+      "type": "selection",
+      "method": "mutual_info",
+      "selected_features": ["feature1", "feature2", "feature3"],
+      "scores": [0.45, 0.38, 0.32]
+    }
+  ],
+  "feature_count": {
+    "original": 10,
+    "generated": 25,
+    "after_selection": 15
+  }
+}
+```
+
+### Analysis Report
+```
+Feature Engineering Summary
+==========================
+
+Input:
+- Rows: 10,000
+- Original features: 10
+- Missing values: 5.2%
+
+Generated Features:
+- Deep Feature Synthesis: 25 features
+- Polynomial features: 15 features
+- Time-series features: 12 features
+- Total generated: 52 features
+
+Transformations:
+✓ Missing values imputed (mean strategy)
+✓ Categorical encoded (one-hot: 3 columns)
+✓ Numerical scaled (standard scaler)
+✓ Log transform applied (2 columns)
+
+Feature Selection:
+- Method: Mutual Information
+- Features before: 62
+- Features after: 20
+- Removed correlated: 8 pairs (r > 0.95)
+- Removed low variance: 3 features
+
+Feature Importance (Top 10):
+1. customer_sum_amount: 0.245
+2. rolling_mean_7: 0.198
+3. age_income_interaction: 0.176
+4. month: 0.154
+5. category_encoded: 0.132
+...
+
+Feature Interactions Detected:
+- age × income: strength 0.82
+- month × dayofweek: strength 0.67
+
+Output:
+- Final feature count: 20
+- Feature matrix saved: engineered_features.csv
+- Definitions saved: feature_pipeline.json
+```
+
+## Best Practices
+
+### 1. Avoid Feature Leakage
+- Never use target in feature generation
+- Separate train/test transformations
+- Use time-aware splits for time-series
+
+### 2. Handle Missing Values First
+```bash
+# Always impute or drop before other transformations
+/ml:feature-engineering \
+  --impute mean \
+  --missing-threshold 0.5 \  # Drop if >50% missing
+  --create-missing-indicators
+```
+
+### 3. Encode Before Scaling
+```bash
+# Correct order: encode → scale → select
+/ml:feature-engineering \
+  --encode onehot \
+  --scale standard \
+  --select mutual_info
+```
+
+### 4. Use Cross-Validation for Selection
+```bash
+# RFECV is more robust than RFE
+/ml:feature-engineering \
+  --select rfecv  # Instead of --select rfe
+```
+
+### 5. Export Definitions for Production
+```bash
+# Always export for reproducibility
+/ml:feature-engineering \
+  --export-definitions pipeline.json
+
+# Apply same transformations to new data
+/ml:feature-engineering \
+  --input new_data.csv \
+  --apply-definitions pipeline.json
+```
+
+### 6. Validate Feature Quality
+```bash
+/ml:feature-engineering \
+  --validate \
+  --variance-threshold 0.01 \
+  --remove-correlated 0.95
+```
+
+## Common Patterns
+
+### Pattern 1: Automated Feature Discovery
+```bash
+# Let Featuretools discover features
+/ml:feature-engineering \
+  --auto-generate \
+  --max-depth 2 \
+  --select mutual_info \
+  --select-top 50
+```
+
+### Pattern 2: Manual Feature Engineering
+```bash
+# Explicit transformations
+/ml:feature-engineering \
+  --encode target \
+  --polynomial-degree 2 \
+  --lags value:1,2,3 \
+  --rolling value:7:mean,std
+```
+
+### Pattern 3: High-Dimensional Data
+```bash
+# Aggressive feature reduction
+/ml:feature-engineering \
+  --variance-threshold 0.05 \
+  --remove-correlated 0.90 \
+  --select lasso \
+  --select-top 100
+```
+
+### Pattern 4: Time-Series Focus
+```bash
+# Time-series specific features
+/ml:feature-engineering \
+  --lags value:1,7,14,30 \
+  --rolling value:7:mean,std,min,max \
+  --datetime-components \
+  --seasonal-period 7 \
+  --impute ffill
+```
+
+## Error Handling
+
+### Missing Required Arguments
+```
+Error: --target is required when using --select or --encode target
+Solution: Specify target column: --target sales
+```
+
+### Incompatible Options
+```
+Error: Cannot use --auto-generate without --entities and --relationships
+Solution: Provide entity definitions or disable --auto-generate
+```
+
+### Feature Leakage Detection
+```
+Warning: Target variable 'sales' appears in feature names
+Recommendation: Review feature generation to prevent leakage
+```
+
+## Integration
+
+### With Model Training
+```bash
+# Feature engineering → Model training
+/ml:feature-engineering \
+  --input data.csv \
+  --target churn \
+  --output features.csv \
+  --export-definitions pipeline.json
+
+/ml:train \
+  --input features.csv \
+  --target churn \
+  --model random-forest
+```
+
+### With Hyperparameter Tuning
+```bash
+# Find best feature engineering params
+for depth in 1 2 3; do
+  /ml:feature-engineering \
+    --auto-generate \
+    --max-depth $depth \
+    --output features_depth_${depth}.csv
+
+  /ml:train --input features_depth_${depth}.csv --evaluate
+done
+```
+
+## Related Commands
+
+- `/ml:data-validation` - Validate data quality before feature engineering
+- `/ml:train` - Train models using engineered features
+- `/ml:evaluate` - Evaluate feature importance
+- `/ml:deploy` - Deploy feature engineering pipeline
+
+## References
+
+- Featuretools Documentation: https://docs.featuretools.com/
+- Scikit-learn Feature Selection: https://scikit-learn.org/stable/modules/feature_selection.html
+- Category Encoders: https://contrib.scikit-learn.org/category_encoders/
+- Feature Engine: https://feature-engine.trainindata.com/