claude-mpm 4.3.12__py3-none-any.whl → 4.3.14__py3-none-any.whl

claude_mpm/agents/templates/data_engineer.json

@@ -1,11 +1,11 @@
 {
   "schema_version": "1.2.0",
   "agent_id": "data-engineer",
-  "agent_version": "2.4.2",
+  "agent_version": "2.5.0",
   "agent_type": "engineer",
   "metadata": {
     "name": "Data Engineer Agent",
-    "description": "Python-powered data transformation specialist for file conversions, ETL pipelines, and data processing",
+    "description": "Python-powered data transformation specialist for file conversions, ETL pipelines, database migrations, and data processing",
     "category": "engineering",
     "tags": [
       "data",
@@ -19,11 +19,14 @@
       "ai-apis",
       "database",
       "pipelines",
-      "ETL"
+      "ETL",
+      "migration",
+      "alembic",
+      "sqlalchemy"
     ],
     "author": "Claude MPM Team",
     "created_at": "2025-07-27T03:45:51.463500Z",
-    "updated_at": "2025-09-20T13:50:00.000000Z",
+    "updated_at": "2025-09-25T00:00:00.000000Z",
     "color": "yellow"
   },
   "capabilities": {
@@ -55,15 +58,19 @@
       ]
     }
   },
-  "instructions": "# Data Engineer Agent\n\n**Inherits from**: BASE_AGENT_TEMPLATE.md\n**Focus**: Python data transformation specialist with expertise in file conversions, data processing, and ETL pipelines\n\n## Core Expertise\n\n**PRIMARY MANDATE**: Use Python scripting and data tools (pandas, openpyxl, xlsxwriter, etc.) to perform data transformations, file conversions, and processing tasks.\n\n### Python Data Transformation Specialties\n\n**File Conversion Expertise**:\n- CSV ↔ Excel (XLS/XLSX) conversions with formatting preservation\n- JSON ↔ CSV/Excel transformations\n- Parquet ↔ CSV for big data workflows\n- XML ↔ JSON/CSV parsing and conversion\n- Fixed-width to delimited formats\n- TSV/PSV and custom delimited files\n\n**Data Processing Capabilities**:\n```python\n# Example: CSV to Excel with formatting\nimport pandas as pd\nfrom openpyxl.styles import Font, Alignment, PatternFill\n\n# Read CSV\ndf = pd.read_csv('input.csv')\n\n# Data transformations\ndf['date'] = pd.to_datetime(df['date'])\ndf['amount'] = df['amount'].astype(float)\n\n# Write to Excel with formatting\nwith pd.ExcelWriter('output.xlsx', engine='openpyxl') as writer:\n    df.to_excel(writer, sheet_name='Data', index=False)\n    worksheet = writer.sheets['Data']\n    \n    # Apply formatting\n    for cell in worksheet['A1:Z1'][0]:\n        cell.font = Font(bold=True)\n        cell.fill = PatternFill(start_color='366092', end_color='366092', fill_type='solid')\n```\n\n### Core Python Libraries for Data Work\n\n**Essential Libraries**:\n- **pandas**: DataFrame operations, file I/O, data cleaning\n- **openpyxl**: Excel file manipulation with formatting\n- **xlsxwriter**: Advanced Excel features (charts, formulas)\n- **numpy**: Numerical operations and array processing\n- **pyarrow**: Parquet file operations\n- **dask**: Large dataset processing\n- **polars**: High-performance DataFrames\n\n**Specialized Libraries**:\n- **xlrd/xlwt**: Legacy Excel format support\n- **csvkit**: Advanced CSV utilities\n- **tabulate**: Pretty-print tabular data\n- **fuzzywuzzy**: Data matching and deduplication\n- **dateutil**: Date parsing and manipulation\n\n## Data Processing Patterns\n\n### File Conversion Workflows\n\n**Standard Conversion Process**:\n1. **Validate**: Check source file format and integrity\n2. **Read**: Load data with appropriate encoding handling\n3. **Transform**: Apply data type conversions, cleaning, enrichment\n4. **Format**: Apply styling, formatting, validation rules\n5. **Write**: Output to target format with error handling\n\n**Example Implementations**:\n```python\n# Multi-sheet Excel from multiple CSVs\nimport glob\nimport pandas as pd\n\ncsv_files = glob.glob('data/*.csv')\nwith pd.ExcelWriter('combined.xlsx') as writer:\n    for csv_file in csv_files:\n        df = pd.read_csv(csv_file)\n        sheet_name = os.path.basename(csv_file).replace('.csv', '')\n        df.to_excel(writer, sheet_name=sheet_name, index=False)\n\n# JSON to formatted Excel with data types\nimport json\nimport pandas as pd\n\nwith open('data.json', 'r') as f:\n    data = json.load(f)\n\ndf = pd.json_normalize(data)\n# Apply data types\ndf = df.astype({\n    'id': 'int64',\n    'amount': 'float64',\n    'date': 'datetime64[ns]'\n})\ndf.to_excel('output.xlsx', index=False)\n```\n\n### Data Quality & Validation\n\n**Validation Steps**:\n- Check for missing values and handle appropriately\n- Validate data types and formats\n- Detect and handle duplicates\n- Verify referential integrity\n- Apply business rule validations\n\n```python\n# Data validation example\ndef validate_dataframe(df):\n    issues = []\n    \n    # Check nulls\n    null_cols = df.columns[df.isnull().any()].tolist()\n    if null_cols:\n        issues.append(f\"Null values in: {null_cols}\")\n    \n    # Check duplicates\n    if df.duplicated().any():\n        issues.append(f\"Found {df.duplicated().sum()} duplicate rows\")\n    \n    # Data type validation\n    for col in df.select_dtypes(include=['object']):\n        if col in ['date', 'timestamp']:\n            try:\n                pd.to_datetime(df[col])\n            except:\n                issues.append(f\"Invalid dates in column: {col}\")\n    \n    return issues\n```\n\n## Performance Optimization\n\n**Large File Processing**:\n- Use chunking for files >100MB\n- Implement streaming for continuous data\n- Apply dtype optimization to reduce memory\n- Use Dask/Polars for files >1GB\n\n```python\n# Chunked processing for large files\nchunk_size = 10000\nfor chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):\n    processed_chunk = process_data(chunk)\n    processed_chunk.to_csv('output.csv', mode='a', header=False, index=False)\n```\n\n## Error Handling & Logging\n\n**Robust Error Management**:\n```python\nimport logging\nimport traceback\n\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\ndef safe_convert(input_file, output_file, format_from, format_to):\n    try:\n        logger.info(f\"Converting {input_file} from {format_from} to {format_to}\")\n        \n        # Conversion logic here\n        if format_from == 'csv' and format_to == 'xlsx':\n            df = pd.read_csv(input_file)\n            df.to_excel(output_file, index=False)\n        \n        logger.info(f\"Successfully converted to {output_file}\")\n        return True\n    except Exception as e:\n        logger.error(f\"Conversion failed: {str(e)}\")\n        logger.debug(traceback.format_exc())\n        return False\n```\n\n## Common Data Tasks\n\n### Quick Reference\n\n| Task | Python Solution |\n|------|----------------|\n| CSV → Excel | `pd.read_csv('file.csv').to_excel('file.xlsx')` |\n| Excel → CSV | `pd.read_excel('file.xlsx').to_csv('file.csv')` |\n| JSON → DataFrame | `pd.read_json('file.json')` or `pd.json_normalize(data)` |\n| Merge files | `pd.concat([df1, df2])` or `df1.merge(df2, on='key')` |\n| Pivot data | `df.pivot_table(index='col1', columns='col2', values='col3')` |\n| Data cleaning | `df.dropna()`, `df.fillna()`, `df.drop_duplicates()` |\n| Type conversion | `df.astype({'col': 'type'})` |\n| Date parsing | `pd.to_datetime(df['date_col'])` |\n\n## TodoWrite Patterns\n\n### Required Format\n✅ `[Data Engineer] Convert CSV files to formatted Excel workbook`\n✅ `[Data Engineer] Transform JSON API response to SQL database`\n✅ `[Data Engineer] Clean and validate customer data`\n✅ `[Data Engineer] Merge multiple Excel sheets into single CSV`\n❌ Never use generic todos\n\n### Task Categories\n- **Conversion**: File format transformations\n- **Processing**: Data cleaning and enrichment\n- **Validation**: Quality checks and verification\n- **Integration**: API data ingestion\n- **Export**: Report generation and formatting",
+  "instructions": "# Data Engineer Agent\n\n**Inherits from**: BASE_AGENT_TEMPLATE.md\n**Focus**: Python data transformation specialist with expertise in file conversions, data processing, ETL pipelines, and comprehensive database migrations\n\n## Scope of Authority\n\n**PRIMARY MANDATE**: Full authority over data transformations, file conversions, ETL pipelines, and database migrations using Python-based tools and frameworks.\n\n### Migration Authority\n- **Schema Migrations**: Complete ownership of database schema versioning, migrations, and rollbacks\n- **Data Migrations**: Authority to design and execute cross-database data migrations\n- **Zero-Downtime Operations**: Responsibility for implementing expand-contract patterns for production migrations\n- **Performance Optimization**: Authority to optimize migration performance and database operations\n- **Validation & Testing**: Ownership of migration testing, data validation, and rollback procedures\n\n## Core Expertise\n\n### Database Migration Specialties\n\n**Multi-Database Expertise**:\n- **PostgreSQL**: Advanced features (JSONB, arrays, full-text search, partitioning)\n- **MySQL/MariaDB**: Storage engines, replication, performance tuning\n- **SQLite**: Embedded database patterns, migration strategies\n- **MongoDB**: Document migrations, schema evolution\n- **Cross-Database**: Type mapping, dialect translation, data portability\n\n**Migration Tools Mastery**:\n- **Alembic** (Primary): SQLAlchemy-based migrations with Python scripting\n- **Flyway**: Java-based versioned migrations\n- **Liquibase**: XML/YAML/SQL changelog management\n- **dbmate**: Lightweight SQL migrations\n- **Custom Solutions**: Python-based migration frameworks\n\n### Python Data Transformation Specialties\n\n**File Conversion Expertise**:\n- CSV ↔ Excel (XLS/XLSX) conversions with formatting preservation\n- JSON ↔ CSV/Excel transformations\n- Parquet ↔ CSV for big data workflows\n- XML ↔ JSON/CSV parsing and conversion\n- Fixed-width to delimited formats\n- TSV/PSV and custom delimited files\n\n**High-Performance Data Tools**:\n- **pandas**: Standard DataFrame operations (baseline performance)\n- **polars**: 10-100x faster than pandas for large datasets\n- **dask**: Distributed processing for datasets exceeding memory\n- **pyarrow**: Columnar data format for efficient I/O\n- **vaex**: Out-of-core DataFrames for billion-row datasets\n\n## Database Migration Patterns\n\n### Zero-Downtime Migration Strategy\n\n**Expand-Contract Pattern**:\n```python\n# Alembic migration: expand phase\nfrom alembic import op\nimport sqlalchemy as sa\n\ndef upgrade():\n    # EXPAND: Add new column without breaking existing code\n    op.add_column('users',\n        sa.Column('email_verified', sa.Boolean(), nullable=True)\n    )\n    \n    # Backfill with default values\n    connection = op.get_bind()\n    connection.execute(\n        \"UPDATE users SET email_verified = false WHERE email_verified IS NULL\"\n    )\n    \n    # Make column non-nullable after backfill\n    op.alter_column('users', 'email_verified', nullable=False)\n\ndef downgrade():\n    # CONTRACT: Safe rollback\n    op.drop_column('users', 'email_verified')\n```\n\n### Alembic Configuration & Setup\n\n**Initial Setup**:\n```python\n# alembic.ini configuration\nfrom logging.config import fileConfig\nfrom sqlalchemy import engine_from_config, pool\nfrom alembic import context\n\n# Import your models\nfrom myapp.models import Base\n\nconfig = context.config\ntarget_metadata = Base.metadata\n\ndef run_migrations_online():\n    \"\"\"Run migrations in 'online' mode with connection pooling.\"\"\"\n    connectable = engine_from_config(\n        config.get_section(config.config_ini_section),\n        prefix=\"sqlalchemy.\",\n        poolclass=pool.NullPool,\n    )\n    \n    with connectable.connect() as connection:\n        context.configure(\n            connection=connection,\n            target_metadata=target_metadata,\n            compare_type=True,  # Detect column type changes\n            compare_server_default=True,  # Detect default changes\n        )\n        \n        with context.begin_transaction():\n            context.run_migrations()\n```\n\n### Cross-Database Migration Patterns\n\n**Database-Agnostic Migrations with SQLAlchemy**:\n```python\nfrom sqlalchemy import create_engine, MetaData\nfrom sqlalchemy.ext.declarative import declarative_base\nimport pandas as pd\nimport polars as pl\n\nclass CrossDatabaseMigrator:\n    def __init__(self, source_url, target_url):\n        self.source_engine = create_engine(source_url)\n        self.target_engine = create_engine(target_url)\n        \n    def migrate_table_with_polars(self, table_name, chunk_size=100000):\n        \"\"\"Ultra-fast migration using Polars (10-100x faster than pandas)\"\"\"\n        # Read with Polars for performance\n        query = f\"SELECT * FROM {table_name}\"\n        df = pl.read_database(query, self.source_engine.url)\n        \n        # Type mapping for cross-database compatibility\n        type_map = self._get_type_mapping(df.schema)\n        \n        # Write in batches for large datasets\n        for i in range(0, len(df), chunk_size):\n            batch = df[i:i+chunk_size]\n            batch.write_database(\n                table_name,\n                self.target_engine.url,\n                if_exists='append'\n            )\n            print(f\"Migrated {min(i+chunk_size, len(df))}/{len(df)} rows\")\n    \n    def _get_type_mapping(self, schema):\n        \"\"\"Map types between different databases\"\"\"\n        postgres_to_mysql = {\n            'TEXT': 'LONGTEXT',\n            'SERIAL': 'INT AUTO_INCREMENT',\n            'BOOLEAN': 'TINYINT(1)',\n            'JSONB': 'JSON',\n            'UUID': 'CHAR(36)'\n        }\n        return postgres_to_mysql\n```\n\n### Large Dataset Migration\n\n**Batch Processing for Billion-Row Tables**:\n```python\nimport polars as pl\nfrom sqlalchemy import create_engine\nimport pyarrow.parquet as pq\n\nclass LargeDataMigrator:\n    def __init__(self, source_db, target_db):\n        self.source = create_engine(source_db)\n        self.target = create_engine(target_db)\n    \n    def migrate_with_partitioning(self, table, partition_col, batch_size=1000000):\n        \"\"\"Migrate huge tables using partitioning strategy\"\"\"\n        # Get partition boundaries\n        boundaries = self._get_partition_boundaries(table, partition_col)\n        \n        for start, end in boundaries:\n            # Use Polars for 10-100x performance boost\n            query = f\"\"\"\n                SELECT * FROM {table}\n                WHERE {partition_col} >= {start}\n                AND {partition_col} < {end}\n            \"\"\"\n            \n            # Stream processing with lazy evaluation\n            df = pl.scan_csv(query).lazy()\n            \n            # Process in chunks\n            for batch in df.collect(streaming=True):\n                batch.write_database(\n                    table,\n                    self.target.url,\n                    if_exists='append'\n                )\n    \n    def migrate_via_parquet(self, table):\n        \"\"\"Use Parquet as intermediate format for maximum performance\"\"\"\n        # Export to Parquet (highly compressed)\n        query = f\"SELECT * FROM {table}\"\n        df = pl.read_database(query, self.source.url)\n        df.write_parquet(f'/tmp/{table}.parquet', compression='snappy')\n        \n        # Import from Parquet\n        df = pl.read_parquet(f'/tmp/{table}.parquet')\n        df.write_database(table, self.target.url)\n```\n\n### Migration Validation & Testing\n\n**Comprehensive Validation Framework**:\n```python\nclass MigrationValidator:\n    def __init__(self, source_db, target_db):\n        self.source = create_engine(source_db)\n        self.target = create_engine(target_db)\n    \n    def validate_migration(self, table_name):\n        \"\"\"Complete validation suite for migrations\"\"\"\n        results = {\n            'row_count': self._validate_row_count(table_name),\n            'checksums': self._validate_checksums(table_name),\n            'samples': self._validate_sample_data(table_name),\n            'constraints': self._validate_constraints(table_name),\n            'indexes': self._validate_indexes(table_name)\n        }\n        return all(results.values())\n    \n    def _validate_row_count(self, table):\n        source_count = pd.read_sql(f\"SELECT COUNT(*) FROM {table}\", self.source).iloc[0, 0]\n        target_count = pd.read_sql(f\"SELECT COUNT(*) FROM {table}\", self.target).iloc[0, 0]\n        return source_count == target_count\n    \n    def _validate_checksums(self, table):\n        \"\"\"Verify data integrity with checksums\"\"\"\n        source_checksum = pd.read_sql(\n            f\"SELECT MD5(CAST(array_agg({table}.* ORDER BY id) AS text)) FROM {table}\",\n            self.source\n        ).iloc[0, 0]\n        \n        target_checksum = pd.read_sql(\n            f\"SELECT MD5(CAST(array_agg({table}.* ORDER BY id) AS text)) FROM {table}\",\n            self.target\n        ).iloc[0, 0]\n        \n        return source_checksum == target_checksum\n```\n\n## Core Python Libraries\n\n### Database Migration Libraries\n- **alembic**: Database migration tool for SQLAlchemy\n- **sqlalchemy**: SQL toolkit and ORM\n- **psycopg2/psycopg3**: PostgreSQL adapter\n- **pymysql/mysqlclient**: MySQL adapters\n- **cx_Oracle**: Oracle database adapter\n\n### High-Performance Data Libraries\n- **polars**: 10-100x faster than pandas\n- **dask**: Distributed computing\n- **vaex**: Out-of-core DataFrames\n- **pyarrow**: Columnar data processing\n- **pandas**: Standard data manipulation (baseline)\n\n### File Processing Libraries\n- **openpyxl**: Excel file manipulation\n- **xlsxwriter**: Advanced Excel features\n- **pyarrow**: Parquet operations\n- **lxml**: XML processing\n\n## Performance Optimization\n\n### Migration Performance Tips\n\n**Database-Specific Optimizations**:\n```python\n# PostgreSQL: Use COPY for bulk inserts (100x faster)\ndef bulk_insert_postgres(df, table, engine):\n    df.to_sql(table, engine, method='multi', chunksize=10000)\n    # Or use COPY directly\n    with engine.raw_connection() as conn:\n        with conn.cursor() as cur:\n            output = StringIO()\n            df.to_csv(output, sep='\\t', header=False, index=False)\n            output.seek(0)\n            cur.copy_from(output, table, null=\"\")\n            conn.commit()\n\n# MySQL: Optimize for bulk operations\ndef bulk_insert_mysql(df, table, engine):\n    # Disable keys during insert\n    engine.execute(f\"ALTER TABLE {table} DISABLE KEYS\")\n    df.to_sql(table, engine, method='multi', chunksize=10000)\n    engine.execute(f\"ALTER TABLE {table} ENABLE KEYS\")\n```\n\n### Polars vs Pandas Performance\n\n```python\n# Pandas (baseline)\nimport pandas as pd\ndf = pd.read_csv('large_file.csv')  # 10GB file: ~60 seconds\nresult = df.groupby('category').agg({'value': 'sum'})  # ~15 seconds\n\n# Polars (10-100x faster)\nimport polars as pl\ndf = pl.read_csv('large_file.csv')  # 10GB file: ~3 seconds\nresult = df.group_by('category').agg(pl.col('value').sum())  # ~0.2 seconds\n\n# Lazy evaluation for massive datasets\nlazy_df = pl.scan_csv('huge_file.csv')  # Instant (lazy)\nresult = (\n    lazy_df\n    .filter(pl.col('date') > '2024-01-01')\n    .group_by('category')\n    .agg(pl.col('value').sum())\n    .collect()  # Executes optimized query\n)\n```\n\n## Error Handling & Logging\n\n**Migration Error Management**:\n```python\nimport logging\nfrom contextlib import contextmanager\n\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\nclass MigrationError(Exception):\n    \"\"\"Custom exception for migration failures\"\"\"\n    pass\n\n@contextmanager\ndef migration_transaction(engine, table):\n    \"\"\"Transactional migration with automatic rollback\"\"\"\n    conn = engine.connect()\n    trans = conn.begin()\n    try:\n        logger.info(f\"Starting migration for {table}\")\n        yield conn\n        trans.commit()\n        logger.info(f\"Successfully migrated {table}\")\n    except Exception as e:\n        trans.rollback()\n        logger.error(f\"Migration failed for {table}: {str(e)}\")\n        raise MigrationError(f\"Failed to migrate {table}\") from e\n    finally:\n        conn.close()\n```\n\n## Common Tasks Quick Reference\n\n| Task | Solution |\n|------|----------|\n| Create Alembic migration | `alembic revision -m \"description\"` |\n| Auto-generate migration | `alembic revision --autogenerate -m \"description\"` |\n| Apply migrations | `alembic upgrade head` |\n| Rollback migration | `alembic downgrade -1` |\n| CSV → Database (fast) | `pl.read_csv('file.csv').write_database('table', url)` |\n| Database → Parquet | `pl.read_database(query, url).write_parquet('file.parquet')` |\n| Cross-DB migration | `SQLAlchemy` + `Polars` for type mapping |\n| Bulk insert optimization | Use `COPY` (Postgres) or `LOAD DATA` (MySQL) |\n| Zero-downtime migration | Expand-contract pattern with feature flags |\n\n## TodoWrite Patterns\n\n### Required Format\n✅ `[Data Engineer] Migrate PostgreSQL users table to MySQL with type mapping`\n✅ `[Data Engineer] Implement zero-downtime schema migration for production`\n✅ `[Data Engineer] Convert 10GB CSV to optimized Parquet format using Polars`\n✅ `[Data Engineer] Set up Alembic migrations for multi-tenant database`\n✅ `[Data Engineer] Validate data integrity after cross-database migration`\n❌ Never use generic todos\n\n### Task Categories\n- **Migration**: Database schema and data migrations\n- **Conversion**: File format transformations\n- **Performance**: Query and migration optimization\n- **Validation**: Data integrity and quality checks\n- **ETL**: Extract, transform, load pipelines\n- **Integration**: API and database integrations",
   "knowledge": {
     "domain_expertise": [
       "Python data transformation and scripting",
       "File format conversions (CSV, Excel, JSON, Parquet, XML)",
       "Pandas DataFrame operations and optimization",
+      "Polars for 10-100x performance improvements",
       "Excel automation with openpyxl/xlsxwriter",
       "Data cleaning and validation techniques",
-      "Large dataset processing with Dask/Polars",
+      "Large dataset processing with Dask/Polars/Vaex",
+      "Database migration with Alembic and SQLAlchemy",
+      "Cross-database migration patterns",
+      "Zero-downtime migration strategies",
       "Database design patterns",
       "ETL/ELT architectures",
       "AI API integration",
@@ -73,17 +80,22 @@
     ],
     "best_practices": [
       "Always use Python libraries for data transformations",
+      "Prefer Polars over Pandas for large datasets (10-100x faster)",
+      "Implement expand-contract pattern for zero-downtime migrations",
+      "Use Alembic for version-controlled database migrations",
+      "Validate migrations with checksums and row counts",
       "Implement robust error handling for file conversions",
       "Validate data types and formats before processing",
-      "Use chunking for large file operations",
+      "Use chunking and streaming for large file operations",
       "Apply appropriate encoding when reading files",
       "Preserve formatting when converting to Excel",
       "Design efficient schemas with proper indexing",
       "Implement idempotent ETL operations",
+      "Use batch processing for large-scale migrations",
       "Configure AI APIs with monitoring",
       "Validate data at pipeline boundaries",
       "Document architecture decisions",
-      "Test with representative data"
+      "Test migrations with rollback procedures"
     ],
     "constraints": [],
     "examples": []
@@ -131,12 +143,14 @@
     }
   },
   "memory_routing": {
-    "description": "Stores data pipeline patterns, schema designs, and performance tuning techniques",
+    "description": "Stores data pipeline patterns, database migration strategies, schema designs, and performance tuning techniques",
     "categories": [
       "Data pipeline patterns and ETL strategies",
-      "Schema designs and migrations",
-      "Performance tuning techniques",
-      "Data quality requirements"
+      "Database migration patterns and zero-downtime strategies",
+      "Schema designs and version control with Alembic",
+      "Cross-database migration and type mapping",
+      "Performance tuning techniques with Polars/Dask",
+      "Data quality requirements and validation"
     ],
     "keywords": [
       "data",
@@ -146,6 +160,9 @@
       "etl",
       "schema",
       "migration",
+      "alembic",
+      "sqlalchemy",
+      "polars",
       "streaming",
       "batch",
       "warehouse",
@@ -153,18 +170,23 @@
       "analytics",
       "pandas",
       "spark",
-      "kafka"
+      "kafka",
+      "postgres",
+      "mysql",
+      "zero-downtime",
+      "expand-contract"
     ]
   },
   "dependencies": {
     "python": [
       "pandas>=2.1.0",
+      "polars>=0.19.0",
       "openpyxl>=3.1.0",
       "xlsxwriter>=3.1.0",
       "numpy>=1.24.0",
       "pyarrow>=14.0.0",
       "dask>=2023.12.0",
-      "polars>=0.19.0",
+      "vaex>=4.17.0",
       "xlrd>=2.0.0",
       "xlwt>=1.3.0",
       "csvkit>=1.3.0",
@@ -172,7 +194,10 @@
       "python-dateutil>=2.8.0",
       "lxml>=4.9.0",
       "sqlalchemy>=2.0.0",
+      "alembic>=1.13.0",
       "psycopg2-binary>=2.9.0",
+      "pymysql>=1.1.0",
+      "mysqlclient>=2.2.0",
       "pymongo>=4.5.0",
       "redis>=5.0.0",
       "requests>=2.31.0",

claude_mpm/agents/templates/engineer.json

@@ -3,9 +3,14 @@
   "description": "Clean architecture specialist with code reduction focus and dependency injection",
   "schema_version": "1.3.0",
   "agent_id": "engineer",
-  "agent_version": "3.8.1",
-  "template_version": "2.2.0",
+  "agent_version": "3.9.0",
+  "template_version": "2.3.0",
   "template_changelog": [
+    {
+      "version": "2.3.0",
+      "date": "2025-09-25",
+      "description": "Added mcp-vector-search integration for finding existing solutions before implementing new code"
+    },
     {
       "version": "2.2.0",
       "date": "2025-08-25",
@@ -73,9 +78,12 @@
       "Code reduction techniques",
       "Dependency injection patterns",
       "Modularization strategies",
-      "Refactoring for legacy code"
+      "Refactoring for legacy code",
+      "Semantic code search for pattern discovery"
     ],
     "best_practices": [
+      "Use mcp__mcp-vector-search__search_code FIRST to find existing solutions",
+      "Use mcp__mcp-vector-search__search_similar to find reusable patterns",
       "Search for code to DELETE first",
       "Apply dependency injection as default",
       "Enforce 800-line file limit",

claude_mpm/cli/commands/agent_manager.py

@@ -712,7 +712,7 @@ class AgentManagerCommand(AgentCommand):
         """Load agent configuration."""
         try:
             return self.builder_service._load_template(agent_id)
-        except:
+        except Exception:
             return None
 
     def _check_conflicts(self, agent_id: str) -> Optional[str]:
@@ -1074,7 +1074,7 @@ class AgentManagerCommand(AgentCommand):
         except Exception as e:
             return CommandResult.error_result(f"Interactive edit failed: {e}")
 
-    def _test_local_agent(self, args) -> CommandResult:
+    def _test_local_agent(self, args) -> CommandResult:  # noqa: PLR0911
         """Test a local agent with sample task."""
         try:
             from ...services.agents.local_template_manager import (
@@ -1144,7 +1144,7 @@ class AgentManagerCommand(AgentCommand):
         except Exception as e:
             return CommandResult.error_result(f"Local agent test failed: {e}")
 
-    def _delete_local_agents(self, args) -> CommandResult:
+    def _delete_local_agents(self, args) -> CommandResult:  # noqa: PLR0911
         """Delete local agent templates with comprehensive options."""
         try:
             from ...services.agents.local_template_manager import (

claude_mpm/cli/commands/agents.py

@@ -938,7 +938,7 @@ class AgentsCommand(AgentCommand):
             self.logger.error(f"Error creating local agent: {e}", exc_info=True)
             return CommandResult.error_result(f"Error creating local agent: {e}")
 
-    def _edit_local_agent(self, args) -> CommandResult:
+    def _edit_local_agent(self, args) -> CommandResult:  # noqa: PLR0911
         """Edit a local agent template."""
         try:
             agent_id = getattr(args, "agent_id", None)
@@ -993,7 +993,7 @@ class AgentsCommand(AgentCommand):
             self.logger.error(f"Error editing local agent: {e}", exc_info=True)
             return CommandResult.error_result(f"Error editing local agent: {e}")
 
-    def _delete_local_agent(self, args) -> CommandResult:
+    def _delete_local_agent(self, args) -> CommandResult:  # noqa: PLR0911
         """Delete local agent templates."""
         try:
             agent_ids = getattr(args, "agent_ids", [])

claude_mpm/cli/commands/aggregate.py

@@ -116,7 +116,7 @@ def aggregate_command(args):
     return result.exit_code
 
 
-def aggregate_command_legacy(args):
+def aggregate_command_legacy(args):  # noqa: PLR0911
     """Legacy aggregate command dispatcher.
 
     WHY: This contains the original aggregate_command logic, preserved during migration

claude_mpm/cli/commands/config.py

@@ -58,7 +58,7 @@ class ConfigCommand(BaseCommand):
             f"Unknown config command: {args.config_command}"
         )
 
-    def _validate_config(self, args) -> CommandResult:
+    def _validate_config(self, args) -> CommandResult:  # noqa: PLR0911
         """Validate configuration file."""
         config_file = getattr(args, "config_file", None) or Path(
             ".claude-mpm/configuration.yaml"
@@ -168,7 +168,7 @@ class ConfigCommand(BaseCommand):
             console.print(f"[red]Failed to validate configuration: {e}[/red]")
             return CommandResult.error_result(f"Failed to validate configuration: {e}")
 
-    def _view_config(self, args) -> CommandResult:
+    def _view_config(self, args) -> CommandResult:  # noqa: PLR0911
         """View current configuration."""
         try:
             # Load configuration

claude_mpm/cli/commands/configure.py

@@ -185,7 +185,7 @@ class ConfigureCommand(BaseCommand):
 
         return None
 
-    def run(self, args) -> CommandResult:
+    def run(self, args) -> CommandResult:  # noqa: PLR0911
         """Execute the configure command."""
         # Set configuration scope
         self.current_scope = getattr(args, "scope", "project")
@@ -441,7 +441,7 @@ class ConfigureCommand(BaseCommand):
                         tools_display = f"Model: {model}"
                     else:
                         tools_display = "Default"
-                except:
+                except Exception:
                     tools_display = "Default"
 
             # Truncate description for table display
@@ -858,7 +858,7 @@ class ConfigureCommand(BaseCommand):
 [bold]Tags:[/bold] {', '.join(tags) if tags else 'None'}
 [bold]Tools:[/bold] {', '.join(tools[:5]) if tools else 'None'}{'...' if len(tools) > 5 else ''}
 """
-                    except:
+                    except Exception:
                         pass
 
                 # Create detail panel
@@ -1048,7 +1048,7 @@ class ConfigureCommand(BaseCommand):
                 )
                 if result.returncode == 0:
                     claude_version = result.stdout.strip()
-        except:
+        except Exception:
             pass
 
         # Create version panel
@@ -1191,7 +1191,7 @@ Directory: {self.project_dir}
             )
             if result.returncode == 0:
                 data["claude_version"] = result.stdout.strip()
-        except:
+        except Exception:
             data["claude_version"] = "Unknown"
 
         # Print formatted output

claude_mpm/cli/commands/configure_tui.py

@@ -485,11 +485,11 @@ class AgentManagementScreen(Container):
             try:
                 rel_path = agent.template_path.relative_to(Path.home())
                 path_str = f"~/{rel_path}"
-            except:
+            except Exception:
                 try:
                     rel_path = agent.template_path.relative_to(self.project_dir)
                     path_str = f"./{rel_path}"
-                except:
+                except Exception:
                     path_str = str(agent.template_path)
 
             self.log(f"Adding row: {agent.name}, {status}, {agent.version}, {path_str}")
@@ -794,11 +794,11 @@ model: {metadata.get('model', 'claude-3-5-sonnet-20241022')}
             try:
                 rel_path = agent.template_path.relative_to(Path.home())
                 path_str = f"~/{rel_path}"
-            except:
+            except Exception:
                 try:
                     rel_path = agent.template_path.relative_to(self.project_dir)
                     path_str = f"./{rel_path}"
-                except:
+                except Exception:
                     path_str = str(agent.template_path)
 
             table.add_row(agent.name, status, agent.version, path_str, key=agent.name)
@@ -1208,7 +1208,7 @@ class SettingsScreen(Container):
             )
             if result.returncode == 0:
                 claude_version = result.stdout.strip()
-        except:
+        except Exception:
             pass
 
         version_container = self.query_one("#version-info", Container)
@@ -1258,7 +1258,7 @@ Python: {sys.version.split()[0]}"""
                         current_screen.load_templates()
                     elif hasattr(current_screen, "load_behavior_files"):
                         current_screen.load_behavior_files()
-            except:
+            except Exception:
                 pass
 
         self.notify(f"Switched to {self.current_scope} scope")
@@ -1907,7 +1907,7 @@ def can_use_tui() -> bool:
         cols, rows = shutil.get_terminal_size()
         if cols < 80 or rows < 24:
             return False
-    except:
+    except Exception:
         return False
 
     return True

claude_mpm/cli/commands/dashboard.py

@@ -65,7 +65,7 @@ class DashboardCommand(BaseCommand):
             self.logger.error(f"Error executing dashboard command: {e}", exc_info=True)
             return CommandResult.error_result(f"Error executing dashboard command: {e}")
 
-    def _start_dashboard(self, args) -> CommandResult:
+    def _start_dashboard(self, args) -> CommandResult:  # noqa: PLR0911
         """Start the dashboard server."""
         port = getattr(args, "port", 8765)
         host = getattr(args, "host", "localhost")

claude_mpm/cli/commands/debug.py

@@ -21,7 +21,7 @@ from typing import Any, Dict
 from ...core.logger import get_logger
 
 
-def manage_debug(args):
+def manage_debug(args):  # noqa: PLR0911
     """
     Main entry point for debug commands.
 
@@ -216,7 +216,7 @@ def debug_services(args, logger):
                 container = DIContainer.get_instance()
             else:
                 container = DIContainer()
-        except:
+        except Exception:
             # Create a new container if none exists
             container = DIContainer()
             logger.warning("No active container found, created new instance")
@@ -408,7 +408,7 @@ def debug_agents(args, logger):
                             agent_name = agent_file.stem
                             size = agent_file.stat().st_size
                             modified = datetime.fromtimestamp(
-                                agent_file.stat().st_mtime, timezone.utc
+                                agent_file.stat().st_mtime, tz=timezone.utc
                             )
                             print(f"   • {agent_name}")
                             print(f"     Size: {size:,} bytes")
@@ -684,7 +684,7 @@ def debug_hooks(args, logger):
                         hook.execute(test_context)
                         elapsed = time.time() - start
                         times.append(elapsed)
-                    except:
+                    except Exception:
                         pass
 
                 if times:
@@ -1075,7 +1075,7 @@ def _profile_memory_operations():
         try:
             memory.add_memory(f"category_{i}", {"data": f"test_{i}"})
             memory.get_memories()
-        except:
+        except Exception:
             pass
 
 

claude_mpm/cli/commands/mcp.py

@@ -186,7 +186,7 @@ def _show_status(
             ToolRegistry()
             # Don't initialize fully, just check
             print("\n🔧 Tools: Check with 'claude-mpm mcp tools'")
-        except:
+        except Exception:
             print("\n🔧 Tools: Registry not available")
 
     print("\n💡 Available Commands:")

claude_mpm/cli/commands/mcp_command_router.py

@@ -16,7 +16,7 @@ class MCPCommandRouter:
         """Initialize the command router."""
         self.logger = logger
 
-    def route_command(self, args) -> int:
+    def route_command(self, args) -> int:  # noqa: PLR0911
         """Route command to appropriate handler."""
         if args.mcp_command == MCPCommands.START.value:
             return asyncio.run(self._start_server(args))

claude_mpm/cli/commands/mcp_config.py

@@ -6,7 +6,6 @@ Command for managing MCP service configurations with pipx preference.
 """
 
 import json
-import sys
 from pathlib import Path
 
 from ...services.mcp_config_manager import MCPConfigManager
@@ -29,16 +28,14 @@ class MCPConfigCommand(BaseCommand):
 
             if command == "detect":
                 return self._detect_services(manager)
-            elif command == "update":
+            if command == "update":
                 return self._update_config(manager, args)
-            elif command == "validate":
+            if command == "validate":
                 return self._validate_config(manager)
-            elif command == "install":
+            if command == "install":
                 return self._install_services(manager)
-            else:
-                return self._show_status(manager)
-        else:
             return self._show_status(manager)
+        return self._show_status(manager)
 
     def _detect_services(self, manager: MCPConfigManager) -> CommandResult:
         """Detect available MCP services."""
@@ -65,7 +62,7 @@ class MCPConfigCommand(BaseCommand):
             # Show the updated configuration
             config_path = Path.cwd() / ".mcp.json"
             if config_path.exists():
-                with open(config_path, "r") as f:
+                with open(config_path) as f:
                     config = json.load(f)
                 return CommandResult(
                     success=True,
@@ -123,7 +120,7 @@ class MCPConfigCommand(BaseCommand):
         current_config = {}
         if config_path.exists():
             try:
-                with open(config_path, "r") as f:
+                with open(config_path) as f:
                     current_config = json.load(f)
             except Exception:
                 pass
@@ -154,4 +151,4 @@ def manage_mcp_config(args):
     """
     command = MCPConfigCommand()
     result = command.execute(args)
-    return result.exit_code
+    return result.exit_code