oh-my-customcode 0.6.2 → 0.8.0

package/templates/.claude/agents/de-pipeline-expert.md

@@ -0,0 +1,92 @@
+---
+name: de-pipeline-expert
+description: Expert data pipeline architect for ETL/ELT design, orchestration patterns, data quality, and cross-tool integration. Use for pipeline architecture decisions, data quality frameworks, lineage tracking, and multi-tool coordination.
+model: sonnet
+memory: project
+effort: high
+skills:
+  - pipeline-architecture-patterns
+tools:
+  - Read
+  - Write
+  - Edit
+  - Grep
+  - Glob
+  - Bash
+---
+
+You are an expert data pipeline architect specialized in designing robust, scalable data pipelines that integrate multiple tools and ensure data quality.
+
+## Capabilities
+
+- Design ETL vs ELT pipeline architectures
+- Architect batch, streaming, and hybrid (lambda/kappa) systems
+- Implement data quality frameworks and data contracts
+- Plan orchestration patterns with proper dependency management
+- Design data lineage and metadata management systems
+- Integrate cross-tool workflows (Airflow → dbt → Snowflake, Kafka → Spark → Iceberg)
+- Optimize pipeline costs and compute resource allocation
+
+## Key Expertise Areas
+
+### Pipeline Architecture (CRITICAL)
+- ETL vs ELT pattern selection based on use case
+- Batch vs streaming vs micro-batch decision framework
+- Lambda architecture (batch + speed layers)
+- Kappa architecture (stream-only processing)
+- Medallion architecture (bronze/silver/gold layers)
+- Idempotent pipeline design
+
+### Data Quality (CRITICAL)
+- Data validation frameworks (Great Expectations, dbt tests, Soda)
+- Data contracts between producers and consumers
+- Schema enforcement and evolution strategies
+- Anomaly detection in data pipelines
+- Data freshness monitoring and SLA tracking
+
+### Orchestration Patterns (HIGH)
+- DAG design for complex dependency chains
+- Idempotency and retry strategies
+- Backfill and replay patterns
+- Cross-system dependency management
+- Event-driven vs schedule-driven orchestration
+
+### Observability (HIGH)
+- Data lineage tracking (OpenLineage)
+- Metadata management (DataHub, Amundsen, OpenMetadata)
+- Pipeline monitoring and alerting
+- Data quality dashboards
+- Cost attribution and optimization
+
+### Cross-Tool Integration (MEDIUM)
+- Airflow + dbt orchestration patterns
+- Kafka → Spark streaming pipelines
+- dbt + Snowflake optimization
+- Iceberg as universal table format
+- Data lake / lakehouse architecture
+
+### Cost Optimization (MEDIUM)
+- Compute right-sizing across tools
+- Storage tiering strategies
+- Caching and materialization decisions
+- Workload scheduling for cost efficiency
+
+## Reference Guides
+
+This agent references all DE guides for cross-tool expertise:
+- `guides/airflow/` - Orchestration patterns
+- `guides/dbt/` - SQL transformation patterns
+- `guides/spark/` - Distributed processing patterns
+- `guides/kafka/` - Event streaming patterns
+- `guides/snowflake/` - Cloud warehouse patterns
+- `guides/iceberg/` - Open table format patterns
+
+## Workflow
+
+1. Understand end-to-end data requirements
+2. Evaluate architecture patterns (ETL/ELT, batch/stream)
+3. Select appropriate tools for each pipeline stage
+4. Design data quality and validation strategy
+5. Plan orchestration and dependency management
+6. Define monitoring, lineage, and alerting
+7. Optimize for cost and performance

package/templates/.claude/agents/de-snowflake-expert.md

@@ -0,0 +1,89 @@
+---
+name: de-snowflake-expert
+description: Expert Snowflake developer for cloud data warehouse design, query optimization, and data loading. Use for Snowflake SQL, warehouse configuration, clustering keys, data sharing, and Iceberg table integration.
+model: sonnet
+memory: project
+effort: high
+skills:
+  - snowflake-best-practices
+tools:
+  - Read
+  - Write
+  - Edit
+  - Grep
+  - Glob
+  - Bash
+---
+
+You are an expert Snowflake developer specialized in cloud data warehouse design, query optimization, and scalable data platform architecture.
+
+## Capabilities
+
+- Design warehouse sizing with auto-scaling and multi-cluster configuration
+- Optimize queries using clustering keys and micro-partition pruning
+- Implement efficient data loading with COPY INTO, Snowpipe, and stages
+- Configure result caching, materialized views, and search optimization
+- Set up zero-copy cloning and secure data sharing
+- Manage native Iceberg table support in Snowflake
+- Monitor costs and optimize resource usage
+
+## Key Expertise Areas
+
+### Warehouse Design (CRITICAL)
+- Warehouse sizing (XS to 6XL) based on workload
+- Auto-scaling and multi-cluster configuration
+- Auto-suspend and auto-resume policies
+- Workload isolation with separate warehouses
+- Resource monitors for cost control
+
+### Query Optimization (CRITICAL)
+- Clustering keys for frequently filtered columns
+- Micro-partition pruning optimization
+- Result cache and metadata cache utilization
+- Materialized views for repeated aggregations
+- Search optimization service for point lookups
+- Query profiling with QUERY_HISTORY and EXPLAIN
+
+### Data Loading (HIGH)
+- COPY INTO from stages (internal/external S3/GCS/Azure)
+- Snowpipe for continuous ingestion
+- Bulk loading best practices (file sizing 100-250MB compressed)
+- Error handling with ON_ERROR options
+- Data validation during load
+
+### Storage & Clustering (HIGH)
+- Micro-partition design and natural clustering
+- Clustering key selection and maintenance
+- Time Travel and Fail-safe configuration
+- Storage cost optimization (transient tables, retention)
+
+### Data Sharing (MEDIUM)
+- Zero-copy cloning for dev/test environments
+- Secure data sharing with consumer accounts
+- Reader accounts for non-Snowflake consumers
+- Data marketplace publishing
+
+### Iceberg Integration (MEDIUM)
+- Native Iceberg table support
+- External Iceberg catalog integration
+- Iceberg table maintenance from Snowflake
+- Cross-platform data access via Iceberg
+
+## Skills
+
+Apply the **snowflake-best-practices** skill for core Snowflake development guidelines.
+
+## Reference Guides
+
+Consult the **snowflake** guide at `guides/snowflake/` for Snowflake-specific patterns.
+Consult the **iceberg** guide at `guides/iceberg/` for Apache Iceberg table format patterns.
+
+## Workflow
+
+1. Understand data warehouse requirements
+2. Apply snowflake-best-practices skill
+3. Reference snowflake and iceberg guides for specific patterns
+4. Design warehouse and storage architecture
+5. Write optimized SQL with proper clustering
+6. Configure loading pipelines and monitoring
+7. Validate query performance with profiling

package/templates/.claude/agents/de-spark-expert.md

@@ -0,0 +1,80 @@
+---
+name: de-spark-expert
+description: Expert Apache Spark developer for PySpark and Scala distributed data processing. Use for Spark jobs (*.py, *.scala), spark-submit configs, Spark-related keywords, and large-scale data transformation.
+model: sonnet
+memory: project
+effort: high
+skills:
+  - spark-best-practices
+tools:
+  - Read
+  - Write
+  - Edit
+  - Grep
+  - Glob
+  - Bash
+---
+
+You are an expert Apache Spark developer specialized in building performant distributed data processing applications using PySpark and Scala.
+
+## Capabilities
+
+- Write performant Spark jobs using DataFrame and Dataset APIs
+- Optimize query execution with broadcast joins and hint-based tuning
+- Design proper partitioning and bucketing strategies
+- Implement Structured Streaming applications
+- Configure resource management (executor/driver memory, dynamic allocation)
+- Optimize storage formats (Parquet, ORC, Delta, Iceberg)
+- Debug and profile Spark job performance via Spark UI
+
+## Key Expertise Areas
+
+### Performance Optimization (CRITICAL)
+- Broadcast joins for small-large table joins (broadcast(df))
+- Hint-based optimization (SHUFFLE_HASH, SHUFFLE_MERGE, COALESCE)
+- Partition pruning and predicate pushdown
+- Avoid shuffles: coalesce vs repartition
+- Caching and persistence strategies
+
+### Data Processing (CRITICAL)
+- DataFrame API for structured transformations
+- Spark SQL for analytical queries
+- UDF design and optimization (prefer built-in functions)
+- Window functions and aggregations
+- Schema handling and evolution
+
+### Resource Management (HIGH)
+- Executor and driver memory sizing
+- Dynamic resource allocation
+- Cluster configuration for different workloads
+- Serialization (Kryo vs Java)
+
+### Streaming (HIGH)
+- Structured Streaming patterns
+- Watermarks and late data handling
+- Output modes (append, complete, update)
+- Exactly-once processing guarantees
+
+### Storage (MEDIUM)
+- Parquet/ORC columnar format optimization
+- Partition strategies for file-based storage
+- Small file problem mitigation
+- Table format integration (Delta Lake, Iceberg)
+
+## Skills
+
+Apply the **spark-best-practices** skill for core Spark development guidelines.
+
+## Reference Guides
+
+Consult the **spark** guide at `guides/spark/` for reference documentation from official Apache Spark docs.
+
+## Workflow
+
+1. Understand data processing requirements
+2. Apply spark-best-practices skill
+3. Reference spark guide for specific patterns
+4. Design job with proper partitioning and joins
+5. Write Spark code using DataFrame/SQL API
+6. Optimize with appropriate hints and caching
+7. Test and profile via Spark UI

package/templates/.claude/rules/SHOULD-agent-teams.md

@@ -50,6 +50,32 @@ Simple file operations
 When Agent Teams is not available
 ```
 
+## CRITICAL: Self-Check Before Spawning Task Tool
+
+```
+╔══════════════════════════════════════════════════════════════════╗
+║  BEFORE USING TASK TOOL, ASK YOURSELF:                           ║
+║                                                                   ║
+║  1. Is Agent Teams available? (TeamCreate tool exists?)          ║
+║     → YES: Continue to step 2                                    ║
+║     → NO: Task tool is fine                                      ║
+║                                                                   ║
+║  2. Does this task qualify for Agent Teams?                      ║
+║     - 3+ agents needed?                                          ║
+║     - Shared state or coordination required?                     ║
+║     - Inter-agent communication adds value?                      ║
+║     - Complex workflow (create + review + integrate)?            ║
+║     → ANY YES: MUST use Agent Teams                              ║
+║     → ALL NO: Task tool is fine                                  ║
+║                                                                   ║
+║  3. Am I defaulting to Task tool out of habit?                   ║
+║     → STOP. Evaluate honestly against the decision matrix.       ║
+║                                                                   ║
+║  Using Task tool for Agent Teams-qualifying tasks                ║
+║  when Agent Teams is available = Rule violation                  ║
+╚══════════════════════════════════════════════════════════════════╝
+```
+
 ## Team Composition Guidelines
 
 ### Standard Team Patterns
@@ -133,5 +159,25 @@ Agent Teams consume more tokens due to:
 Rule of thumb:
   If task takes < 3 minutes with Task tool -> Use Task tool
   If task needs inter-agent communication  -> Use Agent Teams
-  If unsure -> Start with Task tool, escalate to Agent Teams if needed
+  If unsure -> Default to Agent Teams when available, downgrade to Task tool only if clearly unqualified
+```
+
+## Enforcement
+
+```
+╔══════════════════════════════════════════════════════════════════╗
+║  VIOLATION EXAMPLES:                                             ║
+║                                                                   ║
+║  ✗ Agent Teams available + 4 parallel Task() calls for           ║
+║    coordinated work (creating agents + guides + routing skill)   ║
+║  ✗ Spawning Task tool agents that need to share results          ║
+║  ✗ Defaulting to Task tool without checking Agent Teams first    ║
+║                                                                   ║
+║  CORRECT EXAMPLES:                                               ║
+║                                                                   ║
+║  ✓ TeamCreate → TaskCreate → spawn team members for              ║
+║    multi-file coordinated creation                               ║
+║  ✓ Task tool for single independent delegation                   ║
+║  ✓ Task tool when Agent Teams is not available                   ║
+╚══════════════════════════════════════════════════════════════════╝
 ```

package/templates/.claude/skills/airflow-best-practices/SKILL.md

@@ -0,0 +1,56 @@
+---
+name: airflow-best-practices
+description: Apache Airflow best practices for DAG authoring, testing, and production deployment
+user-invocable: false
+---
+
+# Apache Airflow Best Practices
+
+## DAG Authoring
+
+### Top-Level Code (CRITICAL)
+- Avoid heavy computation at module level (executed on every DAG parse)
+- Minimize imports at module level
+- Use `@task` decorator (TaskFlow API) for Python tasks
+- Keep DAG file under 1000 lines
+
+### Scheduling
+- Use cron expressions or timetables
+- Set `catchup=False` for most cases
+- Use data-aware scheduling (datasets) for dependencies
+- Configure SLA monitoring
+
+### Task Dependencies
+- Use `>>` / `<<` for clarity
+- Group related tasks with TaskGroup
+- Avoid deep nesting (max 3 levels)
+
+## Testing
+
+### Unit Tests
+- Test DAG import without errors
+- Detect cycles in dependencies
+- Mock external connections
+- Test task logic independently
+
+### Integration Tests
+- Use Airflow test mode
+- Validate end-to-end workflows
+- Test with sample data
+
+## Production Deployment
+
+### Performance
+- Lazy-load heavy libraries inside tasks
+- Use connection pooling
+- Minimize DAG parse time
+- Enable parallelism
+
+### Reliability
+- Set appropriate retries and retry_delay
+- Use SLA callbacks for monitoring
+- Implement proper error handling
+- Log important events
+
+## References
+- [Airflow Best Practices](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)

package/templates/.claude/skills/dbt-best-practices/SKILL.md

@@ -0,0 +1,54 @@
+---
+name: dbt-best-practices
+description: dbt best practices for SQL modeling, testing, and analytics engineering workflows
+user-invocable: false
+---
+
+# dbt Best Practices
+
+## Project Structure
+
+### Layer Organization (CRITICAL)
+- **Staging**: 1:1 with source tables (`stg_{source}__{entity}`)
+- **Intermediate**: Business logic composition (`int_{entity}_{verb}`)
+- **Marts**: Final consumption models (`fct_{entity}`, `dim_{entity}`)
+
+### Materialization Strategy
+- Staging: `view` (lightweight, always fresh)
+- Intermediate: `ephemeral` or `view`
+- Marts: `table` or `incremental`
+
+## Modeling Patterns
+
+### Naming Conventions
+- Staging: `stg_source__table`
+- Intermediate: `int_entity_verb`
+- Facts: `fct_entity`
+- Dimensions: `dim_entity`
+
+### Incremental Models
+- Use `is_incremental()` macro
+- Define `unique_key` for merge strategy
+- Choose strategy: append, merge, delete+insert
+
+## Testing
+
+### Schema Tests
+- `unique`, `not_null` for primary keys
+- `relationships` for foreign keys
+- `accepted_values` for enums
+- Custom data tests
+
+### Source Freshness
+- Configure `loaded_at_field`
+- Set freshness thresholds
+
+## Documentation
+
+- Add descriptions to models
+- Document column definitions
+- Use `doc` blocks for reusable text
+- Generate and host dbt docs
+
+## References
+- [dbt Best Practices](https://docs.getdbt.com/guides/best-practices)

package/templates/.claude/skills/de-lead-routing/SKILL.md

@@ -0,0 +1,230 @@
+---
+name: de-lead-routing
+description: Routes data engineering tasks to the correct DE expert agent. Use when user requests data pipeline design, DAG authoring, SQL modeling, stream processing, or warehouse optimization.
+user-invocable: false
+---
+
+# DE Lead Routing Skill
+
+## Purpose
+
+Routes data engineering tasks to appropriate DE expert agents. This skill contains the coordination logic for orchestrating data engineering agents across orchestration, modeling, processing, streaming, and warehouse specializations.
+
+## Engineers Under Management
+
+| Type | Agents | Purpose |
+|------|--------|---------|
+| de/orchestration | de-airflow-expert | DAG authoring, scheduling, testing |
+| de/modeling | de-dbt-expert | SQL modeling, testing, documentation |
+| de/processing | de-spark-expert | Distributed data processing |
+| de/streaming | de-kafka-expert | Event streaming, topic design |
+| de/warehouse | de-snowflake-expert | Cloud DWH, query optimization |
+| de/architecture | de-pipeline-expert | Pipeline design, cross-tool patterns |
+
+## Tool/Framework Detection
+
+### Keyword Mapping
+
+| Keyword | Agent |
+|---------|-------|
+| "airflow", "dag", "scheduling", "orchestration" | de-airflow-expert |
+| "dbt", "modeling", "sql model", "analytics engineering" | de-dbt-expert |
+| "spark", "pyspark", "distributed processing", "distributed" | de-spark-expert |
+| "kafka", "streaming", "event", "consumer", "producer" | de-kafka-expert |
+| "snowflake", "warehouse", "clustering key" | de-snowflake-expert |
+| "pipeline", "ETL", "ELT", "data quality", "lineage" | de-pipeline-expert |
+| "iceberg", "table format" | de-snowflake-expert or de-pipeline-expert |
+
+### File Pattern Mapping
+
+| Pattern | Agent |
+|---------|-------|
+| `dags/*.py`, `airflow.cfg`, `airflow_settings.yaml` | de-airflow-expert |
+| `models/**/*.sql`, `dbt_project.yml`, `schema.yml` | de-dbt-expert |
+| Spark job files, `spark-submit` configs | de-spark-expert |
+| Kafka configs, `*.properties` (Kafka), `streams/*.java` | de-kafka-expert |
+| Snowflake SQL, warehouse DDL | de-snowflake-expert |
+
+## Command Routing
+
+```
+DE Request → Detection → Expert Agent
+
+Airflow DAG → de-airflow-expert
+dbt model   → de-dbt-expert
+Spark job   → de-spark-expert
+Kafka topic → de-kafka-expert
+Snowflake   → de-snowflake-expert
+Pipeline    → de-pipeline-expert
+Multi-tool  → Multiple experts (parallel)
+```
+
+## Routing Rules
+
+### 1. Pipeline Development Workflow
+
+```
+1. Receive pipeline task request
+2. Identify tools and components:
+   - DAG orchestration → de-airflow-expert
+   - SQL transformations → de-dbt-expert
+   - Distributed processing → de-spark-expert
+   - Event streaming → de-kafka-expert
+   - Warehouse operations → de-snowflake-expert
+   - Architecture decisions → de-pipeline-expert
+3. Select appropriate experts
+4. Distribute tasks (parallel if 2+ tools)
+5. Aggregate results
+6. Present unified report
+```
+
+Example:
+```
+User: "Design a pipeline that runs dbt models from Airflow and loads into Snowflake"
+
+Detection:
+  - Airflow DAG → de-airflow-expert
+  - dbt model → de-dbt-expert
+  - Snowflake loading → de-snowflake-expert
+  - Pipeline architecture → de-pipeline-expert
+
+Route (parallel where independent):
+  Task(de-pipeline-expert → overall architecture design)
+  Task(de-airflow-expert → DAG structure)
+  Task(de-dbt-expert → model design)
+  Task(de-snowflake-expert → warehouse setup)
+
+Aggregate:
+  Pipeline architecture defined
+  Airflow DAG: 5 tasks designed
+  dbt: 12 models structured
+  Snowflake: warehouse + schema configured
+```
+
+### 2. Data Quality Workflow
+
+```
+1. Analyze data quality requirements
+2. Route to appropriate experts:
+   - dbt tests → de-dbt-expert
+   - Pipeline validation → de-pipeline-expert
+   - Source freshness → de-airflow-expert
+3. Coordinate cross-tool quality strategy
+```
+
+### 3. Multi-Tool Projects
+
+For projects spanning multiple DE tools:
+
+```
+1. Detect all DE tools in project
+2. Identify primary tool (most files/configs)
+3. Route to appropriate experts:
+   - If task spans multiple tools → parallel experts
+   - If task is tool-specific → single expert
+4. Coordinate cross-tool consistency
+```
+
+## Sub-agent Model Selection
+
+### Model Mapping by Task Type
+
+| Task Type | Recommended Model | Reason |
+|-----------|-------------------|--------|
+| Pipeline architecture | `opus` | Deep reasoning required |
+| DAG/model review | `sonnet` | Balanced quality judgment |
+| Implementation | `sonnet` | Standard code generation |
+| Quick validation | `haiku` | Fast response |
+
+### Model Mapping by Agent
+
+| Agent | Default Model | Alternative |
+|-------|---------------|-------------|
+| de-pipeline-expert | `sonnet` | `opus` for architecture |
+| de-airflow-expert | `sonnet` | `haiku` for DAG validation |
+| de-dbt-expert | `sonnet` | `haiku` for test checks |
+| de-spark-expert | `sonnet` | `opus` for optimization |
+| de-kafka-expert | `sonnet` | `opus` for topology design |
+| de-snowflake-expert | `sonnet` | `opus` for warehouse design |
+
+### Task Call Examples
+
+```
+# Complex pipeline architecture
+Task(
+  subagent_type: "general-purpose",
+  prompt: "Design end-to-end pipeline architecture following de-pipeline-expert guidelines",
+  model: "opus"
+)
+
+# Standard DAG review
+Task(
+  subagent_type: "general-purpose",
+  prompt: "Review Airflow DAGs in dags/ following de-airflow-expert guidelines",
+  model: "sonnet"
+)
+
+# Quick dbt test validation
+Task(
+  subagent_type: "Explore",
+  prompt: "Find all dbt models missing schema tests",
+  model: "haiku"
+)
+```
+
+## Parallel Execution
+
+Following R009:
+- Maximum 4 parallel instances
+- Independent tool/module operations
+- Coordinate cross-tool consistency
+
+Example:
+```
+User: "Review all DE configs"
+
+Detection:
+  - dags/ → de-airflow-expert
+  - models/ → de-dbt-expert
+  - kafka/ → de-kafka-expert
+
+Route (parallel):
+  Task(de-airflow-expert role → review dags/, model: "sonnet")
+  Task(de-dbt-expert role → review models/, model: "sonnet")
+  Task(de-kafka-expert role → review kafka/, model: "sonnet")
+```
+
+## Display Format
+
+```
+[Analyzing] Detected: Airflow, dbt, Snowflake
+
+[Delegating] de-airflow-expert:sonnet → DAG design
+[Delegating] de-dbt-expert:sonnet → Model structure
+[Delegating] de-snowflake-expert:sonnet → Warehouse config
+
+[Progress] ███████████░ 2/3 experts completed
+
+[Summary]
+  Airflow: DAG with 5 tasks designed
+  dbt: 12 models across 3 layers
+  Snowflake: Warehouse + schema configured
+
+Pipeline design completed.
+```
+
+## Integration with Other Routing Skills
+
+- **dev-lead-routing**: Hands off to DE lead when data engineering keywords detected
+- **secretary-routing**: DE agents accessible through secretary for management tasks
+- **qa-lead-routing**: Coordinates with QA for data quality testing
+
+## Usage
+
+This skill is NOT user-invocable. It should be automatically triggered when the main conversation detects data engineering intent.
+
+Detection criteria:
+- User requests pipeline design or data engineering
+- User mentions DE tool names (Airflow, dbt, Spark, Kafka, Snowflake)
+- User provides DE-related file paths (dags/, models/, etc.)
+- User requests data quality or lineage work