oh-my-customcode 0.7.0 → 0.9.0

package/templates/.claude/agents/de-spark-expert.md

@@ -0,0 +1,80 @@
+---
+name: de-spark-expert
+description: Expert Apache Spark developer for PySpark and Scala distributed data processing. Use for Spark jobs (*.py, *.scala), spark-submit configs, Spark-related keywords, and large-scale data transformation.
+model: sonnet
+memory: project
+effort: high
+skills:
+  - spark-best-practices
+tools:
+  - Read
+  - Write
+  - Edit
+  - Grep
+  - Glob
+  - Bash
+---
+
+You are an expert Apache Spark developer specialized in building performant distributed data processing applications using PySpark and Scala.
+
+## Capabilities
+
+- Write performant Spark jobs using DataFrame and Dataset APIs
+- Optimize query execution with broadcast joins and hint-based tuning
+- Design proper partitioning and bucketing strategies
+- Implement Structured Streaming applications
+- Configure resource management (executor/driver memory, dynamic allocation)
+- Optimize storage formats (Parquet, ORC, Delta, Iceberg)
+- Debug and profile Spark job performance via Spark UI
+
+## Key Expertise Areas
+
+### Performance Optimization (CRITICAL)
+- Broadcast joins for small-large table joins (broadcast(df))
+- Hint-based optimization (SHUFFLE_HASH, SHUFFLE_MERGE, COALESCE)
+- Partition pruning and predicate pushdown
+- Avoid shuffles: coalesce vs repartition
+- Caching and persistence strategies
+
+### Data Processing (CRITICAL)
+- DataFrame API for structured transformations
+- Spark SQL for analytical queries
+- UDF design and optimization (prefer built-in functions)
+- Window functions and aggregations
+- Schema handling and evolution
+
+### Resource Management (HIGH)
+- Executor and driver memory sizing
+- Dynamic resource allocation
+- Cluster configuration for different workloads
+- Serialization (Kryo vs Java)
+
+### Streaming (HIGH)
+- Structured Streaming patterns
+- Watermarks and late data handling
+- Output modes (append, complete, update)
+- Exactly-once processing guarantees
+
+### Storage (MEDIUM)
+- Parquet/ORC columnar format optimization
+- Partition strategies for file-based storage
+- Small file problem mitigation
+- Table format integration (Delta Lake, Iceberg)
+
+## Skills
+
+Apply the **spark-best-practices** skill for core Spark development guidelines.
+
+## Reference Guides
+
+Consult the **spark** guide at `guides/spark/` for reference documentation from official Apache Spark docs.
+
+## Workflow
+
+1. Understand data processing requirements
+2. Apply spark-best-practices skill
+3. Reference spark guide for specific patterns
+4. Design job with proper partitioning and joins
+5. Write Spark code using DataFrame/SQL API
+6. Optimize with appropriate hints and caching
+7. Test and profile via Spark UI

package/templates/.claude/rules/SHOULD-agent-teams.md

@@ -50,6 +50,32 @@ Simple file operations
 When Agent Teams is not available
 ```
 
+## CRITICAL: Self-Check Before Spawning Task Tool
+
+```
+╔══════════════════════════════════════════════════════════════════╗
+║  BEFORE USING TASK TOOL, ASK YOURSELF:                           ║
+║                                                                   ║
+║  1. Is Agent Teams available? (TeamCreate tool exists?)          ║
+║     → YES: Continue to step 2                                    ║
+║     → NO: Task tool is fine                                      ║
+║                                                                   ║
+║  2. Does this task qualify for Agent Teams?                      ║
+║     - 3+ agents needed?                                          ║
+║     - Shared state or coordination required?                     ║
+║     - Inter-agent communication adds value?                      ║
+║     - Complex workflow (create + review + integrate)?            ║
+║     → ANY YES: MUST use Agent Teams                              ║
+║     → ALL NO: Task tool is fine                                  ║
+║                                                                   ║
+║  3. Am I defaulting to Task tool out of habit?                   ║
+║     → STOP. Evaluate honestly against the decision matrix.       ║
+║                                                                   ║
+║  Using Task tool for Agent Teams-qualifying tasks                ║
+║  when Agent Teams is available = Rule violation                  ║
+╚══════════════════════════════════════════════════════════════════╝
+```
+
 ## Team Composition Guidelines
 
 ### Standard Team Patterns
@@ -133,5 +159,25 @@ Agent Teams consume more tokens due to:
 Rule of thumb:
   If task takes < 3 minutes with Task tool -> Use Task tool
   If task needs inter-agent communication  -> Use Agent Teams
-  If unsure -> Start with Task tool, escalate to Agent Teams if needed
+  If unsure -> Default to Agent Teams when available, downgrade to Task tool only if clearly unqualified
+```
+
+## Enforcement
+
+```
+╔══════════════════════════════════════════════════════════════════╗
+║  VIOLATION EXAMPLES:                                             ║
+║                                                                   ║
+║  ✗ Agent Teams available + 4 parallel Task() calls for           ║
+║    coordinated work (creating agents + guides + routing skill)   ║
+║  ✗ Spawning Task tool agents that need to share results          ║
+║  ✗ Defaulting to Task tool without checking Agent Teams first    ║
+║                                                                   ║
+║  CORRECT EXAMPLES:                                               ║
+║                                                                   ║
+║  ✓ TeamCreate → TaskCreate → spawn team members for              ║
+║    multi-file coordinated creation                               ║
+║  ✓ Task tool for single independent delegation                   ║
+║  ✓ Task tool when Agent Teams is not available                   ║
+╚══════════════════════════════════════════════════════════════════╝
 ```

package/templates/.claude/skills/airflow-best-practices/SKILL.md

@@ -0,0 +1,56 @@
+---
+name: airflow-best-practices
+description: Apache Airflow best practices for DAG authoring, testing, and production deployment
+user-invocable: false
+---
+
+# Apache Airflow Best Practices
+
+## DAG Authoring
+
+### Top-Level Code (CRITICAL)
+- Avoid heavy computation at module level (executed on every DAG parse)
+- Minimize imports at module level
+- Use `@task` decorator (TaskFlow API) for Python tasks
+- Keep DAG file under 1000 lines
+
+### Scheduling
+- Use cron expressions or timetables
+- Set `catchup=False` for most cases
+- Use data-aware scheduling (datasets) for dependencies
+- Configure SLA monitoring
+
+### Task Dependencies
+- Use `>>` / `<<` for clarity
+- Group related tasks with TaskGroup
+- Avoid deep nesting (max 3 levels)
+
+## Testing
+
+### Unit Tests
+- Test DAG import without errors
+- Detect cycles in dependencies
+- Mock external connections
+- Test task logic independently
+
+### Integration Tests
+- Use Airflow test mode
+- Validate end-to-end workflows
+- Test with sample data
+
+## Production Deployment
+
+### Performance
+- Lazy-load heavy libraries inside tasks
+- Use connection pooling
+- Minimize DAG parse time
+- Enable parallelism
+
+### Reliability
+- Set appropriate retries and retry_delay
+- Use SLA callbacks for monitoring
+- Implement proper error handling
+- Log important events
+
+## References
+- [Airflow Best Practices](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)

package/templates/.claude/skills/dbt-best-practices/SKILL.md

@@ -0,0 +1,54 @@
+---
+name: dbt-best-practices
+description: dbt best practices for SQL modeling, testing, and analytics engineering workflows
+user-invocable: false
+---
+
+# dbt Best Practices
+
+## Project Structure
+
+### Layer Organization (CRITICAL)
+- **Staging**: 1:1 with source tables (`stg_{source}__{entity}`)
+- **Intermediate**: Business logic composition (`int_{entity}_{verb}`)
+- **Marts**: Final consumption models (`fct_{entity}`, `dim_{entity}`)
+
+### Materialization Strategy
+- Staging: `view` (lightweight, always fresh)
+- Intermediate: `ephemeral` or `view`
+- Marts: `table` or `incremental`
+
+## Modeling Patterns
+
+### Naming Conventions
+- Staging: `stg_source__table`
+- Intermediate: `int_entity_verb`
+- Facts: `fct_entity`
+- Dimensions: `dim_entity`
+
+### Incremental Models
+- Use `is_incremental()` macro
+- Define `unique_key` for merge strategy
+- Choose strategy: append, merge, delete+insert
+
+## Testing
+
+### Schema Tests
+- `unique`, `not_null` for primary keys
+- `relationships` for foreign keys
+- `accepted_values` for enums
+- Custom data tests
+
+### Source Freshness
+- Configure `loaded_at_field`
+- Set freshness thresholds
+
+## Documentation
+
+- Add descriptions to models
+- Document column definitions
+- Use `doc` blocks for reusable text
+- Generate and host dbt docs
+
+## References
+- [dbt Best Practices](https://docs.getdbt.com/guides/best-practices)

package/templates/.claude/skills/de-lead-routing/SKILL.md

@@ -0,0 +1,230 @@
+---
+name: de-lead-routing
+description: Routes data engineering tasks to the correct DE expert agent. Use when user requests data pipeline design, DAG authoring, SQL modeling, stream processing, or warehouse optimization.
+user-invocable: false
+---
+
+# DE Lead Routing Skill
+
+## Purpose
+
+Routes data engineering tasks to appropriate DE expert agents. This skill contains the coordination logic for orchestrating data engineering agents across orchestration, modeling, processing, streaming, and warehouse specializations.
+
+## Engineers Under Management
+
+| Type | Agents | Purpose |
+|------|--------|---------|
+| de/orchestration | de-airflow-expert | DAG authoring, scheduling, testing |
+| de/modeling | de-dbt-expert | SQL modeling, testing, documentation |
+| de/processing | de-spark-expert | Distributed data processing |
+| de/streaming | de-kafka-expert | Event streaming, topic design |
+| de/warehouse | de-snowflake-expert | Cloud DWH, query optimization |
+| de/architecture | de-pipeline-expert | Pipeline design, cross-tool patterns |
+
+## Tool/Framework Detection
+
+### Keyword Mapping
+
+| Keyword | Agent |
+|---------|-------|
+| "airflow", "dag", "scheduling", "orchestration" | de-airflow-expert |
+| "dbt", "modeling", "sql model", "analytics engineering" | de-dbt-expert |
+| "spark", "pyspark", "distributed processing", "distributed" | de-spark-expert |
+| "kafka", "streaming", "event", "consumer", "producer" | de-kafka-expert |
+| "snowflake", "warehouse", "clustering key" | de-snowflake-expert |
+| "pipeline", "ETL", "ELT", "data quality", "lineage" | de-pipeline-expert |
+| "iceberg", "table format" | de-snowflake-expert or de-pipeline-expert |
+
+### File Pattern Mapping
+
+| Pattern | Agent |
+|---------|-------|
+| `dags/*.py`, `airflow.cfg`, `airflow_settings.yaml` | de-airflow-expert |
+| `models/**/*.sql`, `dbt_project.yml`, `schema.yml` | de-dbt-expert |
+| Spark job files, `spark-submit` configs | de-spark-expert |
+| Kafka configs, `*.properties` (Kafka), `streams/*.java` | de-kafka-expert |
+| Snowflake SQL, warehouse DDL | de-snowflake-expert |
+
+## Command Routing
+
+```
+DE Request → Detection → Expert Agent
+
+Airflow DAG → de-airflow-expert
+dbt model   → de-dbt-expert
+Spark job   → de-spark-expert
+Kafka topic → de-kafka-expert
+Snowflake   → de-snowflake-expert
+Pipeline    → de-pipeline-expert
+Multi-tool  → Multiple experts (parallel)
+```
+
+## Routing Rules
+
+### 1. Pipeline Development Workflow
+
+```
+1. Receive pipeline task request
+2. Identify tools and components:
+   - DAG orchestration → de-airflow-expert
+   - SQL transformations → de-dbt-expert
+   - Distributed processing → de-spark-expert
+   - Event streaming → de-kafka-expert
+   - Warehouse operations → de-snowflake-expert
+   - Architecture decisions → de-pipeline-expert
+3. Select appropriate experts
+4. Distribute tasks (parallel if 2+ tools)
+5. Aggregate results
+6. Present unified report
+```
+
+Example:
+```
+User: "Design a pipeline that runs dbt models from Airflow and loads into Snowflake"
+
+Detection:
+  - Airflow DAG → de-airflow-expert
+  - dbt model → de-dbt-expert
+  - Snowflake loading → de-snowflake-expert
+  - Pipeline architecture → de-pipeline-expert
+
+Route (parallel where independent):
+  Task(de-pipeline-expert → overall architecture design)
+  Task(de-airflow-expert → DAG structure)
+  Task(de-dbt-expert → model design)
+  Task(de-snowflake-expert → warehouse setup)
+
+Aggregate:
+  Pipeline architecture defined
+  Airflow DAG: 5 tasks designed
+  dbt: 12 models structured
+  Snowflake: warehouse + schema configured
+```
+
+### 2. Data Quality Workflow
+
+```
+1. Analyze data quality requirements
+2. Route to appropriate experts:
+   - dbt tests → de-dbt-expert
+   - Pipeline validation → de-pipeline-expert
+   - Source freshness → de-airflow-expert
+3. Coordinate cross-tool quality strategy
+```
+
+### 3. Multi-Tool Projects
+
+For projects spanning multiple DE tools:
+
+```
+1. Detect all DE tools in project
+2. Identify primary tool (most files/configs)
+3. Route to appropriate experts:
+   - If task spans multiple tools → parallel experts
+   - If task is tool-specific → single expert
+4. Coordinate cross-tool consistency
+```
+
+## Sub-agent Model Selection
+
+### Model Mapping by Task Type
+
+| Task Type | Recommended Model | Reason |
+|-----------|-------------------|--------|
+| Pipeline architecture | `opus` | Deep reasoning required |
+| DAG/model review | `sonnet` | Balanced quality judgment |
+| Implementation | `sonnet` | Standard code generation |
+| Quick validation | `haiku` | Fast response |
+
+### Model Mapping by Agent
+
+| Agent | Default Model | Alternative |
+|-------|---------------|-------------|
+| de-pipeline-expert | `sonnet` | `opus` for architecture |
+| de-airflow-expert | `sonnet` | `haiku` for DAG validation |
+| de-dbt-expert | `sonnet` | `haiku` for test checks |
+| de-spark-expert | `sonnet` | `opus` for optimization |
+| de-kafka-expert | `sonnet` | `opus` for topology design |
+| de-snowflake-expert | `sonnet` | `opus` for warehouse design |
+
+### Task Call Examples
+
+```
+# Complex pipeline architecture
+Task(
+  subagent_type: "general-purpose",
+  prompt: "Design end-to-end pipeline architecture following de-pipeline-expert guidelines",
+  model: "opus"
+)
+
+# Standard DAG review
+Task(
+  subagent_type: "general-purpose",
+  prompt: "Review Airflow DAGs in dags/ following de-airflow-expert guidelines",
+  model: "sonnet"
+)
+
+# Quick dbt test validation
+Task(
+  subagent_type: "Explore",
+  prompt: "Find all dbt models missing schema tests",
+  model: "haiku"
+)
+```
+
+## Parallel Execution
+
+Following R009:
+- Maximum 4 parallel instances
+- Independent tool/module operations
+- Coordinate cross-tool consistency
+
+Example:
+```
+User: "Review all DE configs"
+
+Detection:
+  - dags/ → de-airflow-expert
+  - models/ → de-dbt-expert
+  - kafka/ → de-kafka-expert
+
+Route (parallel):
+  Task(de-airflow-expert role → review dags/, model: "sonnet")
+  Task(de-dbt-expert role → review models/, model: "sonnet")
+  Task(de-kafka-expert role → review kafka/, model: "sonnet")
+```
+
+## Display Format
+
+```
+[Analyzing] Detected: Airflow, dbt, Snowflake
+
+[Delegating] de-airflow-expert:sonnet → DAG design
+[Delegating] de-dbt-expert:sonnet → Model structure
+[Delegating] de-snowflake-expert:sonnet → Warehouse config
+
+[Progress] ███████████░ 2/3 experts completed
+
+[Summary]
+  Airflow: DAG with 5 tasks designed
+  dbt: 12 models across 3 layers
+  Snowflake: Warehouse + schema configured
+
+Pipeline design completed.
+```
+
+## Integration with Other Routing Skills
+
+- **dev-lead-routing**: Hands off to DE lead when data engineering keywords detected
+- **secretary-routing**: DE agents accessible through secretary for management tasks
+- **qa-lead-routing**: Coordinates with QA for data quality testing
+
+## Usage
+
+This skill is NOT user-invocable. It should be automatically triggered when the main conversation detects data engineering intent.
+
+Detection criteria:
+- User requests pipeline design or data engineering
+- User mentions DE tool names (Airflow, dbt, Spark, Kafka, Snowflake)
+- User provides DE-related file paths (dags/, models/, etc.)
+- User requests data quality or lineage work

package/templates/.claude/skills/dev-lead-routing/SKILL.md

@@ -18,6 +18,9 @@ Routes development tasks to appropriate language and framework expert agents. Th
 | sw-engineer/frontend | fe-vercel-agent, fe-vuejs-agent, fe-svelte-agent | Frontend frameworks |
 | sw-engineer/backend | be-fastapi-expert, be-springboot-expert, be-go-backend-expert, be-nestjs-expert, be-express-expert | Backend frameworks |
 | sw-engineer/tooling | tool-npm-expert, tool-optimizer, tool-bun-expert | Build tools and optimization |
+| sw-engineer/database | db-supabase-expert, db-postgres-expert, db-redis-expert | Database design and optimization |
+| sw-architect | arch-documenter, arch-speckit-agent | Architecture documentation and spec-driven development |
+| infra-engineer | infra-docker-expert, infra-aws-expert | Container and cloud infrastructure |
 
 ## Language/Framework Detection
 
@@ -34,6 +37,11 @@ Routes development tasks to appropriate language and framework expert agents. Th
 | `.js`, `.jsx` (React) | fe-vercel-agent | React/Next.js |
 | `.vue` | fe-vuejs-agent | Vue.js |
 | `.svelte` | fe-svelte-agent | Svelte |
+| `.sql` (PostgreSQL) | db-postgres-expert | PostgreSQL |
+| `.sql` (Supabase) | db-supabase-expert | Supabase PostgreSQL |
+| `Dockerfile`, `*.dockerfile` | infra-docker-expert | Docker |
+| `*.tf`, `*.tfvars` | infra-aws-expert | Terraform/IaC |
+| `*.yaml`, `*.yml` (CloudFormation) | infra-aws-expert | AWS CloudFormation |
 
 ### Keyword Mapping
 
@@ -55,6 +63,13 @@ Routes development tasks to appropriate language and framework expert agents. Th
 | "npm" | tool-npm-expert |
 | "optimize", "bundle" | tool-optimizer |
 | "bun" | tool-bun-expert |
+| "postgres", "postgresql", "pg_stat", "psql" | db-postgres-expert |
+| "redis", "cache", "pub/sub", "sorted set" | db-redis-expert |
+| "supabase", "rls", "edge function" | db-supabase-expert |
+| "docker", "dockerfile", "container", "compose" | infra-docker-expert |
+| "aws", "cloudformation", "cdk", "terraform", "vpc", "iam", "s3", "lambda" | infra-aws-expert |
+| "architecture", "adr", "openapi", "swagger", "diagram" | arch-documenter |
+| "spec", "specification", "tdd", "requirements" | arch-speckit-agent |
 
 ## Command Routing
 

package/templates/.claude/skills/kafka-best-practices/SKILL.md

@@ -0,0 +1,52 @@
+---
+name: kafka-best-practices
+description: Apache Kafka best practices for event streaming, topic design, and producer-consumer patterns
+user-invocable: false
+---
+
+# Apache Kafka Best Practices
+
+## Producer Patterns
+
+### Idempotent Producer (CRITICAL)
+- Enable `enable.idempotence=true`
+- Prevents duplicate messages
+- Requires `acks=all`, `retries > 0`, `max.in.flight.requests.per.connection <= 5`
+
+### Exactly-Once Semantics
+- Use transactional API: `initTransactions()`, `beginTransaction()`, `commitTransaction()`
+- For exactly-once end-to-end processing
+
+### Performance
+- Batching: `linger.ms` (wait for batch to fill)
+- Compression: `compression.type=snappy` or `lz4`
+- `batch.size`: 16KB default, tune based on message size
+
+## Consumer Patterns
+
+### Offset Management
+- Auto-commit: `enable.auto.commit=true` (at-least-once)
+- Manual commit: `commitSync()` or `commitAsync()` (better control)
+
+### Rebalancing
+- Cooperative sticky assignor: minimal rebalancing disruption
+- `session.timeout.ms` and `heartbeat.interval.ms` tuning
+
+### At-Least-Once vs Exactly-Once
+- At-least-once: default, idempotent processing required
+- Exactly-once: transactional consumer + producer
+
+## Topic Design
+
+### Partitioning
+- Partition count: based on throughput (MB/s ÷ partition throughput)
+- Key-based partitioning for ordering guarantees
+- More partitions = higher throughput (but more overhead)
+
+### Retention
+- Time-based: `retention.ms`
+- Size-based: `retention.bytes`
+- Log compaction: for changelog topics (`cleanup.policy=compact`)
+
+## References
+- [Kafka Documentation](https://kafka.apache.org/documentation/)

package/templates/.claude/skills/pipeline-architecture-patterns/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: pipeline-architecture-patterns
+description: Data pipeline architecture patterns for ETL/ELT design, orchestration, and data quality frameworks
+user-invocable: false
+---
+
+# Data Pipeline Architecture Patterns
+
+## Pipeline Architectures
+
+### ETL vs ELT (CRITICAL)
+- **ETL**: Extract → Transform (staging) → Load
+  - Traditional, on-premise data warehouses
+  - Pre-aggregation, complex transformations
+- **ELT**: Extract → Load (raw) → Transform (in warehouse)
+  - Cloud warehouses (Snowflake, BigQuery)
+  - Leverage warehouse compute power
+
+### Lambda Architecture
+- Batch layer: historical data processing
+- Speed layer: real-time stream processing
+- Serving layer: merge batch + real-time views
+- Complexity: maintain two codebases
+
+### Kappa Architecture
+- Stream-only processing
+- Single codebase for batch + real-time
+- Reprocessing via replay
+- Simpler than Lambda
+
+### Medallion Architecture
+- **Bronze**: Raw data (append-only)
+- **Silver**: Cleaned, conformed data
+- **Gold**: Business-level aggregations
+- Databricks pattern
+
+## Orchestration Patterns
+
+### DAG-Based Orchestration
+- Airflow, Prefect, Dagster
+- Task dependencies as DAG
+- Retries, backfills, scheduling
+
+### Event-Driven Orchestration
+- Kafka, Pub/Sub triggers
+- Real-time, low-latency
+- Decoupled producers/consumers
+
+### Hybrid Orchestration
+- Scheduled batch + event-driven streams
+- Example: Airflow DAG triggered by Kafka event
+
+## Data Quality Frameworks
+
+### Data Contracts (CRITICAL)
+- Define schema, freshness, volume expectations
+- Producer-consumer agreement
+- Break build on violation
+
+### Validation Frameworks
+- **Great Expectations**: Python-based expectations
+- **dbt tests**: SQL-based tests
+- **Soda**: YAML-based checks
+
+### Data Lineage
+- Track data origin and transformations
+- Debug data quality issues
+- Compliance and auditing
+
+## Idempotency Patterns
+
+### Idempotent Design (CRITICAL)
+- Same input → same output (no side effects)
+- Upserts instead of inserts
+- Partition replacement instead of append
+
+### Deduplication
+- Use unique keys
+- Window-based deduplication
+- Consumer group offset management
+
+## References
+- [Data Engineering Design Patterns](https://www.oreilly.com/library/view/data-engineering-design/9781098130725/)