mindforge-cc 10.0.2 → 10.7.0

package/.mindforge/skills/data-governance/SKILL.md

@@ -0,0 +1,42 @@
+---
+name: data-governance
+version: 1.0.0
+min_mindforge_version: 10.6.0
+status: stable
+triggers: data governance framework, data catalog implementation, data lineage tracking, data access control, data quality framework, data stewardship, metadata management, data classification, data retention policy, data discovery platform, data ownership, data compliance framework
+---
+
+# Skill — Data Governance
+
+## When this skill activates
+This skill activates when implementing data catalog systems, establishing data ownership models, building lineage tracking, or designing access control frameworks. Use when organizations need to scale data democratization while maintaining compliance and quality.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Define data classification taxonomy with clear criteria: public, internal, confidential, restricted with handling requirements for each tier
+2. Establish data ownership model: domain owners, data stewards, technical custodians with RACI matrix for responsibilities
+3. Document compliance requirements: GDPR, CCPA, HIPAA, SOC2 with specific technical controls needed for each regulation
+4. Design metadata schema capturing: business definitions, technical specifications, quality metrics, lineage, and access policies
+
+### During implementation
+- Build automated data catalog discovery scanning databases, data lakes, APIs, and file systems to populate metadata repository
+- Implement column-level lineage tracking from source systems through transformations to final consumption with impact analysis capabilities
+- Create role-based access control (RBAC) with attribute-based policies (ABAC) for dynamic access based on data classification and user context
+- Establish data quality framework with profiling rules, validation checks, and quality scores at dataset and column level
+- Implement data retention policies with automated archival and deletion workflows based on regulatory requirements and business rules
+- Build data stewardship workflows for metadata enrichment: business glossary terms, data ownership assignment, quality issue resolution
+- Create audit logging for all data access, modifications, and policy changes with immutable trail for compliance reporting
+
+### After implementation
+- Deploy self-serve data discovery portal with search, business glossary, quality indicators, and access request workflows
+- Generate automated data quality reports with trend analysis, anomaly detection, and stakeholder-specific dashboards
+- Create compliance audit packages with evidence of controls: access logs, retention proof, encryption verification, lineage documentation
+- Build data governance metrics dashboard: catalog coverage, metadata completeness, quality score trends, access request SLA
+
+## Self-check before task completion
+- [ ] Data catalog covers 90%+ of production data assets with accurate business metadata
+- [ ] Lineage tracking provides end-to-end visibility from source to consumption with transformation logic
+- [ ] Access control policies enforced at query time with separation of duties for sensitive data
+- [ ] Data quality framework monitors critical datasets with automated alerting on quality degradation
+- [ ] Compliance documentation generated automatically with evidence trails for audit requirements

package/.mindforge/skills/data-lakehouse/SKILL.md

@@ -0,0 +1,42 @@
+---
+name: data-lakehouse
+version: 1.0.0
+min_mindforge_version: 10.6.0
+status: stable
+triggers: data lakehouse architecture, medallion architecture, schema evolution lakehouse, time travel data, partition optimization, Delta Lake implementation, Iceberg table design, lakehouse query performance, data lakehouse governance, lakehouse ingestion, lakehouse serving layer, lakehouse cost optimization
+---
+
+# Skill — Data Lakehouse
+
+## When this skill activates
+This skill activates when implementing lakehouse architectures combining data lake flexibility with data warehouse performance. Use when building Delta Lake, Iceberg, or Hudi tables with ACID guarantees, schema evolution, and time travel capabilities.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Design medallion architecture layers: bronze (raw ingestion), silver (cleansed/conformed), gold (business-level aggregates) with clear promotion criteria
+2. Select table format (Delta, Iceberg, Hudi) based on requirements: write patterns, query patterns, ecosystem compatibility, and feature needs
+3. Plan partitioning strategy based on query patterns: typically date/time for time-series, geography for location-based, or composite keys avoiding over-partitioning (<1000 partitions)
+4. Define schema evolution policy: additive changes (safe), nullable to required (breaking), type changes (migration required) with versioning strategy
+
+### During implementation
+- Implement ACID transactions for atomic writes: use table format's transaction log (Delta Log, Iceberg metadata) to ensure consistency
+- Configure file sizing for optimal query performance: target 128MB-1GB per file, run regular OPTIMIZE/COMPACT operations to prevent small files
+- Enable time travel with retention policy: maintain snapshots for point-in-time queries and audit, configure vacuum/expire based on compliance needs
+- Design incremental processing patterns: merge/upsert operations for CDC, append for event streams, overwrite partitions for batch updates
+- Implement Z-ordering or clustering on frequently filtered columns (non-partition keys) to improve query performance via data skipping
+- Build schema evolution handlers: automatic schema merging for new columns, validation for breaking changes, schema registry integration
+- Create data quality checkpoints between medallion layers: row counts, null checks, referential integrity, business rule validation with quarantine tables
+
+### After implementation
+- Monitor table health metrics: file count, average file size, partition count, metadata size, and compaction needs
+- Build cost optimization reports: storage by layer, compute for jobs, query costs, and opportunities for partition pruning or materialization
+- Create governance controls: table-level access policies, column masking, row filtering, and audit logging for sensitive data access
+- Generate performance analysis: query patterns, partition pruning effectiveness, file skipping statistics, and optimization recommendations
+
+## Self-check before task completion
+- [ ] Medallion layers clearly defined with data quality gates between bronze → silver → gold promotions
+- [ ] Partitioning strategy optimized for query patterns with validation that queries prune partitions effectively
+- [ ] ACID transactions tested with concurrent writes and failure scenarios to ensure consistency
+- [ ] Schema evolution tested with backward and forward compatibility for common evolution scenarios
+- [ ] File management strategy (OPTIMIZE/VACUUM) scheduled with monitoring for small file accumulation

package/.mindforge/skills/data-mesh/SKILL.md

@@ -0,0 +1,42 @@
+---
+name: data-mesh
+version: 1.0.0
+min_mindforge_version: 10.6.0
+status: stable
+triggers: data mesh architecture, domain data ownership, data product design, federated data governance, self-serve data platform, data mesh implementation, data contract mesh, data domain boundary, mesh interoperability, decentralized data ownership, data product specification, domain-driven data
+---
+
+# Skill — Data Mesh
+
+## When this skill activates
+This skill activates when implementing data mesh architectures with domain-oriented ownership, federated governance, and data-as-a-product principles. Use when centralizing data management becomes a bottleneck and domains need autonomy with interoperability.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Define domain boundaries using domain-driven design: bounded contexts, ubiquitous language, core domains vs supporting domains with ownership mapping
+2. Establish data product specification template: SLAs, schemas, semantics, access controls, versioning, and quality guarantees each domain must provide
+3. Design federated computational governance: global standards (discovery, security, interoperability) enforced through platform, local decisions (tech stack, modeling) owned by domains
+4. Create self-serve data platform capabilities: provisioning automation, observability tools, discovery services, and development environments domains can use independently
+
+### During implementation
+- Build data product registry with standardized metadata: domain owner, SLAs, schemas, sample data, access request process, and consumer feedback
+- Implement data contracts between domains: schema definitions, backward compatibility guarantees, deprecation policies, and breaking change notifications
+- Create domain-agnostic platform services: infrastructure provisioning (IaC templates), CI/CD pipelines, monitoring dashboards, and cost allocation
+- Design data product APIs with consistency: REST for batch, streaming for real-time, query engines for analytical, with versioning and deprecation paths
+- Establish quality frameworks domains must implement: data validation, profiling, lineage tracking, incident response with federated monitoring
+- Build interoperability layer: common data types, standard formats (Parquet, Avro), semantic layer, and cross-domain joins through data products not direct access
+- Implement federated identity and access: domain-owned authorization, centralized authentication, audit logging, and privacy controls enforced at platform level
+
+### After implementation
+- Create data product marketplace: searchable catalog, quality scores, usage analytics, consumer reviews, and onboarding documentation
+- Build platform health metrics: provisioning time, incident resolution SLA, platform uptime, and developer satisfaction scores
+- Generate federated governance reports: compliance by domain, quality trends, cross-domain dependencies, and policy violations
+- Document domain interaction patterns: producer-consumer relationships, data sharing agreements, and conflict resolution processes
+
+## Self-check before task completion
+- [ ] Domain boundaries clearly defined with ownership assignments and RACI matrix for responsibilities
+- [ ] Data products meet platform standards for discoverability, access control, quality, and SLAs
+- [ ] Self-serve platform enables domains to provision, deploy, and monitor data products independently
+- [ ] Federated governance enforces global standards while allowing domain autonomy in implementation
+- [ ] Interoperability tested across domains through data product contracts and APIs, not direct database access

package/.mindforge/skills/data-modeling/SKILL.md

@@ -0,0 +1,107 @@
+---
+name: data-modeling
+version: 1.0.0
+min_mindforge_version: 10.0.7
+status: stable
+triggers: data modeling, dimensional model design, star schema design, snowflake schema design, normalization decision, schema evolution strategy, data contract definition, slowly changing dimension, entity relationship design, data warehouse modeling, schema lifecycle, data lineage mapping
+---
+
+# Data Modeling
+
+## When this skill activates
+
+This skill activates when the user is designing, implementing, or evolving data models.
+This includes entity-relationship design, dimensional modeling (star/snowflake schemas),
+normalization decisions, slowly changing dimension strategies, schema evolution planning,
+data contract definitions between producers and consumers, and data lineage mapping
+across transformation pipelines.
+
+## Mandatory actions
+
+### Before
+
+1. Identify the workload type: OLTP (transactional) vs OLAP (analytical) vs hybrid.
+2. Determine the primary consumers of the data (applications, analysts, ML pipelines).
+3. Assess data volume, velocity, and variety characteristics.
+4. Review existing schemas and their evolution history.
+5. Identify upstream data sources and downstream consumers (lineage context).
+
+### During
+
+**Modeling Phases (Conceptual to Logical to Physical):**
+- **Conceptual:** Business entities and relationships, no implementation details. Stakeholder-readable.
+- **Logical:** Attributes, data types, keys, constraints. Technology-agnostic.
+- **Physical:** Indexes, partitions, storage engines, materialized views. Technology-specific.
+- Always start conceptual, refine to logical, then optimize physical. Never skip phases.
+
+**Normalization (OLTP):**
+- **1NF:** Eliminate repeating groups; atomic values in every column.
+- **2NF:** Remove partial dependencies (all non-key columns depend on the full primary key).
+- **3NF:** Remove transitive dependencies (non-key columns depend only on the key).
+- **BCNF:** Every determinant is a candidate key.
+- Normalize for OLTP (reduces anomalies, ensures consistency).
+- Denormalize for OLAP (reduces joins, improves query performance).
+- Document every denormalization decision with rationale.
+
+**Star Schema (Dimensional Modeling):**
+- **Fact tables:** Measurable events (transactions, clicks, shipments). Contain foreign keys + metrics.
+- **Dimension tables:** Descriptive context (who, what, where, when, how).
+- **Grain definition:** The most atomic level of detail in a fact table. Define grain FIRST.
+- Prefer conformed dimensions (shared across fact tables) for consistency.
+- Junk dimensions: combine low-cardinality flags into a single dimension.
+
+**Snowflake Schema:**
+- Use when dimensions have natural sub-hierarchies (geography: country → state → city).
+- Normalizes dimension tables to reduce redundancy.
+- Trade-off: more joins but less storage and clearer hierarchy.
+- Prefer star schema unless dimension table size or update frequency justifies snowflaking.
+
+**Slowly Changing Dimensions (SCD):**
+- **Type 0:** Fixed, never changes (date of birth).
+- **Type 1:** Overwrite old value. No history preserved. Use for corrections.
+- **Type 2:** Add new row with version tracking (start_date, end_date, is_current). Full history.
+- **Type 3:** Add "previous" column alongside current. Limited history (one prior value).
+- **Type 6 (Hybrid):** Combines Types 1, 2, and 3 for maximum flexibility.
+- Default to Type 2 unless storage or query complexity is a concern.
+
+**Data Contracts:**
+- Agreement between data producer and consumer on schema + semantics + SLA.
+- Schema: field names, types, nullability, constraints.
+- Semantics: business meaning of each field (not just technical definition).
+- SLA: freshness guarantee, completeness threshold, availability window.
+- Enforce contracts via schema validation in pipelines (Great Expectations, dbt tests).
+- Breaking contract changes require notification and migration period.
+
+**Schema Evolution:**
+- **Additive (safe):** New optional columns, new tables, new indexes.
+- **Breaking (dangerous):** Column removal, type changes, renaming, adding NOT NULL without default.
+- Use migration scripts (Flyway, Alembic, Liquibase) for all schema changes.
+- Version schemas and maintain a changelog.
+- Test migrations against production-like data volumes before deploying.
+
+**Data Lineage:**
+- Track data from source → transformation → consumption.
+- Document at column-level granularity for critical fields.
+- Use tools (dbt lineage graph, Apache Atlas, DataHub) for automated discovery.
+- Lineage enables impact analysis (what breaks if this source changes?).
+- Required for regulatory compliance (GDPR: where does PII flow?).
+
+### After
+
+1. Verify grain is explicitly defined for every fact table.
+2. Confirm normalization level matches workload type (OLTP normalized, OLAP denormalized).
+3. Validate SCD strategy is documented for every dimension with mutable attributes.
+4. Ensure data contracts exist between all critical producer-consumer pairs.
+5. Check that schema evolution follows additive-first principles.
+6. Verify lineage is documented for compliance-sensitive data flows.
+
+## Self-check before task completion
+
+- [ ] Modeling followed the conceptual → logical → physical progression.
+- [ ] Normalization level is appropriate for the workload (OLTP vs OLAP).
+- [ ] Fact table grain is explicitly defined and documented.
+- [ ] SCD types are chosen and justified for mutable dimensions.
+- [ ] Data contracts define schema, semantics, and SLA for critical interfaces.
+- [ ] Schema evolution strategy avoids breaking changes without migration.
+- [ ] Data lineage is mapped for compliance-sensitive and business-critical paths.
+- [ ] Physical optimizations (indexes, partitions) are justified by query patterns.

package/.mindforge/skills/data-pipeline-design/SKILL.md

@@ -0,0 +1,171 @@
+---
+name: data-pipeline-design
+version: 1.0.0
+min_mindforge_version: 10.1.1
+status: stable
+triggers: data pipeline design, ETL pipeline, ELT pattern, batch vs streaming pipeline, exactly-once processing pipeline, schema registry pipeline, data quality gate, pipeline orchestration, data ingestion, pipeline backfill, pipeline monitoring, data freshness
+---
+
+# Skill — Data Pipeline Design
+
+## When this skill activates
+Any task involving designing data ingestion, transformation, or delivery pipelines.
+Includes ETL/ELT architecture, batch vs streaming decisions, schema management,
+data quality enforcement, and pipeline orchestration.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Define data contract (schema, freshness SLA, volume, sources, consumers).
+2. Decide batch vs streaming (latency requirement is the primary driver).
+3. Identify exactly-once requirements (financial data = must, analytics = can relax).
+4. Plan schema evolution strategy (backward-compatible changes only).
+
+### During implementation
+- Implement data quality gates before consumers see data.
+- Use schema registry for all structured data exchange.
+- Make all transformations idempotent (safe to re-run).
+- Include dead-letter queues for malformed/failed records.
+- Add lineage tracking (where did this data come from?).
+- Monitor freshness SLA with alerting.
+
+### After implementation
+- Verify backfill capability (can we reprocess historical data?).
+- Test schema evolution (add column, change type) without breaking consumers.
+- Confirm quality gates catch known bad data patterns.
+- Validate freshness SLA is met under normal load.
+- Document data lineage for every output table.
+
+## ETL vs ELT Decision
+
+### ETL (Extract → Transform → Load)
+- Transform before loading into destination.
+- Best for: structured sources, known transformations, data quality at boundary.
+- Tools: Airflow + Python, Spark, custom processors.
+- Advantage: Clean data in warehouse, fewer warehouse compute costs.
+
+### ELT (Extract → Load → Transform)
+- Load raw data, transform in the warehouse/lakehouse.
+- Best for: diverse sources, evolving transformations, exploratory analysis.
+- Tools: Fivetran/Airbyte (extract+load) + dbt (transform).
+- Advantage: Raw data preserved, transformations versioned and testable.
+
+### Decision Matrix
+| Factor | ETL | ELT |
+|--------|-----|-----|
+| Source diversity | Low (known schema) | High (many sources) |
+| Transformation stability | Stable, well-defined | Evolving, experimental |
+| Data volume | Moderate | Very high |
+| Warehouse compute cost | Sensitive | Acceptable |
+| Need raw data access | No | Yes |
+
+## Batch vs Streaming
+
+### Batch Processing
+- Process data in scheduled intervals (hourly, daily).
+- Simpler implementation, easier debugging.
+- Cheaper for high-volume, latency-tolerant workloads.
+- Tools: Airflow, Spark Batch, dbt.
+
+### Stream Processing
+- Process events as they arrive (real-time or near-real-time).
+- Complex: windowing, ordering, late-arriving data.
+- Required when business needs data in <5 minutes.
+- Tools: Kafka Streams, Flink, Spark Structured Streaming.
+
+### Decision: Use streaming only when
+- Business requires <5 minute data freshness.
+- Events must trigger immediate actions (fraud, alerts).
+- Source naturally produces events (clickstream, IoT).
+
+Otherwise, batch is simpler and cheaper.
+
+## Exactly-Once Processing
+
+### Why It's Hard
+Network failures + retries = potential duplicates.
+
+### Strategies
+1. **Idempotent sinks**: Write operations produce same result regardless of repetition (UPSERT, conditional write).
+2. **Deduplication keys**: Assign unique ID to each record, deduplicate at sink.
+3. **Checkpointing**: Record progress markers, resume from checkpoint on failure.
+4. **Transactional outbox**: Atomic write to source + outbox table, separate relay.
+
+### Practical Guarantees
+| Guarantee | Cost | Use When |
+|-----------|------|----------|
+| At-most-once | Lowest | Metrics where loss is acceptable |
+| At-least-once + idempotent sink | Medium | Most pipelines |
+| Exactly-once (Kafka transactions) | Highest | Financial, billing |
+
+## Schema Registry
+
+### Purpose
+- Central source of truth for data schemas.
+- Enforce compatibility between producers and consumers.
+- Enable schema evolution without breaking downstream.
+
+### Compatibility Modes
+- **Backward compatible**: New schema can read old data (add optional fields).
+- **Forward compatible**: Old schema can read new data (remove optional fields).
+- **Full compatible**: Both backward and forward (safest, most restrictive).
+
+### Rules
+- All structured data exchange goes through schema registry.
+- Use Avro or Protobuf (self-describing, compact, evolvable).
+- Test schema changes against compatibility rules in CI.
+- Never break backward compatibility without coordinated migration.
+
+## Data Quality Gates
+
+### Checks to Implement
+| Check | Example | Severity |
+|-------|---------|----------|
+| Not null | Primary keys must exist | CRITICAL |
+| Uniqueness | No duplicate records | CRITICAL |
+| Range | Age between 0-150 | HIGH |
+| Freshness | Data < 1 hour old | HIGH |
+| Volume | Row count ±10% of expected | MEDIUM |
+| Referential | Foreign keys resolve | MEDIUM |
+| Format | Email matches pattern | LOW |
+
+### Implementation
+- Run quality checks BEFORE exposing data to consumers.
+- Quarantine failing records in dead-letter table.
+- Alert on quality degradation trends.
+- Track quality metrics over time (quality score per table).
+
+## Pipeline Orchestration
+
+### Airflow DAG Best Practices
+- One DAG per logical pipeline.
+- Idempotent tasks (re-runnable without side effects).
+- Explicit dependencies (no implicit ordering).
+- SLA alerts for late-running pipelines.
+- Backfill support (catchup=True with idempotent tasks).
+- Retry with exponential backoff for transient failures.
+
+### Monitoring
+- Freshness SLA: alert when data is older than threshold.
+- Pipeline duration: alert on >2x normal runtime.
+- Record count: alert on ±20% deviation from expected.
+- Error rate: alert on >1% record failures.
+
+## Backfill Strategy
+
+### Requirements
+- Every pipeline must support historical reprocessing.
+- Backfill must be idempotent (running twice = same result).
+- Partition by date for efficient backfill of specific ranges.
+- Backfill should not interfere with production pipeline runs.
+
+## Self-check
+- [ ] Data contract defined (schema, freshness, volume).
+- [ ] Batch vs streaming decision justified by latency requirement.
+- [ ] Quality gates implemented before consumer access.
+- [ ] Schema registered and compatibility mode set.
+- [ ] All transformations are idempotent.
+- [ ] Dead-letter queue configured for failures.
+- [ ] Backfill capability tested.
+- [ ] Freshness SLA monitored with alerting.
+- [ ] Data lineage documented.

package/.mindforge/skills/data-privacy-engineering/SKILL.md

@@ -0,0 +1,42 @@
+---
+name: data-privacy-engineering
+version: 1.0.0
+min_mindforge_version: 10.6.0
+status: stable
+triggers: data privacy engineering, differential privacy implementation, anonymization technique, consent management system, privacy-preserving computation, GDPR data engineering, data masking, privacy by design, homomorphic encryption, federated learning privacy, secure multi-party computation, PII detection pipeline
+---
+
+# Skill — Data Privacy Engineering
+
+## When this skill activates
+This skill activates when implementing privacy-preserving data systems, building consent management infrastructure, or applying anonymization techniques. Use when handling sensitive data requires technical controls beyond access restrictions.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Conduct privacy impact assessment: identify PII/sensitive data, data flows, retention requirements, third-party sharing, and regulatory obligations (GDPR, CCPA, HIPAA)
+2. Define privacy requirements: anonymization level (k-anonymity, l-diversity, differential privacy), consent granularity, right-to-erasure scope, and data minimization principles
+3. Select appropriate privacy techniques: tokenization (reversible), hashing (one-way), encryption (protected), differential privacy (statistical), synthetic data (replacement)
+4. Establish privacy testing framework: re-identification risk assessment, privacy budget tracking, consent enforcement verification, and breach simulation
+
+### During implementation
+- Implement automated PII detection pipeline: regex patterns, ML models, NER for unstructured text scanning code, logs, databases, and data lakes
+- Build tokenization service with: format-preserving encryption for display, secure token vault, key rotation, and performance caching for high-throughput
+- Create differential privacy mechanisms: Laplace/Gaussian noise addition calibrated to epsilon budget, query result perturbation, and privacy budget accounting across queries
+- Design consent management system: granular opt-in/opt-out, purpose-specific consent, consent version tracking, and propagation to downstream systems
+- Implement data minimization controls: retention policies with automated deletion, purpose limitation enforcement, and necessity justification for data collection
+- Build privacy-preserving analytics: federated learning for ML without centralized data, secure aggregation for metrics, and homomorphic encryption for computation on encrypted data
+- Create data subject rights workflows: search across systems, export in portable format, deletion with verification, and rectification propagation
+
+### After implementation
+- Generate privacy compliance reports: PII inventory, consent coverage, retention policy enforcement, third-party data sharing audit, and rights request fulfillment SLA
+- Build privacy monitoring dashboards: PII exposure incidents, consent withdrawal rates, privacy budget consumption, and anonymization quality metrics
+- Create breach response procedures: detection, containment, notification timelines, affected user identification, and remediation workflows
+- Document privacy controls: anonymization methods, re-identification risk levels, consent mechanisms, and data retention justifications for audit purposes
+
+## Self-check before task completion
+- [ ] PII detection covers all data stores with automated scanning and alerting on new sensitive data discoveries
+- [ ] Anonymization techniques applied with documented re-identification risk assessment (k-anonymity ≥10 or equivalent)
+- [ ] Consent management enforces purpose limitation with propagation to all downstream processing systems
+- [ ] Differential privacy implementation maintains epsilon budget <1.0 for sensitive aggregations with privacy accounting
+- [ ] Data subject rights workflows tested for completeness across all systems within regulatory SLA (30 days GDPR)

package/.mindforge/skills/database-performance/SKILL.md

@@ -0,0 +1,174 @@
+---
+name: database-performance
+version: 1.0.0
+min_mindforge_version: 0.3.0
+status: stable
+triggers: database performance, query plan analysis, EXPLAIN ANALYZE, index selection, partition pruning, materialized view, query optimization, slow query, index strategy, table scan elimination, join optimization, query profiling
+compose: database-patterns
+---
+
+# Skill — Database Performance
+
+## When this skill activates
+Any task involving slow queries, query optimization, index strategy, EXPLAIN plan
+analysis, partitioning, materialized views, or database profiling.
+
+## Mandatory actions when this skill is active
+
+### Before optimizing
+1. Get the current query execution plan (EXPLAIN ANALYZE, not just EXPLAIN).
+2. Identify the actual bottleneck (do not guess).
+3. Measure baseline performance (p50, p95, p99 latency).
+4. Understand the data distribution (cardinality, skew).
+
+### Reading EXPLAIN ANALYZE output
+
+**Key things to look for:**
+
+| Signal | Meaning | Action |
+|--------|---------|--------|
+| Seq Scan on large table | Full table scan, no index used | Add appropriate index |
+| Nested Loop with high rows | O(n*m) join strategy | Consider Hash Join, add index on join column |
+| Actual rows >> Estimated rows | Stale statistics | Run ANALYZE on the table |
+| Sort with external merge | Not enough work_mem | Increase work_mem or add index for ORDER BY |
+| Filter removing most rows | Index not selective enough | Add more specific index or partial index |
+
+**Node types (best to worst for large tables):**
+1. Index Only Scan — best (reads from index, no table access).
+2. Index Scan — good (uses index, fetches rows from table).
+3. Bitmap Index Scan — okay (for medium selectivity).
+4. Seq Scan — bad on large tables (reads every row).
+
+### Index strategy
+
+**B-tree (default, most common):**
+- Equality: `WHERE status = 'active'`
+- Range: `WHERE created_at > '2025-01-01'`
+- Prefix matching: `WHERE name LIKE 'foo%'`
+- Sorting: `ORDER BY created_at DESC`
+- Composite: `(tenant_id, created_at)` — order matters, left-to-right.
+
+**GIN (Generalized Inverted Index):**
+- JSONB containment: `WHERE data @> '{"key": "value"}'`
+- Array contains: `WHERE tags @> ARRAY['tag1']`
+- Full-text search: `WHERE to_tsvector(body) @@ to_tsquery('search')`
+
+**Partial index (conditional):**
+- Index only rows that match a condition.
+- `CREATE INDEX idx_active_orders ON orders(created_at) WHERE status = 'active'`
+- Smaller, faster, less write overhead.
+
+**Expression index:**
+- Index a computed value.
+- `CREATE INDEX idx_lower_email ON users(LOWER(email))`
+- Query must use the same expression to hit the index.
+
+### Common query anti-patterns
+
+**Functions on indexed columns:**
+```sql
+-- BAD: index on created_at is useless
+WHERE EXTRACT(YEAR FROM created_at) = 2025
+
+-- GOOD: rewrite as range
+WHERE created_at >= '2025-01-01' AND created_at < '2026-01-01'
+```
+
+**OR conditions preventing index use:**
+```sql
+-- BAD: may cause Seq Scan
+WHERE status = 'active' OR status = 'pending'
+
+-- GOOD: use IN
+WHERE status IN ('active', 'pending')
+```
+
+**SELECT * when you need few columns:**
+```sql
+-- BAD: fetches all columns, prevents index-only scan
+SELECT * FROM orders WHERE tenant_id = 'abc'
+
+-- GOOD: select only needed columns
+SELECT id, status, total FROM orders WHERE tenant_id = 'abc'
+```
+
+**Missing LIMIT on unbounded queries:**
+```sql
+-- BAD: may return millions of rows
+SELECT * FROM events WHERE type = 'click'
+
+-- GOOD: always paginate
+SELECT * FROM events WHERE type = 'click' ORDER BY id LIMIT 50
+```
+
+### Materialized views
+
+**When to use:**
+- Expensive aggregations needed frequently (dashboards, reports).
+- Data changes infrequently relative to read frequency.
+- Acceptable staleness (refresh interval is tolerable).
+
+**Implementation:**
+```sql
+CREATE MATERIALIZED VIEW monthly_revenue AS
+SELECT tenant_id, date_trunc('month', created_at) AS month, SUM(amount) AS total
+FROM orders
+WHERE status = 'completed'
+GROUP BY tenant_id, month;
+
+-- Refresh on schedule
+REFRESH MATERIALIZED VIEW CONCURRENTLY monthly_revenue;
+```
+
+**Rules:**
+- Always use CONCURRENTLY (does not lock reads during refresh).
+- Add a unique index for CONCURRENTLY to work.
+- Monitor refresh duration — alert if it exceeds threshold.
+- Consider triggers for real-time materialized views (small tables only).
+
+### Partitioning
+
+**Range partitioning (time-series data):**
+```sql
+CREATE TABLE events (
+  id BIGINT, tenant_id UUID, created_at TIMESTAMPTZ, data JSONB
+) PARTITION BY RANGE (created_at);
+
+CREATE TABLE events_2025_01 PARTITION OF events
+  FOR VALUES FROM ('2025-01-01') TO ('2025-02-01');
+```
+
+**Benefits:**
+- Partition pruning: queries on created_at only scan relevant partitions.
+- Easy data lifecycle: DROP old partitions instead of DELETE (instant, no vacuum).
+- Parallel scan across partitions.
+
+**Hash partitioning (even distribution):**
+- For tables with no natural range key.
+- Distributes rows evenly across N partitions.
+- Good for very large tables that need parallel access.
+
+**Rules:**
+- Partition key must be in every query's WHERE clause for pruning.
+- Too many partitions (>1000) can slow planning.
+- Automate partition creation (don't rely on manual monthly creation).
+
+### Join optimization
+
+- Ensure join columns have indexes on both sides.
+- Small table JOIN large table: ensure small table is the "driving" table.
+- Consider denormalization if a join is on the critical path and never changes.
+- Use CTEs carefully — in PostgreSQL < 12, CTEs are optimization fences.
+
+### Monitoring
+
+- Enable `pg_stat_statements` for query-level statistics.
+- Alert on queries exceeding p95 threshold.
+- Track index usage: `pg_stat_user_indexes` — unused indexes waste write performance.
+- Regular VACUUM and ANALYZE (autovacuum tuning for high-write tables).
+
+## Self-check before task completion
+- [ ] Did I follow the mandatory actions for this skill?
+- [ ] Did I apply the patterns appropriate to the context?
+- [ ] Did I verify the implementation meets the criteria above?
+- [ ] Did I document decisions and trade-offs made?