mindforge-cc 10.0.3 → 11.0.0

package/.mindforge/skills/data-modeling/SKILL.md

@@ -0,0 +1,107 @@
+---
+name: data-modeling
+version: 1.0.0
+min_mindforge_version: 10.0.7
+status: stable
+triggers: data modeling, dimensional model design, star schema design, snowflake schema design, normalization decision, schema evolution strategy, data contract definition, slowly changing dimension, entity relationship design, data warehouse modeling, schema lifecycle, data lineage mapping
+---
+
+# Data Modeling
+
+## When this skill activates
+
+This skill activates when the user is designing, implementing, or evolving data models.
+This includes entity-relationship design, dimensional modeling (star/snowflake schemas),
+normalization decisions, slowly changing dimension strategies, schema evolution planning,
+data contract definitions between producers and consumers, and data lineage mapping
+across transformation pipelines.
+
+## Mandatory actions
+
+### Before
+
+1. Identify the workload type: OLTP (transactional) vs OLAP (analytical) vs hybrid.
+2. Determine the primary consumers of the data (applications, analysts, ML pipelines).
+3. Assess data volume, velocity, and variety characteristics.
+4. Review existing schemas and their evolution history.
+5. Identify upstream data sources and downstream consumers (lineage context).
+
+### During
+
+**Modeling Phases (Conceptual to Logical to Physical):**
+- **Conceptual:** Business entities and relationships, no implementation details. Stakeholder-readable.
+- **Logical:** Attributes, data types, keys, constraints. Technology-agnostic.
+- **Physical:** Indexes, partitions, storage engines, materialized views. Technology-specific.
+- Always start conceptual, refine to logical, then optimize physical. Never skip phases.
+
+**Normalization (OLTP):**
+- **1NF:** Eliminate repeating groups; atomic values in every column.
+- **2NF:** Remove partial dependencies (all non-key columns depend on the full primary key).
+- **3NF:** Remove transitive dependencies (non-key columns depend only on the key).
+- **BCNF:** Every determinant is a candidate key.
+- Normalize for OLTP (reduces anomalies, ensures consistency).
+- Denormalize for OLAP (reduces joins, improves query performance).
+- Document every denormalization decision with rationale.
+
+**Star Schema (Dimensional Modeling):**
+- **Fact tables:** Measurable events (transactions, clicks, shipments). Contain foreign keys + metrics.
+- **Dimension tables:** Descriptive context (who, what, where, when, how).
+- **Grain definition:** The most atomic level of detail in a fact table. Define grain FIRST.
+- Prefer conformed dimensions (shared across fact tables) for consistency.
+- Junk dimensions: combine low-cardinality flags into a single dimension.
+
+**Snowflake Schema:**
+- Use when dimensions have natural sub-hierarchies (geography: country → state → city).
+- Normalizes dimension tables to reduce redundancy.
+- Trade-off: more joins but less storage and clearer hierarchy.
+- Prefer star schema unless dimension table size or update frequency justifies snowflaking.
+
+**Slowly Changing Dimensions (SCD):**
+- **Type 0:** Fixed, never changes (date of birth).
+- **Type 1:** Overwrite old value. No history preserved. Use for corrections.
+- **Type 2:** Add new row with version tracking (start_date, end_date, is_current). Full history.
+- **Type 3:** Add "previous" column alongside current. Limited history (one prior value).
+- **Type 6 (Hybrid):** Combines Types 1, 2, and 3 for maximum flexibility.
+- Default to Type 2 unless storage or query complexity is a concern.
+
+**Data Contracts:**
+- Agreement between data producer and consumer on schema + semantics + SLA.
+- Schema: field names, types, nullability, constraints.
+- Semantics: business meaning of each field (not just technical definition).
+- SLA: freshness guarantee, completeness threshold, availability window.
+- Enforce contracts via schema validation in pipelines (Great Expectations, dbt tests).
+- Breaking contract changes require notification and migration period.
+
+**Schema Evolution:**
+- **Additive (safe):** New optional columns, new tables, new indexes.
+- **Breaking (dangerous):** Column removal, type changes, renaming, adding NOT NULL without default.
+- Use migration scripts (Flyway, Alembic, Liquibase) for all schema changes.
+- Version schemas and maintain a changelog.
+- Test migrations against production-like data volumes before deploying.
+
+**Data Lineage:**
+- Track data from source → transformation → consumption.
+- Document at column-level granularity for critical fields.
+- Use tools (dbt lineage graph, Apache Atlas, DataHub) for automated discovery.
+- Lineage enables impact analysis (what breaks if this source changes?).
+- Required for regulatory compliance (GDPR: where does PII flow?).
+
+### After
+
+1. Verify grain is explicitly defined for every fact table.
+2. Confirm normalization level matches workload type (OLTP normalized, OLAP denormalized).
+3. Validate SCD strategy is documented for every dimension with mutable attributes.
+4. Ensure data contracts exist between all critical producer-consumer pairs.
+5. Check that schema evolution follows additive-first principles.
+6. Verify lineage is documented for compliance-sensitive data flows.
+
+## Self-check before task completion
+
+- [ ] Modeling followed the conceptual → logical → physical progression.
+- [ ] Normalization level is appropriate for the workload (OLTP vs OLAP).
+- [ ] Fact table grain is explicitly defined and documented.
+- [ ] SCD types are chosen and justified for mutable dimensions.
+- [ ] Data contracts define schema, semantics, and SLA for critical interfaces.
+- [ ] Schema evolution strategy avoids breaking changes without migration.
+- [ ] Data lineage is mapped for compliance-sensitive and business-critical paths.
+- [ ] Physical optimizations (indexes, partitions) are justified by query patterns.

package/.mindforge/skills/data-pipeline-design/SKILL.md

@@ -0,0 +1,171 @@
+---
+name: data-pipeline-design
+version: 1.0.0
+min_mindforge_version: 10.1.1
+status: stable
+triggers: data pipeline design, ETL pipeline, ELT pattern, batch vs streaming pipeline, exactly-once processing pipeline, schema registry pipeline, data quality gate, pipeline orchestration, data ingestion, pipeline backfill, pipeline monitoring, data freshness
+---
+
+# Skill — Data Pipeline Design
+
+## When this skill activates
+Any task involving designing data ingestion, transformation, or delivery pipelines.
+Includes ETL/ELT architecture, batch vs streaming decisions, schema management,
+data quality enforcement, and pipeline orchestration.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Define data contract (schema, freshness SLA, volume, sources, consumers).
+2. Decide batch vs streaming (latency requirement is the primary driver).
+3. Identify exactly-once requirements (financial data = must, analytics = can relax).
+4. Plan schema evolution strategy (backward-compatible changes only).
+
+### During implementation
+- Implement data quality gates before consumers see data.
+- Use schema registry for all structured data exchange.
+- Make all transformations idempotent (safe to re-run).
+- Include dead-letter queues for malformed/failed records.
+- Add lineage tracking (where did this data come from?).
+- Monitor freshness SLA with alerting.
+
+### After implementation
+- Verify backfill capability (can we reprocess historical data?).
+- Test schema evolution (add column, change type) without breaking consumers.
+- Confirm quality gates catch known bad data patterns.
+- Validate freshness SLA is met under normal load.
+- Document data lineage for every output table.
+
+## ETL vs ELT Decision
+
+### ETL (Extract → Transform → Load)
+- Transform before loading into destination.
+- Best for: structured sources, known transformations, data quality at boundary.
+- Tools: Airflow + Python, Spark, custom processors.
+- Advantage: Clean data in warehouse, fewer warehouse compute costs.
+
+### ELT (Extract → Load → Transform)
+- Load raw data, transform in the warehouse/lakehouse.
+- Best for: diverse sources, evolving transformations, exploratory analysis.
+- Tools: Fivetran/Airbyte (extract+load) + dbt (transform).
+- Advantage: Raw data preserved, transformations versioned and testable.
+
+### Decision Matrix
+| Factor | ETL | ELT |
+|--------|-----|-----|
+| Source diversity | Low (known schema) | High (many sources) |
+| Transformation stability | Stable, well-defined | Evolving, experimental |
+| Data volume | Moderate | Very high |
+| Warehouse compute cost | Sensitive | Acceptable |
+| Need raw data access | No | Yes |
+
+## Batch vs Streaming
+
+### Batch Processing
+- Process data in scheduled intervals (hourly, daily).
+- Simpler implementation, easier debugging.
+- Cheaper for high-volume, latency-tolerant workloads.
+- Tools: Airflow, Spark Batch, dbt.
+
+### Stream Processing
+- Process events as they arrive (real-time or near-real-time).
+- Complex: windowing, ordering, late-arriving data.
+- Required when business needs data in <5 minutes.
+- Tools: Kafka Streams, Flink, Spark Structured Streaming.
+
+### Decision: Use streaming only when
+- Business requires <5 minute data freshness.
+- Events must trigger immediate actions (fraud, alerts).
+- Source naturally produces events (clickstream, IoT).
+
+Otherwise, batch is simpler and cheaper.
+
+## Exactly-Once Processing
+
+### Why It's Hard
+Network failures + retries = potential duplicates.
+
+### Strategies
+1. **Idempotent sinks**: Write operations produce same result regardless of repetition (UPSERT, conditional write).
+2. **Deduplication keys**: Assign unique ID to each record, deduplicate at sink.
+3. **Checkpointing**: Record progress markers, resume from checkpoint on failure.
+4. **Transactional outbox**: Atomic write to source + outbox table, separate relay.
+
+### Practical Guarantees
+| Guarantee | Cost | Use When |
+|-----------|------|----------|
+| At-most-once | Lowest | Metrics where loss is acceptable |
+| At-least-once + idempotent sink | Medium | Most pipelines |
+| Exactly-once (Kafka transactions) | Highest | Financial, billing |
+
+## Schema Registry
+
+### Purpose
+- Central source of truth for data schemas.
+- Enforce compatibility between producers and consumers.
+- Enable schema evolution without breaking downstream.
+
+### Compatibility Modes
+- **Backward compatible**: New schema can read old data (add optional fields).
+- **Forward compatible**: Old schema can read new data (remove optional fields).
+- **Full compatible**: Both backward and forward (safest, most restrictive).
+
+### Rules
+- All structured data exchange goes through schema registry.
+- Use Avro or Protobuf (self-describing, compact, evolvable).
+- Test schema changes against compatibility rules in CI.
+- Never break backward compatibility without coordinated migration.
+
+## Data Quality Gates
+
+### Checks to Implement
+| Check | Example | Severity |
+|-------|---------|----------|
+| Not null | Primary keys must exist | CRITICAL |
+| Uniqueness | No duplicate records | CRITICAL |
+| Range | Age between 0-150 | HIGH |
+| Freshness | Data < 1 hour old | HIGH |
+| Volume | Row count ±10% of expected | MEDIUM |
+| Referential | Foreign keys resolve | MEDIUM |
+| Format | Email matches pattern | LOW |
+
+### Implementation
+- Run quality checks BEFORE exposing data to consumers.
+- Quarantine failing records in dead-letter table.
+- Alert on quality degradation trends.
+- Track quality metrics over time (quality score per table).
+
+## Pipeline Orchestration
+
+### Airflow DAG Best Practices
+- One DAG per logical pipeline.
+- Idempotent tasks (re-runnable without side effects).
+- Explicit dependencies (no implicit ordering).
+- SLA alerts for late-running pipelines.
+- Backfill support (catchup=True with idempotent tasks).
+- Retry with exponential backoff for transient failures.
+
+### Monitoring
+- Freshness SLA: alert when data is older than threshold.
+- Pipeline duration: alert on >2x normal runtime.
+- Record count: alert on ±20% deviation from expected.
+- Error rate: alert on >1% record failures.
+
+## Backfill Strategy
+
+### Requirements
+- Every pipeline must support historical reprocessing.
+- Backfill must be idempotent (running twice = same result).
+- Partition by date for efficient backfill of specific ranges.
+- Backfill should not interfere with production pipeline runs.
+
+## Self-check
+- [ ] Data contract defined (schema, freshness, volume).
+- [ ] Batch vs streaming decision justified by latency requirement.
+- [ ] Quality gates implemented before consumer access.
+- [ ] Schema registered and compatibility mode set.
+- [ ] All transformations are idempotent.
+- [ ] Dead-letter queue configured for failures.
+- [ ] Backfill capability tested.
+- [ ] Freshness SLA monitored with alerting.
+- [ ] Data lineage documented.

package/.mindforge/skills/data-privacy-engineering/SKILL.md

@@ -0,0 +1,42 @@
+---
+name: data-privacy-engineering
+version: 1.0.0
+min_mindforge_version: 10.6.0
+status: stable
+triggers: data privacy engineering, differential privacy implementation, anonymization technique, consent management system, privacy-preserving computation, GDPR data engineering, data masking, privacy by design, homomorphic encryption, federated learning privacy, secure multi-party computation, PII detection pipeline
+---
+
+# Skill — Data Privacy Engineering
+
+## When this skill activates
+This skill activates when implementing privacy-preserving data systems, building consent management infrastructure, or applying anonymization techniques. Use when handling sensitive data requires technical controls beyond access restrictions.
+
+## Mandatory actions when this skill is active
+
+### Before writing any code
+1. Conduct privacy impact assessment: identify PII/sensitive data, data flows, retention requirements, third-party sharing, and regulatory obligations (GDPR, CCPA, HIPAA)
+2. Define privacy requirements: anonymization level (k-anonymity, l-diversity, differential privacy), consent granularity, right-to-erasure scope, and data minimization principles
+3. Select appropriate privacy techniques: tokenization (reversible), hashing (one-way), encryption (protected), differential privacy (statistical), synthetic data (replacement)
+4. Establish privacy testing framework: re-identification risk assessment, privacy budget tracking, consent enforcement verification, and breach simulation
+
+### During implementation
+- Implement automated PII detection pipeline: regex patterns, ML models, NER for unstructured text scanning code, logs, databases, and data lakes
+- Build tokenization service with: format-preserving encryption for display, secure token vault, key rotation, and performance caching for high-throughput
+- Create differential privacy mechanisms: Laplace/Gaussian noise addition calibrated to epsilon budget, query result perturbation, and privacy budget accounting across queries
+- Design consent management system: granular opt-in/opt-out, purpose-specific consent, consent version tracking, and propagation to downstream systems
+- Implement data minimization controls: retention policies with automated deletion, purpose limitation enforcement, and necessity justification for data collection
+- Build privacy-preserving analytics: federated learning for ML without centralized data, secure aggregation for metrics, and homomorphic encryption for computation on encrypted data
+- Create data subject rights workflows: search across systems, export in portable format, deletion with verification, and rectification propagation
+
+### After implementation
+- Generate privacy compliance reports: PII inventory, consent coverage, retention policy enforcement, third-party data sharing audit, and rights request fulfillment SLA
+- Build privacy monitoring dashboards: PII exposure incidents, consent withdrawal rates, privacy budget consumption, and anonymization quality metrics
+- Create breach response procedures: detection, containment, notification timelines, affected user identification, and remediation workflows
+- Document privacy controls: anonymization methods, re-identification risk levels, consent mechanisms, and data retention justifications for audit purposes
+
+## Self-check before task completion
+- [ ] PII detection covers all data stores with automated scanning and alerting on new sensitive data discoveries
+- [ ] Anonymization techniques applied with documented re-identification risk assessment (k-anonymity ≥10 or equivalent)
+- [ ] Consent management enforces purpose limitation with propagation to all downstream processing systems
+- [ ] Differential privacy implementation maintains epsilon budget <1.0 for sensitive aggregations with privacy accounting
+- [ ] Data subject rights workflows tested for completeness across all systems within regulatory SLA (30 days GDPR)

package/.mindforge/skills/database-performance/SKILL.md

@@ -0,0 +1,174 @@
+---
+name: database-performance
+version: 1.0.0
+min_mindforge_version: 0.3.0
+status: stable
+triggers: database performance, query plan analysis, EXPLAIN ANALYZE, index selection, partition pruning, materialized view, query optimization, slow query, index strategy, table scan elimination, join optimization, query profiling
+compose: database-patterns
+---
+
+# Skill — Database Performance
+
+## When this skill activates
+Any task involving slow queries, query optimization, index strategy, EXPLAIN plan
+analysis, partitioning, materialized views, or database profiling.
+
+## Mandatory actions when this skill is active
+
+### Before optimizing
+1. Get the current query execution plan (EXPLAIN ANALYZE, not just EXPLAIN).
+2. Identify the actual bottleneck (do not guess).
+3. Measure baseline performance (p50, p95, p99 latency).
+4. Understand the data distribution (cardinality, skew).
+
+### Reading EXPLAIN ANALYZE output
+
+**Key things to look for:**
+
+| Signal | Meaning | Action |
+|--------|---------|--------|
+| Seq Scan on large table | Full table scan, no index used | Add appropriate index |
+| Nested Loop with high rows | O(n*m) join strategy | Consider Hash Join, add index on join column |
+| Actual rows >> Estimated rows | Stale statistics | Run ANALYZE on the table |
+| Sort with external merge | Not enough work_mem | Increase work_mem or add index for ORDER BY |
+| Filter removing most rows | Index not selective enough | Add more specific index or partial index |
+
+**Node types (best to worst for large tables):**
+1. Index Only Scan — best (reads from index, no table access).
+2. Index Scan — good (uses index, fetches rows from table).
+3. Bitmap Index Scan — okay (for medium selectivity).
+4. Seq Scan — bad on large tables (reads every row).
+
+### Index strategy
+
+**B-tree (default, most common):**
+- Equality: `WHERE status = 'active'`
+- Range: `WHERE created_at > '2025-01-01'`
+- Prefix matching: `WHERE name LIKE 'foo%'`
+- Sorting: `ORDER BY created_at DESC`
+- Composite: `(tenant_id, created_at)` — order matters, left-to-right.
+
+**GIN (Generalized Inverted Index):**
+- JSONB containment: `WHERE data @> '{"key": "value"}'`
+- Array contains: `WHERE tags @> ARRAY['tag1']`
+- Full-text search: `WHERE to_tsvector(body) @@ to_tsquery('search')`
+
+**Partial index (conditional):**
+- Index only rows that match a condition.
+- `CREATE INDEX idx_active_orders ON orders(created_at) WHERE status = 'active'`
+- Smaller, faster, less write overhead.
+
+**Expression index:**
+- Index a computed value.
+- `CREATE INDEX idx_lower_email ON users(LOWER(email))`
+- Query must use the same expression to hit the index.
+
+### Common query anti-patterns
+
+**Functions on indexed columns:**
+```sql
+-- BAD: index on created_at is useless
+WHERE EXTRACT(YEAR FROM created_at) = 2025
+
+-- GOOD: rewrite as range
+WHERE created_at >= '2025-01-01' AND created_at < '2026-01-01'
+```
+
+**OR conditions preventing index use:**
+```sql
+-- BAD: may cause Seq Scan
+WHERE status = 'active' OR status = 'pending'
+
+-- GOOD: use IN
+WHERE status IN ('active', 'pending')
+```
+
+**SELECT * when you need few columns:**
+```sql
+-- BAD: fetches all columns, prevents index-only scan
+SELECT * FROM orders WHERE tenant_id = 'abc'
+
+-- GOOD: select only needed columns
+SELECT id, status, total FROM orders WHERE tenant_id = 'abc'
+```
+
+**Missing LIMIT on unbounded queries:**
+```sql
+-- BAD: may return millions of rows
+SELECT * FROM events WHERE type = 'click'
+
+-- GOOD: always paginate
+SELECT * FROM events WHERE type = 'click' ORDER BY id LIMIT 50
+```
+
+### Materialized views
+
+**When to use:**
+- Expensive aggregations needed frequently (dashboards, reports).
+- Data changes infrequently relative to read frequency.
+- Acceptable staleness (refresh interval is tolerable).
+
+**Implementation:**
+```sql
+CREATE MATERIALIZED VIEW monthly_revenue AS
+SELECT tenant_id, date_trunc('month', created_at) AS month, SUM(amount) AS total
+FROM orders
+WHERE status = 'completed'
+GROUP BY tenant_id, month;
+
+-- Refresh on schedule
+REFRESH MATERIALIZED VIEW CONCURRENTLY monthly_revenue;
+```
+
+**Rules:**
+- Always use CONCURRENTLY (does not lock reads during refresh).
+- Add a unique index for CONCURRENTLY to work.
+- Monitor refresh duration — alert if it exceeds threshold.
+- Consider triggers for real-time materialized views (small tables only).
+
+### Partitioning
+
+**Range partitioning (time-series data):**
+```sql
+CREATE TABLE events (
+  id BIGINT, tenant_id UUID, created_at TIMESTAMPTZ, data JSONB
+) PARTITION BY RANGE (created_at);
+
+CREATE TABLE events_2025_01 PARTITION OF events
+  FOR VALUES FROM ('2025-01-01') TO ('2025-02-01');
+```
+
+**Benefits:**
+- Partition pruning: queries on created_at only scan relevant partitions.
+- Easy data lifecycle: DROP old partitions instead of DELETE (instant, no vacuum).
+- Parallel scan across partitions.
+
+**Hash partitioning (even distribution):**
+- For tables with no natural range key.
+- Distributes rows evenly across N partitions.
+- Good for very large tables that need parallel access.
+
+**Rules:**
+- Partition key must be in every query's WHERE clause for pruning.
+- Too many partitions (>1000) can slow planning.
+- Automate partition creation (don't rely on manual monthly creation).
+
+### Join optimization
+
+- Ensure join columns have indexes on both sides.
+- Small table JOIN large table: ensure small table is the "driving" table.
+- Consider denormalization if a join is on the critical path and never changes.
+- Use CTEs carefully — in PostgreSQL < 12, CTEs are optimization fences.
+
+### Monitoring
+
+- Enable `pg_stat_statements` for query-level statistics.
+- Alert on queries exceeding p95 threshold.
+- Track index usage: `pg_stat_user_indexes` — unused indexes waste write performance.
+- Regular VACUUM and ANALYZE (autovacuum tuning for high-write tables).
+
+## Self-check before task completion
+- [ ] Did I follow the mandatory actions for this skill?
+- [ ] Did I apply the patterns appropriate to the context?
+- [ ] Did I verify the implementation meets the criteria above?
+- [ ] Did I document decisions and trade-offs made?