@djm204/agent-skills 1.0.0

package/templates/engineering/data-engineering/.cursor/rules/data-quality.mdc

@@ -0,0 +1,78 @@
+---
+description: Data Quality
+alwaysApply: false
+---
+
+# Data Quality
+
+Patterns for validating, monitoring, and ensuring data quality.
+
+## Quality Dimensions
+
+- **Completeness** — required data is present (no unexpected nulls)
+- **Accuracy** — data reflects reality (values in expected ranges)
+- **Consistency** — data agrees across systems (totals match line items)
+- **Timeliness** — data is fresh enough (within SLA)
+- **Uniqueness** — no unwanted duplicates (primary keys are unique)
+- **Validity** — data conforms to business rules (email matches regex)
+
+## Validation Patterns
+
+- **Schema validation** — enforce expected column names, types, and nullability before processing
+- **Null checks** — assert zero nulls in required columns; fail pipeline on violation
+- **Uniqueness checks** — `groupBy(keys).count().filter("count > 1")` must be empty
+- **Range checks** — numeric values within `[min, max]`; flag or reject out-of-range
+- **Referential integrity** — left-anti join to find orphan foreign keys
+
+```python
+def check_required_columns(df, required):
+    for col in required:
+        if df.filter(F.col(col).isNull()).count() > 0:
+            raise DataQualityError(f"Nulls in required column: {col}")
+    return df
+```
+
+## Severity Levels
+
+- **Critical** — pipeline must fail (null PKs, duplicates on unique keys)
+- **Warning** — log and continue (invalid emails, future dates); emit metrics for trending
+
+## Freshness Monitoring
+
+- Define SLA per table (e.g., orders: 2h, daily_sales: 24h)
+- Compare `MAX(timestamp_column)` against current time
+- Alert on SLA breaches with severity matching business impact
+
+## Anomaly Detection
+
+- **Volume** — compare today's record count against rolling 30-day mean ± 3 std devs
+- **Distribution drift** — detect shifts in categorical value distributions (KL divergence)
+- Zero-count partitions or sudden spikes both warrant alerts
+
+## Quarantine Pattern
+
+Separate good and bad records; route failures to a quarantine table for investigation.
+
+```python
+good = df.filter("order_id IS NOT NULL AND total_amount >= 0")
+bad = df.subtract(good)
+if bad.count() > 0:
+    bad.withColumn("_reason", F.lit("validation_failed")) \
+       .write.mode("append").saveAsTable("quarantine.orders")
+```
+
+## DBT Tests
+
+```yaml
+columns:
+  - name: order_id
+    tests: [not_null, unique]
+  - name: status
+    tests: [{accepted_values: {values: [pending, confirmed, shipped]}}]
+```
+
+## Anti-Patterns
+
+- Silently dropping bad records without logging or quarantine
+- Only validating in development; skipping checks in production
+- No freshness monitoring — stale data serves silently

package/templates/engineering/data-engineering/.cursor/rules/overview.mdc

@@ -0,0 +1,49 @@
+---
+description: Data Engineering
+alwaysApply: false
+---
+
+# Data Engineering
+
+Guidelines for building robust, scalable data platforms and pipelines.
+
+## Scope
+
+- Batch pipelines (ETL/ELT) and stream processing
+- Data warehouses, lakehouses, analytics engineering (DBT)
+- Data quality and observability systems
+
+## Key Principles
+
+- **Idempotency is non-negotiable** — every pipeline must produce identical results on re-run
+- **Data quality is a feature** — validate early, monitor continuously, alert proactively
+- **Schema is a contract** — breaking changes require coordination and versioning
+- **Observability over debugging** — instrument everything; never debug production with ad-hoc queries
+- **Cost-aware engineering** — compute and storage have real costs; optimize deliberately
+
+## Data Architecture Layers
+
+- **Bronze/Raw** — exact copy of source, append-only (minutes-hours freshness)
+- **Silver/Curated** — validated, typed, deduplicated (hours freshness)
+- **Gold/Marts** — business-ready aggregates (hours-daily freshness)
+
+## Pipeline Design Essentials
+
+- Delete-then-insert or merge for idempotency — never blind append
+- Use `execution_date`, not `current_timestamp()` for determinism
+- Parameterize source/target tables and dates for testability
+- Support backfill via parameterized execution dates
+- Implement dead letter queues for invalid records
+
+## Anti-Patterns
+
+- Appending without clearing target partition (creates duplicates on re-run)
+- Processing only today's data without a late-arrival window
+- Hardcoded table names and connection strings in pipeline code
+
+## Definition of Done
+
+- [ ] Idempotency verified (re-run produces same result)
+- [ ] Data quality checks implemented and passing
+- [ ] Schema documented; monitoring and alerting configured
+- [ ] Cost estimate understood

package/templates/engineering/data-engineering/.cursor/rules/performance.mdc

@@ -0,0 +1,71 @@
+---
+description: Performance Optimization
+alwaysApply: false
+---
+
+# Performance Optimization
+
+Patterns for efficient, cost-effective data pipelines.
+
+## Partitioning
+
+- Partition by columns used in most query filters (usually date)
+- Target 100K–10M records per partition; avoid over-partitioning
+- Compact small files: `OPTIMIZE table` and set `maxRecordsPerFile`
+
+```python
+# Good: single partition column matching query patterns
+df.write.partitionBy("order_date").saveAsTable("curated.orders")
+
+# Bad: over-partitioning creates millions of tiny files
+df.write.partitionBy("order_date", "customer_id", "product_id").saveAsTable(t)
+```
+
+## Query Optimization
+
+- **Predicate pushdown** — filter on native columns; UDFs block pushdown (full scan)
+- **Column pruning** — `.select()` needed columns before transforms, not after
+- **Broadcast joins** — `broadcast(small_df)` for dimension tables (<100MB)
+- **Aggregate before join** — reduce shuffle by aggregating first, joining second
+- **Avoid unnecessary sorts** — only `orderBy` when needed; `limit` first when possible
+
+```python
+# Good: native filter pushes to storage
+df.filter(F.col("order_date") > F.date_sub(F.current_date(), 7))
+
+# Bad: UDF blocks pushdown, causes full scan
+df.filter(is_recent_udf(F.col("order_date")))
+```
+
+## Caching
+
+- Cache intermediate DataFrames reused multiple times
+- Always `unpersist()` in a `finally` block
+- Use `MEMORY_AND_DISK` for large datasets that may spill
+
+## Shuffle Optimization
+
+- Aggregate before join to reduce shuffle volume
+- `coalesce(n)` before write (no shuffle); `repartition(n)` only when increasing parallelism
+- Tune `spark.sql.shuffle.partitions` (default 200; use `"auto"` on Spark 3.0+)
+
+## Z-Ordering (Delta Lake)
+
+Co-locate data for multi-column filter queries:
+
+```sql
+OPTIMIZE curated.orders ZORDER BY (customer_id, product_id)
+```
+
+## Cost Management
+
+- Use incremental processing; avoid full-refresh when unnecessary
+- Apply data retention: `DELETE` + `VACUUM` old partitions
+- Monitor pipeline compute costs and bytes scanned weekly
+- Prefer `zstd` compression for better ratio
+
+## Anti-Patterns
+
+- Filtering on derived columns that prevent partition pruning
+- Reading all columns then selecting late in the pipeline
+- Caching without cleanup — memory leaks across jobs

package/templates/engineering/data-engineering/.cursor/rules/pipeline-design.mdc

@@ -0,0 +1,79 @@
+---
+description: Pipeline Design
+alwaysApply: false
+---
+
+# Pipeline Design
+
+Patterns for building reliable data pipelines.
+
+## Core Principles
+
+- **Idempotency** — same inputs, same outputs, every time
+- **Determinism** — use `execution_date`, not `current_timestamp()`; seed random operations
+- **Atomicity** — all-or-nothing writes; use Delta/Iceberg transactions or staging+swap
+
+## Pipeline Patterns
+
+### Batch Full Refresh
+Use when source is small or doesn't support incremental.
+
+```python
+df = spark.read.table(source)
+df.write.mode("overwrite").saveAsTable(target)
+```
+
+### Batch Incremental (Watermark)
+Use when data is large and source supports change tracking.
+
+```python
+last_wm = get_watermark(target, wm_col)
+new = spark.read.table(source).filter(F.col(wm_col) > last_wm)
+DeltaTable.forName(spark, target).alias("t") \
+    .merge(new.alias("s"), "t.id = s.id") \
+    .whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()
+```
+
+### CDC (Change Data Capture)
+Use when tracking all changes for point-in-time queries.
+
+```python
+DeltaTable.forName(spark, target).alias("t") \
+    .merge(cdc_events.alias("s"), "t.id = s.id") \
+    .whenMatchedDelete(condition="s.op = 'DELETE'") \
+    .whenMatchedUpdateAll(condition="s.op = 'UPDATE'") \
+    .whenNotMatchedInsertAll(condition="s.op = 'INSERT'").execute()
+```
+
+### Streaming
+Use when low latency required from event streams.
+
+```python
+spark.readStream.format("kafka").option("subscribe", topic).load() \
+    .transform(process) \
+    .writeStream.format("delta") \
+    .option("checkpointLocation", ckpt).toTable(target)
+```
+
+## Late Data Handling
+
+- **Batch**: reprocess a rolling window (e.g., last 3 days) to catch late arrivals
+- **Streaming**: use `.withWatermark("event_time", "1 hour")` to define late-data tolerance
+
+## Orchestration
+
+- Design DAGs with explicit data dependencies between tasks
+- Retry with exponential backoff: 3 retries, 5min initial, 1hr max
+- Support backfill via parameterized `execution_date`
+
+## Error Handling
+
+- **Fail fast** — validate critical assumptions (nulls, empty input) before transforms
+- **Dead letter queue** — route invalid records to a DLQ table for investigation
+- Log record counts at each stage; emit metrics for monitoring
+
+## Anti-Patterns
+
+- Blind append without partition clearing (duplicates on re-run)
+- Hidden dependencies via side effects instead of explicit table reads
+- Hardcoded table names instead of parameterized source/target

package/templates/engineering/data-engineering/.cursor/rules/security.mdc

@@ -0,0 +1,79 @@
+---
+description: Security & Governance
+alwaysApply: false
+---
+
+# Security & Governance
+
+Patterns for securing data and maintaining compliance.
+
+## Data Classification
+
+- **Public** — shareable externally; no special controls
+- **Internal** — business data; requires authentication
+- **Confidential** — sensitive business data; encryption + access control
+- **Restricted** — PII/regulated data; encryption, masking, audit logging required
+
+Tag tables with classification metadata (`data_classification`, `contains_pii`, `data_owner`).
+
+## PII Handling
+
+Strategies by sensitivity:
+- **Hash** (SHA-256) — one-way, for matching without exposing values
+- **Encrypt** (Fernet) — reversible, for authorized access
+- **Mask** — partial display (`jo****@email.com`)
+- **Redact** — replace with `[REDACTED]`
+
+```python
+def mask_pii(df, strategy):
+    for col, method in strategy.items():
+        if col not in df.columns: continue
+        if method == "hash": df = df.withColumn(col, F.sha2(F.col(col), 256))
+        elif method == "redact": df = df.withColumn(col, F.lit("[REDACTED]"))
+    return df
+```
+
+## Access Control
+
+- **RBAC** — create roles (analyst, engineer, admin); grant minimal permissions per role
+- **Column-level** — create masked views for non-privileged users; grant view access, not table access
+- **Row-level** — filter rows in views by user attributes (e.g., region)
+- **Least privilege** — default to restricted; require approval for elevation
+
+```sql
+-- Analysts get masked view, not raw table
+GRANT SELECT ON curated.customers_masked TO data_analyst;
+```
+
+## Audit Logging
+
+- Log all data access: user, table, operation, row count, timestamp
+- Log schema changes: before/after schema, change type, user
+- Append-only audit tables — never delete audit records
+- Review access patterns for anomalies (unusual volume, off-hours access)
+
+## Data Retention
+
+- Define retention policy per layer (raw: 90d, curated: 3yr, audit: 7yr)
+- Archive before delete when required
+- Support GDPR right-to-erasure: delete by customer_id across all tables, log deletion
+
+## Secrets Management
+
+- Never hardcode credentials — use environment variables or secrets managers
+- Support credential rotation without downtime
+- Re-encrypt data when rotating encryption keys
+
+```python
+# Bad: hardcoded
+conn = "postgresql://user:pass@host/db"
+
+# Good: from environment or secrets manager
+conn = f"postgresql://{os.environ['DB_USER']}:{os.environ['DB_PASS']}@{os.environ['DB_HOST']}/db"
+```
+
+## Anti-Patterns
+
+- Granting `ALL PRIVILEGES` to broad groups instead of minimal role-based access
+- Storing PII unencrypted in bronze/raw layer without masking downstream
+- No audit logging on sensitive table access

package/templates/engineering/data-engineering/.cursor/rules/testing.mdc

@@ -0,0 +1,75 @@
+---
+description: Data Pipeline Testing
+alwaysApply: false
+---
+
+# Data Pipeline Testing
+
+Strategies for testing data pipelines effectively.
+
+## Testing Pyramid
+
+- **Unit** — single transformation, fast, test logic in isolation
+- **Integration** — end-to-end pipeline flow, medium speed
+- **Contract** — schema/interface compatibility, prevent breaking changes
+- **Data Quality** — validate production data, slowest
+
+## Unit Tests
+
+Test individual transformations with small, controlled DataFrames.
+
+```python
+def test_calculate_order_total(spark):
+    df = spark.createDataFrame([{"order_id": "1", "quantity": 2, "unit_price": Decimal("10.00")}])
+    assert calculate_order_total(df).collect()[0]["total"] == Decimal("20.00")
+```
+
+## Integration Tests
+
+Test full pipeline flows in isolated test databases. Use fixtures to create/drop DBs.
+
+```python
+def test_idempotency(spark, test_db):
+    for _ in range(2):
+        run_pipeline(source=f"{test_db}.raw", target=f"{test_db}.curated", date=date(2024,1,15))
+    assert spark.table(f"{test_db}.curated").count() == 3  # Not doubled
+```
+
+## Contract Tests
+
+Ensure schema backward compatibility for downstream consumers.
+
+```python
+def test_schema_backward_compatible():
+    schema = spark.table("curated.orders").schema
+    required = {"order_id": StringType(), "total_amount": DecimalType(12,2)}
+    for col, expected_type in required.items():
+        assert col in [f.name for f in schema.fields]
+        assert schema[col].dataType == expected_type
+```
+
+## DBT Tests
+
+```yaml
+models:
+  - name: orders
+    columns:
+      - name: order_id
+        tests: [not_null, unique]
+      - name: customer_id
+        tests: [not_null, {relationships: {to: ref('customers'), field: customer_id}}]
+```
+
+## Best Practices
+
+- Test behavior, not implementation (test WHAT, not HOW)
+- Each test sets up its own data — no shared mutable state
+- Use descriptive names: `test_calculate_total_handles_zero_quantity`
+- Test edge cases: nulls, empty inputs, duplicates, large values
+- Use `assert_df_equality` (chispa) for DataFrame comparisons
+
+## Anti-Patterns
+
+- Only testing the happy path; skipping null/empty/duplicate cases
+- Tests depending on shared state or execution order
+- Testing implementation details (e.g., checking query plans)