@booklib/core 2.0.0

package/skills/data-intensive-patterns/references/review-checklist.md

@@ -0,0 +1,193 @@
+# Data-Intensive Applications Code Review Checklist
+
+Use this checklist when reviewing data-intensive application code. Work through each section
+and flag any violations. Not every section applies to every review — skip sections
+that aren't relevant to the code under review.
+
+---
+
+## 1. Data Modeling
+
+- [ ] Data model fits the application's access patterns (relational, document, graph, event log)
+- [ ] Relationships are modeled appropriately (joins vs. embedding vs. references)
+- [ ] Schema is explicit or schema-on-read strategy is intentional and documented
+- [ ] No impedance mismatch — application objects map cleanly to storage model
+- [ ] Normalization level is appropriate (not over-normalized for a document store, not under-normalized for relational)
+
+**Red flags**: Forcing graph-like traversals through a relational model with recursive joins.
+Storing deeply nested JSON in a relational column then parsing it in application code.
+Document model with many-to-many relationships handled by manual application-side joins.
+
+---
+
+## 2. Storage Engine and Indexing
+
+- [ ] Storage engine matches workload characteristics (write-heavy → LSM; read-heavy → B-tree)
+- [ ] Indexes exist for common query patterns
+- [ ] No unnecessary indexes (each index slows down writes)
+- [ ] Column-oriented storage used for analytical/OLAP workloads
+- [ ] Materialized views or data cubes used where pre-aggregation helps
+- [ ] Compaction strategy is configured appropriately for LSM-based stores
+
+**Red flags**: Full table scans on large tables due to missing indexes. Using a row-oriented
+store for analytical queries scanning millions of rows. Write-heavy workload on a database
+optimized for reads without considering LSM alternatives.
+
+---
+
+## 3. Encoding and Schema Evolution
+
+- [ ] Serialization format supports forward and backward compatibility
+- [ ] Schema registry is in place for Avro/Protobuf-encoded messages
+- [ ] Field tags (Protobuf) or schema resolution (Avro) used for evolution
+- [ ] Old and new code can run simultaneously during rolling deployments
+- [ ] No required fields added in a non-backward-compatible way
+- [ ] Deleted field tags/names are never reused
+
+**Red flags**: Using plain JSON for inter-service communication without versioning.
+Adding required fields to Protobuf definitions in production. Encoding changes that break
+consumers during rolling deployments. No schema registry for Kafka topics.
+
+---
+
+## 4. Replication
+
+- [ ] Replication topology matches consistency and availability requirements
+- [ ] Failover procedure is tested and documented
+- [ ] Replication lag is monitored and handled in application code
+- [ ] Read-after-write consistency is provided where needed (e.g., read from leader after write)
+- [ ] Split-brain protection exists (fencing tokens, epoch numbers)
+- [ ] For multi-leader: conflict resolution strategy is defined and tested
+- [ ] For leaderless: quorum parameters (w, r, n) are tuned for the workload
+
+**Red flags**: Async replication with no monitoring of replication lag. No split-brain protection
+during leader failover. Using LWW for conflict resolution in multi-leader setup where data loss
+is unacceptable. Quorum reads not configured (r + w ≤ n) giving inconsistent reads.
+
+---
+
+## 5. Partitioning
+
+- [ ] Partition key distributes load evenly (no hot partitions)
+- [ ] Partition strategy matches access patterns (key-range for scans, hash for uniform)
+- [ ] Cross-partition queries are minimized or explicitly handled
+- [ ] Secondary index strategy is chosen (local vs global) with trade-offs understood
+- [ ] Rebalancing approach is defined (fixed partitions, dynamic split, proportional to nodes)
+- [ ] Request routing is in place (client-side, routing tier, or coordinator)
+
+**Red flags**: Monotonically increasing keys (timestamps, auto-increment) used as hash partition
+key — all writes go to one partition. Range queries across hash-partitioned data. No plan
+for rebalancing when adding nodes. Scatter-gather queries hitting all partitions for every read.
+
+---
+
+## 6. Transactions and Concurrency
+
+- [ ] Isolation level is appropriate for the consistency requirements
+- [ ] Write skew scenarios are identified and mitigated
+- [ ] Phantom reads are prevented where needed (predicate/index-range locks or SSI)
+- [ ] Long-running transactions are avoided (hold locks briefly)
+- [ ] Deadlock detection or timeout is configured
+- [ ] Optimistic concurrency (CAS, version numbers) used where appropriate
+
+**Red flags**: Using READ COMMITTED where transactions read-then-write based on stale data
+(write skew). SERIALIZABLE isolation everywhere regardless of need (performance waste).
+Missing `SELECT ... FOR UPDATE` where concurrent updates can violate business rules.
+No retry logic for serialization failures under SSI.
+
+---
+
+## 7. Distributed Systems Resilience
+
+- [ ] All remote calls have timeouts configured
+- [ ] Retries use exponential backoff with jitter
+- [ ] Retry operations are idempotent (idempotency keys present)
+- [ ] Circuit breakers protect against cascading failures
+- [ ] Fencing tokens used for distributed locks/leases
+- [ ] No reliance on wall-clock timestamps for ordering across nodes
+- [ ] Network partitions are handled gracefully (not ignored)
+- [ ] Process pauses (GC, etc.) are accounted for in lease/lock design
+
+**Red flags**: HTTP calls without timeouts. Immediate retries without backoff (thundering herd).
+Using System.currentTimeMillis() for conflict resolution across nodes. Distributed locks
+without fencing tokens. Assuming clocks are synchronized across nodes.
+
+---
+
+## 8. Consensus and Coordination
+
+- [ ] Leader election uses a proper consensus protocol (not ad-hoc)
+- [ ] Coordination services (ZooKeeper/etcd) used for leader election and configuration
+- [ ] No hand-rolled consensus or distributed locking
+- [ ] 2PC is avoided for cross-service transactions (use sagas instead)
+- [ ] Uniqueness constraints across partitions use linearizable operations
+
+**Red flags**: Home-grown leader election using database timestamps. Two-phase commit across
+heterogeneous systems. Distributed lock implemented with Redis SET NX without fencing tokens
+or proper expiration handling. Assumption that ZooKeeper watches are instantaneous.
+
+---
+
+## 9. Batch and Stream Processing
+
+- [ ] Batch jobs are idempotent (safe to re-run)
+- [ ] Stream consumers are idempotent (safe to replay)
+- [ ] Exactly-once semantics achieved via idempotency, not by assumption
+- [ ] Processing output goes to a well-defined sink (not side effects scattered in operators)
+- [ ] Backpressure mechanism exists (consumers can signal producers to slow down)
+- [ ] Checkpointing or microbatching configured for stream fault tolerance
+- [ ] Late events / out-of-order events are handled (watermarks, allowed lateness)
+- [ ] Window semantics match business requirements (tumbling, hopping, sliding, session)
+
+**Red flags**: Stream consumer that crashes and loses all progress (no checkpointing).
+Batch job that partially writes output on failure (not atomic). Producer overwhelming consumer
+with no flow control. Using processing time instead of event time for time-sensitive analytics.
+No dead letter queue for malformed messages.
+
+---
+
+## 10. Derived Data and Integration
+
+- [ ] Derived data (caches, indexes, views) is maintained via events or CDC — not dual writes
+- [ ] Transactional outbox pattern used for reliable event publishing
+- [ ] Change Data Capture configured for keeping systems in sync
+- [ ] Event schema versioning strategy exists
+- [ ] Event consumers can bootstrap from scratch (initial snapshot + streaming)
+- [ ] Eventual consistency is acceptable and communicated to users appropriately
+
+**Red flags**: Application code that updates both the primary database and Elasticsearch in
+separate calls (dual write — can diverge on failure). No outbox pattern — events published after
+transaction commit (can be lost on crash). CDC consumer with no mechanism for initial snapshot.
+Derived views that can never be rebuilt from the event log.
+
+---
+
+## 11. Operational Readiness
+
+- [ ] Health check endpoints exist
+- [ ] Key metrics exposed: request rate, latency percentiles (p50, p95, p99), error rate
+- [ ] Distributed tracing instrumented (OpenTelemetry or equivalent)
+- [ ] Structured logging with correlation IDs
+- [ ] Alerts configured for critical failure conditions
+- [ ] Capacity planning considers tail latency (p99, not just averages)
+- [ ] Backpressure and graceful degradation strategies in place
+- [ ] Runbooks exist for common failure scenarios
+
+**Red flags**: Only monitoring averages (hides tail latency issues). No distributed tracing
+across service boundaries. Console.log as only observability. No runbook for leader failover
+or partition rebalancing. No capacity planning for data growth.
+
+---
+
+## Severity Classification
+
+When reporting issues, classify them:
+
+- **Critical**: Data loss risk, correctness issue, or security vulnerability
+  (e.g., dual writes without outbox, missing fencing tokens, no transaction isolation for invariants)
+- **Major**: Reliability or scalability debt that will cause problems at scale
+  (e.g., hot partitions, 2PC across services, no idempotency on retries, wrong storage engine)
+- **Minor**: Best practice deviation with limited immediate impact
+  (e.g., missing health check, no schema registry, suboptimal compaction settings)
+- **Suggestion**: Improvement that would be nice but isn't urgent
+  (e.g., consider CQRS for complex queries, evaluate column store for analytics workload)

package/skills/data-intensive-patterns/scripts/adr.py

@@ -0,0 +1,213 @@
+#!/usr/bin/env python3
+"""
+adr.py - Architecture Decision Record generator for data-intensive systems.
+
+Usage:
+    python adr.py <decision-title>
+    python adr.py                    # interactive mode
+
+Generates:
+    adr-NNN-<slug>.md   - Numbered ADR file with data-intensive-specific sections
+    ADR-INDEX.md        - Running index of all ADRs (appended to)
+
+The ADR includes standard sections plus four data-intensive-specific sections:
+    - Consistency model
+    - Failure mode
+    - Scalability impact
+    - Operability
+
+Based on patterns from "Designing Data-Intensive Applications" by Martin Kleppmann.
+"""
+
+import argparse
+import datetime
+import pathlib
+import re
+import sys
+
+
+def slugify(title: str) -> str:
+    slug = title.lower()
+    slug = re.sub(r"[^a-z0-9]+", "-", slug)
+    slug = slug.strip("-")
+    return slug
+
+
+def next_adr_number(adr_dir: pathlib.Path) -> int:
+    existing = list(adr_dir.glob("adr-[0-9][0-9][0-9]-*.md"))
+    if not existing:
+        return 1
+    numbers = []
+    for p in existing:
+        m = re.match(r"adr-(\d{3})-", p.name)
+        if m:
+            numbers.append(int(m.group(1)))
+    return max(numbers) + 1 if numbers else 1
+
+
+def prompt(question: str, default: str = "") -> str:
+    suffix = f" [{default}]" if default else ""
+    try:
+        answer = input(f"{question}{suffix}: ").strip()
+    except (EOFError, KeyboardInterrupt):
+        print()
+        sys.exit(0)
+    return answer if answer else default
+
+
+def collect_options() -> list[str]:
+    options = []
+    print("Enter up to 4 considered options (leave blank to stop):")
+    for i in range(1, 5):
+        opt = prompt(f"  Option {i}")
+        if not opt:
+            break
+        options.append(opt)
+    return options
+
+
+def render_adr(
+    number: int,
+    title: str,
+    context: str,
+    options: list[str],
+    chosen: str,
+    consequences: str,
+    consistency_model: str,
+    failure_mode: str,
+    scalability_impact: str,
+    operability: str,
+    date: str,
+) -> str:
+    options_text = "\n".join(f"- {opt}" for opt in options) if options else "- (none listed)"
+    return f"""\
+# ADR-{number:03d}: {title}
+
+**Date:** {date}
+**Status:** Proposed
+
+---
+
+## Context
+
+{context}
+
+## Considered Options
+
+{options_text}
+
+## Decision
+
+{chosen}
+
+## Consequences
+
+{consequences}
+
+---
+
+## Data-Intensive Considerations
+
+### Consistency Model
+
+> What consistency guarantees does this choice provide?
+
+{consistency_model}
+
+### Failure Mode
+
+> What happens when this component fails?
+
+{failure_mode}
+
+### Scalability Impact
+
+> How does this scale with data volume?
+
+{scalability_impact}
+
+### Operability
+
+> How observable and maintainable is this choice?
+
+{operability}
+"""
+
+
+def append_to_index(index_path: pathlib.Path, number: int, title: str, filename: str, date: str) -> None:
+    header = "# ADR Index\n\n| # | Title | Date | File |\n|---|-------|------|------|\n"
+    entry = f"| {number:03d} | {title} | {date} | [{filename}]({filename}) |\n"
+    if not index_path.exists():
+        index_path.write_text(header + entry, encoding="utf-8")
+        print(f"Created: {index_path}")
+    else:
+        content = index_path.read_text(encoding="utf-8")
+        index_path.write_text(content + entry, encoding="utf-8")
+        print(f"Updated: {index_path}")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Generate an ADR for data-intensive systems."
+    )
+    parser.add_argument(
+        "title",
+        nargs="?",
+        default="",
+        help="Decision title (will prompt if omitted)",
+    )
+    parser.add_argument(
+        "--output-dir",
+        default=".",
+        help="Directory to write ADR files (default: ./)",
+    )
+    args = parser.parse_args()
+
+    output_dir = pathlib.Path(args.output_dir).resolve()
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    title = args.title.strip() or prompt("Decision title")
+    if not title:
+        print("ERROR: A title is required.")
+        sys.exit(1)
+
+    print()
+    context = prompt("Context (why is this decision needed?)", default="Describe the situation and forces at play.")
+    options = collect_options()
+    chosen = prompt("Chosen option")
+    consequences = prompt("Consequences (trade-offs, risks, next steps)", default="To be determined.")
+    print()
+    print("-- Data-intensive sections --")
+    consistency_model = prompt("Consistency model", default="To be defined.")
+    failure_mode = prompt("Failure mode", default="To be defined.")
+    scalability_impact = prompt("Scalability impact", default="To be defined.")
+    operability = prompt("Operability", default="To be defined.")
+
+    number = next_adr_number(output_dir)
+    date = datetime.date.today().isoformat()
+    filename = f"adr-{number:03d}-{slugify(title)}.md"
+    adr_path = output_dir / filename
+
+    content = render_adr(
+        number=number,
+        title=title,
+        context=context,
+        options=options,
+        chosen=chosen,
+        consequences=consequences,
+        consistency_model=consistency_model,
+        failure_mode=failure_mode,
+        scalability_impact=scalability_impact,
+        operability=operability,
+        date=date,
+    )
+
+    adr_path.write_text(content, encoding="utf-8")
+    print(f"\nWrote: {adr_path}")
+
+    append_to_index(output_dir / "ADR-INDEX.md", number, title, filename, date)
+    print("\nDone.")
+
+
+if __name__ == "__main__":
+    main()

package/skills/data-intensive-patterns/scripts/example.py

@@ -0,0 +1 @@
+

package/skills/data-pipelines/SKILL.md

@@ -0,0 +1,259 @@
+---
+name: data-pipelines
+version: "1.0"
+license: MIT
+tags: [data, etl, pipelines, python]
+description: >
+  Apply Data Pipelines Pocket Reference practices (James Densmore). Covers
+  Infrastructure (Ch 1-2: warehouses, lakes, cloud), Patterns (Ch 3: ETL, ELT,
+  CDC), DB Ingestion (Ch 4: MySQL, PostgreSQL, MongoDB, full/incremental),
+  File Ingestion (Ch 5: CSV, JSON, cloud storage), API Ingestion (Ch 6: REST,
+  pagination, rate limiting), Streaming (Ch 7: Kafka, Kinesis, event-driven),
+  Storage (Ch 8: Redshift, BigQuery, Snowflake), Transforms (Ch 9: SQL, Python,
+  dbt), Validation (Ch 10: Great Expectations, schema checks), Orchestration
+  (Ch 11: Airflow, DAGs, scheduling), Monitoring (Ch 12: SLAs, alerting),
+  Best Practices (Ch 13: idempotency, backfilling, error handling). Trigger on
+  "data pipeline", "ETL", "ELT", "data ingestion", "Airflow", "dbt",
+  "data warehouse", "Kafka streaming", "CDC", "data orchestration".
+---
+
+# Data Pipelines Pocket Reference Skill
+
+You are an expert data engineer grounded in the 13 chapters from
+*Data Pipelines Pocket Reference* (Moving and Processing Data for Analytics)
+by James Densmore. You help developers and data engineers in two modes:
+
+1. **Pipeline Building** — Design and implement data pipelines with idiomatic, production-ready patterns
+2. **Pipeline Review** — Analyze existing pipelines against the book's practices and recommend improvements
+
+## How to Decide Which Mode
+
+- If the user asks you to *build*, *create*, *design*, *implement*, *write*, or *set up* a pipeline → **Pipeline Building**
+- If the user asks you to *review*, *audit*, *improve*, *troubleshoot*, *optimize*, or *analyze* a pipeline → **Pipeline Review**
+- If ambiguous, ask briefly which mode they'd prefer
+
+---
+
+## Mode 1: Pipeline Building
+
+When designing or building data pipelines, follow this decision flow:
+
+### Step 1 — Understand the Requirements
+
+Ask (or infer from context):
+
+- **What data source?** — Database (MySQL, PostgreSQL, MongoDB), files (CSV, JSON, cloud storage), API (REST), streaming (Kafka, Kinesis)?
+- **What destination?** — Data warehouse (Redshift, BigQuery, Snowflake), data lake (S3, GCS), operational database?
+- **What pattern?** — ETL, ELT, CDC, streaming, batch?
+- **What scale?** — Volume, velocity, variety of data? SLA requirements?
+
+### Step 2 — Apply the Right Practices
+
+Read `references/practices-catalog.md` for the full chapter-by-chapter catalog. Quick decision guide by concern:
+
+| Concern | Chapters to Apply |
+|---------|-------------------|
+| Infrastructure and architecture | Ch 1-2: Pipeline types, data warehouses vs data lakes, cloud storage (S3, GCS, Azure Blob), choosing infrastructure |
+| Pipeline patterns and design | Ch 3: ETL vs ELT, change data capture (CDC), full vs incremental extraction, append vs upsert loading |
+| Database ingestion | Ch 4: MySQL/PostgreSQL/MongoDB extraction, full and incremental loads, connection pooling, binary log replication |
+| File-based ingestion | Ch 5: CSV/JSON/flat file parsing, cloud storage integration, file naming conventions, schema detection |
+| API ingestion | Ch 6: REST API extraction, pagination handling, rate limiting, authentication, retry logic, webhook ingestion |
+| Streaming data | Ch 7: Kafka producers/consumers, Kinesis streams, event-driven pipelines, exactly-once semantics, stream processing |
+| Data storage and loading | Ch 8: Warehouse loading patterns (Redshift COPY, BigQuery load, Snowflake stages), partitioning, clustering |
+| Transformations | Ch 9: SQL-based transforms, Python transforms, dbt models, staging/intermediate/mart layers, incremental models |
+| Data validation and testing | Ch 10: Schema validation, data quality checks, Great Expectations, row counts, null checks, referential integrity |
+| Orchestration | Ch 11: Apache Airflow, DAG design, task dependencies, scheduling, sensors, XComs, idempotent tasks |
+| Monitoring and alerting | Ch 12: Pipeline health metrics, SLA tracking, data freshness, logging, alerting strategies, anomaly detection |
+| Best practices | Ch 13: Idempotency, backfilling, error handling, retry strategies, data lineage, documentation |
+
+### Step 3 — Follow Data Pipeline Principles
+
+Every pipeline implementation should honor these principles:
+
+1. **Idempotency always** — Running a pipeline multiple times with the same input produces the same result; use DELETE+INSERT or MERGE patterns
+2. **Incremental over full** — Prefer incremental extraction using timestamps or CDC over full table scans when data volume grows
+3. **ELT over ETL for analytics** — Load raw data into the warehouse first, transform with SQL/dbt; leverage warehouse compute power
+4. **Schema evolution readiness** — Design pipelines to handle schema changes gracefully; use schema detection and validation
+5. **Atomicity in loading** — Use staging tables, transactions, and atomic swaps; never leave destinations in partial states
+6. **Orchestration for dependencies** — Use DAGs (Airflow) to manage task ordering, retries, and failure handling; avoid time-based chaining
+7. **Validate early and often** — Check data quality at ingestion, after transformation, and before serving; use automated assertion frameworks
+8. **Monitor everything** — Track row counts, data freshness, pipeline duration, error rates; alert on SLA breaches
+9. **Design for backfilling** — Parameterize pipelines by date range; make it easy to reprocess historical data
+10. **Document data lineage** — Track where data comes from, how it's transformed, and where it goes; maintain a data catalog
+
+### Step 4 — Build the Pipeline
+
+Follow these guidelines:
+
+- **Production-ready** — Include error handling, retries, logging, monitoring from the start
+- **Configurable** — Externalize connection strings, credentials, date ranges, batch sizes; use environment variables or config files
+- **Testable** — Write unit tests for transformations, integration tests for end-to-end flows
+- **Observable** — Include logging at each stage, metrics collection, alerting hooks
+- **Documented** — README, data dictionary, DAG documentation, runbook for common failures
+
+When building pipelines, produce:
+
+1. **Pattern identification** — Which chapters/concepts apply and why
+2. **Architecture diagram** — Source → Ingestion → Storage → Transform → Serve flow
+3. **Implementation** — Production-ready code with error handling
+4. **Configuration** — Connection configs, scheduling, environment setup
+5. **Monitoring setup** — What to track and alert on
+
+### Pipeline Building Examples
+
+**Example 1 — Database to Warehouse ETL:**
+```
+User: "Create a pipeline to sync MySQL orders to BigQuery"
+
+Apply: Ch 3 (incremental extraction), Ch 4 (MySQL ingestion), Ch 8 (BigQuery loading),
+       Ch 11 (Airflow orchestration), Ch 13 (idempotency)
+
+Generate:
+- Incremental extraction using updated_at timestamp
+- Staging table load with BigQuery load jobs
+- MERGE/upsert into final table for idempotency
+- Airflow DAG with proper scheduling and error handling
+- Row count validation between source and destination
+```
+
+**Example 2 — REST API Ingestion Pipeline:**
+```
+User: "Build a pipeline to ingest data from a paginated REST API"
+
+Apply: Ch 6 (API ingestion, pagination, rate limiting), Ch 5 (JSON handling),
+       Ch 8 (warehouse loading), Ch 10 (validation)
+
+Generate:
+- Paginated API client with retry logic and rate limiting
+- JSON response parsing and flattening
+- Incremental loading with cursor-based pagination
+- Schema validation on ingested records
+- Error handling for API failures and timeouts
+```
+
+**Example 3 — Streaming Pipeline:**
+```
+User: "Set up a Kafka-based streaming pipeline for event data"
+
+Apply: Ch 7 (Kafka, event-driven), Ch 8 (warehouse loading),
+       Ch 12 (monitoring), Ch 13 (exactly-once semantics)
+
+Generate:
+- Kafka consumer group configuration
+- Event deserialization and validation
+- Micro-batch or streaming sink to warehouse
+- Dead letter queue for failed events
+- Consumer lag monitoring and alerting
+```
+
+**Example 4 — dbt Transformation Layer:**
+```
+User: "Create a dbt project for transforming raw e-commerce data"
+
+Apply: Ch 9 (dbt, SQL transforms, staging/mart layers),
+       Ch 10 (data testing), Ch 13 (incremental models)
+
+Generate:
+- Staging models (1:1 with source, renamed/typed)
+- Intermediate models (business logic joins)
+- Mart models (final analytics tables)
+- dbt tests (not_null, unique, relationships, custom)
+- Incremental model configuration with merge strategy
+```
+
+---
+
+## Mode 2: Pipeline Review
+
+When reviewing data pipelines, read `references/review-checklist.md` for the full checklist.
+
+### Review Process
+
+1. **Architecture scan** — Check Ch 1-3: pipeline pattern choice (ETL/ELT/CDC), infrastructure fit, data flow design
+2. **Ingestion scan** — Check Ch 4-7: extraction method, incremental vs full, error handling, source-specific best practices
+3. **Storage scan** — Check Ch 8: loading patterns, partitioning, clustering, staging table usage, atomic loads
+4. **Transform scan** — Check Ch 9: SQL vs Python choice, dbt patterns, layer structure, incremental models
+5. **Quality scan** — Check Ch 10: validation coverage, schema checks, data quality assertions, testing
+6. **Orchestration scan** — Check Ch 11: DAG design, task granularity, dependency management, idempotency
+7. **Operations scan** — Check Ch 12-13: monitoring, alerting, backfill capability, error handling, documentation
+
+### Calibrating Review Tone — Well-Designed vs. Problematic Pipelines
+
+**Before listing issues, assess overall quality:**
+
+- If the pipeline already implements idempotency, incremental extraction, separation of concerns, retry logic, structured logging, and lineage tracking — say so explicitly and lead with praise.
+- **Do NOT manufacture problems** to appear thorough. If a pattern is correct, praise it. Only flag genuine gaps.
+- Frame truly optional improvements as "minor" or "nice-to-have," not "Critical" or "will cause real pain in production."
+- A well-designed pipeline deserves a review that opens with "This is a well-designed pipeline" and highlights what it does right before any suggestions.
+
+**Specific patterns to recognize and praise when present:**
+
+- **ETL function separation** — `extract`, `transform`, `load` as distinct single-responsibility functions (Ch 3: ETL pattern, Ch 11: task granularity) → Praise explicitly.
+- **Generator/batch extraction** — `yield`-based extraction that streams rows in batches rather than fetching everything into memory (Ch 4: streaming extraction, memory efficiency) → Praise explicitly; do NOT suggest it is broken.
+- **Watermark-based incremental extraction** — filtering by timestamp/cursor to avoid full-table scans on reruns (Ch 3-4) → Praise explicitly.
+- **Upsert / ON CONFLICT DO UPDATE** — ensures idempotency and safe reruns (Ch 13) → Praise explicitly.
+- **Retry with exponential backoff** — `run_with_retry` wrappers for transient errors (Ch 13) → Praise explicitly.
+- **Structured logging with row counts** — batch-level `logger.info` with row counts already present (Ch 12: monitoring) → Praise it; do NOT suggest adding logging that already exists.
+- **pipeline_run_id / audit column** — tracking which pipeline run produced each row (Ch 13: data lineage) → Praise explicitly.
+
+### Review Output Format
+
+Structure your review as:
+
+```
+## Summary
+One paragraph: overall pipeline quality, pattern adherence, main concerns.
+If the pipeline is well-designed, say so clearly upfront.
+
+## Strengths
+For each good pattern found:
+- **Pattern**: name and chapter reference
+- **Where**: location in the pipeline
+- **Why it matters**: brief explanation
+
+## Issues
+For each genuine issue found:
+- **Topic**: chapter and concept
+- **Location**: where in the pipeline
+- **Problem**: what's wrong
+- **Fix**: recommended change with code/config snippet
+- **Severity**: Critical / High / Minor (only use Critical or High for real production risks)
+
+## Recommendations
+Priority-ordered list. Frame genuinely minor items as "nice-to-have" or "minor."
+Each recommendation references the specific chapter/concept.
+If no significant issues exist, say so — a short list of minor suggestions is fine.
+```
+
+### Common Data Pipeline Anti-Patterns to Flag
+
+- **Full extraction when incremental suffices** → Ch 3-4: Use timestamp/CDC-based incremental extraction for growing tables
+- **No idempotency** → Ch 13: Pipelines should produce same results when re-run; use DELETE+INSERT or MERGE
+- **Transforming before loading (unnecessary ETL)** → Ch 3: Use ELT pattern; load raw data first, transform in warehouse
+- **No staging tables** → Ch 8: Always load to staging first, validate, then swap/merge to production
+- **Hardcoded credentials** → Ch 13: Use environment variables, secrets managers, or config files
+- **No error handling or retries** → Ch 6, 13: Implement retry logic with exponential backoff for transient failures
+- **Time-based dependencies** → Ch 11: Use DAG-based orchestration (Airflow) instead of cron with time buffers
+- **Missing data validation** → Ch 10: Add row count checks, null checks, schema validation, freshness checks
+- **No monitoring or alerting** → Ch 12: Track pipeline duration, row counts, error rates; alert on SLA breaches
+- **Monolithic pipelines** → Ch 11: Break into small, reusable, testable tasks in a DAG
+- **No backfill support** → Ch 13: Parameterize pipelines by date range; make historical reprocessing easy
+- **Ignoring schema evolution** → Ch 5, 10: Handle new columns, type changes, missing fields gracefully
+- **Unpartitioned warehouse tables** → Ch 8: Partition by date/key for query performance and cost
+- **No data lineage** → Ch 13: Document source-to-destination mappings and transformation logic
+- **Blocking on API rate limits** → Ch 6: Implement rate limit awareness with backoff and queuing
+- **Missing dead letter queues** → Ch 7: Capture failed events/records for inspection and reprocessing
+- **Over-orchestrating** → Ch 11: Not every script needs Airflow; match orchestration complexity to pipeline needs
+
+---
+
+## General Guidelines
+
+- **ELT for analytics, ETL for operational** — Use warehouse compute for analytics transforms; use ETL only when destination can't transform
+- **Incremental by default** — Start with incremental extraction; fall back to full only when necessary
+- **Idempotency is non-negotiable** — Every pipeline must be safely re-runnable without data duplication or corruption
+- **Validate at boundaries** — Check data quality at ingestion, after transformation, and before serving
+- **Orchestrate with DAGs** — Use Airflow or similar tools for dependency management, retries, and scheduling
+- **Monitor proactively** — Don't wait for users to report stale data; alert on freshness, completeness, and accuracy
+- For deeper practice details, read `references/practices-catalog.md` before building pipelines.
+- For review checklists, read `references/review-checklist.md` before reviewing pipelines.

package/skills/data-pipelines/assets/example_asset.txt

@@ -0,0 +1 @@
+