@booklib/core 2.0.0

package/skills/data-pipelines/evals/evals.json

@@ -0,0 +1,45 @@
+{
+  "evals": [
+    {
+      "id": "eval-01-etl-no-error-handling-no-idempotency",
+      "prompt": "Review this ETL script:\n\n```python\nimport psycopg2\nimport requests\n\nSOURCE_DB = 'postgresql://user:pass@source-host/prod'\nDEST_DB = 'postgresql://user:pass@warehouse-host/warehouse'\n\ndef run():\n    src = psycopg2.connect(SOURCE_DB)\n    dst = psycopg2.connect(DEST_DB)\n\n    rows = src.cursor().execute(\n        'SELECT id, customer_id, amount, created_at FROM orders'\n    ).fetchall()\n\n    for row in rows:\n        order_id, customer_id, amount, created_at = row\n        resp = requests.get(f'https://api.exchange.io/rate?currency=EUR')\n        rate = resp.json()['rate']\n        amount_eur = amount * rate\n\n        dst.cursor().execute(\n            'INSERT INTO orders_eur VALUES (%s, %s, %s, %s)',\n            (order_id, customer_id, amount_eur, created_at)\n        )\n\n    dst.commit()\n    src.close()\n    dst.close()\n\nif __name__ == '__main__':\n    run()\n```",
+      "expectations": [
+        "Flags the full-table extraction `SELECT * FROM orders` with no timestamp filter as a non-incremental load that will re-process the entire table on every run; recommends incremental extraction using a watermark (Ch 3-4: incremental over full extraction)",
+        "Flags the absence of idempotency: re-running the script will insert duplicate rows into `orders_eur`; recommends an INSERT ... ON CONFLICT DO NOTHING or MERGE pattern (Ch 13: idempotency is non-negotiable)",
+        "Flags the external API call `requests.get` inside the per-row loop, which issues one HTTP request per order row — an N+1 pattern causing severe performance and rate-limit issues; recommends fetching the exchange rate once before the loop",
+        "Flags no error handling anywhere: if the API call fails, the loop crashes mid-run leaving the destination in a partially loaded state with no indication of progress (Ch 13: error handling and retry strategies)",
+        "Flags hardcoded credentials in source strings; recommends environment variables or a secrets manager (Ch 13: never hardcode credentials)",
+        "Flags no logging of rows processed, errors encountered, or run duration (Ch 12: monitoring and observability)",
+        "Flags the absence of a staging table: data is written directly to the production `orders_eur` table without validation (Ch 8: always load to staging first)"
+      ]
+    },
+    {
+      "id": "eval-02-mixed-transform-and-load",
+      "prompt": "Review this data pipeline script:\n\n```python\nimport pandas as pd\nimport sqlalchemy\n\ndef process_and_load(csv_path: str, db_url: str, table: str):\n    df = pd.read_csv(csv_path)\n\n    # Clean and transform\n    df['email'] = df['email'].str.lower().str.strip()\n    df['revenue'] = df['revenue'].fillna(0)\n    df['signup_date'] = pd.to_datetime(df['signup_date'])\n    df = df[df['revenue'] >= 0]\n    df['revenue_category'] = df['revenue'].apply(\n        lambda x: 'high' if x > 1000 else 'low'\n    )\n    df['country'] = df['country'].str.upper()\n\n    # Enrich with another file\n    regions = pd.read_csv('regions.csv')  # hardcoded path\n    df = df.merge(regions, on='country', how='left')\n\n    # Load directly into the final table\n    engine = sqlalchemy.create_engine(db_url)\n    df.to_sql(table, engine, if_exists='append', index=False)\n    print(f'Loaded {len(df)} rows')\n```",
+      "expectations": [
+        "Flags that transformation logic and loading logic are combined in a single function, violating separation of concerns; recommends splitting into separate extract, transform, and load functions (Ch 3: ETL pattern design, Ch 11: DAG-based task granularity)",
+        "Flags the hardcoded path `'regions.csv'` as a non-configurable dependency that breaks when the file moves; recommends externalizing all paths and inputs as parameters or config (Ch 13: configurable pipelines)",
+        "Flags `if_exists='append'` with no deduplication: re-running appends duplicate rows; recommends staging table + MERGE or using a unique constraint with INSERT OR IGNORE (Ch 13: idempotency)",
+        "Flags no data validation before loading: there is no check that the merge did not produce unexpected nulls in the region column or that row counts match expectations (Ch 10: validate at boundaries)",
+        "Flags no logging beyond a single print statement: recommends structured logging of row counts at each stage, null rates, and merge match rate (Ch 12: monitoring and observability)",
+        "Flags absence of data lineage tracking: no pipeline_run_id or audit column to identify which pipeline run produced each row, making debugging and reruns harder to trace (Ch 13: data lineage)",
+        "Recommends adding a schema validation step after reading the CSV to catch missing or mistyped columns before transformations run (Ch 10: schema validation at ingestion)"
+      ]
+    },
+    {
+      "id": "eval-03-clean-pipeline-with-retry-logging-separation",
+      "prompt": "Review this data pipeline implementation:\n\n```python\nimport logging\nimport time\nfrom datetime import datetime, timedelta\nfrom typing import Iterator\nimport psycopg2\nimport psycopg2.extras\n\nlogger = logging.getLogger(__name__)\n\nBATCH_SIZE = 1000\nMAX_RETRIES = 3\nBACKOFF_BASE = 2\n\n\ndef extract(conn, watermark: datetime) -> Iterator[list]:\n    \"\"\"Yield batches of new orders since the watermark.\"\"\"\n    with conn.cursor(cursor_factory=psycopg2.extras.DictCursor) as cur:\n        cur.execute(\n            'SELECT id, customer_id, amount, created_at '\n            'FROM orders WHERE created_at > %s ORDER BY created_at',\n            (watermark,)\n        )\n        while True:\n            rows = cur.fetchmany(BATCH_SIZE)\n            if not rows:\n                break\n            logger.info('Extracted batch of %d rows', len(rows))\n            yield [dict(r) for r in rows]\n\n\ndef transform(batch: list[dict]) -> list[dict]:\n    \"\"\"Apply business rules: normalize amounts, tag high-value orders.\"\"\"\n    result = []\n    for row in batch:\n        row['amount'] = round(float(row['amount']), 2)\n        row['is_high_value'] = row['amount'] > 500\n        result.append(row)\n    return result\n\n\ndef load(conn, rows: list[dict], run_id: str) -> int:\n    \"\"\"Upsert rows into orders_warehouse; return count of rows loaded.\"\"\"\n    with conn.cursor() as cur:\n        psycopg2.extras.execute_values(\n            cur,\n            '''\n            INSERT INTO orders_warehouse (id, customer_id, amount, is_high_value, created_at, pipeline_run_id)\n            VALUES %s\n            ON CONFLICT (id) DO UPDATE SET\n                amount = EXCLUDED.amount,\n                is_high_value = EXCLUDED.is_high_value,\n                pipeline_run_id = EXCLUDED.pipeline_run_id\n            ''',\n            [(r['id'], r['customer_id'], r['amount'], r['is_high_value'],\n              r['created_at'], run_id) for r in rows]\n        )\n    conn.commit()\n    return len(rows)\n\n\ndef run_with_retry(fn, *args, **kwargs):\n    \"\"\"Retry a function with exponential backoff on transient errors.\"\"\"\n    for attempt in range(1, MAX_RETRIES + 1):\n        try:\n            return fn(*args, **kwargs)\n        except psycopg2.OperationalError as e:\n            if attempt == MAX_RETRIES:\n                raise\n            delay = BACKOFF_BASE ** attempt\n            logger.warning('Attempt %d failed: %s. Retrying in %ds', attempt, e, delay)\n            time.sleep(delay)\n```",
+      "expectations": [
+        "Recognizes this is a well-designed pipeline and says so explicitly",
+        "Praises the clear separation of `extract`, `transform`, and `load` into distinct functions with single responsibilities (Ch 3: ETL pattern, Ch 11: task granularity)",
+        "Praises the watermark-based incremental extraction that avoids full-table scans on reruns (Ch 3-4: incremental extraction)",
+        "Praises the `ON CONFLICT DO UPDATE` upsert ensuring the pipeline is idempotent and safe to re-run (Ch 13: idempotency is non-negotiable)",
+        "Praises the generator-based `extract` function that yields batches, avoiding loading the full result set into memory (Ch 4: streaming extraction, memory efficiency)",
+        "Praises the `run_with_retry` wrapper with exponential backoff for transient database errors (Ch 13: error handling and retry strategies)",
+        "Praises structured logging at the batch level with row counts for observability (Ch 12: monitoring)",
+        "Praises the `pipeline_run_id` column in the load, enabling lineage tracking and debugging of which run produced which rows (Ch 13: data lineage)",
+        "Does NOT manufacture issues to appear thorough; any suggestions are framed as minor optional improvements"
+      ]
+    }
+  ]
+}

package/skills/data-pipelines/evals/results.json

@@ -0,0 +1,13 @@
+{
+  "pass_rate": 0.957,
+  "passed": 22,
+  "total": 23,
+  "baseline_pass_rate": 0.304,
+  "baseline_passed": 7,
+  "baseline_total": 23,
+  "delta": 0.652,
+  "model": "default",
+  "evals_run": 3,
+  "date": "2026-03-28",
+  "non_standard_provider": true
+}

package/skills/data-pipelines/examples/after.md

@@ -0,0 +1,97 @@
+# After
+
+A clean pipeline with separated extract/transform/load functions, idempotent upserts, retry logic, and proper error handling.
+
+```python
+import logging
+import time
+from dataclasses import dataclass
+from datetime import datetime
+from functools import wraps
+
+import psycopg2
+import requests
+from requests.exceptions import RequestException
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class SaleRecord:
+    id: str
+    sale_date: datetime
+    revenue: float
+    region: str
+
+
+def with_retry(max_attempts: int = 3, backoff_seconds: float = 2.0):
+    """Decorator: retry a function on transient failures with exponential backoff."""
+    def decorator(fn):
+        @wraps(fn)
+        def wrapper(*args, **kwargs):
+            for attempt in range(1, max_attempts + 1):
+                try:
+                    return fn(*args, **kwargs)
+                except (RequestException, psycopg2.OperationalError) as exc:
+                    if attempt == max_attempts:
+                        raise
+                    wait = backoff_seconds ** attempt
+                    logger.warning("Attempt %d/%d failed: %s — retrying in %.1fs",
+                                   attempt, max_attempts, exc, wait)
+                    time.sleep(wait)
+        return wrapper
+    return decorator
+
+
+@with_retry(max_attempts=3)
+def extract(api_url: str) -> list[dict]:
+    """Fetch raw sales records from the partner API."""
+    response = requests.get(api_url, timeout=30)
+    response.raise_for_status()
+    return response.json()["sales"]
+
+
+def transform(raw_records: list[dict]) -> list[SaleRecord]:
+    """Parse and normalise raw API records into typed SaleRecord objects."""
+    return [
+        SaleRecord(
+            id=rec["id"],
+            sale_date=datetime.fromisoformat(rec["date"]),
+            revenue=float(rec["amount_usd"]),
+            region=rec["region"].strip().upper(),
+        )
+        for rec in raw_records
+    ]
+
+
+def load(records: list[SaleRecord], dsn: str) -> int:
+    """Upsert records into fact_sales. Idempotent: re-running is safe."""
+    upsert_sql = """
+        INSERT INTO fact_sales (sale_id, sale_date, revenue, region, loaded_at)
+        VALUES (%(id)s, %(sale_date)s, %(revenue)s, %(region)s, NOW())
+        ON CONFLICT (sale_id) DO UPDATE
+            SET revenue   = EXCLUDED.revenue,
+                loaded_at = EXCLUDED.loaded_at
+    """
+    with psycopg2.connect(dsn) as conn, conn.cursor() as cur:
+        cur.executemany(upsert_sql, [vars(r) for r in records])
+        loaded = cur.rowcount
+    logger.info("Upserted %d records into fact_sales", loaded)
+    return loaded
+
+
+def run_pipeline(api_url: str, warehouse_dsn: str) -> None:
+    logger.info("Starting sales pipeline")
+    raw = extract(api_url)
+    records = transform(raw)
+    loaded = load(records, warehouse_dsn)
+    logger.info("Pipeline complete: %d records loaded", loaded)
+```
+
+Key improvements:
+- Extract, transform, and load are separate functions with single responsibilities — each is independently testable and replaceable (Ch 13: Best Practices — separation of concerns)
+- `ON CONFLICT (sale_id) DO UPDATE` makes the load idempotent — re-running the pipeline never creates duplicate rows (Ch 13: Idempotency)
+- `@with_retry` decorator handles transient API and database failures with exponential backoff (Ch 6: API Ingestion — retry logic)
+- `SaleRecord` dataclass replaces a raw dict, providing type safety and named field access in the transform step
+- `psycopg2.connect` used as a context manager ensures the connection and transaction are always closed and committed correctly (Ch 4: Database Ingestion)
+- Structured logging with `logger.info/warning` replaces bare `print` — output is filterable and includes context (Ch 12: Monitoring)

package/skills/data-pipelines/examples/before.md

@@ -0,0 +1,37 @@
+# Before
+
+A Python ETL script that mixes extraction, transformation, and loading in one function with no error handling, no idempotency, and no retry logic.
+
+```python
+import psycopg2
+import requests
+from datetime import datetime
+
+def run_pipeline():
+    # Extract: fetch from API
+    resp = requests.get("https://api.partner.com/sales/export")
+    data = resp.json()
+
+    # Connect to warehouse
+    conn = psycopg2.connect("host=dw user=etl dbname=warehouse")
+    cur = conn.cursor()
+
+    # Transform + Load: all in one loop, no error handling
+    for record in data["sales"]:
+        sale_date = datetime.strptime(record["date"], "%Y-%m-%dT%H:%M:%S")
+        revenue = float(record["amount_usd"])
+        region = record["region"].strip().upper()
+
+        # No upsert — re-running inserts duplicates
+        cur.execute("""
+            INSERT INTO fact_sales (sale_id, sale_date, revenue, region, loaded_at)
+            VALUES (%s, %s, %s, %s, NOW())
+        """, (record["id"], sale_date, revenue, region))
+
+    conn.commit()
+    cur.close()
+    conn.close()
+    print("done")
+
+run_pipeline()
+```

package/skills/data-pipelines/references/api_reference.md

@@ -0,0 +1,301 @@
+# Data Pipelines Pocket Reference — Practices Catalog
+
+Chapter-by-chapter catalog of practices from *Data Pipelines Pocket Reference*
+by James Densmore for pipeline building.
+
+---
+
+## Chapter 1–2: Introduction & Modern Data Infrastructure
+
+### Infrastructure Choices
+- **Data warehouse** — Columnar storage optimized for analytics queries (Redshift, BigQuery, Snowflake); choose when analytics is primary use case
+- **Data lake** — Object storage (S3, GCS, Azure Blob) for raw, unstructured, or semi-structured data; choose when data variety is high or schema is unknown
+- **Hybrid** — Land raw data in lake, load structured subsets into warehouse; common modern pattern
+- **Cloud-native** — Leverage managed services (serverless compute, auto-scaling storage); reduce operational burden
+
+### Pipeline Types
+- **Batch** — Process data in scheduled intervals (hourly, daily); suitable for most analytics use cases
+- **Streaming** — Process data continuously as it arrives; required for real-time dashboards, alerts, event-driven systems
+- **Micro-batch** — Small frequent batches (every few minutes); compromise between batch simplicity and near-real-time latency
+
+---
+
+## Chapter 3: Common Data Pipeline Patterns
+
+### Extraction Patterns
+- **Full extraction** — Extract entire dataset each run; simple but expensive for large tables; use for small reference tables or initial loads
+- **Incremental extraction** — Extract only new/changed records using a high-water mark (timestamp, auto-increment ID, or sequence); preferred for growing datasets
+- **Change data capture (CDC)** — Capture changes from database transaction logs (MySQL binlog, PostgreSQL WAL); lowest latency, captures deletes; use for real-time sync
+
+### Loading Patterns
+- **Full refresh (truncate + load)** — Replace entire destination table; simple, idempotent; use for small tables or when incremental is unreliable
+- **Append** — Insert new records only; use for event/log data that is never updated
+- **Upsert (MERGE)** — Insert new records, update existing ones based on a key; use for mutable dimension data
+- **Delete + Insert by partition** — Delete partition, insert replacement; idempotent and efficient for date-partitioned fact tables
+
+### ETL vs ELT
+- **ETL** — Transform before loading; use when destination has limited compute or when data must be cleansed before storage
+- **ELT** — Load raw data first, transform in destination; preferred for modern cloud warehouses with cheap compute; enables raw data preservation and flexible re-transformation
+
+---
+
+## Chapter 4: Database Ingestion
+
+### MySQL Extraction
+- Use `SELECT ... WHERE updated_at > :last_run` for incremental extraction
+- For full extraction: `SELECT *` with optional `LIMIT/OFFSET` for large tables (prefer streaming cursors)
+- Use read replicas to avoid impacting production database performance
+- Handle MySQL timezone conversions (store/compare in UTC)
+- Connection pooling for concurrent extraction from multiple tables
+
+### PostgreSQL Extraction
+- Similar incremental patterns using timestamp columns
+- Use `COPY TO` for efficient bulk export to CSV/files
+- Leverage PostgreSQL logical replication for CDC
+- Handle PostgreSQL-specific types (arrays, JSON, custom types) during extraction
+
+### MongoDB Extraction
+- Use change streams for real-time CDC
+- For incremental: query by `_id` (ObjectId contains timestamp) or custom timestamp field
+- Handle nested documents: flatten or store as JSON in warehouse
+- Use `mongodump` for full extraction of large collections
+
+### General Database Practices
+- **Connection management** — Use connection pools; close connections promptly; handle timeouts
+- **Query optimization** — Add indexes on extraction columns; limit selected columns; use WHERE clauses
+- **Binary data** — Skip or store references to BLOBs; don't load binary into analytics warehouse
+- **Character encoding** — Ensure UTF-8 throughout the pipeline; handle encoding mismatches at extraction
+
+---
+
+## Chapter 5: File Ingestion
+
+### CSV Files
+- Handle header rows, quoting, escaping, delimiters (not always comma)
+- Detect and handle encoding (UTF-8, Latin-1, Windows-1252)
+- Validate column count per row; log and quarantine malformed rows
+- Use streaming parsers for large files; avoid loading entire file into memory
+
+### JSON Files
+- Handle nested structures: flatten for warehouse loading or store as JSON column
+- Use JSON Lines (newline-delimited JSON) for large datasets
+- Validate against expected schema; handle missing and extra fields
+- Parse dates and timestamps consistently
+
+### Cloud Storage Integration
+- **S3** — Use `aws s3 cp/sync` or boto3; leverage S3 event notifications for trigger-based ingestion
+- **GCS** — Use `gsutil` or google-cloud-storage library; use Pub/Sub for event notifications
+- **Azure Blob** — Use Azure SDK; leverage Event Grid for notifications
+- Use prefix/partition naming: `s3://bucket/table/year=2024/month=01/day=15/`
+- Implement file manifests to track which files have been processed
+
+### File Best Practices
+- **Naming conventions** — Include date, source, and sequence in filenames: `orders_2024-01-15_001.csv`
+- **Compression** — Use gzip or snappy for storage efficiency; most tools handle compressed files natively
+- **Archiving** — Move processed files to archive prefix/bucket; retain for reprocessing capability
+- **Schema detection** — Infer schema from first N rows; validate against expected schema; alert on changes
+
+---
+
+## Chapter 6: API Ingestion
+
+### REST API Patterns
+- **Authentication** — Handle API keys, OAuth tokens, token refresh; store credentials securely
+- **Pagination** — Implement cursor-based, offset-based, or link-header pagination; prefer cursor-based for consistency
+- **Rate limiting** — Respect rate limit headers (X-RateLimit-Remaining, Retry-After); implement backoff
+- **Retry logic** — Retry on 429 (rate limit) and 5xx (server error) with exponential backoff; don't retry on 4xx (client error)
+
+### API Data Handling
+- **JSON response parsing** — Extract relevant fields; handle nested objects and arrays
+- **Incremental fetching** — Use modified_since parameters, cursor tokens, or date range filters
+- **Schema changes** — Handle new fields gracefully; log and alert on missing expected fields
+- **Large responses** — Stream responses for large payloads; paginate aggressively
+
+### Webhook Ingestion
+- Set up HTTP endpoints to receive push notifications
+- Validate webhook signatures for security
+- Acknowledge receipt quickly (200 OK); process asynchronously
+- Implement idempotency using event IDs to handle duplicate deliveries
+
+---
+
+## Chapter 7: Streaming Data
+
+### Apache Kafka
+- **Producers** — Serialize events (Avro, JSON, Protobuf); use keys for partition ordering; configure acknowledgments
+- **Consumers** — Use consumer groups for parallel processing; commit offsets after successful processing; handle rebalances
+- **Topics** — Design topic schemas; set retention policies; partition for throughput and ordering requirements
+- **Exactly-once** — Use idempotent producers + transactional consumers; or implement deduplication downstream
+
+### Amazon Kinesis
+- **Streams** — Configure shard count for throughput; use enhanced fan-out for multiple consumers
+- **Firehose** — Direct-to-S3/Redshift delivery; configure buffering interval and size; transform with Lambda
+
+### Stream Processing Patterns
+- **Windowing** — Tumbling, sliding, session windows for aggregation over time
+- **Watermarks** — Handle late-arriving events; define allowed lateness
+- **State management** — Use state stores for aggregations; handle checkpointing and recovery
+- **Dead letter queues** — Route failed events for inspection and reprocessing; don't lose data silently
+
+---
+
+## Chapter 8: Data Storage and Loading
+
+### Amazon Redshift
+- Use `COPY` command for bulk loading from S3; much faster than INSERT
+- Define `SORTKEY` for frequently filtered columns; `DISTKEY` for join columns
+- Use `UNLOAD` for efficient export back to S3
+- Vacuum and analyze tables after large loads
+
+### Google BigQuery
+- Use load jobs for bulk data; streaming inserts for real-time (more expensive)
+- Partition tables by date (ingestion time or column-based) for cost and performance
+- Cluster tables on frequently filtered columns
+- Use external tables for querying data in GCS without loading
+
+### Snowflake
+- Use stages (internal or external) for file-based loading
+- `COPY INTO` for bulk loads from stages; `SNOWPIPE` for continuous loading
+- Use virtual warehouses sized appropriately for load workloads
+- Leverage Time Travel for data recovery and auditing
+
+### General Loading Practices
+- **Staging tables** — Always load to staging first; validate before merging to production
+- **Atomic swaps** — Use table rename or partition swap for atomic updates
+- **Data types** — Map source types carefully; avoid implicit conversions; use appropriate precision
+- **Compression** — Let warehouse handle compression; load compressed files when supported
+- **Partitioning** — Partition by date for time-series data; by key for lookup tables
+- **Clustering** — Cluster on frequently filtered columns within partitions
+
+---
+
+## Chapter 9: Data Transformations
+
+### SQL-Based Transforms
+- Use CTEs (Common Table Expressions) for readability
+- Prefer window functions over self-joins for ranking, running totals
+- Use CASE expressions for conditional logic
+- Aggregate at the right grain; avoid fan-out joins that multiply rows
+
+### dbt (Data Build Tool)
+- **Staging models** — 1:1 with source tables; rename columns, cast types, filter deleted records
+- **Intermediate models** — Business logic joins, complex calculations, deduplication
+- **Mart models** — Final analytics-ready tables; optimized for dashboard queries
+- **Incremental models** — Process only new/changed data; use `unique_key` for merge strategy
+- **Tests** — `not_null`, `unique`, `accepted_values`, `relationships` on key columns; custom data tests
+- **Sources** — Define sources in YAML; use `source()` macro for lineage tracking; freshness checks
+
+### Python-Based Transforms
+- Use pandas for small-medium datasets; PySpark or Dask for large-scale processing
+- Write pure functions for transformations; make them testable
+- Handle data types explicitly; don't rely on inference for production pipelines
+- Use vectorized operations; avoid row-by-row iteration
+
+### Transform Best Practices
+- **Layered architecture** — Raw → Staging → Intermediate → Mart; each layer has clear purpose
+- **Single responsibility** — Each model/transform does one thing well
+- **Documented logic** — Comment complex business rules; maintain a data dictionary
+- **Version controlled** — All transformation code in git; review changes via PR
+
+---
+
+## Chapter 10: Data Validation and Testing
+
+### Validation Types
+- **Schema validation** — Verify column names, types, and count match expectations
+- **Row count checks** — Compare source and destination row counts; alert on significant discrepancies
+- **Null checks** — Assert key columns are not null; track null percentages for optional columns
+- **Uniqueness checks** — Verify primary key uniqueness in destination tables
+- **Referential integrity** — Check foreign key relationships between tables
+- **Range checks** — Validate values fall within expected ranges (dates, amounts, percentages)
+- **Freshness checks** — Verify data is not stale; alert when max timestamp is older than threshold
+
+### Great Expectations
+- Define expectations as code; version control them
+- Run validations as pipeline steps; fail pipeline on critical expectation failures
+- Generate data documentation from expectations
+- Use checkpoints for scheduled validation runs
+
+### Testing Practices
+- **Unit tests** — Test individual transformation functions with known inputs/outputs
+- **Integration tests** — Test end-to-end pipeline with sample data
+- **Regression tests** — Compare current results against known-good baselines
+- **Data contracts** — Define and enforce schemas between producer and consumer teams
+
+---
+
+## Chapter 11: Orchestration
+
+### Apache Airflow
+- **DAGs** — Define pipelines as Directed Acyclic Graphs; each node is a task
+- **Operators** — Use appropriate operators: PythonOperator, BashOperator, provider operators (BigQueryOperator, S3ToRedshiftOperator)
+- **Scheduling** — Use cron expressions or timedelta; set `start_date` and `catchup` appropriately
+- **Dependencies** — Use `>>` operator or `set_upstream/downstream`; keep DAGs shallow and wide
+- **XComs** — Pass small metadata between tasks (row counts, file paths); NOT large datasets
+- **Sensors** — Wait for external conditions (file arrival, partition availability); use with timeout
+- **Variables and Connections** — Store config in Airflow Variables; credentials in Connections
+- **Pools** — Limit concurrency for resource-constrained tasks (database connections, API rate limits)
+
+### DAG Design Patterns
+- **One pipeline per DAG** — Keep DAGs focused; avoid mega-DAGs that do everything
+- **Idempotent tasks** — Every task can be re-run safely; use `execution_date` for parameterization
+- **Task granularity** — Tasks should be atomic and independently retryable; not too fine (overhead) or coarse (blast radius)
+- **Error handling** — Use `on_failure_callback` for alerting; `retries` and `retry_delay` for transient failures
+- **Backfilling** — Use `airflow backfill` for historical reprocessing; ensure tasks support date parameterization
+
+---
+
+## Chapter 12: Monitoring and Alerting
+
+### Pipeline Health Metrics
+- **Duration** — Track execution time; alert on runs significantly longer than historical average
+- **Row counts** — Track records processed per run; alert on zero rows or dramatic changes
+- **Error rates** — Track failed records, retries, exceptions; alert on elevated error rates
+- **Data freshness** — Track max timestamp in destination; alert when data is staler than SLA
+- **Resource usage** — Track CPU, memory, disk, network; alert on resource exhaustion
+
+### Alerting Strategies
+- **SLA-based** — Define delivery SLAs; alert when pipelines miss their windows
+- **Anomaly-based** — Detect deviations from historical patterns (row counts, durations, values)
+- **Threshold-based** — Alert on fixed thresholds (error rate > 5%, null rate > 10%)
+- **Escalation** — Define severity levels; route alerts appropriately (Slack, PagerDuty, email)
+
+### Logging
+- Log at each pipeline stage: extraction start/end, row counts, load confirmation
+- Include correlation IDs to trace records through the pipeline
+- Store logs centrally for searchability (ELK, CloudWatch, Stackdriver)
+- Retain logs for debugging and audit compliance
+
+---
+
+## Chapter 13: Best Practices
+
+### Idempotency
+- Use DELETE+INSERT by date partition for fact tables
+- Use MERGE/upsert with natural keys for dimension tables
+- Use staging tables as intermediary; clean up on both success and failure
+- Test by running pipeline twice; verify no data duplication
+
+### Backfilling
+- Parameterize all pipelines by date range (start_date, end_date)
+- Use Airflow `execution_date` or equivalent for date-aware runs
+- Test backfill on a small date range before running full historical reprocess
+- Monitor resource usage during backfill; may need to throttle parallelism
+
+### Error Handling
+- Retry transient failures (network timeouts, rate limits) with exponential backoff
+- Fail fast on permanent errors (authentication failure, missing source table)
+- Quarantine bad records; don't let one bad row fail the entire pipeline
+- Send alerts with actionable context (error message, affected table, run ID)
+
+### Data Lineage and Documentation
+- Track source-to-destination mappings for every table
+- Document transformation logic, especially business rules
+- Maintain a data dictionary with column descriptions and types
+- Use tools like dbt docs, DataHub, or Amundsen for automated lineage
+
+### Security
+- Never hardcode credentials; use secrets managers (AWS Secrets Manager, HashiCorp Vault)
+- Encrypt data in transit (TLS) and at rest (warehouse encryption)
+- Use least-privilege IAM roles for pipeline service accounts
+- Audit access to sensitive data; mask PII in non-production environments