@booklib/skills 1.0.0

package/data-pipelines/references/api_reference.md

@@ -0,0 +1,301 @@
+# Data Pipelines Pocket Reference — Practices Catalog
+
+Chapter-by-chapter catalog of practices from *Data Pipelines Pocket Reference*
+by James Densmore for pipeline building.
+
+---
+
+## Chapter 1–2: Introduction & Modern Data Infrastructure
+
+### Infrastructure Choices
+- **Data warehouse** — Columnar storage optimized for analytics queries (Redshift, BigQuery, Snowflake); choose when analytics is primary use case
+- **Data lake** — Object storage (S3, GCS, Azure Blob) for raw, unstructured, or semi-structured data; choose when data variety is high or schema is unknown
+- **Hybrid** — Land raw data in lake, load structured subsets into warehouse; common modern pattern
+- **Cloud-native** — Leverage managed services (serverless compute, auto-scaling storage); reduce operational burden
+
+### Pipeline Types
+- **Batch** — Process data in scheduled intervals (hourly, daily); suitable for most analytics use cases
+- **Streaming** — Process data continuously as it arrives; required for real-time dashboards, alerts, event-driven systems
+- **Micro-batch** — Small frequent batches (every few minutes); compromise between batch simplicity and near-real-time latency
+
+---
+
+## Chapter 3: Common Data Pipeline Patterns
+
+### Extraction Patterns
+- **Full extraction** — Extract entire dataset each run; simple but expensive for large tables; use for small reference tables or initial loads
+- **Incremental extraction** — Extract only new/changed records using a high-water mark (timestamp, auto-increment ID, or sequence); preferred for growing datasets
+- **Change data capture (CDC)** — Capture changes from database transaction logs (MySQL binlog, PostgreSQL WAL); lowest latency, captures deletes; use for real-time sync
+
+### Loading Patterns
+- **Full refresh (truncate + load)** — Replace entire destination table; simple, idempotent; use for small tables or when incremental is unreliable
+- **Append** — Insert new records only; use for event/log data that is never updated
+- **Upsert (MERGE)** — Insert new records, update existing ones based on a key; use for mutable dimension data
+- **Delete + Insert by partition** — Delete partition, insert replacement; idempotent and efficient for date-partitioned fact tables
+
+### ETL vs ELT
+- **ETL** — Transform before loading; use when destination has limited compute or when data must be cleansed before storage
+- **ELT** — Load raw data first, transform in destination; preferred for modern cloud warehouses with cheap compute; enables raw data preservation and flexible re-transformation
+
+---
+
+## Chapter 4: Database Ingestion
+
+### MySQL Extraction
+- Use `SELECT ... WHERE updated_at > :last_run` for incremental extraction
+- For full extraction: `SELECT *` with optional `LIMIT/OFFSET` for large tables (prefer streaming cursors)
+- Use read replicas to avoid impacting production database performance
+- Handle MySQL timezone conversions (store/compare in UTC)
+- Connection pooling for concurrent extraction from multiple tables
+
+### PostgreSQL Extraction
+- Similar incremental patterns using timestamp columns
+- Use `COPY TO` for efficient bulk export to CSV/files
+- Leverage PostgreSQL logical replication for CDC
+- Handle PostgreSQL-specific types (arrays, JSON, custom types) during extraction
+
+### MongoDB Extraction
+- Use change streams for real-time CDC
+- For incremental: query by `_id` (ObjectId contains timestamp) or custom timestamp field
+- Handle nested documents: flatten or store as JSON in warehouse
+- Use `mongodump` for full extraction of large collections
+
+### General Database Practices
+- **Connection management** — Use connection pools; close connections promptly; handle timeouts
+- **Query optimization** — Add indexes on extraction columns; limit selected columns; use WHERE clauses
+- **Binary data** — Skip or store references to BLOBs; don't load binary into analytics warehouse
+- **Character encoding** — Ensure UTF-8 throughout the pipeline; handle encoding mismatches at extraction
+
+---
+
+## Chapter 5: File Ingestion
+
+### CSV Files
+- Handle header rows, quoting, escaping, delimiters (not always comma)
+- Detect and handle encoding (UTF-8, Latin-1, Windows-1252)
+- Validate column count per row; log and quarantine malformed rows
+- Use streaming parsers for large files; avoid loading entire file into memory
+
+### JSON Files
+- Handle nested structures: flatten for warehouse loading or store as JSON column
+- Use JSON Lines (newline-delimited JSON) for large datasets
+- Validate against expected schema; handle missing and extra fields
+- Parse dates and timestamps consistently
+
+### Cloud Storage Integration
+- **S3** — Use `aws s3 cp/sync` or boto3; leverage S3 event notifications for trigger-based ingestion
+- **GCS** — Use `gsutil` or google-cloud-storage library; use Pub/Sub for event notifications
+- **Azure Blob** — Use Azure SDK; leverage Event Grid for notifications
+- Use prefix/partition naming: `s3://bucket/table/year=2024/month=01/day=15/`
+- Implement file manifests to track which files have been processed
+
+### File Best Practices
+- **Naming conventions** — Include date, source, and sequence in filenames: `orders_2024-01-15_001.csv`
+- **Compression** — Use gzip or snappy for storage efficiency; most tools handle compressed files natively
+- **Archiving** — Move processed files to archive prefix/bucket; retain for reprocessing capability
+- **Schema detection** — Infer schema from first N rows; validate against expected schema; alert on changes
+
+---
+
+## Chapter 6: API Ingestion
+
+### REST API Patterns
+- **Authentication** — Handle API keys, OAuth tokens, token refresh; store credentials securely
+- **Pagination** — Implement cursor-based, offset-based, or link-header pagination; prefer cursor-based for consistency
+- **Rate limiting** — Respect rate limit headers (X-RateLimit-Remaining, Retry-After); implement backoff
+- **Retry logic** — Retry on 429 (rate limit) and 5xx (server error) with exponential backoff; don't retry on 4xx (client error)
+
+### API Data Handling
+- **JSON response parsing** — Extract relevant fields; handle nested objects and arrays
+- **Incremental fetching** — Use modified_since parameters, cursor tokens, or date range filters
+- **Schema changes** — Handle new fields gracefully; log and alert on missing expected fields
+- **Large responses** — Stream responses for large payloads; paginate aggressively
+
+### Webhook Ingestion
+- Set up HTTP endpoints to receive push notifications
+- Validate webhook signatures for security
+- Acknowledge receipt quickly (200 OK); process asynchronously
+- Implement idempotency using event IDs to handle duplicate deliveries
+
+---
+
+## Chapter 7: Streaming Data
+
+### Apache Kafka
+- **Producers** — Serialize events (Avro, JSON, Protobuf); use keys for partition ordering; configure acknowledgments
+- **Consumers** — Use consumer groups for parallel processing; commit offsets after successful processing; handle rebalances
+- **Topics** — Design topic schemas; set retention policies; partition for throughput and ordering requirements
+- **Exactly-once** — Use idempotent producers + transactional consumers; or implement deduplication downstream
+
+### Amazon Kinesis
+- **Streams** — Configure shard count for throughput; use enhanced fan-out for multiple consumers
+- **Firehose** — Direct-to-S3/Redshift delivery; configure buffering interval and size; transform with Lambda
+
+### Stream Processing Patterns
+- **Windowing** — Tumbling, sliding, session windows for aggregation over time
+- **Watermarks** — Handle late-arriving events; define allowed lateness
+- **State management** — Use state stores for aggregations; handle checkpointing and recovery
+- **Dead letter queues** — Route failed events for inspection and reprocessing; don't lose data silently
+
+---
+
+## Chapter 8: Data Storage and Loading
+
+### Amazon Redshift
+- Use `COPY` command for bulk loading from S3; much faster than INSERT
+- Define `SORTKEY` for frequently filtered columns; `DISTKEY` for join columns
+- Use `UNLOAD` for efficient export back to S3
+- Vacuum and analyze tables after large loads
+
+### Google BigQuery
+- Use load jobs for bulk data; streaming inserts for real-time (more expensive)
+- Partition tables by date (ingestion time or column-based) for cost and performance
+- Cluster tables on frequently filtered columns
+- Use external tables for querying data in GCS without loading
+
+### Snowflake
+- Use stages (internal or external) for file-based loading
+- `COPY INTO` for bulk loads from stages; `SNOWPIPE` for continuous loading
+- Use virtual warehouses sized appropriately for load workloads
+- Leverage Time Travel for data recovery and auditing
+
+### General Loading Practices
+- **Staging tables** — Always load to staging first; validate before merging to production
+- **Atomic swaps** — Use table rename or partition swap for atomic updates
+- **Data types** — Map source types carefully; avoid implicit conversions; use appropriate precision
+- **Compression** — Let warehouse handle compression; load compressed files when supported
+- **Partitioning** — Partition by date for time-series data; by key for lookup tables
+- **Clustering** — Cluster on frequently filtered columns within partitions
+
+---
+
+## Chapter 9: Data Transformations
+
+### SQL-Based Transforms
+- Use CTEs (Common Table Expressions) for readability
+- Prefer window functions over self-joins for ranking, running totals
+- Use CASE expressions for conditional logic
+- Aggregate at the right grain; avoid fan-out joins that multiply rows
+
+### dbt (Data Build Tool)
+- **Staging models** — 1:1 with source tables; rename columns, cast types, filter deleted records
+- **Intermediate models** — Business logic joins, complex calculations, deduplication
+- **Mart models** — Final analytics-ready tables; optimized for dashboard queries
+- **Incremental models** — Process only new/changed data; use `unique_key` for merge strategy
+- **Tests** — `not_null`, `unique`, `accepted_values`, `relationships` on key columns; custom data tests
+- **Sources** — Define sources in YAML; use `source()` macro for lineage tracking; freshness checks
+
+### Python-Based Transforms
+- Use pandas for small-medium datasets; PySpark or Dask for large-scale processing
+- Write pure functions for transformations; make them testable
+- Handle data types explicitly; don't rely on inference for production pipelines
+- Use vectorized operations; avoid row-by-row iteration
+
+### Transform Best Practices
+- **Layered architecture** — Raw → Staging → Intermediate → Mart; each layer has clear purpose
+- **Single responsibility** — Each model/transform does one thing well
+- **Documented logic** — Comment complex business rules; maintain a data dictionary
+- **Version controlled** — All transformation code in git; review changes via PR
+
+---
+
+## Chapter 10: Data Validation and Testing
+
+### Validation Types
+- **Schema validation** — Verify column names, types, and count match expectations
+- **Row count checks** — Compare source and destination row counts; alert on significant discrepancies
+- **Null checks** — Assert key columns are not null; track null percentages for optional columns
+- **Uniqueness checks** — Verify primary key uniqueness in destination tables
+- **Referential integrity** — Check foreign key relationships between tables
+- **Range checks** — Validate values fall within expected ranges (dates, amounts, percentages)
+- **Freshness checks** — Verify data is not stale; alert when max timestamp is older than threshold
+
+### Great Expectations
+- Define expectations as code; version control them
+- Run validations as pipeline steps; fail pipeline on critical expectation failures
+- Generate data documentation from expectations
+- Use checkpoints for scheduled validation runs
+
+### Testing Practices
+- **Unit tests** — Test individual transformation functions with known inputs/outputs
+- **Integration tests** — Test end-to-end pipeline with sample data
+- **Regression tests** — Compare current results against known-good baselines
+- **Data contracts** — Define and enforce schemas between producer and consumer teams
+
+---
+
+## Chapter 11: Orchestration
+
+### Apache Airflow
+- **DAGs** — Define pipelines as Directed Acyclic Graphs; each node is a task
+- **Operators** — Use appropriate operators: PythonOperator, BashOperator, provider operators (BigQueryOperator, S3ToRedshiftOperator)
+- **Scheduling** — Use cron expressions or timedelta; set `start_date` and `catchup` appropriately
+- **Dependencies** — Use `>>` operator or `set_upstream/downstream`; keep DAGs shallow and wide
+- **XComs** — Pass small metadata between tasks (row counts, file paths); NOT large datasets
+- **Sensors** — Wait for external conditions (file arrival, partition availability); use with timeout
+- **Variables and Connections** — Store config in Airflow Variables; credentials in Connections
+- **Pools** — Limit concurrency for resource-constrained tasks (database connections, API rate limits)
+
+### DAG Design Patterns
+- **One pipeline per DAG** — Keep DAGs focused; avoid mega-DAGs that do everything
+- **Idempotent tasks** — Every task can be re-run safely; use `execution_date` for parameterization
+- **Task granularity** — Tasks should be atomic and independently retryable; not too fine (overhead) or coarse (blast radius)
+- **Error handling** — Use `on_failure_callback` for alerting; `retries` and `retry_delay` for transient failures
+- **Backfilling** — Use `airflow backfill` for historical reprocessing; ensure tasks support date parameterization
+
+---
+
+## Chapter 12: Monitoring and Alerting
+
+### Pipeline Health Metrics
+- **Duration** — Track execution time; alert on runs significantly longer than historical average
+- **Row counts** — Track records processed per run; alert on zero rows or dramatic changes
+- **Error rates** — Track failed records, retries, exceptions; alert on elevated error rates
+- **Data freshness** — Track max timestamp in destination; alert when data is staler than SLA
+- **Resource usage** — Track CPU, memory, disk, network; alert on resource exhaustion
+
+### Alerting Strategies
+- **SLA-based** — Define delivery SLAs; alert when pipelines miss their windows
+- **Anomaly-based** — Detect deviations from historical patterns (row counts, durations, values)
+- **Threshold-based** — Alert on fixed thresholds (error rate > 5%, null rate > 10%)
+- **Escalation** — Define severity levels; route alerts appropriately (Slack, PagerDuty, email)
+
+### Logging
+- Log at each pipeline stage: extraction start/end, row counts, load confirmation
+- Include correlation IDs to trace records through the pipeline
+- Store logs centrally for searchability (ELK, CloudWatch, Stackdriver)
+- Retain logs for debugging and audit compliance
+
+---
+
+## Chapter 13: Best Practices
+
+### Idempotency
+- Use DELETE+INSERT by date partition for fact tables
+- Use MERGE/upsert with natural keys for dimension tables
+- Use staging tables as intermediary; clean up on both success and failure
+- Test by running pipeline twice; verify no data duplication
+
+### Backfilling
+- Parameterize all pipelines by date range (start_date, end_date)
+- Use Airflow `execution_date` or equivalent for date-aware runs
+- Test backfill on a small date range before running full historical reprocess
+- Monitor resource usage during backfill; may need to throttle parallelism
+
+### Error Handling
+- Retry transient failures (network timeouts, rate limits) with exponential backoff
+- Fail fast on permanent errors (authentication failure, missing source table)
+- Quarantine bad records; don't let one bad row fail the entire pipeline
+- Send alerts with actionable context (error message, affected table, run ID)
+
+### Data Lineage and Documentation
+- Track source-to-destination mappings for every table
+- Document transformation logic, especially business rules
+- Maintain a data dictionary with column descriptions and types
+- Use tools like dbt docs, DataHub, or Amundsen for automated lineage
+
+### Security
+- Never hardcode credentials; use secrets managers (AWS Secrets Manager, HashiCorp Vault)
+- Encrypt data in transit (TLS) and at rest (warehouse encryption)
+- Use least-privilege IAM roles for pipeline service accounts
+- Audit access to sensitive data; mask PII in non-production environments

package/data-pipelines/references/review-checklist.md

@@ -0,0 +1,181 @@
+# Data Pipelines Pocket Reference — Pipeline Review Checklist
+
+Systematic checklist for reviewing data pipelines against the 13 chapters
+from *Data Pipelines Pocket Reference* by James Densmore.
+
+---
+
+## 1. Architecture & Patterns (Chapters 1–3)
+
+### Infrastructure
+- [ ] **Ch 1-2 — Appropriate infrastructure** — Is the right storage chosen for the use case (warehouse for analytics, lake for raw/unstructured)?
+- [ ] **Ch 2 — Cloud-native services** — Are managed services used where appropriate to reduce operational burden?
+- [ ] **Ch 2 — Separation of storage and compute** — Is compute scaled independently from storage?
+
+### Pipeline Patterns
+- [ ] **Ch 3 — ETL vs ELT** — Is the right pattern chosen? Is ELT used for analytics workloads with modern warehouses?
+- [ ] **Ch 3 — Full vs incremental** — Is incremental extraction used for growing datasets? Is full extraction justified for small tables?
+- [ ] **Ch 3 — CDC where appropriate** — Is CDC used for real-time sync needs instead of polling?
+- [ ] **Ch 3 — Loading strategy** — Is the right load pattern used (append for events, upsert for dimensions, full refresh for small lookups)?
+
+---
+
+## 2. Data Ingestion (Chapters 4–7)
+
+### Database Ingestion (Ch 4)
+- [ ] **Ch 4 — Read replica usage** — Are extractions running against read replicas, not production databases?
+- [ ] **Ch 4 — Incremental column** — Is there a reliable timestamp or ID column for incremental extraction?
+- [ ] **Ch 4 — Connection management** — Are connections pooled and properly closed? Are timeouts configured?
+- [ ] **Ch 4 — Query efficiency** — Are only needed columns selected? Are WHERE clauses using indexed columns?
+- [ ] **Ch 4 — Large table handling** — Are large extractions chunked or using streaming cursors?
+
+### File Ingestion (Ch 5)
+- [ ] **Ch 5 — Schema validation** — Are file schemas validated before processing? Are malformed rows handled?
+- [ ] **Ch 5 — Encoding handling** — Is character encoding handled consistently (UTF-8)?
+- [ ] **Ch 5 — Cloud storage patterns** — Are files organized with partitioned prefixes? Are processed files archived?
+- [ ] **Ch 5 — File tracking** — Is there a mechanism to track which files have been processed to avoid reprocessing?
+- [ ] **Ch 5 — Compression** — Are files compressed for storage and transfer efficiency?
+
+### API Ingestion (Ch 6)
+- [ ] **Ch 6 — Pagination** — Is pagination implemented correctly? Are all pages fetched? Is cursor-based preferred?
+- [ ] **Ch 6 — Rate limiting** — Are rate limit headers respected? Is backoff implemented for 429 responses?
+- [ ] **Ch 6 — Retry logic** — Are transient errors (5xx, timeouts) retried with exponential backoff? Are 4xx errors not retried?
+- [ ] **Ch 6 — Authentication** — Are credentials stored securely? Are token refreshes handled?
+- [ ] **Ch 6 — Incremental fetching** — Are date/cursor parameters used to fetch only new data?
+
+### Streaming Ingestion (Ch 7)
+- [ ] **Ch 7 — Consumer group design** — Are consumer groups configured for parallel processing? Are rebalances handled?
+- [ ] **Ch 7 — Offset management** — Are offsets committed after successful processing, not before?
+- [ ] **Ch 7 — Serialization** — Are events serialized with a schema (Avro, Protobuf) for evolution support?
+- [ ] **Ch 7 — Dead letter queue** — Are failed events routed to a DLQ for inspection and reprocessing?
+- [ ] **Ch 7 — Exactly-once semantics** — Is deduplication implemented downstream or are idempotent producers used?
+- [ ] **Ch 7 — Backpressure** — Is backpressure handled when consumer can't keep up with producer?
+
+---
+
+## 3. Data Storage & Loading (Chapter 8)
+
+### Loading Patterns
+- [ ] **Ch 8 — Bulk loading** — Are bulk load commands used (COPY, load jobs) instead of row-by-row INSERT?
+- [ ] **Ch 8 — Staging tables** — Is data loaded to staging first, validated, then merged to production?
+- [ ] **Ch 8 — Atomic operations** — Are loads atomic? Is the destination never left in a partial state?
+- [ ] **Ch 8 — Data type mapping** — Are source types mapped correctly to destination types? No implicit conversions?
+
+### Table Design
+- [ ] **Ch 8 — Partitioning** — Are large tables partitioned by date or key? Are queries leveraging partition pruning?
+- [ ] **Ch 8 — Clustering** — Are frequently filtered columns used as cluster keys within partitions?
+- [ ] **Ch 8 — Sort/distribution keys** — For Redshift: are SORTKEY and DISTKEY chosen based on query patterns?
+
+### Warehouse-Specific
+- [ ] **Ch 8 — Redshift COPY** — Is S3-based COPY used instead of INSERT for bulk loads?
+- [ ] **Ch 8 — BigQuery load jobs** — Are load jobs preferred over streaming inserts for batch pipelines?
+- [ ] **Ch 8 — Snowflake stages** — Are stages used for file-based loading? Is SNOWPIPE configured for continuous loads?
+
+---
+
+## 4. Transformations (Chapter 9)
+
+### SQL Transforms
+- [ ] **Ch 9 — CTEs for readability** — Are Common Table Expressions used instead of deeply nested subqueries?
+- [ ] **Ch 9 — Window functions** — Are window functions used for ranking, running totals instead of self-joins?
+- [ ] **Ch 9 — Grain awareness** — Do joins maintain the correct grain? No accidental fan-out producing duplicate rows?
+- [ ] **Ch 9 — Deterministic logic** — Are transforms deterministic (same input → same output)? No reliance on row ordering?
+
+### dbt Patterns
+- [ ] **Ch 9 — Layer structure** — Are models organized into staging → intermediate → mart layers?
+- [ ] **Ch 9 — Staging models** — Are staging models 1:1 with sources? Do they rename, cast, and filter only?
+- [ ] **Ch 9 — Incremental models** — Are large models configured as incremental with proper `unique_key` and merge strategy?
+- [ ] **Ch 9 — Source definitions** — Are sources defined in YAML with `source()` macro? Are freshness checks configured?
+- [ ] **Ch 9 — Model materialization** — Are materializations appropriate (view for light transforms, table for heavy, incremental for large)?
+
+### Python Transforms
+- [ ] **Ch 9 — Appropriate tool** — Is Python used only when SQL is insufficient (ML, complex parsing, API calls)?
+- [ ] **Ch 9 — Vectorized operations** — Are pandas/numpy vectorized operations used instead of row-by-row iteration?
+- [ ] **Ch 9 — Memory management** — Are large datasets processed in chunks or with distributed frameworks (PySpark)?
+- [ ] **Ch 9 — Pure functions** — Are transformation functions pure (no side effects) and independently testable?
+
+---
+
+## 5. Data Validation & Testing (Chapter 10)
+
+### Validation Coverage
+- [ ] **Ch 10 — Schema validation** — Are column names, types, and count validated at ingestion?
+- [ ] **Ch 10 — Row count reconciliation** — Are source and destination row counts compared? Is threshold alerting configured?
+- [ ] **Ch 10 — Null checks** — Are NOT NULL constraints enforced on key columns? Are null percentages tracked?
+- [ ] **Ch 10 — Uniqueness checks** — Is primary key uniqueness verified after loading?
+- [ ] **Ch 10 — Referential integrity** — Are foreign key relationships validated between related tables?
+- [ ] **Ch 10 — Range validation** — Are values checked against expected ranges (dates in past, amounts positive, percentages 0-100)?
+- [ ] **Ch 10 — Freshness checks** — Is data freshness monitored? Are alerts configured for stale data?
+
+### Testing
+- [ ] **Ch 10 — Unit tests** — Are individual transformation functions tested with known inputs/outputs?
+- [ ] **Ch 10 — Integration tests** — Is the end-to-end pipeline tested with sample data?
+- [ ] **Ch 10 — dbt tests** — Are dbt schema tests (not_null, unique, relationships) defined? Are custom data tests used?
+- [ ] **Ch 10 — Regression tests** — Are current results compared against known-good baselines for critical tables?
+
+---
+
+## 6. Orchestration (Chapter 11)
+
+### DAG Design
+- [ ] **Ch 11 — One pipeline per DAG** — Are DAGs focused on a single pipeline? No mega-DAGs?
+- [ ] **Ch 11 — Task granularity** — Are tasks atomic and independently retryable? Not too fine or too coarse?
+- [ ] **Ch 11 — Shallow and wide** — Are DAGs shallow (few sequential steps) and wide (parallel where possible)?
+- [ ] **Ch 11 — No hardcoded dates** — Are dates parameterized using execution_date or equivalent? No hardcoded date strings?
+
+### Airflow Specifics
+- [ ] **Ch 11 — Idempotent tasks** — Can every task be safely re-run without data duplication or side effects?
+- [ ] **Ch 11 — Appropriate operators** — Are provider operators used where available instead of generic PythonOperator?
+- [ ] **Ch 11 — XCom usage** — Are XComs used only for small metadata (file paths, row counts), not large data?
+- [ ] **Ch 11 — Sensor timeouts** — Do sensors have timeouts to avoid indefinite waiting?
+- [ ] **Ch 11 — Error callbacks** — Are `on_failure_callback` configured for alerting on task failures?
+- [ ] **Ch 11 — Retries** — Are retries configured with appropriate delay for transient failures?
+- [ ] **Ch 11 — Pool limits** — Are pools used to limit concurrency for resource-constrained tasks?
+- [ ] **Ch 11 — Catchup configuration** — Is `catchup` set appropriately (True for backfill-supporting DAGs, False otherwise)?
+
+---
+
+## 7. Monitoring & Operations (Chapters 12–13)
+
+### Monitoring
+- [ ] **Ch 12 — Duration tracking** — Is pipeline execution time tracked and alerted on anomalies?
+- [ ] **Ch 12 — Row count monitoring** — Are rows processed per run tracked? Alerts on zero or unusual counts?
+- [ ] **Ch 12 — Error rate tracking** — Are failed records, retries, and exceptions monitored?
+- [ ] **Ch 12 — Data freshness SLA** — Are freshness SLAs defined and monitored? Alerts on breaches?
+- [ ] **Ch 12 — Resource monitoring** — Are CPU, memory, disk, and network usage tracked for pipeline infrastructure?
+
+### Alerting
+- [ ] **Ch 12 — Actionable alerts** — Do alerts include context (error message, affected table, run ID, link to logs)?
+- [ ] **Ch 12 — Severity levels** — Are alerts classified by severity with appropriate routing?
+- [ ] **Ch 12 — Alert fatigue prevention** — Are thresholds tuned to avoid noisy alerts? Are alerts deduplicated?
+
+### Operational Excellence
+- [ ] **Ch 13 — Idempotency** — Are all pipelines idempotent? Can they be re-run without data corruption?
+- [ ] **Ch 13 — Backfill support** — Are pipelines parameterized for date-range backfilling?
+- [ ] **Ch 13 — Error handling** — Are transient errors retried? Are permanent errors failed fast? Are bad records quarantined?
+- [ ] **Ch 13 — Credential security** — Are credentials in secrets managers, not in code or config files?
+- [ ] **Ch 13 — Data lineage** — Is source-to-destination mapping documented? Is transformation logic recorded?
+- [ ] **Ch 13 — Documentation** — Is there a README, data dictionary, and runbook for each pipeline?
+- [ ] **Ch 13 — Version control** — Is all pipeline code in git? Are changes reviewed via PR?
+
+---
+
+## Quick Review Workflow
+
+1. **Architecture pass** — Verify ETL/ELT choice, incremental vs full, infrastructure fit
+2. **Ingestion pass** — Check source-specific best practices, error handling, incremental logic
+3. **Loading pass** — Verify bulk loading, staging tables, partitioning, atomic operations
+4. **Transform pass** — Check SQL quality, dbt patterns, layer structure, determinism
+5. **Validation pass** — Verify data quality checks at each boundary, test coverage
+6. **Orchestration pass** — Check DAG design, idempotency, task granularity, error handling
+7. **Operations pass** — Verify monitoring, alerting, backfill support, documentation
+8. **Prioritize findings** — Rank by severity: data loss risk > data quality > performance > best practices > style
+
+## Severity Levels
+
+| Severity | Description | Example |
+|----------|-------------|---------|
+| **Critical** | Data loss, corruption, or security risk | Non-idempotent pipeline causing duplicates, hardcoded credentials, no staging tables with partial load risk, missing dead letter queue losing events |
+| **High** | Data quality or reliability issues | Missing validation, no error handling, full extraction on large tables, no monitoring or alerting, blocking on rate limits |
+| **Medium** | Performance, maintainability, or operational gaps | Missing partitioning, monolithic DAGs, no backfill support, missing documentation, no incremental models for large tables |
+| **Low** | Best practice improvements, optimization opportunities | Missing compression, suboptimal clustering, verbose logging, minor naming inconsistencies |

package/data-pipelines/scripts/example.py

@@ -0,0 +1 @@
+