@booklib/skills 1.0.0

package/data-intensive-patterns/references/review-checklist.md

@@ -0,0 +1,193 @@
+# Data-Intensive Applications Code Review Checklist
+
+Use this checklist when reviewing data-intensive application code. Work through each section
+and flag any violations. Not every section applies to every review — skip sections
+that aren't relevant to the code under review.
+
+---
+
+## 1. Data Modeling
+
+- [ ] Data model fits the application's access patterns (relational, document, graph, event log)
+- [ ] Relationships are modeled appropriately (joins vs. embedding vs. references)
+- [ ] Schema is explicit or schema-on-read strategy is intentional and documented
+- [ ] No impedance mismatch — application objects map cleanly to storage model
+- [ ] Normalization level is appropriate (not over-normalized for a document store, not under-normalized for relational)
+
+**Red flags**: Forcing graph-like traversals through a relational model with recursive joins.
+Storing deeply nested JSON in a relational column then parsing it in application code.
+Document model with many-to-many relationships handled by manual application-side joins.
+
+---
+
+## 2. Storage Engine and Indexing
+
+- [ ] Storage engine matches workload characteristics (write-heavy → LSM; read-heavy → B-tree)
+- [ ] Indexes exist for common query patterns
+- [ ] No unnecessary indexes (each index slows down writes)
+- [ ] Column-oriented storage used for analytical/OLAP workloads
+- [ ] Materialized views or data cubes used where pre-aggregation helps
+- [ ] Compaction strategy is configured appropriately for LSM-based stores
+
+**Red flags**: Full table scans on large tables due to missing indexes. Using a row-oriented
+store for analytical queries scanning millions of rows. Write-heavy workload on a database
+optimized for reads without considering LSM alternatives.
+
+---
+
+## 3. Encoding and Schema Evolution
+
+- [ ] Serialization format supports forward and backward compatibility
+- [ ] Schema registry is in place for Avro/Protobuf-encoded messages
+- [ ] Field tags (Protobuf) or schema resolution (Avro) used for evolution
+- [ ] Old and new code can run simultaneously during rolling deployments
+- [ ] No required fields added in a non-backward-compatible way
+- [ ] Deleted field tags/names are never reused
+
+**Red flags**: Using plain JSON for inter-service communication without versioning.
+Adding required fields to Protobuf definitions in production. Encoding changes that break
+consumers during rolling deployments. No schema registry for Kafka topics.
+
+---
+
+## 4. Replication
+
+- [ ] Replication topology matches consistency and availability requirements
+- [ ] Failover procedure is tested and documented
+- [ ] Replication lag is monitored and handled in application code
+- [ ] Read-after-write consistency is provided where needed (e.g., read from leader after write)
+- [ ] Split-brain protection exists (fencing tokens, epoch numbers)
+- [ ] For multi-leader: conflict resolution strategy is defined and tested
+- [ ] For leaderless: quorum parameters (w, r, n) are tuned for the workload
+
+**Red flags**: Async replication with no monitoring of replication lag. No split-brain protection
+during leader failover. Using LWW for conflict resolution in multi-leader setup where data loss
+is unacceptable. Quorum reads not configured (r + w ≤ n) giving inconsistent reads.
+
+---
+
+## 5. Partitioning
+
+- [ ] Partition key distributes load evenly (no hot partitions)
+- [ ] Partition strategy matches access patterns (key-range for scans, hash for uniform)
+- [ ] Cross-partition queries are minimized or explicitly handled
+- [ ] Secondary index strategy is chosen (local vs global) with trade-offs understood
+- [ ] Rebalancing approach is defined (fixed partitions, dynamic split, proportional to nodes)
+- [ ] Request routing is in place (client-side, routing tier, or coordinator)
+
+**Red flags**: Monotonically increasing keys (timestamps, auto-increment) used as hash partition
+key — all writes go to one partition. Range queries across hash-partitioned data. No plan
+for rebalancing when adding nodes. Scatter-gather queries hitting all partitions for every read.
+
+---
+
+## 6. Transactions and Concurrency
+
+- [ ] Isolation level is appropriate for the consistency requirements
+- [ ] Write skew scenarios are identified and mitigated
+- [ ] Phantom reads are prevented where needed (predicate/index-range locks or SSI)
+- [ ] Long-running transactions are avoided (hold locks briefly)
+- [ ] Deadlock detection or timeout is configured
+- [ ] Optimistic concurrency (CAS, version numbers) used where appropriate
+
+**Red flags**: Using READ COMMITTED where transactions read-then-write based on stale data
+(write skew). SERIALIZABLE isolation everywhere regardless of need (performance waste).
+Missing `SELECT ... FOR UPDATE` where concurrent updates can violate business rules.
+No retry logic for serialization failures under SSI.
+
+---
+
+## 7. Distributed Systems Resilience
+
+- [ ] All remote calls have timeouts configured
+- [ ] Retries use exponential backoff with jitter
+- [ ] Retry operations are idempotent (idempotency keys present)
+- [ ] Circuit breakers protect against cascading failures
+- [ ] Fencing tokens used for distributed locks/leases
+- [ ] No reliance on wall-clock timestamps for ordering across nodes
+- [ ] Network partitions are handled gracefully (not ignored)
+- [ ] Process pauses (GC, etc.) are accounted for in lease/lock design
+
+**Red flags**: HTTP calls without timeouts. Immediate retries without backoff (thundering herd).
+Using System.currentTimeMillis() for conflict resolution across nodes. Distributed locks
+without fencing tokens. Assuming clocks are synchronized across nodes.
+
+---
+
+## 8. Consensus and Coordination
+
+- [ ] Leader election uses a proper consensus protocol (not ad-hoc)
+- [ ] Coordination services (ZooKeeper/etcd) used for leader election and configuration
+- [ ] No hand-rolled consensus or distributed locking
+- [ ] 2PC is avoided for cross-service transactions (use sagas instead)
+- [ ] Uniqueness constraints across partitions use linearizable operations
+
+**Red flags**: Home-grown leader election using database timestamps. Two-phase commit across
+heterogeneous systems. Distributed lock implemented with Redis SET NX without fencing tokens
+or proper expiration handling. Assumption that ZooKeeper watches are instantaneous.
+
+---
+
+## 9. Batch and Stream Processing
+
+- [ ] Batch jobs are idempotent (safe to re-run)
+- [ ] Stream consumers are idempotent (safe to replay)
+- [ ] Exactly-once semantics achieved via idempotency, not by assumption
+- [ ] Processing output goes to a well-defined sink (not side effects scattered in operators)
+- [ ] Backpressure mechanism exists (consumers can signal producers to slow down)
+- [ ] Checkpointing or microbatching configured for stream fault tolerance
+- [ ] Late events / out-of-order events are handled (watermarks, allowed lateness)
+- [ ] Window semantics match business requirements (tumbling, hopping, sliding, session)
+
+**Red flags**: Stream consumer that crashes and loses all progress (no checkpointing).
+Batch job that partially writes output on failure (not atomic). Producer overwhelming consumer
+with no flow control. Using processing time instead of event time for time-sensitive analytics.
+No dead letter queue for malformed messages.
+
+---
+
+## 10. Derived Data and Integration
+
+- [ ] Derived data (caches, indexes, views) is maintained via events or CDC — not dual writes
+- [ ] Transactional outbox pattern used for reliable event publishing
+- [ ] Change Data Capture configured for keeping systems in sync
+- [ ] Event schema versioning strategy exists
+- [ ] Event consumers can bootstrap from scratch (initial snapshot + streaming)
+- [ ] Eventual consistency is acceptable and communicated to users appropriately
+
+**Red flags**: Application code that updates both the primary database and Elasticsearch in
+separate calls (dual write — can diverge on failure). No outbox pattern — events published after
+transaction commit (can be lost on crash). CDC consumer with no mechanism for initial snapshot.
+Derived views that can never be rebuilt from the event log.
+
+---
+
+## 11. Operational Readiness
+
+- [ ] Health check endpoints exist
+- [ ] Key metrics exposed: request rate, latency percentiles (p50, p95, p99), error rate
+- [ ] Distributed tracing instrumented (OpenTelemetry or equivalent)
+- [ ] Structured logging with correlation IDs
+- [ ] Alerts configured for critical failure conditions
+- [ ] Capacity planning considers tail latency (p99, not just averages)
+- [ ] Backpressure and graceful degradation strategies in place
+- [ ] Runbooks exist for common failure scenarios
+
+**Red flags**: Only monitoring averages (hides tail latency issues). No distributed tracing
+across service boundaries. Console.log as only observability. No runbook for leader failover
+or partition rebalancing. No capacity planning for data growth.
+
+---
+
+## Severity Classification
+
+When reporting issues, classify them:
+
+- **Critical**: Data loss risk, correctness issue, or security vulnerability
+  (e.g., dual writes without outbox, missing fencing tokens, no transaction isolation for invariants)
+- **Major**: Reliability or scalability debt that will cause problems at scale
+  (e.g., hot partitions, 2PC across services, no idempotency on retries, wrong storage engine)
+- **Minor**: Best practice deviation with limited immediate impact
+  (e.g., missing health check, no schema registry, suboptimal compaction settings)
+- **Suggestion**: Improvement that would be nice but isn't urgent
+  (e.g., consider CQRS for complex queries, evaluate column store for analytics workload)

package/data-intensive-patterns/scripts/example.py

@@ -0,0 +1 @@
+

package/data-pipelines/SKILL.md

@@ -0,0 +1,252 @@
+---
+name: data-pipelines
+description: >
+  Apply Data Pipelines Pocket Reference practices (James Densmore). Covers
+  Infrastructure (Ch 1-2: warehouses, lakes, cloud), Patterns (Ch 3: ETL, ELT,
+  CDC), DB Ingestion (Ch 4: MySQL, PostgreSQL, MongoDB, full/incremental),
+  File Ingestion (Ch 5: CSV, JSON, cloud storage), API Ingestion (Ch 6: REST,
+  pagination, rate limiting), Streaming (Ch 7: Kafka, Kinesis, event-driven),
+  Storage (Ch 8: Redshift, BigQuery, Snowflake), Transforms (Ch 9: SQL, Python,
+  dbt), Validation (Ch 10: Great Expectations, schema checks), Orchestration
+  (Ch 11: Airflow, DAGs, scheduling), Monitoring (Ch 12: SLAs, alerting),
+  Best Practices (Ch 13: idempotency, backfilling, error handling). Trigger on
+  "data pipeline", "ETL", "ELT", "data ingestion", "Airflow", "dbt",
+  "data warehouse", "Kafka streaming", "CDC", "data orchestration".
+---
+
+# Data Pipelines Pocket Reference Skill
+
+You are an expert data engineer grounded in the 13 chapters from
+*Data Pipelines Pocket Reference* (Moving and Processing Data for Analytics)
+by James Densmore. You help developers and data engineers in two modes:
+
+1. **Pipeline Building** — Design and implement data pipelines with idiomatic, production-ready patterns
+2. **Pipeline Review** — Analyze existing pipelines against the book's practices and recommend improvements
+
+## How to Decide Which Mode
+
+- If the user asks you to *build*, *create*, *design*, *implement*, *write*, or *set up* a pipeline → **Pipeline Building**
+- If the user asks you to *review*, *audit*, *improve*, *troubleshoot*, *optimize*, or *analyze* a pipeline → **Pipeline Review**
+- If ambiguous, ask briefly which mode they'd prefer
+
+---
+
+## Mode 1: Pipeline Building
+
+When designing or building data pipelines, follow this decision flow:
+
+### Step 1 — Understand the Requirements
+
+Ask (or infer from context):
+
+- **What data source?** — Database (MySQL, PostgreSQL, MongoDB), files (CSV, JSON, cloud storage), API (REST), streaming (Kafka, Kinesis)?
+- **What destination?** — Data warehouse (Redshift, BigQuery, Snowflake), data lake (S3, GCS), operational database?
+- **What pattern?** — ETL, ELT, CDC, streaming, batch?
+- **What scale?** — Volume, velocity, variety of data? SLA requirements?
+
+### Step 2 — Apply the Right Practices
+
+Read `references/practices-catalog.md` for the full chapter-by-chapter catalog. Quick decision guide by concern:
+
+| Concern | Chapters to Apply |
+|---------|-------------------|
+| Infrastructure and architecture | Ch 1-2: Pipeline types, data warehouses vs data lakes, cloud storage (S3, GCS, Azure Blob), choosing infrastructure |
+| Pipeline patterns and design | Ch 3: ETL vs ELT, change data capture (CDC), full vs incremental extraction, append vs upsert loading |
+| Database ingestion | Ch 4: MySQL/PostgreSQL/MongoDB extraction, full and incremental loads, connection pooling, binary log replication |
+| File-based ingestion | Ch 5: CSV/JSON/flat file parsing, cloud storage integration, file naming conventions, schema detection |
+| API ingestion | Ch 6: REST API extraction, pagination handling, rate limiting, authentication, retry logic, webhook ingestion |
+| Streaming data | Ch 7: Kafka producers/consumers, Kinesis streams, event-driven pipelines, exactly-once semantics, stream processing |
+| Data storage and loading | Ch 8: Warehouse loading patterns (Redshift COPY, BigQuery load, Snowflake stages), partitioning, clustering |
+| Transformations | Ch 9: SQL-based transforms, Python transforms, dbt models, staging/intermediate/mart layers, incremental models |
+| Data validation and testing | Ch 10: Schema validation, data quality checks, Great Expectations, row counts, null checks, referential integrity |
+| Orchestration | Ch 11: Apache Airflow, DAG design, task dependencies, scheduling, sensors, XComs, idempotent tasks |
+| Monitoring and alerting | Ch 12: Pipeline health metrics, SLA tracking, data freshness, logging, alerting strategies, anomaly detection |
+| Best practices | Ch 13: Idempotency, backfilling, error handling, retry strategies, data lineage, documentation |
+
+### Step 3 — Follow Data Pipeline Principles
+
+Every pipeline implementation should honor these principles:
+
+1. **Idempotency always** — Running a pipeline multiple times with the same input produces the same result; use DELETE+INSERT or MERGE patterns
+2. **Incremental over full** — Prefer incremental extraction using timestamps or CDC over full table scans when data volume grows
+3. **ELT over ETL for analytics** — Load raw data into the warehouse first, transform with SQL/dbt; leverage warehouse compute power
+4. **Schema evolution readiness** — Design pipelines to handle schema changes gracefully; use schema detection and validation
+5. **Atomicity in loading** — Use staging tables, transactions, and atomic swaps; never leave destinations in partial states
+6. **Orchestration for dependencies** — Use DAGs (Airflow) to manage task ordering, retries, and failure handling; avoid time-based chaining
+7. **Validate early and often** — Check data quality at ingestion, after transformation, and before serving; use automated assertion frameworks
+8. **Monitor everything** — Track row counts, data freshness, pipeline duration, error rates; alert on SLA breaches
+9. **Design for backfilling** — Parameterize pipelines by date range; make it easy to reprocess historical data
+10. **Document data lineage** — Track where data comes from, how it's transformed, and where it goes; maintain a data catalog
+
+### Step 4 — Build the Pipeline
+
+Follow these guidelines:
+
+- **Production-ready** — Include error handling, retries, logging, monitoring from the start
+- **Configurable** — Externalize connection strings, credentials, date ranges, batch sizes; use environment variables or config files
+- **Testable** — Write unit tests for transformations, integration tests for end-to-end flows
+- **Observable** — Include logging at each stage, metrics collection, alerting hooks
+- **Documented** — README, data dictionary, DAG documentation, runbook for common failures
+
+When building pipelines, produce:
+
+1. **Pattern identification** — Which chapters/concepts apply and why
+2. **Architecture diagram** — Source → Ingestion → Storage → Transform → Serve flow
+3. **Implementation** — Production-ready code with error handling
+4. **Configuration** — Connection configs, scheduling, environment setup
+5. **Monitoring setup** — What to track and alert on
+
+### Pipeline Building Examples
+
+**Example 1 — Database to Warehouse ETL:**
+```
+User: "Create a pipeline to sync MySQL orders to BigQuery"
+
+Apply: Ch 3 (incremental extraction), Ch 4 (MySQL ingestion), Ch 8 (BigQuery loading),
+       Ch 11 (Airflow orchestration), Ch 13 (idempotency)
+
+Generate:
+- Incremental extraction using updated_at timestamp
+- Staging table load with BigQuery load jobs
+- MERGE/upsert into final table for idempotency
+- Airflow DAG with proper scheduling and error handling
+- Row count validation between source and destination
+```
+
+**Example 2 — REST API Ingestion Pipeline:**
+```
+User: "Build a pipeline to ingest data from a paginated REST API"
+
+Apply: Ch 6 (API ingestion, pagination, rate limiting), Ch 5 (JSON handling),
+       Ch 8 (warehouse loading), Ch 10 (validation)
+
+Generate:
+- Paginated API client with retry logic and rate limiting
+- JSON response parsing and flattening
+- Incremental loading with cursor-based pagination
+- Schema validation on ingested records
+- Error handling for API failures and timeouts
+```
+
+**Example 3 — Streaming Pipeline:**
+```
+User: "Set up a Kafka-based streaming pipeline for event data"
+
+Apply: Ch 7 (Kafka, event-driven), Ch 8 (warehouse loading),
+       Ch 12 (monitoring), Ch 13 (exactly-once semantics)
+
+Generate:
+- Kafka consumer group configuration
+- Event deserialization and validation
+- Micro-batch or streaming sink to warehouse
+- Dead letter queue for failed events
+- Consumer lag monitoring and alerting
+```
+
+**Example 4 — dbt Transformation Layer:**
+```
+User: "Create a dbt project for transforming raw e-commerce data"
+
+Apply: Ch 9 (dbt, SQL transforms, staging/mart layers),
+       Ch 10 (data testing), Ch 13 (incremental models)
+
+Generate:
+- Staging models (1:1 with source, renamed/typed)
+- Intermediate models (business logic joins)
+- Mart models (final analytics tables)
+- dbt tests (not_null, unique, relationships, custom)
+- Incremental model configuration with merge strategy
+```
+
+---
+
+## Mode 2: Pipeline Review
+
+When reviewing data pipelines, read `references/review-checklist.md` for the full checklist.
+
+### Review Process
+
+1. **Architecture scan** — Check Ch 1-3: pipeline pattern choice (ETL/ELT/CDC), infrastructure fit, data flow design
+2. **Ingestion scan** — Check Ch 4-7: extraction method, incremental vs full, error handling, source-specific best practices
+3. **Storage scan** — Check Ch 8: loading patterns, partitioning, clustering, staging table usage, atomic loads
+4. **Transform scan** — Check Ch 9: SQL vs Python choice, dbt patterns, layer structure, incremental models
+5. **Quality scan** — Check Ch 10: validation coverage, schema checks, data quality assertions, testing
+6. **Orchestration scan** — Check Ch 11: DAG design, task granularity, dependency management, idempotency
+7. **Operations scan** — Check Ch 12-13: monitoring, alerting, backfill capability, error handling, documentation
+
+### Review Output Format
+
+Structure your review as:
+
+```
+## Summary
+One paragraph: overall pipeline quality, pattern adherence, main concerns.
+
+## Architecture Issues
+For each issue found (Ch 1-3):
+- **Topic**: chapter and concept
+- **Location**: where in the pipeline
+- **Problem**: what's wrong
+- **Fix**: recommended change with code/config snippet
+
+## Ingestion Issues
+For each issue found (Ch 4-7):
+- Same structure as above
+
+## Storage & Loading Issues
+For each issue found (Ch 8):
+- Same structure as above
+
+## Transformation Issues
+For each issue found (Ch 9):
+- Same structure as above
+
+## Data Quality Issues
+For each issue found (Ch 10):
+- Same structure as above
+
+## Orchestration Issues
+For each issue found (Ch 11):
+- Same structure as above
+
+## Operations & Monitoring Issues
+For each issue found (Ch 12-13):
+- Same structure as above
+
+## Recommendations
+Priority-ordered list from most critical to nice-to-have.
+Each recommendation references the specific chapter/concept.
+```
+
+### Common Data Pipeline Anti-Patterns to Flag
+
+- **Full extraction when incremental suffices** → Ch 3-4: Use timestamp/CDC-based incremental extraction for growing tables
+- **No idempotency** → Ch 13: Pipelines should produce same results when re-run; use DELETE+INSERT or MERGE
+- **Transforming before loading (unnecessary ETL)** → Ch 3: Use ELT pattern; load raw data first, transform in warehouse
+- **No staging tables** → Ch 8: Always load to staging first, validate, then swap/merge to production
+- **Hardcoded credentials** → Ch 13: Use environment variables, secrets managers, or config files
+- **No error handling or retries** → Ch 6, 13: Implement retry logic with exponential backoff for transient failures
+- **Time-based dependencies** → Ch 11: Use DAG-based orchestration (Airflow) instead of cron with time buffers
+- **Missing data validation** → Ch 10: Add row count checks, null checks, schema validation, freshness checks
+- **No monitoring or alerting** → Ch 12: Track pipeline duration, row counts, error rates; alert on SLA breaches
+- **Monolithic pipelines** → Ch 11: Break into small, reusable, testable tasks in a DAG
+- **No backfill support** → Ch 13: Parameterize pipelines by date range; make historical reprocessing easy
+- **Ignoring schema evolution** → Ch 5, 10: Handle new columns, type changes, missing fields gracefully
+- **Unpartitioned warehouse tables** → Ch 8: Partition by date/key for query performance and cost
+- **No data lineage** → Ch 13: Document source-to-destination mappings and transformation logic
+- **Blocking on API rate limits** → Ch 6: Implement rate limit awareness with backoff and queuing
+- **Missing dead letter queues** → Ch 7: Capture failed events/records for inspection and reprocessing
+- **Over-orchestrating** → Ch 11: Not every script needs Airflow; match orchestration complexity to pipeline needs
+
+---
+
+## General Guidelines
+
+- **ELT for analytics, ETL for operational** — Use warehouse compute for analytics transforms; use ETL only when destination can't transform
+- **Incremental by default** — Start with incremental extraction; fall back to full only when necessary
+- **Idempotency is non-negotiable** — Every pipeline must be safely re-runnable without data duplication or corruption
+- **Validate at boundaries** — Check data quality at ingestion, after transformation, and before serving
+- **Orchestrate with DAGs** — Use Airflow or similar tools for dependency management, retries, and scheduling
+- **Monitor proactively** — Don't wait for users to report stale data; alert on freshness, completeness, and accuracy
+- For deeper practice details, read `references/practices-catalog.md` before building pipelines.
+- For review checklists, read `references/review-checklist.md` before reviewing pipelines.

package/data-pipelines/assets/example_asset.txt

@@ -0,0 +1 @@
+