@booklib/core 2.0.0

package/skills/data-pipelines/references/review-checklist.md

@@ -0,0 +1,181 @@
+# Data Pipelines Pocket Reference — Pipeline Review Checklist
+
+Systematic checklist for reviewing data pipelines against the 13 chapters
+from *Data Pipelines Pocket Reference* by James Densmore.
+
+---
+
+## 1. Architecture & Patterns (Chapters 1–3)
+
+### Infrastructure
+- [ ] **Ch 1-2 — Appropriate infrastructure** — Is the right storage chosen for the use case (warehouse for analytics, lake for raw/unstructured)?
+- [ ] **Ch 2 — Cloud-native services** — Are managed services used where appropriate to reduce operational burden?
+- [ ] **Ch 2 — Separation of storage and compute** — Is compute scaled independently from storage?
+
+### Pipeline Patterns
+- [ ] **Ch 3 — ETL vs ELT** — Is the right pattern chosen? Is ELT used for analytics workloads with modern warehouses?
+- [ ] **Ch 3 — Full vs incremental** — Is incremental extraction used for growing datasets? Is full extraction justified for small tables?
+- [ ] **Ch 3 — CDC where appropriate** — Is CDC used for real-time sync needs instead of polling?
+- [ ] **Ch 3 — Loading strategy** — Is the right load pattern used (append for events, upsert for dimensions, full refresh for small lookups)?
+
+---
+
+## 2. Data Ingestion (Chapters 4–7)
+
+### Database Ingestion (Ch 4)
+- [ ] **Ch 4 — Read replica usage** — Are extractions running against read replicas, not production databases?
+- [ ] **Ch 4 — Incremental column** — Is there a reliable timestamp or ID column for incremental extraction?
+- [ ] **Ch 4 — Connection management** — Are connections pooled and properly closed? Are timeouts configured?
+- [ ] **Ch 4 — Query efficiency** — Are only needed columns selected? Are WHERE clauses using indexed columns?
+- [ ] **Ch 4 — Large table handling** — Are large extractions chunked or using streaming cursors?
+
+### File Ingestion (Ch 5)
+- [ ] **Ch 5 — Schema validation** — Are file schemas validated before processing? Are malformed rows handled?
+- [ ] **Ch 5 — Encoding handling** — Is character encoding handled consistently (UTF-8)?
+- [ ] **Ch 5 — Cloud storage patterns** — Are files organized with partitioned prefixes? Are processed files archived?
+- [ ] **Ch 5 — File tracking** — Is there a mechanism to track which files have been processed to avoid reprocessing?
+- [ ] **Ch 5 — Compression** — Are files compressed for storage and transfer efficiency?
+
+### API Ingestion (Ch 6)
+- [ ] **Ch 6 — Pagination** — Is pagination implemented correctly? Are all pages fetched? Is cursor-based preferred?
+- [ ] **Ch 6 — Rate limiting** — Are rate limit headers respected? Is backoff implemented for 429 responses?
+- [ ] **Ch 6 — Retry logic** — Are transient errors (5xx, timeouts) retried with exponential backoff? Are 4xx errors not retried?
+- [ ] **Ch 6 — Authentication** — Are credentials stored securely? Are token refreshes handled?
+- [ ] **Ch 6 — Incremental fetching** — Are date/cursor parameters used to fetch only new data?
+
+### Streaming Ingestion (Ch 7)
+- [ ] **Ch 7 — Consumer group design** — Are consumer groups configured for parallel processing? Are rebalances handled?
+- [ ] **Ch 7 — Offset management** — Are offsets committed after successful processing, not before?
+- [ ] **Ch 7 — Serialization** — Are events serialized with a schema (Avro, Protobuf) for evolution support?
+- [ ] **Ch 7 — Dead letter queue** — Are failed events routed to a DLQ for inspection and reprocessing?
+- [ ] **Ch 7 — Exactly-once semantics** — Is deduplication implemented downstream or are idempotent producers used?
+- [ ] **Ch 7 — Backpressure** — Is backpressure handled when consumer can't keep up with producer?
+
+---
+
+## 3. Data Storage & Loading (Chapter 8)
+
+### Loading Patterns
+- [ ] **Ch 8 — Bulk loading** — Are bulk load commands used (COPY, load jobs) instead of row-by-row INSERT?
+- [ ] **Ch 8 — Staging tables** — Is data loaded to staging first, validated, then merged to production?
+- [ ] **Ch 8 — Atomic operations** — Are loads atomic? Is the destination never left in a partial state?
+- [ ] **Ch 8 — Data type mapping** — Are source types mapped correctly to destination types? No implicit conversions?
+
+### Table Design
+- [ ] **Ch 8 — Partitioning** — Are large tables partitioned by date or key? Are queries leveraging partition pruning?
+- [ ] **Ch 8 — Clustering** — Are frequently filtered columns used as cluster keys within partitions?
+- [ ] **Ch 8 — Sort/distribution keys** — For Redshift: are SORTKEY and DISTKEY chosen based on query patterns?
+
+### Warehouse-Specific
+- [ ] **Ch 8 — Redshift COPY** — Is S3-based COPY used instead of INSERT for bulk loads?
+- [ ] **Ch 8 — BigQuery load jobs** — Are load jobs preferred over streaming inserts for batch pipelines?
+- [ ] **Ch 8 — Snowflake stages** — Are stages used for file-based loading? Is SNOWPIPE configured for continuous loads?
+
+---
+
+## 4. Transformations (Chapter 9)
+
+### SQL Transforms
+- [ ] **Ch 9 — CTEs for readability** — Are Common Table Expressions used instead of deeply nested subqueries?
+- [ ] **Ch 9 — Window functions** — Are window functions used for ranking, running totals instead of self-joins?
+- [ ] **Ch 9 — Grain awareness** — Do joins maintain the correct grain? No accidental fan-out producing duplicate rows?
+- [ ] **Ch 9 — Deterministic logic** — Are transforms deterministic (same input → same output)? No reliance on row ordering?
+
+### dbt Patterns
+- [ ] **Ch 9 — Layer structure** — Are models organized into staging → intermediate → mart layers?
+- [ ] **Ch 9 — Staging models** — Are staging models 1:1 with sources? Do they rename, cast, and filter only?
+- [ ] **Ch 9 — Incremental models** — Are large models configured as incremental with proper `unique_key` and merge strategy?
+- [ ] **Ch 9 — Source definitions** — Are sources defined in YAML with `source()` macro? Are freshness checks configured?
+- [ ] **Ch 9 — Model materialization** — Are materializations appropriate (view for light transforms, table for heavy, incremental for large)?
+
+### Python Transforms
+- [ ] **Ch 9 — Appropriate tool** — Is Python used only when SQL is insufficient (ML, complex parsing, API calls)?
+- [ ] **Ch 9 — Vectorized operations** — Are pandas/numpy vectorized operations used instead of row-by-row iteration?
+- [ ] **Ch 9 — Memory management** — Are large datasets processed in chunks or with distributed frameworks (PySpark)?
+- [ ] **Ch 9 — Pure functions** — Are transformation functions pure (no side effects) and independently testable?
+
+---
+
+## 5. Data Validation & Testing (Chapter 10)
+
+### Validation Coverage
+- [ ] **Ch 10 — Schema validation** — Are column names, types, and count validated at ingestion?
+- [ ] **Ch 10 — Row count reconciliation** — Are source and destination row counts compared? Is threshold alerting configured?
+- [ ] **Ch 10 — Null checks** — Are NOT NULL constraints enforced on key columns? Are null percentages tracked?
+- [ ] **Ch 10 — Uniqueness checks** — Is primary key uniqueness verified after loading?
+- [ ] **Ch 10 — Referential integrity** — Are foreign key relationships validated between related tables?
+- [ ] **Ch 10 — Range validation** — Are values checked against expected ranges (dates in past, amounts positive, percentages 0-100)?
+- [ ] **Ch 10 — Freshness checks** — Is data freshness monitored? Are alerts configured for stale data?
+
+### Testing
+- [ ] **Ch 10 — Unit tests** — Are individual transformation functions tested with known inputs/outputs?
+- [ ] **Ch 10 — Integration tests** — Is the end-to-end pipeline tested with sample data?
+- [ ] **Ch 10 — dbt tests** — Are dbt schema tests (not_null, unique, relationships) defined? Are custom data tests used?
+- [ ] **Ch 10 — Regression tests** — Are current results compared against known-good baselines for critical tables?
+
+---
+
+## 6. Orchestration (Chapter 11)
+
+### DAG Design
+- [ ] **Ch 11 — One pipeline per DAG** — Are DAGs focused on a single pipeline? No mega-DAGs?
+- [ ] **Ch 11 — Task granularity** — Are tasks atomic and independently retryable? Not too fine or too coarse?
+- [ ] **Ch 11 — Shallow and wide** — Are DAGs shallow (few sequential steps) and wide (parallel where possible)?
+- [ ] **Ch 11 — No hardcoded dates** — Are dates parameterized using execution_date or equivalent? No hardcoded date strings?
+
+### Airflow Specifics
+- [ ] **Ch 11 — Idempotent tasks** — Can every task be safely re-run without data duplication or side effects?
+- [ ] **Ch 11 — Appropriate operators** — Are provider operators used where available instead of generic PythonOperator?
+- [ ] **Ch 11 — XCom usage** — Are XComs used only for small metadata (file paths, row counts), not large data?
+- [ ] **Ch 11 — Sensor timeouts** — Do sensors have timeouts to avoid indefinite waiting?
+- [ ] **Ch 11 — Error callbacks** — Are `on_failure_callback` configured for alerting on task failures?
+- [ ] **Ch 11 — Retries** — Are retries configured with appropriate delay for transient failures?
+- [ ] **Ch 11 — Pool limits** — Are pools used to limit concurrency for resource-constrained tasks?
+- [ ] **Ch 11 — Catchup configuration** — Is `catchup` set appropriately (True for backfill-supporting DAGs, False otherwise)?
+
+---
+
+## 7. Monitoring & Operations (Chapters 12–13)
+
+### Monitoring
+- [ ] **Ch 12 — Duration tracking** — Is pipeline execution time tracked and alerted on anomalies?
+- [ ] **Ch 12 — Row count monitoring** — Are rows processed per run tracked? Alerts on zero or unusual counts?
+- [ ] **Ch 12 — Error rate tracking** — Are failed records, retries, and exceptions monitored?
+- [ ] **Ch 12 — Data freshness SLA** — Are freshness SLAs defined and monitored? Alerts on breaches?
+- [ ] **Ch 12 — Resource monitoring** — Are CPU, memory, disk, and network usage tracked for pipeline infrastructure?
+
+### Alerting
+- [ ] **Ch 12 — Actionable alerts** — Do alerts include context (error message, affected table, run ID, link to logs)?
+- [ ] **Ch 12 — Severity levels** — Are alerts classified by severity with appropriate routing?
+- [ ] **Ch 12 — Alert fatigue prevention** — Are thresholds tuned to avoid noisy alerts? Are alerts deduplicated?
+
+### Operational Excellence
+- [ ] **Ch 13 — Idempotency** — Are all pipelines idempotent? Can they be re-run without data corruption?
+- [ ] **Ch 13 — Backfill support** — Are pipelines parameterized for date-range backfilling?
+- [ ] **Ch 13 — Error handling** — Are transient errors retried? Are permanent errors failed fast? Are bad records quarantined?
+- [ ] **Ch 13 — Credential security** — Are credentials in secrets managers, not in code or config files?
+- [ ] **Ch 13 — Data lineage** — Is source-to-destination mapping documented? Is transformation logic recorded?
+- [ ] **Ch 13 — Documentation** — Is there a README, data dictionary, and runbook for each pipeline?
+- [ ] **Ch 13 — Version control** — Is all pipeline code in git? Are changes reviewed via PR?
+
+---
+
+## Quick Review Workflow
+
+1. **Architecture pass** — Verify ETL/ELT choice, incremental vs full, infrastructure fit
+2. **Ingestion pass** — Check source-specific best practices, error handling, incremental logic
+3. **Loading pass** — Verify bulk loading, staging tables, partitioning, atomic operations
+4. **Transform pass** — Check SQL quality, dbt patterns, layer structure, determinism
+5. **Validation pass** — Verify data quality checks at each boundary, test coverage
+6. **Orchestration pass** — Check DAG design, idempotency, task granularity, error handling
+7. **Operations pass** — Verify monitoring, alerting, backfill support, documentation
+8. **Prioritize findings** — Rank by severity: data loss risk > data quality > performance > best practices > style
+
+## Severity Levels
+
+| Severity | Description | Example |
+|----------|-------------|---------|
+| **Critical** | Data loss, corruption, or security risk | Non-idempotent pipeline causing duplicates, hardcoded credentials, no staging tables with partial load risk, missing dead letter queue losing events |
+| **High** | Data quality or reliability issues | Missing validation, no error handling, full extraction on large tables, no monitoring or alerting, blocking on rate limits |
+| **Medium** | Performance, maintainability, or operational gaps | Missing partitioning, monolithic DAGs, no backfill support, missing documentation, no incremental models for large tables |
+| **Low** | Best practice improvements, optimization opportunities | Missing compression, suboptimal clustering, verbose logging, minor naming inconsistencies |

package/skills/data-pipelines/scripts/example.py

@@ -0,0 +1 @@
+

package/skills/data-pipelines/scripts/new_pipeline.py

@@ -0,0 +1,444 @@
+#!/usr/bin/env python3
+"""
+new_pipeline.py — Scaffold a new data pipeline with extract/transform/load structure.
+Usage: python new_pipeline.py <pipeline-name> [--source csv|api|db] [--target db|file|api]
+"""
+
+import argparse
+import os
+import sys
+from pathlib import Path
+from string import Template
+
+# ---------------------------------------------------------------------------
+# File templates
+# ---------------------------------------------------------------------------
+
+EXTRACT_CSV = '''\
+"""extract.py — Extract data from a CSV source."""
+
+import csv
+import logging
+import time
+from pathlib import Path
+from functools import wraps
+
+logger = logging.getLogger(__name__)
+
+
+def retry(max_attempts=3, delay=2.0, exceptions=(Exception,)):
+    """Retry decorator with exponential backoff."""
+    def decorator(fn):
+        @wraps(fn)
+        def wrapper(*args, **kwargs):
+            for attempt in range(1, max_attempts + 1):
+                try:
+                    return fn(*args, **kwargs)
+                except exceptions as exc:
+                    if attempt == max_attempts:
+                        raise
+                    wait = delay * (2 ** (attempt - 1))
+                    logger.warning("Attempt %d failed: %s. Retrying in %.1fs...", attempt, exc, wait)
+                    time.sleep(wait)
+        return wrapper
+    return decorator
+
+
+@retry(max_attempts=3, exceptions=(OSError,))
+def extract(source_path: str) -> list[dict]:
+    """Read rows from a CSV file. Returns a list of dicts."""
+    path = Path(source_path)
+    if not path.exists():
+        raise FileNotFoundError(f"Source file not found: {path}")
+    logger.info("Extracting from %s", path)
+    with path.open(newline="", encoding="utf-8") as fh:
+        reader = csv.DictReader(fh)
+        rows = list(reader)
+    logger.info("Extracted %d rows", len(rows))
+    return rows
+'''
+
+EXTRACT_API = '''\
+"""extract.py — Extract data from an HTTP API source."""
+
+import json
+import logging
+import time
+import urllib.error
+import urllib.request
+from functools import wraps
+
+logger = logging.getLogger(__name__)
+
+BASE_URL = "https://api.example.com/data"
+API_KEY = ""  # Set via environment variable in production
+
+
+def retry(max_attempts=3, delay=2.0, exceptions=(Exception,)):
+    """Retry decorator with exponential backoff."""
+    def decorator(fn):
+        @wraps(fn)
+        def wrapper(*args, **kwargs):
+            for attempt in range(1, max_attempts + 1):
+                try:
+                    return fn(*args, **kwargs)
+                except exceptions as exc:
+                    if attempt == max_attempts:
+                        raise
+                    wait = delay * (2 ** (attempt - 1))
+                    logger.warning("Attempt %d failed: %s. Retrying in %.1fs...", attempt, exc, wait)
+                    time.sleep(wait)
+        return wrapper
+    return decorator
+
+
+@retry(max_attempts=3, exceptions=(urllib.error.URLError, OSError))
+def extract(endpoint: str = BASE_URL) -> list[dict]:
+    """Fetch JSON records from an API endpoint. Returns a list of dicts."""
+    logger.info("Extracting from %s", endpoint)
+    req = urllib.request.Request(endpoint, headers={"Accept": "application/json"})
+    with urllib.request.urlopen(req, timeout=30) as response:
+        data = json.loads(response.read())
+    records = data if isinstance(data, list) else data.get("results", data.get("items", []))
+    logger.info("Extracted %d records", len(records))
+    return records
+'''
+
+EXTRACT_DB = '''\
+"""extract.py — Extract data from a database source."""
+
+import logging
+import sqlite3
+import time
+from functools import wraps
+
+logger = logging.getLogger(__name__)
+
+DB_PATH = "source.db"
+QUERY = "SELECT * FROM source_table"
+
+
+def retry(max_attempts=3, delay=2.0, exceptions=(Exception,)):
+    """Retry decorator with exponential backoff."""
+    def decorator(fn):
+        @wraps(fn)
+        def wrapper(*args, **kwargs):
+            for attempt in range(1, max_attempts + 1):
+                try:
+                    return fn(*args, **kwargs)
+                except exceptions as exc:
+                    if attempt == max_attempts:
+                        raise
+                    wait = delay * (2 ** (attempt - 1))
+                    logger.warning("Attempt %d failed: %s. Retrying in %.1fs...", attempt, exc, wait)
+                    time.sleep(wait)
+        return wrapper
+    return decorator
+
+
+@retry(max_attempts=3, exceptions=(sqlite3.OperationalError,))
+def extract(db_path: str = DB_PATH, query: str = QUERY) -> list[dict]:
+    """Query records from a SQLite database. Returns a list of dicts."""
+    logger.info("Connecting to %s", db_path)
+    conn = sqlite3.connect(db_path)
+    conn.row_factory = sqlite3.Row
+    try:
+        cursor = conn.execute(query)
+        rows = [dict(row) for row in cursor.fetchall()]
+    finally:
+        conn.close()
+    logger.info("Extracted %d rows", len(rows))
+    return rows
+'''
+
+TRANSFORM_TEMPLATE = '''\
+"""transform.py — Transform extracted records."""
+
+import logging
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+
+def _clean_record(record: dict[str, Any]) -> dict[str, Any]:
+    """Strip whitespace from string values and drop empty fields."""
+    cleaned = {}
+    for key, value in record.items():
+        if isinstance(value, str):
+            value = value.strip()
+        if value not in (None, "", []):
+            cleaned[key] = value
+    return cleaned
+
+
+def _validate_record(record: dict[str, Any]) -> bool:
+    """Return True if the record is valid. Customize required fields here."""
+    # TODO: add field-specific validation
+    return bool(record)
+
+
+def transform(records: list[dict]) -> list[dict]:
+    """Clean, validate, and reshape records for loading."""
+    logger.info("Transforming %d records", len(records))
+    output = []
+    skipped = 0
+    for record in records:
+        cleaned = _clean_record(record)
+        if not _validate_record(cleaned):
+            skipped += 1
+            continue
+        # TODO: add field mappings / enrichment here
+        output.append(cleaned)
+    if skipped:
+        logger.warning("Skipped %d invalid records", skipped)
+    logger.info("Transformed %d records", len(output))
+    return output
+'''
+
+LOAD_DB = '''\
+"""load.py — Idempotent load into a SQLite database using upsert."""
+
+import logging
+import sqlite3
+from typing import Any
+
+logger = logging.getLogger(__name__)
+
+DB_PATH = "output.db"
+TABLE = "$pipeline_name"
+# Define a unique key column used for upsert conflict detection
+UNIQUE_KEY = "id"
+
+
+def _ensure_table(conn: sqlite3.Connection, sample: dict[str, Any]) -> None:
+    columns = ", ".join(
+        f"{col} TEXT" if col != UNIQUE_KEY else f"{col} TEXT PRIMARY KEY"
+        for col in sample
+    )
+    conn.execute(f"CREATE TABLE IF NOT EXISTS {TABLE} ({columns})")
+    conn.commit()
+
+
+def load(records: list[dict]) -> int:
+    """Upsert records into SQLite. Returns number of rows written."""
+    if not records:
+        logger.info("No records to load.")
+        return 0
+    logger.info("Loading %d records into %s:%s", len(records), DB_PATH, TABLE)
+    conn = sqlite3.connect(DB_PATH)
+    try:
+        _ensure_table(conn, records[0])
+        cols = ", ".join(records[0].keys())
+        placeholders = ", ".join("?" for _ in records[0])
+        sql = (
+            f"INSERT OR REPLACE INTO {TABLE} ({cols}) VALUES ({placeholders})"
+        )
+        conn.executemany(sql, [list(r.values()) for r in records])
+        conn.commit()
+    finally:
+        conn.close()
+    logger.info("Loaded %d records", len(records))
+    return len(records)
+'''
+
+LOAD_FILE = '''\
+"""load.py — Write records to a CSV or JSON file (idempotent by overwrite)."""
+
+import csv
+import json
+import logging
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+OUTPUT_PATH = "$pipeline_name_output.csv"
+
+
+def load(records: list[dict], output_path: str = OUTPUT_PATH) -> int:
+    """Write records to a file. Overwrites to ensure idempotency."""
+    if not records:
+        logger.info("No records to load.")
+        return 0
+    path = Path(output_path)
+    path.parent.mkdir(parents=True, exist_ok=True)
+    if path.suffix == ".json":
+        path.write_text(json.dumps(records, indent=2, default=str), encoding="utf-8")
+    else:
+        with path.open("w", newline="", encoding="utf-8") as fh:
+            writer = csv.DictWriter(fh, fieldnames=records[0].keys())
+            writer.writeheader()
+            writer.writerows(records)
+    logger.info("Wrote %d records to %s", len(records), path)
+    return len(records)
+'''
+
+LOAD_API = '''\
+"""load.py — POST records to an API endpoint (idempotent with dedup key)."""
+
+import json
+import logging
+import urllib.error
+import urllib.request
+
+logger = logging.getLogger(__name__)
+
+TARGET_URL = "https://api.example.com/ingest"
+BATCH_SIZE = 100
+
+
+def _post_batch(batch: list[dict]) -> None:
+    payload = json.dumps(batch).encode("utf-8")
+    req = urllib.request.Request(
+        TARGET_URL,
+        data=payload,
+        method="POST",
+        headers={"Content-Type": "application/json"},
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            status = resp.status
+            logger.info("Batch of %d posted — HTTP %d", len(batch), status)
+    except urllib.error.HTTPError as exc:
+        logger.error("HTTP error %d posting batch: %s", exc.code, exc.reason)
+        raise
+
+
+def load(records: list[dict]) -> int:
+    """POST records in batches. Returns total records sent."""
+    if not records:
+        logger.info("No records to load.")
+        return 0
+    total = 0
+    for i in range(0, len(records), BATCH_SIZE):
+        batch = records[i:i + BATCH_SIZE]
+        _post_batch(batch)
+        total += len(batch)
+    logger.info("Loaded %d records via API", total)
+    return total
+'''
+
+PIPELINE_TEMPLATE = '''\
+"""pipeline.py — Orchestrator: extract → transform → load."""
+
+import logging
+import sys
+import time
+
+from extract import extract
+from transform import transform
+from load import load
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s  %(levelname)-8s  %(name)s  %(message)s",
+    datefmt="%Y-%m-%dT%H:%M:%S",
+)
+logger = logging.getLogger("$pipeline_name")
+
+
+def run() -> int:
+    """Run the full pipeline. Returns exit code (0=success, 1=failure)."""
+    start = time.monotonic()
+    logger.info("Pipeline '$pipeline_name' starting")
+    try:
+        raw = extract()
+        records = transform(raw)
+        count = load(records)
+        elapsed = time.monotonic() - start
+        logger.info(
+            "Pipeline complete — %d records loaded in %.2fs", count, elapsed
+        )
+        return 0
+    except Exception as exc:
+        logger.exception("Pipeline failed: %s", exc)
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(run())
+'''
+
+REQUIREMENTS_TEMPLATE = '''\
+# Runtime dependencies for $pipeline_name pipeline
+# Add your project-specific packages below.
+
+# Uncomment as needed:
+# requests>=2.31          # for API sources/targets
+# psycopg2-binary>=2.9    # for PostgreSQL
+# pymysql>=1.1            # for MySQL
+# pandas>=2.0             # for complex transformations
+# pydantic>=2.0           # for record validation
+'''
+
+# ---------------------------------------------------------------------------
+# Source/target template selection
+# ---------------------------------------------------------------------------
+
+EXTRACT_TEMPLATES = {"csv": EXTRACT_CSV, "api": EXTRACT_API, "db": EXTRACT_DB}
+LOAD_TEMPLATES = {"db": LOAD_DB, "file": LOAD_FILE, "api": LOAD_API}
+
+
+def render(template_str: str, pipeline_name: str) -> str:
+    safe_name = pipeline_name.replace("-", "_")
+    return Template(template_str).safe_substitute(pipeline_name=safe_name)
+
+
+def create_pipeline(name: str, source: str, target: str) -> None:
+    base = Path(name)
+    if base.exists():
+        print(f"Error: directory '{base}' already exists. Choose a different name.")
+        sys.exit(1)
+    base.mkdir(parents=True)
+
+    files = {
+        "extract.py": render(EXTRACT_TEMPLATES[source], name),
+        "transform.py": render(TRANSFORM_TEMPLATE, name),
+        "load.py": render(LOAD_TEMPLATES[target], name),
+        "pipeline.py": render(PIPELINE_TEMPLATE, name),
+        "requirements.txt": render(REQUIREMENTS_TEMPLATE, name),
+    }
+
+    created = []
+    for filename, content in files.items():
+        path = base / filename
+        path.write_text(content, encoding="utf-8")
+        created.append(str(path))
+
+    print(f"\nPipeline '{name}' created successfully!\n")
+    print(f"  Source  : {source}")
+    print(f"  Target  : {target}")
+    print(f"\nFiles created:")
+    for f in created:
+        print(f"  {f}")
+    print(f"\nNext steps:")
+    print(f"  1. cd {name}")
+    print(f"  2. Review extract.py and update source configuration")
+    print(f"  3. Customize transform.py with your business logic")
+    print(f"  4. Review load.py and configure target destination")
+    print(f"  5. pip install -r requirements.txt   # add packages as needed")
+    print(f"  6. python pipeline.py")
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Scaffold a new data pipeline (extract → transform → load)"
+    )
+    parser.add_argument("name", help="Pipeline name (used as directory name)")
+    parser.add_argument(
+        "--source",
+        choices=["csv", "api", "db"],
+        default="csv",
+        help="Data source type (default: csv)",
+    )
+    parser.add_argument(
+        "--target",
+        choices=["db", "file", "api"],
+        default="db",
+        help="Data target type (default: db)",
+    )
+    args = parser.parse_args()
+    create_pipeline(args.name, args.source, args.target)
+
+
+if __name__ == "__main__":
+    main()