@orchestrator-claude/definitions 3.32.2 → 3.34.0

package/skills/python-data-engineering/SKILL.md

@@ -0,0 +1,792 @@
+---
+name: python-data-engineering
+description: Python data engineering patterns para ETL pipelines, schema validators, fuzzy matchers, Excel generators. Use when implementing Python+pandas+rapidfuzz+openpyxl+azure-storage-blob projects. Cobre pytest discipline, mypy --strict, ruff config, Pydantic v2, pandas idioms, fuzzy matching tuning, Azure SDK Python, structured logging, government CSV parsing. Não cobre application backends Python web (FastAPI/Django) — usar /backend skills para isso.
+---
+
+# Python Data Engineering — Patterns and Conventions
+
+This skill is the **canonical reference** for Python data engineering work in this org. It is loaded by the `python-data-engineer` agent and is also available to any other agent dispatching Python+data tasks.
+
+For deep pandas reference beyond what's here, see also: <https://jeffallan.github.io/claude-skills/skills/data-ml/pandas-pro/> (external skill, not maintained by this org).
+
+## 1. Project Conventions
+
+### Project layout (pyproject.toml-based)
+
+```
+project-name/
+├── pyproject.toml              # PEP 517/518 — single source for deps, tools config
+├── .gitignore                  # blocks *.xlsx, .cache/, .env, __pycache__/, .mypy_cache/
+├── src/
+│   └── package_name/           # snake_case package
+│       ├── __init__.py
+│       └── module.py
+├── tests/
+│   ├── conftest.py             # shared fixtures
+│   ├── unit/
+│   ├── integration/
+│   └── acceptance/
+└── docs/
+```
+
+**Why `src/` layout** (not flat): forces install-then-test pattern, prevents accidental imports from working tree, mypy + ruff treat correctly.
+
+### `pyproject.toml` baseline
+
+```toml
+[project]
+name = "package-name"
+version = "0.1.0"
+requires-python = ">=3.11"
+dependencies = [
+    "pandas>=2.1",
+    "pydantic>=2.5",
+    "pydantic-settings>=2.1",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.4",
+    "pytest-cov>=4.1",
+    "pytest-mock>=3.12",
+    "ruff>=0.1",
+    "mypy>=1.7",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "UP", "B", "SIM", "RUF"]
+# E/F: pycodestyle errors + pyflakes
+# I: isort
+# N: pep8-naming
+# UP: pyupgrade
+# B: bugbear (common bug patterns)
+# SIM: simplify
+# RUF: ruff-specific
+
+[tool.mypy]
+strict = true
+python_version = "3.11"
+warn_return_any = true
+warn_unused_ignores = true
+disallow_untyped_defs = true
+disallow_any_explicit = false  # allow Any explicitly only when annotated
+
+[tool.pytest.ini_options]
+markers = [
+    "integration: requires external services (Azurite, etc.)",
+    "acceptance: requires real data — KPI gates",
+]
+addopts = "--strict-markers --cov-fail-under=80"
+```
+
+**Rule**: never disable a ruff lint rule globally without comment. If needed, `# noqa: <code>  # reason: …` per-line.
+
+## 2. TDD with pytest
+
+### Test layout
+
+| Type | Path | Marker | Runs in CI? |
+|---|---|---|---|
+| Unit | `tests/unit/test_<module>.py` | (none) | yes, fast |
+| Integration | `tests/integration/test_<feature>.py` | `@pytest.mark.integration` | yes (with services) |
+| Acceptance | `tests/acceptance/test_kpi_<name>.py` | `@pytest.mark.acceptance` | no — manual only |
+
+### Test function naming
+
+```python
+def test_<unit>_<condition>_<expected_behavior>() -> None:
+    """One-line docstring describing intent."""
+    ...
+```
+
+Examples:
+- `test_parse_inep_csv_with_valid_2023_schema_returns_dataframe`
+- `test_parse_inep_csv_missing_required_column_raises_schema_error`
+- `test_crossjoin_stage1_exact_match_resolves_70_percent`
+
+### Fixture patterns
+
+```python
+# tests/conftest.py
+import pytest
+from pathlib import Path
+
+@pytest.fixture
+def inep_sample_2023(tmp_path: Path) -> Path:
+    """Provides a 20-row synthetic INEP 2023 CSV."""
+    src = Path("tests/fixtures/inep_sample_2023.csv")
+    dst = tmp_path / "inep.csv"
+    dst.write_bytes(src.read_bytes())
+    return dst
+```
+
+**Rule**: never modify fixture files in tests. Always copy to `tmp_path` first.
+
+### Parametrization for table-driven tests
+
+```python
+@pytest.mark.parametrize(
+    "qt_presencial,qt_ead,expected",
+    [
+        (10000, 0, "PRESENCIAL"),
+        (0, 10000, "EAD"),
+        (5000, 5000, "HIBRIDO"),
+        (8000, 2000, "PRESENCIAL"),   # 80% presencial
+        (2000, 8000, "EAD"),           # 80% ead
+        (4000, 4000, "HIBRIDO"),       # 50/50 fits 30-70 mix range
+        (0, 0, "SEM_DADOS"),
+    ],
+)
+def test_derive_modalidade_predominante(qt_presencial: int, qt_ead: int, expected: str) -> None:
+    assert derive_modalidade(qt_presencial, qt_ead) == expected
+```
+
+### Mocking with pytest-mock
+
+```python
+def test_fetch_latest_inep_retries_on_503(mocker: MockerFixture) -> None:
+    mock_get = mocker.patch("redbull_sfm.inep_ingest.requests.get")
+    mock_get.side_effect = [
+        mocker.Mock(status_code=503),
+        mocker.Mock(status_code=503),
+        mocker.Mock(status_code=200, content=b"..."),
+    ]
+    df, year = fetch_latest_inep()
+    assert mock_get.call_count == 3
+```
+
+**Rule**: prefer `mocker.patch` over `unittest.mock.patch` for cleaner teardown.
+
+### Coverage gate
+
+```bash
+pytest tests/unit/ --cov=src --cov-fail-under=80 --cov-report=term-missing
+```
+
+Required: ≥80% branch coverage on new code. CI fails below threshold.
+
+## 3. Type Hints + mypy --strict
+
+### Essential rules
+
+```python
+# Always type function signatures
+def parse(path: Path, year: int) -> pd.DataFrame:
+    ...
+
+# Use Protocol for structural typing (duck typing)
+from typing import Protocol
+
+class CsvReader(Protocol):
+    def read(self, path: Path) -> pd.DataFrame: ...
+
+# Use TypedDict for dict-like structured data
+from typing import TypedDict
+
+class RunReportDict(TypedDict):
+    year_used: int
+    kpis: dict[str, float]
+    duration_seconds: float
+```
+
+### dataclass vs Pydantic BaseModel — decision tree
+
+| Use case | Choose |
+|---|---|
+| Internal data, no validation needed | `@dataclass(frozen=True)` |
+| External data parsing + validation | `pydantic.BaseModel` |
+| Settings from env | `pydantic_settings.BaseSettings` |
+| Performance-critical hot path | `@dataclass(slots=True, frozen=True)` |
+| Cross-field validation | `BaseModel` with `@model_validator(mode="after")` |
+
+### Avoid `Any`
+
+```python
+# BAD
+def process(data: Any) -> Any: ...
+
+# GOOD
+from typing import TypeVar
+T = TypeVar("T", bound=pd.DataFrame)
+def process(data: T) -> T: ...
+```
+
+If `Any` is unavoidable, annotate with comment:
+```python
+def from_external_lib(x: Any) -> int:  # third-party returns Any-typed
+    return int(x)
+```
+
+## 4. Pydantic v2 Patterns
+
+### Schema validation pattern
+
+```python
+from pydantic import BaseModel, Field, field_validator
+from typing import Literal
+
+class INEPRecord(BaseModel):
+    """Single row of MICRODADOS_CADASTRO_IES_<year>.csv."""
+
+    CO_IES: int = Field(..., gt=0, description="Código MEC da IES")
+    NO_IES: str = Field(..., min_length=3)
+    NO_MUNICIPIO_IES: str = Field(..., min_length=2)
+    SG_UF_IES: str = Field(..., pattern=r"^[A-Z]{2}$")
+    TP_REDE: Literal[1, 2, 3, 4]  # 1=pub federal, 2=pub estadual, etc.
+    TP_SITUACAO: Literal[1, 2, 3]  # 1=ativa, 2=extinta, 3=desativada
+    QT_MAT_PRESENCIAL: int = Field(..., ge=0)
+    QT_MAT_EAD: int = Field(..., ge=0)
+
+    @field_validator("NO_IES", "NO_MUNICIPIO_IES")
+    @classmethod
+    def strip_and_upper(cls, v: str) -> str:
+        return v.strip().upper()
+```
+
+### Cross-field validation
+
+```python
+from pydantic import model_validator
+
+class MatchResultMetadata(BaseModel):
+    total_input_universities: int
+    matched_count: int
+    unmatched_count: int
+
+    @model_validator(mode="after")
+    def matched_plus_unmatched_equals_input(self) -> "MatchResultMetadata":
+        if self.matched_count + self.unmatched_count != self.total_input_universities:
+            raise ValueError("matched + unmatched must equal input total")
+        return self
+```
+
+### Settings management (replaces os.getenv boilerplate)
+
+```python
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+class AzureSettings(BaseSettings):
+    """Auto-loads from env vars: AZURE_SAS_INPUT, AZURE_SAS_OUTPUT, AZURE_STORAGE_ACCOUNT."""
+
+    model_config = SettingsConfigDict(env_prefix="AZURE_", env_file=".env")
+
+    sas_input: str = Field(..., min_length=20)
+    sas_output: str = Field(..., min_length=20)
+    storage_account: str = Field(..., pattern=r"^[a-z0-9]{3,24}$")
+    input_container: str = "redbull-sfm-input"
+    output_container: str = "redbull-sfm-output"
+
+# Usage
+settings = AzureSettings()  # raises ValidationError if env missing
+```
+
+### Serialization
+
+```python
+report = RunReport(year_used=2023, kpis={"K-1": 2543}, duration_seconds=47.3)
+json_str = report.model_dump_json(indent=2)  # NOT json.dumps(asdict(report))
+dict_form = report.model_dump()
+```
+
+### Validation error handling
+
+```python
+from pydantic import ValidationError
+
+try:
+    record = INEPRecord.model_validate(row_dict)
+except ValidationError as e:
+    # Log structured error — each field with its own error
+    log.error("inep_row_invalid", row_idx=idx, errors=e.errors())
+    continue  # skip malformed row
+```
+
+**Rule**: `e.errors()` returns list of dicts — preserve via structured logging, do NOT just `str(e)`.
+
+## 5. Pandas Idioms
+
+For deep pandas patterns beyond this section, see jeffallan claude-skills `pandas-pro` (external reference).
+
+### Method chaining (readability)
+
+```python
+# BAD — temporary variables, harder to follow
+df_clean = df.dropna(subset=["CO_IES"])
+df_filtered = df_clean[df_clean["TP_SITUACAO"] == 1]
+df_typed = df_filtered.astype({"TP_REDE": "category"})
+
+# GOOD — chained
+df_typed = (
+    df
+    .dropna(subset=["CO_IES"])
+    .query("TP_SITUACAO == 1")
+    .astype({"TP_REDE": "category"})
+)
+```
+
+### dtype management (memory + correctness)
+
+```python
+# After read_csv, immediately enforce dtypes
+df = pd.read_csv(
+    path,
+    encoding="latin-1",
+    dtype={
+        "CO_IES": "int64",
+        "TP_REDE": "Int8",       # nullable int8, accepts NaN
+        "TP_SITUACAO": "Int8",
+        "NO_IES": "string",      # not "object"
+        "SG_UF_IES": "category",  # low-cardinality
+    },
+)
+```
+
+**Rule**: never let pandas auto-infer dtype on production code paths. Always explicit.
+
+### Avoiding SettingWithCopyWarning
+
+```python
+# BAD
+df_active = df[df["TP_SITUACAO"] == 1]
+df_active["MODALIDADE"] = "..."  # warning
+
+# GOOD
+df_active = df[df["TP_SITUACAO"] == 1].copy()
+df_active["MODALIDADE"] = "..."
+
+# BETTER
+df_active = df.query("TP_SITUACAO == 1").assign(MODALIDADE="...")
+```
+
+### Validation after transforms
+
+```python
+# Always check invariants explicitly after non-trivial transforms
+df_agg = df.groupby("UNIVERSITY").agg(...)
+assert df_agg.index.is_unique, "aggregation produced duplicate UNIVERSITY keys"
+assert len(df_agg) == 430, f"expected 430 universities, got {len(df_agg)}"
+```
+
+### Merge validation
+
+```python
+# Always specify validate when merging to catch unexpected cardinality
+result = pd.merge(
+    df_mmm,
+    df_inep,
+    on="match_key",
+    how="left",
+    validate="one_to_one",  # raises if not 1:1
+    indicator=True,         # adds _merge column for debugging
+)
+unmatched = result.query("_merge == 'left_only'")
+```
+
+## 6. Fuzzy Matching com rapidfuzz
+
+### Algorithm selection
+
+| Algorithm | Use case | Pitfall |
+|---|---|---|
+| `fuzz.ratio` | Strings same length, char-level diff | Bad for word reordering |
+| `fuzz.token_sort_ratio` | Word reordering OK ("FED ALAGOAS" vs "ALAGOAS FED") | Slow on long strings |
+| `fuzz.partial_ratio` | Prefixes/substrings ("UFAL" matches "UNIVERSIDADE FEDERAL DE ALAGOAS") | False positives in short strings |
+| `fuzz.WRatio` | "Smart" weighted combo — when unsure | Less predictable, harder to tune threshold |
+
+### Threshold tuning
+
+Start with **85** as default threshold. Calibrate via known-truth golden set:
+
+```python
+# Calibration script — run once with ~20-30 known pairs
+from rapidfuzz import fuzz
+
+golden_pairs = [
+    ("UNIVERSIDADE FEDERAL DE ALAGOAS", "UFAL"),  # expected: match
+    ("USP", "Universidade de Sao Paulo"),         # expected: match
+    ("Universidade Anhanguera", "UNESP"),         # expected: no match
+    ...
+]
+
+for s1, s2 in golden_pairs:
+    print(f"token_sort: {fuzz.token_sort_ratio(s1, s2):3d}  partial: {fuzz.partial_ratio(s1, s2):3d}  {s1[:30]} | {s2[:30]}")
+```
+
+Pick threshold where positive-pairs score ≥ T and negative-pairs score < T.
+
+### Performance — process in batches
+
+```python
+from rapidfuzz import process
+
+# Instead of N*M loop (slow):
+for mmm_name in mmm_names:
+    for inep_name in inep_names:
+        score = fuzz.token_sort_ratio(mmm_name, inep_name)
+
+# Use vectorized cdist (fast):
+import numpy as np
+matrix = process.cdist(mmm_names, inep_names, scorer=fuzz.token_sort_ratio)
+# matrix[i, j] is score between mmm_names[i] and inep_names[j]
+best_matches = np.argmax(matrix, axis=1)
+best_scores = matrix[np.arange(len(mmm_names)), best_matches]
+```
+
+### Normalization BEFORE fuzzy
+
+```python
+from unidecode import unidecode
+
+def normalize(s: str) -> str:
+    """Apply BEFORE feeding to fuzz: removes accents, casing, punctuation."""
+    return unidecode(s.lower().strip()).replace("-", " ").replace(",", " ")
+
+# Then fuzz on normalized strings
+score = fuzz.token_sort_ratio(normalize(s1), normalize(s2))
+```
+
+## 7. Schema Evolution Patterns
+
+### Versioned validators
+
+```python
+# schemas.py
+INEP_2023_REQUIRED_COLS = frozenset({
+    "CO_IES", "NO_IES", "NO_MUNICIPIO_IES", "SG_UF_IES",
+    "TP_REDE", "TP_SITUACAO", "QT_MAT_PRESENCIAL", "QT_MAT_EAD",
+})
+
+INEP_2024_REQUIRED_COLS = INEP_2023_REQUIRED_COLS  # presumed stable; update when published
+
+def validate_schema(df: pd.DataFrame, year: int) -> None:
+    expected = {
+        2023: INEP_2023_REQUIRED_COLS,
+        2024: INEP_2024_REQUIRED_COLS,
+    }.get(year)
+    if expected is None:
+        raise SchemaError(f"unknown INEP year {year}")
+    missing = expected - set(df.columns)
+    if missing:
+        raise SchemaError(f"INEP {year} missing required cols: {sorted(missing)}")
+    # extras are OK — warn but don't fail
+    extras = set(df.columns) - expected
+    if extras:
+        log.warning("inep_schema_extras_detected", year=year, extras=sorted(extras))
+```
+
+### Backward-compat strategy
+
+When INEP schema changes between years:
+1. Add `INEP_<NEW_YEAR>_REQUIRED_COLS` to schemas.py (don't modify old)
+2. If a column was renamed, add a normalization step in `parse_inep_csv` that maps old → new name
+3. If a column was removed, add a "synthesized" column from other data
+4. Never drop support for old schemas until you confirm no one needs to re-process old years
+
+## 8. Azure SDK Python (azure-storage-blob)
+
+### Async client (preferred for I/O)
+
+```python
+from azure.storage.blob.aio import BlobServiceClient
+from azure.core.exceptions import ResourceNotFoundError
+
+async def download_blob(container: str, blob: str, sas_url: str) -> bytes:
+    async with BlobServiceClient(account_url=sas_url) as service:
+        async with service.get_blob_client(container=container, blob=blob) as client:
+            try:
+                stream = await client.download_blob()
+                return await stream.readall()
+            except ResourceNotFoundError:
+                log.error("blob_not_found", container=container, blob=blob)
+                raise
+```
+
+### SAS handling
+
+```python
+# NEVER log full SAS URL — redact the query string
+def redact_sas(url: str) -> str:
+    """Remove ?sv=...&sig=... from URL for safe logging."""
+    return url.split("?")[0] + "?<redacted>"
+```
+
+### Retry policy
+
+```python
+from azure.core.pipeline.policies import RetryPolicy
+
+# Built-in retry with exp backoff
+service = BlobServiceClient(
+    account_url=sas_url,
+    retry_total=3,
+    retry_backoff_factor=2,
+    retry_backoff_max=30,
+)
+```
+
+### Error types to handle explicitly
+
+| Exception | When | Resposta |
+|---|---|---|
+| `ResourceNotFoundError` | Blob doesn't exist | Hard fail, log container/blob |
+| `ResourceExistsError` | Overwrite without flag | Either set `overwrite=True` or rename |
+| `ClientAuthenticationError` | SAS expired/invalid | Hard fail, alert (need new SAS) |
+| `ServiceRequestError` | Network/transient | Retry (already covered by RetryPolicy) |
+
+## 9. openpyxl Deep
+
+### Conditional formatting
+
+```python
+from openpyxl.formatting.rule import CellIsRule
+from openpyxl.styles import PatternFill
+
+ws.conditional_formatting.add(
+    "F2:F500",  # MATCH_SCORE column
+    CellIsRule(
+        operator="lessThan",
+        formula=["85"],
+        fill=PatternFill(start_color="FFCDD2", end_color="FFCDD2", fill_type="solid"),  # red
+    ),
+)
+ws.conditional_formatting.add(
+    "F2:F500",
+    CellIsRule(
+        operator="between",
+        formula=["85", "89"],
+        fill=PatternFill(start_color="FFF9C4", end_color="FFF9C4", fill_type="solid"),  # yellow
+    ),
+)
+```
+
+### AutoFilter
+
+```python
+ws.auto_filter.ref = ws.dimensions  # filter buttons on all columns
+```
+
+### NamedStyle for consistent headers
+
+```python
+from openpyxl.styles import NamedStyle, Font, PatternFill, Alignment
+
+header_style = NamedStyle(name="header")
+header_style.font = Font(bold=True, color="FFFFFF")
+header_style.fill = PatternFill(start_color="001E60", end_color="001E60", fill_type="solid")  # Red Bull blue
+header_style.alignment = Alignment(horizontal="center", vertical="center")
+
+# Register once on workbook
+wb.add_named_style(header_style)
+
+# Apply
+for cell in ws[1]:  # row 1 = header
+    cell.style = "header"
+```
+
+### Freeze panes
+
+```python
+ws.freeze_panes = "A2"  # freeze row 1
+```
+
+### Performance — write_only mode for large outputs
+
+```python
+from openpyxl import Workbook
+wb = Workbook(write_only=True)  # ~10x faster, but read_only=False after save
+ws = wb.create_sheet("IES_FULL")
+ws.append(headers)
+for row in rows_iter:
+    ws.append(row)
+wb.save(path)
+```
+
+## 10. Structured Logging (structlog)
+
+### Config (call once at startup)
+
+```python
+import structlog
+import logging
+import sys
+
+def configure_logging(level: str = "INFO") -> None:
+    logging.basicConfig(format="%(message)s", stream=sys.stdout, level=level)
+    structlog.configure(
+        processors=[
+            structlog.contextvars.merge_contextvars,
+            structlog.processors.add_log_level,
+            structlog.processors.TimeStamper(fmt="iso", utc=True),
+            structlog.processors.StackInfoRenderer(),
+            structlog.processors.format_exc_info,
+            structlog.processors.JSONRenderer(),
+        ],
+        wrapper_class=structlog.make_filtering_bound_logger(logging.INFO),
+        cache_logger_on_first_use=True,
+    )
+```
+
+### Usage patterns
+
+```python
+log = structlog.get_logger()
+
+# Bind context at function entry
+log = log.bind(workflow="annual-update", year=2023)
+
+# Log with structured fields (NOT f-strings)
+log.info("inep_download_started", url=url, container=container)
+# BAD: log.info(f"Downloading from {url}")
+
+# Use exception() to capture stack
+try:
+    ...
+except SchemaError:
+    log.exception("inep_schema_validation_failed", year=year)
+    raise
+```
+
+### Required fields in all logs
+
+| Field | Always | Why |
+|---|---|---|
+| `event` | yes (first arg) | snake_case action name |
+| `workflow` | yes | enables filtering across runs |
+| `year` | yes | enables filtering across years |
+| Module-specific kwargs | as relevant | structured grep |
+
+**Rule**: never log secrets, SAS URLs, or PII. Use `redact_sas()` helper.
+
+## 11. INEP / Government CSV Patterns
+
+### Encoding fallback
+
+```python
+def read_csv_resilient(path: Path) -> pd.DataFrame:
+    """Try latin-1 → cp1252 → utf-8. INEP files are usually latin-1."""
+    for encoding in ("latin-1", "cp1252", "utf-8"):
+        try:
+            return pd.read_csv(path, encoding=encoding, sep=";")
+        except UnicodeDecodeError:
+            continue
+    raise ValueError(f"could not decode {path} with any known encoding")
+```
+
+### Malformed rows
+
+```python
+# Pandas: skip bad rows but log them
+df = pd.read_csv(
+    path,
+    encoding="latin-1",
+    sep=";",
+    on_bad_lines=lambda bad_line: (
+        log.warning("inep_bad_row_skipped", line=bad_line),
+        None,  # skip
+    )[1],
+    engine="python",  # required for on_bad_lines callable
+)
+```
+
+### Year discovery from MEC portal
+
+```python
+import re
+import requests
+
+def discover_latest_inep_year() -> int:
+    """Scrape MEC portal for the most recent published Censo year.
+
+    Falls back to (current_calendar_year - 2) if scraping fails — INEP publishes
+    with ~1y lag, so 2026 Q1 usually has 2024 latest, but conservatively assume 2023.
+    """
+    try:
+        resp = requests.get(
+            "https://www.gov.br/inep/pt-br/acesso-a-informacao/dados-abertos/microdados/censo-da-educacao-superior",
+            timeout=30,
+        )
+        years = sorted(set(int(y) for y in re.findall(r"\b(20\d{2})\b", resp.text)), reverse=True)
+        return years[0] if years else 2023
+    except requests.RequestException:
+        log.warning("inep_year_discovery_failed", fallback=2023)
+        return 2023
+```
+
+### Data quality assertions per row
+
+```python
+def validate_inep_row(row: dict[str, Any]) -> bool:
+    """Quick sanity checks per row. Reject if any fail."""
+    if row.get("CO_IES", 0) <= 0:
+        return False
+    if not re.match(r"^[A-Z]{2}$", str(row.get("SG_UF_IES", ""))):
+        return False
+    if row.get("QT_MAT_PRESENCIAL", -1) < 0 or row.get("QT_MAT_EAD", -1) < 0:
+        return False
+    return True
+```
+
+## 12. Quality Gates
+
+### Pre-commit checklist (run in order)
+
+```bash
+ruff format --check src tests       # NO uncommitted formatting drift
+ruff check src tests                # 0 lint errors
+mypy --strict src                   # 0 type errors
+pytest tests/unit --cov=src --cov-fail-under=80
+pytest tests/integration            # requires Azurite
+# acceptance tests run manually only — never in CI
+```
+
+### CI pipeline structure (GHA)
+
+```yaml
+jobs:
+  lint-and-types:
+    steps:
+      - ruff format --check src tests
+      - ruff check src tests
+      - mypy --strict src
+  unit:
+    needs: lint-and-types
+    steps:
+      - pytest tests/unit --cov=src --cov-fail-under=80
+  integration:
+    needs: unit
+    services:
+      azurite:
+        image: mcr.microsoft.com/azure-storage/azurite
+    steps:
+      - pytest tests/integration
+```
+
+### Definition of Done per module
+
+A module is **DONE** when:
+1. All unit tests pass (TDD red → green)
+2. Type-check passes (`mypy --strict <module>`)
+3. Lint passes (`ruff check <module>`)
+4. Coverage ≥80% branch
+5. At least one integration test covers the module if it has I/O
+6. Module exports documented in `__init__.py`
+7. Acceptance tests exist for KPI-bearing functionality (if applicable)
+
+## Cross-references
+
+- TDD discipline (TS-flavored): `/tdd-discipline` — different stack, **do NOT use for Python projects**
+- Project conventions (stack-agnostic): `/project-conventions`
+- Checkpoint protocol: `/checkpoint-protocol`
+- KB lookup: `/kb-lookup` — for org-specific patterns
+- External pandas deep dive: <https://jeffallan.github.io/claude-skills/skills/data-ml/pandas-pro/>
+
+## When NOT to use this skill
+
+- Python web backends (FastAPI/Django) — use backend-specific skills if available
+- Pure scripting (no schemas, no data transformations) — overkill
+- TS/React projects — wrong language entirely
+- Azure infra (Bicep, GHA OIDC) — use `/azure-prepare`, `/azure-deploy` instead