jumpstart-mode 1.1.13 → 1.1.14

package/.github/skills/python-developer/references/production-readiness.md

@@ -0,0 +1,103 @@
+# Production Readiness Checklist
+
+Use this before any AI Innovation Pod delivery handoff or go-live.
+
+## Environment & Configuration
+
+- [ ] All secrets in Key Vault / secret manager — not in `.env` files on servers
+- [ ] `.env` files gitignored and not present on production servers
+- [ ] `ENVIRONMENT=production` set
+- [ ] Debug endpoints disabled (`docs_url=None`, `openapi_url=None`)
+- [ ] `CORS_ORIGINS` is explicitly limited — not `["*"]`
+- [ ] All required environment variables documented in `.env.example`
+- [ ] `validate-env` CLI command passes
+
+## Database
+
+- [ ] Alembic migrations run as a separate step before app start
+- [ ] Database URL uses a least-privilege user (not postgres superuser)
+- [ ] Connection pool sized appropriately (`pool_size=10`, `max_overflow=20`)
+- [ ] `pool_pre_ping=True` set
+- [ ] Database backup and restore procedure documented and tested
+- [ ] pgvector extension enabled if embeddings are stored
+
+## Application
+
+- [ ] Health endpoint `/health` returns 200
+- [ ] Readiness endpoint `/ready` checks DB connectivity
+- [ ] Request timeouts set (via reverse proxy or middleware)
+- [ ] File upload limits enforced (size + MIME type)
+- [ ] Pagination limits capped (no unbounded queries)
+- [ ] Error responses do not expose stack traces or internal details
+
+## Auth & Authorization
+
+- [ ] JWT secret is at least 32 random bytes
+- [ ] Tokens have expiry (`exp` claim set)
+- [ ] Auth failures return 401, not 403 or 500
+- [ ] Every route that touches user data checks ownership
+
+## Logging & Observability
+
+- [ ] Structured JSON logging enabled
+- [ ] Logs shipped to centralized store (Azure Monitor, Datadog, etc.)
+- [ ] Correlation IDs injected on every request
+- [ ] LLM calls log: model, input_tokens, output_tokens, elapsed_ms, prompt_template
+- [ ] Error rate alerting configured
+- [ ] Health endpoint monitored with uptime alert
+
+## AI-Specific
+
+- [ ] LLM provider credentials stored in Key Vault
+- [ ] `max_tokens` set on every completion call (prevent runaway cost)
+- [ ] Token usage logged per request
+- [ ] Prompt injection mitigations in place (user input delimited)
+- [ ] RAG responses include citations / source IDs
+- [ ] Evaluation dataset exists with ≥ 10 golden Q&A pairs
+- [ ] Baseline retrieval recall measured
+- [ ] Human-in-the-loop gates defined for high-risk actions
+- [ ] Feedback capture mechanism in place
+
+## Security
+
+- [ ] `pip-audit` passes — no critical CVEs in dependencies
+- [ ] `bandit` scan reviewed — no high-severity findings
+- [ ] `detect-secrets scan` passes — no committed secrets
+- [ ] Input validation on all user-facing endpoints (Pydantic)
+- [ ] File upload path traversal prevention in place
+- [ ] SQL injection prevention: no raw string interpolation in queries
+- [ ] SSRF prevention: user-supplied URLs validated against allowlist
+
+## Workers & Background Jobs
+
+- [ ] Celery workers run with `acks_late=True`
+- [ ] Dead letter queue or retry limit configured
+- [ ] Job status visible to users or operators (dashboard or API)
+- [ ] Worker crash does not leave jobs in "running" state permanently
+
+## Deployment
+
+- [ ] Docker image builds with no root user
+- [ ] Container health check configured
+- [ ] Rollback procedure documented (previous image tag available)
+- [ ] Database migration rollback procedure tested
+- [ ] Zero-downtime deployment strategy confirmed (rolling deploy or blue/green)
+- [ ] Auto-scaling limits set (prevent runaway cost)
+
+## Documentation
+
+- [ ] `README.md` includes setup, env vars, and run instructions
+- [ ] API documented (OpenAPI schema exported or docs URL noted for internal use)
+- [ ] Known limitations and out-of-scope items documented
+- [ ] On-call or escalation contact documented
+
+## Handoff Checklist (AI Pod to Client)
+
+- [ ] `.env.example` with all required variables
+- [ ] Infrastructure provisioning scripts or Bicep/Terraform
+- [ ] Migration runbook
+- [ ] Deployment runbook
+- [ ] Monitoring dashboard link
+- [ ] Backup/restore procedure
+- [ ] Evaluation results from pre-launch baseline
+- [ ] At least one team member on the client side trained on the eval process

package/.github/skills/python-developer/references/pydantic-settings.md

@@ -0,0 +1,227 @@
+# Pydantic & Settings Patterns
+
+## Settings Management with pydantic-settings
+
+Use `pydantic-settings` for all application configuration.
+It reads from environment variables, `.env` files, and supports secrets from files.
+
+```python
+# app/core/config.py
+from pydantic import Field, field_validator
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class Settings(BaseSettings):
+    # App identity
+    app_name: str = "AI Pod Service"
+    app_version: str = "0.1.0"
+    environment: str = Field(default="local", pattern="^(local|staging|production)$")
+    debug: bool = False
+    log_level: str = Field(default="INFO", pattern="^(DEBUG|INFO|WARNING|ERROR|CRITICAL)$")
+
+    # Server
+    host: str = "0.0.0.0"
+    port: int = Field(default=8000, ge=1, le=65535)
+    cors_origins: list[str] = ["http://localhost:3000"]
+
+    # Database
+    database_url: str  # required — no default means it must be set
+
+    # Auth
+    jwt_secret: str  # required
+
+    # Azure OpenAI
+    azure_openai_endpoint: str | None = None
+    azure_openai_api_key: str | None = None
+    azure_openai_deployment: str = "gpt-4o"
+    azure_openai_embedding_deployment: str = "text-embedding-3-small"
+
+    model_config = SettingsConfigDict(
+        env_file=".env",
+        env_file_encoding="utf-8",
+        extra="ignore",
+        case_sensitive=False,
+    )
+
+    @field_validator("database_url")
+    @classmethod
+    def validate_database_url(cls, v: str) -> str:
+        if not v.startswith(("postgresql://", "postgresql+psycopg://", "sqlite://")):
+            raise ValueError("DATABASE_URL must use postgresql or sqlite scheme.")
+        return v
+
+
+# Singleton — import this everywhere
+settings = Settings()
+```
+
+## Pattern: Config as dependency
+
+For testing, inject settings via FastAPI's dependency system:
+
+```python
+from functools import lru_cache
+from fastapi import Depends
+
+@lru_cache
+def get_settings() -> Settings:
+    return Settings()
+
+# In routes or services:
+def my_service(config: Settings = Depends(get_settings)) -> ...:
+    ...
+
+# In tests — override the dependency:
+def test_with_custom_settings():
+    app.dependency_overrides[get_settings] = lambda: Settings(environment="test")
+```
+
+## Pydantic Request Models
+
+```python
+from pydantic import BaseModel, Field, field_validator
+from typing import Annotated
+
+
+class DocumentCreate(BaseModel):
+    title: Annotated[str, Field(min_length=1, max_length=500, strip_whitespace=True)]
+    content: Annotated[str, Field(min_length=1, max_length=500_000)]
+    source_uri: str | None = None
+    tags: list[str] = Field(default_factory=list, max_length=20)
+
+    @field_validator("tags")
+    @classmethod
+    def tags_must_be_lowercase(cls, v: list[str]) -> list[str]:
+        return [tag.lower().strip() for tag in v]
+```
+
+## Pydantic Response Models
+
+```python
+from pydantic import BaseModel, ConfigDict
+from datetime import datetime
+
+
+class DocumentRead(BaseModel):
+    model_config = ConfigDict(from_attributes=True)  # enables ORM mode
+
+    id: str
+    title: str
+    content: str
+    source_uri: str | None
+    tags: list[str]
+    created_at: datetime
+    updated_at: datetime | None
+```
+
+## Nested Models
+
+```python
+class ChunkRead(BaseModel):
+    model_config = ConfigDict(from_attributes=True)
+    id: str
+    text: str
+    chunk_index: int
+    embedding_model: str | None
+
+
+class DocumentWithChunks(DocumentRead):
+    chunks: list[ChunkRead] = Field(default_factory=list)
+```
+
+## Discriminated Unions (agent tool results)
+
+```python
+from typing import Literal
+from pydantic import BaseModel
+
+
+class TextResult(BaseModel):
+    type: Literal["text"] = "text"
+    content: str
+
+
+class ErrorResult(BaseModel):
+    type: Literal["error"] = "error"
+    message: str
+    code: str
+
+
+ToolResult = TextResult | ErrorResult
+```
+
+## Secrets Handling
+
+```python
+from pydantic import SecretStr
+
+
+class Settings(BaseSettings):
+    database_password: SecretStr  # Never logged or serialized
+
+    @property
+    def database_url(self) -> str:
+        return f"postgresql://user:{self.database_password.get_secret_value()}@host/db"
+```
+
+## Environment-Specific Config
+
+```bash
+# .env.example (committed)
+APP_NAME=AI Pod Service
+ENVIRONMENT=local
+DATABASE_URL=postgresql://postgres:postgres@localhost:5432/aipod
+JWT_SECRET=change-me-in-production
+AZURE_OPENAI_ENDPOINT=
+AZURE_OPENAI_API_KEY=
+
+# .env (gitignored)
+# Copy from .env.example and fill in real values
+```
+
+## Model Serialization
+
+```python
+doc = DocumentRead(...)
+
+# To dict
+doc.model_dump()                          # Python dict
+doc.model_dump(exclude_none=True)         # Drop None fields
+doc.model_dump(include={"id", "title"})   # Partial
+
+# To JSON
+doc.model_dump_json()
+
+# From ORM model
+doc = DocumentRead.model_validate(orm_doc)   # ORM -> Pydantic
+```
+
+## Common Validation Patterns
+
+```python
+from pydantic import AnyUrl, EmailStr, field_validator
+import re
+
+
+class UserCreate(BaseModel):
+    email: EmailStr
+    webhook_url: AnyUrl | None = None
+    name: str
+
+    @field_validator("name")
+    @classmethod
+    def name_no_special_chars(cls, v: str) -> str:
+        if not re.match(r"^[\w\s\-']+$", v):
+            raise ValueError("Name contains invalid characters.")
+        return v.strip()
+```
+
+## Anti-Patterns to Avoid
+
+| Anti-pattern | Correct approach |
+|-------------|-----------------|
+| `os.environ.get("KEY")` scattered everywhere | Single `Settings` object |
+| Defaults that are secrets | Mark as required (no default) |
+| `dict` or `Any` as request body | Explicit Pydantic model |
+| Returning ORM objects directly from routes | Always return a Pydantic response model |
+| `model.dict()` (deprecated) | Use `model.model_dump()` |

package/.github/skills/python-developer/references/rag-pipeline-patterns.md

@@ -0,0 +1,307 @@
+# RAG Pipeline Patterns
+
+## Core Principle
+
+A RAG pipeline is not a demo. Before writing a single ingestion function, confirm:
+
+- Source IDs are defined and stable
+- Content hashes will prevent re-ingestion of unchanged content
+- Chunk metadata is specified (source_id, chunk_index, page, section, created_at)
+- Citation traceability is understood (how does the answer link back to a source?)
+- An evaluation dataset strategy exists (even 10 golden QA pairs is enough to start)
+
+If any of these are missing, stop and define them first.
+
+## Pipeline Stages
+
+```
+1. Source Discovery        → list files, URLs, SharePoint items, DB records
+2. Document Extraction     → text from PDF, DOCX, XLSX, HTML, Markdown
+3. Text Cleanup            → strip artifacts, normalize whitespace, detect language
+4. Chunking                → fixed-size, sentence, semantic, or structural
+5. Metadata Enrichment     → add source_id, title, page, section, created_at
+6. Embedding Generation    → call embedding API; batch calls for throughput
+7. Storage                 → upsert vectors + metadata in pgvector or vector DB
+8. Retrieval               → semantic search, keyword search, or hybrid
+9. Reranking               → cross-encoder or score threshold filter
+10. Answer Generation      → LLM call with retrieved context + citations
+11. Citation Grounding     → map claims to source chunks
+12. Evaluation             → retrieval recall, answer quality, citation accuracy
+13. Feedback Capture       → user thumbs up/down, corrections
+```
+
+## Chunk Data Model
+
+```python
+# app/models/chunk.py
+from sqlmodel import Field, SQLModel, Column
+from sqlalchemy import Text
+from pgvector.sqlalchemy import Vector
+import uuid
+
+
+class DocumentChunk(SQLModel, table=True):
+    __tablename__ = "document_chunks"
+
+    id: str = Field(default_factory=lambda: str(uuid.uuid4()), primary_key=True)
+    document_id: str = Field(foreign_key="documents.id", index=True)
+    chunk_index: int
+    text: str = Field(sa_column=Column(Text))
+    char_count: int
+    page_number: int | None = None
+    section_heading: str | None = None
+    content_hash: str = Field(max_length=64, index=True)
+
+    embedding: list[float] | None = Field(default=None, sa_column=Column(Vector(1536)))
+    embedding_model: str | None = None
+    embedded_at: str | None = None
+```
+
+## Document Extraction
+
+```python
+# app/integrations/extractors.py
+from pathlib import Path
+import fitz  # PyMuPDF
+import docx
+
+
+def extract_pdf(path: Path) -> list[dict]:
+    """Returns list of {page: int, text: str}"""
+    doc = fitz.open(str(path))
+    pages = []
+    for i, page in enumerate(doc):
+        text = page.get_text("text")
+        if text.strip():
+            pages.append({"page": i + 1, "text": text})
+    return pages
+
+
+def extract_docx(path: Path) -> str:
+    doc = docx.Document(str(path))
+    return "\n\n".join(p.text for p in doc.paragraphs if p.text.strip())
+```
+
+## Chunking
+
+```python
+# app/services/chunking.py
+from dataclasses import dataclass
+import hashlib
+
+
+@dataclass
+class Chunk:
+    text: str
+    chunk_index: int
+    char_count: int
+    content_hash: str
+    page_number: int | None = None
+    section_heading: str | None = None
+
+
+def chunk_text(
+    text: str,
+    chunk_size: int = 512,
+    overlap: int = 64,
+    page_number: int | None = None,
+) -> list[Chunk]:
+    words = text.split()
+    chunks = []
+    start = 0
+    idx = 0
+
+    while start < len(words):
+        end = start + chunk_size
+        chunk_words = words[start:end]
+        chunk_text = " ".join(chunk_words)
+
+        chunks.append(Chunk(
+            text=chunk_text,
+            chunk_index=idx,
+            char_count=len(chunk_text),
+            content_hash=hashlib.sha256(chunk_text.encode()).hexdigest(),
+            page_number=page_number,
+        ))
+
+        idx += 1
+        start = end - overlap
+
+    return chunks
+```
+
+## Embedding Service
+
+```python
+# app/services/embedding.py
+from typing import Protocol
+import hashlib
+import time
+import structlog
+
+log = structlog.get_logger(__name__)
+
+
+class EmbeddingClient(Protocol):
+    def embed_batch(self, texts: list[str]) -> list[list[float]]: ...
+
+
+class EmbeddingService:
+    def __init__(self, client: EmbeddingClient, model_name: str) -> None:
+        self.client = client
+        self.model_name = model_name
+
+    def embed_chunks(self, chunks: list[str]) -> list[list[float]]:
+        start = time.time()
+        embeddings = self.client.embed_batch(chunks)
+        log.info(
+            "embeddings_generated",
+            count=len(chunks),
+            model=self.model_name,
+            elapsed_ms=round((time.time() - start) * 1000),
+        )
+        return embeddings
+```
+
+## Retrieval
+
+```python
+# app/repositories/retrieval.py
+from sqlmodel import Session, select
+from app.models.chunk import DocumentChunk
+
+
+class RetrievalRepository:
+    def __init__(self, session: Session) -> None:
+        self.session = session
+
+    def semantic_search(
+        self,
+        query_embedding: list[float],
+        top_k: int = 10,
+    ) -> list[DocumentChunk]:
+        results = self.session.exec(
+            select(DocumentChunk)
+            .where(DocumentChunk.embedding.is_not(None))
+            .order_by(DocumentChunk.embedding.cosine_distance(query_embedding))
+            .limit(top_k)
+        ).all()
+        return list(results)
+```
+
+## Answer Generation with Citations
+
+```python
+# app/services/rag.py
+from dataclasses import dataclass
+from app.integrations.llm import LlmClient, CompletionRequest
+from app.models.chunk import DocumentChunk
+
+
+@dataclass
+class RagResponse:
+    answer: str
+    sources: list[dict]   # [{chunk_id, document_id, text_snippet, score}]
+    model: str
+    input_tokens: int | None
+
+
+class RagService:
+    def __init__(self, llm: LlmClient) -> None:
+        self.llm = llm
+
+    def answer(self, question: str, chunks: list[DocumentChunk]) -> RagResponse:
+        context = "\n\n".join(
+            f"[SOURCE {i+1}] {c.text}" for i, c in enumerate(chunks)
+        )
+        system_prompt = (
+            "You are a helpful assistant. Answer the question using only the provided sources. "
+            "Cite sources as [SOURCE N]. If the answer is not in the sources, say so."
+        )
+        user_prompt = f"Question: {question}\n\nSources:\n{context}"
+
+        response = self.llm.complete(CompletionRequest(
+            system_prompt=system_prompt,
+            user_prompt=user_prompt,
+        ))
+
+        return RagResponse(
+            answer=response.text,
+            sources=[
+                {
+                    "chunk_id": c.id,
+                    "document_id": c.document_id,
+                    "text_snippet": c.text[:200],
+                }
+                for c in chunks
+            ],
+            model=response.model,
+            input_tokens=response.input_tokens,
+        )
+```
+
+## Ingestion Pipeline Orchestration
+
+```python
+# pipelines/ingest_documents.py
+import typer
+from pathlib import Path
+from sqlmodel import Session
+from app.core.database import engine
+from app.services.ingestion import IngestionService
+
+app = typer.Typer()
+
+
+@app.command()
+def ingest(source_path: str = typer.Argument(..., help="Path to documents folder")) -> None:
+    """Ingest documents from a local folder into the vector store."""
+    folder = Path(source_path)
+    if not folder.is_dir():
+        typer.echo(f"Error: {source_path} is not a directory.", err=True)
+        raise typer.Exit(code=1)
+
+    with Session(engine) as session:
+        service = IngestionService(session)
+        result = service.ingest_folder(folder)
+        typer.echo(f"Ingested {result.created} documents, skipped {result.skipped} unchanged.")
+
+
+if __name__ == "__main__":
+    app()
+```
+
+## Evaluation Schema
+
+Every RAG system needs a golden dataset. Start with 10 Q&A pairs from real user questions.
+
+```python
+# evaluation/schemas.py
+from pydantic import BaseModel
+
+
+class EvalCase(BaseModel):
+    question: str
+    expected_answer: str
+    expected_source_ids: list[str]   # documents that should be retrieved
+
+
+class EvalResult(BaseModel):
+    question: str
+    actual_answer: str
+    retrieved_source_ids: list[str]
+    expected_source_ids: list[str]
+    retrieval_hit: bool              # at least one expected source retrieved
+    answer_score: float | None       # LLM-as-judge 0.0–1.0
+```
+
+## RAG Warnings
+
+Do not deploy a RAG system without:
+
+- [ ] Content hash deduplication on ingestion
+- [ ] Citation grounding in every response
+- [ ] Retrieval logging (what was retrieved per query)
+- [ ] At least 10 eval cases with measured retrieval recall
+- [ ] Re-ingestion strategy for updated documents
+- [ ] Source deletion that also removes chunks and embeddings