groundwork-method 0.0.1 → 0.10.0

package/src/generators/python-microservice/docs/principles/stack/python/documentation.md

@@ -0,0 +1,240 @@
+---
+title: Code Documentation
+description: How to document Python code — structure over comments, Pydantic as the primary documentation layer.
+status: active
+last_reviewed: 2026-05-26
+---
+
+# Code Documentation
+
+## TL;DR
+
+Structure documents more reliably than comments. Type annotations are verified by `mypy`. Pydantic `Field(description=...)` flows directly into OpenAPI. Test names are executable documentation. Docstrings and inline comments are a last resort — used only when the code alone cannot convey the why.
+
+---
+
+## Structure Over Comments
+
+Comments are promises that no linter can verify. When code changes, `mypy` catches type errors and `pytest` catches behavioural regressions — but stale docstrings silently mislead. In a codebase where AI agents drive significant code velocity, comment drift is not hypothetical; it is the default outcome.
+
+**The documentation hierarchy:**
+
+1. **Type annotations** — `mypy`/`pyright` reject incorrect types. Zero drift risk.
+2. **Pydantic `Field(description=...)`** — part of the code; flows into OpenAPI, Swagger UI, and validation errors.
+3. **Naming** — descriptive function and variable names. Refactor first; comment last.
+4. **Test names** — executable documentation verified by CI on every run.
+5. **Docstrings** — one-line summaries for route handlers (Swagger) and complex public APIs.
+6. **Inline "why" comments** — last resort for genuinely non-obvious decisions.
+
+Everything at levels 1–4 is verified by tooling. Levels 5–6 are human promises with drift risk. Minimise them.
+
+---
+
+## Pydantic as the Primary Documentation Layer
+
+In a FastAPI service, Pydantic models are the primary documentation layer. `Field(description=...)` generates OpenAPI specs, Swagger UI field descriptions, and validation error messages — all from a single source of truth that cannot drift because it is the code:
+
+```python
+from pydantic import BaseModel, Field
+from typing import Annotated
+
+class MeetingCreate(BaseModel):
+    """Request payload for creating a new meeting."""
+
+    title: Annotated[str, Field(
+        max_length=200,
+        description="Human-readable meeting title",
+        examples=["Sprint Planning"],
+    )]
+    host_id: Annotated[str, Field(
+        description="UUID of the user hosting the meeting",
+        pattern=r"^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$",
+    )]
+    scheduled_at: Annotated[datetime | None, Field(
+        default=None,
+        description="ISO 8601 datetime for scheduled start. Omit for instant meetings.",
+        examples=["2026-09-17T14:00:00Z"],
+    )]
+```
+
+| Approach | Drift risk | Tooling integration |
+|----------|-----------|---------------------|
+| Docstring listing fields | **High** — nothing verifies accuracy | None |
+| `Field(description=...)` | **Zero** — it is the code | OpenAPI, Swagger UI, validation errors, IDE hover |
+
+**Never duplicate `Field(description=...)` content in a class docstring.** The class docstring gets one line describing the model's domain role — nothing more:
+
+```python
+# Good
+class MeetingCreate(BaseModel):
+    """Request payload for creating a new meeting."""
+    title: Annotated[str, Field(description="Human-readable meeting title")]
+
+# Bad — docstring duplicates what Field already expresses
+class MeetingCreate(BaseModel):
+    """Request payload for creating a new meeting.
+    Attributes:
+        title: Human-readable meeting title, max 200 chars.
+        host_id: UUID of the hosting user.
+    """
+```
+
+---
+
+## Route Handler Docstrings
+
+FastAPI renders route handler docstrings in Swagger UI. That is the only reason they exist.
+
+```python
+@router.post("/meetings", response_model=MeetingResponse, status_code=201)
+async def create_meeting(
+    body: Annotated[MeetingCreate, Body()],
+    auth: Annotated[AuthContext, Depends(require_auth)],
+) -> MeetingResponse:
+    """Creates a new meeting. Requires `meetings:write` scope."""
+```
+
+One line is usually enough. Add a second sentence only when the behaviour has a non-obvious aspect that the signature cannot convey:
+
+```python
+async def create_meeting(...) -> MeetingResponse:
+    """Creates a new meeting. Requires `meetings:write` scope.
+
+    If `scheduled_at` is provided, the meeting starts in `scheduled` status.
+    Otherwise it starts immediately in `active` status.
+    """
+```
+
+Do not document parameters in the route handler docstring — they are documented via Pydantic `Field` and `Annotated`.
+
+---
+
+## What to Document
+
+| What | How |
+|------|-----|
+| Pydantic model fields | `Field(description=...)` |
+| Pydantic model class | One-line docstring describing its domain role |
+| Route handler | One-line docstring for Swagger UI |
+| Complex public function with non-obvious contract | Minimal docstring covering edge cases and error conditions types cannot express |
+| Side effects invisible in the signature | Note in docstring (event publishing, cache invalidation) |
+| Module when the filename is not self-explanatory | 1–3 line module docstring |
+
+---
+
+## What Not to Document
+
+**`__init__` with typed parameters:**
+
+```python
+# Bad — the types say everything
+class MeetingService:
+    def __init__(self, repo: MeetingRepository, notifier: Notifier) -> None:
+        """Initializes MeetingService with repo and notifier."""
+
+# Good
+class MeetingService:
+    """Orchestrates meeting lifecycle operations."""
+
+    def __init__(self, repo: MeetingRepository, notifier: Notifier) -> None:
+        self._repo = repo
+        self._notifier = notifier
+```
+
+**Simple functions where the signature is self-evident:**
+
+```python
+# Bad — restating the signature
+def get_meeting(meeting_id: str) -> Meeting | None:
+    """Gets a meeting by its ID, returning Meeting or None."""
+
+# Good — no docstring needed
+def get_meeting(meeting_id: str) -> Meeting | None:
+    ...
+```
+
+**`Args`/`Returns` sections that duplicate type hints:**
+
+```python
+# Bad
+def process(audio_url: str, diarize: bool = True) -> TranscriptResult:
+    """
+    Args:
+        audio_url (str): URL of the audio file.
+        diarize (bool): Whether to diarize. Defaults to True.
+    Returns:
+        TranscriptResult: The transcript result.
+    """
+
+# Good — add only what types cannot convey
+def process(audio_url: str, diarize: bool = True) -> TranscriptResult:
+    """Processes audio and returns a structured transcript.
+
+    Raises:
+        AudioDownloadError: If the URL is unreachable.
+        DiarizationError: If diarization is requested but HF_TOKEN is missing.
+    """
+```
+
+---
+
+## Inline Comments
+
+Inline comments within function bodies are justified only for genuinely non-obvious "why" that cannot be expressed through naming or structure:
+
+```python
+async def transcribe(self, audio: AudioSegment) -> TranscriptResult:
+    audio = audio.resample(16000) if audio.sample_rate != 16000 else audio
+
+    # Batch size 16 determined via GPU memory profiling on T4 instances.
+    segments = self._model.transcribe(audio.numpy(), batch_size=16)
+
+    # HACK(alice): WhisperX returns negative start times on clips <2s.
+    # Clamp to zero until upstream fix. See whisperX#XXX.
+    for seg in segments:
+        seg.start = max(0.0, seg.start)
+
+    return TranscriptResult(segments=segments)
+```
+
+The resample call needs no comment — the conditional is clear. The batch size and the clamping hack are genuinely non-obvious and justify comments.
+
+---
+
+## In-Code Markers
+
+Use a consistent format that includes the author and an issue reference:
+
+```python
+# TODO(bob): Switch to streaming transcription when WhisperX v4 adds support.
+#            Current batch approach buffers full audio in memory. Issue #123.
+
+# FIXME(carol): Race condition on concurrent requests for same audio_id.
+#               Needs distributed lock via Redis. Issue #456.
+
+# HACK(dave): Pyannote v3.1 returns empty segments for audio < 500ms.
+#             Remove after v3.2 upgrade.
+```
+
+A marker without an issue reference is a marker that will never be resolved.
+
+---
+
+## Module Docstrings
+
+Module docstrings orient developers navigating the codebase. Keep them to 1–3 lines:
+
+```python
+"""Meeting domain entities and business rules.
+
+Domain entities have no knowledge of persistence, transport, or external services.
+"""
+```
+
+Skip when the filename is self-explanatory. Add one to `__init__.py` only when it re-exports public symbols:
+
+```python
+"""Core domain layer exports."""
+from .meeting import Meeting
+from .participant import Participant
+```

package/src/generators/python-microservice/docs/principles/stack/python/mcp.md

@@ -0,0 +1,147 @@
+---
+title: MCP Servers in Python
+description: How to build FastMCP servers in Python — tools, resources, prompts, HTTP transport, and integration with FastAPI.
+status: active
+last_reviewed: 2026-05-26
+---
+
+# MCP Servers in Python
+
+## TL;DR
+
+We expose service capabilities to AI agents through FastMCP servers that live in the entrypoint layer. Tools delegate to services — they contain no business logic. Docstrings and type hints are the schema. `stdio` is for local tooling only; production uses HTTP transport.
+
+---
+
+## Why MCP
+
+The Model Context Protocol gives AI agents a standardised way to discover and invoke service capabilities. Rather than hardcoding API calls into agent prompts, MCP exposes named, schema-validated tools that agents can select and call dynamically.
+
+The architectural position is the same as any other entrypoint: the FastMCP server validates inputs, delegates to services, and returns domain outputs. It owns no logic.
+
+---
+
+## Server Construction
+
+```python
+from fastmcp import FastMCP
+
+mcp = FastMCP(
+    name="my-service",
+    instructions="One sentence describing the capabilities this server exposes.",
+)
+```
+
+Initialise the MCP server at module level. Wire it into the FastAPI lifespan so its lifecycle is managed alongside the rest of the application (see [Async Patterns](async.md) — Application Lifespan).
+
+**Use HTTP transport in production.** The `stdio` transport is for local CLI tooling and development only. A backend service fleet requires HTTP (SSE or Streamable HTTP) so multiple agents can connect concurrently:
+
+```python
+mcp = FastMCP(name="my-service", transport="streamable-http")
+```
+
+---
+
+## Tools
+
+A tool is a callable an agent can invoke. Expose service methods as tools — not provider methods, not internal utilities.
+
+```python
+@mcp.tool()
+async def transcribe_audio(
+    audio_url: str,
+    language: str = "en",
+) -> TranscriptionResult:
+    """
+    Transcribe audio from a GCS URL and return a structured transcript.
+
+    Returns segments with speaker labels, confidence scores, and timestamps.
+    Use this when you need to convert audio into text for downstream analysis.
+
+    Args:
+        audio_url: GCS URI of the audio file (e.g. gs://bucket/file.mp3).
+        language: BCP-47 language code. Defaults to English.
+    """
+    return await transcription_service.transcribe(audio_url, language=language)
+```
+
+**The docstring is the tool schema.** FastMCP generates the JSON Schema an agent uses for tool selection and parameter construction from the function signature and docstring. A vague or missing docstring produces a schema the agent cannot reliably use.
+
+Write the docstring as if the reader is an LLM deciding whether and how to invoke the tool:
+
+- **First line:** what the tool does
+- **Second paragraph:** when to use it — this is the selection signal
+- **`Args`:** every parameter with its expected format and valid values
+
+**Return domain types, not SDK types.** Tool return types must serialise cleanly to JSON. Domain Pydantic models serialise correctly. SDK response objects do not — and passing them across the entrypoint boundary leaks an implementation detail into the core, breaking the inward-dependency rule.
+
+**Type hints are enforced at runtime.** FastMCP validates inputs against the function signature before the tool is called. Missing or incorrect type hints cause silent schema degradation that only surfaces when an agent tries to invoke the tool.
+
+---
+
+## Resources
+
+A resource exposes data an agent can read without invoking a workflow:
+
+```python
+@mcp.resource("items://{item_id}")
+async def get_item(item_id: str) -> Item:
+    """
+    Retrieve a completed item by ID.
+
+    Read this before calling processing tools to avoid redundant work.
+    """
+    return await item_repository.get(item_id)
+```
+
+Resources are read-only. An agent browses them for context before deciding which tool to call. Keep them cheap — no inference, no heavy computation, no side effects.
+
+---
+
+## Prompts
+
+A prompt is a structured message template for guiding agent behaviour in a specific, repeatable workflow:
+
+```python
+@mcp.prompt()
+def analyse_prompt(item_id: str, focus: str) -> str:
+    """Structure an analysis task over a specific item."""
+    return f"""
+    You are analysing item {item_id}.
+    Focus: {focus}
+
+    Start by reading the resource at items://{item_id}.
+    Then use the available tools to complete your analysis.
+    Report findings with direct references to the source material.
+    """.strip()
+```
+
+---
+
+## Mounting on FastAPI
+
+Mount the MCP server as an ASGI sub-application so observability middleware, lifespan, and dependency injection stay aligned:
+
+```python
+from fastapi import FastAPI
+from myservice.entrypoints.mcp.server import mcp
+
+def create_app() -> FastAPI:
+    app = FastAPI(lifespan=lifespan)
+    app.mount("/mcp", mcp.get_asgi_app())
+    return app
+```
+
+---
+
+## Anti-Patterns
+
+**Business logic in a tool body.** The tool body is one line: a service call. Logic belongs in the domain, not in an MCP decorator.
+
+**SDK types in tool signatures.** A parameter or return type from an external SDK will fail to serialise or expose implementation details to the agent. Use domain Pydantic models.
+
+**Missing or vague docstrings.** An agent cannot reliably select or invoke a tool with a one-word description. The docstring is the interface — treat it with the same rigour as the type signature.
+
+**`stdio` transport in production.** Single-process, single-client only. Cannot be load-balanced and does not support concurrent agent connections.
+
+**Resources that run inference.** Resources are reads from storage. Anything that triggers model inference or heavy computation belongs in a tool.

package/src/generators/python-microservice/docs/principles/stack/python/resilience.md

@@ -0,0 +1,193 @@
+---
+title: Resilience
+description: Timeouts, retries, circuit breakers, graceful shutdown, and health checks for Python microservices.
+status: active
+last_reviewed: 2026-05-26
+---
+
+# Resilience
+
+## TL;DR
+
+Every outbound call has a timeout. Transient failures retry with jitter. Sustained provider failures open a circuit breaker. The service shuts down gracefully — in-flight requests complete before the process exits.
+
+---
+
+## Why This Matters
+
+ML inference is slow, expensive, and fails in ways that general web services don't encounter. A single hung provider request without a timeout holds a connection open until the client gives up — which may be never. An API rate limit without retries converts a transient error into a user-visible failure. A redeployment without graceful shutdown drops in-flight requests mid-stream.
+
+These are not edge cases to harden after launch. They are the predictable failure modes of a service that calls external APIs, and the patterns to handle them are straightforward.
+
+---
+
+## Timeouts
+
+Set explicit timeouts on every outbound HTTP call. `httpx.AsyncClient` accepts granular timeout configuration:
+
+```python
+import httpx
+
+client = httpx.AsyncClient(
+    timeout=httpx.Timeout(
+        connect=5.0,   # Time to establish the TCP connection
+        read=30.0,     # Time to receive the full response body
+        write=10.0,    # Time to send the request body
+        pool=5.0,      # Time waiting for a connection from the pool
+    )
+)
+```
+
+Own the timeout at the provider level. The provider controls the connection; the timeout is part of its contract, not a concern of the service that calls it.
+
+`read` timeout is the critical one for ML providers. The default httpx timeout covers connection establishment only — a streaming response that hangs mid-transfer will wait indefinitely without an explicit `read` value.
+
+---
+
+## Retries
+
+Use `tenacity` for retry logic. It handles exponential backoff, jitter, and async correctly, and keeps retry policy visible at the call site:
+
+```python
+from tenacity import (
+    retry,
+    retry_if_exception_type,
+    stop_after_attempt,
+    wait_exponential_jitter,
+)
+from app.core.exceptions import TransientInferenceError
+
+@retry(
+    retry=retry_if_exception_type(TransientInferenceError),
+    wait=wait_exponential_jitter(initial=1, max=30),
+    stop=stop_after_attempt(3),
+)
+async def _call_with_retry(self, payload: dict) -> dict:
+    return await self._client.post(payload)
+```
+
+**Jitter is not optional.** Exponential backoff without jitter causes a thundering herd: all failed requests retry at the same moment, amplifying the load on a provider that is already struggling. `wait_exponential_jitter` adds random variance to break the synchronisation.
+
+**Only retry transient errors.** Map provider errors to domain exceptions before retrying. A `400 Bad Request` is a permanent failure — retrying wastes time and burns API budget. A `503 Service Unavailable` or a network timeout is transient:
+
+```python
+class AssemblyAIProvider:
+    async def transcribe(self, audio_uri: str) -> TranscriptionResult:
+        try:
+            return await self._call_with_retry(audio_uri)
+        except aai.errors.BadRequestError as e:
+            raise PermanentInferenceError("Invalid audio format", cause=e)
+        except (aai.errors.ServerError, httpx.TimeoutException) as e:
+            raise TransientInferenceError("AssemblyAI unavailable", cause=e)
+```
+
+The `retry_if_exception_type` check ensures only `TransientInferenceError` triggers a retry. Permanent errors surface immediately.
+
+---
+
+## Circuit Breakers
+
+A circuit breaker prevents a failing dependency from cascading into the rest of the service. When a provider fails repeatedly, the circuit opens and subsequent calls fail fast — protecting system resources and giving the provider time to recover.
+
+Use the `circuitbreaker` library:
+
+```python
+from circuitbreaker import circuit
+from app.core.exceptions import TransientInferenceError
+
+class OpenAIProvider:
+    @circuit(
+        failure_threshold=5,
+        recovery_timeout=30,
+        expected_exception=TransientInferenceError,
+    )
+    async def complete(self, prompt: str) -> str:
+        return await self._client.complete(prompt)
+```
+
+After 5 consecutive `TransientInferenceError` exceptions, the circuit opens. For the next 30 seconds, calls fail immediately without reaching the provider. After the recovery timeout, one probe request goes through — if it succeeds, the circuit closes; if not, it opens again.
+
+**Set `expected_exception` explicitly.** A circuit breaker that trips on all exceptions — including domain logic errors — produces false opens and makes the service appear unavailable when the problem is the caller, not the provider.
+
+---
+
+## Graceful Shutdown
+
+The service handles `SIGTERM` by draining in-flight requests before exiting. FastAPI and Uvicorn implement this correctly when `lifespan` is used:
+
+1. Kubernetes sends `SIGTERM`
+2. Uvicorn stops accepting new connections
+3. In-flight requests and `BackgroundTasks` complete
+4. `lifespan` cleanup runs: pools close, clients disconnect
+5. Process exits with code 0
+
+The only requirement on the application side: keep the `lifespan` shutdown path fast and non-blocking. Avoid long-running cleanup that delays exit beyond the Kubernetes `terminationGracePeriodSeconds`:
+
+```python
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    resources = await initialise_resources()
+    yield
+    # Shutdown must complete before Kubernetes force-kills the pod
+    await asyncio.wait_for(resources.close(), timeout=4.0)
+```
+
+---
+
+## Health and Readiness Probes
+
+Expose two endpoints. Kubernetes uses them to route traffic correctly:
+
+**`/healthz` — liveness.** The process is running and not deadlocked. Returns `200` unconditionally if the event loop is responding. No external dependency checks — a slow database must not cause a liveness failure, which would trigger a pod restart.
+
+**`/readyz` — readiness.** The service can handle traffic. Returns `200` only after startup completes (database pool initialised, Pub/Sub subscription established). Returns `503` during startup and during graceful shutdown so Kubernetes stops routing new requests to the pod before it terminates.
+
+```python
+@router.get("/healthz")
+async def liveness() -> dict:
+    return {"status": "ok"}
+
+@router.get("/readyz")
+async def readiness(state: AppState = Depends(get_app_state)) -> dict:
+    if not state.ready:
+        raise HTTPException(status_code=503, detail="not ready")
+    return {"status": "ready"}
+```
+
+Set `state.ready = False` at the start of the graceful shutdown sequence — before the process begins draining. This prevents Kubernetes from sending new requests to a pod that is about to exit.
+
+---
+
+## Graceful Degradation
+
+When a non-critical step fails, return a partial result rather than a hard error. A transcription without speaker labels is useful. An empty response is not.
+
+Define degradation explicitly at the service layer:
+
+```python
+async def transcribe_with_diarization(self, audio_uri: str) -> TranscriptionResult:
+    transcript = await self.transcriber.transcribe(audio_uri)
+
+    try:
+        speakers = await self.diarizer.diarise(transcript)
+        return transcript.with_speakers(speakers)
+    except DiarizationUnavailableError:
+        logger.warning("diarization.unavailable", meeting_id=transcript.meeting_id)
+        return transcript
+```
+
+Log every degraded response with structured context. Silent degradation — returning a reduced result without a log or metric — makes the failure mode invisible in production.
+
+---
+
+## Anti-Patterns
+
+**No timeout on provider calls.** Without an explicit `read` timeout, a hung provider holds the connection open indefinitely. Every `httpx.AsyncClient` needs timeout configuration.
+
+**Retrying non-transient errors.** A `400` or domain `ValidationError` will never succeed on retry. Map errors to domain types first; only `TransientInferenceError` retries.
+
+**Circuit breaker on all exceptions.** Trips on caller errors, producing false opens that make the service appear down when the provider is healthy.
+
+**Long-running lifespan shutdown.** Cleanup that blocks for more than a few seconds will be force-killed by Kubernetes, leaving resources in an inconsistent state.
+
+**Silent degradation.** A partial result returned without a structured log is invisible to the on-call engineer debugging why a non-critical feature is missing.