@nurix/etna 0.18.4 → 0.18.5

package/data/harnesses/hr/skills/evaluator/criteria/deployment-readiness.md

@@ -0,0 +1,414 @@
+# Deployment-Readiness — Extraction Plan
+
+- **Sub-criterion:** 1.3 (AI-only)
+- **Weight:** 12.5 pts (4 Code AI × 12.5 = 50 pts half)
+- **Contenders:** Filesystem probes + Hadolint + **observability markers** (structured-logging library presence, metrics/tracing library presence, `/health` endpoint detection) + LLM README/Dockerfile read (layered)
+- **Source quote:** _"Can this actually ship? Docker, config, health checks, or clear instructions to run it outside localhost."_
+
+Bands are **absolute** — per-signal thresholds in the _Banding_ section. Gates this recipe owns: **`DEMO`** (`deployment_readiness ≤ 1` when all 4 run-section/deploy signals are false OR README claims a deploy artifact that doesn't exist in the tree) and **`OBSERVABILITY`** (caps `deployment_readiness ≤ 3` when structured-logging library is absent AND metrics/tracing library is absent). Structured-logging detection: presence of `structlog` / `winston` / `pino` / `log-slog` / `zap` in deps OR JSON log format detected in code. Metrics/tracing detection: OpenTelemetry SDK / Prometheus client / Datadog agent / statsd client / a custom `/metrics` endpoint. Gate fires only when **both** categories are absent — a repo with structured logging but no metrics still ships above 3. Raw dump: `raw/deployment-readiness-observability-markers.json` with fields `{logging_lib: "<name>"|null, metrics_lib: "<name>"|null, has_health_endpoint: bool, json_log_detected: bool}`.
+
+> Five signals, integrated by the scorer. Serverless-native repos (no Dockerfile) can reach band 5 via PaaS/managed-platform path — see bands. No weighted-math internal split — the LLM emits a single 0–5 band and cites the most load-bearing signal.
+
+---
+
+## Extraction recipe
+
+### Signal 1 — Containerization (Docker OR devcontainer OR compose-with-image)
+
+```bash
+# Presence
+ls <repo>/Dockerfile \
+   <repo>/docker-compose.yml \
+   <repo>/docker-compose.yaml \
+   <repo>/compose.yaml \
+   <repo>/compose.override.yml \
+   <repo>/.dockerignore \
+   <repo>/.devcontainer/devcontainer.json \
+   <repo>/devcontainer.json 2>/dev/null
+
+# Dockerfile features (case-insensitive — Docker accepts `FROM ... as builder` too)
+grep -liE '^HEALTHCHECK'       <repo>/Dockerfile 2>/dev/null  # healthcheck
+grep -liE '^FROM[^#]*\bAS\b'   <repo>/Dockerfile 2>/dev/null  # multi-stage
+grep -liE '^USER[[:space:]]'   <repo>/Dockerfile 2>/dev/null  # non-root
+
+# Compose with inline image (no build:) → legitimate containerization without Dockerfile
+grep -lE '^\s*image:' <repo>/docker-compose.y*ml <repo>/compose.yaml 2>/dev/null
+
+# Lint (single Go binary; JSON out) — only if Dockerfile exists
+if [ -f <repo>/Dockerfile ]; then
+  hadolint --format json <repo>/Dockerfile > /tmp/hadolint.json
+fi
+```
+
+Capture:
+
+- `has_dockerfile`, `has_compose`, `has_dockerignore`, `has_devcontainer`
+- `compose_has_inline_image` (compose without Dockerfile is legitimate)
+- `docker_has_healthcheck`, `docker_multi_stage`, `docker_non_root`
+- `hadolint_findings` (total), `hadolint_by_severity` (`error`/`warning`/`info`/`style`)
+- `has_container_artifact` = `has_dockerfile OR has_devcontainer OR compose_has_inline_image`
+
+### Signal 2 — Config hygiene
+
+```bash
+# env-example presence
+ls <repo>/{.env.example,.env.sample,.env.template} 2>/dev/null
+
+# .env gitignored?
+grep -E '^\.env(\.|$)' <repo>/.gitignore 2>/dev/null
+
+# Any tracked .env file? (should be zero)
+find <repo> -maxdepth 3 -name '.env' \
+  -not -path '*/node_modules/*' -not -path '*/.venv/*'
+```
+
+Capture: `has_env_example`, `env_in_gitignore`, `tracked_env_files` (list; should be empty).
+
+### Signal 3 — Deploy artifacts (GHA / PaaS / IaC)
+
+```bash
+# GHA deploy workflows
+ls <repo>/.github/workflows/ 2>/dev/null \
+  | grep -iE 'deploy|release|publish|ship'
+
+# PaaS configs — broad 2026 landscape
+ls <repo>/{\
+vercel.json,netlify.toml,\
+fly.toml,\
+railway.json,railway.toml,\
+render.yaml,\
+Procfile,app.json,app.yaml,apprunner.yaml,\
+wrangler.toml,wrangler.jsonc,\
+serverless.yml,template.yaml,cdk.json,sst.config.ts,Pulumi.yaml,\
+modal.toml,\
+.replit,replit.nix,\
+nixpacks.toml,project.toml,\
+supabase/config.toml,\
+.do/app.yaml,\
+koyeb.yaml,zeabur.json,\
+dagger.json,porter.yaml\
+} 2>/dev/null
+
+# Cloudflare Pages markers (no single file; check for any)
+ls <repo>/{_routes.json,_headers,_redirects} 2>/dev/null
+
+# IaC directories
+ls -d <repo>/{k8s,kubernetes,manifests,helm,terraform,pulumi,cdk} 2>/dev/null
+
+# Helm chart detection
+ls <repo>/Chart.yaml 2>/dev/null
+
+# Kustomize
+find <repo> -maxdepth 3 -name 'kustomization.yaml' 2>/dev/null
+```
+
+Capture:
+
+- `deploy_workflow_files` (list)
+- `paas_configs` (list of present files; extend the list as new deploy targets appear)
+- `iac_dirs` (list of present dirs)
+- `has_any_deploy_artifact` = bool (OR across above + Signal 1 containerization)
+
+### Signal 4 — Health checks + localhost counter-signal
+
+```bash
+# Health route definitions (broad regex, all stacks)
+grep -rnE '["'\''/](health(z)?|readyz|livez|liveness|readiness|ping|status)["'\''/\s(]' \
+  <repo>/src <repo>/app <repo>/internal 2>/dev/null \
+  | head -20
+
+# K8s liveness/readiness probes (if iac_dirs includes k8s/)
+grep -rE 'livenessProbe|readinessProbe' <repo>/k8s <repo>/helm 2>/dev/null
+
+# Localhost hardcoding in non-test source (counter-signal)
+# Exclude lines that reference env vars (fine: `process.env.X || 'localhost:3000'` is a dev default, not hardcoding)
+grep -rnE 'localhost:[0-9]+|127\.0\.0\.1:[0-9]+|http://localhost' \
+  <repo>/src <repo>/app \
+  --include='*.py' --include='*.ts' --include='*.tsx' \
+  --include='*.js' --include='*.jsx' --include='*.go' \
+  --exclude-dir=node_modules --exclude-dir=__tests__ \
+  --exclude-dir=.next --exclude-dir=dist --exclude-dir=build \
+  2>/dev/null \
+  | grep -vE 'process\.env|os\.environ|ENV\[|getenv|import\.meta\.env|viper\.Get|os\.Getenv' \
+  | wc -l
+```
+
+Capture:
+
+- `health_route_hits` (count; 0 = none)
+- `k8s_probes_present` (bool)
+- `localhost_hardcoded_count` (non-test source, env-var-filtered)
+
+### Signal 5 — README read (mechanical + LLM two-signal)
+
+**Mechanical — README section heading presence:**
+
+```bash
+grep -iE '^##+\s*(Deploy|Deployment|Production|Hosting|Running in production|Getting Started|Installation|Quickstart|Usage|How to run|Setup)' \
+  <repo>/README.md <repo>/readme.md 2>/dev/null
+```
+
+Capture: `readme_has_run_section` (bool — any matching heading present).
+
+**LLM — quality read:** Read and score:
+
+- **README.md** (or `readme.md` / `README.rst`)
+- **Dockerfile** (if present)
+
+Score each on:
+
+- Does the README teach a stranger to run the app **outside localhost**? (deploy target named, deploy command given, env vars documented?)
+- Is the Dockerfile production-shaped (multi-stage, slim base, pinned versions, non-root, HEALTHCHECK)?
+- Anti-signals: only `npm run dev` / `python app.py`; no env docs; localhost-only language.
+
+Capture:
+
+- `llm_readme_score` (0–5)
+- `llm_dockerfile_score` (0–5 or `null` if absent)
+- `llm_has_run_section` (bool)
+- `llm_confidence` (`high`/`medium`/`low`)
+- `llm_notes` (one sentence)
+
+### Signal 6 — Observability markers (OBSERVABILITY gate trigger)
+
+Ships-to-prod requires signal — structured logs and metrics/traces. This signal looks for evidence that either is wired into the app. **Both** must be absent for the gate to fire; one present means the repo has the basics.
+
+```bash
+# Structured-logging library — check deps (manifest) + imports (code) + JSON-log
+# format heuristic. Any one match lets the repo off the gate's logging arm.
+LOG_LIBS="structlog|loguru|winston|pino|log-slog|uber-go/zap|go.uber.org/zap"
+
+# Dependency declaration — one of:
+if [ -f <repo>/pyproject.toml ]; then
+  grep -iE "$LOG_LIBS" <repo>/pyproject.toml 2>/dev/null
+fi
+if [ -f <repo>/requirements.txt ]; then
+  grep -iE "$LOG_LIBS" <repo>/requirements.txt 2>/dev/null
+fi
+if [ -f <repo>/package.json ]; then
+  jq -r '(.dependencies // {}) + (.devDependencies // {}) | keys[]' \
+    <repo>/package.json 2>/dev/null | grep -iE "$LOG_LIBS"
+fi
+if [ -f <repo>/go.mod ]; then
+  grep -iE "$LOG_LIBS" <repo>/go.mod 2>/dev/null
+fi
+
+# Import sighting in source — covers vendored or pinned-but-renamed deps.
+grep -rnE "import.*\b($LOG_LIBS)\b|from\s+($LOG_LIBS)" \
+  <repo>/src <repo>/app <repo>/internal \
+  --include='*.py' --include='*.ts' --include='*.tsx' \
+  --include='*.js' --include='*.jsx' --include='*.go' \
+  --exclude-dir=node_modules --exclude-dir=.venv \
+  2>/dev/null | head -20
+
+# JSON log-format heuristic — bare `json.dumps(...)` inside what looks like a
+# log call. Weak signal; participates only as a fallback. Evaluator should
+# confirm with LLM when this is the only logging-arm match.
+grep -rnE '(logger|log)\.(info|warn|error|debug)\([^)]*json\.(dumps|stringify)' \
+  <repo>/src <repo>/app \
+  --include='*.py' --include='*.ts' --include='*.js' --include='*.go' \
+  --exclude-dir=node_modules 2>/dev/null | head -5
+
+# Metrics / tracing library — OpenTelemetry SDK, Prometheus client,
+# Datadog agent, statsd client, or a custom /metrics endpoint.
+METRICS_LIBS="opentelemetry|otel-|prom-client|prometheus_client|prometheus-client|ddtrace|datadog|statsd|hot-shots"
+
+# Dependency + import sighting (same pattern as logging)
+if [ -f <repo>/pyproject.toml ]; then
+  grep -iE "$METRICS_LIBS" <repo>/pyproject.toml 2>/dev/null
+fi
+if [ -f <repo>/package.json ]; then
+  jq -r '(.dependencies // {}) + (.devDependencies // {}) | keys[]' \
+    <repo>/package.json 2>/dev/null | grep -iE "$METRICS_LIBS"
+fi
+if [ -f <repo>/go.mod ]; then
+  grep -iE "$METRICS_LIBS" <repo>/go.mod 2>/dev/null
+fi
+grep -rnE "import.*\b($METRICS_LIBS)\b|from\s+($METRICS_LIBS)" \
+  <repo>/src <repo>/app <repo>/internal \
+  --include='*.py' --include='*.ts' --include='*.tsx' \
+  --include='*.js' --include='*.jsx' --include='*.go' \
+  --exclude-dir=node_modules --exclude-dir=.venv \
+  2>/dev/null | head -20
+
+# Custom /metrics endpoint — Prometheus convention. Matches route definitions.
+grep -rnE '["/]metrics["/\s(]' <repo>/src <repo>/app \
+  <repo>/internal --include='*.py' --include='*.ts' \
+  --include='*.js' --include='*.go' \
+  --exclude-dir=node_modules 2>/dev/null | head -10
+```
+
+Assemble observability markers JSON (the sub-agent writes the interpreted fields, not the raw grep):
+
+```json
+{
+  "logging_lib": "structlog" | null,
+  "logging_detected_via": "manifest" | "import" | "json-format-heuristic" | null,
+  "metrics_lib": "opentelemetry" | null,
+  "metrics_detected_via": "manifest" | "import" | "metrics-endpoint" | null,
+  "has_metrics_endpoint": true | false,
+  "has_health_endpoint": true | false,
+  "json_log_detected": true | false
+}
+```
+
+Capture:
+
+- `logging_lib` — name of the structured-logging library, or `null` if none found
+- `metrics_lib` — name of the metrics/tracing library, or `null` if none found
+- `has_metrics_endpoint` — bool (custom `/metrics` route found)
+- `has_json_log_format` — bool (heuristic match; low-confidence tiebreaker only)
+- `observability_logging_present` = `(logging_lib != null) OR has_json_log_format`
+- `observability_metrics_present` = `(metrics_lib != null) OR has_metrics_endpoint`
+
+**OBSERVABILITY gate trigger:** `observability_logging_present == false` AND `observability_metrics_present == false`. Both must be absent for the gate to fire — one present is enough to avoid the cap. Caps `deployment_readiness ≤ 3`.
+
+---
+
+## Raw dumps (flat — files under `raw/` named `deployment-readiness-*`)
+
+Each signal writes its raw output to the flat `raw/` directory at the workspace root. Absent files = not extracted / not applicable. Aggregates go to `scorecard.yaml` under `data.`.
+
+| File | Source signal | Shape / note |
+| --- | --- | --- |
+| `raw/deployment-readiness-artifacts.json` | Signals 1 + 3 + 4 | Combined filesystem probe: `{container: {dockerfile, compose, devcontainer, healthcheck, multi_stage, non_root}, deploy: {gha_workflows, paas_configs, iac_dirs, helm, kustomize}, env: {env_example, env_in_gitignore, tracked_env_files}, health: {routes, k8s_probes}}` |
+| `raw/deployment-readiness-hadolint.json` | Signal 1 | Raw `hadolint --format json` output (absent when no Dockerfile) |
+| `raw/deployment-readiness-localhost-refs.txt` | Signal 4 | `grep -rnE 'localhost:...\|127.0.0.1:...'` output with `file:line`, after env-var exclusion (absent when count = 0) |
+| `raw/deployment-readiness-llm-read.md` | Signal 5 | LLM's README + Dockerfile analysis: `{readme_score, dockerfile_score, has_run_section, confidence, one_sentence}` plus a short paragraph of rationale |
+| `raw/deployment-readiness-observability-markers.json` | Signal 6 | Structured object: `{logging_lib, logging_detected_via, metrics_lib, metrics_detected_via, has_metrics_endpoint, has_health_endpoint, json_log_detected}` — feeds the OBSERVABILITY gate decision |
+
+---
+
+## Aggregation → 0–5 band
+
+Scorer integrates the 5 signals. Rough directionality:
+
+| Signal | Pulls score UP | Pulls score DOWN |
+| --- | --- | --- |
+| `has_container_artifact` + (Hadolint-clean if Dockerfile) | Present + 0 Hadolint errors | Absent or > 5 Hadolint errors |
+| `docker_has_healthcheck` / `docker_non_root` / `docker_multi_stage` | All true | None true |
+| `has_env_example` + `env_in_gitignore` + zero tracked `.env` | All clean | Any violation |
+| `has_any_deploy_artifact` (container / GHA / PaaS / IaC) | ≥ 2 distinct artifacts | None |
+| `health_route_hits` or `k8s_probes_present` or Docker `HEALTHCHECK` | ≥ 1 present | 0 |
+| `localhost_hardcoded_count` | 0 in non-test source | > 3 |
+| `llm_readme_score` | 4–5 | 0–2 |
+| `llm_dockerfile_score` | 4–5 | 0–2 |
+
+**Bands (absolute):**
+
+| Band | Path A — container-first | Path B — serverless-native |
+| --- | --- | --- |
+| **5** | Dockerfile Hadolint-clean with multi-stage + USER + HEALTHCHECK + `.env.example` + health route + deploy workflow + README LLM ≥ 4 + zero localhost hardcoding | PaaS/managed-platform config + deploy workflow + `.env.example` + health route (HTTP or framework) + README LLM ≥ 4 + zero localhost hardcoding |
+| **4** | Strong container path, one weaker signal | Strong serverless path, one weaker signal |
+| **3** | Some container or platform artifact; env vars referenced; mixed other signals |
+| **2** | Minimal — one ship artifact present but weak README / no health / no env docs |
+| **1** | DEMO gate fires (see below); localhost-only repo |
+| **0** | Suppressed (DEMO gate caps at 1) |
+
+Serverless-native repos (Vercel/Fly/Cloudflare/Modal) reach band 5 via Path B — do not penalize absent Dockerfile if a PaaS config carries the deploy artifact.
+
+---
+
+## DEMO gate — two-signal agreement
+
+Gate fires only when **all three** are absent AND the LLM + mechanical README read agree:
+
+```
+has_container_artifact == false
+AND has_any_deploy_artifact == false   (i.e. no GHA deploy, no PaaS config, no IaC)
+AND readme_has_run_section == false    (mechanical heading grep)
+AND llm_has_run_section == false        (LLM read)
+```
+
+**Why two-signal:** a single LLM judgment as a hard gate means one noisy call caps the score. Requiring heading grep + LLM agreement makes the gate robust to LLM noise. If `llm_confidence == low`, defer to mechanical-only (skip the LLM side of the AND).
+
+When fired, emit `gates_triggered: [DEMO]` and cap `deployment_readiness ≤ 1`.
+
+---
+
+## OBSERVABILITY gate — both arms absent
+
+Gate fires only when **both** observability arms are absent:
+
+```
+observability_logging_present == false
+  AND
+observability_metrics_present == false
+```
+
+That is: the repo ships neither a structured-logging library (structlog / winston / pino / log-slog / zap / a detected JSON-log format) **nor** a metrics/tracing library (OpenTelemetry / Prometheus client / Datadog / statsd / a custom `/metrics` endpoint).
+
+**Why both-arms:** enterprise observability is "logs AND metrics", but a single-binary CLI or a thin FaaS handler may legitimately ship with just structured logs and skip metrics (or vice versa). Demanding both is punitive on small, valid shapes. The gate catches the repo that's shipping without _any_ production-grade signal — `print()`-only logging and no metrics — which is the failure we care about.
+
+When fired, emit `gates_triggered: [OBSERVABILITY]` and cap `deployment_readiness ≤ 3`. Absent arm names ship in the evidence string: `"no structured logger AND no metrics/tracing — cap at 3"`.
+
+---
+
+## Fallback — signals missing
+
+- No Dockerfile → drop Hadolint lint; rely on Signals 2, 3, 4, 5 + `has_devcontainer` / `compose_has_inline_image` as container signal.
+- Hadolint not installed → drop lint component; rely on LLM Dockerfile read.
+- Fully serverless (Vercel/Fly only, no Dockerfile) → Path B; PaaS config in Signal 3 carries deploy artifact weight.
+- LLM low-confidence → DEMO gate falls back to mechanical-only (README heading grep + artifact absence).
+
+Annotate evidence with `"Signals used: [...]"` and `"Path: A | B"` so the audit trail is explicit.
+
+---
+
+## Output (what the scorer emits)
+
+```yaml
+sub_criterion: deployment_readiness
+score: 3
+evidence: "Dockerfile present (multi-stage, non-root, no HEALTHCHECK; Hadolint 2 warnings); .env.example + .env gitignored; no deploy workflow; README has install steps but no outside-localhost deploy section; 4 hardcoded localhost refs. Path A."
+data:
+  has_dockerfile: true
+  has_compose: false
+  has_dockerignore: true
+  has_devcontainer: false
+  compose_has_inline_image: false
+  has_container_artifact: true
+  docker_has_healthcheck: false
+  docker_multi_stage: true
+  docker_non_root: true
+  hadolint_findings: 2
+  hadolint_by_severity: { error: 0, warning: 2, info: 0, style: 0 }
+  has_env_example: true
+  env_in_gitignore: true
+  tracked_env_files: []
+  deploy_workflow_files: []
+  paas_configs: []
+  iac_dirs: []
+  has_any_deploy_artifact: true # Dockerfile counts as containerization
+  health_route_hits: 0
+  k8s_probes_present: false
+  localhost_hardcoded_count: 4
+  readme_has_run_section: true
+  llm_readme_score: 2
+  llm_dockerfile_score: 3
+  llm_has_run_section: true
+  llm_confidence: medium
+  logging_lib: structlog
+  logging_detected_via: manifest
+  metrics_lib: null
+  metrics_detected_via: null
+  has_metrics_endpoint: false
+  has_json_log_format: true
+  observability_logging_present: true
+  observability_metrics_present: false
+  path: A
+  signals_used: [filesystem, hadolint, llm, observability-markers]
+  gate_demo_fired: false
+  gate_observability_fired: false
+```
+
+---
+
+## Caveats
+
+- File presence ≠ working artifact. Only runtime verification (a separate manual pass) catches broken Dockerfiles / deploy workflows.
+- Hadolint catches form, not intent. A "clean" Dockerfile that installs nothing still lints green.
+- PaaS probe list needs periodic update — new deploy targets appear faster than the list evolves.
+- LLM sample is 1 README. Repos with multi-file docs (`docs/`, `CONTRIBUTING.md`, wiki) may be under-scored on the interpretive signal. Consider glob-and-concat for docs-heavy repos.
+- Localhost grep is a noisy counter-signal even with env-var exclusion — LLM notes should adjudicate when the count is > 0 but reasonable.
+- Two-signal DEMO gate may still false-negative on repos whose README has a heading like `## Getting Started` that only says `npm run dev` (LLM catches this; mechanical does not). That's why LLM is in the AND, not replaced.