@balpal4495/quorum 0.2.0 → 0.3.0

package/README.md

@@ -65,6 +65,12 @@ When a decision is made, your AI stages a Chronicle entry using `oracle.propose(
 
 Commit `.chronicle/committed/` to git. Future sessions — and your teammates' sessions — start with that context.
 
+### Every merged PR creates a Chronicle proposal automatically
+
+A GitHub Actions workflow fires when any PR merges to main. It creates a Chronicle proposal capturing the decision, which files changed, and any explicitly deferred items from the PR description. The proposal sits in `proposals/` until you commit it — nothing is auto-indexed.
+
+This means the gap between "PR merged" and "Chronicle knows about it" is now zero.
+
 ---
 
 ## Real examples
@@ -111,15 +117,36 @@ gaps: ["no lock strategy documented", "no rollback plan"]
 council_brief: challenge
 ```
 
-Council's Chairman gives a verdict:
-
-```
-satisfied: false
-verdict: "On a table this size, a naive ALTER TABLE takes an exclusive lock for minutes.
-          Specify a shadow column pattern or pg_repack. No rollback plan documented."
+Council's Chairman gives a structured verdict:
+
+```json
+{
+  "satisfied": false,
+  "blockers": [
+    {
+      "issue": "Naive ALTER TABLE takes an exclusive lock for minutes on a 50M-row table",
+      "evidence": ["db-017"],
+      "required_fix": "Use shadow column pattern or pg_repack. Add rollback path."
+    }
+  ],
+  "warnings": [],
+  "advisor_split": { "proceed": 0, "redesign": 4, "investigate-more": 1 }
+}
 ```
 
-The agent revises the plan. You approve the Chronicle entry once it's solid. The reasoning is on record for the next time someone touches that table.
+The agent revises the plan. You approve the Chronicle entry once it's solid. The reasoning — including alternatives considered and why they were rejected — is on record for the next time someone touches that table:
+
+```json
+{
+  "decision": "Use shadow column pattern for NOT NULL migration on users table",
+  "alternatives_considered": ["naive ALTER TABLE", "pg_repack"],
+  "rejected_reason": ["ALTER TABLE takes exclusive lock for minutes on 50M rows"],
+  "scope": ["database", "migrations"],
+  "affected_areas": ["db/migrations/", "src/models/user.ts"],
+  "validation_plan": ["Confirm 100% backfill before applying NOT NULL constraint", "Test rollback path on staging"],
+  "review_after": "2026-08-01"
+}
+```
 
 ---
 
@@ -130,14 +157,96 @@ Four portable TypeScript modules installed into `quorum/modules/`:
 | Module | What it does |
 |---|---|
 | **Oracle** | Query and write interface to Chronicle. No LLM required. |
-| **Jury** | Evaluates a proposed design against Chronicle evidence. Returns a confidence score. |
-| **Council** | A panel of advisors challenges the design independently, reviewers critique anonymously, a Chairman gives a final verdict. |
+| **Jury** | Evaluates a proposed design against Chronicle evidence. Returns a decomposed confidence score and hard-blocker gaps. |
+| **Council** | A panel of advisors challenges the design independently, reviewers critique anonymously, a Chairman gives a structured verdict with blockers and warnings. |
 | **Sentinel** | Shows which files the AI knows nothing about, flags stale knowledge, and posts a coverage map on every PR. |
 
 The modules live in your repo — readable by any AI working in the codebase. Nothing is hidden in `node_modules`.
 
 ---
 
+## How Jury works
+
+Before calling the LLM, Jury runs a **deterministic preflight** — no LLM required — that checks whether the design touches sensitive areas (auth, database migrations, crypto, payments, PII, secrets), mentions a rollback strategy, and whether any refuted Chronicle entries conflict with the design. These facts are injected into the Jury prompt as hard ground truth.
+
+The LLM then scores the design across four dimensions:
+
+| Dimension | What it measures |
+|---|---|
+| Evidence support | Do validated Chronicle entries confirm this approach works here? |
+| Feasibility | Do Chronicle entries suggest this is achievable? |
+| Risk | How well does the design address known failure modes? |
+| Completeness | Does the design cover the full outcome? |
+
+Confidence is recomputed as the exact average of those four scores — the LLM's stated confidence is discarded. Jury also separates `blocking_gaps` (must resolve before proceeding) from `gaps` (useful but not critical).
+
+---
+
+## How Council works
+
+Before running the full panel, a **risk classifier** reads the design text and Chronicle evidence and assigns a risk level:
+
+| Risk | Council mode | LLM calls |
+|---|---|---|
+| Low | 1 advisor + 1 reviewer | 4 |
+| Medium | 1 advisor + 2 reviewers | 5 |
+| High | 5 advisors + 5 reviewers | 12 |
+| Critical | 5 advisors + 5 reviewers (+ human architecture flag) | 12 |
+
+Auth, crypto, payments, and data deletion trigger Critical. Database migrations, PII, permissions trigger High. Cache, queues, deployments trigger Medium. Everything else is Low.
+
+The Chairman's verdict is **structured**:
+
+```json
+{
+  "blockers": [
+    {
+      "issue": "No rollback plan for destructive migration",
+      "evidence": ["db-017"],
+      "required_fix": "Add shadow-column migration and rollback path before execution"
+    }
+  ],
+  "warnings": [
+    {
+      "issue": "No integration test for token expiry edge case",
+      "suggested_fix": "Add test covering token rotation during concurrent requests"
+    }
+  ],
+  "advisor_split": { "proceed": 2, "redesign": 2, "investigate-more": 1 }
+}
+```
+
+Blockers must be resolved before the human gate. Warnings can be ticketed. High `advisor_split` disagreement is surfaced explicitly — it means genuine uncertainty, not a safe proceed.
+
+Every Oracle ID cited in the verdict is also validated against the evidence pack that was actually sent. Hallucinated citations are flagged in `citation_validation.hallucinated_ids` and stripped from the Chronicle proposal.
+
+---
+
+## Eval suite
+
+`evals/` contains canonical test cases — known-bad proposals that Council should block and known-good ones it should pass:
+
+| Case | Expected outcome |
+|---|---|
+| Naive NOT NULL migration on large table | Block — no lock strategy |
+| HS256 JWT when RS256 was already chosen | Block — cites refuted entry auth-022 |
+| PII fields logged to stdout | Block — GDPR violation in evidence |
+| Payment charge without idempotency key | Block — duplicate charge risk |
+| Redis sessions (previously removed) | Block — memory overhead already documented |
+| Cache without stampede protection | Block — prior incident in Chronicle |
+| Safe internal rename | Proceed — low risk, no conflicts |
+| RS256 JWT (approved pattern) | Proceed — matches validated Chronicle entry |
+| Migration with rollback + shadow column | Proceed — addresses documented failure mode |
+| Novel WebSocket design, no evidence | Investigate-more — no Chronicle evidence either way |
+
+Deterministic assertions (preflight, risk classifier) run on every CI pass. LLM-dependent assertions (confidence bounds, Council recommendation) activate with `EVAL_LLM=1`.
+
+```bash
+npx vitest run evals/
+```
+
+---
+
 ## Sentinel — coverage and drift
 
 Sentinel surfaces two things Chronicle can't tell you about itself.

package/SETUP.md

@@ -197,11 +197,17 @@ If the directory is not created, re-check that `setup()` is being awaited correc
 Confirm the modules are working in this environment:
 
 ```bash
+# Module unit tests
 npx vitest run quorum/modules/
+
+# Eval suite — deterministic assertions, no LLM required
+npx vitest run quorum/evals/
 ```
 
 All tests should pass. If they fail due to missing dependencies, re-run Step 3.
 
+The eval suite runs canonical test cases (known-bad proposals that should block, known-good ones that should pass) through the deterministic preflight and risk classifier. These pass without any LLM. If you later want to test Jury confidence and Council recommendations against a real LLM, set `EVAL_LLM=1` when running.
+
 ---
 
 ## Step 9 — Report what was done

package/evals/__tests__/eval.test.ts

@@ -0,0 +1,31 @@
+/**
+ * Eval suite — runs all cases from evals/cases/ through deterministic checks.
+ *
+ * Deterministic assertions (preflight, risk classifier) run on every CI pass.
+ * LLM-dependent assertions (jury confidence, council recommendation) are skipped
+ * unless EVAL_LLM env var is set — they are too slow and costly for standard CI.
+ *
+ * To run with a real LLM locally:
+ *   EVAL_LLM=1 OPENAI_API_KEY=sk-... npx vitest run evals/
+ */
+import { describe, it, expect } from "vitest"
+import path from "path"
+import { loadCases, runCase } from "../runner"
+
+describe("eval suite — deterministic checks", async () => {
+  const cases = await loadCases(path.join(__dirname, "../cases"))
+
+  for (const evalCase of cases) {
+    it(`[${evalCase.id}] ${evalCase.description}`, async () => {
+      // No LLM — only runs deterministic assertions (preflight + risk classifier)
+      const result = await runCase(evalCase)
+
+      if (!result.passed) {
+        // Surface all failures clearly
+        throw new Error(
+          `Eval case "${evalCase.id}" failed:\n${result.failures.map(f => `  • ${f}`).join("\n")}`,
+        )
+      }
+    })
+  }
+})

package/evals/cases/auth_hs256_rejected.json

@@ -0,0 +1,46 @@
+{
+  "id": "auth_hs256_rejected",
+  "description": "Proposing HS256 JWT when RS256 was already chosen — should block",
+  "outcome": "Add JWT authentication to the API",
+  "design": "Use HS256 symmetric JWT tokens with a shared secret stored in environment variables",
+  "oracle_evidence": [
+    {
+      "id": "auth-022",
+      "key_insight": "HS256 JWT rejected — no way to rotate keys without invalidating all active sessions",
+      "decision": "HS256 JWT rejected — no way to rotate keys without invalidating all active sessions. Use RS256.",
+      "schema_version": 2,
+      "affected_areas": ["src/auth/", "src/middleware/"],
+      "scope": ["auth", "sessions"],
+      "status": "refuted",
+      "confidence": 0.91,
+      "source_module": "council",
+      "evidence_cited": [],
+      "timestamp": "2025-03-01T09:00:00Z"
+    },
+    {
+      "id": "auth-031",
+      "key_insight": "RS256 with short-lived tokens and refresh rotation in httpOnly cookies is the approved pattern",
+      "decision": "RS256 with 15-min access tokens and refresh rotation stored in httpOnly cookies is the approved auth pattern",
+      "schema_version": 2,
+      "affected_areas": ["src/auth/", "src/middleware/"],
+      "scope": ["auth", "sessions"],
+      "status": "validated",
+      "confidence": 0.88,
+      "source_module": "council",
+      "evidence_cited": ["auth-022"],
+      "timestamp": "2025-03-15T11:00:00Z"
+    }
+  ],
+  "expected": {
+    "jury_max_confidence": 0.45,
+    "council_recommendation": "redesign",
+    "must_flag": ["key rotation", "HS256"],
+    "must_cite": ["auth-022"],
+    "risk_level": "critical",
+    "preflight_expects": {
+      "touches_sensitive_area": true,
+      "sensitive_areas_include": ["auth"],
+      "chronicle_conflicts": ["auth-022"]
+    }
+  }
+}

package/evals/cases/auth_rs256_valid.json

@@ -0,0 +1,30 @@
+{
+  "id": "auth_rs256_valid",
+  "description": "Proposing the already-approved RS256 pattern — should proceed",
+  "outcome": "Add JWT authentication to the API",
+  "design": "RS256 tokens with 15-minute expiry and refresh rotation stored in httpOnly cookies, matching the approved pattern in Chronicle",
+  "oracle_evidence": [
+    {
+      "id": "auth-031",
+      "key_insight": "RS256 with short-lived tokens and refresh rotation in httpOnly cookies is the approved pattern",
+      "decision": "RS256 with 15-min access tokens and refresh rotation stored in httpOnly cookies is the approved auth pattern",
+      "schema_version": 2,
+      "affected_areas": ["src/auth/", "src/middleware/"],
+      "scope": ["auth", "sessions"],
+      "status": "validated",
+      "confidence": 0.88,
+      "source_module": "council",
+      "evidence_cited": [],
+      "timestamp": "2025-03-15T11:00:00Z"
+    }
+  ],
+  "expected": {
+    "jury_min_confidence": 0.65,
+    "council_recommendation": "proceed",
+    "must_not_flag": ["key rotation problem"],
+    "risk_level": "critical",
+    "preflight_expects": {
+      "touches_sensitive_area": true
+    }
+  }
+}

package/evals/cases/cache_missing_lock.json

@@ -0,0 +1,31 @@
+{
+  "id": "cache_missing_lock",
+  "description": "Cache implementation missing stampede protection — should warn or block",
+  "outcome": "Cache expensive product catalogue queries in Redis",
+  "design": "On cache miss, fetch from database and write to Redis with a 5-minute TTL. No locking strategy.",
+  "oracle_evidence": [
+    {
+      "id": "cache-008",
+      "key_insight": "Redis cache without stampede protection caused DB overload during peak traffic",
+      "decision": "Redis cache without stampede protection caused DB overload — all cache misses hit DB simultaneously during spikes",
+      "schema_version": 2,
+      "affected_areas": ["src/cache/", "src/api/products.ts"],
+      "scope": ["cache", "performance"],
+      "status": "refuted",
+      "confidence": 0.85,
+      "source_module": "council",
+      "evidence_cited": [],
+      "timestamp": "2025-02-20T14:00:00Z"
+    }
+  ],
+  "expected": {
+    "jury_max_confidence": 0.60,
+    "council_recommendation": "redesign",
+    "must_flag": ["stampede", "lock"],
+    "must_cite": ["cache-008"],
+    "risk_level": "medium",
+    "preflight_expects": {
+      "chronicle_conflicts": ["cache-008"]
+    }
+  }
+}

package/evals/cases/db_naive_not_null.json

@@ -0,0 +1,32 @@
+{
+  "id": "db_naive_not_null",
+  "description": "Naive NOT NULL migration on a large table — should block",
+  "outcome": "Add a NOT NULL region column to the users table",
+  "design": "Run ALTER TABLE users ADD COLUMN region TEXT NOT NULL DEFAULT 'uk'",
+  "oracle_evidence": [
+    {
+      "id": "db-017",
+      "key_insight": "Avoid naive locking migrations on large tables",
+      "decision": "Avoid naive locking migrations on large tables — use shadow column pattern or pg_repack",
+      "schema_version": 2,
+      "affected_areas": ["db/migrations", "src/models/user.ts"],
+      "scope": ["database", "migrations"],
+      "status": "refuted",
+      "confidence": 0.91,
+      "source_module": "council",
+      "evidence_cited": [],
+      "timestamp": "2025-01-10T10:00:00Z"
+    }
+  ],
+  "expected": {
+    "jury_max_confidence": 0.55,
+    "council_recommendation": "redesign",
+    "must_flag": ["lock strategy", "rollback"],
+    "must_cite": ["db-017"],
+    "risk_level": "high",
+    "preflight_expects": {
+      "rollback_mentioned": false,
+      "chronicle_conflicts": ["db-017"]
+    }
+  }
+}

package/evals/cases/logging_pii_leak.json

@@ -0,0 +1,32 @@
+{
+  "id": "logging_pii_leak",
+  "description": "Logging request bodies that contain PII — should block",
+  "outcome": "Add structured request logging to the API for debugging",
+  "design": "Log full request body including email, name, and any user-submitted fields to stdout using JSON format",
+  "oracle_evidence": [
+    {
+      "id": "sec-003",
+      "key_insight": "Logging PII fields violates GDPR and our data retention policy",
+      "decision": "Logging PII fields (email, name, address) violates GDPR Article 5 — all request logging must scrub or omit PII fields",
+      "schema_version": 2,
+      "affected_areas": ["src/middleware/logger.ts", "src/api/"],
+      "scope": ["pii", "compliance", "logging"],
+      "status": "validated",
+      "confidence": 0.95,
+      "source_module": "council",
+      "evidence_cited": [],
+      "timestamp": "2025-01-05T08:00:00Z"
+    }
+  ],
+  "expected": {
+    "jury_max_confidence": 0.40,
+    "council_recommendation": "redesign",
+    "must_flag": ["PII", "GDPR", "email"],
+    "must_cite": ["sec-003"],
+    "risk_level": "high",
+    "preflight_expects": {
+      "touches_sensitive_area": true,
+      "sensitive_areas_include": ["pii"]
+    }
+  }
+}

package/evals/cases/migration_with_rollback.json

@@ -0,0 +1,43 @@
+{
+  "id": "migration_with_rollback",
+  "description": "DB migration that explicitly addresses rollback and uses safe pattern — should proceed",
+  "outcome": "Add a NOT NULL region column to the users table",
+  "design": "Use shadow column pattern: add region TEXT NULLABLE, backfill via batched update, then add NOT NULL constraint after 100% fill confirmed. Rollback: drop shadow column. Uses pg_repack to avoid exclusive locks.",
+  "oracle_evidence": [
+    {
+      "id": "db-017",
+      "key_insight": "Avoid naive locking migrations on large tables — use shadow column pattern or pg_repack",
+      "decision": "Avoid naive locking migrations on large tables — use shadow column pattern or pg_repack",
+      "schema_version": 2,
+      "affected_areas": ["db/migrations", "src/models/user.ts"],
+      "scope": ["database", "migrations"],
+      "status": "refuted",
+      "confidence": 0.91,
+      "source_module": "council",
+      "evidence_cited": [],
+      "timestamp": "2025-01-10T10:00:00Z"
+    },
+    {
+      "id": "db-019",
+      "key_insight": "Shadow column pattern with batched backfill is the approved approach for NOT NULL migrations",
+      "decision": "Shadow column pattern with batched backfill is the approved approach for large NOT NULL migrations",
+      "schema_version": 2,
+      "affected_areas": ["db/migrations"],
+      "scope": ["database", "migrations"],
+      "status": "validated",
+      "confidence": 0.87,
+      "source_module": "council",
+      "evidence_cited": ["db-017"],
+      "timestamp": "2025-02-01T12:00:00Z"
+    }
+  ],
+  "expected": {
+    "jury_min_confidence": 0.65,
+    "council_recommendation": "proceed",
+    "risk_level": "high",
+    "preflight_expects": {
+      "rollback_mentioned": true,
+      "chronicle_conflicts": ["db-017"]
+    }
+  }
+}

package/evals/cases/no_evidence_novel_design.json

@@ -0,0 +1,16 @@
+{
+  "id": "no_evidence_novel_design",
+  "description": "Novel design with no Chronicle evidence either way — should investigate-more",
+  "outcome": "Implement real-time collaboration features using WebSockets",
+  "design": "Use Socket.io for bi-directional communication, Redis pub/sub for multi-instance message fanout, and optimistic UI updates with conflict resolution via last-write-wins",
+  "oracle_evidence": [],
+  "expected": {
+    "jury_max_confidence": 0.65,
+    "council_recommendation": "investigate-more",
+    "risk_level": "medium",
+    "preflight_expects": {
+      "touches_sensitive_area": false,
+      "chronicle_conflicts": []
+    }
+  }
+}

package/evals/cases/payment_no_idempotency.json

@@ -0,0 +1,33 @@
+{
+  "id": "payment_no_idempotency",
+  "description": "Payment charge without idempotency key — should block",
+  "outcome": "Implement one-click repurchase for customers",
+  "design": "On button click, POST /api/charge with the stored card token and amount. Retry on network failure up to 3 times.",
+  "oracle_evidence": [
+    {
+      "id": "pay-004",
+      "key_insight": "Payment charges without idempotency keys caused duplicate charges during network retries",
+      "decision": "All payment charge requests must include a Stripe idempotency key — retries without idempotency keys caused duplicate charges in production",
+      "schema_version": 2,
+      "affected_areas": ["src/payments/", "src/api/checkout.ts"],
+      "scope": ["payments", "stripe"],
+      "status": "refuted",
+      "confidence": 0.97,
+      "source_module": "council",
+      "evidence_cited": [],
+      "timestamp": "2025-04-01T16:00:00Z"
+    }
+  ],
+  "expected": {
+    "jury_max_confidence": 0.40,
+    "council_recommendation": "redesign",
+    "must_flag": ["idempotency", "duplicate charge"],
+    "must_cite": ["pay-004"],
+    "risk_level": "critical",
+    "preflight_expects": {
+      "touches_sensitive_area": true,
+      "sensitive_areas_include": ["payments"],
+      "chronicle_conflicts": ["pay-004"]
+    }
+  }
+}

package/evals/cases/redis_session_rejected.json

@@ -0,0 +1,32 @@
+{
+  "id": "redis_session_rejected",
+  "description": "Proposing Redis sessions when they were already removed — should block",
+  "outcome": "Implement user session management",
+  "design": "Store session data in Redis with a 30-minute TTL and auto-extend on activity. Use express-session with connect-redis.",
+  "oracle_evidence": [
+    {
+      "id": "auth-015",
+      "key_insight": "Redis sessions removed due to memory overhead at scale and operational complexity",
+      "decision": "Redis sessions removed — memory overhead at scale was unsustainable and operational complexity (Redis cluster, failover) added too much risk",
+      "schema_version": 2,
+      "affected_areas": ["src/auth/", "src/middleware/session.ts"],
+      "scope": ["auth", "sessions", "infrastructure"],
+      "status": "refuted",
+      "confidence": 0.89,
+      "source_module": "council",
+      "evidence_cited": [],
+      "timestamp": "2025-02-10T15:00:00Z"
+    }
+  ],
+  "expected": {
+    "jury_max_confidence": 0.50,
+    "council_recommendation": "redesign",
+    "must_flag": ["memory overhead", "Redis"],
+    "must_cite": ["auth-015"],
+    "risk_level": "critical",
+    "preflight_expects": {
+      "touches_sensitive_area": true,
+      "chronicle_conflicts": ["auth-015"]
+    }
+  }
+}

package/evals/cases/safe_refactor.json

@@ -0,0 +1,17 @@
+{
+  "id": "safe_refactor",
+  "description": "Low-risk internal refactor with no sensitive areas — should proceed without friction",
+  "outcome": "Rename internal helper functions in the reporting module for consistency",
+  "design": "Rename generateCsvReport to exportReportAsCsv and generatePdfReport to exportReportAsPdf in src/reports/. Update all callers. No behaviour change.",
+  "oracle_evidence": [],
+  "expected": {
+    "jury_min_confidence": 0.70,
+    "council_recommendation": "proceed",
+    "risk_level": "low",
+    "preflight_expects": {
+      "touches_sensitive_area": false,
+      "rollback_mentioned": false,
+      "chronicle_conflicts": []
+    }
+  }
+}