diogenes 0.1.0 → 0.1.1

data/docs/contributing.md

@@ -0,0 +1,228 @@
+# Contributing to Diogenes
+
+Diogenes is opinionated by design. Before contributing, read the philosophy section of the README and the architecture notes in CLAUDE.md. A PR that conflicts with the core philosophy — even if the code is excellent — will not be merged.
+
+---
+
+## What We're Looking For
+
+**Good contributions:**
+- New gates that encode a real, defensible constraint
+- Improvements to grounding verification accuracy or the verifier prompt
+- New eval matchers that cover failure modes existing matchers don't
+- Drift detection improvements — staleness scoring, webhook integrations
+- Better failure messages — they should be actionable, not cryptic
+- Dashboard improvements that surface existing data more clearly
+- Bug fixes with accompanying regression tests
+- Documentation that makes the framework clearer
+
+**Not a good fit:**
+- LLM client integrations (Diogenes is provider-agnostic by design)
+- RAG pipeline or embedding implementations
+- Gates that can be bypassed with configuration
+- Anything that makes a gate failure a warning instead of an error
+- Database-backed eval golden pairs
+- Authentication for the dashboard
+
+If you're unsure whether your idea fits, open a discussion issue before writing code.
+
+---
+
+## Getting Started
+
+```bash
+git clone https://github.com/your-org/diogenes
+cd diogenes
+bundle install
+bundle exec rspec
+```
+
+The test suite should pass with no configuration. If it doesn't, open an issue.
+
+To run the full suite including Rails integration tests:
+
+```bash
+bundle exec rspec
+bundle exec rspec spec/rails
+```
+
+---
+
+## Contributing to Gates
+
+Gates are the core primitive. Adding one is a significant decision — each gate adds cognitive load to every team that adopts the gem. New gates should address a failure mode that existing gates don't cover.
+
+### Steps
+
+1. Create `lib/diogenes/gates/your_gate.rb` inheriting from `Diogenes::Gates::Base`
+2. Implement `#valid?` and `#failure_message`
+3. Add any cross-gate incompatibilities to `lib/diogenes/gates/compatibility.rb`
+4. Register in `Diogenes::Feature`
+5. Write unit specs in `spec/diogenes/gates/your_gate_spec.rb`
+6. Document in `docs/framework.md` following the existing format
+7. Add to `docs/examples.md` if the gate meaningfully changes a verdict
+
+### Gate Design Rules
+
+- Gates fail loudly or pass. No warnings, no degraded modes.
+- `#failure_message` must include the feature class name, the gate name, and a plain-English instruction for what to change.
+- Gates are validated at boot. If your gate requires runtime information to validate, reconsider the design.
+- Default configuration should be the most conservative option.
+
+### Versioning
+
+Adding a new gate incompatibility to the compatibility matrix is a **minor** bump if it catches configurations that were silently wrong. It is a **major** bump if it invalidates configurations that were intentionally valid.
+
+---
+
+## Contributing to Grounding Verification
+
+The grounding verifier has two separate concerns: the prompt and the logic. Keep them separate.
+
+### The Prompt (`lib/diogenes/grounding/prompt.rb`)
+
+The prompt is versioned independently. When you change it:
+
+- Increment `Diogenes::Grounding::Prompt::VERSION`
+- Audit records store the prompt version — old records remain interpretable
+- Add a spec that the new prompt correctly classifies a fixture set of (context, response) pairs as supported/unsupported/contradicted
+
+Don't change the prompt to improve performance on one case without running the full fixture set. A prompt that is better on average but worse on contradictions is not an improvement.
+
+### The Verifier Logic (`lib/diogenes/grounding/verifier.rb`)
+
+The verifier accepts any LLM callable. Tests use a stub:
+
+```ruby
+Diogenes::Grounding::Verifier.stub(result: :pass)
+Diogenes::Grounding::Verifier.stub(result: :flag, unsupported: ["claim text"])
+```
+
+Never make live LLM calls in specs.
+
+### The Result (`lib/diogenes/grounding/result.rb`)
+
+`Diogenes::Grounding::Result` is a value object. Adding fields is a minor bump. Removing or renaming fields is a major bump — audit records store serialized results.
+
+---
+
+## Contributing to Drift Detection
+
+### Staleness Scoring (`lib/diogenes/drift/staleness_score.rb`)
+
+The staleness score is a pure function of timestamps and diff size. Keep it that way — no database access, no I/O. Specs use fixed timestamps.
+
+```ruby
+score = Diogenes::Drift::StalenessScore.calculate(
+  source_updated_at: 47.days.ago,
+  indexed_at: 89.days.ago,
+  diff_size: :major
+)
+```
+
+### Re-index Jobs
+
+Diogenes ships a base job class. The host app subclasses it with embedding logic. Never add embedding implementation to the base class.
+
+```ruby
+# What we ship
+class Diogenes::Drift::ReindexJob < ApplicationJob
+  def perform(document_id)
+    raise NotImplementedError, "Subclass with your embedding logic"
+  end
+end
+
+# What the host app writes
+class ReindexDocumentJob < Diogenes::Drift::ReindexJob
+  def perform(document_id)
+    # fetch, embed, store
+  end
+end
+```
+
+### Webhooks
+
+Drift alert webhooks send a standard payload. If you're adding webhook support for a new event type, use the existing payload shape and add a `type` field. Don't introduce a new shape.
+
+---
+
+## Contributing to the Eval Runner
+
+### Adding Matchers (`lib/diogenes/evals/matchers.rb`)
+
+Matchers are composable predicates. A new matcher should:
+
+- Accept a response struct and return true/false
+- Have a clear, descriptive failure message when it returns false
+- Be usable inside `all_of` and `one_of`
+- Be tested against fixture responses, never live LLM calls
+
+```ruby
+module Diogenes
+  module Evals
+    module Matchers
+      def your_matcher(arg)
+        ->(result) {
+          # evaluate result, return true/false
+        }
+      end
+    end
+  end
+end
+```
+
+### Golden Pairs Live in Code
+
+Don't add database storage for golden pairs. They are version-controlled files. If a contributor wants to propose database-backed pairs, the answer is no — visibility in code review is a feature, not a limitation.
+
+### Regression Detection
+
+The regression detector (`lib/diogenes/evals/regression.rb`) stores the last passing response and diffs it against the first failing one. If you're improving diff quality, use the fixture set in `spec/fixtures/evals/regressions/`. Don't introduce a dependency on a diff library — the current implementation is intentionally minimal.
+
+---
+
+## Contributing to the Dashboard
+
+The dashboard engine has no authentication and no CSS framework dependency. Keep it that way.
+
+### Adding a New View
+
+Views use ERB. No ViewComponent, no Stimulus, no Turbo (unless it's already a dependency by the time you're reading this — check the gemspec). Interactivity should be achievable with standard Rails UJS or vanilla JS.
+
+### Adding a New Route
+
+Add to `lib/diogenes/engine/config/routes.rb`. Follow the existing namespace pattern. Every new route needs a corresponding controller action and view, even if the view is a stub.
+
+### What the Dashboard Must Never Do
+
+- Make LLM calls
+- Trigger re-indexing without an explicit user action
+- Display raw query or response content (hashes only — PII protection)
+- Require specific authentication middleware
+
+---
+
+## Pull Request Guidelines
+
+- One logical change per PR
+- Tests for all new behaviour
+- Updated documentation if you're changing or adding functionality
+- A clear description of *why*, not just *what*
+
+PRs that add code without updating relevant docs will be sent back.
+
+---
+
+## Versioning
+
+Diogenes follows semantic versioning strictly.
+
+- **Patch:** bug fixes, documentation, internal refactors
+- **Minor:** new gates, new matchers, new dashboard views, non-breaking additions
+- **Major:** changes to gate semantics, changes to audit record shape, removal of features, breaking DSL changes
+
+---
+
+## Code of Conduct
+
+Be direct. Be specific. Assume good intent. If you're reviewing a PR, explain your reasoning — "this conflicts with the philosophy" is not sufficient without explaining which part and why.

data/docs/dashboard.md

@@ -0,0 +1,365 @@
+# The Diogenes Dashboard
+
+The Diogenes dashboard is a Rails engine that mounts into your existing application and gives you a live view of AI feature health across three dimensions: grounding verification, document drift, and eval regressions.
+
+Think of it as Sidekiq Web for AI accountability — one place to see whether your AI features are doing what you think they're doing.
+
+---
+
+## Mounting
+
+```ruby
+# config/routes.rb
+mount Diogenes::Engine => '/diogenes'
+```
+
+Diogenes has no opinions about authentication. Protect the route the same way you protect any sensitive admin surface:
+
+```ruby
+# With Devise + admin constraint
+authenticate :user, ->(u) { u.admin? } do
+  mount Diogenes::Engine => '/diogenes'
+end
+
+# With HTTP basic auth via a constraint
+constraints Diogenes::AdminConstraint.new(ENV['DIOGENES_PASSWORD']) do
+  mount Diogenes::Engine => '/diogenes'
+end
+
+# With a custom Rack middleware
+mount Diogenes::Engine => '/diogenes', constraints: YourAuthConstraint
+```
+
+---
+
+## Engine Routes
+
+```ruby
+# lib/diogenes/engine/routes.rb
+
+Diogenes::Engine.routes.draw do
+  root to: 'dashboard#overview'
+
+  namespace :dashboard do
+    # Grounding
+    get  'grounding',                    to: 'grounding#index'
+    get  'grounding/:feature',           to: 'grounding#show'
+    get  'grounding/:feature/:id',       to: 'grounding#verification'
+
+    # Drift
+    get  'drift',                        to: 'drift#index'
+    get  'drift/:index_name',            to: 'drift#show'
+    post 'drift/:index_name/reindex',    to: 'drift#reindex'
+    post 'drift/:document_id/reindex',   to: 'drift#reindex_document'
+
+    # Evals
+    get  'evals',                        to: 'evals#index'
+    get  'evals/:feature',               to: 'evals#show'
+    post 'evals/:feature/run',           to: 'evals#run'
+    get  'evals/:feature/:run_id',       to: 'evals#run_result'
+  end
+end
+```
+
+---
+
+## Controller Structure
+
+```
+lib/diogenes/engine/
+  app/
+    controllers/
+      diogenes/
+        dashboard_controller.rb     # Overview tab
+        dashboard/
+          grounding_controller.rb
+          drift_controller.rb
+          evals_controller.rb
+    views/
+      diogenes/
+        dashboard/
+          overview.html.erb
+          grounding/
+            index.html.erb
+            show.html.erb
+            verification.html.erb
+          drift/
+            index.html.erb
+            show.html.erb
+          evals/
+            index.html.erb
+            show.html.erb
+            run_result.html.erb
+```
+
+---
+
+## The Overview Tab
+
+`GET /diogenes`
+
+The entry point. One row per gated feature, showing aggregate health across all three dimensions. Red features surface immediately.
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│ DIOGENES                                              Last updated: 2 min ago│
+├──────────────────────┬────────┬─────────────┬──────────────┬────────────────┤
+│ Feature              │ Gates  │ Grounding   │ Drift        │ Evals          │
+├──────────────────────┼────────┼─────────────┼──────────────┼────────────────┤
+│ SupportAssistant     │ 5/5 ✓  │ 94% clean   │ 2 stale docs │ 47/50 passing  │
+│ CodebaseOracle       │ 5/5 ✓  │ 98% clean   │ All current  │ 30/30 passing  │
+│ ComplianceSearcher   │ 5/5 ✓  │ 87% clean ⚠ │ 11 stale docs│ 18/25 passing ✗│
+└──────────────────────┴────────┴─────────────┴──────────────┴────────────────┘
+```
+
+`ComplianceSearcher` has a low grounding rate, stale documents, and a failing eval suite. That's the feature that needs attention today — and you can see it before a user does.
+
+---
+
+## The Grounding Tab
+
+`GET /diogenes/dashboard/grounding`
+
+Per-feature grounding verification history. Shows flag rates over time and a live feed of recent verifications.
+
+```
+SupportAssistant — Grounding History (last 30 days)
+
+Flag rate: 6.2%  ▼ from 8.1% last month
+
+Recent verifications:
+  ✓ Supported    "You can cancel your subscription from the billing page..."
+  ✓ Supported    "The enterprise tier includes unlimited seats..."
+  ⚠ Unsupported  "Refunds are processed within 24 hours..."  → flagged for review
+  ✗ Contradicted "The API rate limit is 100 req/min..."      → blocked, not served
+```
+
+### Verification Detail
+
+`GET /diogenes/dashboard/grounding/:feature/:id`
+
+Drill into any verification to see the full picture:
+
+```
+Verification #8472 — SupportAssistant
+Query:    "How long does a refund take?"
+Status:   ⚠ FLAGGED — unsupported claim detected
+
+Retrieved context:
+  [chunk 1] "Refund requests are reviewed within 2 business days..."  ← source: refund-policy.md
+  [chunk 2] "Enterprise customers receive priority support..."         ← source: enterprise-terms.md
+
+AI response:
+  "Refunds are typically processed within 24 hours."
+
+Verdict:
+  supported:     []
+  unsupported:   ["processed within 24 hours"]   ← no chunk supports this timeframe
+  contradicted:  []
+
+Action taken: flagged_for_review
+Reviewed by:  sarah@company.com (approved with edit)
+Final sent:   "Refunds are reviewed within 2 business days..."
+```
+
+This is the audit trail that makes human-in-the-loop real. Every verification, every flag, every human decision — all linked.
+
+---
+
+## The Drift Tab
+
+`GET /diogenes/dashboard/drift`
+
+A staleness leaderboard — every indexed document, ranked by how long since its embedding was updated versus when the source last changed.
+
+```
+Document Drift — All Indexes
+
+  ● 11 documents require attention
+
+  CRITICAL (source changed > 30 days ago, not re-indexed)
+  ┌─────────────────────────────────┬──────────────┬──────────────┬──────────┐
+  │ Document                        │ Source updated│ Last indexed │ Staleness│
+  ├─────────────────────────────────┼──────────────┼──────────────┼──────────┤
+  │ refund-policy.md                │ 47 days ago  │ 89 days ago  │ ████████ │
+  │ enterprise-pricing.md           │ 33 days ago  │ 102 days ago │ ████████ │
+  └─────────────────────────────────┴──────────────┴──────────────┴──────────┘
+
+  WARNING (source changed 7–30 days ago)
+  ┌─────────────────────────────────┬──────────────┬──────────────┬──────────┐
+  │ api-rate-limits.md              │ 12 days ago  │ 45 days ago  │ █████░░░ │
+  │ ...                             │              │              │          │
+  └─────────────────────────────────┴──────────────┴──────────────┴──────────┘
+
+  [Re-index all critical]   [Re-index all warnings]
+```
+
+### Re-indexing
+
+Clicking re-index queues an `ActiveJob`. The job is defined in the host app — Diogenes provides the interface and the tracking, not the embedding logic:
+
+```ruby
+# config/initializers/diogenes.rb
+Diogenes.configure do |config|
+  config.drift.reindex_job = ReindexDocumentJob
+  config.drift.staleness_thresholds = {
+    warning:  7.days,
+    critical: 30.days
+  }
+  config.drift.alert_webhook = ENV['DIOGENES_ALERT_WEBHOOK']
+end
+```
+
+### How Drift Is Detected
+
+Diogenes tracks two timestamps per indexed document:
+
+- `source_updated_at` — populated by the host app when indexing, updated by a webhook or scheduled check
+- `indexed_at` — set by Diogenes when an embedding is stored
+
+The staleness score is a function of the gap between them, weighted by how much the source changed (line diff if available, timestamp delta otherwise).
+
+```ruby
+# Inform Diogenes that a source document has changed
+Diogenes::Drift.source_updated(
+  document_id: 'refund-policy-v2',
+  updated_at: Time.current,
+  diff_size: :major   # :minor, :moderate, :major — affects staleness weight
+)
+```
+
+---
+
+## The Evals Tab
+
+`GET /diogenes/dashboard/evals`
+
+The hardest unsolved problem in production AI is knowing whether your feature is getting better or worse over time. The evals tab is Diogenes's answer: define golden question/answer pairs, run them on a schedule, track pass rates over time, and alert on regression.
+
+```
+Evals — All Features
+
+  SupportAssistant    47/50 passing   94%  ▲ from 91% last week   [Run now]
+  CodebaseOracle      30/30 passing  100%  → unchanged             [Run now]
+  ComplianceSearcher  18/25 passing   72%  ▼ from 88% last week ✗ [Run now]
+```
+
+### Defining Golden Pairs
+
+Golden pairs live in your codebase as Ruby, not in a database. They're version-controlled, reviewed in PRs, and treated as first-class tests:
+
+```ruby
+# test/diogenes/evals/support_assistant_evals.rb
+
+Diogenes::Evals.define(SupportAssistant) do
+  eval "basic refund question" do
+    query    "How do I request a refund?"
+    expects  all_of(
+      grounded_in("refund-policy"),
+      contains("billing page"),
+      does_not_contain("24 hours")   # known wrong answer to guard against
+    )
+  end
+
+  eval "enterprise tier question" do
+    query    "Does the enterprise plan include SSO?"
+    expects  all_of(
+      grounded_in("enterprise-terms"),
+      semantically_similar_to("Yes, SSO is included in all enterprise plans")
+    )
+  end
+
+  eval "question with no good answer" do
+    query    "What is the API rate limit for the legacy v1 endpoints?"
+    expects  one_of(
+      low_confidence_response,
+      routes_to_human_review
+    )
+  end
+end
+```
+
+### Matchers
+
+| Matcher | What it checks |
+|---|---|
+| `grounded_in(source)` | Retrieved context includes chunks from the named source |
+| `contains(text)` | Response includes the specified text or close semantic equivalent |
+| `does_not_contain(text)` | Response does not include the specified text |
+| `semantically_similar_to(text)` | Response embedding is within threshold cosine distance |
+| `low_confidence_response` | Feature returned a flagged or low-confidence result |
+| `routes_to_human_review` | Feature queued the output for human review |
+| `all_of(*matchers)` | All matchers must pass |
+| `one_of(*matchers)` | At least one matcher must pass |
+
+### Running Evals
+
+Evals can be run on demand from the dashboard, via a Rake task, or on a schedule:
+
+```bash
+# On demand
+bundle exec rake diogenes:evals:run[SupportAssistant]
+bundle exec rake diogenes:evals:run   # all features
+```
+
+```ruby
+# Scheduled via your existing job infrastructure
+class RunDiogenesEvalsJob < ApplicationJob
+  def perform
+    Diogenes::Evals.run_all
+  end
+end
+```
+
+### Regression Detection
+
+When a passing eval starts failing, Diogenes records the regression point — exactly which run it broke on — and stores a diff between the last passing response and the first failing one. The evals detail view surfaces this:
+
+```
+ComplianceSearcher — Eval Regression Detected
+
+  "What is our data retention policy?"
+
+  Status: ✗ FAILING since run #142 (3 days ago)
+
+  Last passing response (run #141):
+    "Data is retained for 7 years per our compliance obligations..."
+
+  Current failing response (run #144):
+    "Data retention policies vary by region..."
+
+  Likely cause: 'data-retention-policy.md' updated 4 days ago, not re-indexed
+                → See drift tab
+```
+
+That last line is the integration that makes the dashboard coherent: a failing eval points to a stale document in the drift tab, which explains the low grounding score on the grounding tab. Three separate signals, one root cause.
+
+---
+
+## Configuration Reference
+
+```ruby
+# config/initializers/diogenes.rb
+
+Diogenes.configure do |config|
+  # Grounding
+  config.grounding.default_threshold      = 0.8
+  config.grounding.verifier_llm           = -> (prompt) { YourLLMClient.complete(prompt) }
+
+  # Drift
+  config.drift.reindex_job                = ReindexDocumentJob
+  config.drift.staleness_thresholds       = { warning: 7.days, critical: 30.days }
+  config.drift.alert_webhook              = ENV['DIOGENES_ALERT_WEBHOOK']
+  config.drift.check_interval             = 1.hour
+
+  # Evals
+  config.evals.schedule                   = '0 9 * * *'   # daily at 9am, if using sidekiq-cron
+  config.evals.alert_on_regression        = true
+  config.evals.alert_webhook              = ENV['DIOGENES_ALERT_WEBHOOK']
+  config.evals.passing_threshold          = 0.9           # alert if pass rate drops below 90%
+
+  # Audit log
+  config.audit.retention_days             = 90
+  config.audit.pii_scrubber               = Diogenes::PII::DefaultScrubber
+end
+```