omamori 0.1.4 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bafa258377208d15c678d88752804be6963159d28b83c13958de28bdd8bf8fa7
-  data.tar.gz: 9739ae26075e9e5c0f2f87eafd68f2ed6cce0baee719a4c31c1042e8b1c8abf9
+  metadata.gz: 8f3533e1d6c54d0dc29d8c7eb569e9931cdd55b8214180e7380f3cf7080c0caa
+  data.tar.gz: ed4ca270d8a0d490b1c756e6508509609af5c495094ea3dd81e196baa91e6d66
 SHA512:
-  metadata.gz: 373c41e264bc7548b3b4f37a526ed98fcefdd6a51fc6eaa8571b18d7688e131b4852c7e4ce839c7cdd43e540c3bf83b7466a1a66c29cea7c5d6e03dcd59f5cac
-  data.tar.gz: 102078caed02ab830885759ddf21bc8430b15c6a829a5f4ed7c1fcca99360ac707bb938ab19a8f2f9a5ed8ad2ccd86427acfaecc1da6d55a48c27338ae753454
+  metadata.gz: 9fa320c3f0bf6f02cf9f76888a691b24322101e3c8c6787a3e1397f66445248e8edb8a8f6265ceae6870b4fd4aed966dc70425ae02d83d736b6ac972b327d4ce
+  data.tar.gz: 7a6cad8d83abeb40a5e833aadebf3ac0ee10a5629c64df88707852cdc11b42c55b06131dfe5299b0e3393370785b8237ceee74637a1f11f61382c1f687e55e91

data/CHANGELOG.md

@@ -0,0 +1,25 @@
+# Changelog
+
+## 0.2.0 (2026-10-25)
+
+### Breaking changes
+
+- **There is no default reporter.** Choose one with `omamori init --reporter ...` (or `--reporter` on `omamori scan`). Without one, `omamori scan` stops and shows how to run `init`. An 0.1 config that only sets `api_key` keeps using Gemini, now with `gemini-3.8-flash`.
+- `omamori init` takes the settings as flags (`--reporter`, `--reporter-model`, `--classifier`, `--classifier-model`, `--language`, `--ci`, `--force`) and requires `--reporter`. It no longer overwrites an existing `.omamorirc` unless `--force` is given.
+- Without a classifier, all checks are now run by default (0.1 ran 6 main checks). The checks are now 35: `server_error_information_exposure` is merged into `overly_detailed_errors`, and `dependency_trojan_package` (malicious packages) is no longer judged by AI. Use bundler-audit (run by omamori) for known vulnerable versions, and a scanner with a malicious-package database (such as OSV-Scanner or GitHub dependency review) for malicious packages.
+- Checks about something being absent (CSP and other security headers, production settings, two-factor authentication, password policy, audit logging) now ask whether the change weakens or removes an existing protection, instead of whether the application has it.
+
+### Added
+
+- **Classifier**: a fast decision model (TypeSafe Jev, OpenAI Decisions, or one on Ollama) checks each risk as a yes/no question, and only the flagged checks are sent to the reporter. Several risks are also asked as concrete sub-checks. The default threshold is 0.5.
+- **Reporter choice**: any provider RubyLLM supports, or a Claude Code (`claude -p`) / Codex (`codex exec`) subscription. A fallback reporter (`reporter.fallback`) is asked when the reporter refuses or fails.
+- `omamori doctor`: sends the configured classifier and reporters one small request each and says what is missing.
+- `omamori scan --base REF`: scans the diff between REF and HEAD (what a pull request contains).
+- CI configurations generated by `init --ci` / `ci-setup` pass the API keys your classifier and reporter need.
+- An eval set (`evals/`) with AI-written cases, a holdout set, real security fixes from the GitHub Advisory Database, and ordinary commits for measuring false alarms.
+- README sections on getting started and on setting omamori up with an AI agent.
+
+### Fixed
+
+- A reporter refusing to analyze content (Gemini sometimes refuses security analysis) was shown as "no risks". Refused or failed checks are now kept as flags, so nothing is silently passed.
+- The `ci-setup` command was not recognized.

data/Gemfile

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-source "https://rubygems.org"
+source 'https://rubygems.org'
 
 # Specify your gem's dependencies in omamori.gemspec
 gemspec

data/Gemfile.lock

@@ -1,17 +1,18 @@
 PATH
   remote: .
   specs:
-    omamori (0.1.4)
+    omamori (0.2.0)
       brakeman (~> 7.0)
       bundler-audit (~> 0.9.2)
       colorize (~> 0.8)
       dotenv (~> 2.0)
-      ruby-gemini-api (~> 0.1.1)
+      ruby_llm (~> 2.1)
 
 GEM
   remote: https://rubygems.org/
   specs:
     ast (2.4.3)
+    base64 (0.3.0)
     brakeman (7.0.2)
       racc
     bundler-audit (0.9.2)
@@ -24,14 +25,17 @@ GEM
       faraday-net_http (>= 2.0, < 3.5)
       json
       logger
-    faraday-multipart (1.1.0)
+    faraday-multipart (1.2.0)
       multipart-post (~> 2.0)
     faraday-net_http (3.4.0)
       net-http (>= 0.5.0)
+    faraday-retry (2.4.0)
+      faraday (~> 2.0)
     json (2.12.0)
     language_server-protocol (3.17.0.5)
     lint_roller (1.1.0)
     logger (1.7.0)
+    marcel (2.1.0)
     multipart-post (2.4.1)
     net-http (0.6.0)
       uri
@@ -71,16 +75,24 @@ GEM
     rubocop-ast (1.44.1)
       parser (>= 3.3.7.2)
       prism (~> 1.4)
-    ruby-gemini-api (0.1.1)
-      faraday (~> 2.0)
-      faraday-multipart (~> 1.0)
-      json (~> 2.0)
     ruby-progressbar (1.13.0)
+    ruby_llm (2.1.1)
+      base64
+      faraday (>= 1.10.0, != 2.0.0)
+      faraday-multipart (>= 1)
+      faraday-net_http (>= 1)
+      faraday-retry (>= 1)
+      json (< 4)
+      marcel (>= 1.0, < 3)
+      schematist (~> 1.1)
+      zeitwerk (~> 2)
+    schematist (1.1.0)
     thor (1.3.2)
     unicode-display_width (3.1.4)
       unicode-emoji (~> 4.0, >= 4.0.4)
     unicode-emoji (4.0.4)
     uri (1.0.3)
+    zeitwerk (2.8.3)
 
 PLATFORMS
   ruby

data/README.md

@@ -9,59 +9,111 @@ However, please note that this gem is merely a "protective charm" and does not g
 It might help prevent vulnerabilities that could occur even after human review.
 
 Caution:
-AI analysis can uncover vulnerabilities that static analysis alone might miss, but the accuracy of diagnostics may vary depending on the AI model’s "mood."
-Running the analysis multiple times may help reduce false negatives.
-
+AI analysis can uncover vulnerabilities that static analysis alone might miss, but LLM results tend to vary from run to run.
 
 ## Features
 
-- Scan staged changes (`git diff --staged`), the entire codebase, or specified files and directories for security risks.
+- Scan staged changes (`git diff --staged`), a branch diff (`--base`), the entire codebase, or specified files and directories.
+- Fast checklist: each of 35 vulnerability types is checked as a yes/no question by a decision model (TypeSafe Jev, OpenAI Decisions, ...), and an LLM writes a detailed report only for the flagged checks.
+- Choose the classifier and the reporter independently. The reporter can be any API RubyLLM supports (Gemini, Claude, OpenAI, ...) or a Claude Code (`claude -p`) / Codex (`codex exec`) subscription.
 - Integrates with static analysis tools like Brakeman and Bundler-Audit.
-- Utilizes the Gemini API for advanced code analysis to detect vulnerabilities.
 - Supports multiple report formats (console, HTML, JSON).
-- Configurable via a `.omamorirc` file.
-- Exclusion of files and directories via a `.omamoriignore` file (except in `diff` mode).
+- `omamori init` runs with flags only and `omamori doctor` checks the setup, so an AI agent can set omamori up for you.
+- Generates GitHub Actions / GitLab CI configuration.
+- An eval set (`evals/`) for measuring detection accuracy.
+- Configuration via a `.omamorirc` file, and exclusions via a `.omamoriignore` file.
 
-## Installation
+## How It Works
 
-Add this line to your application's Gemfile:
+Omamori's AI analysis has two roles.
 
-```ruby
-gem 'omamori'
-```
+- **Classifier:** A fast decision model that checks each of the 35 risks as a yes/no question, such as TypeSafe Jev or OpenAI Decisions. Optional.
+- **Reporter:** The LLM that writes vulnerability reports. Without a classifier, the reporter checks every risk itself in one prompt.
 
-And then execute:
+With a classifier, a scan works like this:
 
-```bash
-bundle install
-```
+1. The diff is split per file. For each file, all checks (35 by default) are sent to the classifier in a single request. Files are processed in parallel.
+2. Checks whose probability is below the threshold pass immediately.
+3. Only the remaining checks are sent to the reporter, which writes a report following that vulnerability's detection steps.
 
-Or install it yourself as:
+The questions sent to the classifier contain the same vulnerability definitions and detection steps that the reporter receives. For XSS, SQL injection, mass assignment, CSRF, open redirect, missing audit logging, insufficient encryption, and insecure deserialization, the classifier also answers sub-checks: concrete patterns that alone make the risk real (for XSS, for example, "is a URL from users put into `href` without checking its scheme?"). The highest probability among the overall question and its sub-checks is used, and the matched sub-checks are passed on to the reporter.
 
-```bash
-gem install omamori
-```
+Risks about something being absent (CSP and other security headers, production settings, two-factor authentication, password policy, audit logging) are judged as "does this change weaken or remove an existing protection?". Whether the application had the protection in the first place cannot be told from a diff that does not touch it. If the classifier fails for a file, every check for that file is escalated to the reporter. If the reporter cannot write a report, the classifier's flag is kept in the results, so nothing is silently skipped.
 
-## Usage
+## Getting Started
 
-### Initialization
+1. Add omamori to your Gemfile and install it.
 
-To generate an initial configuration file (`.omamorirc`), run:
+    ```ruby
+    gem 'omamori', group: :development, require: false
+    ```
 
-```bash
-omamori init
-```
+    ```bash
+    bundle install
+    ```
+
+2. Choose a reporter and write the config file. The provider and model are given as flags (there are no interactive questions).
 
-Edit the generated `.omamorirc` and `.omamoriignore` files to configure your Gemini API key, preferred model, checks to perform, and files/directories to exclude from scanning, and other settings.
+    ```bash
+    bundle exec omamori init --reporter claude_code                                   # Claude Code subscription (claude -p)
+    bundle exec omamori init --reporter codex                                         # Codex subscription (codex exec)
+    bundle exec omamori init --reporter anthropic --reporter-model claude-sonnet-5-5  # Any API RubyLLM supports
+    ```
+
+    Add `--classifier typesafe` to use a classifier, `--language ja` for reports in Japanese, and `--ci github_actions` (or `gitlab_ci`) to also generate a CI configuration. This writes `.omamorirc` and `.omamoriignore`, then prints the next steps (such as the environment variables to set). An existing `.omamorirc` is not overwritten unless you pass `--force`.
+
+3. Set the environment variables it printed, and check the setup.
+
+    ```bash
+    bundle exec omamori doctor
+    ```
+
+    This sends the classifier and the reporter one small request each, and says what is missing (an API key, or installing or logging in to a CLI) and how to fix it. It exits with 0 when everything works and 1 otherwise.
+
+4. Scan.
+
+    ```bash
+    bundle exec omamori scan --base main
+    ```
+
+There is no default reporter. If neither `.omamorirc` nor `--reporter` sets one, `omamori scan` stops and shows how to run `init`.
+
+### API Keys
+
+API keys are passed through environment variables, named after RubyLLM's settings in upper case (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, `OPENROUTER_API_KEY`, `TYPESAFE_API_KEY` for the Jev classifier, ...). `omamori init` prints the ones your setup uses. The `claude_code` / `codex` reporters use the installed CLI's login, so no API key is needed.
+
+Code diffs are sent to the classifier and reporter providers you choose. For code that must not leave your environment, choose a model that runs locally, such as one on Ollama.
+
+## Setting Up with an AI Agent
+
+When you ask an AI coding agent (Claude Code, Codex, ...) to set up omamori, it can follow these steps:
+
+1. Add `gem 'omamori', group: :development, require: false` to the Gemfile and run `bundle install`
+2. Choose the reporter: `claude_code` if the `claude` command exists, `codex` if `codex` exists. Otherwise, ask the developer which API provider and model they use
+3. Ask the developer whether to use a classifier (`--classifier typesafe`, ...). It makes scans faster and cheaper and raises fewer false alarms, but needs that provider's API key and sends the code to that provider as well
+4. Run `bundle exec omamori init --reporter ... [--reporter-model ...] [--classifier ...] [--language ja] [--ci github_actions]`
+5. Ask the developer to set the environment variables it printed. Do not ask them to paste API keys into the chat, and do not write keys into files
+6. Run `bundle exec omamori doctor`, fixing what it reports, until it exits with 0
+7. Run `bundle exec omamori scan --base main` and report the results. If a CI configuration was generated, also tell the developer the secret names to register (printed by `init`)
+
+## Usage
 
 ### Scanning
 
+Any setting in `.omamorirc` can be changed for one run with flags (see Choosing the Classifier and Reporter).
+
 Scan staged changes (default):
 
 ```bash
 omamori scan
 ```
 
+Scan the changes on the current branch (`main...HEAD`, i.e. what a pull request contains; useful in CI):
+
+```bash
+omamori scan --base main
+```
+
 Scan the entire codebase:
 
 ```bash
@@ -69,60 +121,101 @@ omamori scan --all
 ```
 This mode respects the `.omamoriignore` file.
 
-Specify output format (console, html, json):
+Scan specific files or directories:
 
 ```bash
-bundle exec omamori scan --format html
-bundle exec omamori scan --all --format json
+omamori scan app/controllers/users_controller.rb app/models/user.rb config/routes.rb lib/
 ```
+This mode also respects the `.omamoriignore` file.
 
-### Scan Specific Files/Directories
+Specify the output format (console, html, json):
 
-You can specify particular files or directories to scan:
+```bash
+omamori scan --format html
+omamori scan --all --format json
+```
+
+Run only AI analysis, skipping static analysis tools:
 
 ```bash
-omamori scan <file_path1> <file_path2> ... <directory_path1> ...
+omamori scan --ai
 ```
 
-Example:
+### Choosing the Classifier and Reporter
+
+Both roles are set with a `provider` and a `model`, on the command line or in `.omamorirc` (see [Configuration](#configuration)).
 
 ```bash
-omamori scan app/controllers/users_controller.rb app/models/user.rb config/routes.rb lib/
+omamori scan --base main --classifier typesafe --reporter gemini --reporter-model gemini-3.8-flash   # Jev decides, Gemini reports
+omamori scan --base main --classifier typesafe --reporter claude_code    # Jev decides, claude -p reports
+omamori scan --base main --classifier openai --reporter anthropic --reporter-model claude-sonnet-5-5
+omamori scan --reporter codex                                            # No classifier; codex exec checks everything
 ```
 
-This mode respects the `.omamoriignore` file.
+**Do you need a classifier?** It is optional. In our measurements with `claude -p` (sonnet) as the reporter, on real security fixes from Rails applications and on ordinary commits from open-source Rails apps, the reporter alone caught at least as many vulnerabilities as the classifier pipeline. The classifier made scans faster and cheaper (the reporter is called only for flagged checks) and raised fewer false alarms on changes that look risky but are safe. Cheaper reporters have not been measured on their own.
+
+**Classifier providers**
+
+| provider | Decision model | API key |
+| --- | --- | --- |
+| `typesafe` | TypeSafe Jev (`jev-1.13.0` when `model` is omitted) | `TYPESAFE_API_KEY` |
+| `openai` | OpenAI Decisions API (`gpt-6-luna` when `model` is omitted) | `OPENAI_API_KEY` |
+| `ollama` | A decision model running on Ollama (such as Clef). `model` and the server's `base_url` (or `OLLAMA_API_BASE`) are required | None |
+| `none` | No classifier (default) | None |
+
+With `base_url`, you can point a classifier at any other server that speaks the SystemOne protocol.
+
+**Reporter providers**
 
-### AI Analysis Only
+| provider | What it calls |
+| --- | --- |
+| A RubyLLM provider (`gemini`, `anthropic`, `openai`, `openrouter`, `ollama`, ...) | An API through [RubyLLM](https://rubyllm.com). `model` is required |
+| `claude_code` | `claude -p` (subscription) with all tools disabled, in an empty temporary directory |
+| `codex` | `codex exec` (subscription) in a read-only sandbox, in an empty temporary directory |
 
-To perform only AI analysis without running static analysis tools, use the `--ai` option:
+In `.omamorirc`, the reporter's `provider` can be omitted, in which case it is inferred from the model name. Set `provider` when the model is too new to be in RubyLLM's model registry.
+
+A reporter may refuse to analyze some content (Gemini sometimes refuses security analysis). A refused check is not silently passed; the classifier's flag is kept. With a fallback reporter (`reporter.fallback`), refused or failed checks are asked again there, and only those.
 
 ```bash
-omamori scan --ai
+omamori scan --base main --classifier typesafe --reporter-fallback claude_code --reporter-fallback-model sonnet
 ```
 
+Diffs contain code from outside sources, so `claude_code` / `codex` run without the ability to execute commands. Because subscriptions have tighter rate limits, reports are written 3 at a time when a CLI is used (change with `reporter.concurrency`).
+
 ## Configuration
 
 The `.omamorirc` file in the project root directory allows you to customize Omamori's behavior.
 
-Here's a detailed breakdown of the configuration options:
-
 ```yaml
 # .omamorirc
 # Configuration file for omamori gem
 
-# Gemini API Key (required for AI analysis)
-# You can also set this via the GEMINI_API_KEY environment variable
-api_key: YOUR_GEMINI_API_KEY # Replace with your actual API key
-
-# Gemini Model to use (optional, default: gemini-2.5-flash-preview-04-17)
-model: gemini-2.5-flash-preview-04-17
-
-# Security checks to enable (optional, default: all implemented checks)
+# Classifier (optional). Without it, the reporter checks every risk. Can be overridden with --classifier
+# classifier:
+#   provider: typesafe      # typesafe / openai / ollama
+#   model: jev-1.13.0       # Pin a version (not jev-latest) if you tune the threshold
+#   threshold: 0.5          # Checks at or above this probability go to the reporter
+#   concurrency: 8          # Parallel requests to the classifier
+#   base_url: https://api.typesafe.ai   # Optional. Another SystemOne server
+#   api_key: ...            # Optional. Environment variables (TYPESAFE_API_KEY, ...) are recommended
+
+# Reporter (required; written by omamori init). Can be overridden with --reporter
+reporter:
+  provider: anthropic       # A RubyLLM provider, or claude_code / codex
+  model: claude-sonnet-5-5  # Required for API providers. For CLIs, omit it to use the CLI's default model
+  # concurrency: 8           # Reports written in parallel (default 3 for CLIs, 8 for APIs)
+  # command: /path/to/claude # Optional. Path to the claude_code / codex CLI
+  # api_key: ...             # Optional. Environment variables (ANTHROPIC_API_KEY, ...) are recommended
+  # fallback:                # Optional. Asked only when the reporter refuses or fails
+  #   provider: claude_code
+  #   model: sonnet
+
+# Security checks to run (optional; default: all 35)
 # checks:
-#   xss: true
-#   csrf: true
-#   idor: true
-#   ... # Add other checks as they become available
+#   - xss
+#   - csrf
+#   - idor
 
 # Custom prompt templates (optional)
 # prompt_templates:
@@ -145,9 +238,9 @@ model: gemini-2.5-flash-preview-04-17
 # language: ja
 ```
 
-*   `api_key`: Your API key for accessing the Gemini API. Can also be set via the `GEMINI_API_KEY` environment variable.
-*   `model`: The Gemini model to use for AI analysis. Defaults to `gemini-2.5-flash-preview-04-17`.
-*   `checks`: Configure which types of security checks to enable. By default, all implemented checks are enabled. You can selectively enable/disable checks here (e.g., `xss: true`, `csrf: false`).
+*   `classifier`: `provider`, `model`, `threshold` (default 0.5), `concurrency` (default 8), `base_url`, `api_key`.
+*   `reporter`: `provider`, `model`, `concurrency`, `command`, `api_key`, `fallback` (a fallback reporter with its own `provider`, `model`, ...).
+*   `checks`: The list of security checks to run. The former `server_error_information_exposure` is merged into `overly_detailed_errors`. `dependency_trojan_package` (malicious packages) is no longer judged by AI, because it cannot be judged reliably from code: use bundler-audit (run by omamori) for known vulnerable versions, and a scanner with a malicious-package database, such as OSV-Scanner or GitHub dependency review.
 *   `prompt_templates`: Define custom prompt templates for AI analysis.
 *   `report`: Configure report output settings.
     *   `output_path`: The output directory or prefix for HTML and JSON reports.
@@ -157,16 +250,33 @@ model: gemini-2.5-flash-preview-04-17
     *   `bundler_audit`: Additional command-line options for Bundler-Audit.
 *   `language`: Language setting for the details provided in AI analysis reports. Defaults to English (`en`).
 
+The older keys (`engine`, `llm`, `model`, `provider`, `api_key`, `decision`, `claude`, `codex`) and options (`--engine`, `--llm`) still work. A 0.1 config that only sets `api_key` keeps using Gemini (`gemini-3.8-flash`).
+
 ## .omamoriignore File
 
-The `.omamoriignore` file allows you to exclude specific files and directories from the scan target. Its behavior is similar to a `.gitignore` file, but with the following considerations:
+The `.omamoriignore` file allows you to exclude specific files and directories from the scan target. However, if the `--force-scan-ignored` option is used, the settings in this file will be ignored. Its behavior is similar to a `.gitignore` file, but with the following considerations:
 
 *   **Effective Modes:** It is only effective in `--all` mode or when scanning specified files/directories.
-*   **Ineffective Mode:** In `diff` mode (i.e., `omamori scan` without arguments), the `.omamoriignore` file is ignored. This is because `diff` mode only targets changes retrieved by `git diff`.
-*   **Format:** Each line specifies one pattern. Lines starting with `#` are treated as comments. It is recommended to append a `/` to directory patterns (e.g., `vendor/`). Patterns are evaluated using simple prefix matching (e.g., `config/initializers` matches `config/initializers/devise.rb`). Wildcards (`*`) are not currently supported.
+*   **Ineffective Mode:** In `diff` mode (i.e., `omamori scan` without arguments, or with `--base`), the `.omamoriignore` file is ignored. This is because `diff` mode only targets changes retrieved by `git diff`.
+*   **Format:** Each line specifies one pattern. Lines starting with `#` are treated as comments.
+    *   A pattern ending with `/` excludes everything under that directory (e.g., `vendor/bundle/`).
+    *   Other patterns are evaluated as globs against the path relative to the project root or the file name (e.g., `*.log`, `spec/fixtures/*.rb`).
 
 When you run the `omamori init` command, a `.omamoriignore` file pre-filled with patterns for common files and directories to ignore in a Rails project will be generated. You can edit this file as needed.
 
+## Using in CI
+
+Pass `--ci` to `omamori init`, or generate the configuration later from `.omamorirc`:
+
+```bash
+omamori ci-setup --ci github_actions   # .github/workflows/omamori_scan.yml
+omamori ci-setup --ci gitlab_ci        # .gitlab-ci.yml
+```
+
+The generated configuration scans the branch diff with `--base` on pull requests (merge requests) and the entire codebase with `--all` otherwise. It fetches the full history so the diff can be taken. It passes the environment variables your classifier and reporter need, and prints the names of the secrets (CI/CD variables on GitLab) to register.
+
+`claude_code` / `codex` use the CLI's login on the machine. The generated configuration does not install or log in a CLI on the CI runner: in CI, add `--reporter PROVIDER --reporter-model MODEL` to the scan command to use an API reporter, or set up the runner yourself.
+
 ## Demo Files
 
 The `demo` directory contains example files with known vulnerabilities that can be used to demonstrate Omamori's capabilities.
@@ -181,15 +291,30 @@ To run Omamori on the demo files, you need to stage the changes in the `demo` di
     ```bash
     git add demo_
     ```
-3.  Now you can run Omamori on the staged demo files:
+3.  Now you can run Omamori on the staged demo files (set it up with `omamori init --reporter ...` first):
     ```bash
     omamori scan
     ```
+    To try a classifier (Jev):
+    ```bash
+    omamori scan --classifier typesafe
+    ```
     Or to scan all files in the demo directory (after staging them):
     ```bash
     omamori scan --all
     ```
 
+## Measuring Detection Accuracy
+
+`evals/` contains code changes with known answers. For each of the 35 risks there are two vulnerable changes and two "near miss" changes that look similar but are safe, plus a set of benign changes. XSS, SQL injection, mass assignment, IDOR, CSRF, and open redirect also have cases for specific patterns that are easy to miss. Use it to see how misses and false positives change when you switch the classifier or reporter.
+
+```bash
+bundle exec ruby evals/run.rb --classifier typesafe --runs 3
+bundle exec ruby evals/run.rb --classifier none --reporter claude_code --reporter-model sonnet
+```
+
+See [evals/README.md](evals/README.md) for details.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub. This project is intended to be a safe, welcoming space for collaboration.