completion-kit 0.29.3 → 0.30.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 12931f9b21433398c80f8676655a8dcaf075638dfd2015fe30e94f4f9b318bb5
4
- data.tar.gz: 15869d3565bef4b44d618f0bb3e04649201cb488505160f0fc7a1170eee348df
3
+ metadata.gz: 7a88e90a186d2b535eb032d4ae9ecbae34605c622cc015f605044e5aaf570598
4
+ data.tar.gz: bcf073d39956a5ccd1ac1ccf2e95a69eba4e08998228b2d3ef264c1708daa523
5
5
  SHA512:
6
- metadata.gz: 2aa2cf27bdde25b06fc7b30294623ffafee6f0f1a7cd2b4a9f4990407743638c07e5f79459f813516c452fbd6d5f2aa12d0c638aa702865db76df69f7435aa93
7
- data.tar.gz: 8ea6c48e88708cc574d8d43c2099ef332aba04eb94ba75fc0c3b50a283d17d0ac3a40e3e5b292c8a3dc4bd9be7c2b18526133dc2b000f3330880b9483026621b
6
+ metadata.gz: 99a8354ff64bb6339c44a5074787292d112eba92c9e9ba133b9e8fb3ab7364535f3a32b8e5e8d012264aaffea3572482b753f7e1c136a8ee303d55ef5232d3b3
7
+ data.tar.gz: 6de209e19ec52332cfa510e8abf2e33298bd674ff8b1fba239039cc64215e620a59f30cf6745990c5da448e40f49aa84cdb722dba2afd2f7985788469fb97e09
data/LICENSE CHANGED
@@ -12,7 +12,10 @@ Additional Use Grant: You may use the Licensed Work for any purpose,
12
12
  hosted or managed service whose primary value is the
13
13
  functionality of the Licensed Work itself.
14
14
 
15
- Change Date: 2029-04-25
15
+ Change Date: Three years from the date each version of the
16
+ Licensed Work is first publicly released.
17
+ Versions 0.3.0 through 0.29.3 keep the Change
18
+ Date of 2029-04-25 they were released with.
16
19
 
17
20
  Change License: GNU General Public License (GPL) Version 3
18
21
 
data/README.md CHANGED
@@ -10,27 +10,50 @@
10
10
 
11
11
  Your prompts need tests too.
12
12
 
13
- Run every prompt against real data. Score each output with an LLM judge against criteria you define. Change anything: the prompt, the model, the temperature, the dataset. Re-run and see exactly what got better and what broke. When the scores tell you something's off, CompletionKit suggests an improved prompt based on the judge's actual feedback on your runs. You inspect the diff, apply it as a new version, and verify the improvement.
13
+ Run every prompt against real data. Score each output against criteria you define, with an LLM judge or a plain check. Change anything: the prompt, the model, the temperature, the dataset. Re-run and see exactly what got better and what broke. When the scores tell you something's off, CompletionKit suggests an improved prompt based on the judge's feedback on your runs. You inspect the diff, apply it as a new version, and check the improvement.
14
14
 
15
15
  It's the difference between "this prompt seems to work" and "this prompt scores 4.3 out of 5 across 200 inputs, up from 3.8 last version."
16
16
 
17
17
  **[Start on completionkit.com →](https://completionkit.com)** | **[RubyGems](https://rubygems.org/gems/completion-kit)**
18
18
 
19
- > **Just want to use it?** [CompletionKit Cloud](https://completionkit.com) is the same engine, fully hosted — zero install, no Rails ops, plans at [completionkit.com/pricing](https://completionkit.com/pricing).
19
+ ![The CompletionKit dashboard, showing workspace totals, run activity over the last 14 days, the worst-scoring metric, version-over-version score changes, and recent runs](https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/screenshots/dashboard.png)
20
20
 
21
- ![The CompletionKit dashboard — workspace totals, run activity over the last 14 days, the worst-scoring metric, version-over-version score changes, and recent runs](https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/screenshots/dashboard.png)
21
+ ## How it works
22
+
23
+ <p align="center">
24
+ <img src="https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/diagrams/workflow.png" alt="CompletionKit workflow: a prompt and a dataset feed a run against a model, an LLM judge scores each output on your rubric, low scores drive an AI-suggested rewrite, and the new prompt version re-runs so you can compare" width="820" />
25
+ </p>
26
+
27
+ It's a loop. Each pass leaves you with a score you can compare against the last one.
28
+
29
+ 1. **Create a prompt** with `{{variable}}` placeholders.
30
+ 2. **Upload a dataset.** A CSV whose column headers match the variable names.
31
+ 3. **Run it** against a model and score the outputs against your metrics.
32
+ 4. **Iterate.** Change the prompt, the model, the temperature, or the dataset and re-run. CompletionKit versions your prompts, so you can always compare against earlier results.
33
+ 5. **Get suggestions.** When scores drop, ask CompletionKit for an improved prompt. The suggestion comes from the judge's feedback on each response, not generic prompt advice. Inspect the diff and apply it as a new version.
34
+
35
+ ## Concepts
36
+
37
+ - **Prompt.** A versioned template with `{{variable}}` placeholders. Editing a prompt that has already run creates a new version, so earlier results stay reproducible.
38
+ - **Dataset.** A CSV of real inputs. Each row becomes one test case.
39
+ - **Run.** One execution of a prompt against a dataset. It records every input (model, temperature, metrics) and stores every output and score.
40
+ - **Response.** The model's output for one dataset row, with its scores attached.
41
+ - **Metric.** What you score each response on. A judge metric has an instruction, evaluation steps and a 1 to 5 scale, and an LLM judge applies it. A check metric is a pass or fail test with no LLM involved: contains, does not contain, equals, regex, valid JSON, JSON field equals, length bounds, list overlap, number bounds, and number equals.
42
+ - **Metric group.** A set of metrics you can apply to a run together.
43
+ - **Tag.** A label for prompts, runs, metrics and datasets. Filter any list by tag (`?tag[]=...`).
44
+ - **Provider credential.** An API key for a model provider. It is encrypted at rest and never returned through the API.
22
45
 
23
46
  ## Three ways to run it
24
47
 
25
- Same engine, same UI, same REST API and MCP server — pick the deployment that fits. The first two are stack-agnostic: you run CompletionKit as a product and talk to it over HTTP and MCP, whatever language your own app is written in. The third is for teams already building on Rails.
48
+ Every option has the same engine, UI, REST API and MCP server. The first two work with any stack: you run CompletionKit as a product and talk to it over HTTP and MCP, whatever language your app is written in. The third is for teams already on Rails.
26
49
 
27
- ### 1. Hosted — [completionkit.com](https://completionkit.com) (recommended)
50
+ ### 1. Hosted at [completionkit.com](https://completionkit.com) (recommended)
28
51
 
29
- The fastest path. Sign up and you're running on the same engine you'd self-host, without touching a Rails app. No `db:migrate`, no Puma, no Solid Queue, no provider key management — multi-tenant workspaces, your team logs in, you go. Plans at [completionkit.com/pricing](https://completionkit.com/pricing).
52
+ Sign up and start testing prompts. There's no Rails app to run, no migrations, no job worker and no servers to keep up. Your whole team shares a workspace. Plans are at [completionkit.com/pricing](https://completionkit.com/pricing).
30
53
 
31
- ### 2. Self-hosted — the bundled standalone app
54
+ ### 2. Self-hosted standalone app
32
55
 
33
- Run it on your own infra as a self-contained product. There's nothing to integrate and no Ruby to write — once it's up, you drive everything through the web UI, the REST API, and the MCP server, from whatever stack your own app is built in. It needs Postgres and a host that can run the app (Fly, Render, Heroku, Docker, …).
56
+ Run it on your own servers as a self-contained product. There's nothing to integrate and no Ruby to write: once it's up, you drive it through the web UI, the REST API and the MCP server. It needs Postgres and a host that can run a Rails app.
34
57
 
35
58
  ```bash
36
59
  git clone https://github.com/homemade-software-inc/completion-kit.git
@@ -40,7 +63,7 @@ bin/rails completion_kit:install:migrations
40
63
  bin/rails db:migrate
41
64
  ```
42
65
 
43
- Run **both** a web server and a Solid Queue worker. In two terminals:
66
+ Run a web server and a Solid Queue worker, in two terminals:
44
67
 
45
68
  ```bash
46
69
  bin/rails server
@@ -50,11 +73,11 @@ bin/rails server
50
73
  bin/jobs
51
74
  ```
52
75
 
53
- Or with [foreman](https://github.com/ddollar/foreman) in one terminal: `foreman start -f Procfile.dev`.
76
+ Or run both in one terminal with [foreman](https://github.com/ddollar/foreman): `foreman start -f Procfile.dev`.
54
77
 
55
- Visit `http://localhost:3000`. Add a provider credential (Settings), create a prompt, upload a CSV dataset, and run it. See [Deploying self-hosted](#deploying-self-hosted) for the production-env setup.
78
+ Visit `http://localhost:3000`. Add a provider credential under Settings, create a prompt, upload a CSV dataset, and run it. See [Deploying self-hosted](#deploying-self-hosted) when you're ready for production.
56
79
 
57
- ### 3. Rails engine — mount into your existing Rails app
80
+ ### 3. Mounted in your Rails app
58
81
 
59
82
  ```ruby
60
83
  gem "completion-kit"
@@ -65,22 +88,24 @@ bin/rails generate completion_kit:install
65
88
  bin/rails db:migrate
66
89
  ```
67
90
 
68
- The engine mounts at `/completion_kit`. Generate / judge flows enqueue Active Job jobs (`CompletionKit::GenerateRowJob`, `CompletionKit::JudgeReviewJob`, `CompletionKit::RunCompletionCheckJob`), so your host app needs an Active Job adapter that actually processes them — Solid Queue, Sidekiq, GoodJob, etc. The `:async` adapter is **not** suitable for production: it runs jobs in the web Puma's thread pool with no durability and no retry, and a long LLM call will block request handling.
91
+ The engine mounts at `/completion_kit`. Runs work through Active Job (`CompletionKit::GenerateRowJob`, `CompletionKit::JudgeReviewJob`, `CompletionKit::RunCompletionCheckJob` and others), so your app needs a real job backend such as Solid Queue, Sidekiq or GoodJob. Don't use the `:async` adapter in production. It runs jobs inside Puma's thread pool with no durability or retries, and a long LLM call will hold up web requests.
92
+
93
+ #### Using your own layout
69
94
 
70
- **Host-app layout integration.** If your host app overrides the engine layout (e.g. `layout "application"` on engine controllers, or rendering engine views inside your own shell), include both the engine's stylesheet and JavaScript in that layout:
95
+ If your app renders the engine's pages inside its own layout, include the engine's stylesheet and JavaScript there:
71
96
 
72
97
  ```erb
73
98
  <%= stylesheet_link_tag "completion_kit/application", media: "all" %>
74
99
  <%= javascript_include_tag "completion_kit/application", defer: true, nonce: true %>
75
100
  ```
76
101
 
77
- Without the JavaScript include, in-page behaviours silently fail: clickable table rows, dialogs, copy buttons, the API token reveal, the run form, live run status, live tag-breadcrumb updates, relative-time ticking, CSV row hover-expand, model-refresh progress, focus-first-error, and local-time formatting.
102
+ Without the JavaScript, much of the UI quietly stops working: clickable rows, dialogs, copy buttons, the run form, live run status and local times among them.
78
103
 
79
- **Content Security Policy.** The engine's pages have no inline event handlers or inline scripts, and the engine layout puts the request nonce on its script tags, so the pages work under a nonce-based `script-src`. The views still use inline `style` attributes, so leave `style-src` out of the nonce directives and allow `'unsafe-inline'` there for now.
104
+ #### Content Security Policy
80
105
 
81
- Use a nonce that lasts the whole session, not a fresh random one per request. Turbo Drive re-creates the scripts in each new page body with the nonce from the `csp-nonce` meta tag, but the browser keeps enforcing the policy header of the first page it loaded, so a per-request nonce blocks nonce scripts after the first Drive visit.
106
+ The engine's pages have no inline scripts or inline event handlers, and the engine layout puts the request nonce on its script tags, so they work under a nonce-based `script-src`. The views still use inline `style` attributes, so keep `style-src` out of the nonce directives and allow `'unsafe-inline'` there for now.
82
107
 
83
- Keep a random nonce in the session, and skip it for API controllers so cookie-less API and MCP calls get no session cookie, as the example below does. The Rails default, `->(request) { request.session.id.to_s }`, also lasts the session, but it is empty on a brand-new visitor's first response, so a policy that admits scripts only by nonce blocks that first page's scripts until the session exists.
108
+ Use a nonce that lasts the whole session. Turbo Drive re-creates scripts on each visit using the nonce from the `csp-nonce` meta tag, but the browser keeps enforcing the policy from the first page it loaded, so a fresh nonce per request blocks scripts after the first visit. The Rails default nonce, the session id, is empty on a new visitor's first response, so it blocks that first page's scripts. Keep a random nonce in the session instead, and skip API controllers so API and MCP calls don't get a session cookie:
84
109
 
85
110
  ```ruby
86
111
  Rails.application.configure do
@@ -101,25 +126,25 @@ end
101
126
 
102
127
  ## Providers
103
128
 
104
- CompletionKit discovers available models from each provider's API automatically.
129
+ CompletionKit finds the available models from each provider's API.
105
130
 
106
131
  | Provider | Env vars | What it covers |
107
132
  |----------|----------|----------------|
108
- | **OpenAI** | `OPENAI_API_KEY` | GPT-5, GPT-4.1, GPT-4o, etc. |
133
+ | **OpenAI** | `OPENAI_API_KEY` | GPT-5, GPT-4.1, GPT-4o and the rest |
109
134
  | **Anthropic** | `ANTHROPIC_API_KEY` | Claude Opus, Sonnet, Haiku |
110
- | **Ollama / local endpoint** | `OLLAMA_API_ENDPOINT` (default: `http://localhost:11434/v1`), `OLLAMA_API_KEY` (optional) | Any model you've `ollama pull`-ed, or any OpenAI-compatible local server (vLLM, LM Studio, llama.cpp) |
111
- | **OpenRouter** | `OPENROUTER_API_KEY` | 100+ models from 30+ providers through one API key |
112
- | **Azure AI Foundry** | No env var — add via the Providers UI / a `ProviderCredential` (`api_key` + `api_endpoint` required, `api_version` optional) | Azure OpenAI deployments and Foundry projects |
135
+ | **Ollama or a local endpoint** | `OLLAMA_API_ENDPOINT` (default `http://localhost:11434/v1`), `OLLAMA_API_KEY` (optional) | Any model you've pulled with Ollama, or any OpenAI-compatible local server (vLLM, LM Studio, llama.cpp) |
136
+ | **OpenRouter** | `OPENROUTER_API_KEY` | 100+ models from 30+ providers through one key |
137
+ | **Azure AI Foundry** | None. Add it as a provider credential with `api_key` and `api_endpoint` (`api_version` is optional) | Azure OpenAI deployments and Foundry projects |
113
138
 
114
- Set these as environment variables or configure them in the generated initializer. You can also add provider credentials through the web UI under Settings.
139
+ Set these as environment variables, in the generated initializer, or as provider credentials under Settings in the web UI.
115
140
 
116
- The Ollama key from `OLLAMA_API_KEY` or `config.ollama_api_key` goes only to the Ollama endpoint from `OLLAMA_API_ENDPOINT` or `config.ollama_api_endpoint`, or to `http://localhost:11434/v1` when neither is set and `config.allow_loopback_endpoints` is on, which is the default. A provider credential with no key of its own gets that key only while it points at that same endpoint. Point it anywhere else and CompletionKit sends no key, so store the key on the credential itself. A key stored on a credential always goes to that credential's endpoint. Two endpoints match when their scheme, host, port and path match. The case of the scheme and host, a default port written out, a trailing slash and a trailing `/v1` make no difference. A blank stored key counts as no key.
141
+ The Ollama key from `OLLAMA_API_KEY` or `config.ollama_api_key` is only ever sent to the Ollama endpoint you configured, or to `http://localhost:11434/v1` when you configured none and loopback endpoints are allowed. A provider credential without its own key borrows that key only while it points at that same endpoint. Point it anywhere else and CompletionKit sends no key, so store the key on the credential. A key stored on a credential always goes to that credential's endpoint. Endpoints match on scheme, host, port and path; letter case, a written-out default port, a trailing slash and a trailing `/v1` don't count. A blank stored key counts as no key.
117
142
 
118
- ### Encryption
143
+ Custom model endpoints and CSV imports by URL may reach the server's own loopback interface by default, so a local Ollama works out of the box. If the people using CompletionKit shouldn't reach services on the app server itself, set `config.allow_loopback_endpoints = false`. Private, link-local and reserved addresses are always refused.
119
144
 
120
- Provider API keys are encrypted at rest using [Active Record encryption](https://guides.rubyonrails.org/active_record_encryption.html). You need three encryption keys configured before the app will boot in production.
145
+ ### Encryption
121
146
 
122
- Generate them:
147
+ Provider API keys are encrypted at rest with [Active Record encryption](https://guides.rubyonrails.org/active_record_encryption.html). The app won't boot in production until three encryption keys are set. Generate them:
123
148
 
124
149
  ```bash
125
150
  bin/rails db:encryption:init
@@ -133,13 +158,13 @@ COMPLETION_KIT_ENCRYPTION_DETERMINISTIC_KEY=<generated value>
133
158
  COMPLETION_KIT_ENCRYPTION_KEY_DERIVATION_SALT=<generated value>
134
159
  ```
135
160
 
136
- Or add them to `config/credentials.yml.enc` under `active_record_encryption`. In development, the standalone app uses built-in fallback values so you can skip this step locally.
161
+ You can put them in `config/credentials.yml.enc` under `active_record_encryption` instead. In development the standalone app falls back to built-in values, so you can skip this locally.
137
162
 
138
163
  ## Authentication
139
164
 
140
- CompletionKit requires authentication in any deployed environment. In development and test, routes are open by default (with a log warning); every other environment returns 403 until auth is configured.
165
+ CompletionKit needs authentication anywhere it's deployed. In development and test the routes are open, with a warning in the log. Every other environment returns 403 until you configure one of these two modes.
141
166
 
142
- ### Basic Auth (recommended for simple setups)
167
+ ### Basic auth
143
168
 
144
169
  ```ruby
145
170
  CompletionKit.configure do |c|
@@ -148,7 +173,7 @@ CompletionKit.configure do |c|
148
173
  end
149
174
  ```
150
175
 
151
- ### Custom Auth (Devise, etc.)
176
+ ### Your own auth (Devise and the like)
152
177
 
153
178
  ```ruby
154
179
  CompletionKit.configure do |c|
@@ -156,11 +181,9 @@ CompletionKit.configure do |c|
156
181
  end
157
182
  ```
158
183
 
159
- Only one mode can be active.
160
-
161
- ### Reviewer name
184
+ ### Reviewer names
162
185
 
163
- When someone records a verdict on a judge's score in the web UI, CompletionKit saves it under a reviewer name. Each reviewer keeps one verdict per response and metric, so give every person their own name. Otherwise their verdicts overwrite each other. Set `current_reviewer` to a callable that receives the controller and returns the signed-in person's name:
186
+ When someone agrees or disagrees with a judge's score in the web UI, CompletionKit saves that verdict under a reviewer name. Each reviewer keeps one verdict per response and metric, so give each person their own name or their verdicts overwrite each other. Set `current_reviewer` to a callable that takes the controller and returns the signed-in person's name:
164
187
 
165
188
  ```ruby
166
189
  CompletionKit.configure do |c|
@@ -169,7 +192,9 @@ CompletionKit.configure do |c|
169
192
  end
170
193
  ```
171
194
 
172
- The callable must return a String or nil. Any other value, such as a user record, raises `CompletionKit::ConfigurationError`. If `current_reviewer` is unset or returns a blank value, verdicts are saved under `username`, or `"operator"` when no username is set. The web UI does not read the `X-Remote-User` header on its own. If a proxy in front of your app sets that header, pass it through:
195
+ It must return a String or nil. Anything else, such as a user record, raises `CompletionKit::ConfigurationError`. When `current_reviewer` is unset or returns a blank value, verdicts go under `username`, or `"operator"` if there's no username.
196
+
197
+ The web UI doesn't read the `X-Remote-User` header by itself. If a proxy in front of your app sets it, pass it through:
173
198
 
174
199
  ```ruby
175
200
  CompletionKit.configure do |c|
@@ -177,71 +202,46 @@ CompletionKit.configure do |c|
177
202
  end
178
203
  ```
179
204
 
180
- Only use this header when a trusted proxy always overwrites it. Otherwise any client can send it and claim any reviewer name.
205
+ Only do this when a trusted proxy always overwrites that header. Otherwise any client can send it and claim any name.
181
206
 
182
207
  The REST API and the MCP server take the reviewer name from their own `created_by` parameter.
183
208
 
184
- ## Rate limiting
209
+ ## Limits
210
+
211
+ ### Rate limits
185
212
 
186
- The REST API, the MCP endpoint, and the web UI are rate limited per IP, per minute. The defaults are generous; tune them in the initializer:
213
+ The REST API, the MCP endpoint and the web UI are rate limited per IP, per minute. The defaults are generous:
187
214
 
188
215
  ```ruby
189
216
  CompletionKit.configure do |c|
190
- c.api_rate_limit = 120 # REST API + MCP, requests per minute (default 120)
191
- c.web_rate_limit = 300 # web UI, requests per minute (default 300)
217
+ c.api_rate_limit = 120 # REST API and MCP, requests per minute
218
+ c.web_rate_limit = 300 # web UI, requests per minute
192
219
  end
193
220
  ```
194
221
 
195
- Limiting uses `Rails.cache`. A shared cache store (Solid Cache, Redis) throttles accurately across multiple app instances; a per-process store still throttles each instance independently.
222
+ Limits are counted in `Rails.cache`. A shared store such as Solid Cache or Redis counts across all your app instances. A per-process store limits each instance on its own.
196
223
 
197
- ## Input limits
224
+ ### Input limits
198
225
 
199
- Uploads, prompt templates, and regex check patterns have limits, so one large or hostile input cannot tie up a worker. Tune them in the initializer:
226
+ Uploads, prompt templates and regex checks have limits, so one huge or hostile input can't tie up a worker:
200
227
 
201
228
  ```ruby
202
229
  CompletionKit.configure do |c|
203
230
  c.max_upload_bytes = 25 * 1024 * 1024 # largest dataset CSV or import file (default 25 MB)
204
231
  c.max_template_bytes = 1024 * 1024 # largest prompt template (default 1 MB)
205
- c.regex_timeout = 1.0 # seconds a regex check may spend matching one row (default 1.0)
232
+ c.regex_timeout = 1.0 # seconds a regex check may spend on one row (default 1.0)
206
233
  end
207
234
  ```
208
235
 
209
- A prompt whose template is over `max_template_bytes` fails validation when its template is saved. Prompts saved before a lower limit keep working until their template changes.
210
-
211
- The first line of a dataset's CSV is its header row. A dataset fails validation when its CSV is saved with a header row that is blank or longer than 64 KB. The 64 KB limit is fixed, not a setting.
236
+ A prompt over `max_template_bytes` fails validation when it's saved. Prompts saved before you lowered the limit keep working until their template changes.
212
237
 
213
- A dataset also fails validation when its CSV is saved with bytes that are not valid UTF-8, so export spreadsheets as UTF-8 CSV.
238
+ A dataset fails validation when its header row is blank or longer than 64 KB (a fixed limit), or when the CSV isn't valid UTF-8. Export spreadsheets as UTF-8 CSV.
214
239
 
215
- A regex check that runs past its time limit fails that row and says it timed out. The limit is `regex_timeout`, or your app's `Regexp.timeout` when that is shorter, so the engine never loosens a stricter limit your app set. The engine sets the limit on each pattern and never changes the process-wide `Regexp.timeout`. Set `regex_timeout` to a finite number of seconds of at least `0.001`, or to `nil` to use only `Regexp.timeout`. Any other value, such as `0` or `Float::INFINITY`, raises an `ArgumentError` when the initializer runs.
216
-
217
- ## How it works
218
-
219
- <p align="center">
220
- <img src="https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/diagrams/workflow.png" alt="CompletionKit workflow: a prompt and a dataset feed a run against a model, an LLM judge scores each output on your rubric, low scores drive an AI-suggested rewrite, and the new prompt version re-runs so you can compare" width="820" />
221
- </p>
222
-
223
- It's a loop. Each pass leaves you with a score you can compare against the last one.
224
-
225
- 1. **Create a prompt** with `{{variable}}` placeholders
226
- 2. **Upload a dataset.** A CSV where column headers match the variable names.
227
- 3. **Run it** against a model and score outputs with an LLM judge against criteria you define.
228
- 4. **Iterate.** Change the prompt, the model, the temperature, or the dataset and re-run. CompletionKit versions your prompts so you can always compare against previous results.
229
- 5. **Get suggestions.** When scores drop, ask CompletionKit for an AI-generated improvement. The suggestion is based on the judge's actual per-response feedback, not generic prompt-engineering advice. Inspect the diff and apply it as a new version.
230
-
231
- ## Concepts
232
-
233
- - **Prompt.** A versioned template with `{{variable}}` placeholders. Editing a prompt that's already been run creates a new version, so earlier results stay reproducible.
234
- - **Dataset.** A CSV of real inputs. Each row becomes one test case.
235
- - **Run.** One execution of a prompt against a dataset. Captures every input (model, temperature, metrics) and stores all outputs and scores.
236
- - **Response.** The model's output for one dataset row, with reviews attached.
237
- - **Metric.** An evaluation dimension with a name, instruction, evaluation steps, and a 1-5 star scoring scale. The LLM judge uses this to score each response.
238
- - **Metric Group.** A reusable group of metrics you can apply to a run as a set.
239
- - **Tag.** A domain label you can attach to prompts, runs, metrics, and datasets. Auto-assigned from a 10-color palette. Filter any index page by tag (`?tag[]=...`).
240
- - **Provider Credential.** An API key for a model provider. Encrypted at rest, never returned through the API.
240
+ A regex check that runs out of time fails that row and says so. The limit is `regex_timeout`, or your app's `Regexp.timeout` if that's shorter, so the engine never loosens a stricter limit you set. It sets the limit per pattern and never touches the process-wide `Regexp.timeout`. Set `regex_timeout` to at least `0.001` seconds, or to `nil` to rely on `Regexp.timeout` alone. Any other value, such as `0` or `Float::INFINITY`, raises an `ArgumentError` when the initializer runs.
241
241
 
242
242
  ## REST API
243
243
 
244
- Every resource is accessible via a bearer-token JSON API:
244
+ Every resource is available through a JSON API with a bearer token:
245
245
 
246
246
  ```ruby
247
247
  CompletionKit.configure { |c| c.api_token = ENV["COMPLETION_KIT_API_TOKEN"] }
@@ -252,11 +252,11 @@ curl -H "Authorization: Bearer $TOKEN" \
252
252
  http://localhost:3000/completion_kit/api/v1/prompts
253
253
  ```
254
254
 
255
- Visit `/completion_kit/api_reference` in your running app for per-endpoint docs with copy-to-clipboard curl examples.
255
+ Visit `/completion_kit/api_reference` in your running app for docs on every endpoint, with curl examples you can copy.
256
256
 
257
257
  ## MCP server
258
258
 
259
- CompletionKit runs a [Model Context Protocol](https://modelcontextprotocol.io) server at `/completion_kit/mcp`, exposing every resource as tools that MCP-aware clients (Claude Code, Cursor, etc.) can drive directly:
259
+ CompletionKit runs a [Model Context Protocol](https://modelcontextprotocol.io) server at `/completion_kit/mcp`. It exposes every resource as tools that MCP clients such as Claude Code and Cursor can use directly:
260
260
 
261
261
  ```json
262
262
  {
@@ -269,21 +269,21 @@ CompletionKit runs a [Model Context Protocol](https://modelcontextprotocol.io) s
269
269
  }
270
270
  ```
271
271
 
272
- The in-app API reference page has install snippets you can copy straight into your MCP client config.
272
+ The API reference page in the app has snippets you can paste into your MCP client's config.
273
273
 
274
274
  ## Deploying self-hosted
275
275
 
276
- Any Rails-friendly host works (Fly, Heroku, Render, Docker, etc.). Point it at a Postgres instance via `DATABASE_URL`, set your provider env vars, and run `cd standalone && bin/rails db:migrate` on each deploy.
276
+ Any host that runs Rails will do: Fly, Render, Heroku, Docker and so on. Point it at Postgres with `DATABASE_URL`, set your provider env vars, and run `cd standalone && bin/rails db:migrate` on each deploy.
277
277
 
278
278
  | Variable | Purpose | Default |
279
279
  |----------|---------|---------|
280
- | `COMPLETION_KIT_API_TOKEN` | Bearer token for REST API and MCP | (none, API disabled) |
281
- | `COMPLETION_KIT_USERNAME` | Web UI login username | `admin` |
282
- | `COMPLETION_KIT_PASSWORD` | Web UI login password | (none, open in dev) |
280
+ | `COMPLETION_KIT_API_TOKEN` | Bearer token for the REST API and MCP | None, so the API is off |
281
+ | `COMPLETION_KIT_USERNAME` | Web UI username | `admin` |
282
+ | `COMPLETION_KIT_PASSWORD` | Web UI password | None, so it's open in development only |
283
283
 
284
- You also need the three `COMPLETION_KIT_ENCRYPTION_*` keys from the [Encryption](#encryption) section above.
284
+ You also need the three `COMPLETION_KIT_ENCRYPTION_*` keys from [Encryption](#encryption).
285
285
 
286
- When the gem ships a new migration, install it locally and commit before pushing:
286
+ When a new gem version ships a migration, install it locally and commit it before you deploy:
287
287
 
288
288
  ```bash
289
289
  cd standalone
@@ -294,13 +294,13 @@ git add db/migrate/ && git commit -m "install new engine migration"
294
294
 
295
295
  ### Docker
296
296
 
297
- The standalone app ships a `Dockerfile`, so you can self-host it without a Ruby toolchain on the host. Build with the **repository root** as the context — the app depends on the engine source alongside it:
297
+ The standalone app ships a `Dockerfile`, so you can run it without Ruby on the host. Build from the repository root, since the app needs the engine source next to it:
298
298
 
299
299
  ```bash
300
300
  docker build -f standalone/Dockerfile -t completion-kit .
301
301
  ```
302
302
 
303
- CompletionKit needs a Rails secret (`SECRET_KEY_BASE`) and three Active Record encryption keys. With Docker there's no Rails toolchain on the host to run `bin/rails db:encryption:init`, so generate them with `openssl`. Generate them **once** and keep them stable — if the encryption keys change, provider credentials already stored in the database can no longer be decrypted. Write everything to an env file:
303
+ CompletionKit needs a Rails secret (`SECRET_KEY_BASE`) and the three encryption keys. Without Ruby on the host you can't run `bin/rails db:encryption:init`, so generate them with `openssl` and write everything to an env file:
304
304
 
305
305
  ```bash
306
306
  cat > completion-kit.env <<EOF
@@ -312,58 +312,25 @@ COMPLETION_KIT_ENCRYPTION_KEY_DERIVATION_SALT=$(openssl rand -hex 32)
312
312
  EOF
313
313
  ```
314
314
 
315
- `openssl rand` runs as the file is written, so each line gets a real random value. Keep `completion-kit.env` out of version control and back it up somewhere safe.
315
+ Generate these **once** and keep them. If the encryption keys change, the provider keys already in the database can't be decrypted. Keep `completion-kit.env` out of version control and back it up somewhere safe.
316
316
 
317
- Run the web process and a job worker from the same image, both pointed at that file:
317
+ Run the web process and a job worker from the same image and the same env file, so both share the same secrets:
318
318
 
319
319
  ```bash
320
320
  docker run -d -p 3000:3000 --env-file completion-kit.env completion-kit
321
321
  docker run -d --env-file completion-kit.env completion-kit ./bin/jobs
322
322
  ```
323
323
 
324
- Both processes must share the same `SECRET_KEY_BASE` and encryption keys — the single env file guarantees that. The web container runs `db:prepare` on boot, so migrations apply on first start and on every deploy.
325
-
326
- ## Multi-tenant host apps (advanced)
327
-
328
- For hosts that mount CompletionKit in a multi-tenant app, two optional hooks scope engine records per tenant without forking the engine:
329
-
330
- ```ruby
331
- CompletionKit.configure do |config|
332
- config.tenant_scope = -> {
333
- org = Current.organization&.id
334
- org ? where(organization_id: org) : where("1=0")
335
- }
336
- config.tenant_scope_columns = [:organization_id]
337
- end
338
- ```
339
-
340
- `tenant_scope` runs as each engine model's `default_scope` (use `unscoped` to bypass). `tenant_scope_columns` is appended to every engine uniqueness validation. Adding the tenant columns and composite unique indexes lives in your host migrations. Both defaults (`nil`, `[]`) are no-ops. The `completion_kit:mark_interrupted_runs_failed` rake task bypasses `tenant_scope` and marks in-flight runs in every tenant, so a host that also enforces tenancy in the database, for example with row-level security, must run it as a database role that can see every tenant's rows.
341
-
342
- Custom model endpoints and CSV imports by URL may reach the server's own loopback interface by default, so a local Ollama works out of the box. If the people using the engine should not reach services on the app server itself, set `config.allow_loopback_endpoints = false`. Private, link-local and reserved addresses are always refused.
343
-
344
- One hook lets a host apply its own display rule everywhere the engine lists, counts, or traverses runs, without overriding controllers or views:
345
-
346
- ```ruby
347
- CompletionKit.configure do |config|
348
- config.runs_display_scope = -> { where(created_at: 90.days.ago..) }
349
- config.runs_display_footer_partial = "runs/display_notice"
350
- end
351
- ```
352
-
353
- `runs_display_scope` is a callable evaluated against a `Run` relation, in the same bare-`where` style as `tenant_scope` (it runs via `instance_exec`, so write it zero-arg with the relation as `self`, like a Rails `scope` lambda). It must return a relation: a callable that returns `nil` or anything non-chainable raises when a list renders. The engine applies it through `Run.display_scoped` at every run list and count it owns (the runs index, prompt and dataset show pages, the compare picker, new-run tag defaults, the v1 API index and its `X-Total-Count`, the MCP `runs_list` tool, the dashboard and API reference recent-runs panels, the dashboard "Runs" stat-card count, and provider-credential usage stats) and through `Run.visible_run_ids` for child records that traverse runs (the metric trust-panel sample and the agreement examples shown on a metric page). Use it for a list-only display rule rather than a global `default_scope`, which would null `Run` associations everywhere they are traversed.
354
-
355
- Deliberately exempt, because they must still see every run: id-addressed single-run lookups (`runs#show`, the MCP `runs_get` tool, the v1 API show), delete-confirmation cascade counts, the auto-generated run-name counter, the dashboard's stat-card gate (it keys off the unscoped run total so a workspace whose runs the display scope excludes entirely still sees its activity cards rather than a blank dashboard), and the judge few-shot seeding that learns from corrected examples even on runs the display scope excludes.
356
-
357
- `runs_display_footer_partial` names a partial rendered below the runs list on the dashboard, the index, and the prompt and dataset show pages; it receives the shown runs as a `runs` local. When the display scope excludes every run one of those surfaces would list, the surface says "No runs to show." and the footer partial renders beneath it. The engine cannot know why your scope excludes runs, so use the partial to explain your display rule: your host owns that rule in `runs_display_scope` and is the only side that can describe or count what it left out. Both default to `nil` (no-ops), leaving standalone behaviour unchanged.
324
+ The web container runs `db:prepare` on boot, so migrations apply on first start and on every deploy.
358
325
 
359
326
  ## Contributing
360
327
 
361
- See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, testing, and pull request guidelines.
328
+ See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, testing and pull request guidelines.
362
329
 
363
330
  ## License
364
331
 
365
- CompletionKit 0.3.0 and later are licensed under the [Business Source License 1.1](LICENSE). You may use CompletionKit freely for any purpose, including production, except to offer it (or a derivative) to third parties as a hosted or managed service whose primary value is CompletionKit itself. Three years after each release, that version automatically re-licenses to GPL-3.
332
+ CompletionKit 0.3.0 and later are licensed under the [Business Source License 1.1](LICENSE). You may use CompletionKit for any purpose, including production, except offering it (or a derivative) to others as a hosted or managed service whose main value is CompletionKit itself. Each version becomes GPL-3 three years after it's released. Versions 0.3.0 through 0.29.3 shipped with a fixed date and become GPL-3 on April 25, 2029.
366
333
 
367
334
  CompletionKit 0.2.x and earlier remain available under the [MIT License](https://github.com/homemade-software-inc/completion-kit/blob/v0.2.0/MIT-LICENSE).
368
335
 
369
- For alternative licensing, contact hello@homemade.software.
336
+ For other licensing, contact hello@homemade.software.
@@ -3,6 +3,7 @@ module CompletionKit
3
3
  module V1
4
4
  class BaseController < ActionController::API
5
5
  rate_limit to: CompletionKit.config.api_rate_limit, within: 1.minute,
6
+ by: -> { CompletionKit.config.api_rate_limit_key.call(request) },
6
7
  with: -> { render_error("Rate limit exceeded", status: :too_many_requests) }
7
8
  before_action :authenticate_api!
8
9
  rescue_from ConfigurationError, with: :render_configuration_error
@@ -5,7 +5,8 @@ module CompletionKit
5
5
 
6
6
  TOOLS = {
7
7
  "agreements_list" => {
8
- description: "List agreements. Filter by run_id, response_id, metric_id, or created_by.",
8
+ description: "List human reviews of the LLM judge's scores: each agree, disagree or borderline verdict, with any corrected score and note. Filter by run_id, response_id, metric_id, or created_by. Use it to see where people disagreed with the judge before improving a metric's rubric. It returns each verdict, oldest first, with its run, response, metric and metric version ids, who recorded it and when.",
9
+ annotations: Base::READ_ONLY,
9
10
  inputSchema: {
10
11
  type: "object",
11
12
  properties: {
@@ -19,7 +20,8 @@ module CompletionKit
19
20
  handler: :list
20
21
  },
21
22
  "agreements_create" => {
22
- description: "Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.",
23
+ description: "Record whether a person agrees with the LLM judge's score on one response for one metric, so you can measure how far to trust the judge. The response must belong to the given run. Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'. This upserts an agreement for (run, response, metric, created_by): a new verdict with the same created_by (default \"mcp\") replaces the earlier one, including its corrected score and note, and leaving either out clears it. The verdict is filed under the metric's current judge version, which is the version judges_compare counts it for, even when an earlier version scored that response. It works only for llm_judge metrics, because checks have nothing to calibrate. It returns the saved verdict.",
24
+ annotations: Base::DESTRUCTIVE,
23
25
  inputSchema: {
24
26
  type: "object",
25
27
  properties: {
@@ -4,6 +4,12 @@ module CompletionKit
4
4
  DEFAULT_PAGE_LIMIT = 50
5
5
  MAX_PAGE_LIMIT = 500
6
6
 
7
+ READ_ONLY = {readOnlyHint: true, destructiveHint: false, openWorldHint: false}.freeze
8
+ ADDITIVE = {readOnlyHint: false, destructiveHint: false, openWorldHint: false}.freeze
9
+ DESTRUCTIVE = {readOnlyHint: false, destructiveHint: true, openWorldHint: false}.freeze
10
+ OPEN_WORLD_ADDITIVE = {readOnlyHint: false, destructiveHint: false, openWorldHint: true}.freeze
11
+ OPEN_WORLD_DESTRUCTIVE = {readOnlyHint: false, destructiveHint: true, openWorldHint: true}.freeze
12
+
7
13
  def page_bounds(args)
8
14
  limit = args["limit"].to_i
9
15
  limit = DEFAULT_PAGE_LIMIT if limit <= 0
@@ -14,7 +20,7 @@ module CompletionKit
14
20
  end
15
21
 
16
22
  def definitions
17
- self::TOOLS.map { |name, config| {name: name, description: config[:description], inputSchema: config[:inputSchema]} }
23
+ self::TOOLS.map { |name, config| {name: name, description: config[:description], inputSchema: config[:inputSchema], annotations: config[:annotations]} }
18
24
  end
19
25
 
20
26
  def call(name, arguments)
@@ -9,17 +9,20 @@ module CompletionKit
9
9
 
10
10
  TOOLS = {
11
11
  "datasets_list" => {
12
- description: "List all datasets",
12
+ description: "List every test dataset in your CompletionKit account, newest first. Returns each dataset's id, name, full CSV data, tags and timestamps, so the result can be large when your datasets are big. Use it to find a dataset to test a prompt against or to score AI responses on. To read one dataset, use datasets_get.",
13
+ annotations: Base::READ_ONLY,
13
14
  inputSchema: {type: "object", properties: {}, required: []},
14
15
  handler: :list
15
16
  },
16
17
  "datasets_get" => {
17
- description: "Get a dataset by ID",
18
+ description: "Get one test dataset by its ID. Returns the dataset's name, its full CSV data (header row first, then one test case per row), its tags and its timestamps. Use it to check a dataset's columns and rows before you test a prompt or start a run against it.",
19
+ annotations: Base::READ_ONLY,
18
20
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
19
21
  handler: :get
20
22
  },
21
23
  "datasets_create" => {
22
- description: "Create a dataset with CSV data. First row is the header. Two column names are recognized specially: \"expected_output\" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and \"actual_output\" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.",
24
+ description: "Create a test dataset from CSV text included in the call. The first row is the header, and each later row is one test case. Every column is available to the prompt as a variable. Two column names are recognized specially: \"expected_output\" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and \"actual_output\" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Optional tag_names labels the dataset. Tag names are lowercased and may contain only unaccented letters, numbers, spaces, hyphens and underscores, up to 64 characters, and tags that don't exist yet are created. Returns the new dataset. The CSV is rejected if it is empty, is not valid UTF-8, has a blank header row or a header row over 64 KB, or is larger than the upload limit (25 MB by default). If the CSV is too big to put in the call, use datasets_create_from_url, which downloads files up to 10MB.",
25
+ annotations: Base::ADDITIVE,
23
26
  inputSchema: {
24
27
  type: "object",
25
28
  properties: {name: {type: "string"}, csv_data: {type: "string"}, tag_names: {type: "array", items: {type: "string"}}},
@@ -28,7 +31,8 @@ module CompletionKit
28
31
  handler: :create
29
32
  },
30
33
  "datasets_update" => {
31
- description: "Update a dataset",
34
+ description: "Change a test dataset's name, CSV data or tags. Passing csv_data replaces the whole CSV, and the dataset keeps no copy of the old version. Passing tag_names replaces all of the dataset's tags. Responses from runs that already ran keep the row values they used, but any run started after the change reads the new rows. If you remove or rename a column that a run needs (a variable in its prompt, its expected_column or its output column), that run cannot be started, finished, regraded, retried or rerun until the column is back. To add rows, read the dataset with datasets_get first and send back the full CSV. Returns the updated dataset. If tag_names is rejected, the name and CSV changes in the same call are still saved.",
35
+ annotations: Base::DESTRUCTIVE,
32
36
  inputSchema: {
33
37
  type: "object",
34
38
  properties: {id: {type: "integer"}, name: {type: "string"}, csv_data: {type: "string"}, tag_names: {type: "array", items: {type: "string"}}},
@@ -37,12 +41,14 @@ module CompletionKit
37
41
  handler: :update
38
42
  },
39
43
  "datasets_delete" => {
40
- description: "Delete a dataset",
44
+ description: "Permanently delete a test dataset. This also deletes every run that used it, with those runs' responses, scores, suggestions and human agreement labels. Prompts and metrics are kept. This cannot be undone, so confirm with the user before calling it. Returns a short confirmation.",
45
+ annotations: Base::DESTRUCTIVE,
41
46
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
42
47
  handler: :delete
43
48
  },
44
49
  "datasets_create_from_url" => {
45
- description: "Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the \"expected_output\" (answer key) and \"actual_output\" (pre-made output) columns are recognized specially, overridable per run.",
50
+ description: "Create a test dataset by downloading a CSV file from a public http(s) URL instead of pasting it into the call. Use this for large datasets: the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked, so it cannot point at private network addresses. Redirects are not followed and return an error, and files over 10MB are rejected. The first row is the header. The \"expected_output\" (answer key) and \"actual_output\" (pre-made output) columns are recognized specially, and both are overridable per run. Optional tag_names labels the dataset. Returns the new dataset, or an error that says why the URL, the download or the CSV was rejected.",
51
+ annotations: Base::OPEN_WORLD_ADDITIVE,
46
52
  inputSchema: {
47
53
  type: "object",
48
54
  properties: {
@@ -5,7 +5,8 @@ module CompletionKit
5
5
 
6
6
  TOOLS = {
7
7
  "promptfoo_import" => {
8
- description: "Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.",
8
+ description: "Import a promptfoo config (the full promptfooconfig.yaml text) so you can test its prompts and score their responses in CompletionKit. It creates a new prompt for each inline prompt template, one dataset from the tests' vars, and metrics from the assert blocks in tests and defaultTest, with identical asserts imported once. llm-rubric, g-eval, model-graded-closedqa and factuality become LLM judge metrics. contains, icontains, not-contains, not-icontains, equals, regex and is-json become deterministic check metrics. Prompts use the model named by the first provider, or gpt-4o if none is given, and are created even if no saved provider key serves that model. Prompts given as file:// references are skipped, so paste the file's contents instead. It only adds new records, never changes existing ones and does not start a run. Providers are only matched by name against your saved provider keys and reported as matched or unmatched. Any apiKey in the config is ignored, so remove keys before pasting. Returns a summary of what was created and of the prompts and assert types it skipped, with the reason for each. Assert options such as threshold, weight and provider, defaultTest vars, and asserts with no type are dropped without being listed in the summary. If a prompt or the dataset cannot be saved, the whole import is rolled back and nothing is created.",
9
+ annotations: Base::ADDITIVE,
9
10
  inputSchema: {
10
11
  type: "object",
11
12
  properties: {
@@ -5,7 +5,8 @@ module CompletionKit
5
5
 
6
6
  TOOLS = {
7
7
  "judges_replay" => {
8
- description: "Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.",
8
+ description: "Check an LLM judge against outputs you already have: set up a scoring run that grades a dataset column of existing outputs with one metric's current judge and the judge_model you name (wraps runs_create with prompt_id omitted and output_column supplied, defaulting to actual_output). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts. It returns the new run.",
9
+ annotations: Base::ADDITIVE,
9
10
  inputSchema: {
10
11
  type: "object",
11
12
  properties: {
@@ -20,7 +21,8 @@ module CompletionKit
20
21
  handler: :replay
21
22
  },
22
23
  "judges_compare" => {
23
- description: "Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.",
24
+ description: "Compare two versions of one metric's LLM judge to see which agrees more with human reviewers. It computes stats from the agreement verdicts already recorded under each version and saves nothing; verdicts on runs older than the account's history window are not counted. It requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). It returns, for each version, the number of verdicts, the agreement rate with its confidence range, borderline rate, mean absolute error and kappa, plus the change from A to B and a recommendation. The recommendation is need_more_data under 30 verdicts combined, recommend when B's agreement rate beats A's by more than 3 points, hold when it trails A's by more than 3 points, and no_change otherwise or when one version has no verdicts. It is unavailable for check metrics.",
25
+ annotations: Base::READ_ONLY,
24
26
  inputSchema: {
25
27
  type: "object",
26
28
  properties: {
@@ -5,17 +5,20 @@ module CompletionKit
5
5
 
6
6
  TOOLS = {
7
7
  "metric_groups_list" => {
8
- description: "List all metric groups",
8
+ description: "List every metric group in your workspace, newest first. A metric group is a saved, ordered set of metrics that you can attach to a run in one step, so runs are scored the same way each time. Returns each group's id, name, description, metric_ids, tags and timestamps. The full list comes back with no paging. Use it to find a metric_group_id for runs_create or runs_update.",
9
+ annotations: Base::READ_ONLY,
9
10
  inputSchema: {type: "object", properties: {}, required: []},
10
11
  handler: :list
11
12
  },
12
13
  "metric_groups_get" => {
13
- description: "Get a metric group by ID",
14
+ description: "Get one metric group by id. Returns its id, name, description, the ids of its metrics in order, tags and timestamps. Use metrics_get on each id to see how each metric scores AI responses.",
15
+ annotations: Base::READ_ONLY,
14
16
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
15
17
  handler: :get
16
18
  },
17
19
  "metric_groups_create" => {
18
- description: "Create a metric group",
20
+ description: "Create a metric group: a named, ordered set of metrics that you can attach to a run in one step with metric_group_id, so every run of a prompt is scored the same way. Send name, and optionally description, metric_ids in the order you want them and tag_names (any new tags are created). Each id must be a metric in this workspace, or the whole call fails. A run copies the group's metrics when you attach the group, so later changes to the group don't reach that run. Returns the saved group.",
21
+ annotations: Base::ADDITIVE,
19
22
  inputSchema: {
20
23
  type: "object",
21
24
  properties: {
@@ -28,7 +31,8 @@ module CompletionKit
28
31
  handler: :create
29
32
  },
30
33
  "metric_groups_update" => {
31
- description: "Update a metric group",
34
+ description: "Change a metric group's name, description, metrics or tags. This overwrites the group in place. metric_ids replaces the group's whole metric list in the order given, and an empty list removes every metric. Each id must be a metric in this workspace. Leave metric_ids out to keep the current list. tag_names replaces the whole tag set. Runs already created keep the metrics they had. Returns the updated group.",
35
+ annotations: Base::DESTRUCTIVE,
32
36
  inputSchema: {
33
37
  type: "object",
34
38
  properties: {
@@ -41,7 +45,8 @@ module CompletionKit
41
45
  handler: :update
42
46
  },
43
47
  "metric_groups_delete" => {
44
- description: "Delete a metric group",
48
+ description: "Permanently delete a metric group. Its tags are unlinked, though the tags themselves stay. The metrics in it, and any runs scored with it, stay as they are. This cannot be undone. Returns a confirmation message.",
49
+ annotations: Base::DESTRUCTIVE,
45
50
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
46
51
  handler: :delete
47
52
  }
@@ -5,7 +5,8 @@ module CompletionKit
5
5
 
6
6
  TOOLS = {
7
7
  "metric_versions_list" => {
8
- description: "List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.",
8
+ description: "List every version of one metric's judge or check, drafts and published, newest first. Use it to see how a judge changed over time, find drafts suggested by metrics_suggest_variants or saved from edits in the web app, or pick an older version to restore with metric_versions_publish. Each row carries id, version_number, state (draft or published), source, the current flag marking the live version, and timestamps, plus that version's instruction and rubric bands (judges) or check_config (checks). source is edit or suggestion for versions made those ways, and null for a version saved automatically the first time the metric scored a response or got a verdict. The current version can differ from the metric's live definition if metrics_update changed the metric after that version was saved.",
9
+ annotations: Base::READ_ONLY,
9
10
  inputSchema: {
10
11
  type: "object",
11
12
  properties: {
@@ -16,7 +17,8 @@ module CompletionKit
16
17
  handler: :list
17
18
  },
18
19
  "metric_versions_publish" => {
19
- description: "Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.",
20
+ description: "Make a metric version the live judge or check that scores new AI responses. It works both to publish a draft and to switch back to an older published version. In one step it marks this version current, marks every other version of the metric not current, and writes this version's instruction and rubric bands (or check_config, for a check) onto the metric, replacing what the metric had. Responses already scored keep their scores. Rescore a run with runs_regrade if you want new ones. Human verdicts stay with the version they were given on. Other published versions stay saved, so you can switch back by publishing one, but a live definition that was never saved as a version is lost. A check version with an invalid check_config is refused. Returns the published version.",
21
+ annotations: Base::DESTRUCTIVE,
20
22
  inputSchema: {
21
23
  type: "object",
22
24
  properties: {
@@ -27,7 +29,8 @@ module CompletionKit
27
29
  handler: :publish
28
30
  },
29
31
  "metric_versions_dismiss" => {
30
- description: "Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published version, publish a different one as current instead.",
32
+ description: "Permanently delete a draft version of a metric, whether it came from an edit in the web app or from metrics_suggest_variants. Use it to throw away a suggested judge rewrite you don't want. Published versions are refused. To stop using a published version, publish a different one as current instead. This cannot be undone. Returns the deleted version's id with destroyed: true.",
33
+ annotations: Base::DESTRUCTIVE,
31
34
  inputSchema: {
32
35
  type: "object",
33
36
  properties: {
@@ -44,17 +44,20 @@ module CompletionKit
44
44
 
45
45
  TOOLS = {
46
46
  "metrics_list" => {
47
- description: "List all metrics",
47
+ description: "List every metric in your workspace, newest first. A metric is how CompletionKit scores AI responses. It is either an LLM judge, which grades each response from 1 to 5 (5 is best) against an instruction and rubric, or a check, which scores a response by an exact rule such as contains, equals, regex, valid JSON, a JSON path value, length or number bounds, or list overlap. Returns each metric's id, name, key, type, tags and timestamps, plus its instruction and rubric bands (judges) or its check_config (checks). The full list comes back with no paging. Use it to find a metric id before you score a run, edit a judge or build a metric group.",
48
+ annotations: Base::READ_ONLY,
48
49
  inputSchema: {type: "object", properties: {}, required: []},
49
50
  handler: :list
50
51
  },
51
52
  "metrics_get" => {
52
- description: "Get a metric by ID",
53
+ description: "Get one metric by id. Returns its id, name, key, type, tags and timestamps, plus its live judge instruction and 1 to 5 rubric bands (for an LLM judge) or its check_config (for a check). Use it to read exactly how a metric scores AI responses before you change it or explain a score. To see its saved versions and drafts, use metric_versions_list.",
54
+ annotations: Base::READ_ONLY,
53
55
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
54
56
  handler: :get
55
57
  },
56
58
  "metrics_create" => {
57
- description: "Create a metric with evaluation criteria. #{CHECK_CONFIG_HINT}",
59
+ description: "Create a metric that scores AI responses. By default it is an LLM judge. Send name and instruction (what a good response looks like), and optionally rubric_bands describing 1 to 5 stars, where 5 is best. Any band you leave out gets a default description. For an exact rule instead of a judge, see the check details at the end. The metric's name must be unique in the workspace, ignoring case, spaces and most punctuation, and a renamed metric keeps its original name reserved. A clash fails with the error \"Key has already been taken\". tag_names adds tags and creates any that don't exist yet. Saving a metric scores nothing by itself. Attach it to a run with runs_create or runs_update, then score responses with runs_generate or runs_regrade. Returns the saved metric. #{CHECK_CONFIG_HINT}",
60
+ annotations: Base::ADDITIVE,
58
61
  inputSchema: {
59
62
  type: "object",
60
63
  properties: {
@@ -69,7 +72,8 @@ module CompletionKit
69
72
  handler: :create
70
73
  },
71
74
  "metrics_update" => {
72
- description: "Update a metric. #{CHECK_CONFIG_HINT}",
75
+ description: "Change a metric's name, judge instruction, rubric bands, check_config, type or tags. This overwrites the metric in place, and the new definition scores every response graded after it. It saves no draft or new version. If the metric already has a live version, scores and verdicts given after the change are still filed under that version, so judges_compare can't tell the old and new definitions apart. Responses already scored keep their old scores until you rescore the run with runs_regrade. Nothing tests the change before it goes live. Fields you leave out stay as they are, but a field you send replaces that whole field: rubric_bands resets any band you leave out to its default description, not its old one, and check_config replaces the whole config. tag_names replaces the whole tag set. A rename keeps the metric's original key, so its old name stays reserved. metric_type cannot change once the metric is in a run, has scores or has versions. Returns the updated metric. #{CHECK_CONFIG_HINT}",
76
+ annotations: Base::DESTRUCTIVE,
73
77
  inputSchema: {
74
78
  type: "object",
75
79
  properties: {
@@ -84,12 +88,14 @@ module CompletionKit
84
88
  handler: :update
85
89
  },
86
90
  "metrics_delete" => {
87
- description: "Delete a metric",
91
+ description: "Permanently delete a metric. This also deletes all of its versions and drafts and every human agree, disagree or borderline verdict recorded against them. It removes the metric from every run and metric group and unlinks its tags, though the tags themselves stay. Scores it already gave stay on their responses but are no longer linked to the metric. This cannot be undone. Returns a confirmation message.",
92
+ annotations: Base::DESTRUCTIVE,
88
93
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
89
94
  handler: :delete
90
95
  },
91
96
  "metrics_suggest_variants" => {
92
- description: "Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source=\"suggestion\". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.",
97
+ description: "Get AI suggestions to improve an LLM judge that disagrees with people. It sends the judge's current instruction and rubric, plus up to 8 recent responses where a person disagreed with the judge's score and up to 6 that a person marked borderline (each input and output cut to 200 characters), to your own model provider. The model is asked for rewrites of the instruction, the rubric or both that close the gap. Each rewrite is saved as a draft metric version with source \"suggestion\", and earlier suggestion drafts are kept. The live judge does not change until you publish a draft with metric_versions_publish. Nothing tests a draft before you publish it. Once it is live and people have recorded verdicts on it with agreements_create, judges_compare shows whether it agrees with people more often than the version it replaced. Returns the saved drafts. It works best after you have recorded disagreements with agreements_create. Checks are refused, because they are exact and have nothing to suggest. count defaults to 1 and is capped at 3. model picks the model that writes the rewrites and defaults to your judge model. The call uses your provider key and runs while you wait. It fails with an error if no judging model is set up or the model you name is unknown. If the model's reply can't be read, nothing is saved and you get a \"no parseable variants\" error. A call your provider refuses, such as one with a bad key, usually ends in that same error.",
98
+ annotations: Base::ADDITIVE,
93
99
  inputSchema: {
94
100
  type: "object",
95
101
  properties: {
@@ -5,17 +5,20 @@ module CompletionKit
5
5
 
6
6
  TOOLS = {
7
7
  "prompts_list" => {
8
- description: "List all prompts",
8
+ description: "List every prompt in CompletionKit, newest first. Each version of a prompt is its own entry, so drafts and older versions appear next to the published one. Versions of the same prompt share a family_key, and current=true marks the published version that apps get when they fetch the prompt by name or family_key. Returns each version's id, name, description, template, llm_model, family_key, version_number, current, timestamps and tags. Use it to find a prompt's ID before you test it, edit it, compare its versions or ask for a suggested improvement.",
9
+ annotations: Base::READ_ONLY,
9
10
  inputSchema: {type: "object", properties: {}, required: []},
10
11
  handler: :list
11
12
  },
12
13
  "prompts_get" => {
13
- description: "Get a prompt by ID",
14
+ description: "Get one prompt version by its numeric ID. Returns its name, description, template text with {{variable}} placeholders, the model that answers it (llm_model), family_key, version_number, whether it is the published version (current), timestamps and tags. Use it to read a prompt before you edit it, test it in a run or compare it with another version.",
15
+ annotations: Base::READ_ONLY,
14
16
  inputSchema: {type: "object", properties: {id: {type: "integer", description: "Prompt ID"}}, required: ["id"]},
15
17
  handler: :get
16
18
  },
17
19
  "prompts_create" => {
18
- description: "Create a prompt",
20
+ description: "Create a new prompt to test. Give it a name, a template with {{variable}} placeholders that match your dataset's column names, and the model that should answer it (llm_model). The model name is checked against the models your saved provider keys serve. If CompletionKit has not found any models from a saved key yet, any model name is accepted, and runs fail until a key for that model's provider is added. The new prompt is version 1 of a new prompt family and is published right away. If another published prompt already has the same name, apps that fetch prompts by name get this new one instead. To change an existing prompt, use prompts_update so its versions stay together. tag_names creates any tags that don't exist yet. Returns the saved prompt with its ID. To test it, create a run with runs_create and start it with runs_generate.",
21
+ annotations: Base::DESTRUCTIVE,
19
22
  inputSchema: {
20
23
  type: "object",
21
24
  properties: {
@@ -28,7 +31,8 @@ module CompletionKit
28
31
  handler: :create
29
32
  },
30
33
  "prompts_update" => {
31
- description: "Update a prompt. Changing the template or model of the published version, or editing any version that already has runs, creates a new DRAFT version (current=false) instead of changing it, so an agent's edits don't go live without a gate. Run and compare the draft, then promote it with prompts_publish. An unpublished draft with no runs, or a name, description or tag change on a published version with no runs, is updated in place.",
34
+ description: "Edit a prompt's name, description, template, model or tags. Changing the template or model of the published version, or editing any version that already has runs, creates a new DRAFT version (current=false) instead of changing it, so an agent's edits don't go live without a gate. The draft copies the old version's name, description, template and model, with your changes applied. Run and compare the draft, then promote it with prompts_publish. An unpublished draft with no runs, or a name, description or tag change on a published version with no runs, is updated in place and the old values are not kept. Renaming a published version in place takes effect at once, so apps that fetch it by its old name stop finding it, while fetches by family_key or ID still work. tag_names replaces the whole tag list. A new draft does not copy the old version's tags, so pass tag_names to keep them, or the draft has no tags, even after it is published. Returns the saved version, which has a new ID when a draft was created.",
35
+ annotations: Base::DESTRUCTIVE,
32
36
  inputSchema: {
33
37
  type: "object",
34
38
  properties: {
@@ -41,17 +45,20 @@ module CompletionKit
41
45
  handler: :update
42
46
  },
43
47
  "prompts_delete" => {
44
- description: "Delete a prompt",
48
+ description: "Permanently delete one prompt version by ID, along with every run of that version and all of those runs' responses, scores, judge reviews, agreements and suggestions. Other versions of the prompt, and your datasets, metrics and tags, are kept. Apps that fetch this version by its ID get a not-found error. If you delete the published version, no other version is published in its place, so apps that fetch the prompt by family_key get a not-found error, and apps that fetch it by name get a not-found error or another published prompt with the same name, until you publish another version with prompts_publish. This cannot be undone. Returns a confirmation message.",
49
+ annotations: Base::DESTRUCTIVE,
45
50
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
46
51
  handler: :delete
47
52
  },
48
53
  "prompts_publish" => {
49
- description: "Publish a prompt version, making it the current version",
54
+ description: "Publish a prompt version, making it the current version. Apps that fetch the prompt by its name or family_key get this version from their next request, and the version that was published before stops being current. Use it to ship a draft from prompts_update once a run shows it scores at least as well, or to roll back by publishing an older version. Returns the published version.",
55
+ annotations: Base::DESTRUCTIVE,
50
56
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
51
57
  handler: :publish
52
58
  },
53
59
  "prompts_suggest_improvement" => {
54
- description: "Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).",
60
+ description: "Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. It reads up to 20 of the run's responses with their scores and reviews, then asks the prompt's own model, on your saved provider key, for reasoning plus a rewritten template that keeps the {{variables}}. It then tests the rewrite on up to 30 of the run's judge-scored rows and scores the new answers with the run's LLM-judge metrics and judge model. Returns suggestion_id, prompt_id, reasoning, suggested_template, original_template, validation (before and after averages and how many rows improved, regressed or stayed the same) and net_negative, which is true when the rewrite's average score fell or more rows got worse than better. If validation.tested is 0, nothing was checked and net_negative is false. That happens when the run has no LLM-judge metrics, has no judge-scored rows with both input and output, or every test call failed. It saves the result as a Suggestion. The prompt itself is not changed, so pass suggested_template to prompts_update to save it as a draft version. Requires a run that has a prompt (not a scoring-only run). Each call can make dozens of model calls on your saved provider keys and can take a while.",
61
+ annotations: Base::ADDITIVE,
55
62
  inputSchema: {
56
63
  type: "object",
57
64
  properties: {run_id: {type: "integer", description: "The run whose results ground the improvement."}},
@@ -5,17 +5,20 @@ module CompletionKit
5
5
 
6
6
  TOOLS = {
7
7
  "provider_credentials_list" => {
8
- description: "List all provider credentials (API keys are not exposed)",
8
+ description: "List the model provider connections saved in your CompletionKit account (OpenAI, Anthropic, Ollama or another OpenAI-compatible endpoint, OpenRouter, Azure AI Foundry). Returns each one's id, provider, custom endpoint, API version and timestamps. API keys are not exposed. Use it to see which providers are connected before you test a prompt, start a run or use an LLM judge. It does not show whether a key is still stored or whether the provider's models loaded.",
9
+ annotations: Base::READ_ONLY,
9
10
  inputSchema: {type: "object", properties: {}, required: []},
10
11
  handler: :list
11
12
  },
12
13
  "provider_credentials_get" => {
13
- description: "Get a provider credential by ID (API key is not exposed)",
14
+ description: "Get one saved model provider connection by its ID. Returns its provider, custom endpoint, API version and timestamps. The API key is not exposed. Use it to check how a provider is set up before you change it.",
15
+ annotations: Base::READ_ONLY,
14
16
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
15
17
  handler: :get
16
18
  },
17
19
  "provider_credentials_create" => {
18
- description: "Create a provider credential",
20
+ description: "Save an API key for a model provider (openai, anthropic, ollama for Ollama or any other OpenAI-compatible endpoint, openrouter or azure_foundry) so CompletionKit can run your prompts and LLM judges with it. You can save one credential per provider. azure_foundry requires api_endpoint, ollama needs api_endpoint for its models to load, and any api_endpoint must be an http(s) URL that resolves to a public address. The key is stored encrypted and is never returned. Saving starts a background job that lists the provider's models. For every provider except openrouter, the job then sends each model that has not been tested yet a short test prompt to check it can generate text and, if it can, a second one to check it can judge, and these requests use your key. Returns the saved credential without the key.",
21
+ annotations: Base::OPEN_WORLD_ADDITIVE,
19
22
  inputSchema: {
20
23
  type: "object",
21
24
  properties: {
@@ -29,7 +32,8 @@ module CompletionKit
29
32
  handler: :create
30
33
  },
31
34
  "provider_credentials_update" => {
32
- description: "Update a provider credential. Changing provider or api_endpoint on a credential with a stored key requires api_key in the same call, so a stored key never moves to a new provider or endpoint.",
35
+ description: "Change a saved model provider connection: its API key, provider, custom endpoint or API version. A new api_key replaces the stored key, and the old key cannot be recovered. An empty api_key clears the stored key. Changing provider or api_endpoint on a credential with a stored key requires api_key in the same call, so a stored key never moves to a new provider or endpoint. Each provider can have only one credential. Any api_endpoint must be an http(s) URL that resolves to a public address. Each successful save runs model discovery again in the background. It lists the provider's models and, for every provider except openrouter, sends each model that has not been tested yet a short test prompt to check it can generate text and, if it can, a second one to check it can judge, using the key. Models the provider no longer lists are marked retired, and prompts or judges set to a retired model can fail to run. Returns the credential without the key.",
36
+ annotations: Base::OPEN_WORLD_DESTRUCTIVE,
33
37
  inputSchema: {
34
38
  type: "object",
35
39
  properties: {
@@ -42,7 +46,8 @@ module CompletionKit
42
46
  handler: :update
43
47
  },
44
48
  "provider_credentials_delete" => {
45
- description: "Delete a provider credential",
49
+ description: "Permanently delete a saved model provider connection and its stored API key, and remove that provider's models from your model list. It is refused with an error while any current prompt version uses one of that provider's models, or while any run, including past runs, used one of them as its judge. This cannot be undone, so confirm with the user before calling it. Returns a short confirmation.",
50
+ annotations: Base::DESTRUCTIVE,
46
51
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
47
52
  handler: :delete
48
53
  }
@@ -10,11 +10,8 @@ module CompletionKit
10
10
 
11
11
  TOOLS = {
12
12
  "responses_list" => {
13
- description: "List responses for a run, in row order. Returns " \
14
- "{total, limit, offset, returned, responses}. Defaults to #{Base::DEFAULT_PAGE_LIMIT} rows " \
15
- "because full payloads are large: use \"fields\" to drop the bodies, \"min_score\"/\"max_score\" " \
16
- "to isolate low scorers, and sort \"score_asc\" to read the worst rows first. For per-metric " \
17
- "averages of the whole run use runs_get instead of aggregating here.",
13
+ description: "List the AI responses in a run with their scores, in row order, to see which test cases scored well or badly and why. Each row has the input, the response_text, the expected_output, an average judge score from 1 to 5 (5 is best), and each metric's review with judge feedback or a check's pass/fail. Returns {total, limit, offset, returned, responses}. Defaults to #{Base::DEFAULT_PAGE_LIMIT} rows because full payloads are large: use \"fields\" to drop the bodies, \"min_score\"/\"max_score\" to isolate low scorers, and sort \"score_asc\" to read the worst rows first. Rows with no judge score, such as rows graded only by checks, are left out by a score filter and listed last by a score sort. For per-metric averages of the whole run use runs_get instead of aggregating here.",
14
+ annotations: Base::READ_ONLY,
18
15
  inputSchema: {
19
16
  type: "object",
20
17
  properties: {
@@ -32,7 +29,8 @@ module CompletionKit
32
29
  handler: :list
33
30
  },
34
31
  "responses_get" => {
35
- description: "Get a specific response",
32
+ description: "Get one response from a run in full, by run_id and response id. The response must belong to that run. Returns the input, the complete response_text, the expected_output, the average judge score from 1 to 5 (5 is best), each metric's review with the judge's score and feedback or the check's pass/fail, the row's status and attempts, and any provider error. Use it to read a single test case closely after responses_list points you to it.",
33
+ annotations: Base::READ_ONLY,
36
34
  inputSchema: {
37
35
  type: "object",
38
36
  properties: {run_id: {type: "integer"}, id: {type: "integer"}},
@@ -22,19 +22,20 @@ module CompletionKit
22
22
 
23
23
  TOOLS = {
24
24
  "runs_list" => {
25
- description: "List all runs",
25
+ description: "List your test runs, newest first. A run tests one prompt version against a dataset, or grades existing outputs, and scores each response with your metrics. It returns each run's id, name, status (pending, running, completed or failed), prompt and dataset ids, judge model, average score (1 to 5, 5 is best), check pass rate, per-metric averages and progress. Runs older than the account's history window may be left out. Use it to find a run's id or see recent results, then call runs_get for one run.",
26
+ annotations: Base::READ_ONLY,
26
27
  inputSchema: {type: "object", properties: {}, required: []},
27
28
  handler: :list
28
29
  },
29
30
  "runs_get" => {
30
- description: "Get a run by ID, including \"metric_averages\": a per-metric breakdown with each metric's " \
31
- "average score (or pass rate for checks), how many rows it graded, and how many scored low. " \
32
- "Use this to find the metric dragging a prompt down without listing responses.",
31
+ description: "Get one test run's results by id: status, progress, average score (1 to 5, 5 is best), check pass rate, the ids of failed responses, any failure reason, and \"metric_averages\", a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to see how a prompt scored and to find the metric dragging a prompt down without listing responses. Poll it after runs_generate, runs_regrade or runs_rerun until status is completed or failed. The result includes a warning when no metrics are attached or when the judge's scores are not reproducible.",
32
+ annotations: Base::READ_ONLY,
33
33
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
34
34
  handler: :get
35
35
  },
36
36
  "runs_create" => {
37
- description: "Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.",
37
+ description: "Set up a test run that scores a prompt's responses on a dataset. Give prompt_id and dataset_id to test a prompt: each dataset row fills the prompt's variables, the prompt's model answers it, and that answer gets scored. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones. Attach metrics with metric_ids or metric_group_id, and set judge_model, or LLM-judged metrics are skipped; with no metrics the run scores nothing. It fails if the dataset is missing a column the run needs, such as one the prompt uses, or if no configured provider serves judge_model. This only creates the run; call runs_generate to start it. It returns the new run, with a warning if no metrics are attached or if judge_temperature is above 0, which makes scores irreproducible.",
38
+ annotations: Base::ADDITIVE,
38
39
  inputSchema: {
39
40
  type: "object",
40
41
  properties: {
@@ -54,7 +55,8 @@ module CompletionKit
54
55
  handler: :create
55
56
  },
56
57
  "runs_update" => {
57
- description: "Update a run",
58
+ description: "Change a test run's settings: name, dataset, judge model, temperature, judge temperature, max tokens, output and expected columns, metrics or tags. The prompt cannot be changed. Passing metric_ids or metric_group_id replaces the run's whole metric list, and tag_names replaces all of its tags. It does not regenerate or rescore anything, and existing responses and scores stay as they are; call runs_regrade to rescore with the new metrics, or runs_rerun for a fresh run. It returns the updated run with the same warnings as runs_get.",
59
+ annotations: Base::DESTRUCTIVE,
58
60
  inputSchema: {
59
61
  type: "object",
60
62
  properties: {
@@ -74,27 +76,32 @@ module CompletionKit
74
76
  handler: :update
75
77
  },
76
78
  "runs_delete" => {
77
- description: "Delete a run",
79
+ description: "Permanently delete a test run and everything in it: its generated responses, their scores and judge feedback, the human agreement verdicts recorded on it (which also drops them from the judge's agreement stats), and any prompt suggestions made from it. This cannot be undone. The prompt, dataset and metrics are kept. Confirm with the user before calling. It returns a short confirmation message.",
80
+ annotations: Base::DESTRUCTIVE,
78
81
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
79
82
  handler: :delete
80
83
  },
81
84
  "runs_generate" => {
82
- description: "Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.",
85
+ description: "Start a test run so it produces and scores responses. It is required for every run, including score-only runs (no prompt): it generates responses with the prompt when there is one, and otherwise copies the graded dataset column and grades it. In a prompt run each dataset row goes to the prompt's model, and in either kind of run each LLM-judged metric then scores the result with the run's judge model. LLM-judged metrics are skipped when the run has no judge model. These model calls go to the providers you configured, which may charge for them; checks run without a provider call. The work runs in the background, so poll runs_get until status is completed or failed, and do not call this tool again to check progress. Only a pending or failed run can start, and any refused start marks the run as failed. Calling it on a running or completed run therefore does not start it but marks that run as failed, and a second call would then delete its results. Restarting a failed run first deletes its earlier responses, their scores and any human verdicts on them. For a completed run use runs_rerun, runs_retry_failures or runs_regrade instead. It returns the run with its progress, or the reason it could not start, such as a missing provider key or an empty dataset.",
86
+ annotations: Base::DESTRUCTIVE,
83
87
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
84
88
  handler: :generate
85
89
  },
86
90
  "runs_regrade" => {
87
- description: "Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.",
91
+ description: "Re-score a test run's existing responses with its currently attached metrics, without regenerating. Use it after attaching or editing metrics on an already-generated run. It first clears every existing score and judge comment on the run's succeeded responses, including scores for metrics no longer attached, so the earlier scores are lost. If the run has checks but no judge model, its LLM-judged scores are cleared and not replaced. Human verdicts already recorded on the run are kept even though people gave them on the earlier scores, and agreement stats then pair them with the new scores. LLM-judged metrics call your judge model's provider; checks do not. The work runs in the background, so poll runs_get for progress. It returns the run, or an error when no configured provider serves the judge model, or when there is nothing to re-grade because the run has no succeeded responses, no metrics, or only LLM-judged metrics and no working judge model.",
92
+ annotations: Base::DESTRUCTIVE,
88
93
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
89
94
  handler: :regrade
90
95
  },
91
96
  "runs_rerun" => {
92
- description: "Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.",
97
+ description: "Copy a test run and start the copy, with the same prompt version, dataset, metrics, judge model, settings and tags, under a new automatic name. Use it when the judge changed and you want a clean run instead of mixing versions, or to repeat a test. The original run and its scores stay untouched. The copy runs in the background and calls your model providers the same way runs_generate does. To test a different prompt version, create a new run with runs_create instead. It returns the new run; poll runs_get with its id for progress. If the prompt's model or the judge model has no configured provider, or the dataset no longer has a column the run needs, it returns an error and creates nothing. If the copy is created but cannot start, for example because the dataset is empty, it returns the reason and the copy stays as a failed run.",
98
+ annotations: Base::ADDITIVE,
93
99
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
94
100
  handler: :rerun
95
101
  },
96
102
  "runs_retry_failures" => {
97
- description: "Re-run only the failed responses of a run, optionally limited to specific response ids via \"only\".",
103
+ description: "Retry only the parts of a test run that failed, such as provider timeouts, rate limits or judge errors. Failed responses are generated again and failed scores are retried, while responses and scores that succeeded are left alone; optionally limit it to specific response ids via \"only\". It is refused when the judge has changed since the run executed, meaning a metric on the run now has a newer version than the one that scored it, because a retry would mix versions in the same run; use runs_rerun instead. The retries run in the background and call your model providers, except for checks. It returns the run with its progress, or the run unchanged when nothing failed.",
104
+ annotations: Base::ADDITIVE,
98
105
  inputSchema: {type: "object", properties: {id: {type: "integer"}, only: {type: "array", items: {type: "integer"}}}, required: ["id"]},
99
106
  handler: :retry_failures
100
107
  }
@@ -5,17 +5,20 @@ module CompletionKit
5
5
 
6
6
  TOOLS = {
7
7
  "tags_list" => {
8
- description: "List all tags",
8
+ description: "List every tag in CompletionKit, in name order. Returns each tag's id, name, color and timestamps. Tags label prompts, datasets, metrics, metric groups and runs. Use this to see which labels exist, or to find a tag's ID before renaming or deleting it.",
9
+ annotations: Base::READ_ONLY,
9
10
  inputSchema: {type: "object", properties: {}, required: []},
10
11
  handler: :list
11
12
  },
12
13
  "tags_get" => {
13
- description: "Get a tag by ID",
14
+ description: "Get one tag by ID. Returns its id, name, color and timestamps. Use it to check a tag's current name before you rename it or pass it in tag_names.",
15
+ annotations: Base::READ_ONLY,
14
16
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
15
17
  handler: :get
16
18
  },
17
19
  "tags_create" => {
18
- description: "Create a tag. Color is auto-assigned.",
20
+ description: "Create a tag for labelling prompts, datasets, metrics, metric groups and runs. The name is trimmed and lowercased. It may contain only the letters a to z, digits, spaces, hyphens and underscores (accented and non-Latin letters are rejected), can be up to 64 characters, and must not match an existing tag. Color is auto-assigned. Returns the new tag. You don't need this tool before tagging a record, because the create and update tools for those records take tag_names and create missing tags for you.",
21
+ annotations: Base::ADDITIVE,
19
22
  inputSchema: {
20
23
  type: "object",
21
24
  properties: {name: {type: "string"}},
@@ -24,7 +27,8 @@ module CompletionKit
24
27
  handler: :create
25
28
  },
26
29
  "tags_update" => {
27
- description: "Rename a tag.",
30
+ description: "Rename a tag. Pass the new name; leaving it out returns an error. The name follows the same rules as tags_create and can't match another tag, because tags are not merged. The new name shows at once on every prompt, dataset, metric, metric group and run that carries the tag. Returns the updated tag.",
31
+ annotations: Base::DESTRUCTIVE,
28
32
  inputSchema: {
29
33
  type: "object",
30
34
  properties: {id: {type: "integer"}, name: {type: "string"}},
@@ -33,7 +37,8 @@ module CompletionKit
33
37
  handler: :update
34
38
  },
35
39
  "tags_delete" => {
36
- description: "Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.",
40
+ description: "Delete a tag. Removes the tag from every linked metric, metric group, prompt, run, and dataset; those records themselves are kept. This cannot be undone. Use tags_update instead if you only want to rename it. Returns a confirmation message.",
41
+ annotations: Base::DESTRUCTIVE,
37
42
  inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
38
43
  handler: :delete
39
44
  }
@@ -372,11 +372,9 @@
372
372
  <div class="ck-api-endpoint" style="padding-top: 1rem;">
373
373
  <p class="ck-kicker" style="margin-bottom: 0.5rem;">MCP tools</p>
374
374
  <div class="ck-mcp-tools">
375
- <div class="ck-mcp-tool"><code class="ck-mcp-tool__name">tags_list</code><span class="ck-mcp-tool__desc">List all tags</span></div>
376
- <div class="ck-mcp-tool"><code class="ck-mcp-tool__name">tags_get</code><span class="ck-mcp-tool__desc">Get a tag by ID</span></div>
377
- <div class="ck-mcp-tool"><code class="ck-mcp-tool__name">tags_create</code><span class="ck-mcp-tool__desc">Create a tag (name required)</span></div>
378
- <div class="ck-mcp-tool"><code class="ck-mcp-tool__name">tags_update</code><span class="ck-mcp-tool__desc">Update a tag's name</span></div>
379
- <div class="ck-mcp-tool"><code class="ck-mcp-tool__name">tags_delete</code><span class="ck-mcp-tool__desc">Delete a tag and remove all its taggings</span></div>
375
+ <% CompletionKit::McpTools::Tags.definitions.each do |tool| %>
376
+ <div class="ck-mcp-tool"><code class="ck-mcp-tool__name"><%= tool[:name] %></code><span class="ck-mcp-tool__desc"><%= tool[:description] %></span></div>
377
+ <% end %>
380
378
  </div>
381
379
  <p class="ck-meta-copy" style="margin-top: 0.75rem;">The existing <code>metrics_create</code>, <code>metrics_update</code>, <code>prompts_create</code>, <code>prompts_update</code>, <code>runs_create</code>, <code>runs_update</code>, <code>datasets_create</code>, <code>datasets_create_from_url</code>, <code>datasets_update</code>, <code>metric_groups_create</code>, and <code>metric_groups_update</code> tools all accept a <code>tag_names</code> parameter with the same auto-create and replace semantics as the REST API.</p>
382
380
  </div>
@@ -2,7 +2,7 @@
2
2
  <% pc_delete_id = "ck_delete_provider_#{provider_credential.id}" %>
3
3
  <% pc_delete_confirm = "Delete the #{provider_credential.display_provider} provider and its discovered models? This can't be undone." %>
4
4
  <% end %>
5
- <%= form_with(model: provider_credential, local: true) do |form| %>
5
+ <%= form_with(model: provider_credential, url: local_assigns[:url], local: true) do |form| %>
6
6
  <% if provider_credential.errors.any? %>
7
7
  <div class="ck-flash ck-flash--alert" role="alert">
8
8
  <p class="ck-flash__title"><%= pluralize(provider_credential.errors.count, "problem") %> prevented this provider credential from being saved.</p>
@@ -66,7 +66,9 @@
66
66
  <% if provider_credential.persisted? %>
67
67
  <%= ck_delete_trigger(form_id: pc_delete_id, label: "Delete provider", confirm: pc_delete_confirm) %>
68
68
  <% end %>
69
- <%= link_to "Cancel", provider_credentials_path, class: ck_button_classes(:light, variant: :outline), tabindex: "0" %>
69
+ <% unless local_assigns[:cancel_path] == false %>
70
+ <%= link_to "Cancel", local_assigns[:cancel_path] || provider_credentials_path, class: ck_button_classes(:light, variant: :outline), tabindex: "0" %>
71
+ <% end %>
70
72
  <%= form.submit(provider_credential.persisted? ? "Save provider" : "Create provider", class: ck_button_classes(:dark)) %>
71
73
  </div>
72
74
 
@@ -1,3 +1,3 @@
1
1
  module CompletionKit
2
- VERSION = "0.29.3"
2
+ VERSION = "0.30.0"
3
3
  end
@@ -13,7 +13,7 @@ module CompletionKit
13
13
  attr_accessor :tenant_scope, :tenant_scope_columns
14
14
  attr_accessor :api_reference_authentication_partial
15
15
  attr_accessor :runs_display_scope, :runs_display_footer_partial
16
- attr_accessor :api_rate_limit, :web_rate_limit, :max_upload_bytes
16
+ attr_accessor :api_rate_limit, :api_rate_limit_key, :web_rate_limit, :max_upload_bytes
17
17
  attr_accessor :max_template_bytes
18
18
  attr_reader :regex_timeout
19
19
  attr_accessor :allow_loopback_endpoints
@@ -33,6 +33,7 @@ module CompletionKit
33
33
  @medium_quality_threshold = 3
34
34
 
35
35
  @api_rate_limit = 120
36
+ @api_rate_limit_key = ->(request) { request.remote_ip }
36
37
  @web_rate_limit = 300
37
38
  @max_upload_bytes = 25 * 1024 * 1024
38
39
  @max_template_bytes = 1024 * 1024
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: completion-kit
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.29.3
4
+ version: 0.30.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Damien Bastin