completion-kit 0.5.28 → 0.5.30

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e5b7b1f667eb85b17d7ae6302671129a1e68821bffd1a83f667a7336776f5927
-  data.tar.gz: 2f0e3642574d60e69a466bf0fe097c3287a104ba2a049c33afa6462a83dccaf2
+  metadata.gz: 1bc6277357aee1869602b6c290d958c794788d14efc01d0c4caf1529d3567d69
+  data.tar.gz: a1e60f9278e2bc42c4e1bb487e850f7568a3b134b46e8dbc6b6b3d0bea79c469
 SHA512:
-  metadata.gz: 19bc73fa1f422a5bee002c10681966fa263fe8a69f8928bb48d9630d53feca65721cbc81b3b3415c4dac9d60dc67a616dde7ac1d5e3154b2aa136b73e25a7077
-  data.tar.gz: 5ba36b81ebc2a7df4421423dc8614c624c73f8dde0477bf4a579e914a36ed7c36821840f8c2aed963eb04a50cf840db14119a81a794a251576026731648e49d8
+  metadata.gz: fb4ffbbdb89589e0eca37f995ebc342bd6a6063160ca00cffa25abc956532a2b34aa7e6ce7b7c852802bcf5ed74cc4c49cb7f817ab36dbe07608bf18dd52dc1d
+  data.tar.gz: 61ef079b27c73bb749778e182052ac9c1e6824f1d432635210b647f716f10c5364f6ff8ec9b45cbc274d00b3d93d58b37613c61953596fd7ccf6fe89df1b88f0

data/README.md

@@ -18,19 +18,19 @@ It's the difference between "this prompt seems to work" and "this prompt scores
 
 > **Just want to use it?** [CompletionKit Cloud](https://completionkit.com) is the same engine, fully hosted — zero install, no Rails ops, plans at [completionkit.com/pricing](https://completionkit.com/pricing).
 
-![Test run with scored results](https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/screenshots/test-run.png)
+![The CompletionKit dashboard — workspace totals, run activity over the last 14 days, the worst-scoring metric, version-over-version score changes, and recent runs](https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/screenshots/dashboard.png)
 
 ## Three ways to run it
 
-Same engine, same UI, same REST API and MCP server — pick the deployment that fits.
+Same engine, same UI, same REST API and MCP server — pick the deployment that fits. The first two are stack-agnostic: you run CompletionKit as a product and talk to it over HTTP and MCP, whatever language your own app is written in. The third is for teams already building on Rails.
 
 ### 1. Hosted — [completionkit.com](https://completionkit.com) (recommended)
 
 The fastest path. Sign up and you're running on the same engine you'd self-host, without touching a Rails app. No `db:migrate`, no Puma, no Solid Queue, no provider key management — multi-tenant workspaces, your team logs in, you go. Plans at [completionkit.com/pricing](https://completionkit.com/pricing).
 
-### 2. Self-hosted — the bundled standalone Rails app
+### 2. Self-hosted — the bundled standalone app
 
-Run it on your own infra. No existing Rails app required; Postgres + any Rails-friendly host (Fly, Render, Heroku, Docker, …).
+Run it on your own infra as a self-contained product. There's nothing to integrate and no Ruby to write — once it's up, you drive everything through the web UI, the REST API, and the MCP server, from whatever stack your own app is built in. It needs Postgres and a host that can run the app (Fly, Render, Heroku, Docker, …).
 
 ```bash
 git clone https://github.com/homemade-software-inc/completion-kit.git
@@ -111,7 +111,7 @@ Or add them to `config/credentials.yml.enc` under `active_record_encryption`. In
 
 ## Authentication
 
-CompletionKit requires authentication in production. In development, routes are open by default (with a log warning).
+CompletionKit requires authentication in any deployed environment. In development and test, routes are open by default (with a log warning); every other environment returns 403 until auth is configured.
 
 ### Basic Auth (recommended for simple setups)
 
@@ -132,8 +132,27 @@ end
 
 Only one mode can be active.
 
+## Rate limiting
+
+The REST API, the MCP endpoint, and the web UI are rate limited per IP, per minute. The defaults are generous; tune them in the initializer:
+
+```ruby
+CompletionKit.configure do |c|
+  c.api_rate_limit = 120  # REST API + MCP, requests per minute (default 120)
+  c.web_rate_limit = 300  # web UI, requests per minute (default 300)
+end
+```
+
+Limiting uses `Rails.cache`. A shared cache store (Solid Cache, Redis) throttles accurately across multiple app instances; a per-process store still throttles each instance independently.
+
 ## How it works
 
+<p align="center">
+  <img src="https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/diagrams/workflow.png" alt="CompletionKit workflow: a prompt and a dataset feed a run against a model, an LLM judge scores each output on your rubric, low scores drive an AI-suggested rewrite, and the new prompt version re-runs so you can compare" width="820" />
+</p>
+
+It's a loop. Each pass leaves you with a score you can compare against the last one.
+
 1. **Create a prompt** with `{{variable}}` placeholders
 2. **Upload a dataset.** A CSV where column headers match the variable names.
 3. **Run it** against a model and score outputs with an LLM judge against criteria you define.
@@ -204,6 +223,37 @@ bin/rails db:migrate
 git add db/migrate/ && git commit -m "install new engine migration"
 ```
 
+### Docker
+
+The standalone app ships a `Dockerfile`, so you can self-host it without a Ruby toolchain on the host. Build with the **repository root** as the context — the app depends on the engine source alongside it:
+
+```bash
+docker build -f standalone/Dockerfile -t completion-kit .
+```
+
+CompletionKit needs a Rails secret (`SECRET_KEY_BASE`) and three Active Record encryption keys. With Docker there's no Rails toolchain on the host to run `bin/rails db:encryption:init`, so generate them with `openssl`. Generate them **once** and keep them stable — if the encryption keys change, provider credentials already stored in the database can no longer be decrypted. Write everything to an env file:
+
+```bash
+cat > completion-kit.env <<EOF
+DATABASE_URL=postgres://user:pass@host/completionkit
+SECRET_KEY_BASE=$(openssl rand -hex 64)
+COMPLETION_KIT_ENCRYPTION_PRIMARY_KEY=$(openssl rand -hex 32)
+COMPLETION_KIT_ENCRYPTION_DETERMINISTIC_KEY=$(openssl rand -hex 32)
+COMPLETION_KIT_ENCRYPTION_KEY_DERIVATION_SALT=$(openssl rand -hex 32)
+EOF
+```
+
+`openssl rand` runs as the file is written, so each line gets a real random value. Keep `completion-kit.env` out of version control and back it up somewhere safe.
+
+Run the web process and a job worker from the same image, both pointed at that file:
+
+```bash
+docker run -d -p 3000:3000 --env-file completion-kit.env completion-kit
+docker run -d --env-file completion-kit.env completion-kit ./bin/jobs
+```
+
+Both processes must share the same `SECRET_KEY_BASE` and encryption keys — the single env file guarantees that. The web container runs `db:prepare` on boot, so migrations apply on first start and on every deploy.
+
 ## Multi-tenant host apps (advanced)
 
 For hosts that mount CompletionKit in a multi-tenant app, two optional hooks scope engine records per tenant without forking the engine:

data/app/assets/javascripts/completion_kit/application.js

@@ -158,3 +158,9 @@ document.addEventListener("turbo:before-stream-render", function(event) {
     if (status) { status.textContent = 'Models updated.'; setTimeout(function() { status.textContent = ' '; }, 3000); }
   }
 });
+
+document.addEventListener("click", function(e) {
+  document.querySelectorAll("details.ck-nav-menu[open], details.ck-settings-menu[open], details.ck-flyout[open]").forEach(function(menu) {
+    if (!menu.contains(e.target)) menu.removeAttribute("open");
+  });
+});

data/app/assets/stylesheets/completion_kit/application.css

@@ -4345,6 +4345,16 @@ a.tag-mark {
 .ck-settings-menu__trigger::-webkit-details-marker {
   display: none;
 }
+.ck-settings-menu__trigger svg {
+  display: block;
+  width: 18px;
+  height: 18px;
+}
+.ck-settings-menu__trigger:focus-visible {
+  outline: none;
+  border-color: var(--ck-accent);
+  color: var(--ck-accent);
+}
 .ck-settings-menu__panel {
   position: absolute;
   right: 0;
@@ -4374,6 +4384,16 @@ a.tag-mark {
 .ck-settings-menu__item:hover {
   background: var(--ck-surface-hover);
 }
+.ck-settings-menu__panel form {
+  margin: 0;
+}
+.ck-settings-menu__panel button.ck-settings-menu__item {
+  width: 100%;
+  text-align: left;
+  background: none;
+  border: 0;
+  cursor: pointer;
+}
 
 /* Settings page kicker — small label above the page title on settings pages */
 .ck-settings-kicker {
@@ -4961,3 +4981,71 @@ a.tag-mark {
     max-height: 60vh;
   }
 }
+
+.ck-concept {
+  position: relative;
+  display: inline;
+}
+.ck-concept__toggle {
+  margin: 0 0 0 0.15rem;
+  padding: 0;
+  border: 0;
+  background: none;
+  font: inherit;
+  line-height: 0;
+  color: var(--ck-dim);
+  cursor: pointer;
+}
+.ck-concept__toggle:hover,
+.ck-concept__toggle:focus {
+  color: var(--ck-accent);
+}
+.ck-concept__icon {
+  width: 0.92em;
+  height: 0.92em;
+  vertical-align: -0.12em;
+}
+.ck-concept__pop {
+  display: none;
+  position: absolute;
+  top: calc(100% + 0.4rem);
+  left: 0;
+  z-index: 40;
+  width: 19rem;
+  max-width: calc(100vw - 1.5rem);
+  padding: 0.7rem 0.8rem;
+  background: var(--ck-bg-strong);
+  border: 1px solid var(--ck-line-strong);
+  border-radius: var(--ck-radius);
+  box-shadow: 0 16px 34px rgba(0, 0, 0, 0.5);
+  white-space: normal;
+}
+.ck-concept__toggle:hover + .ck-concept__pop,
+.ck-concept__toggle:focus + .ck-concept__pop,
+.ck-concept__pop:hover {
+  display: block;
+}
+.ck-concept__name {
+  display: block;
+  margin-bottom: 0.3rem;
+  font-family: var(--ck-mono);
+  font-size: 0.72rem;
+  font-weight: 700;
+  letter-spacing: 0.06em;
+  text-transform: uppercase;
+  color: var(--ck-text);
+}
+.ck-concept__body {
+  display: block;
+  font-size: 0.8rem;
+  line-height: 1.5;
+  color: var(--ck-muted);
+}
+@media (max-width: 640px) {
+  .ck-concept__pop {
+    position: fixed;
+    inset: auto 0.75rem 0.75rem 0.75rem;
+    width: auto;
+    max-width: none;
+  }
+}

data/app/controllers/completion_kit/api/v1/base_controller.rb

@@ -2,6 +2,8 @@ module CompletionKit
   module Api
     module V1
       class BaseController < ActionController::API
+        rate_limit to: CompletionKit.config.api_rate_limit, within: 1.minute,
+                   with: -> { render json: {error: "Rate limit exceeded"}, status: :too_many_requests }
         before_action :authenticate_api!
 
         private

data/app/controllers/completion_kit/application_controller.rb

@@ -3,6 +3,8 @@ module CompletionKit
     helper Heroicons::IconsHelper
     layout "completion_kit/application"
 
+    rate_limit to: CompletionKit.config.web_rate_limit, within: 1.minute,
+               with: -> { render plain: "Rate limit exceeded. Please slow down.", status: :too_many_requests }
     before_action :authenticate_completion_kit!
 
     private

data/app/views/layouts/completion_kit/application.html.erb

@@ -27,21 +27,21 @@
         <%= link_to "Metrics", metrics_path, class: request.path.start_with?(metrics_path) || request.path.start_with?(metric_groups_path) ? ck_button_classes(:dark) : ck_button_classes(:light, variant: :outline) %>
         <%= link_to "Datasets", datasets_path, class: active.(datasets_path) %>
         <%= link_to "Runs", runs_path, class: active.(runs_path) %>
-        <% settings_active = request.path.start_with?(provider_credentials_path) || request.path.start_with?(tags_path) || request.path.start_with?(onboarding_path) %>
+        <% settings_active = request.path.start_with?(provider_credentials_path) || request.path.start_with?(tags_path) || request.path.start_with?(onboarding_path) || request.path.start_with?(api_reference_path) %>
         <details class="ck-settings-menu">
           <summary class="<%= settings_active ? ck_button_classes(:dark) : ck_button_classes(:light, variant: :outline) %> ck-settings-menu__trigger" aria-label="Settings">
-            Settings
+            <%= heroicon_tag "cog-6-tooth", variant: :outline, size: 18, "aria-hidden": "true" %>
           </summary>
           <div class="ck-settings-menu__panel" role="menu">
+            <%= link_to "Getting started", onboarding_path(reset: 1), class: "ck-settings-menu__item" %>
+            <%= link_to "API", api_reference_path, class: "ck-settings-menu__item" %>
             <%= link_to "Providers", provider_credentials_path, class: "ck-settings-menu__item" %>
             <%= link_to "Tags", tags_path, class: "ck-settings-menu__item" %>
-            <%= link_to "Getting started", onboarding_path(reset: 1), class: "ck-settings-menu__item" %>
+            <% if main_app.respond_to?(:logout_path) %>
+              <%= button_to "Sign out", main_app.logout_path, method: :delete, class: "ck-settings-menu__item" %>
+            <% end %>
           </div>
         </details>
-        <%= link_to "API", api_reference_path, class: active.(api_reference_path) %>
-        <% if main_app.respond_to?(:logout_path) %>
-          <%= button_to "Log out", main_app.logout_path, method: :delete, class: ck_button_classes(:light, variant: :outline) %>
-        <% end %>
       </nav>
       </details>
     </div>

data/lib/completion_kit/version.rb

@@ -1,3 +1,3 @@
 module CompletionKit
-  VERSION = "0.5.28"
+  VERSION = "0.5.30"
 end

data/lib/completion_kit.rb

@@ -10,6 +10,7 @@ module CompletionKit
     attr_accessor :username, :password, :auth_strategy, :api_token
     attr_accessor :tenant_scope, :tenant_scope_columns
     attr_accessor :api_reference_authentication_partial
+    attr_accessor :api_rate_limit, :web_rate_limit
 
     def initialize
       @openai_api_key = ENV['OPENAI_API_KEY']
@@ -21,6 +22,9 @@ module CompletionKit
       @high_quality_threshold = 4
       @medium_quality_threshold = 3
 
+      @api_rate_limit = 120
+      @web_rate_limit = 300
+
       @api_reference_authentication_partial = "completion_kit/api_reference/authentication"
     end
 

data/lib/generators/completion_kit/templates/initializer.rb

@@ -44,4 +44,10 @@ CompletionKit.configure do |config|
   # Web UI Authentication
   # config.username = "admin"
   # config.password = ENV['COMPLETION_KIT_PASSWORD']
+
+  # Rate limiting (requests per minute, per IP)
+  # The REST API and MCP endpoint share api_rate_limit; the web UI uses
+  # web_rate_limit. Both are enforced through Rails.cache.
+  # config.api_rate_limit = 120
+  # config.web_rate_limit = 300
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: completion-kit
 version: !ruby/object:Gem::Version
-  version: 0.5.28
+  version: 0.5.30
 platform: ruby
 authors:
 - Damien Bastin