completion-kit 0.5.9 → 0.5.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2e1641413d1ed8d27bb6344094b788f33ab521e4d738bf1468fba148185a58f6
-  data.tar.gz: 60b3676c0b8100430a7841a6281b5656184198583b40565c0ef5f7b24663ebc4
+  metadata.gz: ed531ae29162bb91d2c463c3ff4eb20b5da469b9b7a21baddf5054a0ccc15041
+  data.tar.gz: b86aea95b2e1cf73abf6514093565dc07b12dc0f4fe5c5c5c8b80db3fbdfa83d
 SHA512:
-  metadata.gz: 003f3af4e5eaa28bc5c9b800e3788a63134a8bc9750d170ed03c6c1707b2e57675c79e25994abd763fb133eaec484f07133e12a74be9179f90c360b1d290dafd
-  data.tar.gz: 6b73b15e6c1eb9af3d8f5228997553a7489db9c948a2e7298d2c9b40eb19b27a431af84b52368716bac0705f4c62acc2b5b7a8f937192445416b131c59b661f9
+  metadata.gz: 04ae500020e71d52c41073c36a6741bc47b94a06ceec6548720d6022a60ce7422be8a354d627c39a7be8174af2ce65219041c5d99ad175157c7bf4b4eaf8f056
+  data.tar.gz: 261daeeb1555b3aecb8e2e18edb7f14ebdc37f974c2713ecbe12c43a281e8109edc45222be0f6039c325a0428e89c1c11a1a7104f0a36ace2b618fb2ef1cb7e8

data/README.md

@@ -14,21 +14,23 @@ Run every prompt against real data. Score each output with an LLM judge against
 
 It's the difference between "this prompt seems to work" and "this prompt scores 4.3 out of 5 across 200 inputs, up from 3.8 last version."
 
-**[completionkit.com](https://completionkit.com)** | **[RubyGems](https://rubygems.org/gems/completion-kit)**
+**[Start on completionkit.com →](https://completionkit.com)** | **[RubyGems](https://rubygems.org/gems/completion-kit)**
 
-> **CompletionKit Cloud** — hosted, managed CompletionKit with zero setup. Same engine, run for you. See plans at [completionkit.com/pricing](https://completionkit.com/pricing).
+> **Just want to use it?** [CompletionKit Cloud](https://completionkit.com) is the same engine, fully hosted — zero install, no Rails ops, plans at [completionkit.com/pricing](https://completionkit.com/pricing).
 
 ![Test run with scored results](https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/screenshots/test-run.png)
 
-## Quick Start
+## Three ways to run it
 
-### Use CompletionKit Cloud
+Same engine, same UI, same REST API and MCP server — pick the deployment that fits.
 
-The fastest way to start — no install, no servers to run. Sign up at [completionkit.com](https://completionkit.com) and you get the same engine you'd self-host, hosted for you. Best fit if you want to skip the Rails ops.
+### 1. Hosted — [completionkit.com](https://completionkit.com) (recommended)
 
-### Or run the standalone app
+The fastest path. Sign up and you're running on the same engine you'd self-host, without touching a Rails app. No `db:migrate`, no Puma, no Solid Queue, no provider key management — multi-tenant workspaces, your team logs in, you go. Plans at [completionkit.com/pricing](https://completionkit.com/pricing).
 
-Self-host the same engine. No existing Rails app needed.
+### 2. Self-hosted — the bundled standalone Rails app
+
+Run it on your own infra. No existing Rails app required; Postgres + any Rails-friendly host (Fly, Render, Heroku, Docker, …).
 
 ```bash
 git clone https://github.com/homemade-software-inc/completion-kit.git
@@ -38,7 +40,7 @@ bin/rails completion_kit:install:migrations
 bin/rails db:migrate
 ```
 
-Then run **both** processes — a web server and a Solid Queue worker. In two terminals:
+Run **both** a web server and a Solid Queue worker. In two terminals:
 
 ```bash
 bin/rails server
@@ -50,9 +52,9 @@ bin/jobs
 
 Or with [foreman](https://github.com/ddollar/foreman) in one terminal: `foreman start -f Procfile.dev`.
 
-Visit `http://localhost:3000`. Add a provider credential (Settings), create a prompt, upload a CSV dataset, and run it.
+Visit `http://localhost:3000`. Add a provider credential (Settings), create a prompt, upload a CSV dataset, and run it. See [Deploying self-hosted](#deploying-self-hosted) for the production-env setup.
 
-### Or mount as an engine in your existing Rails app
+### 3. Rails engine — mount into your existing Rails app
 
 ```ruby
 gem "completion-kit"
@@ -63,11 +65,9 @@ bin/rails generate completion_kit:install
 bin/rails db:migrate
 ```
 
-The engine mounts at `/completion_kit` in your app. CompletionKit's generate and judge flows enqueue Active Job jobs (`CompletionKit::GenerateRowJob`, `CompletionKit::JudgeReviewJob`, `CompletionKit::RunCompletionCheckJob`), so your host app needs an Active Job adapter that actually processes them — Solid Queue, Sidekiq, GoodJob, etc. The `:async` adapter is **not** suitable for production: it runs jobs in the web Puma's thread pool with no durability and no retry, and a long LLM call will block request handling.
-
-### Host-app layout integration
+The engine mounts at `/completion_kit`. Generate / judge flows enqueue Active Job jobs (`CompletionKit::GenerateRowJob`, `CompletionKit::JudgeReviewJob`, `CompletionKit::RunCompletionCheckJob`), so your host app needs an Active Job adapter that actually processes them — Solid Queue, Sidekiq, GoodJob, etc. The `:async` adapter is **not** suitable for production: it runs jobs in the web Puma's thread pool with no durability and no retry, and a long LLM call will block request handling.
 
-If your host app overrides the engine layout (e.g. `layout "application"` on engine controllers, or rendering engine views inside your own shell), include both the engine's stylesheet and JavaScript in that layout:
+**Host-app layout integration.** If your host app overrides the engine layout (e.g. `layout "application"` on engine controllers, or rendering engine views inside your own shell), include both the engine's stylesheet and JavaScript in that layout:
 
 ```erb
 <%= stylesheet_link_tag "completion_kit/application", media: "all" %>
@@ -183,7 +183,7 @@ CompletionKit runs a [Model Context Protocol](https://modelcontextprotocol.io) s
 
 The in-app API reference page has install snippets you can copy straight into your MCP client config.
 
-## Deploying the standalone app
+## Deploying self-hosted
 
 Any Rails-friendly host works (Fly, Heroku, Render, Docker, etc.). Point it at a Postgres instance via `DATABASE_URL`, set your provider env vars, and run `cd standalone && bin/rails db:migrate` on each deploy.
 

data/app/assets/stylesheets/completion_kit/application.css

@@ -126,13 +126,21 @@ form.button_to {
   font-weight: 700;
   letter-spacing: 0.02em;
   text-decoration: none;
-  color: var(--ck-accent);
+  color: #3AD0E6;
 }
 
 .ck-brand img {
   display: block;
 }
 
+.ck-brand__name {
+  padding-top: 0.75rem;
+}
+
+.ck-brand__kit {
+  color: #AFEDF7;
+}
+
 .ck-topbar__copy {
   display: none;
 }
@@ -262,7 +270,7 @@ form.button_to {
 .ck-meta-copy,
 .ck-note,
 .ck-hint {
-  font-size: 0.95rem;
+  font-size: 0.9rem;
   line-height: 1.6;
 }
 
@@ -1269,6 +1277,13 @@ tr:hover .ck-chip--publish {
   color: var(--ck-accent);
 }
 
+/* the main prompt template block on prompts/show — bigger padding + a
+   touch more line-height since this is the page's primary content */
+.ck-code--prompt {
+  padding: 1.5rem;
+  line-height: 1.75;
+}
+
 .ck-note-box {
   background: var(--ck-surface-soft);
   border: 1px solid var(--ck-line);
@@ -1522,6 +1537,9 @@ tr:hover .ck-chip--publish {
   display: grid;
   gap: 0.4rem;
 }
+.ck-field[hidden] {
+  display: none;
+}
 
 .ck-field--spacious {
   margin-top: 0.3rem;
@@ -1855,7 +1873,18 @@ tr:hover .ck-chip--publish {
   background: var(--ck-bg-strong);
   overflow: auto;
   max-height: 60vh;
+  scrollbar-width: thin;
+  scrollbar-color: var(--ck-line-strong) transparent;
 }
+.ck-csv-table-wrap::-webkit-scrollbar { width: 10px; height: 10px; }
+.ck-csv-table-wrap::-webkit-scrollbar-track { background: transparent; }
+.ck-csv-table-wrap::-webkit-scrollbar-thumb {
+  background: var(--ck-line-strong);
+  border-radius: 5px;
+  border: 2px solid var(--ck-bg-strong);
+}
+.ck-csv-table-wrap::-webkit-scrollbar-thumb:hover { background: var(--ck-muted); }
+.ck-csv-table-wrap::-webkit-scrollbar-corner { background: transparent; }
 
 .ck-modal__body .ck-csv-table-wrap {
   margin-top: 0;
@@ -2763,8 +2792,10 @@ select.ck-input {
 
 /* the metrics field stacks several sub-sections (hint, groups, divider, tag
    filter, checkboxes) — give it more vertical breathing room than a plain field,
-   and extra separation from the run-tags field that follows it */
-#metrics-field {
+   and extra separation from the run-tags field that follows it. Only when the
+   checkboxes are actually present, though — when there are no metrics the field
+   is just "label + warning" and should be a normal compact field. */
+#metrics-field:has(.ck-metric-checkboxes) {
   gap: 0.85rem;
   margin-bottom: 1.25rem;
 }
@@ -3571,7 +3602,7 @@ a.ck-metric-group-pill {
 }
 
 .ck-mcp-tool__desc {
-  font-size: 0.8rem;
+  font-size: 0.9rem;
   color: var(--ck-muted);
 }
 
@@ -3595,7 +3626,7 @@ a.ck-metric-group-pill {
   gap: 0.5rem;
   padding: 0.6rem 0.85rem;
   font-family: var(--ck-mono);
-  font-size: 0.78rem;
+  font-size: 0.9rem;
   font-weight: 500;
   color: var(--ck-text);
   border-bottom: 1px solid var(--ck-line);
@@ -3669,7 +3700,7 @@ a.ck-metric-group-pill {
 }
 
 .ck-api-prompt-card__desc {
-  font-size: 0.78rem;
+  font-size: 0.9rem;
   color: var(--ck-muted);
   margin: 0.2rem 0 0;
   line-height: 1.4;

data/app/controllers/completion_kit/api/v1/runs_controller.rb

@@ -76,7 +76,7 @@ module CompletionKit
         end
 
         def run_params
-          params.permit(:name, :prompt_id, :dataset_id, :judge_model, :temperature,
+          params.permit(:name, :prompt_id, :dataset_id, :judge_model, :temperature, :output_column,
             metric_ids: [], tag_names: [])
         end
       end

data/app/controllers/completion_kit/api_reference_controller.rb

@@ -2,6 +2,12 @@ module CompletionKit
   class ApiReferenceController < ApplicationController
     def index
       @published_prompts = Prompt.current_versions.order(name: :asc)
+      @recent_runs = Run.includes(:prompt).order(created_at: :desc).limit(10)
+      @datasets = Dataset.order(name: :asc)
+      @metrics = Metric.order(name: :asc)
+      @metric_groups = MetricGroup.includes(:metrics).order(name: :asc)
+      @tags = Tag.order(name: :asc)
+      @provider_credentials = ProviderCredential.order(:provider)
       @token = CompletionKit.config.api_token
       @base_url = request.base_url + request.script_name
     end

data/app/controllers/completion_kit/datasets_controller.rb

@@ -9,6 +9,16 @@ module CompletionKit
 
     def show
       @runs = @dataset.runs.includes(:prompt, :responses).order(created_at: :desc)
+      respond_to do |format|
+        format.html
+        format.csv do
+          slug = @dataset.name.to_s.parameterize.presence || "dataset-#{@dataset.id}"
+          send_data @dataset.csv_data.to_s,
+                    type: "text/csv",
+                    filename: "#{slug}.csv",
+                    disposition: "attachment"
+        end
+      end
     end
 
     def new

data/app/controllers/completion_kit/mcp_controller.rb

@@ -17,7 +17,7 @@ module CompletionKit
       end
 
       session_id = request.headers["Mcp-Session-Id"]
-      unless session_id && Rails.cache.exist?("mcp_session:#{session_id}")
+      unless McpSession.active?(session_id)
         render json: jsonrpc_error(request_body["id"], -32000, "Session not initialized. Send initialize first."), status: :bad_request
         return
       end
@@ -40,7 +40,7 @@ module CompletionKit
 
     def destroy
       session_id = request.headers["Mcp-Session-Id"]
-      Rails.cache.delete("mcp_session:#{session_id}") if session_id
+      McpSession.destroy_session(session_id) if session_id
       head :ok
     end
 

data/app/controllers/completion_kit/runs_controller.rb

@@ -84,6 +84,7 @@ module CompletionKit
         dataset_id: @run.dataset_id,
         judge_model: @run.judge_model,
         temperature: @run.temperature,
+        output_column: @run.output_column,
         tag_names: @run.tag_names,
         status: "pending"
       )
@@ -108,6 +109,11 @@ module CompletionKit
     end
 
     def suggest
+      if @run.prompt.nil?
+        redirect_to run_path(@run), alert: "Judge-only runs don't have a prompt to improve."
+        return
+      end
+
       service = PromptImprovementService.new(@run)
       result = service.suggest
       suggestion = @run.suggestions.create!(
@@ -159,13 +165,13 @@ module CompletionKit
     end
 
     def run_params
-      params.require(:run).permit(:name, :prompt_id, :dataset_id, :judge_model, :temperature, metric_ids: [], tag_names: [])
+      params.require(:run).permit(:name, :prompt_id, :dataset_id, :judge_model, :temperature, :output_column, metric_ids: [], tag_names: [])
     end
 
     # Editing a run that already has results forks a new run — but only when a
     # field that affects generation or judging changed. Renaming or retagging is
     # pure metadata and updates the run in place.
-    GENERATION_RUN_FIELDS = %i[prompt_id dataset_id judge_model temperature].freeze
+    GENERATION_RUN_FIELDS = %i[prompt_id dataset_id judge_model temperature output_column].freeze
 
     def run_generation_changed?
       GENERATION_RUN_FIELDS.each do |field|

data/app/jobs/completion_kit/judge_review_job.rb

@@ -54,7 +54,7 @@ module CompletionKit
       evaluation = judge.evaluate(
         response.response_text,
         response.expected_output,
-        run.prompt.template,
+        run.prompt&.template,
         criteria: metric.instruction.to_s,
         rubric_text: metric.display_rubric_text,
         input_data: response.input_data

data/app/models/completion_kit/mcp_session.rb

@@ -0,0 +1,29 @@
+module CompletionKit
+  # MCP session marker — one row per active client session, kept in the
+  # database so sessions survive Puma restarts, deploys, and Rails.cache
+  # eviction. Expired rows are opportunistically pruned on every new
+  # session start, so the table stays bounded by recent activity.
+  class McpSession < ApplicationRecord
+    self.table_name = "completion_kit_mcp_sessions"
+
+    SESSION_TTL = 1.hour
+
+    def self.start!
+      prune_expired!
+      create!(session_id: SecureRandom.uuid, expires_at: SESSION_TTL.from_now).session_id
+    end
+
+    def self.active?(session_id)
+      return false if session_id.blank?
+      where(session_id: session_id).where("expires_at > ?", Time.current).exists?
+    end
+
+    def self.destroy_session(session_id)
+      where(session_id: session_id).delete_all
+    end
+
+    def self.prune_expired!
+      where("expires_at < ?", Time.current).delete_all
+    end
+  end
+end

data/app/models/completion_kit/run.rb

@@ -5,7 +5,7 @@ module CompletionKit
 
     STATUSES = %w[pending running completed failed].freeze
 
-    belongs_to :prompt
+    belongs_to :prompt, optional: true
     belongs_to :dataset, optional: true
     has_many :responses, dependent: :destroy
     has_many :run_metrics, -> { order(:position) }, dependent: :destroy
@@ -15,10 +15,18 @@ module CompletionKit
     validates :name, presence: true
     validates :status, inclusion: { in: STATUSES }
     validate :dataset_supplies_prompt_variables
+    validate :judge_only_run_supplies_output_column
 
     before_validation :set_default_status, on: :create
     before_validation :set_auto_name, on: :create
 
+    # A judge-only run grades a pre-existing column on the dataset instead of
+    # generating new outputs. No prompt is attached; the response text is read
+    # from row[output_column]; no LLM generation happens.
+    def judge_only?
+      prompt.nil?
+    end
+
     def missing_dataset_variables
       return [] unless prompt
       vars = prompt.variables
@@ -89,9 +97,14 @@ module CompletionKit
 
       return fail_with_summary!("Dataset has no rows") if rows.empty?
 
-      client = LlmClient.for_model(prompt.llm_model, ApiConfig.for_model(prompt.llm_model))
-      unless client.configured?
-        return fail_with_summary!("LLM API not configured: #{client.configuration_errors.join(', ')}")
+      if judge_only?
+        column = output_column.presence || "actual_output"
+        return fail_with_summary!("Dataset has no \"#{column}\" column") unless dataset && dataset.headers.include?(column)
+      else
+        client = LlmClient.for_model(prompt.llm_model, ApiConfig.for_model(prompt.llm_model))
+        unless client.configured?
+          return fail_with_summary!("LLM API not configured: #{client.configuration_errors.join(', ')}")
+        end
       end
 
       transaction do
@@ -105,14 +118,27 @@ module CompletionKit
         )
         rows.each_with_index do |row, index|
           input = row.empty? ? nil : row.to_json
-          response = responses.create!(
+          attrs = {
             status: "pending",
             row_index: index,
             input_data: input,
             expected_output: row["expected_output"]
-          )
-          GenerateRowJob.perform_later(id, response.id)
+          }
+          if judge_only?
+            attrs[:status] = "succeeded"
+            attrs[:response_text] = row[output_column.presence || "actual_output"].to_s
+          end
+
+          response = responses.create!(attrs)
+
+          if judge_only?
+            metrics.each { |m| JudgeReviewJob.perform_later(response.id, m.id) } if judge_configured?
+          else
+            GenerateRowJob.perform_later(id, response.id)
+          end
         end
+
+        RunCompletionCheckJob.perform_later(id) if judge_only?
       end
 
       broadcast_ui
@@ -168,6 +194,7 @@ module CompletionKit
       {
         id: id, name: name, status: status, prompt_id: prompt_id,
         dataset_id: dataset_id, judge_model: judge_model, temperature: temperature,
+        output_column: output_column,
         created_at: created_at, updated_at: updated_at,
         responses_count: responses.count, avg_score: avg_score,
         progress_current: snap[:generated_done],
@@ -274,10 +301,14 @@ module CompletionKit
 
     def set_auto_name
       return if name.present?
-      return unless prompt.present?
 
-      count = Run.where(prompt_id: prompt_id).count + 1
-      self.name = "#{prompt.name} — v#{prompt.version_number} ##{count}"
+      if prompt.present?
+        count = Run.where(prompt_id: prompt_id).count + 1
+        self.name = "#{prompt.name} — v#{prompt.version_number} ##{count}"
+      elsif dataset.present?
+        count = Run.where(prompt_id: nil, dataset_id: dataset.id).count + 1
+        self.name = "#{dataset.name} — judge-only ##{count}"
+      end
     end
 
     def dataset_supplies_prompt_variables
@@ -290,5 +321,19 @@ module CompletionKit
         errors.add(:dataset_id, "is missing columns required by the prompt: #{missing.join(', ')}")
       end
     end
+
+    def judge_only_run_supplies_output_column
+      return if prompt.present?
+
+      if dataset.nil?
+        errors.add(:dataset_id, "is required for a judge-only run (no prompt)")
+        return
+      end
+
+      column = output_column.presence || "actual_output"
+      unless dataset.headers.include?(column)
+        errors.add(:output_column, "\"#{column}\" is not a column on dataset \"#{dataset.name}\"")
+      end
+    end
   end
 end

data/app/services/completion_kit/mcp_dispatcher.rb

@@ -6,10 +6,8 @@ module CompletionKit
     PROTOCOL_VERSION = "2025-03-26"
 
     def self.initialize_session
-      session_id = SecureRandom.uuid
-      Rails.cache.write("mcp_session:#{session_id}", true, expires_in: 1.hour)
       {
-        session_id: session_id,
+        session_id: McpSession.start!,
         protocolVersion: PROTOCOL_VERSION,
         serverInfo: {name: "CompletionKit", version: CompletionKit::VERSION},
         capabilities: {tools: {listChanged: false}}

data/app/services/completion_kit/mcp_tools/runs.rb

@@ -15,16 +15,17 @@ module CompletionKit
           handler: :get
         },
         "runs_create" => {
-          description: "Create a run",
+          description: "Create a run. Omit prompt_id and provide output_column for a judge-only run that grades a pre-existing dataset column instead of generating new outputs.",
           inputSchema: {
             type: "object",
             properties: {
               name: {type: "string"}, prompt_id: {type: "integer"},
               dataset_id: {type: "integer"}, judge_model: {type: "string"},
+              output_column: {type: "string", description: "Dataset column to grade when prompt_id is omitted; defaults to \"actual_output\"."},
               metric_ids: {type: "array", items: {type: "integer"}},
               tag_names: {type: "array", items: {type: "string"}}
             },
-            required: ["name", "prompt_id"]
+            required: ["name"]
           },
           handler: :create
         },
@@ -35,6 +36,7 @@ module CompletionKit
             properties: {
               id: {type: "integer"}, name: {type: "string"},
               dataset_id: {type: "integer"}, judge_model: {type: "string"},
+              output_column: {type: "string"},
               metric_ids: {type: "array", items: {type: "integer"}},
               tag_names: {type: "array", items: {type: "string"}}
             },
@@ -63,7 +65,7 @@ module CompletionKit
       end
 
       def self.create(args)
-        run = Run.new(args.slice("name", "prompt_id", "dataset_id", "judge_model"))
+        run = Run.new(args.slice("name", "prompt_id", "dataset_id", "judge_model", "output_column"))
         if run.save
           run.replace_metrics!(args["metric_ids"])
           run.update!(tag_names: args["tag_names"]) if args.key?("tag_names")
@@ -75,7 +77,7 @@ module CompletionKit
 
       def self.update(args)
         run = Run.find(args["id"])
-        if run.update(args.except("id", "metric_ids", "tag_names").slice("name", "dataset_id", "judge_model"))
+        if run.update(args.except("id", "metric_ids", "tag_names").slice("name", "dataset_id", "judge_model", "output_column"))
           run.replace_metrics!(args["metric_ids"]) if args.key?("metric_ids")
           run.update!(tag_names: args["tag_names"]) if args.key?("tag_names")
           text_result(run.reload.as_json)