completion-kit 0.28.44 → 0.28.45

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b8caf20114009989549212499117e59591314054fcfd2423dbfc6f1ad9c6d5c5
-  data.tar.gz: 73c7cde193a4041f8eb6b2300b45794458527ee710927ede840898a347e469ba
+  metadata.gz: 7c6a9a433f596def84c665f6b6fbe2b5aa4ed1422ac923f20a2a2d246aad02c4
+  data.tar.gz: 3a0ad110587d51bad3b41f89439da3efa79c199887d58b76703b0d46d16d7f6d
 SHA512:
-  metadata.gz: c944f32bec9a6b4879dd97b364e95e7935274dc9ffa430d8a484f4388c9332dbf83231efa8d1a27f45bbfe2a1c5bba42e4feab3c110a7ef9075b155b55148181
-  data.tar.gz: 1a98282e19cbaaf949a4083f049fdcb1af26ad3c27d7426644eb3eac95cfcfc1b1810de4100ab5d6796bd79c5a1d9c272ea2d6b9473f3e25c36d96ba041afaac
+  metadata.gz: 5f17ea2b059df5b6e2656c9a80a60d87ccd5a55b7d1d2a2599ca3b913227eca49d19a983015765b52accc0287a8e07e60dedb3e1d92d2744e81f5803a32f5b19
+  data.tar.gz: 673b44c3f1eb7edaf963acb85a0451067ed1f6fa9ee73fee156518c031e8fb2b74786aa119c88d39281cc36e4a5e56da3ab8354a3c0411fe1eca800ed96b275e

data/README.md

@@ -133,6 +133,29 @@ end
 
 Only one mode can be active.
 
+### Reviewer name
+
+When someone records a verdict on a judge's score in the web UI, CompletionKit saves it under a reviewer name. Each reviewer keeps one verdict per response and metric, so give every person their own name. Otherwise their verdicts overwrite each other. Set `current_reviewer` to a callable that receives the controller and returns the signed-in person's name:
+
+```ruby
+CompletionKit.configure do |c|
+  c.auth_strategy = ->(controller) { controller.authenticate_user! }
+  c.current_reviewer = ->(controller) { controller.current_user&.email }
+end
+```
+
+The callable must return a String or nil. Any other value, such as a user record, raises `CompletionKit::ConfigurationError`. If `current_reviewer` is unset or returns a blank value, verdicts are saved under `username`, or `"operator"` when no username is set. The web UI does not read the `X-Remote-User` header on its own. If a proxy in front of your app sets that header, pass it through:
+
+```ruby
+CompletionKit.configure do |c|
+  c.current_reviewer = ->(controller) { controller.request.headers["X-Remote-User"] }
+end
+```
+
+Only use this header when a trusted proxy always overwrites it. Otherwise any client can send it and claim any reviewer name.
+
+The REST API and the MCP server take the reviewer name from their own `created_by` parameter.
+
 ## Rate limiting
 
 The REST API, the MCP endpoint, and the web UI are rate limited per IP, per minute. The defaults are generous; tune them in the initializer:
@@ -269,22 +292,24 @@ CompletionKit.configure do |config|
 end
 ```
 
-`tenant_scope` runs as each engine model's `default_scope` (use `unscoped` to bypass). `tenant_scope_columns` is appended to every engine uniqueness validation. Adding the tenant columns and composite unique indexes lives in your host migrations. Both defaults (`nil`, `[]`) are no-ops.
+`tenant_scope` runs as each engine model's `default_scope` (use `unscoped` to bypass). `tenant_scope_columns` is appended to every engine uniqueness validation. Adding the tenant columns and composite unique indexes lives in your host migrations. Both defaults (`nil`, `[]`) are no-ops. The `completion_kit:mark_interrupted_runs_failed` rake task bypasses `tenant_scope` and marks in-flight runs in every tenant, so a host that also enforces tenancy in the database, for example with row-level security, must run it as a database role that can see every tenant's rows.
+
+Custom model endpoints and CSV imports by URL may reach the server's own loopback interface by default, so a local Ollama works out of the box. If the people using the engine should not reach services on the app server itself, set `config.allow_loopback_endpoints = false`. Private, link-local and reserved addresses are always refused.
 
-One hook lets a host apply run-history retention everywhere the engine lists, counts, or traverses runs, without overriding controllers or views:
+One hook lets a host apply its own display rule everywhere the engine lists, counts, or traverses runs, without overriding controllers or views:
 
 ```ruby
 CompletionKit.configure do |config|
   config.runs_display_scope = -> { where(created_at: 90.days.ago..) }
-  config.runs_display_footer_partial = "runs/retention_notice"
+  config.runs_display_footer_partial = "runs/display_notice"
 end
 ```
 
-`runs_display_scope` is a callable evaluated against a `Run` relation, in the same bare-`where` style as `tenant_scope` (it runs via `instance_exec`, so write it zero-arg with the relation as `self`, like a Rails `scope` lambda). It must return a relation: a callable that returns `nil` or anything non-chainable raises when a list renders. The engine applies it through `Run.display_scoped` at every run list and count it owns (the runs index, prompt and dataset show pages, the compare picker, new-run tag defaults, the v1 API index and its `X-Total-Count`, the MCP `runs_list` tool, the dashboard and API reference recent-runs panels, the dashboard "Runs" stat-card count, and provider-credential usage stats) and through `Run.visible_run_ids` for child records that traverse runs (the metric trust-panel sample and the agreement examples shown on a metric page). Use it for list-only retention rather than a global `default_scope`, which would null `Run` associations everywhere they are traversed.
+`runs_display_scope` is a callable evaluated against a `Run` relation, in the same bare-`where` style as `tenant_scope` (it runs via `instance_exec`, so write it zero-arg with the relation as `self`, like a Rails `scope` lambda). It must return a relation: a callable that returns `nil` or anything non-chainable raises when a list renders. The engine applies it through `Run.display_scoped` at every run list and count it owns (the runs index, prompt and dataset show pages, the compare picker, new-run tag defaults, the v1 API index and its `X-Total-Count`, the MCP `runs_list` tool, the dashboard and API reference recent-runs panels, the dashboard "Runs" stat-card count, and provider-credential usage stats) and through `Run.visible_run_ids` for child records that traverse runs (the metric trust-panel sample and the agreement examples shown on a metric page). Use it for a list-only display rule rather than a global `default_scope`, which would null `Run` associations everywhere they are traversed.
 
-Deliberately exempt, because they must still see every run: id-addressed single-run lookups (`runs#show`, the MCP `runs_get` tool, the v1 API show), delete-confirmation cascade counts, the auto-generated run-name counter, the dashboard's stat-card gate (it keys off the unscoped run total so a long-tenured workspace whose runs have all aged past retention still sees its activity cards rather than a blank dashboard), and the judge few-shot seeding that learns from corrected examples even on hidden runs.
+Deliberately exempt, because they must still see every run: id-addressed single-run lookups (`runs#show`, the MCP `runs_get` tool, the v1 API show), delete-confirmation cascade counts, the auto-generated run-name counter, the dashboard's stat-card gate (it keys off the unscoped run total so a workspace whose runs the display scope excludes entirely still sees its activity cards rather than a blank dashboard), and the judge few-shot seeding that learns from corrected examples even on runs the display scope excludes.
 
-`runs_display_footer_partial` names a partial rendered below the runs list on the dashboard, the index, and the prompt and dataset show pages; it receives the shown runs as a `runs` local. Use it for a notice like "older runs are hidden, upgrade to see them" — your host owns the retention rule in `runs_display_scope`, so it computes the hidden count itself. Both default to `nil` (no-ops), leaving standalone behaviour unchanged.
+`runs_display_footer_partial` names a partial rendered below the runs list on the dashboard, the index, and the prompt and dataset show pages; it receives the shown runs as a `runs` local. When the display scope excludes every run one of those surfaces would list, the surface says "No runs to show." and the footer partial renders beneath it. The engine cannot know why your scope excludes runs, so use the partial to explain your display rule: your host owns that rule in `runs_display_scope` and is the only side that can describe or count what it left out. Both default to `nil` (no-ops), leaving standalone behaviour unchanged.
 
 ## Contributing
 

data/app/assets/stylesheets/completion_kit/application.css

@@ -286,6 +286,11 @@ form.button_to {
   margin-top: 0.45rem;
 }
 
+.ck-kicker + .ck-empty,
+.ck-split + .ck-empty {
+  margin-top: 0.5rem;
+}
+
 .ck-pagination {
   display: flex;
   align-items: center;

data/app/controllers/completion_kit/agreements_controller.rb

@@ -5,7 +5,7 @@ module CompletionKit
     before_action :reject_check_metric, only: [:create]
 
     def create
-      created_by = agreement_creator
+      created_by = ck_current_reviewer
       existing = Agreement.find_by(
         run_id: @run.id, response_id: @response.id, metric_id: @metric.id, created_by: created_by
       )
@@ -74,9 +74,5 @@ module CompletionKit
     def review_for_metric
       @response.reviews.find_by(metric_id: @metric.id)
     end
-
-    def agreement_creator
-      request.env["HTTP_X_REMOTE_USER"].presence || CompletionKit.config.username.presence || "operator"
-    end
   end
 end

data/app/controllers/completion_kit/api/v1/base_controller.rb

@@ -5,9 +5,14 @@ module CompletionKit
         rate_limit to: CompletionKit.config.api_rate_limit, within: 1.minute,
                    with: -> { render_error("Rate limit exceeded", status: :too_many_requests) }
         before_action :authenticate_api!
+        rescue_from ConfigurationError, with: :render_configuration_error
 
         private
 
+        def render_configuration_error(error)
+          render_error(error.message, status: :unprocessable_entity)
+        end
+
         def render_error(message, status:, details: nil)
           payload = { error: message }
           payload[:details] = details if details.present?

data/app/controllers/completion_kit/api/v1/metric_groups_controller.rb

@@ -15,9 +15,8 @@ module CompletionKit
         end
 
         def create
-          metric_group = MetricGroup.new(metric_group_params.except(:metric_ids))
+          metric_group = MetricGroup.new(metric_group_attributes)
           if metric_group.save
-            metric_group.replace_metrics!(params[:metric_ids]) if params.key?(:metric_ids)
             render json: metric_group.reload, status: :created
           else
             render_validation_errors(metric_group)
@@ -25,8 +24,7 @@ module CompletionKit
         end
 
         def update
-          if @metric_group.update(metric_group_params.except(:metric_ids))
-            @metric_group.replace_metrics!(params[:metric_ids]) if params.key?(:metric_ids)
+          if @metric_group.update(metric_group_attributes)
             render json: @metric_group.reload
           else
             render_validation_errors(@metric_group)
@@ -49,6 +47,10 @@ module CompletionKit
         def metric_group_params
           params.permit(:name, :description, metric_ids: [], tag_names: [])
         end
+
+        def metric_group_attributes
+          metric_group_params.except(:metric_ids).merge(selected_metric_ids: metric_group_params[:metric_ids])
+        end
       end
     end
   end

data/app/controllers/completion_kit/api/v1/metrics_controller.rb

@@ -48,9 +48,9 @@ module CompletionKit
             return
           end
 
-          MetricVersion.drafts.where(metric_id: @metric.id, source: "suggestion").destroy_all
           generator = MetricVariantGenerator.new(@metric, count: params[:count].to_i, model: params[:model])
           variants = generator.call
+          MetricVersion.drafts.where(metric_id: @metric.id, source: "suggestion").destroy_all
           if variants.empty?
             render_error("The model returned no usable variants. Try again with a different model.", status: :unprocessable_entity)
             return

data/app/controllers/completion_kit/api/v1/runs_controller.rb

@@ -18,9 +18,8 @@ module CompletionKit
         end
 
         def create
-          run = Run.new(run_params.except(:metric_ids))
+          run = Run.new(run_attributes)
           if run.save
-            run.replace_metrics!(params[:metric_ids])
             render json: run.reload, status: :created
           else
             render_validation_errors(run)
@@ -28,8 +27,7 @@ module CompletionKit
         end
 
         def update
-          if @run.update(run_params.except(:metric_ids))
-            @run.replace_metrics!(params[:metric_ids]) if params.key?(:metric_ids)
+          if @run.update(run_attributes)
             render json: @run.reload
           else
             render_validation_errors(@run)
@@ -65,13 +63,15 @@ module CompletionKit
           else
             render_error(new_run.failure_summary || "Could not start the new run.", status: :unprocessable_entity)
           end
+        rescue ActiveRecord::RecordInvalid => e
+          render_validation_errors(e.record)
         end
 
         def regrade
           if @run.regrade!
             render json: @run.reload, status: :accepted
           else
-            render_error("Nothing to re-grade. The run has no succeeded responses or no metrics attached.", status: :unprocessable_entity)
+            render_error(@run.errors.full_messages.to_sentence.presence || "Nothing to re-grade. The run has no succeeded responses or no metrics attached.", status: :unprocessable_entity)
           end
         end
 
@@ -133,6 +133,10 @@ module CompletionKit
           params.permit(:name, :prompt_id, :dataset_id, :judge_model, :temperature, :max_tokens, :judge_temperature, :output_column, :expected_column,
             metric_ids: [], tag_names: [])
         end
+
+        def run_attributes
+          run_params.except(:metric_ids).merge(selected_metric_ids: run_params[:metric_ids])
+        end
       end
     end
   end

data/app/controllers/completion_kit/application_controller.rb

@@ -8,9 +8,22 @@ module CompletionKit
     rate_limit to: CompletionKit.config.web_rate_limit, within: 1.minute,
                with: -> { render plain: "Rate limit exceeded. Please slow down.", status: :too_many_requests }
     before_action :authenticate_completion_kit!
+    helper_method :ck_current_reviewer
 
     private
 
+    def ck_current_reviewer
+      @ck_current_reviewer ||= begin
+        cfg = CompletionKit.config
+        name = cfg.current_reviewer&.call(self)
+        unless name.nil? || name.is_a?(String)
+          raise CompletionKit::ConfigurationError,
+            "current_reviewer must return a String or nil, got #{name.class}. Return a name such as current_user.email."
+        end
+        name.presence || cfg.username.presence || "operator"
+      end
+    end
+
     def workspace_ready?
       CompletionKit::Onboarding::Checklist.new.complete? ||
         cookies[ONBOARDING_DISMISS_COOKIE].present?

data/app/controllers/completion_kit/mcp_controller.rb

@@ -2,6 +2,8 @@ module CompletionKit
   class McpController < Api::V1::BaseController
     PUBLIC_METHODS = %w[initialize notifications/initialized tools/list].freeze
 
+    FOREIGN_KEY_MESSAGE = "This change references a record that does not exist or is still in use.".freeze
+
     skip_before_action :authenticate_api!, only: [:stream, :handle], raise: false
 
     def stream
@@ -39,17 +41,18 @@ module CompletionKit
     rescue McpDispatcher::InvalidParams => e
       render json: jsonrpc_error(request_body.dig("id"), -32602, e.message), status: :ok
     rescue ActiveRecord::RecordNotFound => e
+      render json: jsonrpc_error(request_body.dig("id"), -32602, not_found_message(e)), status: :ok
+    rescue ActiveRecord::RecordInvalid => e
       render json: jsonrpc_error(request_body.dig("id"), -32602, e.message), status: :ok
-    rescue ActiveRecord::RecordInvalid, ActiveRecord::InvalidForeignKey => e
-      render json: jsonrpc_error(request_body.dig("id"), -32602, e.message), status: :ok
+    rescue ActiveRecord::InvalidForeignKey => e
+      Rails.error.report(e, handled: true, context: { controller: "CompletionKit::McpController" })
+      render json: jsonrpc_error(request_body.dig("id"), -32602, FOREIGN_KEY_MESSAGE), status: :ok
     rescue StandardError => e
       Rails.error.report(e, handled: true, context: { controller: "CompletionKit::McpController" })
       render json: jsonrpc_error(request_body.dig("id"), -32603, "Internal error"), status: :ok
     end
 
     def destroy
-      session_id = request.headers["Mcp-Session-Id"]
-      McpSession.destroy_session(session_id) if session_id
       head :ok
     end
 
@@ -61,6 +64,11 @@ module CompletionKit
       !performed?
     end
 
+    def not_found_message(error)
+      return "Record not found" unless error.model.present? && error.id.present?
+      "#{error.model.demodulize.titleize} #{Array(error.id).join(", ")} not found"
+    end
+
     def jsonrpc_response(id, result)
       {jsonrpc: "2.0", id: id, result: result}
     end

data/app/controllers/completion_kit/metric_groups_controller.rb

@@ -20,11 +20,10 @@ module CompletionKit
     end
 
     def create
-      @metric_group = MetricGroup.new(metric_group_params.except(:metric_ids))
+      @metric_group = MetricGroup.new(metric_group_attributes)
       @metrics = Metric.includes(:tags).order(:name)
 
       if @metric_group.save
-        @metric_group.replace_metrics!(metric_group_params[:metric_ids])
         redirect_to metric_group_path(@metric_group), notice: "Metric group was successfully created."
       else
         render :new, status: :unprocessable_entity
@@ -34,8 +33,7 @@ module CompletionKit
     def update
       @metrics = Metric.includes(:tags).order(:name)
 
-      if @metric_group.update(metric_group_params.except(:metric_ids))
-        @metric_group.replace_metrics!(metric_group_params[:metric_ids])
+      if @metric_group.update(metric_group_attributes)
         redirect_to metric_group_path(@metric_group), notice: "Metric group was successfully updated."
       else
         render :edit, status: :unprocessable_entity
@@ -56,5 +54,9 @@ module CompletionKit
     def metric_group_params
       params.require(:metric_group).permit(:name, :description, metric_ids: [], tag_names: [])
     end
+
+    def metric_group_attributes
+      metric_group_params.except(:metric_ids).merge(selected_metric_ids: metric_group_params[:metric_ids])
+    end
   end
 end

data/app/controllers/completion_kit/provider_credentials_controller.rb

@@ -29,7 +29,11 @@ module CompletionKit
     end
 
     def update
-      if @provider_credential.update(provider_credential_params)
+      attributes = provider_credential_update_params
+      remove = attributes[:remove_api_key] == "1"
+      @remove_api_key = remove && attributes[:api_key].blank?
+
+      if @provider_credential.update(with_api_key_choice(attributes.except(:remove_api_key), remove: remove))
         redirect_to provider_credentials_path, notice: "Provider credential was successfully updated."
       else
         render :edit, status: :unprocessable_entity
@@ -72,5 +76,19 @@ module CompletionKit
     def provider_credential_params
       params.require(:provider_credential).permit(:provider, :api_key, :api_endpoint, :api_version)
     end
+
+    def provider_credential_update_params
+      params.require(:provider_credential).permit(:provider, :api_key, :api_endpoint, :api_version, :remove_api_key)
+    end
+
+    def with_api_key_choice(attributes, remove:)
+      if attributes[:api_key].present?
+        attributes
+      elsif remove
+        attributes.merge(api_key: nil)
+      else
+        attributes.except(:api_key)
+      end
+    end
   end
 end

data/app/controllers/completion_kit/runs_controller.rb

@@ -40,9 +40,8 @@ module CompletionKit
     end
 
     def create
-      @run = Run.new(run_params.except(:metric_ids))
+      @run = Run.new(run_attributes)
       if @run.save
-        @run.replace_metrics!(params[:run][:metric_ids])
         redirect_to run_path(@run), notice: "Run was successfully created."
       else
         load_form_collections
@@ -52,20 +51,18 @@ module CompletionKit
 
     def update
       if @run.responses.any? && run_generation_changed?
-        attrs = run_params.except(:metric_ids).to_h
+        attrs = run_attributes.to_h
         attrs.delete("name") if attrs["name"].to_s == @run.name.to_s
         new_run = Run.new(attrs.merge(status: "pending"))
         if new_run.save
-          new_run.replace_metrics!(params[:run][:metric_ids]) if params[:run].key?(:metric_ids)
           redirect_to run_path(new_run), notice: "Saved as a new run. The previous run and its results are preserved."
         else
-          @run.assign_attributes(run_params.except(:metric_ids))
+          @run.assign_attributes(run_attributes)
           new_run.errors.each { |error| @run.errors.add(error.attribute, error.message) }
           load_form_collections
           render :edit, status: :unprocessable_entity
         end
-      elsif @run.update(run_params.except(:metric_ids))
-        @run.replace_metrics!(params[:run][:metric_ids]) if params[:run].key?(:metric_ids)
+      elsif @run.update(run_attributes)
         redirect_to run_path(@run), notice: "Run saved."
       else
         load_form_collections
@@ -106,7 +103,7 @@ module CompletionKit
       if @run.regrade!
         redirect_to run_path(@run), notice: "Re-grading existing responses against the current metrics."
       else
-        redirect_to run_path(@run), alert: "Nothing to re-grade. The run has no succeeded responses or no metrics attached."
+        redirect_to run_path(@run), alert: @run.errors.full_messages.to_sentence.presence || "Nothing to re-grade. The run has no succeeded responses or no metrics attached."
       end
     end
 
@@ -117,6 +114,10 @@ module CompletionKit
       else
         redirect_to run_path(new_run), alert: new_run.failure_summary || "Could not start the new run."
       end
+    rescue ActiveRecord::RecordInvalid => e
+      redirect_to run_path(@run), alert: "This run can't be rerun. #{e.record.errors.full_messages.to_sentence}."
+    rescue ConfigurationError => e
+      redirect_to run_path(@run), alert: e.message
     end
 
     # The web process drives the live run page, because worker-side Turbo
@@ -226,6 +227,10 @@ module CompletionKit
       params.require(:run).permit(:name, :prompt_id, :dataset_id, :judge_model, :temperature, :max_tokens, :judge_temperature, :output_column, :expected_column, metric_ids: [], tag_names: [])
     end
 
+    def run_attributes
+      run_params.except(:metric_ids).merge(selected_metric_ids: run_params[:metric_ids])
+    end
+
     # Editing a run that already has results forks a new run — but only when a
     # field that affects generation or judging changed. Renaming or retagging is
     # pure metadata and updates the run in place.

data/app/jobs/completion_kit/generate_row_job.rb

@@ -76,7 +76,7 @@ module CompletionKit
         error_provider: nil, error_class: nil, error_status: nil, error_message: nil
       )
 
-      if run.llm_judge_configured?
+      if run.judge_model.present?
         run.llm_metrics.each do |metric|
           JudgeReviewJob.perform_later(response.id, metric.id, run.id)
         end
@@ -101,6 +101,7 @@ module CompletionKit
       response = Response.find_by(id: @response_id)
       return unless response
 
+      error = EndpointError.from(error)
       response.update!(
         status: "failed",
         error_provider: provider_for(response),

data/app/jobs/completion_kit/judge_review_job.rb

@@ -117,6 +117,7 @@ module CompletionKit
       response = Response.find_by(id: @response_id)
       return unless response
 
+      error = EndpointError.from(error)
       review = response.reviews.find_or_initialize_by(metric_id: @metric_id)
       review.assign_attributes(
         metric_name: review.metric_name || Metric.find_by(id: @metric_id)&.name || "(deleted metric)",

data/app/jobs/completion_kit/metric_suggestion_job.rb

@@ -17,10 +17,9 @@ module CompletionKit
       return unless @metric
       return if @metric.check?
 
-      MetricVersion.drafts.where(metric_id: @metric.id, source: "suggestion").destroy_all
-
       generator = MetricVariantGenerator.new(@metric, count: 1)
       variants = generator.call
+      MetricVersion.drafts.where(metric_id: @metric.id, source: "suggestion").destroy_all
       if variants.empty?
         broadcast_status(@metric, partial: "completion_kit/metrics/suggestion_failed", locals: { metric: @metric })
         return

data/app/jobs/completion_kit/model_discovery_job.rb

@@ -8,26 +8,23 @@ module CompletionKit
       30 * executions
     end
 
+    rescue_from(StandardError) do |error|
+      report_and_record_failure!(error)
+    end
+
     retry_on Faraday::TimeoutError,
              Faraday::ConnectionFailed,
-             wait: :polynomially_longer, attempts: 5
+             wait: :polynomially_longer, attempts: 5 do |job, error|
+      job.send(:report_and_record_failure!, error)
+    end
 
     retry_on CompletionKit::RateLimitError,
-             wait: method(:rate_limit_wait), attempts: 5
+             wait: ->(executions) { rate_limit_wait(executions) }, attempts: 5 do |job, error|
+      job.send(:report_and_record_failure!, error)
+    end
 
     discard_on ActiveJob::DeserializationError
 
-    rescue_from(StandardError) do |error|
-      unless error.is_a?(CompletionKit::ModelDiscoveryService::DiscoveryError)
-        Rails.error.report(error, handled: true, context: { job: self.class.name, provider_credential_id: arguments.first })
-      end
-      credential = ProviderCredential.find_by(id: arguments.first)
-      next unless credential
-      credential.update_columns(discovery_status: "failed", discovery_error: error.message.to_s.truncate(500))
-      credential.reload
-      credential.broadcast_discovery_progress
-    end
-
     def perform(provider_credential_id, force: false)
       credential = ProviderCredential.find_by(id: provider_credential_id)
       return unless credential
@@ -57,5 +54,23 @@ module CompletionKit
       credential.reload
       credential.broadcast_discovery_complete
     end
+
+    private
+
+    def report_and_record_failure!(error)
+      unless error.is_a?(CompletionKit::ModelDiscoveryService::DiscoveryError)
+        Rails.error.report(error, handled: true, context: { job: self.class.name, provider_credential_id: arguments.first })
+      end
+      record_failure!(error)
+    end
+
+    def record_failure!(error)
+      credential = ProviderCredential.find_by(id: arguments.first)
+      return unless credential
+
+      credential.update_columns(discovery_status: "failed", discovery_error: EndpointError.from(error).message.to_s.truncate(500))
+      credential.reload
+      credential.broadcast_discovery_progress
+    end
   end
 end

data/app/models/completion_kit/metric_group.rb

@@ -1,6 +1,7 @@
 module CompletionKit
   class MetricGroup < ApplicationRecord
     include CompletionKit::Taggable
+    include CompletionKit::MetricSelection
 
     self.table_name = "completion_kit_metric_groups"
 
@@ -14,9 +15,10 @@ module CompletionKit
     end
 
     def replace_metrics!(metric_ids)
-      return unless metric_ids
+      return if metric_ids.nil?
+
       metric_group_memberships.delete_all
-      Array(metric_ids).reject(&:blank?).each_with_index do |metric_id, index|
+      normalize_metric_ids(metric_ids).each_with_index do |metric_id, index|
         metric_group_memberships.create!(metric_id: metric_id, position: index + 1)
       end
     end

data/app/models/completion_kit/metric_group_membership.rb

@@ -3,7 +3,7 @@ module CompletionKit
     self.table_name = "completion_kit_metric_group_memberships"
 
     belongs_to :metric_group, class_name: "CompletionKit::MetricGroup", foreign_key: "metric_group_id"
-    belongs_to :metric
+    belongs_to :metric, optional: false
 
     validates :metric_id, uniqueness: { scope: :metric_group_id }
 

data/app/models/completion_kit/prompt.rb

@@ -5,7 +5,7 @@ module CompletionKit
     has_many :runs, dependent: :destroy
     has_many :responses, through: :runs
 
-    attr_accessor :cloned_from_llm_model
+    attr_accessor :cloned_from_llm_model, :allow_unconfigured_provider
 
     validates :name, presence: true
     validates :template, presence: true
@@ -105,12 +105,13 @@ module CompletionKit
     end
 
     def llm_model_usable_for_generation
-      rows = Model.where(model_id: llm_model)
-      return if rows.empty?
-      return if rows.where(supports_generation: true, status: "active").exists?
-      return unless rows.where(supports_generation: false).exists?
+      return if Model.for_generation.exists?(model_id: llm_model)
 
-      errors.add(:llm_model, "is not available for generating responses")
+      if Model.exists?(model_id: llm_model, supports_generation: false)
+        errors.add(:llm_model, "is not available for generating responses")
+      elsif !allow_unconfigured_provider && !ApiConfig.servable_model?(llm_model)
+        errors.add(:llm_model, %("#{llm_model}" is not served by any configured provider))
+      end
     end
 
     def assign_family_key

data/app/models/completion_kit/provider_credential.rb

@@ -27,6 +27,7 @@ module CompletionKit
     validates :provider, tenant_scoped_uniqueness: true
     validates :api_endpoint, presence: true, if: :azure_foundry?
     validate :api_endpoint_not_internal
+    validate :api_key_reentered_for_new_destination, on: :update
 
     after_save :enqueue_discovery
     before_destroy :ensure_not_in_use, prepend: true
@@ -36,6 +37,15 @@ module CompletionKit
       provider == "azure_foundry"
     end
 
+    def api_key_stored?
+      api_key_in_database.present?
+    end
+
+    def api_key_ending
+      key = api_key_in_database.to_s
+      key.last(4) if key.length > 8
+    end
+
     def config_hash
       {
         provider: provider,
@@ -180,14 +190,21 @@ module CompletionKit
       Rails.logger.error("[CompletionKit] discovery broadcast render failed: #{e.class}: #{e.message}")
     end
 
+    def api_key_reentered_for_new_destination
+      return if api_key_came_from_user? || api_key.blank?
+      return unless provider_changed? || api_endpoint_in_database.presence != api_endpoint.presence
+
+      errors.add(:api_key, "must be entered again when the provider or endpoint changes")
+    end
+
     def api_endpoint_not_internal
       return if api_endpoint.blank?
 
       issues = ProviderEndpoint.validate(api_endpoint)
       if issues.include?(:invalid_url)
         errors.add(:api_endpoint, "must be a valid http or https URL")
-      elsif issues.include?(:unsafe_host)
-        errors.add(:api_endpoint, "must not point at a private or internal address")
+      elsif issues.any?
+        errors.add(:api_endpoint, "must resolve to a public address")
       end
     end
   end