completion-kit 0.1.0.rc1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cdecf15deb685a524186a2bd4ba48268e10475da0d2cc2914969318893268f70
-  data.tar.gz: ce395abf147434f9a825f79902e47b171b074038d20c06bf7b86b80ba70eaa00
+  metadata.gz: 621db92d1653ef6d326f46a164593ea731817000771aeaa649263fcf5d0a35e4
+  data.tar.gz: 7fbee94e29658df3508530710600a862e18b84d3b0c7b7b0f9d6ee96de1a6bbd
 SHA512:
-  metadata.gz: 9d13fd1d1863c87ca0f7ed78eb1853ac921ff39862b84c678e5b5c9977f1832edcfb50c52b2371f2dc88ad6360be22a78d39ef53385239939a1ac9297df444ff
-  data.tar.gz: bf95df5b178ccfe455350d216a894836a85cfeff5c117d0206b87211ef638c1d690790350168b5a9ad84e8e3895add8c89acdea98738a1c0d2025115679d939f
+  metadata.gz: c76b3407321ed225a97516ace2a14f39786514c2fcc393e105c2be4c04f572a01395058d9af0854164135321b2cea3b711d9675431c9d62e19fe6ca235c514df
+  data.tar.gz: a175cf106b5ccfb2d6605c70be85c3fe96a52962187a3a50de6bfe75322c06a4aa818b8a17ac65208479acb6eba51b1de5fe9004505ed2068881faebfb7c1334

data/README.md

@@ -1,29 +1,44 @@
 <p align="center">
-  <img src="https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/logo.png" alt="CompletionKit logo" width="120" />
+  <img src="https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/logo.png" alt="CompletionKit" width="360" />
 </p>
 
-# CompletionKit
-
-[![CI](https://github.com/homemade-software-inc/completion-kit/actions/workflows/ci.yml/badge.svg)](https://github.com/homemade-software-inc/completion-kit/actions/workflows/ci.yml)
-![coverage](https://img.shields.io/badge/coverage-100%25-brightgreen)
-![dependencies](https://img.shields.io/badge/dependencies-7-blue)
-[![Dependabot](https://img.shields.io/badge/dependabot-enabled-blue?logo=dependabot)](https://github.com/homemade-software-inc/completion-kit/network/updates)
-
-You need to know whether your prompts produce the output you expect, consistently, across real data. CompletionKit gives you that, inside your Rails app.
+<p align="center">
+  <a href="https://github.com/homemade-software-inc/completion-kit/actions/workflows/ci.yml"><img src="https://github.com/homemade-software-inc/completion-kit/actions/workflows/ci.yml/badge.svg" alt="CI" /></a>
+  <img src="https://img.shields.io/badge/coverage-100%25-brightgreen" alt="coverage" />
+</p>
 
-Mount the engine, bring your prompts and datasets, and every input runs through a model you pick. Each output is scored against your own metrics and rubrics by an LLM-as-judge. When you change a prompt, re-run the same dataset and see exactly what got better and what broke — and when the scores tell you something's off, CompletionKit can suggest an improved version of the prompt based on the reviews, which you inspect as a diff and apply as a new version.
+Your prompts need tests too.
 
-Drive it from the web UI, from the REST API, or from Claude Code and other MCP-aware agents via the built-in Model Context Protocol server. All three share the same state — your prompts, runs, datasets, and scores are one source of truth.
+Run every prompt against real data. Score each output with an LLM judge against criteria you define. Change anything: the prompt, the model, the temperature, the dataset. Re-run and see exactly what got better and what broke. When the scores tell you something's off, CompletionKit suggests an improved prompt based on the judge's actual feedback on your runs. You inspect the diff, apply it as a new version, and verify the improvement.
 
 It's the difference between "this prompt seems to work" and "this prompt scores 4.3 out of 5 across 200 inputs, up from 3.8 last version."
 
+**[completionkit.com](https://completionkit.com)** | **[RubyGems](https://rubygems.org/gems/completion-kit)**
+
 ![Prompts index](https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/screenshots/prompts.png)
 
 ![Prompt detail with metrics and rubrics](https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/screenshots/prompt-detail.png)
 
 ![Test run with scored results](https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/screenshots/test-run.png)
 
-## Setup
+## Quick Start
+
+### Run the standalone app
+
+The fastest way to start. No existing Rails app needed.
+
+```bash
+git clone https://github.com/homemade-software-inc/completion-kit.git
+cd completion-kit/standalone
+bundle install
+bin/rails completion_kit:install:migrations
+bin/rails db:migrate
+bin/rails server
+```
+
+Visit `http://localhost:3000`. Add a provider credential (Settings), create a prompt, upload a CSV dataset, and run it.
+
+### Or mount as an engine in your existing Rails app
 
 ```ruby
 gem "completion-kit"
@@ -34,26 +49,40 @@ bin/rails generate completion_kit:install
 bin/rails db:migrate
 ```
 
-Set your provider keys via environment variables or the generated initializer:
+The engine mounts at `/completion_kit` in your app.
 
-```bash
-OPENAI_API_KEY=...
-ANTHROPIC_API_KEY=...
-LLAMA_API_KEY=...
-LLAMA_API_ENDPOINT=...
-```
+## Providers
+
+CompletionKit discovers available models from each provider's API automatically.
+
+| Provider | Env vars | What it covers |
+|----------|----------|----------------|
+| **OpenAI** | `OPENAI_API_KEY` | GPT-5, GPT-4.1, GPT-4o, etc. |
+| **Anthropic** | `ANTHROPIC_API_KEY` | Claude Opus, Sonnet, Haiku |
+| **Ollama / local endpoint** | `OLLAMA_API_ENDPOINT` (default: `http://localhost:11434/v1`) | Any model you've `ollama pull`-ed, or any OpenAI-compatible local server (vLLM, LM Studio, llama.cpp) |
+| **OpenRouter** | `OPENROUTER_API_KEY` | 100+ models from 30+ providers through one API key |
+
+Set these as environment variables or configure them in the generated initializer. You can also add provider credentials through the web UI under Settings.
 
-Available models are discovered dynamically from each provider's API.
+### Encryption
 
-### Encryption keys
+Provider API keys are encrypted at rest using [Active Record encryption](https://guides.rubyonrails.org/active_record_encryption.html). You need three encryption keys configured before the app will boot in production.
 
-Provider API keys are stored using [Rails Active Record encryption](https://guides.rubyonrails.org/active_record_encryption.html), so the host app must have encryption keys configured. If you haven't set them up already:
+Generate them:
 
 ```bash
 bin/rails db:encryption:init
 ```
 
-Copy the generated keys into `config/credentials.yml.enc` under `active_record_encryption`, or set the equivalent environment variables. CompletionKit won't boot without valid keys in production.
+Then set them as environment variables:
+
+```bash
+COMPLETION_KIT_ENCRYPTION_PRIMARY_KEY=<generated value>
+COMPLETION_KIT_ENCRYPTION_DETERMINISTIC_KEY=<generated value>
+COMPLETION_KIT_ENCRYPTION_KEY_DERIVATION_SALT=<generated value>
+```
+
+Or add them to `config/credentials.yml.enc` under `active_record_encryption`. In development, the standalone app uses built-in fallback values so you can skip this step locally.
 
 ## Authentication
 
@@ -62,7 +91,6 @@ CompletionKit requires authentication in production. In development, routes are
 ### Basic Auth (recommended for simple setups)
 
 ```ruby
-# config/initializers/completion_kit.rb
 CompletionKit.configure do |c|
   c.username = "admin"
   c.password = ENV["COMPLETION_KIT_PASSWORD"]
@@ -72,58 +100,49 @@ end
 ### Custom Auth (Devise, etc.)
 
 ```ruby
-# config/initializers/completion_kit.rb
 CompletionKit.configure do |c|
   c.auth_strategy = ->(controller) { controller.authenticate_user! }
 end
 ```
 
-Only one mode can be active — setting both raises a `ConfigurationError`.
+Only one mode can be active.
+
+## How it works
+
+1. **Create a prompt** with `{{variable}}` placeholders
+2. **Upload a dataset.** A CSV where column headers match the variable names.
+3. **Run it** against a model and score outputs with an LLM judge against criteria you define.
+4. **Iterate.** Change the prompt, the model, the temperature, or the dataset and re-run. CompletionKit versions your prompts so you can always compare against previous results.
+5. **Get suggestions.** When scores drop, ask CompletionKit for an AI-generated improvement. The suggestion is based on the judge's actual per-response feedback, not generic prompt-engineering advice. Inspect the diff and apply it as a new version.
 
-## Usage
+## Concepts
 
-1. Create a prompt with `{{variable}}` placeholders
-2. Create a test run and paste CSV data (headers match variable names)
-3. Generate outputs, run AI review, inspect scored results
+- **Prompt.** A versioned template with `{{variable}}` placeholders. Publishing freezes the template; editing a published prompt creates a new version.
+- **Dataset.** A CSV of real inputs. Each row becomes one test case.
+- **Run.** One execution of a prompt against a dataset. Captures every input (model, temperature, metrics) and stores all outputs and scores.
+- **Response.** The model's output for one dataset row, with reviews attached.
+- **Metric.** An evaluation dimension with a name, instruction, evaluation steps, and a 1-5 star scoring scale. The LLM judge uses this to score each response.
+- **Metric Group.** A reusable group of metrics you can apply to a run as a set.
+- **Provider Credential.** An API key for a model provider. Encrypted at rest, never returned through the API.
 
-## Programmatic access
+## REST API
 
-CompletionKit exposes every resource through both a REST JSON API and an MCP server. Both share the same bearer-token auth, so configure once and use either interface:
+Every resource is accessible via a bearer-token JSON API:
 
 ```ruby
-# config/initializers/completion_kit.rb
 CompletionKit.configure { |c| c.api_token = ENV["COMPLETION_KIT_API_TOKEN"] }
 ```
 
-### Concepts
-
-These are the objects you'll work with, whether through the UI, the REST API, or the MCP server:
-
-- **Prompt** — A named, versioned template with `{{variable}}` placeholders. Publishing a prompt freezes its template so runs always reference a known version; editing a published prompt creates a new version.
-- **Dataset** — A CSV of real inputs. Column headers match the prompt's `{{variable}}` names, and each row becomes one test case.
-- **Run** — A single execution of a prompt against a dataset. Tracks progress, stores outputs, and records which metrics were used for scoring.
-- **Response** — The model's output for one row of the dataset, with any reviews attached.
-- **Metric** — One evaluation dimension: a name, an instruction, evaluation steps, and 1–5-star rubric bands. The judge uses a metric to score a response.
-- **Criteria** — A named, reusable bundle of metrics you can apply to a run in one step.
-- **Provider Credential** — An API key for a model provider (OpenAI, Anthropic, Ollama, OpenRouter). Encrypted at rest using Rails' Active Record encryption, and never returned through the API.
-
-### REST API
-
 ```bash
 curl -H "Authorization: Bearer $TOKEN" \
   http://localhost:3000/completion_kit/api/v1/prompts
-
-curl -X POST http://localhost:3000/completion_kit/api/v1/prompts \
-  -H "Authorization: Bearer $TOKEN" \
-  -H "Content-Type: application/json" \
-  -d '{"name":"summarizer","template":"Summarize: {{text}}","llm_model":"gpt-4.1"}'
 ```
 
-Mount the engine, then visit **`/completion_kit/api_reference`** in your running app for per-endpoint documentation with copy-to-clipboard curl examples pre-filled with your token.
+Visit `/completion_kit/api_reference` in your running app for per-endpoint docs with copy-to-clipboard curl examples.
 
-### MCP server
+## MCP server
 
-CompletionKit also runs a [Model Context Protocol](https://modelcontextprotocol.io) server at the `/mcp` path within the engine mount, exposing the same resources as 36 tools (one per CRUD action plus process actions like `runs_generate` and `prompts_publish`). Point Claude Code, Cursor, or any other MCP client at it:
+CompletionKit runs a [Model Context Protocol](https://modelcontextprotocol.io) server at `/completion_kit/mcp`, exposing every resource as tools that MCP-aware clients (Claude Code, Cursor, etc.) can drive directly:
 
 ```json
 {
@@ -136,40 +155,21 @@ CompletionKit also runs a [Model Context Protocol](https://modelcontextprotocol.
 }
 ```
 
-The in-app API reference page also ships install snippets you can copy straight into your MCP client config.
-
-## Standalone App
-
-CompletionKit ships with a standalone Rails app you can deploy as a hosted service.
-
-### Quick Start
-
-```bash
-cd standalone
-bundle install
-bin/rails completion_kit:install:migrations
-bin/rails db:migrate
-bin/rails server
-```
-
-Visit `http://localhost:3000` for the home page, or `http://localhost:3000/completion_kit` for the engine UI.
+The in-app API reference page has install snippets you can copy straight into your MCP client config.
 
-### Configuration
+## Deploying the standalone app
 
-Set environment variables:
+Any Rails-friendly host works (Fly, Heroku, Render, Docker, etc.). Point it at a Postgres instance via `DATABASE_URL`, set your provider env vars, and run `cd standalone && bin/rails db:migrate` on each deploy.
 
 | Variable | Purpose | Default |
 |----------|---------|---------|
-| `COMPLETION_KIT_API_TOKEN` | Bearer token for REST API and MCP access | (none — API disabled) |
+| `COMPLETION_KIT_API_TOKEN` | Bearer token for REST API and MCP | (none, API disabled) |
 | `COMPLETION_KIT_USERNAME` | Web UI login username | `admin` |
-| `COMPLETION_KIT_PASSWORD` | Web UI login password | (none — open in dev) |
-| `DATABASE_URL` | PostgreSQL connection string (production) | SQLite in dev |
+| `COMPLETION_KIT_PASSWORD` | Web UI login password | (none, open in dev) |
 
-### Deploying
+You also need the three `COMPLETION_KIT_ENCRYPTION_*` keys from the [Encryption](#encryption) section above.
 
-Any Rails-friendly host works — Fly, Heroku, Render, self-managed Docker, etc. Point your host at a Postgres instance via `DATABASE_URL`, set the environment variables above, and run `cd standalone && bin/rails db:migrate` on each deploy.
-
-When the gem ships a new engine migration, install it into your standalone app locally and commit the generated file before pushing:
+When the gem ships a new migration, install it locally and commit before pushing:
 
 ```bash
 cd standalone
@@ -178,15 +178,26 @@ bin/rails db:migrate
 git add db/migrate/ && git commit -m "install new engine migration"
 ```
 
-That way your host's `db:migrate` picks up the new file on the next deploy. Don't run `completion_kit:install:migrations` on the host itself — migration files are source artifacts, they belong in git.
+## Multi-tenant host apps (advanced)
 
-## Development
+For hosts that mount CompletionKit in a multi-tenant app, two optional hooks scope engine records per tenant without forking the engine:
 
-```bash
-bundle install
-bundle exec rspec
+```ruby
+CompletionKit.configure do |config|
+  config.tenant_scope = -> {
+    org = Current.organization&.id
+    org ? where(organization_id: org) : where("1=0")
+  }
+  config.tenant_scope_columns = [:organization_id]
+end
 ```
 
+`tenant_scope` runs as each engine model's `default_scope` (use `unscoped` to bypass). `tenant_scope_columns` is appended to every engine uniqueness validation. Adding the tenant columns and composite unique indexes lives in your host migrations. Both defaults (`nil`, `[]`) are no-ops.
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, testing, and pull request guidelines.
+
 ## License
 
 [MIT](https://opensource.org/licenses/MIT)

data/app/controllers/completion_kit/api/v1/metric_groups_controller.rb

@@ -0,0 +1,53 @@
+module CompletionKit
+  module Api
+    module V1
+      class MetricGroupsController < BaseController
+        before_action :set_metric_group, only: [:show, :update, :destroy]
+
+        def index
+          render json: MetricGroup.order(created_at: :desc)
+        end
+
+        def show
+          render json: @metric_group
+        end
+
+        def create
+          metric_group = MetricGroup.new(metric_group_params.except(:metric_ids))
+          if metric_group.save
+            metric_group.replace_metrics!(params[:metric_ids]) if params.key?(:metric_ids)
+            render json: metric_group.reload, status: :created
+          else
+            render json: {errors: metric_group.errors}, status: :unprocessable_entity
+          end
+        end
+
+        def update
+          if @metric_group.update(metric_group_params.except(:metric_ids))
+            @metric_group.replace_metrics!(params[:metric_ids]) if params.key?(:metric_ids)
+            render json: @metric_group.reload
+          else
+            render json: {errors: @metric_group.errors}, status: :unprocessable_entity
+          end
+        end
+
+        def destroy
+          @metric_group.destroy!
+          head :no_content
+        end
+
+        private
+
+        def set_metric_group
+          @metric_group = MetricGroup.find(params[:id])
+        rescue ActiveRecord::RecordNotFound
+          not_found
+        end
+
+        def metric_group_params
+          params.permit(:name, :description, metric_ids: [])
+        end
+      end
+    end
+  end
+end

data/app/controllers/completion_kit/api/v1/metrics_controller.rb

@@ -43,7 +43,7 @@ module CompletionKit
         end
 
         def metric_params
-          params.permit(:name, :instruction, evaluation_steps: [], rubric_bands: [:stars, :description])
+          params.permit(:name, :instruction, rubric_bands: [:stars, :description])
         end
       end
     end

data/app/controllers/completion_kit/api/v1/runs_controller.rb

@@ -15,7 +15,7 @@ module CompletionKit
         def create
           run = Run.new(run_params.except(:metric_ids))
           if run.save
-            replace_run_metrics(run, params[:metric_ids])
+            run.replace_metrics!(params[:metric_ids])
             render json: run.reload, status: :created
           else
             render json: {errors: run.errors}, status: :unprocessable_entity
@@ -24,7 +24,7 @@ module CompletionKit
 
         def update
           if @run.update(run_params.except(:metric_ids))
-            replace_run_metrics(@run, params[:metric_ids]) if params.key?(:metric_ids)
+            @run.replace_metrics!(params[:metric_ids]) if params.key?(:metric_ids)
             render json: @run.reload
           else
             render json: {errors: @run.errors}, status: :unprocessable_entity
@@ -57,14 +57,6 @@ module CompletionKit
         def run_params
           params.permit(:name, :prompt_id, :dataset_id, :judge_model, :temperature, metric_ids: [])
         end
-
-        def replace_run_metrics(run, metric_ids)
-          return unless metric_ids
-          run.run_metrics.delete_all
-          Array(metric_ids).reject(&:blank?).each_with_index do |metric_id, index|
-            run.run_metrics.create!(metric_id: metric_id, position: index + 1)
-          end
-        end
       end
     end
   end

data/app/controllers/completion_kit/metric_groups_controller.rb

@@ -0,0 +1,59 @@
+module CompletionKit
+  class MetricGroupsController < ApplicationController
+    before_action :set_metric_group, only: [:show, :edit, :update, :destroy]
+
+    def index
+      @metric_groups = MetricGroup.includes(:metrics).order(:name)
+    end
+
+    def show
+    end
+
+    def new
+      @metric_group = MetricGroup.new
+      @metrics = Metric.order(:name)
+    end
+
+    def edit
+      @metrics = Metric.order(:name)
+    end
+
+    def create
+      @metric_group = MetricGroup.new(metric_group_params.except(:metric_ids))
+      @metrics = Metric.order(:name)
+
+      if @metric_group.save
+        @metric_group.replace_metrics!(metric_group_params[:metric_ids])
+        redirect_to metric_group_path(@metric_group), notice: "Metric group was successfully created."
+      else
+        render :new, status: :unprocessable_entity
+      end
+    end
+
+    def update
+      @metrics = Metric.order(:name)
+
+      if @metric_group.update(metric_group_params.except(:metric_ids))
+        @metric_group.replace_metrics!(metric_group_params[:metric_ids])
+        redirect_to metric_group_path(@metric_group), notice: "Metric group was successfully updated."
+      else
+        render :edit, status: :unprocessable_entity
+      end
+    end
+
+    def destroy
+      @metric_group.destroy
+      redirect_to metric_groups_path, notice: "Metric group was successfully destroyed."
+    end
+
+    private
+
+    def set_metric_group
+      @metric_group = MetricGroup.find(params[:id])
+    end
+
+    def metric_group_params
+      params.require(:metric_group).permit(:name, :description, metric_ids: [])
+    end
+  end
+end

data/app/controllers/completion_kit/metrics_controller.rb

@@ -3,7 +3,7 @@ module CompletionKit
     before_action :set_metric, only: [:show, :edit, :update, :destroy]
 
     def index
-      @metrics = Metric.order(:name)
+      @metrics = Metric.includes(:metric_groups).order(:name)
     end
 
     def show
@@ -46,7 +46,7 @@ module CompletionKit
     end
 
     def metric_params
-      params.require(:metric).permit(:name, :instruction, evaluation_steps: [], rubric_bands: [:stars, :description])
+      params.require(:metric).permit(:name, :instruction, rubric_bands: [:stars, :description])
     end
   end
 end

data/app/controllers/completion_kit/runs_controller.rb

@@ -35,7 +35,7 @@ module CompletionKit
     def create
       @run = Run.new(run_params.except(:metric_ids))
       if @run.save
-        replace_run_metrics(@run, params[:run][:metric_ids])
+        @run.replace_metrics!(params[:run][:metric_ids])
         redirect_to run_path(@run), notice: "Run was successfully created."
       else
         load_form_collections
@@ -46,10 +46,10 @@ module CompletionKit
     def update
       if @run.responses.any?
         new_run = Run.create!(run_params.except(:metric_ids).to_h.merge(status: "pending"))
-        replace_run_metrics(new_run, params[:run][:metric_ids]) if params[:run].key?(:metric_ids)
+        new_run.replace_metrics!(params[:run][:metric_ids]) if params[:run].key?(:metric_ids)
         redirect_to run_path(new_run), notice: "Saved as a new run. The previous run and its results are preserved."
       elsif @run.update(run_params.except(:metric_ids))
-        replace_run_metrics(@run, params[:run][:metric_ids]) if params[:run].key?(:metric_ids)
+        @run.replace_metrics!(params[:run][:metric_ids]) if params[:run].key?(:metric_ids)
         redirect_to run_path(@run), notice: "Run saved."
       else
         load_form_collections
@@ -112,7 +112,7 @@ module CompletionKit
     def load_form_collections
       @prompts = Prompt.order(:name)
       @datasets = Dataset.order(:name)
-      @criterias = Criteria.includes(:metrics).order(:name)
+      @metric_groups = MetricGroup.includes(:metrics).order(:name)
       @all_metrics = Metric.order(:name)
     end
 
@@ -120,12 +120,5 @@ module CompletionKit
       params.require(:run).permit(:name, :prompt_id, :dataset_id, :judge_model, :temperature, metric_ids: [])
     end
 
-    def replace_run_metrics(run, metric_ids)
-      return unless metric_ids
-      run.run_metrics.delete_all
-      Array(metric_ids).reject(&:blank?).each_with_index do |metric_id, index|
-        run.run_metrics.create!(metric_id: metric_id, position: index + 1)
-      end
-    end
   end
 end

data/app/helpers/completion_kit/application_helper.rb

@@ -76,15 +76,8 @@ module CompletionKit
       end
     end
 
-    PROVIDER_LABELS = {
-      "openai" => "OpenAI",
-      "anthropic" => "Anthropic",
-      "ollama" => "Ollama / local endpoint",
-      "openrouter" => "OpenRouter"
-    }.freeze
-
     def ck_provider_label(provider)
-      PROVIDER_LABELS[provider.to_s] || provider.to_s.titleize
+      CompletionKit::ProviderCredential::PROVIDER_LABELS[provider.to_s] || provider.to_s.titleize
     end
 
     def ck_grouped_models(models, selected = nil)

data/app/models/completion_kit/application_record.rb

@@ -1,5 +1,12 @@
 module CompletionKit
   class ApplicationRecord < ActiveRecord::Base
     self.abstract_class = true
+
+    TenantScopedUniquenessValidator = CompletionKit::TenantScopedUniquenessValidator
+
+    default_scope do
+      scope_proc = CompletionKit.config.tenant_scope
+      scope_proc ? instance_exec(&scope_proc) : all
+    end
   end
 end

data/app/models/completion_kit/metric.rb

@@ -8,15 +8,14 @@ module CompletionKit
       { "stars" => 1, "description" => "Fails to meet the criteria. Major errors or completely off-target." }
     ].freeze
 
-    has_many :criteria_memberships, dependent: :destroy
-    has_many :criterias, through: :criteria_memberships, source: :criteria
+    has_many :metric_group_memberships, dependent: :destroy
+    has_many :metric_groups, through: :metric_group_memberships, source: :metric_group
     has_many :reviews, dependent: :nullify
 
     serialize :rubric_bands, coder: JSON
-    serialize :evaluation_steps, coder: JSON
 
     validates :name, presence: true
-    validates :key, uniqueness: true, allow_nil: true
+    validates :key, tenant_scoped_uniqueness: { allow_nil: true }
 
     before_validation :generate_key
     before_validation :normalize_rubric_bands
@@ -74,7 +73,7 @@ module CompletionKit
     def as_json(options = {})
       {
         id: id, name: name, key: key, instruction: instruction,
-        evaluation_steps: evaluation_steps, rubric_bands: rubric_bands,
+        rubric_bands: rubric_bands,
         created_at: created_at, updated_at: updated_at
       }
     end
@@ -86,7 +85,6 @@ module CompletionKit
     end
 
     def set_defaults
-      self.evaluation_steps ||= []
       self.rubric_bands = self.class.default_rubric_bands if rubric_bands.blank?
     end
 

data/app/models/completion_kit/metric_group.rb

@@ -0,0 +1,30 @@
+module CompletionKit
+  class MetricGroup < ApplicationRecord
+    self.table_name = "completion_kit_metric_groups"
+
+    has_many :metric_group_memberships, -> { order(:position, :id) }, dependent: :destroy
+    has_many :metrics, through: :metric_group_memberships
+
+    validates :name, presence: true
+
+    def ordered_metrics
+      metric_group_memberships.includes(:metric).map(&:metric).compact
+    end
+
+    def replace_metrics!(metric_ids)
+      return unless metric_ids
+      metric_group_memberships.delete_all
+      Array(metric_ids).reject(&:blank?).each_with_index do |metric_id, index|
+        metric_group_memberships.create!(metric_id: metric_id, position: index + 1)
+      end
+    end
+
+    def as_json(options = {})
+      {
+        id: id, name: name, description: description,
+        created_at: created_at, updated_at: updated_at,
+        metric_ids: metric_ids
+      }
+    end
+  end
+end

data/app/models/completion_kit/metric_group_membership.rb

@@ -0,0 +1,20 @@
+module CompletionKit
+  class MetricGroupMembership < ApplicationRecord
+    self.table_name = "completion_kit_metric_group_memberships"
+
+    belongs_to :metric_group, class_name: "CompletionKit::MetricGroup", foreign_key: "metric_group_id"
+    belongs_to :metric
+
+    validates :metric_id, uniqueness: { scope: :metric_group_id }
+
+    before_validation :set_default_position
+
+    private
+
+    def set_default_position
+      return if position.present? || metric_group.blank?
+
+      self.position = metric_group.metric_group_memberships.maximum(:position).to_i + 1
+    end
+  end
+end

data/app/models/completion_kit/model.rb

@@ -3,7 +3,7 @@ module CompletionKit
     STATUSES = %w[active retired failed].freeze
 
     validates :provider, presence: true
-    validates :model_id, presence: true, uniqueness: { scope: :provider }
+    validates :model_id, presence: true, tenant_scoped_uniqueness: { scope: :provider }
     validates :status, presence: true, inclusion: { in: STATUSES }
 
     scope :active, -> { where(status: "active") }

data/app/models/completion_kit/provider_credential.rb

@@ -22,7 +22,8 @@ module CompletionKit
       PROVIDER_LABELS[provider] || provider.titleize
     end
 
-    validates :provider, presence: true, inclusion: { in: PROVIDERS }, uniqueness: true
+    validates :provider, presence: true, inclusion: { in: PROVIDERS }
+    validates :provider, tenant_scoped_uniqueness: true
 
     after_save :enqueue_discovery