truffler 0.1.4 → 0.1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3be74707295fff06a13ad0b12bbfb2d876f851893f0fff5f3f49b14c368bdb9b
-  data.tar.gz: 59a2e7d7e57669e8ffa35eadf656cc4e52fbae6cc47798f9ae4d01a43f870109
+  metadata.gz: 1e678dd275fc608a97a93cc9eb7a881ae686c4cc428bfe2a823310f2d5a4b559
+  data.tar.gz: 3145a1087183f63dae1baf3bd38237761760f368b79828860af802c81caea690
 SHA512:
-  metadata.gz: e71fd94ea16d17b9ca2e2f219591b51dfc94ab82f749bd21d5d277389039e6fa30f3c29130dd07c2b8aecd588abe0e1d91e8c6e70b66d199b485a77a90cc0a59
-  data.tar.gz: b3ea709b2d26f377b3c7731e7cc020d5249780bfbc756ad1f01dde3506a542f80b34bc7e8846387527d9311cbbe0015c826ea90916fdd251fdaa596efef123c6
+  metadata.gz: ebe167d14a66fc74e219e1a5f9206ba158018418b87437b212e51eb6e356c27dd35a5312e443f6cd108a21f57c223e7219c39f5ed45d44ac643aae1feb08a103
+  data.tar.gz: 2352317c03e9c9e8b91597efdf3aab02ea09ca54bb9ff3bb9bcf1fc0a214d1b23d62f1542ef3c7f6866e720f7e7624aa9eab23bea04fe6fe86c649a20e5faaaf

data/CHANGELOG.md

@@ -1,5 +1,37 @@
 # Changelog
 
+## [0.1.6]
+
+A fix from happyhappy production and follow-ups from 0.1.5.
+
+- Zero-result relaxation. With a Cora product filter applied, the word "email" became a `Source: email` filter chip and results dropped from 50 to 0 because the tenant has no email sources. When a keystroke search under the encoding's filters returns nothing but would match without them, those filters now become soft boosts: they stay in the intent vector so matching records rank first, but nothing requires them. The words they consumed become keywords again. Truffler first relaxes only the filters no record in the tenant carries at their threshold and keeps the rest, then every filter if that still finds nothing. It never relaxes into "every record in the tenant" when no word, exact match, or vector match remains. The fallback costs one extra query, only on an empty result; non-empty keystrokes stay one SELECT. `Result#relaxed_labels` lists the relaxed storage keys, and their chips carry `relaxed: true`. Chips the searcher removed stay removed. Smart search candidate filtering falls back the same way instead of reranking an empty set, and records `relaxed_labels` on the run (`run.to_h`).
+- New `config.skip_empty_options` (default false). When true, query encoding offers Jev only the choice options the tenant has label rows for (at or above `choice_min_probability`) and leaves out choice labels with none. The present-option set is digested into the encoding cache key and cached for 5 minutes, so a new option reaches encoding within that TTL and keystrokes stay one SELECT.
+- `truffler:install` and `truffler:upgrade` migration templates pass rubocop-rails-omakase (`[ :a, :b ]` array brackets). A test renders every template and runs RuboCop with the omakase config on the output.
+- `Records::BackfillSpend` no longer caches a missing `truffler_backfill_spends.tenant_key` for the life of the process. On a miss it reloads the column information at most once a minute, so workers started before `db:migrate` move to tenant ledgers without a restart.
+- `Backfill.status` works with an `index_scope` that orders. It plucked distinct tenants over `index_relation` without `reorder(nil)`, which raised on Postgres (`for SELECT DISTINCT, ORDER BY expressions must appear in select list`).
+- A per-tenant choice `options:` callable runs once per label and tenant for each keystroke search, query encoding, labeler batch, and Smart run step, instead of at every vocabulary, fingerprint, and wording read. In happyhappy that was about 8 times per keystroke, and 14 per encoding and 21 per labeler batch in the test model. The new `Truffler::Current.scope` memoizes for one unit of work and clears when it ends, so nothing is shared between searches, jobs, or tenants.
+- Backfill spend ledgers key on the Jev-asked labels only (`Vocabulary#ledger_version`). Changing a supplied (`from:`) label, such as adding an option to a supplied product choice, no longer starts a fresh ledger; the backfill rewrites only that label with no Jev call and no charge. Changing an asked label still starts a new ledger. Existing ledgers keep counting: a model without supplied labels keeps the same key, and one with supplied labels has its pre-0.1.6 row taken over by the first backfill.
+
+## [0.1.5]
+
+Fixes from the Cora integration at Postgres scale, and two from happyhappy's 0.1.4 upgrade.
+
+- Tenant-scoped indexing: `index_if ->(record) { ... }` and `index_scope ->(relation) { ... }` in `truffler do`, plus `config.tenant_enabled = ->(model, tenant_key) { ... }`. Records outside them, or in a disabled tenant, are never labeled, embedded or backfilled. The after-commit hooks, `Labeling::Queue`, the labeler, both backfills, lens backfills, `LabelFlushJob`, `ResumeJob` and `EmbedJob` all honor them. `BackfillJob` takes `tenant_key:`, over-cap demotion and `ResumeJob` enqueue one job per tenant, and `truffler:backfill` and `truffler:status` accept `TENANT=`.
+- `Embeddings::Backfill` pages per tenant with a limited `NOT EXISTS` anti-join instead of a whole-table `NOT IN`.
+- The backfill spend ledger is per tenant for tenant-scoped models (`truffler_backfill_spends.tenant_key`; `config.backfill_spend_cap_scope = :tenant`, the default, or `:app`), so `backfill_spend_cap` applies to each tenant. Existing installs: `rails g truffler:upgrade && rails db:migrate`.
+- `rails g truffler:upgrade` is safe to rerun. It skips migrations already present or applied and writes only the missing ones, with no conflicts.
+- Choice labels store a row only for options at or above `config.choice_min_probability` (default 0.05; `nil` stores all), plus the most likely option. Missing options read as 0.0 everywhere, and no migration is needed.
+- An out-of-shape `from:` answer settles: nothing is stored for that label, the record is marked labeled, and `truffler.supplied_label_failed` fires with `permanent: true`. A raising `from:` still retries and fails after `max_attempts`.
+- Postgres: upserts no longer assign `updated_at` twice (Rails 8.1 raised `PG::SyntaxError` on every live enqueue and embedding write). The suite now also runs on Postgres with pgvector in CI (`TRUFFLER_DATABASE_URL`). As a side effect, re-enqueuing an already pending row no longer refreshes its `updated_at`.
+- Keystroke SQL at scale on Postgres. A `keyword` or `exact` callable that returns a relation runs once as `id = ANY(ARRAY(subquery))` instead of a hashed `IN` over every tenant row; returning an Array of ids is the documented fast path. Label-only and filtered searches add up label scores in one grouped subquery joined on `record_id` instead of a subquery per row: 120 ms p50 before, 51 ms after, for 50k records. New installs get `INCLUDE (record_id)` on `index_truffler_labels_for_search`, and `truffler:upgrade` adds it on existing Postgres installs. On Postgres, `NeighborStore` scores text from the tenant's top-K neighbors (`ORDER BY embedding <=> q LIMIT k`, default 200); `NeighborStore.new(k:, top_k:)` is new, and `config.vector_store` accepts a store instance. Records outside the top `k` get no text score. The README covers `SET LOCAL jit = off` for the keystroke transaction.
+- New `invite_on_pending_encoding` declaration (default true): a model with a `keyword` source, such as a blind index, still shows the `:encoding_pending` Smart search row while a first-time query's encoding is in flight. `Result#local_weak?` keeps the backup provider off when the keyword list is strong.
+- New `Truffler::Clients::Evaluator` for clients that read `schema.questions` and return an evaluation object without `to_h`, such as Cora's `TypeSafeClient`. The Cora checklist in `docs/host-integration.md` now shows it instead of the broken `Callable` line.
+- A query word that names any declared label key, option key or option search-text word, applied or not, is never dropped as filler. `email` and `emails` are no longer default `config.filler_words`.
+- Removing the last label chip no longer makes filler nouns required keywords while a time phrase still applies.
+- A lens backfill never pages disabled tenants (filtered in SQL, so a capped `LensBackfillJob` cannot refetch them forever), and the labeler never deletes a disabled tenant's state rows; it returns them to pending at backfill priority, so re-enabling relabels only what is stale.
+- Removing chips keeps declared label and option words (option search text included) as keywords, the same as the encoder and the cold-cache path.
+- `truffler:status` and `Backfill.status(model, tenant_key:)` count only records inside `index_scope` and enabled tenants, and only their state rows, so a finished partial rollout reports nothing missing or pending.
+
 ## [0.1.4]
 
 Fixes from happyhappy production.

data/README.md

@@ -51,6 +51,10 @@ bin/rails db:migrate
 
 The migration creates only that table and skips it if it already exists. Until you run it, backfills log one warning and cap spend per run, as 0.1.1 did.
 
+0.1.5 adds `truffler_backfill_spends.tenant_key` for per-tenant spend ledgers. The same `truffler:upgrade` command writes that migration. Until you run it, every tenant shares the model's app-wide ledger and a warning is logged once. Workers already running when you migrate pick up the column within a minute, without a restart.
+
+`truffler:upgrade` is safe to rerun. It skips any migration already in `db/migrate` or already applied to the database, and writes only the missing ones.
+
 Truffler digests user keys and query misses with `secret_key_base`. Rails supplies it automatically; outside Rails, set `config.secret_key_base`.
 
 ## Declaring a model
@@ -84,6 +88,7 @@ class Email < ApplicationRecord
     surface :inbox, explicit_action: :enter     # :enter, :key, or :row
     ranking label: 1.0, text: 1.0, keyword: 0.5, exact: 1.0
     weak_below 3                                # fewer keystroke results than this counts as weak
+    invite_on_pending_encoding true             # Smart search row while a first-time query's encoding is in flight
   end
 end
 ```
@@ -91,12 +96,13 @@ end
 Here is what each option does:
 
 - `filter_at` makes a label a hard filter at that probability when the query asks for it. `boost` is the label's weight when the query prefers it. `filter_weight:` (default 0) is how much intent weight a filter adds on top.
-- Choice `options:` may be a callable of the tenant key, which gives each tenant its own vocabulary. A tenant it gives no options (`{}` or nil) simply lacks the label: it is neither asked nor encoded there. The option name `truffler:none` is reserved (see `Truffler::NO_OPTION` below).
+- Choice `options:` may be a callable of the tenant key, which gives each tenant its own vocabulary. A tenant it gives no options (`{}` or nil) simply lacks the label: it is neither asked nor encoded there. The callable runs once per label and tenant for each keystroke search, query encoding, labeler batch, and Smart run step (`Truffler::Current.scope`), never shared between them. The option name `truffler:none` is reserved (see `Truffler::NO_OPTION` below).
 - An option's value may be `{ description: "...", search: "..." }` instead of a plain description, e.g. `options: { "prod-a1" => { description: "Cora, the AI email assistant that drafts replies...", search: "Cora email assistant" } }`. Jev labels records with the description; query words are matched against the short search text (else the description), and the request state's `option_names` carries it. A per-tenant callable may return the same shape. Editing a search text never stales labels or triggers a backfill; it only changes the query-encoding cache key, so queries are re-encoded.
 - `watch :column, ...` relabels every label when one of those columns changes. Saving a record only relabels on columns that changed, so a `reads` field backed by a method (a conversation built from messages, say) needs the columns it is built from in `watch`.
-- `keyword` also accepts a single callable, `->(scope, tokens) { relation }`.
+- `keyword` also accepts a single callable, `->(scope, tokens) { relation }`. It, and each `exact` callable, may return an Array of ids instead of a relation, which is far faster on a large tenant (see [Keystroke search at scale](#keystroke-search-at-scale)).
 - `embeddings column: :my_vector` searches a vector column you maintain yourself. Truffler never writes it.
 - `embeddings` refuses to send encrypted fields to the embedding provider unless you pass `allow_encrypted: true`.
+- `invite_on_pending_encoding` (default true) shows the Smart search row with reason `:encoding_pending` while a query's first encoding is in flight, even when a `keyword` source matched. A model with no `keyword` source always does. Pass `false` to invite only on weak or empty results.
 
 `label key, type, **options` takes:
 
@@ -149,7 +155,10 @@ end
 | `:choice` | one option string (meaning 1.0 for it), or `{ option => probability }` | one row per option, `label:option`; options left out store 0.0 |
 | `:score` | a level index into `legend:` | `index / (levels - 1)`, as for Jev scores |
 
-`nil` stores nothing: the record reads as missing that label, not as 0. An answer out of shape (an undeclared option, a probability outside 0..1, a level past the legend) or a `from:` that raises stores nothing for that label and emits `truffler.supplied_label_failed` with the label key and error class. Rows already stored keep serving until the next good write.
+`nil` stores nothing: the record reads as missing that label, not as 0. Both kinds of failure emit `truffler.supplied_label_failed` with the label key, the error class, and `permanent`:
+
+- An answer out of shape (an undeclared option, a probability outside 0..1, a level past the legend) is permanent (`permanent: true`). It is treated like `nil`: nothing is stored for that label, earlier rows are cleared, and the record is marked labeled, so backfills do not claim it again. A later valid answer, through `watch:` or `truffler_refresh_labels!`, is written as usual.
+- A `from:` that raises is transient (`permanent: false`). The record stays pending at backfill priority and is retried for free, then marked failed after `max_attempts`. Rows already stored keep serving until the next good write.
 
 Supplied labels never reach Jev. They are never in a labeling request, take no budget slot, cost nothing, and do not count toward a lens or backfill spend cap. A flush or backfill where only supplied labels are stale makes no Jev call. The labeling job writes them before it asks Jev anything, so a Jev outage or a budget denial never holds them back. Otherwise they behave like asked labels: same `truffler_labels` rows, the same filters, boosts, chips, and label vectors, and query encoding asks Jev how a query uses them (using `description:`), within the one encoding call per query.
 
@@ -168,11 +177,17 @@ The default client is `Truffler::Clients::RubyLLMTypeSafe`. It needs ruby_llm 2
 If you already have a TypeSafe client, wrap it in `Truffler::Clients::Callable`. The wrapped object must respond to `evaluate(state:, schema:)`, and may also accept `model:`. It returns either the answers hash or `{"answers" => ..., "model" => ..., "usage" => {"input_tokens" => ...}}`:
 
 ```ruby
-config.client = Truffler::Clients::Callable.new(TypeSafeClient.new)
+config.client = Truffler::Clients::Callable.new(MyJevClient.new)
 ```
 
 `schema:` holds the questions in TypeSafe wire shape. When a response carries no token count, Truffler estimates it from the request size and flags the estimate. Errors reach you as `Truffler::ClientError`, carrying only the HTTP status and the error class name.
 
+If your client takes a schema object (it calls `schema.questions`) and returns an evaluation object with `answers`, `model`, and `input_tokens` readers rather than a hash, use `Truffler::Clients::Evaluator` instead. For any other shape, subclass `Truffler::Clients::Base` and implement `perform` (see [docs/host-integration.md](docs/host-integration.md)).
+
+```ruby
+config.client = Truffler::Clients::Evaluator.new(TypeSafeClient.new)
+```
+
 ## Keystroke search
 
 ```ruby
@@ -184,17 +199,21 @@ result = Email.truffler(params[:q], tenant: Current.account.id, scope: Current.a
 
 A keystroke search makes no network call. It reads the query encoding and query vector from the cache. When the cache misses, it enqueues `EncodeQueryJob`, and the next keystroke or reload picks up the result.
 
-Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, or the chosen option key, ignoring case and plurals or sharing their first three letters when both words have four letters or more, so "angry" names `anger`; or, matched exactly, a word of that option's display name or description) counts as naming the label, and common stopwords and `config.filler_words` (generic nouns such as "customers", "users", "emails") are dropped. They are kept only when dropping them would leave no keyword, no applied label, and no time phrase, so "customers in the last 3 hours" lists the window's records while a lone "customers" still searches text; the same rule applies before the encoding is cached. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
+Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, or the chosen option key, ignoring case and plurals or sharing their first three letters when both words have four letters or more, so "angry" names `anger`; or, matched exactly, a word of that option's display name or description) counts as naming the label, and common stopwords and `config.filler_words` (generic nouns such as "customers", "users", "items") are dropped. A word that names any declared label (a word of its key, of an option key, or of an option's search text), applied or not, is never dropped as filler. Filler is kept only when dropping them would leave no keyword, no applied label, and no time phrase, so "customers in the last 3 hours" lists the window's records while a lone "customers" still searches text; the same rule applies before the encoding is cached. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
 
 When the searcher removes a chip, the words that named only that label (or only labels that are now all removed) become keywords again, so removing the `urgent` chip from "urgent refunds" searches for both words. A word that named a label or option key only by shared prefix ("urgently" for `urgent`) also adds a small keyword score (a quarter of the keyword weight) while its label applies, ranking records whose text contains it higher without requiring it.
 
+When a search under the encoding's filters returns nothing but would match without them, the filters relax instead of leaving an empty list. With a product filter applied, "email" can become a `Source: email` filter in a tenant with no email sources. Truffler first relaxes only the filters no record in the tenant carries at their threshold, keeping the rest (the product filter stays). If that still finds nothing, it relaxes every filter. A relaxed filter becomes a soft boost, so records that do match it still rank first, and the words it consumed become keywords again. A filter is never relaxed into "every record in the tenant": if no word comes back and there is no exact or vector match, the list stays empty. Relaxation runs one extra query, only when the result is empty. `result.relaxed_labels` lists the relaxed storage keys, and their chips keep `kind: :filter` with `relaxed: true`, so the UI can say "No email sources; showing keyword matches". Chips the searcher removed stay removed.
+
 Time phrases are handled in Ruby and never asked of Jev: `today`, `yesterday`, `this week`, `last week`, `this month`, `last month`, `past|last N hours|days|weeks|months`, `last hour`, `past hour`, `past week`, `past month`, and `since monday` through `since sunday`. "Past/last N units" is a rolling window ending now; "this/last week" and "this/last month" are calendar windows. The first one in a query limits results to records whose `arrived_at` column falls in that window, its words are not keywords, and it shows as a chip `{key: "time", label: "time", kind: :time, name: "This week"}`. Pass `"time"` in `suppressed:` to drop it. Weeks start on `Date.beginning_of_week`, and the window is computed from `Time.current` (or a `clock:` callable passed to the search, for tests).
 
 The returned `Truffler::Search::Result` exposes:
 
 - `records` and `ids`.
-- `chips`: `[{key:, label:, kind: :filter | :boost | :time, name:}]`.
+- `chips`: `[{key:, label:, kind: :filter | :boost | :time, name:}]`, plus `relaxed: true` on a filter chip that was relaxed.
+- `relaxed_labels`: storage keys of filters relaxed because they left nothing to show (empty otherwise).
 - `invite_row`: `{query:, reason: :weak | :empty | :encoding_pending}` or nil.
+- `local_weak?`: whether the list is too weak to stand alone (starts the backup provider). A pending encoding on a model with a `keyword` source invites Smart search without making the list weak.
 - `encoding_status`: `:cached`, `:pending`, or `:none`.
 - `watermark` and `new_matches_count`.
 - `explicit_action`: the surface's declared action.
@@ -207,6 +226,24 @@ For the "N new matches" row on a later request, pass the watermark back in:
 Email.jev_new_matches_count(params[:q], tenant: account.id, scope: account.emails, user: current_user, since: Time.iso8601(params[:since]))
 ```
 
+### Keystroke search at scale
+
+The keystroke query is shaped so a large tenant (tens of thousands of records) stays fast on Postgres:
+
+- Return ids from `keyword` and `exact` callables when you can, e.g. `.limit(2_000).pluck(:id)` from a blind index. Ids become a literal `IN (...)` list. A returned relation is rendered as `id = ANY(ARRAY(subquery))` on Postgres (plain `IN (subquery)` elsewhere), so it runs once instead of as a hashed filter over every tenant row, but the id list is still faster.
+- Label-only and filtered searches, where every record past the tenant and filters is a candidate, read label scores from one grouped aggregate over `truffler_labels` joined on `record_id`, never a subquery per row. New installs get `index_truffler_labels_for_search` with `INCLUDE (record_id)`, which makes that aggregate an index-only scan. On an existing Postgres install, `bin/rails generate truffler:upgrade && bin/rails db:migrate` adds it (built concurrently, then the old index is dropped). It is optional: the grouped aggregate already replaces the per-row subquery, and the index only saves heap reads.
+- With embeddings on Postgres, text similarity comes from the tenant's top `k` neighbors (`ORDER BY embedding <=> q LIMIT k`), joined on `record_id`. An HNSW index on `truffler_embeddings.embedding` serves it. Records outside the top `k` score no text similarity; see `vector_store` under [Configuration](#configuration).
+- The plans these queries produce are costed high enough that Postgres JIT often compiles them, which can add 0.3 to 0.9 s to a query that runs in tens of milliseconds. Turn JIT off for the keystroke only:
+
+  ```ruby
+  result = Email.transaction do
+    Email.connection.execute("SET LOCAL jit = off")
+    Email.truffler(params[:q], tenant: account.id, scope: account.emails, user: current_user)
+  end
+  ```
+
+  `SET LOCAL` lasts until the transaction ends, so other queries keep JIT. `result.records` is loaded inside the block.
+
 ## Smart search and streaming
 
 The searcher's explicit action (the `explicit_action` their surface declares) starts a Smart run:
@@ -226,7 +263,7 @@ The keystroke list is left untouched. The job then does the following:
 
 1. Starts the provider backup, if one is declared.
 2. Takes a rerank slot under the per-user cap. If no slot is available, the run pauses.
-3. Waits up to `encoding_deadline` for an in-flight query encoding and applies its filters.
+3. Waits up to `encoding_deadline` for an in-flight query encoding and applies its filters. If they leave no candidate, they relax as on the keystroke instead of reranking an empty set. `run.relaxed_labels` and `run.to_h[:relaxed_labels]` list them.
 4. Fans out one `RerankChunkJob` per `rerank_chunk_size` candidates.
 
 Each chunk appends to the buckets and pings. When the searcher edits the query or accepts or removes a chip, call `Email.jev_cancel_smart_search(tenant:, user:, surface:)`.
@@ -341,12 +378,12 @@ Set these in `Truffler.configure do |config| ... end`.
 | `max_field_chars`, `request_token_budget`, `max_questions_per_request` | 4,000, 48,000, 200 | Request packing limits. |
 | `queue_name` | `:default` | Queue for every Truffler job. |
 | `cost_per_million_tokens` | 0.042 | Jev input price, used in usage events and estimates. |
-| `backfill_spend_cap` | 5.0 | Dollar cap on backfill spend per model and vocabulary version, shared by `BackfillJob` chains, `ResumeJob` backfills, and `truffler:backfill` runs (see Backfill). `nil` disables it; for the rake task, `SPEND_CAP=none` does. Supplied labels cost nothing and are still written once it is reached. |
+| `backfill_spend_cap` | 5.0 | Dollar cap on backfill spend per model (per tenant for tenant-scoped models; see `backfill_spend_cap_scope`) and vocabulary version, shared by `BackfillJob` chains, `ResumeJob` backfills, and `truffler:backfill` runs (see Backfill). `nil` disables it; for the rake task, `SPEND_CAP=none` does. Supplied labels cost nothing and are still written once it is reached. |
 | `resume_pending_after` | 5 minutes | How long before `ResumeJob` treats work as stuck. |
 | `embedder` | `Embeddings::RubyLLMEmbedder.new` | Any `Embeddings::Embedder` subclass. The default calls `RubyLLM.embed`, which works on ruby_llm 1.x and 2. |
 | `embedding_cost_per_million_tokens` | 0.02 | Embedding price. |
-| `vector_store` | `:auto` | `:neighbor` (pgvector `<=>`, or sqlite-vec `vec_distance_cosine` when you load the extension), `:ruby` (exact cosine in Ruby), or `:auto` (neighbor when available, otherwise Ruby). A model declaring `embeddings column:` always reads its own column. |
-| `filler_words` | `customer(s) people person user(s) message(s) email(s) item(s) stuff thing(s)` | Generic nouns never required as keywords on their own (matched ignoring plurals). Replace the list or extend it (`config.filler_words += %w[ticket]`). |
+| `vector_store` | `:auto` | `:neighbor` (pgvector `<=>`, or sqlite-vec `vec_distance_cosine` when you load the extension), `:ruby` (exact cosine in Ruby), or `:auto` (neighbor when available, otherwise Ruby). Or a store instance, e.g. `Truffler::Embeddings::NeighborStore.new(k: 500)` or your own `VectorStore` subclass. On Postgres the neighbor store scores text from the tenant's top `k` (default 200) neighbors; `NeighborStore.new(top_k: false)` scores every row exactly. A model declaring `embeddings column:` always reads its own column. |
+| `filler_words` | `customer(s) people person user(s) message(s) item(s) stuff thing(s)` | Generic nouns never required as keywords on their own (matched ignoring plurals). Replace the list or extend it (`config.filler_words += %w[ticket]`). |
 | `encoding_prefetch` | `QueryEncoding::Prefetch.new` | Cache-miss hook, called as `call(model, query, cache_key:, tenant_key:, user_key:)`. |
 | `encoding_deadline` | 1.0 | Seconds a Smart run waits for an in-flight query encoding. |
 | `rerank_depth`, `rerank_chunk_size`, `rerank_max_field_chars` | 30, 10, 1,200 | Candidates reranked, candidates per Jev request, characters per field sent. |
@@ -358,6 +395,10 @@ Set these in `Truffler.configure do |config| ... end`.
 | `lenses.creators`, `lenses.authorize_lens`, `lenses.proposals` | `:developers`, nil, false | Lens policy (see Lenses). |
 | `lenses.spend_cap_usd`, `lenses.sample_size`, `lenses.max_questions`, `lenses.expire_after` | 1.0, 20, 8, 30 days | Lens limits. |
 | `lenses.generator`, `lenses.drafter_model`, `lenses.user_key` | `RubyLLMGenerator`, nil, `"User:42"` style | Drafting model seam and user key mapping. |
+| `tenant_enabled` | nil (every tenant) | `->(model, tenant_key) { ... }`, asked for tenant-scoped models only. A disabled tenant is never labeled, embedded, or backfilled (see Indexing only some tenants). |
+| `backfill_spend_cap_scope` | `:tenant` | `:tenant` keeps a spend ledger per tenant for tenant-scoped models, so `backfill_spend_cap` applies to each tenant. `:app` keeps one ledger per model. Unscoped models always use one. |
+| `choice_min_probability` | 0.05 | Choice labels store a row only for options at or above this probability, plus the most likely option. A missing option reads as 0.0 in filters, boosts, label vectors, and contributions. `nil` stores every option. No migration is needed: rows written earlier stay until their record is relabeled. |
+| `skip_empty_options` | false | When true, query encoding offers Jev only the choice options the tenant has label rows for (at or above `choice_min_probability`, or above 0.0 when that is `nil`), and leaves out choice labels with none, so a query cannot filter on an option no record carries. The present-option set is part of the encoding cache key and is read through the cache store for 5 minutes (`Truffler::QueryEncoding::PresentOptions::TTL`), so keystrokes stay one query and a new option reaches encoding within 5 minutes. |
 
 ## Jobs to schedule
 
@@ -396,9 +437,35 @@ truffler_expire_lenses:
 |---|---|
 | `SPEND_CAP=20` or `none` | Overrides `backfill_spend_cap` for this run. |
 | `MAX_DURATION=600` | Stops after that many seconds with `paused` and the cursor. Rerun to continue. |
-| `RESET_SPEND=1` | Zeroes the spend recorded for the current vocabulary version before starting. |
+| `RESET_SPEND=1` | Zeroes the spend recorded for the current vocabulary version before starting (every tenant's ledger when no `TENANT` is given). |
+| `TENANT=42` | Backfills only that tenant, against that tenant's ledger. `truffler:status` takes it too, to print that tenant's spend. |
+
+The spend cap holds across runs. Truffler records backfill spend per model and ledger version in `truffler_backfill_spends`, and reserves each request's estimate against the cap in SQL. A rerun, a `ResumeJob` backfill, and overlapping `BackfillJob` chains all draw on the same total, so together they stop at `backfill_spend_cap`. Changing what Jev is asked (a reworded question, a new asked label, or an activated lens) starts a new total. Supplied (`from:`) labels cost nothing and never do: adding an option to a supplied choice keeps the ledger, and the next backfill rewrites only that label with no Jev call. Ledgers written before 0.1.6 keep counting. A model without supplied labels keeps the same key, and one with supplied labels has its old row taken over on the first backfill, so no spend is reset. Lens backfills are capped separately by each lens's `spend_cap_usd`. `BackfillJob` reschedules itself after a denial with the same backoff. In code, `Truffler::Labeling::Backfill.new(Email).run(wait: true, max_duration: 600)` does what the task does.
+
+For tenant-scoped models the ledger is per tenant (`backfill_spend_cap_scope :tenant`, the default), so the cap applies to each tenant. A whole-model run skips a tenant once its ledger reaches the cap, keeps labeling the others, and ends with `spend_cap_reached`. A tenant run (`TENANT=`, or `BackfillJob.perform_later("Email", tenant_key: "42")`) stops at that tenant's cap. `ResumeJob` and the flush job's over-cap demotion enqueue one `BackfillJob` per tenant.
+
+### Indexing only some tenants
+
+For a per-account rollout, or to leave some records out, declare which records Truffler indexes:
+
+```ruby
+truffler do
+  tenant :account_id
+  # ...
+  index_if ->(email) { !email.spam? }                  # single records: after-commit hooks, Queue, EmbedJob
+  index_scope ->(relation) { relation.where(spam: false) } # batch paths: backfills and sweeps
+end
+
+Truffler.configure do |config|
+  config.tenant_enabled = ->(model, tenant_key) { Account.search_enabled?(tenant_key) }
+end
+```
 
-The spend cap holds across runs. Truffler records backfill spend per model and app-wide vocabulary version in `truffler_backfill_spends`, and reserves each request's estimate against the cap in SQL. A rerun, a `ResumeJob` backfill, and overlapping `BackfillJob` chains all draw on the same total, so together they stop at `backfill_spend_cap`. Changing the vocabulary (a reworded question, a new label, or an activated lens) starts a new total. Lens backfills are capped separately by each lens's `spend_cap_usd`. `BackfillJob` reschedules itself after a denial with the same backoff. In code, `Truffler::Labeling::Backfill.new(Email).run(wait: true, max_duration: 600)` does what the task does.
+- The after-commit label and embed hooks skip records that are not indexable: `index_if` rejects them or their tenant is disabled. No state row or job is created.
+- `Labeling::Backfill`, `Embeddings::Backfill`, and lens backfills page only over `index_scope` and skip disabled tenants. The labeler drops claimed records that are no longer indexable instead of labeling them.
+- `index_if` and `index_scope` should select the same records. Keep `index_scope` index-friendly, because backfills page over it.
+- The `ResumeJob` embedding sweep runs tenant by tenant over the enabled tenants that have records in `index_scope`. Each tenant pass is a `NOT EXISTS` anti-join limited to the sweep size, not a `NOT IN` over the table.
+- Stored labels of a tenant you disable stay in place and keep serving search. When you re-enable the tenant, run `truffler:backfill` with `TENANT=` for it.
 
 ## Privacy
 

data/lib/generators/truffler/install/templates/migration.rb.tt

@@ -9,9 +9,9 @@ class CreateTrufflerTables < ActiveRecord::Migration<%= migration_version %>
       t.string :fingerprint, null: false
       t.datetime :labeled_at, null: false
     end
-    add_index :truffler_labels, [:record_type, :record_id, :label_key], unique: true,
+    add_index :truffler_labels, [ :record_type, :record_id, :label_key ], unique: true,
       name: "index_truffler_labels_on_record_and_label"
-    add_index :truffler_labels, [:record_type, :tenant_key, :label_key, :value],
+    add_index :truffler_labels, [ :record_type, :tenant_key, :label_key, :value ], include: [ :record_id ],
       name: "index_truffler_labels_for_search"
 
     create_table :truffler_record_states do |t|
@@ -29,9 +29,9 @@ class CreateTrufflerTables < ActiveRecord::Migration<%= migration_version %>
       t.datetime :embedded_at
       t.timestamps
     end
-    add_index :truffler_record_states, [:record_type, :record_id], unique: true,
+    add_index :truffler_record_states, [ :record_type, :record_id ], unique: true,
       name: "index_truffler_record_states_on_record"
-    add_index :truffler_record_states, [:record_type, :tenant_key, :status, :priority],
+    add_index :truffler_record_states, [ :record_type, :tenant_key, :status, :priority ],
       name: "index_truffler_record_states_for_claims"
 
     create_table :truffler_embeddings do |t|
@@ -49,9 +49,9 @@ class CreateTrufflerTables < ActiveRecord::Migration<%= migration_version %>
       t.string :label_vocabulary_version
       t.timestamps
     end
-    add_index :truffler_embeddings, [:record_type, :record_id], unique: true,
+    add_index :truffler_embeddings, [ :record_type, :record_id ], unique: true,
       name: "index_truffler_embeddings_on_record"
-    add_index :truffler_embeddings, [:record_type, :tenant_key], name: "index_truffler_embeddings_on_tenant"
+    add_index :truffler_embeddings, [ :record_type, :tenant_key ], name: "index_truffler_embeddings_on_tenant"
 
     create_table :truffler_query_misses do |t|
       t.string :record_type, null: false
@@ -61,7 +61,7 @@ class CreateTrufflerTables < ActiveRecord::Migration<%= migration_version %>
       t.text :query_text
       t.datetime :created_at, null: false
     end
-    add_index :truffler_query_misses, [:record_type, :tenant_key, :created_at],
+    add_index :truffler_query_misses, [ :record_type, :tenant_key, :created_at ],
       name: "index_truffler_query_misses_on_tenant"
     add_index :truffler_query_misses, :created_at
 
@@ -86,7 +86,7 @@ class CreateTrufflerTables < ActiveRecord::Migration<%= migration_version %>
       t.datetime :last_used_at
       t.timestamps
     end
-    add_index :truffler_lenses, [:record_type, :status, :scope_type, :tenant_key],
+    add_index :truffler_lenses, [ :record_type, :status, :scope_type, :tenant_key ],
       name: "index_truffler_lenses_for_visibility"
 
     create_table :truffler_lens_versions do |t|
@@ -103,17 +103,20 @@ class CreateTrufflerTables < ActiveRecord::Migration<%= migration_version %>
       t.datetime :activated_at
       t.datetime :created_at, null: false
     end
-    add_index :truffler_lens_versions, [:lens_id, :number], unique: true,
+    add_index :truffler_lens_versions, [ :lens_id, :number ], unique: true,
       name: "index_truffler_lens_versions_on_lens_and_number"
 
     create_table :truffler_backfill_spends do |t|
       t.string :record_type, null: false
+      t.string :tenant_key
       t.string :vocabulary_version, null: false
       t.float :spent_usd, null: false, default: 0.0
       t.integer :requests, null: false, default: 0
       t.timestamps
     end
-    add_index :truffler_backfill_spends, [:record_type, :vocabulary_version], unique: true,
-      name: "index_truffler_backfill_spends_on_ledger"
+    add_index :truffler_backfill_spends, [ :record_type, :tenant_key, :vocabulary_version ], unique: true,
+      name: "index_truffler_backfill_spends_on_tenant_ledger"
+    add_index :truffler_backfill_spends, [ :record_type, :vocabulary_version ], unique: true, where: "tenant_key IS NULL",
+      name: "index_truffler_backfill_spends_on_app_ledger"
   end
 end

data/lib/generators/truffler/upgrade/templates/backfill_spends_migration.rb.tt

@@ -9,7 +9,7 @@ class CreateTrufflerBackfillSpends < ActiveRecord::Migration<%= migration_versio
       t.integer :requests, null: false, default: 0
       t.timestamps
     end
-    add_index :truffler_backfill_spends, [:record_type, :vocabulary_version], unique: true,
+    add_index :truffler_backfill_spends, [ :record_type, :vocabulary_version ], unique: true,
       name: "index_truffler_backfill_spends_on_ledger"
   end
 

data/lib/generators/truffler/upgrade/templates/backfill_spends_tenant_key_migration.rb.tt

@@ -0,0 +1,25 @@
+class AddTenantKeyToTrufflerBackfillSpends < ActiveRecord::Migration<%= migration_version %>
+  def up
+    add_column :truffler_backfill_spends, :tenant_key, :string unless column_exists?(:truffler_backfill_spends, :tenant_key)
+    if index_name_exists?(:truffler_backfill_spends, "index_truffler_backfill_spends_on_ledger")
+      remove_index :truffler_backfill_spends, name: "index_truffler_backfill_spends_on_ledger"
+    end
+    unless index_name_exists?(:truffler_backfill_spends, "index_truffler_backfill_spends_on_tenant_ledger")
+      add_index :truffler_backfill_spends, [ :record_type, :tenant_key, :vocabulary_version ], unique: true,
+        name: "index_truffler_backfill_spends_on_tenant_ledger"
+    end
+    unless index_name_exists?(:truffler_backfill_spends, "index_truffler_backfill_spends_on_app_ledger")
+      add_index :truffler_backfill_spends, [ :record_type, :vocabulary_version ], unique: true, where: "tenant_key IS NULL",
+        name: "index_truffler_backfill_spends_on_app_ledger"
+    end
+  end
+
+  def down
+    execute "DELETE FROM truffler_backfill_spends WHERE tenant_key IS NOT NULL" if column_exists?(:truffler_backfill_spends, :tenant_key)
+    remove_index :truffler_backfill_spends, name: "index_truffler_backfill_spends_on_tenant_ledger", if_exists: true
+    remove_index :truffler_backfill_spends, name: "index_truffler_backfill_spends_on_app_ledger", if_exists: true
+    remove_column :truffler_backfill_spends, :tenant_key, if_exists: true
+    add_index :truffler_backfill_spends, [ :record_type, :vocabulary_version ], unique: true,
+      name: "index_truffler_backfill_spends_on_ledger", if_not_exists: true
+  end
+end

data/lib/generators/truffler/upgrade/templates/labels_search_covering_migration.rb.tt

@@ -0,0 +1,32 @@
+class CoverTrufflerLabelsForSearch < ActiveRecord::Migration<%= migration_version %>
+  disable_ddl_transaction!
+
+  COVERING = "index_truffler_labels_for_search_covering".freeze
+  COLUMNS = [ :record_type, :tenant_key, :label_key, :value ].freeze
+
+  def up
+    return unless postgres? && !covered?
+
+    add_index :truffler_labels, COLUMNS, include: [ :record_id ], name: COVERING, algorithm: :concurrently
+    remove_index :truffler_labels, name: "index_truffler_labels_for_search", algorithm: :concurrently, if_exists: true
+  end
+
+  def down
+    return unless postgres? && index_name_exists?(:truffler_labels, COVERING)
+
+    add_index :truffler_labels, COLUMNS, name: "index_truffler_labels_for_search", algorithm: :concurrently,
+      if_not_exists: true
+    remove_index :truffler_labels, name: COVERING, algorithm: :concurrently
+  end
+
+  private
+
+  def postgres?
+    connection.adapter_name.match?(/postg/i)
+  end
+
+  def covered?
+    index_name_exists?(:truffler_labels, COVERING) ||
+      select_value("SELECT indexdef FROM pg_indexes WHERE indexname = 'index_truffler_labels_for_search'").to_s.include?("INCLUDE")
+  end
+end

data/lib/generators/truffler/upgrade/upgrade_generator.rb

@@ -3,19 +3,65 @@ require "rails/generators/active_record"
 
 module Truffler
   module Generators
-    # Adds the tables a newer Truffler needs to an app installed with an older
-    # one. Each migration skips a table that already exists.
+    # Adds the tables and columns a newer Truffler needs to an app installed
+    # with an older one. Safe to rerun: a step is skipped when db/migrate
+    # already holds a migration of that name or the database already has
+    # what it adds, so only missing migrations are written. Each migration
+    # also skips a table, column, or index that already exists.
     class UpgradeGenerator < Rails::Generators::Base
       include ActiveRecord::Generators::Migration
 
       source_root File.expand_path("templates", __dir__)
 
+      class_attribute :schema_connection, default: -> { ActiveRecord::Base.connection }
+
       def create_backfill_spends_migration
-        migration_template "backfill_spends_migration.rb.tt", File.join(db_migrate_path, "create_truffler_backfill_spends.rb")
+        add_migration "backfill_spends_migration.rb.tt", "create_truffler_backfill_spends" do |connection|
+          connection.table_exists?(:truffler_backfill_spends)
+        end
+      end
+
+      def create_backfill_spends_tenant_key_migration
+        add_migration "backfill_spends_tenant_key_migration.rb.tt", "add_tenant_key_to_truffler_backfill_spends" do |connection|
+          connection.column_exists?(:truffler_backfill_spends, :tenant_key)
+        end
+      end
+
+      # Postgres only: the 0.1.5 covering index that makes label-only ranking
+      # an index-only scan. Other databases and installs that already have it
+      # skip the step.
+      def create_labels_search_covering_migration
+        add_migration "labels_search_covering_migration.rb.tt", "cover_truffler_labels_for_search" do |connection|
+          !connection.adapter_name.match?(/postg/i) || labels_search_covered?(connection)
+        end
       end
 
       private
 
+      def labels_search_covered?(connection)
+        connection.index_name_exists?(:truffler_labels, "index_truffler_labels_for_search_covering") ||
+          connection.select_value("SELECT indexdef FROM pg_indexes WHERE indexname = 'index_truffler_labels_for_search'").to_s.include?("INCLUDE")
+      end
+
+      def add_migration(template, name, &applied)
+        existing = self.class.migration_exists?(File.join(destination_root, db_migrate_path), name)
+        if existing
+          say_status :skip, "#{name} (#{File.basename(existing)} exists)", :yellow
+        elsif schema_applied?(&applied)
+          say_status :skip, "#{name} (already in the database)", :yellow
+        else
+          migration_template template, File.join(db_migrate_path, "#{name}.rb")
+        end
+      end
+
+      # Without a database (not configured, not created yet) every step counts
+      # as missing; the migrations themselves are no-ops where already applied.
+      def schema_applied?
+        yield schema_connection.call
+      rescue ActiveRecord::ActiveRecordError
+        false
+      end
+
       def migration_version
         "[#{ActiveRecord::Migration.current_version}]"
       end

data/lib/tasks/truffler.rake

@@ -27,40 +27,55 @@ namespace :truffler do
 
   describe_cursor = ->(cursor) { cursor.nil? ? "none" : cursor }
 
+  resolve_tenant = ->(value) { value.to_s.strip.presence }
+
+  describe_model = ->(model, tenant_key) { tenant_key ? "#{model.name} (tenant #{tenant_key})" : model.name }
+
   desc "Backfill stale, missing, and failed labels for a model, waiting out budget denials " \
-    "(SPEND_CAP=dollars or none; default config.backfill_spend_cap; MAX_DURATION=seconds; RESET_SPEND=1 for a fresh spend ledger)"
+    "(TENANT=key for one tenant; SPEND_CAP=dollars or none; default config.backfill_spend_cap; MAX_DURATION=seconds; " \
+    "RESET_SPEND=1 for a fresh spend ledger)"
   task :backfill, [ :model ] => :setup do |_, args|
     model = resolve_model.call(args[:model])
+    tenant_key = resolve_tenant.call(ENV.fetch("TENANT", nil))
+    name = describe_model.call(model, tenant_key)
     spend_cap = resolve_spend_cap.call(ENV.fetch("SPEND_CAP", nil))
     max_duration = resolve_max_duration.call(ENV.fetch("MAX_DURATION", nil))
     if ENV.fetch("RESET_SPEND", nil) == "1"
-      Truffler::Labeling::Backfill.reset_spend!(model)
-      puts "#{model.name}: fresh spend ledger for the current vocabulary version"
+      ledger_tenant = model.truffler_definition.ledger_tenant(tenant_key)
+      all_tenants = ledger_tenant.nil? && !model.truffler_definition.ledger_tenant("").nil?
+      Truffler::Labeling::Backfill.reset_spend!(model, tenant_key: ledger_tenant, all_tenants: all_tenants)
+      puts all_tenants ? "#{name}: fresh spend ledgers for every tenant" : "#{name}: fresh spend ledger for the current vocabulary version"
     end
     progress = lambda do |so_far, delay|
-      puts "#{model.name}: waiting #{format('%.1f', delay)}s for backfill budget (#{so_far.labeled} labeled, " \
+      puts "#{name}: waiting #{format('%.1f', delay)}s for backfill budget (#{so_far.labeled} labeled, " \
         "$#{format('%.6f', so_far.cost)} spent, cursor #{describe_cursor.call(so_far.cursor)})"
     end
-    result = Truffler::Labeling::Backfill.new(model, spend_cap: spend_cap).run(wait: true, max_duration: max_duration, progress: progress)
-    summary = "#{model.name}: #{result.status}, #{result.labeled} labeled in #{result.requests} requests, $#{format('%.6f', result.cost)}"
+    result = Truffler::Labeling::Backfill.new(model, tenant_key: tenant_key, spend_cap: spend_cap)
+      .run(wait: true, max_duration: max_duration, progress: progress)
+    summary = "#{name}: #{result.status}, #{result.labeled} labeled in #{result.requests} requests, $#{format('%.6f', result.cost)}"
     summary += ", cursor #{describe_cursor.call(result.cursor)}" if result.status == :paused
     puts summary
   end
 
-  print_status = lambda do |model|
-    puts model.name
+  print_status = lambda do |model, tenant_key = nil|
+    puts describe_model.call(model, tenant_key)
     Truffler::Labeling::Backfill.status(model).each { |key, count| puts format("  %-9s %d", key, count) }
-    if Truffler::Records::BackfillSpend.available?
-      ledger = Truffler::Labeling::Backfill.spend(model)
+    ledger_tenant = model.truffler_definition.ledger_tenant(tenant_key)
+    if Truffler::Records::BackfillSpend.available? && Truffler::Records::BackfillSpend.tenant_ledgers? &&
+        model.truffler_definition.ledger_tenant("") && ledger_tenant.nil?
+      puts format("  %-9s %s", "spent", "per tenant; pass TENANT=key")
+    elsif Truffler::Records::BackfillSpend.available?
+      ledger = Truffler::Labeling::Backfill.spend(model, tenant_key: ledger_tenant)
       cap = Truffler.config.backfill_spend_cap
       puts format("  %-9s $%.6f in %d requests (vocabulary %s, cap %s)", "spent", ledger&.spent_usd.to_f, ledger&.requests.to_i,
-        Truffler::Labeling::Backfill.ledger_version(model).first(12), cap ? format("$%.2f", cap) : "none")
+        Truffler::Labeling::Backfill.ledger_version(model, ledger_tenant).first(12), cap ? format("$%.2f", cap) : "none")
     else
       puts format("  %-9s %s", "spent", "not tracked across runs; run bin/rails g truffler:upgrade && bin/rails db:migrate")
     end
   end
 
-  desc "Print a model's labeling counts by status and staleness (every registered Truffler model when none is named)"
+  desc "Print a model's labeling counts by status and staleness (every registered Truffler model when none is named; " \
+    "TENANT=key for a tenant's spend ledger)"
   task :status, [ :model ] => :setup do |_, args|
     if args[:model].to_s.strip.empty?
       Rails.application.eager_load! if defined?(Rails.application) && Rails.application
@@ -68,7 +83,7 @@ namespace :truffler do
       abort "No Truffler models are registered" if models.empty?
       models.each(&print_status)
     else
-      print_status.call(resolve_model.call(args[:model]))
+      print_status.call(resolve_model.call(args[:model]), resolve_tenant.call(ENV.fetch("TENANT", nil)))
     end
   end
 end

data/lib/truffler/benchmark/runner.rb

@@ -223,7 +223,7 @@ module Truffler
         failures = @dataset.injections.filter_map do |injection|
           clean = stored[injection.clean_id]
           twin = stored[injection.record.id]
-          changed = (clean.keys | twin.keys).reject { |key| Metrics.agree?(clean[key], twin[key], tolerance) }
+          changed = (clean.keys | twin.keys).reject { |key| Metrics.agree?(clean.fetch(key, 0.0), twin.fetch(key, 0.0), tolerance) }
           next if changed.empty?
 
           { "fixture" => injection.id, "clean_id" => injection.clean_id, "twin_id" => injection.record.id,

data/lib/truffler/clients/evaluator.rb

@@ -0,0 +1,44 @@
+module Truffler
+  module Clients
+    # Wraps a host client whose `evaluate(state:, schema:)` takes a schema
+    # object (it reads `schema.questions`) and returns an evaluation object
+    # with `answers`, `model`, and `input_tokens` (or a `usage` hash) readers
+    # instead of a hash, such as Cora's TypeSafeClient and its
+    # TypeSafe::Evaluation. The schema also reads like the question hash, so
+    # a client that treats it as a Hash works too. The pinned model is passed
+    # when the host's method accepts `model:`.
+    #
+    #   config.client = Truffler::Clients::Evaluator.new(TypeSafeClient.new)
+    class Evaluator < Callable
+      Schema = Data.define(:questions) do
+        delegate :[], :each, :keys, :size, :empty?, :as_json, :to_json, to: :questions
+
+        def ids
+          questions.keys
+        end
+
+        def to_h
+          questions
+        end
+      end
+
+      def perform(state:, questions:, model:)
+        response(super(state: state, questions: Schema.new(questions: questions), model: model))
+      end
+
+      private
+
+      def response(evaluation)
+        return evaluation if evaluation.is_a?(Hash)
+
+        tokens = evaluation.try(:input_tokens) || usage_tokens(evaluation.try(:usage))
+        { "answers" => evaluation.answers.to_h, "model" => evaluation.try(:model),
+          "usage" => ({ "input_tokens" => tokens } if tokens) }.compact
+      end
+
+      def usage_tokens(usage)
+        usage.is_a?(Hash) ? usage.with_indifferent_access[:input_tokens] : usage.try(:input_tokens)
+      end
+    end
+  end
+end