truffler 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 28c9f27b6811a398815e7cc1026b0d1ed0ef1caf546bd820a59be4efdd4dcffa
+  data.tar.gz: 1889959cc8dfd3f32af19a37dda640673e08d6459f5a5e7722747cd32a7422fd
+SHA512:
+  metadata.gz: 76974d97ae716de75fc3cc1992868c2cabbd94c6851d6a85267fb5c3c1d18ca21f2f998c6ae1d135192081bd7b544351801fc88d33cde4070c56464a01be4d98
+  data.tar.gz: 8a35786e72b7e5b2996ce915da8419cb8d0d0873c8839829155b7d831f043caf4df9a9be29b9219f75d2403abbbd2c915a09b06b31e0135566a0dbe53f14774e

data/CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+## 0.1.0
+
+- `truffler do ... end` declarations on Active Record models: tenant, fields, noul/choice/score labels, keyword and exact sources, embeddings, a provider, surfaces, ranking weights, and `weak_below`.
+- Labeling on commit through a tenant-grouped queue, with packed Jev requests, per-record fingerprints, relabeling when watched fields change, `ResumeJob`, and a spend-capped backfill (`rake truffler:backfill`, `rake truffler:status`).
+- Host-supplied label answers: `label ..., from: ->(record) { ... }` stores answers your app already computes (with optional `watch:`, `version:`, and `description:`) and never asks Jev about them: no request, no budget slot, no spend, and written even while Jev is down. They filter, boost, fill label vectors, and take part in query encoding like asked labels. `record.truffler_refresh_labels!` rewrites them on demand.
+- One account-wide Jev budget with headroom, priority ceilings (live, encode, rerank, backfill), per-user caps, and a per-tenant live cap.
+- Jev clients for `ruby_llm-typesafe`, a `Callable` for host clients, a fake, and cassettes. `truffler.*` notifications carry only ids, numbers, and digests.
+- Optional text embeddings, and label vectors stored as named-dimension embeddings. Vector stores for pgvector/sqlite-vec (`neighbor`), exact cosine in Ruby, or a host-maintained column.
+- Keystroke search in one SQL query with hybrid dot-product scoring, chips, the Smart search invite row, the encoding status, and a watermark for "new matches".
+- Query encoding with deduplicated prefetch, cached late answers, and per-user caching when lenses are visible.
+- Smart search: a candidate snapshot, chunked Jev rerank into append-only Strong/Possible/Unlikely buckets, pause and cancel, and data-free Action Cable pings, plus a generated `TrufflerChannel`.
+- Provider backup search for exact-text or weak-local queries, in its own section.
+- Query miss log with distinct-user gating, retention (`PruneQueryMissesJob`), and `rake truffler:suggestions`.
+- Lenses: drafting, preview, version comparison, activation with backfill, regenerate, restore, history, proposals from misses, expiry (`ExpireLensesJob`), a creators policy, and the `authorize_lens` hook.
+- `rake truffler:bench` benchmark with synthetic fixtures, cassette replay, packed-batch agreement, and injection checks.
+- Install generator, README, and host integration guide.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Kieran Klaassen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,425 @@
+# Truffler
+
+Intent search for Rails apps, built on your own database and TypeSafe Jev.
+
+When a record is saved, Truffler asks Jev the typed questions you declared about it (yes/no "nouls", choices, and scores) and stores each answer as a number in `truffler_labels`. Taken together, a record's answers form a named-dimension embedding: one float per label, and one per option for choice labels. Optional text embeddings sit beside them. A search query is encoded the same way: for each label, Jev decides whether the query filters on it, prefers it, or ignores it. Keystroke search then ranks records in one SQL query that adds up a weighted dot product over the label dimensions, text similarity, keyword hits, and exact-source hits:
+
+```
+score = w_label * SUM(intent weight * label value) + w_text * text similarity
+      + w_keyword * keyword hit + w_exact * exact hit
+```
+
+Truffler also provides:
+
+- **Smart search.** When the searcher asks for it explicitly, Jev reranks the top candidates in chunks. Results stream into Strong, Possible, and Unlikely buckets, announced by data-free Action Cable pings.
+- **Provider backup.** An external search (Gmail, for example) runs in its own section when the query asks for exact text or the local results are weak.
+- **Lenses.** A permitted user describes a kind of search in plain language. Truffler drafts label questions for it, previews them on a sample, and activates them as new, versioned dimensions in that user's scope.
+
+Truffler ships no UI components. It returns plain Ruby data, which your controllers pass to your own views. [docs/host-integration.md](docs/host-integration.md) maps every field to the UI state it drives.
+
+Requirements: Ruby 3.2+, Rails (Active Record, Active Job, Active Support) 7.2 or 8.x, a job backend, and a cache store shared by web and job processes that supports `increment` (Solid Cache, Redis, or Memcached).
+
+## Install
+
+```ruby
+# Gemfile
+gem "truffler"
+gem "ruby_llm-typesafe" # the default Jev client; skip it if you wrap your own client
+```
+
+```bash
+bin/rails generate truffler:install
+bin/rails db:migrate
+```
+
+The generator writes three files:
+
+- A migration that creates `truffler_labels`, `truffler_record_states`, `truffler_embeddings`, `truffler_query_misses`, `truffler_lenses`, and `truffler_lens_versions`.
+- `config/initializers/truffler.rb`.
+- `app/channels/truffler_channel.rb`.
+
+It takes two options. `--record-id-type=bigint|integer|string|uuid` must match your primary keys. `--vector-dimensions=N` stores embeddings in a pgvector column of that width; it needs Postgres and the `neighbor` gem, which provides the `t.vector` column type.
+
+Truffler digests user keys and query misses with `secret_key_base`. Rails supplies it automatically; outside Rails, set `config.secret_key_base`.
+
+## Declaring a model
+
+Include `Truffler::Model` and declare what Jev reads and asks:
+
+```ruby
+class Email < ApplicationRecord
+  include Truffler::Model
+
+  truffler do
+    tenant :account_id                          # every search and Jev request stays inside one tenant
+    reads :subject, :body, :sender_name         # the fields Jev sees
+
+    label :needs_action, :noul, question: "Does this email need the reader to act or reply?",
+      criteria: { true => "Asks for a reply, a decision, a payment, or a task", false => "FYI, receipts, newsletters" },
+      filter_at: 0.6, boost: 2.0
+    label :urgent, :noul, question: "Is this email time-sensitive?", boost: 2.0
+    label :category, :choice, question: "Which category fits this email?",
+      options: { billing: "Invoices, receipts, and payments", travel: "Trips and bookings", other: nil },
+      filter_at: 0.5
+    label :importance, :score, question: "How important is this email to the reader?",
+      legend: { 0 => "Ignorable", 1 => "Worth a look", 2 => "Must read" }
+
+    keyword :subject, :body                     # LIKE match on plaintext columns
+    exact :sender, ->(scope, token) { scope.where(sender_email: token) }
+    embeddings dimensions: 256                  # optional text embeddings
+    provider :gmail, label: "Gmail", search: ->(query, tenant:, user:) { GmailSearch.call(query, account_id: tenant, user_key: user) }
+    order :received_at, :desc
+    arrived_at :received_at                     # drives "N new matches" and lens samples (default created_at)
+    surface :inbox, explicit_action: :enter     # :enter, :key, or :row
+    ranking label: 1.0, text: 1.0, keyword: 0.5, exact: 1.0
+    weak_below 3                                # fewer keystroke results than this counts as weak
+  end
+end
+```
+
+Here is what each option does:
+
+- `filter_at` makes a label a hard filter at that probability when the query asks for it. `boost` is the label's weight when the query prefers it. `filter_weight:` (default 0) is how much intent weight a filter adds on top.
+- Choice `options:` may be a callable of the tenant key, which gives each tenant its own vocabulary.
+- `keyword` also accepts a single callable, `->(scope, tokens) { relation }`.
+- `embeddings column: :my_vector` searches a vector column you maintain yourself. Truffler never writes it.
+- `embeddings` refuses to send encrypted fields to the embedding provider unless you pass `allow_encrypted: true`.
+
+`label key, type, **options` takes:
+
+| Option | Applies to | Meaning |
+|---|---|---|
+| `question:` | asked labels (required); optional with `from:` | What Jev is asked about each record. |
+| `criteria:` | `:noul` | `{ true => "...", false => "..." }` guidance for Jev. |
+| `options:` | `:choice` (required) | Option names, `{ option => description }`, or a callable of the tenant key. |
+| `legend:` | `:score` (required) | Two or more ordered levels, as an array or `{ index => description }`. |
+| `filter_at:`, `boost:`, `filter_weight:` | all | Filter threshold, boost weight, and the intent weight a filter adds. |
+| `description:` | all | The label's wording in query encoding. Defaults to the question, then to the key. |
+| `from:` | all | `->(record) { answer }`. Makes the label host-supplied; Jev is never asked. See [Labels you already compute](#labels-you-already-compute). |
+| `watch:` | with `from:` | Extra columns whose change refreshes the label, in addition to `reads`. |
+| `version:` | with `from:` | Any value; changing it rewrites the label for every record on the next backfill. |
+
+Keys must be lowercase snake case without a double underscore, and `lens` is reserved.
+
+Saving a record enqueues labeling (and embedding, when enabled) after commit, but only when a field in `reads`, the tenant column, or a supplied label's `watch:` column changed. Destroying a record removes its labels, state, and embeddings.
+
+### Labels you already compute
+
+If your app already classifies records, declare those answers with `from:` instead of asking Jev again. happyhappy, for example, stores sentiment, anger, category, product, actionability, and author role on each item's classification:
+
+```ruby
+class Item < ApplicationRecord
+  include Truffler::Model
+
+  truffler do
+    tenant :workspace_id
+    reads :title, :body
+
+    label :sentiment, :choice, options: %w[positive neutral negative mixed],
+      from: ->(item) { item.classification&.sentiment }, filter_at: 0.5
+    label :anger, :noul, from: ->(item) { item.classification&.anger }, boost: 2.0
+    label :category, :choice, options: ->(workspace_id) { Category.names_for(workspace_id) },
+      from: ->(item) { item.classification&.category_probabilities }, description: "what the feedback is about"
+    label :actionability, :score, legend: %w[none vague clear], from: ->(item) { item.classification&.actionability },
+      watch: [ :triaged_at ], version: 2
+
+    label :needs_reply, :noul, question: "Does this item ask the team for a reply?"   # still asked of Jev
+  end
+end
+```
+
+`from:` receives the record and returns the answer in the shape Jev answers are stored in:
+
+| Type | `from:` returns | Stored |
+|---|---|---|
+| `:noul` | a probability from 0 to 1, or `true`/`false` | the probability under `label` |
+| `:choice` | one option string (meaning 1.0 for it), or `{ option => probability }` | one row per option, `label:option`; options left out store 0.0 |
+| `:score` | a level index into `legend:` | `index / (levels - 1)`, as for Jev scores |
+
+`nil` stores nothing: the record reads as missing that label, not as 0. An answer out of shape (an undeclared option, a probability outside 0..1, a level past the legend) or a `from:` that raises stores nothing for that label and emits `truffler.supplied_label_failed` with the label key and error class. Rows already stored keep serving until the next good write.
+
+Supplied labels never reach Jev. They are never in a labeling request, take no budget slot, cost nothing, and do not count toward a lens or backfill spend cap. A flush or backfill where only supplied labels are stale makes no Jev call. The labeling job writes them before it asks Jev anything, so a Jev outage or a budget denial never holds them back. Otherwise they behave like asked labels: same `truffler_labels` rows, the same filters, boosts, chips, and label vectors, and query encoding asks Jev how a query uses them (using `description:`), within the one encoding call per query.
+
+To keep them fresh:
+
+- Changing a `reads` field, the tenant column, or one of the label's `watch:` columns refreshes it after commit. Stored values keep serving until the labeling job rewrites them.
+- Call `record.truffler_refresh_labels!` when the answers change somewhere truffler cannot see, such as the job that runs your classifier after insert. It rewrites the record's supplied labels immediately, with no Jev call.
+- Bump `version:` when the logic behind `from:` changes. That changes the vocabulary version, so `rake truffler:backfill` rewrites the label for every record, again at no cost.
+
+A supplied label's fingerprint digests its type, options, legend, description, and `version:`, never the Jev model, so changing `config.model` does not rewrite supplied labels.
+
+## Clients
+
+The default client is `Truffler::Clients::RubyLLMTypeSafe`. It needs ruby_llm 2 and `ruby_llm-typesafe`, and it calls `RubyLLM.chat(model:, provider: :typesafe).with_schema(...)`.
+
+If you already have a TypeSafe client, wrap it in `Truffler::Clients::Callable`. The wrapped object must respond to `evaluate(state:, schema:)`, and may also accept `model:`. It returns either the answers hash or `{"answers" => ..., "model" => ..., "usage" => {"input_tokens" => ...}}`:
+
+```ruby
+config.client = Truffler::Clients::Callable.new(TypeSafeClient.new)
+```
+
+`schema:` holds the questions in TypeSafe wire shape. When a response carries no token count, Truffler estimates it from the request size and flags the estimate. Errors reach you as `Truffler::ClientError`, carrying only the HTTP status and the error class name.
+
+## Keystroke search
+
+```ruby
+result = Email.truffler(params[:q], tenant: Current.account.id, scope: Current.account.emails, user: Current.user,
+  suppressed: params[:removed_chips], surface: :inbox)
+```
+
+`tenant:` is required on a model that declares a tenant. `scope:` must be a relation of the model, and results never leave it. `suppressed:` takes the keys of chips the searcher removed. The call also accepts `limit:` (default 50) and `weights:`, which overrides `ranking` for one call.
+
+A keystroke search makes no network call. It reads the query encoding and query vector from the cache. When the cache misses, it enqueues `EncodeQueryJob`, and the next keystroke or reload picks up the result.
+
+The returned `Truffler::Search::Result` exposes:
+
+- `records` and `ids`.
+- `chips`: `[{key:, label:, kind: :filter | :boost, name:}]`.
+- `invite_row`: `{query:, reason: :weak | :empty | :encoding_pending}` or nil.
+- `encoding_status`: `:cached`, `:pending`, or `:none`.
+- `watermark` and `new_matches_count`.
+- `explicit_action`: the surface's declared action.
+- `score(record)`, `breakdown(record)`, and `contributions(record)`, for debugging.
+- `promoted_ids(run)` and `smart_ranking_paused?(run)`, used alongside a Smart run.
+
+For the "N new matches" row on a later request, pass the watermark back in:
+
+```ruby
+Email.jev_new_matches_count(params[:q], tenant: account.id, scope: account.emails, user: current_user, since: Time.iso8601(params[:since]))
+```
+
+## Smart search and streaming
+
+The searcher's explicit action (the `explicit_action` their surface declares) starts a Smart run:
+
+```ruby
+run = Email.jev_smart_search(params[:q], tenant: account.id, scope: account.emails, user: Current.user,
+  surface: :inbox, suppressed: params[:removed_chips])
+```
+
+Starting a run happens in three steps:
+
+1. Truffler takes a snapshot of candidate ids from your scope. The keystroke ranking comes first, topped up with the newest records when no encoding exists yet.
+2. It supersedes the searcher's previous run.
+3. It enqueues `SmartSearchJob`.
+
+The keystroke list is left untouched. The job then does the following:
+
+1. Starts the provider backup, if one is declared.
+2. Takes a rerank slot under the per-user cap. If no slot is available, the run pauses.
+3. Waits up to `encoding_deadline` for an in-flight query encoding and applies its filters.
+4. Fans out one `RerankChunkJob` per `rerank_chunk_size` candidates.
+
+Each chunk appends to the buckets and pings. When the searcher edits the query or accepts or removes a chip, call `Email.jev_cancel_smart_search(tenant:, user:, surface:)`.
+
+Runs live in the cache store for `smart_run_ttl`. Load one with `Truffler::SmartSearch.find(run_id, user: Current.user, tenant: Current.account.id)`, passing the same `user:` and `tenant:` you gave `jev_smart_search`. A run that belongs to another searcher or tenant reads as expired, so a leaked run id shows nothing. `run.to_h` is the whole `smart` prop: ids, scores, and states, never record data.
+
+### Pings and Inertia partial reloads
+
+The generated `TrufflerChannel` streams `truffler:<user_key>`. The user key is `"User:42"` for a record and `to_s` for anything else, and it must match the `user:` you pass to searches. Each ping is `{run_id, section, changed_at}` with `section` either `"smart"` or `"provider"`. No result data travels over the socket.
+
+The client ignores pings for other runs and reloads only the Smart prop, following happyhappy's `use-mood-stream.ts` pattern:
+
+```ts
+consumer.subscriptions.create({ channel: "TrufflerChannel" }, {
+  received({ run_id }: { run_id: string; section: string }) {
+    if (run_id === currentRunId) router.reload({ only: ["smart"] })
+  },
+})
+```
+
+On the server, the controller renders the run as a lazy prop, so a partial reload recomputes only that prop:
+
+```ruby
+run = params[:run_id] && Truffler::SmartSearch.find(params[:run_id], user: Current.user, tenant: account.id)
+
+render inertia: "Emails/Index", props: {
+  emails: -> { serialize(result.records) },
+  chips: result.chips,
+  invite_row: result.invite_row,
+  smart: -> { run&.to_h },
+}
+```
+
+The provider section is part of `smart` (`sections.provider`), so reloading `smart` covers both kinds of ping. To render a bucket, load its ids through your own scope, for example `account.emails.where(id: ids)`, and keep the order the ids arrive in. To route pings through something other than `ActionCable.server`, set `config.broadcaster` to any object that responds to `broadcast(stream, payload)`.
+
+## Provider backup
+
+A model can declare one `provider`. Its `search:` callable receives `(query, tenant:, user:)` and returns an array of result hashes. The provider runs during the Smart run under either of two conditions:
+
+- **Exact text.** The query contains a quoted phrase, digits, an email address, or an identifier such as `INV-4471`.
+- **Weak local results.** The keystroke results were weak.
+
+Its state appears in `run.provider_section`, which has one of the statuses `:absent`, `:pending`, `:results`, `:empty`, or `:unavailable`. Each provider search is scoped to the run's own tenant and user. Results are stored in the cache store for the run's TTL, so return only the fields your UI needs.
+
+## Lenses
+
+A lens is a set of drafted label questions for one model and scope: the whole app (`Scope.app`), one tenant (`Scope.tenant(key)`), or one user within a tenant (`Scope.user(tenant_key, user_key)`). A lens's answers are stored as `lens:<id>:<label>` rows in `truffler_labels`. Filters, label vectors, staleness, backfill, and query encoding treat them exactly like declared labels.
+
+```ruby
+Lenses = Truffler::Lenses
+scope = Lenses::Scope.tenant(account.id)
+
+draft = Lenses::Drafter.draft("customers asking for a refund", model: Email, scope: scope)
+preview = Lenses::Previewer.preview(draft, relation: account.emails, by: Current.user)
+preview.distribution                # {label => {bucket => count}} on a sample of recent records
+preview.examples                    # up to 5 record ids per bucket
+preview.estimate                    # records, requests, input_tokens, cost_usd, duration_seconds, within_cap
+
+lens = Lenses::Activator.activate(draft, by: Current.user)   # enqueues LensBackfillJob
+
+version = lens.regenerate(by: Current.user, description: "customers asking for a refund or a credit")
+comparison = Lenses::Previewer.compare(version, lens, relation: account.emails, by: Current.user)
+comparison.shift                    # per-label bucket and mean movement on the same sample
+Lenses::Activator.activate(version, by: Current.user)
+
+lens.restore!(1, by: Current.user)  # copies version 1 forward as a new active version
+lens.history                        # number, status, label keys, reused keys, author digests, and times
+lens.promote!                       # prints `label ...` lines to paste into the model's declaration
+```
+
+Some rules that apply throughout:
+
+- The drafting model sees the description and the existing label vocabulary, never record text.
+- Search keeps using the active version until a new one is activated.
+- Labels written by an earlier version keep serving searches until they are relabeled.
+- Each lens has its own spend cap. Previews run at encode priority and fail with `LensSpendCapExceeded` when the lens would exceed its cap.
+- A lens nobody searches with for `expire_after` expires.
+- `Lenses::Proposer.propose(Email, tenant_key: account.id.to_s)` drafts `proposed` lenses from query-miss clusters when `config.lenses.proposals` is on. A proposed lens changes nothing until someone activates it.
+
+Who may change lenses is set by `config.lenses.creators`:
+
+- `:developers` (the default) allows app, tenant, and user lenses.
+- `:tenant_users` allows tenant and user lenses.
+- `:each_user` allows only personal lenses, and only for their own user.
+
+The `authorize_lens` hook always has the final word. Without it, every change raises `Truffler::NotAuthorized`:
+
+```ruby
+config.lenses.creators = :tenant_users
+config.lenses.authorize_lens = ->(user, scope) { user.admin? || scope.tenant_key == user.account_id.to_s }
+```
+
+For personal lenses, build the scope with the same key your searches use: `Scope.user(account.id, Truffler::Search::Keystroke.user_key(Current.user))`.
+
+## Configuration
+
+Set these in `Truffler.configure do |config| ... end`.
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `model` | `"jev-latest"` | Jev model pin. Changing it makes every asked label stale; supplied (`from:`) labels are unaffected. |
+| `client` | `Clients::RubyLLMTypeSafe.new` | Jev client (see Clients). |
+| `cache_store` | `Rails.cache` | Budget counters, encodings, Smart runs. Must be shared and support `increment`. |
+| `requests_per_minute` | 1,200 (`TYPESAFE_REQUESTS_PER_MINUTE`) | TypeSafe account limit. |
+| `headroom` | 0.25 | Share of the limit left for your app's other Jev calls. |
+| `priority_ceilings` | `{live: 1.0, encode: 0.9, rerank: 0.75, backfill: 0.5}` | Share of the gem's per-second budget each priority may fill. Lower priorities give way first. |
+| `user_caps` | `{encode: 30, rerank: 10}` | Per-user requests per minute. A denied rerank pauses the run. |
+| `tenant_live_cap` | 120 | Records per tenant per minute labeled live. The rest wait at backfill priority. |
+| `max_wait` | 5.0 | Seconds a live labeling call may wait for a budget slot. |
+| `batch_size`, `grouping_window` | 10, 0 | Records per labeling request and the flush delay. |
+| `max_attempts` | 5 | Labeling attempts per record before it is marked failed. |
+| `max_field_chars`, `request_token_budget`, `max_questions_per_request` | 4,000, 48,000, 200 | Request packing limits. |
+| `queue_name` | `:default` | Queue for every Truffler job. |
+| `cost_per_million_tokens` | 0.042 | Jev input price, used in usage events and estimates. |
+| `backfill_spend_cap` | nil | Dollar cap per backfill run. Supplied labels cost nothing and are still written once it is reached. |
+| `resume_pending_after` | 5 minutes | How long before `ResumeJob` treats work as stuck. |
+| `embedder` | `Embeddings::RubyLLMEmbedder.new` | Any `Embeddings::Embedder` subclass. The default calls `RubyLLM.embed`, which works on ruby_llm 1.x and 2. |
+| `embedding_cost_per_million_tokens` | 0.02 | Embedding price. |
+| `vector_store` | `:auto` | `:neighbor` (pgvector `<=>`, or sqlite-vec `vec_distance_cosine` when you load the extension), `:ruby` (exact cosine in Ruby), or `:auto` (neighbor when available, otherwise Ruby). A model declaring `embeddings column:` always reads its own column. |
+| `encoding_prefetch` | `QueryEncoding::Prefetch.new` | Cache-miss hook, called as `call(model, query, cache_key:, tenant_key:, user_key:)`. |
+| `encoding_deadline` | 1.0 | Seconds a Smart run waits for an in-flight query encoding. |
+| `rerank_depth`, `rerank_chunk_size`, `rerank_max_field_chars` | 30, 10, 1,200 | Candidates reranked, candidates per Jev request, characters per field sent. |
+| `smart_thresholds` | `{strong: 0.70, possible: 0.35}` | Bucket cutoffs on the relevance probability. |
+| `smart_run_ttl`, `smart_candidate_pool` | 15 minutes, 200 | Run lifetime and snapshot size cap. |
+| `broadcaster` | `ActionCable.server` | Ping target. |
+| `miss_retention`, `miss_min_distinct_users` | 30 days, 5 | Miss log retention, and the number of distinct users a miss cluster needs before anyone sees it. |
+| `secret_key_base` | Rails' | Keys user digests, miss digests, and Smart query encryption. |
+| `lenses.creators`, `lenses.authorize_lens`, `lenses.proposals` | `:developers`, nil, false | Lens policy (see Lenses). |
+| `lenses.spend_cap_usd`, `lenses.sample_size`, `lenses.max_questions`, `lenses.expire_after` | 1.0, 20, 8, 30 days | Lens limits. |
+| `lenses.generator`, `lenses.drafter_model`, `lenses.user_key` | `RubyLLMGenerator`, nil, `"User:42"` style | Drafting model seam and user key mapping. |
+
+## Jobs to schedule
+
+Truffler enqueues most of its own jobs. Run a worker for `config.queue_name` and schedule these yourself:
+
+| Job or task | When |
+|---|---|
+| `Truffler::Jobs::ResumeJob` | Every few minutes. Requeues failed and stuck labeling after an outage or a crashed worker, and enqueues up to 1,000 missing or stale embeddings per model per hour. |
+| `Truffler::Jobs::PruneQueryMissesJob` | Daily. Enforces `miss_retention`. |
+| `Truffler::Jobs::ExpireLensesJob` | Daily. Expires lenses unused for `lenses.expire_after`. |
+| `bin/rails "truffler:backfill[Email]"` (`SPEND_CAP=5`) or `Truffler::Jobs::BackfillJob.perform_later("Email")` | After adopting Truffler, changing a declaration, or changing the model pin. |
+| `Truffler::Embeddings::Backfill.new(Email).enqueue` | After enabling embeddings or changing the embedding model, width, or fields, to re-embed everything now instead of through the `ResumeJob` sweep. It enqueues 1,000 jobs at a time; pass `limit:` to cap the total. |
+
+`bin/rails "truffler:status[Email]"` prints labeling counts. `bin/rails "truffler:suggestions[Email]"` prints candidate questions drawn from logged query misses.
+
+With Solid Queue, for example:
+
+```yaml
+# config/recurring.yml
+truffler_resume:
+  class: Truffler::Jobs::ResumeJob
+  schedule: every 5 minutes
+truffler_prune_misses:
+  class: Truffler::Jobs::PruneQueryMissesJob
+  schedule: every day at 3am
+truffler_expire_lenses:
+  class: Truffler::Jobs::ExpireLensesJob
+  schedule: every day at 3am
+```
+
+## Privacy
+
+These guarantees hold for every model, and matter most for models that use Active Record encryption:
+
+- Decrypted text exists only in process memory during a labeling, embedding, encoding, rerank, or preview call.
+- Tables, job arguments, and logs carry only ids, numbers, and digests. The same goes for `truffler.*` notifications: `jev_call`, `search`, `smart_search`, `embed_call`, `lens_draft`, and the rest.
+- Client errors keep only the error class and HTTP status.
+- On encrypted models, pending query payloads and Smart run queries in the cache are encrypted with a key derived from `secret_key_base`. Search notifications carry a query digest instead of the query.
+- On encrypted models, the miss log and lens descriptions keep Active Record ciphertext when encryption is configured, and nothing otherwise. Miss clusters are shown only once they reach `miss_min_distinct_users`.
+- Embeddings refuse encrypted fields unless you opt in with `allow_encrypted: true`, or declare a `column:` you fill yourself.
+- `keyword` runs `LIKE` against columns, so it cannot see inside encrypted columns. Use `exact` with deterministic encryption for blind-index lookups.
+- A Smart rerank sends only candidates from the snapshot of your scope, one tenant per request, with fields truncated to `rerank_max_field_chars`. Query and record text appear only in the request state, marked as untrusted data.
+
+## Cost and rate limits
+
+Jev costs $0.042 per million input tokens. The default account limit is 1,200 requests per minute. With the default 25% headroom, Truffler takes up to 900 requests per minute, or 15 per second, split by priority ceilings in this order: live labeling, then query encoding, then rerank, then backfill.
+
+Here is what a typical session consumes:
+
+- **Labeling.** Each labeling request packs `batch_size` records from one tenant.
+- **Query encoding.** Each query is encoded once per vocabulary version and then cached.
+- **Smart search.** A run costs `rerank_depth / rerank_chunk_size` requests, 3 by default.
+
+Every call emits `truffler.jev_call` with input tokens (estimated and flagged when the client reports none), cost, model, and latency, so you can meter spend yourself. For a higher throughput ceiling, raise `TYPESAFE_REQUESTS_PER_MINUTE` once TypeSafe raises your account limit.
+
+## Benchmark
+
+```bash
+bundle exec rake truffler:bench                           # replay committed cassettes (CI runs this)
+bundle exec rake truffler:bench MODE=synthetic BENCH_RECORDS=5000
+bundle exec rake truffler:bench PARAMS=bench/params.yml OUT=tmp/bench.json
+TYPESAFE_API_KEY=... bundle exec rake truffler:bench MODE=record JEV=live
+```
+
+The benchmark loads fixtures into its own SQLite database and labels them through the real pipeline. It prints a JSON report with the following:
+
+- Recall and precision for intent and exact-text queries.
+- Keystroke p50 and p95 latency.
+- Labeling throughput.
+- Cost per labeled record and per query.
+- Packed-batch agreement and the adopted batch size.
+- Prompt-injection checks for labels and rerank.
+- Rerank bucket counts.
+
+It exits non-zero when a check fails. `bench/params.yml` holds the tunable thresholds, boosts, and blend weights.
+
+## Development
+
+```bash
+bundle install
+bundle exec rake test
+bundle exec rubocop
+```
+
+Tests run on in-memory SQLite with fake clients. Any live Jev call raises `Truffler::LiveCallInTest`.

data/bench/cassettes/004cad52c71fc8d847bf388eb9c2c288e1169b65ec4a28ce040cb7fea7c55f04.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"billing","confidence":0.8225,"probabilities":{"billing":0.8225,"newsletter":0.0444,"personal":0.0444,"travel":0.0444,"work":0.0444},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r001__needs_action":{"noul":0.0573,"type":"noul"},"r001__urgent":{"noul":0.0257,"type":"noul"}},"model":"synthetic-jev","request_hash":"004cad52c71fc8d847bf388eb9c2c288e1169b65ec4a28ce040cb7fea7c55f04","usage":{"input_tokens":436}}

data/bench/cassettes/0547d3d23c3fbaae86772ffa0e4e96475aa152a2556b8fa4bc7568effcb81a28.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"personal","confidence":0.7978,"probabilities":{"billing":0.0506,"newsletter":0.0506,"personal":0.7978,"travel":0.0506,"work":0.0506},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r001__needs_action":{"noul":0.8113,"type":"noul"},"r001__urgent":{"noul":0.3778,"type":"noul"},"r002__category":{"choice":"work","confidence":0.8259,"probabilities":{"billing":0.0435,"newsletter":0.0435,"personal":0.0435,"travel":0.0435,"work":0.8259},"type":"choice"},"r002__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r002__needs_action":{"noul":0.8905,"type":"noul"},"r002__urgent":{"noul":0.7213,"type":"noul"},"r003__category":{"choice":"billing","confidence":0.7929,"probabilities":{"billing":0.7929,"newsletter":0.0518,"personal":0.0518,"travel":0.0518,"work":0.0518},"type":"choice"},"r003__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r003__needs_action":{"noul":0.891,"type":"noul"},"r003__urgent":{"noul":0.4753,"type":"noul"},"r004__category":{"choice":"work","confidence":0.8143,"probabilities":{"billing":0.0464,"newsletter":0.0464,"personal":0.0464,"travel":0.0464,"work":0.8143},"type":"choice"},"r004__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r004__needs_action":{"noul":0.1103,"type":"noul"},"r004__urgent":{"noul":0.024,"type":"noul"},"r005__category":{"choice":"work","confidence":0.8285,"probabilities":{"billing":0.0429,"newsletter":0.0429,"personal":0.0429,"travel":0.0429,"work":0.8285},"type":"choice"},"r005__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r005__needs_action":{"noul":0.1037,"type":"noul"},"r005__urgent":{"noul":0.0564,"type":"noul"},"r006__category":{"choice":"personal","confidence":0.7725,"probabilities":{"billing":0.0569,"newsletter":0.0569,"personal":0.7725,"travel":0.0569,"work":0.0569},"type":"choice"},"r006__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r006__needs_action":{"noul":0.7797,"type":"noul"},"r006__urgent":{"noul":0.3717,"type":"noul"},"r007__category":{"choice":"travel","confidence":0.7932,"probabilities":{"billing":0.0517,"newsletter":0.0517,"personal":0.0517,"travel":0.7932,"work":0.0517},"type":"choice"},"r007__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r007__needs_action":{"noul":0.1218,"type":"noul"},"r007__urgent":{"noul":0.1914,"type":"noul"},"r008__category":{"choice":"travel","confidence":0.8153,"probabilities":{"billing":0.0462,"newsletter":0.0462,"personal":0.0462,"travel":0.8153,"work":0.0462},"type":"choice"},"r008__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r008__needs_action":{"noul":0.8375,"type":"noul"},"r008__urgent":{"noul":0.9082,"type":"noul"},"r009__category":{"choice":"travel","confidence":0.812,"probabilities":{"billing":0.047,"newsletter":0.047,"personal":0.047,"travel":0.812,"work":0.047},"type":"choice"},"r009__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r009__needs_action":{"noul":0.8489,"type":"noul"},"r009__urgent":{"noul":0.8866,"type":"noul"},"r010__category":{"choice":"work","confidence":0.8034,"probabilities":{"billing":0.0492,"newsletter":0.0492,"personal":0.0492,"travel":0.0492,"work":0.8034},"type":"choice"},"r010__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r010__needs_action":{"noul":0.8752,"type":"noul"},"r010__urgent":{"noul":0.6943,"type":"noul"}},"model":"synthetic-jev","request_hash":"0547d3d23c3fbaae86772ffa0e4e96475aa152a2556b8fa4bc7568effcb81a28","usage":{"input_tokens":3531}}

data/bench/cassettes/07aa23d166ab3500178e3f3c0838183051b093507bef88e1e2d29efdf5759ee6.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"travel","confidence":0.8125,"probabilities":{"billing":0.0469,"newsletter":0.0469,"personal":0.0469,"travel":0.8125,"work":0.0469},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r001__needs_action":{"noul":0.8712,"type":"noul"},"r001__urgent":{"noul":0.8741,"type":"noul"}},"model":"synthetic-jev","request_hash":"07aa23d166ab3500178e3f3c0838183051b093507bef88e1e2d29efdf5759ee6","usage":{"input_tokens":440}}

data/bench/cassettes/0858c5006c0bfac3e970ddaff3c139f6c110dbe3d8539fd95263706c637cdc8e.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"billing","confidence":0.8192,"probabilities":{"billing":0.8192,"newsletter":0.0452,"personal":0.0452,"travel":0.0452,"work":0.0452},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r001__needs_action":{"noul":0.9025,"type":"noul"},"r001__urgent":{"noul":0.8583,"type":"noul"},"r002__category":{"choice":"billing","confidence":0.7741,"probabilities":{"billing":0.7741,"newsletter":0.0565,"personal":0.0565,"travel":0.0565,"work":0.0565},"type":"choice"},"r002__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r002__needs_action":{"noul":0.065,"type":"noul"},"r002__urgent":{"noul":0.049,"type":"noul"},"r003__category":{"choice":"travel","confidence":0.8219,"probabilities":{"billing":0.0445,"newsletter":0.0445,"personal":0.0445,"travel":0.8219,"work":0.0445},"type":"choice"},"r003__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r003__needs_action":{"noul":0.1646,"type":"noul"},"r003__urgent":{"noul":0.1702,"type":"noul"},"r004__category":{"choice":"travel","confidence":0.8075,"probabilities":{"billing":0.0481,"newsletter":0.0481,"personal":0.0481,"travel":0.8075,"work":0.0481},"type":"choice"},"r004__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r004__needs_action":{"noul":0.8251,"type":"noul"},"r004__urgent":{"noul":0.8935,"type":"noul"},"r005__category":{"choice":"work","confidence":0.8234,"probabilities":{"billing":0.0441,"newsletter":0.0441,"personal":0.0441,"travel":0.0441,"work":0.8234},"type":"choice"},"r005__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r005__needs_action":{"noul":0.888,"type":"noul"},"r005__urgent":{"noul":0.6915,"type":"noul"},"r006__category":{"choice":"work","confidence":0.7839,"probabilities":{"billing":0.054,"newsletter":0.054,"personal":0.054,"travel":0.054,"work":0.7839},"type":"choice"},"r006__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r006__needs_action":{"noul":0.1048,"type":"noul"},"r006__urgent":{"noul":0.0444,"type":"noul"},"r007__category":{"choice":"personal","confidence":0.7725,"probabilities":{"billing":0.0569,"newsletter":0.0569,"personal":0.7725,"travel":0.0569,"work":0.0569},"type":"choice"},"r007__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r007__needs_action":{"noul":0.7797,"type":"noul"},"r007__urgent":{"noul":0.3717,"type":"noul"},"r008__category":{"choice":"personal","confidence":0.7886,"probabilities":{"billing":0.0529,"newsletter":0.0529,"personal":0.7886,"travel":0.0529,"work":0.0529},"type":"choice"},"r008__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r008__needs_action":{"noul":0.0428,"type":"noul"},"r008__urgent":{"noul":0.0375,"type":"noul"},"r009__category":{"choice":"newsletter","confidence":0.8016,"probabilities":{"billing":0.0496,"newsletter":0.8016,"personal":0.0496,"travel":0.0496,"work":0.0496},"type":"choice"},"r009__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r009__needs_action":{"noul":0.0394,"type":"noul"},"r009__urgent":{"noul":0.0,"type":"noul"},"r010__category":{"choice":"billing","confidence":0.7726,"probabilities":{"billing":0.7726,"newsletter":0.0569,"personal":0.0569,"travel":0.0569,"work":0.0569},"type":"choice"},"r010__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r010__needs_action":{"noul":0.8982,"type":"noul"},"r010__urgent":{"noul":0.8671,"type":"noul"}},"model":"synthetic-jev","request_hash":"0858c5006c0bfac3e970ddaff3c139f6c110dbe3d8539fd95263706c637cdc8e","usage":{"input_tokens":3521}}

data/bench/cassettes/0c0eb7c27d76bd0b9263690a63cd92d6b68c305826318555fd5b084a09970b78.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"personal","confidence":0.8027,"probabilities":{"billing":0.0493,"newsletter":0.0493,"personal":0.8027,"travel":0.0493,"work":0.0493},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r001__needs_action":{"noul":0.8213,"type":"noul"},"r001__urgent":{"noul":0.4195,"type":"noul"}},"model":"synthetic-jev","request_hash":"0c0eb7c27d76bd0b9263690a63cd92d6b68c305826318555fd5b084a09970b78","usage":{"input_tokens":430}}

data/bench/cassettes/0fada0c0a103882e2895f7f455d73d2cbe2b4a81bf743bb19b2b4f6e0130126b.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"work","confidence":0.7918,"probabilities":{"billing":0.0521,"newsletter":0.0521,"personal":0.0521,"travel":0.0521,"work":0.7918},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r001__needs_action":{"noul":0.0878,"type":"noul"},"r001__urgent":{"noul":0.034,"type":"noul"},"r002__category":{"choice":"personal","confidence":0.7842,"probabilities":{"billing":0.054,"newsletter":0.054,"personal":0.7842,"travel":0.054,"work":0.054},"type":"choice"},"r002__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r002__needs_action":{"noul":0.7832,"type":"noul"},"r002__urgent":{"noul":0.3762,"type":"noul"},"r003__category":{"choice":"personal","confidence":0.8173,"probabilities":{"billing":0.0457,"newsletter":0.0457,"personal":0.8173,"travel":0.0457,"work":0.0457},"type":"choice"},"r003__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r003__needs_action":{"noul":0.4445,"type":"noul"},"r003__urgent":{"noul":0.0372,"type":"noul"},"r004__category":{"choice":"newsletter","confidence":0.8104,"probabilities":{"billing":0.0474,"newsletter":0.8104,"personal":0.0474,"travel":0.0474,"work":0.0474},"type":"choice"},"r004__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r004__needs_action":{"noul":0.0467,"type":"noul"},"r004__urgent":{"noul":0.0067,"type":"noul"},"r005__category":{"choice":"personal","confidence":0.7808,"probabilities":{"billing":0.0548,"newsletter":0.0548,"personal":0.7808,"travel":0.0548,"work":0.0548},"type":"choice"},"r005__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r005__needs_action":{"noul":0.8272,"type":"noul"},"r005__urgent":{"noul":0.3827,"type":"noul"}},"model":"synthetic-jev","request_hash":"0fada0c0a103882e2895f7f455d73d2cbe2b4a81bf743bb19b2b4f6e0130126b","usage":{"input_tokens":1779}}

data/bench/cassettes/1117d79ac7c43b21beaf75e510461f91cd4ecc3cdbbe461f1794644af4aaf8f3.json

@@ -0,0 +1 @@
+{"answers":{"c001__relevance":{"noul":0.0,"type":"noul"},"c002__relevance":{"noul":0.0,"type":"noul"}},"model":"synthetic-jev","request_hash":"1117d79ac7c43b21beaf75e510461f91cd4ecc3cdbbe461f1794644af4aaf8f3","usage":{"input_tokens":436}}

data/bench/cassettes/11b7444a074cca4d0ef1535db0cf0edefe4c34eac4be3c593c9c2f33d3b038b8.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"work","confidence":0.7904,"probabilities":{"billing":0.0524,"newsletter":0.0524,"personal":0.0524,"travel":0.0524,"work":0.7904},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r001__needs_action":{"noul":0.8789,"type":"noul"},"r001__urgent":{"noul":0.6915,"type":"noul"},"r002__category":{"choice":"work","confidence":0.8104,"probabilities":{"billing":0.0474,"newsletter":0.0474,"personal":0.0474,"travel":0.0474,"work":0.8104},"type":"choice"},"r002__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r002__needs_action":{"noul":0.9278,"type":"noul"},"r002__urgent":{"noul":0.7102,"type":"noul"},"r003__category":{"choice":"work","confidence":0.8266,"probabilities":{"billing":0.0434,"newsletter":0.0434,"personal":0.0434,"travel":0.0434,"work":0.8266},"type":"choice"},"r003__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r003__needs_action":{"noul":0.8918,"type":"noul"},"r003__urgent":{"noul":0.6986,"type":"noul"},"r004__category":{"choice":"personal","confidence":0.8208,"probabilities":{"billing":0.0448,"newsletter":0.0448,"personal":0.8208,"travel":0.0448,"work":0.0448},"type":"choice"},"r004__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r004__needs_action":{"noul":0.0408,"type":"noul"},"r004__urgent":{"noul":0.0458,"type":"noul"},"r005__category":{"choice":"personal","confidence":0.7967,"probabilities":{"billing":0.0508,"newsletter":0.0508,"personal":0.7967,"travel":0.0508,"work":0.0508},"type":"choice"},"r005__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r005__needs_action":{"noul":0.0645,"type":"noul"},"r005__urgent":{"noul":0.0284,"type":"noul"},"r006__category":{"choice":"travel","confidence":0.7933,"probabilities":{"billing":0.0517,"newsletter":0.0517,"personal":0.0517,"travel":0.7933,"work":0.0517},"type":"choice"},"r006__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r006__needs_action":{"noul":0.1244,"type":"noul"},"r006__urgent":{"noul":0.1961,"type":"noul"},"r007__category":{"choice":"personal","confidence":0.7742,"probabilities":{"billing":0.0565,"newsletter":0.0565,"personal":0.7742,"travel":0.0565,"work":0.0565},"type":"choice"},"r007__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r007__needs_action":{"noul":0.8159,"type":"noul"},"r007__urgent":{"noul":0.4012,"type":"noul"},"r008__category":{"choice":"travel","confidence":0.7855,"probabilities":{"billing":0.0536,"newsletter":0.0536,"personal":0.0536,"travel":0.7855,"work":0.0536},"type":"choice"},"r008__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r008__needs_action":{"noul":0.8406,"type":"noul"},"r008__urgent":{"noul":0.8831,"type":"noul"},"r009__category":{"choice":"billing","confidence":0.7992,"probabilities":{"billing":0.7992,"newsletter":0.0502,"personal":0.0502,"travel":0.0502,"work":0.0502},"type":"choice"},"r009__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r009__needs_action":{"noul":0.9102,"type":"noul"},"r009__urgent":{"noul":0.8601,"type":"noul"},"r010__category":{"choice":"travel","confidence":0.8016,"probabilities":{"billing":0.0496,"newsletter":0.0496,"personal":0.0496,"travel":0.8016,"work":0.0496},"type":"choice"},"r010__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r010__needs_action":{"noul":0.8473,"type":"noul"},"r010__urgent":{"noul":0.9068,"type":"noul"}},"model":"synthetic-jev","request_hash":"11b7444a074cca4d0ef1535db0cf0edefe4c34eac4be3c593c9c2f33d3b038b8","usage":{"input_tokens":3511}}

data/bench/cassettes/11d00371d1b993675be00463159164536f1f8ed90a48652b0935eb9c36f8b684.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"personal","confidence":0.7831,"probabilities":{"billing":0.0542,"newsletter":0.0542,"personal":0.7831,"travel":0.0542,"work":0.0542},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r001__needs_action":{"noul":0.0485,"type":"noul"},"r001__urgent":{"noul":0.0715,"type":"noul"}},"model":"synthetic-jev","request_hash":"11d00371d1b993675be00463159164536f1f8ed90a48652b0935eb9c36f8b684","usage":{"input_tokens":425}}

data/bench/cassettes/1456ebe12616ce961d68a14eada62778a7271ebd3d1b4cb6eefd6857651c7afc.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"personal","confidence":0.7831,"probabilities":{"billing":0.0542,"newsletter":0.0542,"personal":0.7831,"travel":0.0542,"work":0.0542},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r001__needs_action":{"noul":0.0485,"type":"noul"},"r001__urgent":{"noul":0.0715,"type":"noul"}},"model":"synthetic-jev","request_hash":"1456ebe12616ce961d68a14eada62778a7271ebd3d1b4cb6eefd6857651c7afc","usage":{"input_tokens":424}}

data/bench/cassettes/14c3693f64b0e9ea7e470277fccae6ab501d404185eec5bb9074ef4b75c572c0.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"personal","confidence":0.7896,"probabilities":{"billing":0.0526,"newsletter":0.0526,"personal":0.7896,"travel":0.0526,"work":0.0526},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r001__needs_action":{"noul":0.82,"type":"noul"},"r001__urgent":{"noul":0.3702,"type":"noul"},"r002__category":{"choice":"work","confidence":0.7839,"probabilities":{"billing":0.054,"newsletter":0.054,"personal":0.054,"travel":0.054,"work":0.7839},"type":"choice"},"r002__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r002__needs_action":{"noul":0.1048,"type":"noul"},"r002__urgent":{"noul":0.0444,"type":"noul"},"r003__category":{"choice":"personal","confidence":0.8208,"probabilities":{"billing":0.0448,"newsletter":0.0448,"personal":0.8208,"travel":0.0448,"work":0.0448},"type":"choice"},"r003__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r003__needs_action":{"noul":0.0408,"type":"noul"},"r003__urgent":{"noul":0.0458,"type":"noul"},"r004__category":{"choice":"billing","confidence":0.7833,"probabilities":{"billing":0.7833,"newsletter":0.0542,"personal":0.0542,"travel":0.0542,"work":0.0542},"type":"choice"},"r004__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":1.0,"1":0.0,"2":0.0},"score":0,"type":"score"},"r004__needs_action":{"noul":0.0336,"type":"noul"},"r004__urgent":{"noul":0.0657,"type":"noul"},"r005__category":{"choice":"personal","confidence":0.7725,"probabilities":{"billing":0.0569,"newsletter":0.0569,"personal":0.7725,"travel":0.0569,"work":0.0569},"type":"choice"},"r005__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r005__needs_action":{"noul":0.7797,"type":"noul"},"r005__urgent":{"noul":0.3717,"type":"noul"},"r006__category":{"choice":"billing","confidence":0.8193,"probabilities":{"billing":0.8193,"newsletter":0.0452,"personal":0.0452,"travel":0.0452,"work":0.0452},"type":"choice"},"r006__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r006__needs_action":{"noul":0.5286,"type":"noul"},"r006__urgent":{"noul":0.223,"type":"noul"},"r007__category":{"choice":"work","confidence":0.8234,"probabilities":{"billing":0.0441,"newsletter":0.0441,"personal":0.0441,"travel":0.0441,"work":0.8234},"type":"choice"},"r007__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r007__needs_action":{"noul":0.888,"type":"noul"},"r007__urgent":{"noul":0.6915,"type":"noul"},"r008__category":{"choice":"work","confidence":0.8163,"probabilities":{"billing":0.0459,"newsletter":0.0459,"personal":0.0459,"travel":0.0459,"work":0.8163},"type":"choice"},"r008__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r008__needs_action":{"noul":0.1228,"type":"noul"},"r008__urgent":{"noul":0.0797,"type":"noul"},"r009__category":{"choice":"travel","confidence":0.785,"probabilities":{"billing":0.0537,"newsletter":0.0537,"personal":0.0537,"travel":0.785,"work":0.0537},"type":"choice"},"r009__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r009__needs_action":{"noul":0.1577,"type":"noul"},"r009__urgent":{"noul":0.2185,"type":"noul"},"r010__category":{"choice":"work","confidence":0.7812,"probabilities":{"billing":0.0547,"newsletter":0.0547,"personal":0.0547,"travel":0.0547,"work":0.7812},"type":"choice"},"r010__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r010__needs_action":{"noul":0.8861,"type":"noul"},"r010__urgent":{"noul":0.6774,"type":"noul"}},"model":"synthetic-jev","request_hash":"14c3693f64b0e9ea7e470277fccae6ab501d404185eec5bb9074ef4b75c572c0","usage":{"input_tokens":3515}}

data/bench/cassettes/14fe9f6d8055d41f5d970dc32ab640c213ea8782d365c82d32b9884ba4f28c67.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"work","confidence":0.8121,"probabilities":{"billing":0.047,"newsletter":0.047,"personal":0.047,"travel":0.047,"work":0.8121},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":1.0,"2":0.0},"score":1,"type":"score"},"r001__needs_action":{"noul":0.0957,"type":"noul"},"r001__urgent":{"noul":0.0274,"type":"noul"}},"model":"synthetic-jev","request_hash":"14fe9f6d8055d41f5d970dc32ab640c213ea8782d365c82d32b9884ba4f28c67","usage":{"input_tokens":437}}

data/bench/cassettes/1a1d8557e4698426c2467aa573a240357715d3d9e3fa77bf066672a8b6bf9a86.json

@@ -0,0 +1 @@
+{"answers":{"r001__category":{"choice":"billing","confidence":0.7922,"probabilities":{"billing":0.7922,"newsletter":0.052,"personal":0.052,"travel":0.052,"work":0.052},"type":"choice"},"r001__importance":{"confidence":1.0,"legend":{"0":"0","1":"1","2":"2"},"probabilities":{"0":0.0,"1":0.0,"2":1.0},"score":2,"type":"score"},"r001__needs_action":{"noul":0.8768,"type":"noul"},"r001__urgent":{"noul":0.2418,"type":"noul"}},"model":"synthetic-jev","request_hash":"1a1d8557e4698426c2467aa573a240357715d3d9e3fa77bf066672a8b6bf9a86","usage":{"input_tokens":444}}

data/bench/cassettes/1b5a8aaaf98305560588246cd51f53c6c185a951f8018c26bf588954d9de1cea.json

@@ -0,0 +1 @@
+{"answers":{"c001__relevance":{"noul":0.0,"type":"noul"},"c002__relevance":{"noul":0.0,"type":"noul"}},"model":"synthetic-jev","request_hash":"1b5a8aaaf98305560588246cd51f53c6c185a951f8018c26bf588954d9de1cea","usage":{"input_tokens":438}}