truffler 0.1.0 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 28c9f27b6811a398815e7cc1026b0d1ed0ef1caf546bd820a59be4efdd4dcffa
-  data.tar.gz: 1889959cc8dfd3f32af19a37dda640673e08d6459f5a5e7722747cd32a7422fd
+  metadata.gz: 99d7f61714bf653e012b5cea1f9d57cfbde9d390aae048ba0d52387292a37159
+  data.tar.gz: b9826342d2552dae2b1cd9d671ac4d4d448f5fcbe3822b21e5788b7d272365b0
 SHA512:
-  metadata.gz: 76974d97ae716de75fc3cc1992868c2cabbd94c6851d6a85267fb5c3c1d18ca21f2f998c6ae1d135192081bd7b544351801fc88d33cde4070c56464a01be4d98
-  data.tar.gz: 8a35786e72b7e5b2996ce915da8419cb8d0d0873c8839829155b7d831f043caf4df9a9be29b9219f75d2403abbbd2c915a09b06b31e0135566a0dbe53f14774e
+  metadata.gz: 414ddacdd2e195a077792a36b3c1e8050a8e086a99ccb7ffed065776826f6af2c83da47835ce36d663db166b664e2559a01f2f92dcc6c493e120bbcd60ae89f8
+  data.tar.gz: 272d2d762fdcd5e0f6a6d0cec08ca332619a14e6ab8b3b94a5f2670e16091bf0c75395866a5a77ede42e7d22483a95252cbf6b0cee3565cacfb91d2ae5c17569

data/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## [0.1.2]
+
+Fixes from happyhappy's production `churn_risk` backfill and its 0.1.1 upgrade.
+
+- `rake truffler:backfill` waits out budget denials instead of ending on `budget_denied`. It backs off 1 s, doubling to 30 s (longer when the budget's new `Decision#retry_after` hint says so), and retries from the same cursor until the backfill completes or reaches the spend cap. `MAX_DURATION=<seconds>` stops it with `paused` and the cursor. While waiting it prints counts, spend, and the cursor, never record text. In code: `Labeling::Backfill#run(wait: true, max_duration:, sleeper:, clock:, progress:)`. `BackfillJob` reschedules denials with the same backoff (it used to wait a fixed 30 s), counting consecutive denials in its arguments.
+- The backfill spend cap holds across runs. Spend is recorded per model and app-wide vocabulary version in a new `truffler_backfill_spends` table. Each Jev request reserves its estimate against the cap in SQL, so rerunning the rake task or overlapping `BackfillJob` chains can no longer spend more than `backfill_spend_cap` for one vocabulary version. Before, overlapping chains could spend up to twice the cap. A vocabulary change starts a new ledger. `truffler:status` prints the spend for the current version, and `RESET_SPEND=1` zeroes it before a backfill. Lens backfills still charge only their lens row.
+- Upgrading from 0.1.1: run `bin/rails generate truffler:upgrade && bin/rails db:migrate` to add `truffler_backfill_spends`. Until you do, backfills log one warning and cap spend per run as in 0.1.1. Fresh installs get the table from `truffler:install`.
+- Time phrases now include hours ("last 3 hours", "past hour", "last hour"), "past week", "past month", and "past/last N months" as rolling windows ending now; "this/last week" and "this/last month" keep their calendar meaning. Chips and suppression are unchanged.
+- A query word now names a choice option by the words of its display name or description (minus stopwords), not only its key, so "spiral" names option `p_17` described as "Spiral writing tool" when the encoding applies it. The request state's label vocabulary adds `option_names` (option key to display name) next to `options`.
+
+## [0.1.1]
+
+Fixes from the first host integration (happyhappy).
+
+- Live queries no longer return zero results next to a label filter. Query encoding sends the label vocabulary (descriptions and choice option names) in the request state; a keyword that names an applied label or its option (including a shared first-three-letter stem, so "angry" names `anger`) becomes a label term and common stopwords become filler; and when a label filter applies, keyword hits only add to the score instead of being required.
+- Declaring `truffler` on a model whose table does not exist yet (a fresh database during `db:prepare`) no longer raises. Column checks run on the first labeling or search instead and raise `DefinitionError` then if columns are still unknown.
+- Query encoding's "no option" answer is the reserved `Truffler::NO_OPTION` (`"truffler:none"`), so a host choice option named `none` can be filtered. Host options may not use the reserved name.
+- Time phrases (`today`, `yesterday`, `this/last week`, `this/last month`, `past/last N days/weeks`, `since <weekday>`) are parsed locally, never asked of Jev or matched as keywords, and limit results on the `arrived_at` column. They show as a removable `kind: :time` chip (suppress `"time"`); keystroke search accepts a `clock:` for tests.
+- `watch:` works on asked labels, not only on `from:` labels, and a model-level `watch :column, ...` relabels every label when those columns change, for `reads` fields backed by methods.
+- `Clients::Fake` answers unscripted choices with their neutral option (`ignore`, `Truffler::NO_OPTION`, `keyword`) when present, so unscripted host tests no longer filter on every label.
+- `config.backfill_spend_cap` defaults to 5.0 USD for `BackfillJob`, `ResumeJob` backfills, and `rake truffler:backfill`. Set it to `nil`, or pass `SPEND_CAP=none`, to disable it; an unparseable `SPEND_CAP` aborts.
+- A per-tenant choice whose options callable returns `{}` or nil for a tenant is left out of that tenant's vocabulary (not asked, not encoded) instead of raising.
+
 ## 0.1.0
 
 - `truffler do ... end` declarations on Active Record models: tenant, fields, noul/choice/score labels, keyword and exact sources, embeddings, a provider, surfaces, ranking weights, and `weak_below`.

data/README.md

@@ -34,12 +34,23 @@ bin/rails db:migrate
 
 The generator writes three files:
 
-- A migration that creates `truffler_labels`, `truffler_record_states`, `truffler_embeddings`, `truffler_query_misses`, `truffler_lenses`, and `truffler_lens_versions`.
+- A migration that creates `truffler_labels`, `truffler_record_states`, `truffler_embeddings`, `truffler_query_misses`, `truffler_lenses`, `truffler_lens_versions`, and `truffler_backfill_spends`.
 - `config/initializers/truffler.rb`.
 - `app/channels/truffler_channel.rb`.
 
 It takes two options. `--record-id-type=bigint|integer|string|uuid` must match your primary keys. `--vector-dimensions=N` stores embeddings in a pgvector column of that width; it needs Postgres and the `neighbor` gem, which provides the `t.vector` column type.
 
+### Upgrading from 0.1.1
+
+0.1.2 adds the `truffler_backfill_spends` table, which keeps the backfill spend cap across runs. Add it with:
+
+```bash
+bin/rails generate truffler:upgrade
+bin/rails db:migrate
+```
+
+The migration creates only that table and skips it if it already exists. Until you run it, backfills log one warning and cap spend per run, as 0.1.1 did.
+
 Truffler digests user keys and query misses with `secret_key_base`. Rails supplies it automatically; outside Rails, set `config.secret_key_base`.
 
 ## Declaring a model
@@ -80,7 +91,8 @@ end
 Here is what each option does:
 
 - `filter_at` makes a label a hard filter at that probability when the query asks for it. `boost` is the label's weight when the query prefers it. `filter_weight:` (default 0) is how much intent weight a filter adds on top.
-- Choice `options:` may be a callable of the tenant key, which gives each tenant its own vocabulary.
+- Choice `options:` may be a callable of the tenant key, which gives each tenant its own vocabulary. A tenant it gives no options (`{}` or nil) simply lacks the label: it is neither asked nor encoded there. The option name `truffler:none` is reserved (see `Truffler::NO_OPTION` below).
+- `watch :column, ...` relabels every label when one of those columns changes. Saving a record only relabels on columns that changed, so a `reads` field backed by a method (a conversation built from messages, say) needs the columns it is built from in `watch`.
 - `keyword` also accepts a single callable, `->(scope, tokens) { relation }`.
 - `embeddings column: :my_vector` searches a vector column you maintain yourself. Truffler never writes it.
 - `embeddings` refuses to send encrypted fields to the embedding provider unless you pass `allow_encrypted: true`.
@@ -96,12 +108,12 @@ Here is what each option does:
 | `filter_at:`, `boost:`, `filter_weight:` | all | Filter threshold, boost weight, and the intent weight a filter adds. |
 | `description:` | all | The label's wording in query encoding. Defaults to the question, then to the key. |
 | `from:` | all | `->(record) { answer }`. Makes the label host-supplied; Jev is never asked. See [Labels you already compute](#labels-you-already-compute). |
-| `watch:` | with `from:` | Extra columns whose change refreshes the label, in addition to `reads`. |
+| `watch:` | all | Extra columns whose change refreshes (or, for an asked label, re-asks) just this label, in addition to `reads`. |
 | `version:` | with `from:` | Any value; changing it rewrites the label for every record on the next backfill. |
 
 Keys must be lowercase snake case without a double underscore, and `lens` is reserved.
 
-Saving a record enqueues labeling (and embedding, when enabled) after commit, but only when a field in `reads`, the tenant column, or a supplied label's `watch:` column changed. Destroying a record removes its labels, state, and embeddings.
+Saving a record enqueues labeling (and embedding, when enabled) after commit, but only when a column-backed field in `reads`, the tenant column, a model-level `watch` column, or a label's `watch:` column changed. Destroying a record removes its labels, state, and embeddings.
 
 ### Labels you already compute
 
@@ -171,10 +183,14 @@ result = Email.truffler(params[:q], tenant: Current.account.id, scope: Current.a
 
 A keystroke search makes no network call. It reads the query encoding and query vector from the cache. When the cache misses, it enqueues `EncodeQueryJob`, and the next keystroke or reload picks up the result.
 
+Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, the chosen option, or a word of that option's display name or description, ignoring case and plurals, or a word of four letters or more sharing its first three letters, so "angry" names `anger`) counts as naming the label, and common stopwords are dropped. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
+
+Time phrases are handled in Ruby and never asked of Jev: `today`, `yesterday`, `this week`, `last week`, `this month`, `last month`, `past|last N hours|days|weeks|months`, `last hour`, `past hour`, `past week`, `past month`, and `since monday` through `since sunday`. "Past/last N units" is a rolling window ending now; "this/last week" and "this/last month" are calendar windows. The first one in a query limits results to records whose `arrived_at` column falls in that window, its words are not keywords, and it shows as a chip `{key: "time", label: "time", kind: :time, name: "This week"}`. Pass `"time"` in `suppressed:` to drop it. Weeks start on `Date.beginning_of_week`, and the window is computed from `Time.current` (or a `clock:` callable passed to the search, for tests).
+
 The returned `Truffler::Search::Result` exposes:
 
 - `records` and `ids`.
-- `chips`: `[{key:, label:, kind: :filter | :boost, name:}]`.
+- `chips`: `[{key:, label:, kind: :filter | :boost | :time, name:}]`.
 - `invite_row`: `{query:, reason: :weak | :empty | :encoding_pending}` or nil.
 - `encoding_status`: `:cached`, `:pending`, or `:none`.
 - `watermark` and `new_matches_count`.
@@ -322,7 +338,7 @@ Set these in `Truffler.configure do |config| ... end`.
 | `max_field_chars`, `request_token_budget`, `max_questions_per_request` | 4,000, 48,000, 200 | Request packing limits. |
 | `queue_name` | `:default` | Queue for every Truffler job. |
 | `cost_per_million_tokens` | 0.042 | Jev input price, used in usage events and estimates. |
-| `backfill_spend_cap` | nil | Dollar cap per backfill run. Supplied labels cost nothing and are still written once it is reached. |
+| `backfill_spend_cap` | 5.0 | Dollar cap on backfill spend per model and vocabulary version, shared by `BackfillJob` chains, `ResumeJob` backfills, and `truffler:backfill` runs (see Backfill). `nil` disables it; for the rake task, `SPEND_CAP=none` does. Supplied labels cost nothing and are still written once it is reached. |
 | `resume_pending_after` | 5 minutes | How long before `ResumeJob` treats work as stuck. |
 | `embedder` | `Embeddings::RubyLLMEmbedder.new` | Any `Embeddings::Embedder` subclass. The default calls `RubyLLM.embed`, which works on ruby_llm 1.x and 2. |
 | `embedding_cost_per_million_tokens` | 0.02 | Embedding price. |
@@ -348,10 +364,10 @@ Truffler enqueues most of its own jobs. Run a worker for `config.queue_name` and
 | `Truffler::Jobs::ResumeJob` | Every few minutes. Requeues failed and stuck labeling after an outage or a crashed worker, and enqueues up to 1,000 missing or stale embeddings per model per hour. |
 | `Truffler::Jobs::PruneQueryMissesJob` | Daily. Enforces `miss_retention`. |
 | `Truffler::Jobs::ExpireLensesJob` | Daily. Expires lenses unused for `lenses.expire_after`. |
-| `bin/rails "truffler:backfill[Email]"` (`SPEND_CAP=5`) or `Truffler::Jobs::BackfillJob.perform_later("Email")` | After adopting Truffler, changing a declaration, or changing the model pin. |
+| `bin/rails "truffler:backfill[Email]"` (see Backfill) or `Truffler::Jobs::BackfillJob.perform_later("Email")` | After adopting Truffler, changing a declaration, or changing the model pin. |
 | `Truffler::Embeddings::Backfill.new(Email).enqueue` | After enabling embeddings or changing the embedding model, width, or fields, to re-embed everything now instead of through the `ResumeJob` sweep. It enqueues 1,000 jobs at a time; pass `limit:` to cap the total. |
 
-`bin/rails "truffler:status[Email]"` prints labeling counts. `bin/rails "truffler:suggestions[Email]"` prints candidate questions drawn from logged query misses.
+`bin/rails "truffler:status[Email]"` prints labeling counts and the backfill spend for the current vocabulary version. `bin/rails "truffler:suggestions[Email]"` prints candidate questions drawn from logged query misses.
 
 With Solid Queue, for example:
 
@@ -368,6 +384,18 @@ truffler_expire_lenses:
   schedule: every day at 3am
 ```
 
+### Backfill
+
+`bin/rails "truffler:backfill[Email]"` labels missing, stale, and failed records inline, newest first, at backfill priority. Backfill has the lowest share of the request budget, so the task waits whenever the budget turns it away. It backs off 1 s, doubling to 30 s (or longer if the budget says a slot frees later), and resumes from the same cursor. It ends when every record is current (`complete`) or the spend cap is reached (`spend_cap_reached`). While it waits it prints the records labeled, the spend, and the cursor, never record text. Environment variables:
+
+| Variable | Effect |
+|---|---|
+| `SPEND_CAP=20` or `none` | Overrides `backfill_spend_cap` for this run. |
+| `MAX_DURATION=600` | Stops after that many seconds with `paused` and the cursor. Rerun to continue. |
+| `RESET_SPEND=1` | Zeroes the spend recorded for the current vocabulary version before starting. |
+
+The spend cap holds across runs. Truffler records backfill spend per model and app-wide vocabulary version in `truffler_backfill_spends`, and reserves each request's estimate against the cap in SQL. A rerun, a `ResumeJob` backfill, and overlapping `BackfillJob` chains all draw on the same total, so together they stop at `backfill_spend_cap`. Changing the vocabulary (a reworded question, a new label, or an activated lens) starts a new total. Lens backfills are capped separately by each lens's `spend_cap_usd`. `BackfillJob` reschedules itself after a denial with the same backoff. In code, `Truffler::Labeling::Backfill.new(Email).run(wait: true, max_duration: 600)` does what the task does.
+
 ## Privacy
 
 These guarantees hold for every model, and matter most for models that use Active Record encryption:
@@ -423,3 +451,5 @@ bundle exec rubocop
 ```
 
 Tests run on in-memory SQLite with fake clients. Any live Jev call raises `Truffler::LiveCallInTest`.
+
+In your own tests, `Truffler::Clients::Fake` answers unscripted questions neutrally: nouls no, scores the lowest level, and choices their neutral option when they have one (`ignore` for a query's label intent, `Truffler::NO_OPTION` for an option question, `keyword` for a word role), otherwise the first option. An unscripted query encoding therefore applies no labels; script the ones a test needs, e.g. `fake.answer("intent__needs_action", "filter")`.

data/lib/generators/truffler/install/templates/migration.rb.tt

@@ -105,5 +105,15 @@ class CreateTrufflerTables < ActiveRecord::Migration<%= migration_version %>
     end
     add_index :truffler_lens_versions, [:lens_id, :number], unique: true,
       name: "index_truffler_lens_versions_on_lens_and_number"
+
+    create_table :truffler_backfill_spends do |t|
+      t.string :record_type, null: false
+      t.string :vocabulary_version, null: false
+      t.float :spent_usd, null: false, default: 0.0
+      t.integer :requests, null: false, default: 0
+      t.timestamps
+    end
+    add_index :truffler_backfill_spends, [:record_type, :vocabulary_version], unique: true,
+      name: "index_truffler_backfill_spends_on_ledger"
   end
 end

data/lib/generators/truffler/upgrade/templates/backfill_spends_migration.rb.tt

@@ -0,0 +1,19 @@
+class CreateTrufflerBackfillSpends < ActiveRecord::Migration<%= migration_version %>
+  def up
+    return if table_exists?(:truffler_backfill_spends)
+
+    create_table :truffler_backfill_spends do |t|
+      t.string :record_type, null: false
+      t.string :vocabulary_version, null: false
+      t.float :spent_usd, null: false, default: 0.0
+      t.integer :requests, null: false, default: 0
+      t.timestamps
+    end
+    add_index :truffler_backfill_spends, [:record_type, :vocabulary_version], unique: true,
+      name: "index_truffler_backfill_spends_on_ledger"
+  end
+
+  def down
+    drop_table :truffler_backfill_spends, if_exists: true
+  end
+end

data/lib/generators/truffler/upgrade/upgrade_generator.rb

@@ -0,0 +1,24 @@
+require "rails/generators"
+require "rails/generators/active_record"
+
+module Truffler
+  module Generators
+    # Adds the tables a newer Truffler needs to an app installed with an older
+    # one. Each migration skips a table that already exists.
+    class UpgradeGenerator < Rails::Generators::Base
+      include ActiveRecord::Generators::Migration
+
+      source_root File.expand_path("templates", __dir__)
+
+      def create_backfill_spends_migration
+        migration_template "backfill_spends_migration.rb.tt", File.join(db_migrate_path, "create_truffler_backfill_spends.rb")
+      end
+
+      private
+
+      def migration_version
+        "[#{ActiveRecord::Migration.current_version}]"
+      end
+    end
+  end
+end

data/lib/tasks/truffler.rake

@@ -9,12 +9,42 @@ namespace :truffler do
     model
   end
 
-  desc "Backfill stale, missing, and failed labels for a model (SPEND_CAP=dollars)"
+  resolve_spend_cap = lambda do |value|
+    case value.to_s.strip.downcase
+    when "" then Truffler.config.backfill_spend_cap
+    when "none" then nil
+    else Float(value, exception: false) || abort("SPEND_CAP must be a dollar amount or none, got #{value.inspect}")
+    end
+  end
+
+  resolve_max_duration = lambda do |value|
+    next if value.to_s.strip.empty?
+
+    seconds = Float(value, exception: false)
+    abort "MAX_DURATION must be a number of seconds, got #{value.inspect}" unless seconds&.positive?
+    seconds
+  end
+
+  describe_cursor = ->(cursor) { cursor.nil? ? "none" : cursor }
+
+  desc "Backfill stale, missing, and failed labels for a model, waiting out budget denials " \
+    "(SPEND_CAP=dollars or none; default config.backfill_spend_cap; MAX_DURATION=seconds; RESET_SPEND=1 for a fresh spend ledger)"
   task :backfill, [ :model ] => :setup do |_, args|
     model = resolve_model.call(args[:model])
-    spend_cap = ENV["SPEND_CAP"].presence&.to_f || Truffler.config.backfill_spend_cap
-    result = Truffler::Labeling::Backfill.new(model, spend_cap: spend_cap).run
-    puts "#{model.name}: #{result.status}, #{result.labeled} labeled in #{result.requests} requests, $#{format('%.6f', result.cost)}"
+    spend_cap = resolve_spend_cap.call(ENV.fetch("SPEND_CAP", nil))
+    max_duration = resolve_max_duration.call(ENV.fetch("MAX_DURATION", nil))
+    if ENV.fetch("RESET_SPEND", nil) == "1"
+      Truffler::Labeling::Backfill.reset_spend!(model)
+      puts "#{model.name}: fresh spend ledger for the current vocabulary version"
+    end
+    progress = lambda do |so_far, delay|
+      puts "#{model.name}: waiting #{format('%.1f', delay)}s for backfill budget (#{so_far.labeled} labeled, " \
+        "$#{format('%.6f', so_far.cost)} spent, cursor #{describe_cursor.call(so_far.cursor)})"
+    end
+    result = Truffler::Labeling::Backfill.new(model, spend_cap: spend_cap).run(wait: true, max_duration: max_duration, progress: progress)
+    summary = "#{model.name}: #{result.status}, #{result.labeled} labeled in #{result.requests} requests, $#{format('%.6f', result.cost)}"
+    summary += ", cursor #{describe_cursor.call(result.cursor)}" if result.status == :paused
+    puts summary
   end
 
   desc "Print a model's labeling counts by status and staleness"
@@ -22,5 +52,13 @@ namespace :truffler do
     model = resolve_model.call(args[:model])
     puts model.name
     Truffler::Labeling::Backfill.status(model).each { |key, count| puts format("  %-9s %d", key, count) }
+    if Truffler::Records::BackfillSpend.available?
+      ledger = Truffler::Labeling::Backfill.spend(model)
+      cap = Truffler.config.backfill_spend_cap
+      puts format("  %-9s $%.6f in %d requests (vocabulary %s, cap %s)", "spent", ledger&.spent_usd.to_f, ledger&.requests.to_i,
+        Truffler::Labeling::Backfill.ledger_version(model).first(12), cap ? format("$%.2f", cap) : "none")
+    else
+      puts format("  %-9s %s", "spent", "not tracked across runs; run bin/rails g truffler:upgrade && bin/rails db:migrate")
+    end
   end
 end

data/lib/truffler/budget.rb

@@ -8,12 +8,17 @@ module Truffler
   #
   # Outcomes: :granted takes a slot; :demoted means a tenant is over its live
   # cap and its records should wait at backfill priority (no slot is taken);
-  # :denied means skip, pause, or reschedule. Only live callers wait, up to
-  # max_wait. A cache that cannot count (the null store) never blocks.
+  # :denied means skip, pause, or reschedule, with `retry_after` seconds until
+  # the denying window rolls over. Only live callers wait, up to max_wait. A
+  # cache that cannot count (the null store) never blocks.
   class Budget
     PRIORITIES = %i[live encode rerank backfill].freeze
 
-    Decision = Data.define(:outcome, :priority, :reason) do
+    Decision = Data.define(:outcome, :priority, :reason, :retry_after) do
+      def initialize(outcome:, priority:, reason:, retry_after: nil)
+        super
+      end
+
       def granted? = outcome == :granted
       def demoted? = outcome == :demoted
       def denied? = outcome == :denied
@@ -38,9 +43,9 @@ module Truffler
         return Decision.new(:demoted, :backfill, :tenant_cap) unless take(tenant_counter(tenant_key), records, config.tenant_live_cap, taken)
       end
       if (cap = config.user_caps[priority]) && user_key
-        return deny(priority, :user_cap, taken) unless take(user_counter(priority, user_key), 1, cap, taken)
+        return deny(priority, :user_cap, taken, until_next_minute) unless take(user_counter(priority, user_key), 1, cap, taken)
       end
-      return deny(priority, :exhausted, taken) unless take_second(priority)
+      return deny(priority, :exhausted, taken, until_next_second) unless take_second(priority)
 
       Decision.new(:granted, priority, nil)
     end
@@ -54,7 +59,7 @@ module Truffler
 
       cap = config.user_caps[priority]
       return Decision.new(:granted, priority, nil) unless cap && user_key
-      return deny(priority, :user_cap, []) unless take(user_counter(priority, user_key), 1, cap)
+      return deny(priority, :user_cap, [], until_next_minute) unless take(user_counter(priority, user_key), 1, cap)
 
       Decision.new(:granted, priority, nil)
     end
@@ -97,10 +102,20 @@ module Truffler
       end
     end
 
-    def deny(priority, reason, taken)
+    def deny(priority, reason, taken, retry_after)
       taken.each { |key, amount| @cache.decrement(key, amount, expires_in: 2.minutes) }
       Instrumentation.instrument(:budget_denied, priority: priority)
-      Decision.new(:denied, priority, reason)
+      Decision.new(:denied, priority, reason, retry_after)
+    end
+
+    def until_next_second
+      now = @clock.call
+      now.floor + 1 - now
+    end
+
+    def until_next_minute
+      now = @clock.call
+      (minute + 1) * 60 - now
     end
 
     def tenant_counter(tenant_key)

data/lib/truffler/clients/fake.rb

@@ -9,8 +9,14 @@ module Truffler
     #   fake.answer(:urgency, 2)                        # score level
     #   fake.answer(:spam) { |tag, state| state.dig("records", tag, "body").include?("$$$") ? 0.9 : 0.1 }
     #
-    # Unscripted questions answer no, the first option, or the lowest level.
+    # Unscripted questions answer no, the lowest level, or for a choice its
+    # neutral option when it has one (`ignore` for a query-encoding intent,
+    # `Truffler::NO_OPTION` for an option question, `keyword` for a word
+    # role), else the first option. So an unscripted query encoding applies
+    # no label.
     class Fake < Base
+      NEUTRAL_OPTIONS = [ "ignore", NO_OPTION, "keyword" ].freeze
+
       attr_reader :calls
 
       def initialize(model: nil, &default)
@@ -67,7 +73,7 @@ module Truffler
         probabilities = options.to_h { |option| [ option, 0.0 ] }
         case value
         when Hash then probabilities.merge!(value.transform_keys(&:to_s).transform_values(&:to_f))
-        when nil then probabilities[options.first] = 1.0
+        when nil then probabilities[(NEUTRAL_OPTIONS & options).first || options.first] = 1.0
         else probabilities[value.to_s] = 1.0
         end
         pick, confidence = probabilities.max_by { |_, probability| probability }

data/lib/truffler/configuration.rb

@@ -3,6 +3,7 @@ module Truffler
     DEFAULT_REQUESTS_PER_MINUTE = 1_200
     DEFAULT_PRIORITY_CEILINGS = { live: 1.0, encode: 0.9, rerank: 0.75, backfill: 0.5 }.freeze
     DEFAULT_USER_CAPS = { encode: 30, rerank: 10 }.freeze
+    DEFAULT_BACKFILL_SPEND_CAP = 5.0
 
     attr_accessor :model, :cost_per_million_tokens, :requests_per_minute, :headroom, :priority_ceilings,
       :user_caps, :tenant_live_cap, :max_wait, :batch_size, :grouping_window, :max_attempts,
@@ -35,7 +36,7 @@ module Truffler
       @queue_name = :default
       @miss_retention = 30.days
       @miss_min_distinct_users = 5
-      @backfill_spend_cap = nil
+      @backfill_spend_cap = DEFAULT_BACKFILL_SPEND_CAP
       @resume_pending_after = 5.minutes
       @vector_store = :auto
       @embedding_cost_per_million_tokens = 0.02

data/lib/truffler/definition.rb

@@ -4,7 +4,7 @@ module Truffler
     EXPLICIT_ACTIONS = %i[enter key row].freeze
     DEFAULT_EMBEDDINGS = { model: "text-embedding-3-small", dimensions: 256 }.freeze
 
-    attr_reader :model, :fields, :labels, :exact_sources, :providers, :surfaces
+    attr_reader :model, :fields, :labels, :exact_sources, :providers, :surfaces, :watch_columns
     attr_accessor :tenant_column, :keyword, :embeddings, :order, :arrived_at_column
 
     def initialize(model)
@@ -14,6 +14,7 @@ module Truffler
       @exact_sources = {}
       @providers = {}
       @surfaces = {}
+      @watch_columns = []
       @arrived_at_column = "created_at"
     end
 
@@ -39,6 +40,13 @@ module Truffler
       labels.values.select(&:supplied?)
     end
 
+    # Columns whose change relabels every label: the column-backed fields,
+    # the tenant column, and the model-level `watch` columns. Method-backed
+    # fields never show up in saved_changes, so they need `watch`.
+    def relabel_columns
+      [ *fields, tenant_column, *watch_columns ].compact
+    end
+
     def per_tenant_vocabulary?
       labels.each_value.any?(&:per_tenant?)
     end
@@ -71,13 +79,28 @@ module Truffler
       Array(model.try(:encrypted_attributes)).map(&:to_s) & fields
     end
 
+    # Column checks wait for the table: a model declared at boot on a fresh
+    # database is checked on its first labeling or search instead.
     def validate!
       raise DefinitionError, "#{model.name}: declare the fields Jev reads with `reads`" if fields.empty?
 
-      check_columns([ tenant_column, *fields, *Array(keyword).grep(String), *supplied_labels.flat_map(&:watch) ].compact.uniq)
+      @columns_deferred = !table_available?
+      validate_columns! unless @columns_deferred
       check_embeddings if embeddings
     end
 
+    def validate_columns!
+      return if @columns_checked
+
+      if @columns_deferred
+        return unless table_available?
+
+        model.reset_column_information
+      end
+      check_columns([ tenant_column, *fields, *Array(keyword).grep(String), *watch_columns, *labels.values.flat_map(&:watch) ].compact.uniq)
+      @columns_checked = true
+    end
+
     DEFAULT_RANKING = { label: 1.0, text: 1.0, keyword: 0.5, exact: 1.0, min_similarity: 0.0 }.freeze
     DEFAULT_WEAK_BELOW = 3
 
@@ -95,6 +118,12 @@ module Truffler
 
     private
 
+    def table_available?
+      model.connection.data_source_exists?(model.table_name)
+    rescue ActiveRecord::ActiveRecordError
+      false
+    end
+
     def check_columns(names)
       columns = model.attribute_names
       missing = names.reject { |name| columns.include?(name) || model.method_defined?(name) }
@@ -124,6 +153,10 @@ module Truffler
         @definition.fields.concat(fields.map(&:to_s))
       end
 
+      def watch(*columns)
+        @definition.watch_columns.concat(columns.map(&:to_s))
+      end
+
       def label(key, type, **options)
         @definition.add_label(LabelDefinition.new(key, type, **options))
       end

data/lib/truffler/errors.rb

@@ -7,7 +7,17 @@ module Truffler
   class LiveCallInTest < Error; end
   class IncompleteAnswers < Error; end
   class CassetteMiss < Error; end
-  class BudgetExhausted < Error; end
+
+  # Carries the budget's hint of how many seconds until a retry can succeed.
+  class BudgetExhausted < Error
+    attr_reader :retry_after
+
+    def initialize(message = nil, retry_after: nil)
+      @retry_after = retry_after
+      super(message)
+    end
+  end
+
   class TenantMismatch < Error; end
   class NotAuthorized < Error; end
   class InvalidLens < Error; end

data/lib/truffler/jobs/backfill_job.rb

@@ -2,20 +2,21 @@ module Truffler
   module Jobs
     # Backfills one model's stale, missing, failed, and demoted labels at
     # backfill priority. Arguments are the record type, the id cursor, the
-    # spend so far, the cap, and the retry attempt, never record text. A
-    # budget denial or a page limit reschedules the job from its cursor; the
-    # spend cap ends it. A Jev error reschedules it with backoff and the spend
+    # spend so far, the cap, the retry attempt, and the count of budget
+    # denials in a row, never record text. A budget denial reschedules the job
+    # from its cursor after the same backoff a waiting Labeling::Backfill uses
+    # (reset whenever a run lands work); a page limit reschedules it at once;
+    # the spend cap ends it. A Jev error reschedules it with backoff and the spend
     # already made, so retries cannot push past the cap; after MAX_ATTEMPTS
     # the released rows wait for the ResumeJob sweep.
     class BackfillJob < ActiveJob::Base
       MAX_PAGES = 20
       MAX_ATTEMPTS = 10
-      BUDGET_WAIT = 30.seconds
 
       queue_as { Truffler.config.queue_name }
 
       def perform(record_type, cursor: nil, spent: 0.0, spend_cap: Truffler.config.backfill_spend_cap, max_pages: MAX_PAGES,
-        attempt: 0)
+        attempt: 0, denials: 0)
         model = record_type.safe_constantize
         return unless model.respond_to?(:truffler_definition) && model.truffler_definition
 
@@ -25,7 +26,7 @@ module Truffler
 
         follow_up = { cursor: result.cursor, spent: spent + result.cost, spend_cap: spend_cap, max_pages: max_pages }
         case result.status
-        when :budget_denied then self.class.set(wait: BUDGET_WAIT).perform_later(record_type, **follow_up)
+        when :budget_denied then retry_after_denial(record_type, follow_up, result, denials)
         when :paused then self.class.perform_later(record_type, **follow_up)
         when :client_error then retry_later(record_type, follow_up, attempt + 1)
         when :complete, :spend_cap_reached then nil
@@ -35,6 +36,12 @@ module Truffler
 
       private
 
+      def retry_after_denial(record_type, follow_up, result, denials)
+        denials = 0 if result.labeled.positive? || result.requests.positive?
+        wait = Labeling::Backfill.backoff(denials, result.retry_after)
+        self.class.set(wait: wait).perform_later(record_type, **follow_up, denials: denials + 1)
+      end
+
       def retry_later(record_type, follow_up, attempt)
         return if attempt >= MAX_ATTEMPTS
 

data/lib/truffler/label_definition.rb

@@ -2,12 +2,14 @@ module Truffler
   # One typed label question. Nouls store their probability, scores their
   # normalized position, and choices one row per option ("label:option") with
   # that option's probability. Choice options may be a callable of the tenant
-  # key, which makes the vocabulary per-tenant.
+  # key, which makes the vocabulary per-tenant; a tenant it gives no options
+  # (`{}` or nil) simply does not have the label.
   #
   # A label with `from:` is supplied by the host: its answer is read from the
   # record in the shape Jev answers normalize to and Jev is never asked. Its
-  # question is optional, `watch:` names extra columns that refresh it, and
-  # `version:` forces a refresh when its logic changes.
+  # question is optional and `version:` forces a refresh when its logic
+  # changes. On any label, `watch:` names extra columns whose change
+  # refreshes (or re-asks) just that label.
   class LabelDefinition
     TYPES = %i[noul choice score].freeze
     KEY = /\A[a-z][a-z0-9_]*\z/
@@ -50,9 +52,15 @@ module Truffler
     def options(tenant_key = nil)
       options = per_tenant? ? @options.call(tenant_key) : @options
       options = Array(options).to_h { |option| [ option, nil ] } unless options.is_a?(Hash)
-      raise DefinitionError, "#{key}: choice options for tenant #{tenant_key.inspect} are empty" if options.empty?
+      options = options.to_h { |option, description| [ option.to_s, description ] }
+      raise DefinitionError, "#{key}: the option name #{NO_OPTION} is reserved" if options.key?(NO_OPTION)
 
-      options.to_h { |option, description| [ option.to_s, description ] }
+      options
+    end
+
+    # False for a per-tenant choice with no options for this tenant.
+    def available?(tenant_key = nil)
+      type != :choice || !per_tenant? || options(tenant_key).any?
     end
 
     def question(tenant_key = nil)
@@ -139,7 +147,7 @@ module Truffler
     def validate_supplied!
       unless supplied?
         raise DefinitionError, "#{key}: a question is required" if instructions.blank?
-        raise DefinitionError, "#{key}: watch: and version: need from:" if @watch.any? || !@version.nil?
+        raise DefinitionError, "#{key}: version: needs from:" unless @version.nil?
 
         return
       end