truffler 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 28c9f27b6811a398815e7cc1026b0d1ed0ef1caf546bd820a59be4efdd4dcffa
-  data.tar.gz: 1889959cc8dfd3f32af19a37dda640673e08d6459f5a5e7722747cd32a7422fd
+  metadata.gz: 78736a9bfd83f38c6623b52904121d19a15c33f9d53bad271c1021968749c11d
+  data.tar.gz: 68f40b4143278a1ab48102aaa5d0146fc00c71885f332180dacdd2495e6e0a7d
 SHA512:
-  metadata.gz: 76974d97ae716de75fc3cc1992868c2cabbd94c6851d6a85267fb5c3c1d18ca21f2f998c6ae1d135192081bd7b544351801fc88d33cde4070c56464a01be4d98
-  data.tar.gz: 8a35786e72b7e5b2996ce915da8419cb8d0d0873c8839829155b7d831f043caf4df9a9be29b9219f75d2403abbbd2c915a09b06b31e0135566a0dbe53f14774e
+  metadata.gz: 2b98ccc3b180c487994c17eb6ed194e300ecace2948b549952a7b731e9bd5364c74a59656e93d8235bd6499aa41cb0926ea6ccf80acb9eed69e4946f5219b133
+  data.tar.gz: 860b3932fa42544edd3bee2ecf65ec3e90ae0aa8d9d9c6d8c1ea073dd9e5a0b7d7e4af78ab28dcd469aced26a47aa73e91b148878b98ea999fe8cf7cee5b3e5a

data/CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## [0.1.1]
+
+Fixes from the first host integration (happyhappy).
+
+- Live queries no longer return zero results next to a label filter. Query encoding sends the label vocabulary (descriptions and choice option names) in the request state; a keyword that names an applied label or its option (including a shared first-three-letter stem, so "angry" names `anger`) becomes a label term and common stopwords become filler; and when a label filter applies, keyword hits only add to the score instead of being required.
+- Declaring `truffler` on a model whose table does not exist yet (a fresh database during `db:prepare`) no longer raises. Column checks run on the first labeling or search instead and raise `DefinitionError` then if columns are still unknown.
+- Query encoding's "no option" answer is the reserved `Truffler::NO_OPTION` (`"truffler:none"`), so a host choice option named `none` can be filtered. Host options may not use the reserved name.
+- Time phrases (`today`, `yesterday`, `this/last week`, `this/last month`, `past/last N days/weeks`, `since <weekday>`) are parsed locally, never asked of Jev or matched as keywords, and limit results on the `arrived_at` column. They show as a removable `kind: :time` chip (suppress `"time"`); keystroke search accepts a `clock:` for tests.
+- `watch:` works on asked labels, not only on `from:` labels, and a model-level `watch :column, ...` relabels every label when those columns change, for `reads` fields backed by methods.
+- `Clients::Fake` answers unscripted choices with their neutral option (`ignore`, `Truffler::NO_OPTION`, `keyword`) when present, so unscripted host tests no longer filter on every label.
+- `config.backfill_spend_cap` defaults to 5.0 USD for `BackfillJob`, `ResumeJob` backfills, and `rake truffler:backfill`. Set it to `nil`, or pass `SPEND_CAP=none`, to disable it; an unparseable `SPEND_CAP` aborts.
+- A per-tenant choice whose options callable returns `{}` or nil for a tenant is left out of that tenant's vocabulary (not asked, not encoded) instead of raising.
+
 ## 0.1.0
 
 - `truffler do ... end` declarations on Active Record models: tenant, fields, noul/choice/score labels, keyword and exact sources, embeddings, a provider, surfaces, ranking weights, and `weak_below`.

data/README.md

@@ -80,7 +80,8 @@ end
 Here is what each option does:
 
 - `filter_at` makes a label a hard filter at that probability when the query asks for it. `boost` is the label's weight when the query prefers it. `filter_weight:` (default 0) is how much intent weight a filter adds on top.
-- Choice `options:` may be a callable of the tenant key, which gives each tenant its own vocabulary.
+- Choice `options:` may be a callable of the tenant key, which gives each tenant its own vocabulary. A tenant it gives no options (`{}` or nil) simply lacks the label: it is neither asked nor encoded there. The option name `truffler:none` is reserved (see `Truffler::NO_OPTION` below).
+- `watch :column, ...` relabels every label when one of those columns changes. Saving a record only relabels on columns that changed, so a `reads` field backed by a method (a conversation built from messages, say) needs the columns it is built from in `watch`.
 - `keyword` also accepts a single callable, `->(scope, tokens) { relation }`.
 - `embeddings column: :my_vector` searches a vector column you maintain yourself. Truffler never writes it.
 - `embeddings` refuses to send encrypted fields to the embedding provider unless you pass `allow_encrypted: true`.
@@ -96,12 +97,12 @@ Here is what each option does:
 | `filter_at:`, `boost:`, `filter_weight:` | all | Filter threshold, boost weight, and the intent weight a filter adds. |
 | `description:` | all | The label's wording in query encoding. Defaults to the question, then to the key. |
 | `from:` | all | `->(record) { answer }`. Makes the label host-supplied; Jev is never asked. See [Labels you already compute](#labels-you-already-compute). |
-| `watch:` | with `from:` | Extra columns whose change refreshes the label, in addition to `reads`. |
+| `watch:` | all | Extra columns whose change refreshes (or, for an asked label, re-asks) just this label, in addition to `reads`. |
 | `version:` | with `from:` | Any value; changing it rewrites the label for every record on the next backfill. |
 
 Keys must be lowercase snake case without a double underscore, and `lens` is reserved.
 
-Saving a record enqueues labeling (and embedding, when enabled) after commit, but only when a field in `reads`, the tenant column, or a supplied label's `watch:` column changed. Destroying a record removes its labels, state, and embeddings.
+Saving a record enqueues labeling (and embedding, when enabled) after commit, but only when a column-backed field in `reads`, the tenant column, a model-level `watch` column, or a label's `watch:` column changed. Destroying a record removes its labels, state, and embeddings.
 
 ### Labels you already compute
 
@@ -171,10 +172,14 @@ result = Email.truffler(params[:q], tenant: Current.account.id, scope: Current.a
 
 A keystroke search makes no network call. It reads the query encoding and query vector from the cache. When the cache misses, it enqueues `EncodeQueryJob`, and the next keystroke or reload picks up the result.
 
+Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, or the chosen option, ignoring case and plurals, or a word of four letters or more sharing its first three letters, so "angry" names `anger`) counts as naming the label, and common stopwords are dropped. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
+
+Time phrases are handled in Ruby and never asked of Jev: `today`, `yesterday`, `this week`, `last week`, `this month`, `last month`, `past|last N days|weeks`, and `since monday` through `since sunday`. The first one in a query limits results to records whose `arrived_at` column falls in that window, its words are not keywords, and it shows as a chip `{key: "time", label: "time", kind: :time, name: "This week"}`. Pass `"time"` in `suppressed:` to drop it. Weeks start on `Date.beginning_of_week`, and the window is computed from `Time.current` (or a `clock:` callable passed to the search, for tests).
+
 The returned `Truffler::Search::Result` exposes:
 
 - `records` and `ids`.
-- `chips`: `[{key:, label:, kind: :filter | :boost, name:}]`.
+- `chips`: `[{key:, label:, kind: :filter | :boost | :time, name:}]`.
 - `invite_row`: `{query:, reason: :weak | :empty | :encoding_pending}` or nil.
 - `encoding_status`: `:cached`, `:pending`, or `:none`.
 - `watermark` and `new_matches_count`.
@@ -322,7 +327,7 @@ Set these in `Truffler.configure do |config| ... end`.
 | `max_field_chars`, `request_token_budget`, `max_questions_per_request` | 4,000, 48,000, 200 | Request packing limits. |
 | `queue_name` | `:default` | Queue for every Truffler job. |
 | `cost_per_million_tokens` | 0.042 | Jev input price, used in usage events and estimates. |
-| `backfill_spend_cap` | nil | Dollar cap per backfill run. Supplied labels cost nothing and are still written once it is reached. |
+| `backfill_spend_cap` | 5.0 | Dollar cap per backfill run (`BackfillJob` chains, `ResumeJob` backfills, and `truffler:backfill`). `nil` disables it; for the rake task, `SPEND_CAP=none` does. Supplied labels cost nothing and are still written once it is reached. |
 | `resume_pending_after` | 5 minutes | How long before `ResumeJob` treats work as stuck. |
 | `embedder` | `Embeddings::RubyLLMEmbedder.new` | Any `Embeddings::Embedder` subclass. The default calls `RubyLLM.embed`, which works on ruby_llm 1.x and 2. |
 | `embedding_cost_per_million_tokens` | 0.02 | Embedding price. |
@@ -348,7 +353,7 @@ Truffler enqueues most of its own jobs. Run a worker for `config.queue_name` and
 | `Truffler::Jobs::ResumeJob` | Every few minutes. Requeues failed and stuck labeling after an outage or a crashed worker, and enqueues up to 1,000 missing or stale embeddings per model per hour. |
 | `Truffler::Jobs::PruneQueryMissesJob` | Daily. Enforces `miss_retention`. |
 | `Truffler::Jobs::ExpireLensesJob` | Daily. Expires lenses unused for `lenses.expire_after`. |
-| `bin/rails "truffler:backfill[Email]"` (`SPEND_CAP=5`) or `Truffler::Jobs::BackfillJob.perform_later("Email")` | After adopting Truffler, changing a declaration, or changing the model pin. |
+| `bin/rails "truffler:backfill[Email]"` (`SPEND_CAP=20`, or `none`; default `backfill_spend_cap`) or `Truffler::Jobs::BackfillJob.perform_later("Email")` | After adopting Truffler, changing a declaration, or changing the model pin. |
 | `Truffler::Embeddings::Backfill.new(Email).enqueue` | After enabling embeddings or changing the embedding model, width, or fields, to re-embed everything now instead of through the `ResumeJob` sweep. It enqueues 1,000 jobs at a time; pass `limit:` to cap the total. |
 
 `bin/rails "truffler:status[Email]"` prints labeling counts. `bin/rails "truffler:suggestions[Email]"` prints candidate questions drawn from logged query misses.
@@ -423,3 +428,5 @@ bundle exec rubocop
 ```
 
 Tests run on in-memory SQLite with fake clients. Any live Jev call raises `Truffler::LiveCallInTest`.
+
+In your own tests, `Truffler::Clients::Fake` answers unscripted questions neutrally: nouls no, scores the lowest level, and choices their neutral option when they have one (`ignore` for a query's label intent, `Truffler::NO_OPTION` for an option question, `keyword` for a word role), otherwise the first option. An unscripted query encoding therefore applies no labels; script the ones a test needs, e.g. `fake.answer("intent__needs_action", "filter")`.

data/lib/tasks/truffler.rake

@@ -9,10 +9,18 @@ namespace :truffler do
     model
   end
 
-  desc "Backfill stale, missing, and failed labels for a model (SPEND_CAP=dollars)"
+  resolve_spend_cap = lambda do |value|
+    case value.to_s.strip.downcase
+    when "" then Truffler.config.backfill_spend_cap
+    when "none" then nil
+    else Float(value, exception: false) || abort("SPEND_CAP must be a dollar amount or none, got #{value.inspect}")
+    end
+  end
+
+  desc "Backfill stale, missing, and failed labels for a model (SPEND_CAP=dollars or none; default config.backfill_spend_cap)"
   task :backfill, [ :model ] => :setup do |_, args|
     model = resolve_model.call(args[:model])
-    spend_cap = ENV["SPEND_CAP"].presence&.to_f || Truffler.config.backfill_spend_cap
+    spend_cap = resolve_spend_cap.call(ENV.fetch("SPEND_CAP", nil))
     result = Truffler::Labeling::Backfill.new(model, spend_cap: spend_cap).run
     puts "#{model.name}: #{result.status}, #{result.labeled} labeled in #{result.requests} requests, $#{format('%.6f', result.cost)}"
   end

data/lib/truffler/clients/fake.rb

@@ -9,8 +9,14 @@ module Truffler
     #   fake.answer(:urgency, 2)                        # score level
     #   fake.answer(:spam) { |tag, state| state.dig("records", tag, "body").include?("$$$") ? 0.9 : 0.1 }
     #
-    # Unscripted questions answer no, the first option, or the lowest level.
+    # Unscripted questions answer no, the lowest level, or for a choice its
+    # neutral option when it has one (`ignore` for a query-encoding intent,
+    # `Truffler::NO_OPTION` for an option question, `keyword` for a word
+    # role), else the first option. So an unscripted query encoding applies
+    # no label.
     class Fake < Base
+      NEUTRAL_OPTIONS = [ "ignore", NO_OPTION, "keyword" ].freeze
+
       attr_reader :calls
 
       def initialize(model: nil, &default)
@@ -67,7 +73,7 @@ module Truffler
         probabilities = options.to_h { |option| [ option, 0.0 ] }
         case value
         when Hash then probabilities.merge!(value.transform_keys(&:to_s).transform_values(&:to_f))
-        when nil then probabilities[options.first] = 1.0
+        when nil then probabilities[(NEUTRAL_OPTIONS & options).first || options.first] = 1.0
         else probabilities[value.to_s] = 1.0
         end
         pick, confidence = probabilities.max_by { |_, probability| probability }

data/lib/truffler/configuration.rb

@@ -3,6 +3,7 @@ module Truffler
     DEFAULT_REQUESTS_PER_MINUTE = 1_200
     DEFAULT_PRIORITY_CEILINGS = { live: 1.0, encode: 0.9, rerank: 0.75, backfill: 0.5 }.freeze
     DEFAULT_USER_CAPS = { encode: 30, rerank: 10 }.freeze
+    DEFAULT_BACKFILL_SPEND_CAP = 5.0
 
     attr_accessor :model, :cost_per_million_tokens, :requests_per_minute, :headroom, :priority_ceilings,
       :user_caps, :tenant_live_cap, :max_wait, :batch_size, :grouping_window, :max_attempts,
@@ -35,7 +36,7 @@ module Truffler
       @queue_name = :default
       @miss_retention = 30.days
       @miss_min_distinct_users = 5
-      @backfill_spend_cap = nil
+      @backfill_spend_cap = DEFAULT_BACKFILL_SPEND_CAP
       @resume_pending_after = 5.minutes
       @vector_store = :auto
       @embedding_cost_per_million_tokens = 0.02

data/lib/truffler/definition.rb

@@ -4,7 +4,7 @@ module Truffler
     EXPLICIT_ACTIONS = %i[enter key row].freeze
     DEFAULT_EMBEDDINGS = { model: "text-embedding-3-small", dimensions: 256 }.freeze
 
-    attr_reader :model, :fields, :labels, :exact_sources, :providers, :surfaces
+    attr_reader :model, :fields, :labels, :exact_sources, :providers, :surfaces, :watch_columns
     attr_accessor :tenant_column, :keyword, :embeddings, :order, :arrived_at_column
 
     def initialize(model)
@@ -14,6 +14,7 @@ module Truffler
       @exact_sources = {}
       @providers = {}
       @surfaces = {}
+      @watch_columns = []
       @arrived_at_column = "created_at"
     end
 
@@ -39,6 +40,13 @@ module Truffler
       labels.values.select(&:supplied?)
     end
 
+    # Columns whose change relabels every label: the column-backed fields,
+    # the tenant column, and the model-level `watch` columns. Method-backed
+    # fields never show up in saved_changes, so they need `watch`.
+    def relabel_columns
+      [ *fields, tenant_column, *watch_columns ].compact
+    end
+
     def per_tenant_vocabulary?
       labels.each_value.any?(&:per_tenant?)
     end
@@ -71,13 +79,28 @@ module Truffler
       Array(model.try(:encrypted_attributes)).map(&:to_s) & fields
     end
 
+    # Column checks wait for the table: a model declared at boot on a fresh
+    # database is checked on its first labeling or search instead.
     def validate!
       raise DefinitionError, "#{model.name}: declare the fields Jev reads with `reads`" if fields.empty?
 
-      check_columns([ tenant_column, *fields, *Array(keyword).grep(String), *supplied_labels.flat_map(&:watch) ].compact.uniq)
+      @columns_deferred = !table_available?
+      validate_columns! unless @columns_deferred
       check_embeddings if embeddings
     end
 
+    def validate_columns!
+      return if @columns_checked
+
+      if @columns_deferred
+        return unless table_available?
+
+        model.reset_column_information
+      end
+      check_columns([ tenant_column, *fields, *Array(keyword).grep(String), *watch_columns, *labels.values.flat_map(&:watch) ].compact.uniq)
+      @columns_checked = true
+    end
+
     DEFAULT_RANKING = { label: 1.0, text: 1.0, keyword: 0.5, exact: 1.0, min_similarity: 0.0 }.freeze
     DEFAULT_WEAK_BELOW = 3
 
@@ -95,6 +118,12 @@ module Truffler
 
     private
 
+    def table_available?
+      model.connection.data_source_exists?(model.table_name)
+    rescue ActiveRecord::ActiveRecordError
+      false
+    end
+
     def check_columns(names)
       columns = model.attribute_names
       missing = names.reject { |name| columns.include?(name) || model.method_defined?(name) }
@@ -124,6 +153,10 @@ module Truffler
         @definition.fields.concat(fields.map(&:to_s))
       end
 
+      def watch(*columns)
+        @definition.watch_columns.concat(columns.map(&:to_s))
+      end
+
       def label(key, type, **options)
         @definition.add_label(LabelDefinition.new(key, type, **options))
       end

data/lib/truffler/label_definition.rb

@@ -2,12 +2,14 @@ module Truffler
   # One typed label question. Nouls store their probability, scores their
   # normalized position, and choices one row per option ("label:option") with
   # that option's probability. Choice options may be a callable of the tenant
-  # key, which makes the vocabulary per-tenant.
+  # key, which makes the vocabulary per-tenant; a tenant it gives no options
+  # (`{}` or nil) simply does not have the label.
   #
   # A label with `from:` is supplied by the host: its answer is read from the
   # record in the shape Jev answers normalize to and Jev is never asked. Its
-  # question is optional, `watch:` names extra columns that refresh it, and
-  # `version:` forces a refresh when its logic changes.
+  # question is optional and `version:` forces a refresh when its logic
+  # changes. On any label, `watch:` names extra columns whose change
+  # refreshes (or re-asks) just that label.
   class LabelDefinition
     TYPES = %i[noul choice score].freeze
     KEY = /\A[a-z][a-z0-9_]*\z/
@@ -50,9 +52,15 @@ module Truffler
     def options(tenant_key = nil)
       options = per_tenant? ? @options.call(tenant_key) : @options
       options = Array(options).to_h { |option| [ option, nil ] } unless options.is_a?(Hash)
-      raise DefinitionError, "#{key}: choice options for tenant #{tenant_key.inspect} are empty" if options.empty?
+      options = options.to_h { |option, description| [ option.to_s, description ] }
+      raise DefinitionError, "#{key}: the option name #{NO_OPTION} is reserved" if options.key?(NO_OPTION)
 
-      options.to_h { |option, description| [ option.to_s, description ] }
+      options
+    end
+
+    # False for a per-tenant choice with no options for this tenant.
+    def available?(tenant_key = nil)
+      type != :choice || !per_tenant? || options(tenant_key).any?
     end
 
     def question(tenant_key = nil)
@@ -139,7 +147,7 @@ module Truffler
     def validate_supplied!
       unless supplied?
         raise DefinitionError, "#{key}: a question is required" if instructions.blank?
-        raise DefinitionError, "#{key}: watch: and version: need from:" if @watch.any? || !@version.nil?
+        raise DefinitionError, "#{key}: version: needs from:" unless @version.nil?
 
         return
       end

data/lib/truffler/labeling/backfill.rb

@@ -58,6 +58,7 @@ module Truffler
       def initialize(model, spend_cap: Truffler.config.backfill_spend_cap, batch_size: Truffler.config.batch_size,
         page_size: nil, cursor: nil, spent: 0.0, client: Truffler.config.client, budget: Budget.new)
         @model = model
+        model.truffler_definition.validate_columns!
         @batch_size = batch_size
         @page_size = page_size || batch_size * 5
         @cursor = cursor

data/lib/truffler/labeling/labeler.rb

@@ -14,6 +14,7 @@ module Truffler
 
       def initialize(model, client: Truffler.config.client, budget: Budget.new)
         @model = model
+        model.truffler_definition.validate_columns!
         @client = client
         @budget = budget
       end

data/lib/truffler/labeling/queue.rb

@@ -10,6 +10,7 @@ module Truffler
 
       def initialize(model, config: Truffler.config)
         @model = model
+        model.truffler_definition.validate_columns!
         @config = config
       end
 

data/lib/truffler/labeling/supplied.rb

@@ -12,6 +12,7 @@ module Truffler
 
       def initialize(model)
         @model = model
+        model.truffler_definition.validate_columns!
         @failed_ids = Set.new
       end
 

data/lib/truffler/lenses/drafter.rb

@@ -84,7 +84,8 @@ module Truffler
       # regenerated.
       def vocabulary_for(model, scope, lens)
         definition = model.truffler_definition
-        declared = definition.labels.to_h { |key, label| [ key, label.question(scope.tenant_key) ] }
+        declared = definition.labels.select { |_, label| label.available?(scope.tenant_key) }
+          .to_h { |key, label| [ key, label.question(scope.tenant_key) ] }
         visible = Lenses.visible_lenses(model, tenant_key: scope.tenant_key, user_digest: scope.user_digest)
         visible = visible.reject { |other| other.id == lens&.id }
         declared.merge(visible.each_with_object({}) { |other, all| all.merge!(other.storage_questions) })

data/lib/truffler/model.rb

@@ -62,8 +62,9 @@ module Truffler
     # outside the watched columns, e.g. from the job that classified it.
     def truffler_refresh_labels!
       definition = self.class.truffler_definition
-      keys = definition.supplied_labels.map(&:key)
-      Labeling::Supplied.new(self.class).write([ [ self, keys ] ], tenant_key: definition.tenant_key_for(self)) if keys.any?
+      tenant_key = definition.tenant_key_for(self)
+      keys = definition.supplied_labels.select { |label| label.available?(tenant_key) }.map(&:key)
+      Labeling::Supplied.new(self.class).write([ [ self, keys ] ], tenant_key: tenant_key) if keys.any?
       self
     end
 
@@ -77,10 +78,10 @@ module Truffler
       end
 
       changed = saved_changes.keys
-      if changed.intersect?([ *definition.fields, definition.tenant_column ].compact)
+      if changed.intersect?(definition.relabel_columns)
         truffler_expire_labels
       else
-        watched = definition.supplied_labels.select { |label| changed.intersect?(label.watch) }
+        watched = definition.labels.values.select { |label| changed.intersect?(label.watch) }
         return if watched.empty?
 
         truffler_expire_labels(watched.map(&:key))
@@ -103,7 +104,7 @@ module Truffler
     def truffler_enqueue_embedding
       definition = self.class.truffler_definition
       return unless Embeddings.managed?(definition)
-      return unless previously_new_record? || saved_changes.keys.intersect?([ *definition.fields, definition.tenant_column ].compact)
+      return unless previously_new_record? || saved_changes.keys.intersect?(definition.relabel_columns)
 
       Jobs::EmbedJob.perform_later(self.class.polymorphic_name, id)
     end

data/lib/truffler/query_encoding/encoder.rb

@@ -2,12 +2,19 @@ module Truffler
   module QueryEncoding
     # Encodes one pending query (KTD9). It asks a fixed question set: each
     # label gets `filter | boost | ignore`, each choice label also gets its
-    # options plus `none`, and each of the first 12 word tokens gets
+    # options plus `Truffler::NO_OPTION`, and each of the first 12 word tokens gets
     # `keyword | label_term | filler`. Exact-text tokens (digits, dates,
     # quoted phrases, emails, identifiers) are keywords decided locally and
     # never asked (R18). Query text travels only in `state` ("query" and
     # "tokens"); a token question names its word by position, `tokens[n]`,
-    # so searcher text never lands in an instruction (R8).
+    # so searcher text never lands in an instruction (R8). The label
+    # vocabulary rides along in `state["labels"]` so a token question can
+    # tell a word that names a label from text to match.
+    #
+    # Jev's word roles are then reconciled locally: a keyword that names a
+    # label the query applies (its key, a word of its key, or the chosen
+    # option, ignoring case and plurals, or sharing its first three letters) becomes a label term, and a common
+    # stopword becomes filler.
     #
     # Answers become a `Search::Encoding` with the KTD20 intent vector: boost
     # gives the declared boost, filter narrows and adds `filter_weight`
@@ -22,10 +29,14 @@ module Truffler
       }.freeze
       TOKEN_ROLES = {
         "keyword" => "A word to match in the record text",
-        "label_term" => "A word that names one of the labels above rather than text to match",
+        "label_term" => "A word that names one of the labels in `labels`, or one of its options, rather than text to match",
         "filler" => "A word that carries no meaning for the search"
       }.freeze
-      NO_OPTION = "none".freeze
+      NO_OPTION = Truffler::NO_OPTION
+      STOPWORDS = %w[
+        a about all an and any are at be by for from have i in is it me my now of on or our please so some that the their
+        them there they this to up us was we what when where which who why with you your
+      ].to_set.freeze
 
       Request = Data.define(:state, :questions, :token_ids, :exact_tokens, :unasked_tokens)
 
@@ -54,16 +65,20 @@ module Truffler
             criteria: options)
         end
 
-        words = query.tokens.each_with_index.reject { |token, _| query.exact_tokens.include?(token) }
+        words = query.tokens.each_with_index.reject do |token, position|
+          query.exact_tokens.include?(token) || query.time_position?(position)
+        end
         asked = words.first(MAX_TOKEN_QUESTIONS)
         token_ids = asked.to_h do |_token, position|
           id = :"token__#{position}"
-          questions.choice(id, instructions: %(In the search query, what is the word tokens[#{position}]?), criteria: TOKEN_ROLES)
+          questions.choice(id, instructions: %(In the search query, what is the word tokens[#{position}]? The labels it may name, ) +
+            %(with their options, are in `labels`.), criteria: TOKEN_ROLES)
           [ position, id.to_s ]
         end
 
-        Request.new(state: { "query" => query.normalized, "tokens" => query.tokens }, questions: questions.to_h, token_ids: token_ids,
-          exact_tokens: query.exact_tokens, unasked_tokens: words.drop(MAX_TOKEN_QUESTIONS).map(&:first))
+        state = { "query" => query.normalized, "tokens" => query.tokens, "labels" => vocabulary_state(labels, tenant_key) }
+        Request.new(state: state, questions: questions.to_h, token_ids: token_ids, exact_tokens: query.exact_tokens,
+          unasked_tokens: words.drop(MAX_TOKEN_QUESTIONS).map(&:first))
       end
 
       # Encodes the query pending under `cache_key`. Returns the encoding, or
@@ -103,6 +118,7 @@ module Truffler
         filters = {}
         boosts = {}
         intent = {}
+        terms = []
         labels(model, tenant_key, user_key).each_value do |label|
           key = storage_key(label, answers, tenant_key)
           next unless key
@@ -113,15 +129,15 @@ module Truffler
             intent[key] = label.filter_weight
           when "boost"
             boosts[key] = intent[key] = label.boost || DEFAULT_BOOST
+          else next
           end
+          terms.concat(label_terms(key))
         end
 
-        roles = request.token_ids.transform_values { |id| answers.choice(id) }
         query = Search::Query.new(request.state["query"])
-        keyword_tokens = query.tokens.each_with_index.filter_map do |token, position|
-          token if roles.fetch(position, "keyword") == "keyword"
-        end
-        label_term_tokens = roles.filter_map { |position, role| query.tokens[position] if role == "label_term" }
+        roles = reconcile(query, request.token_ids.transform_values { |id| answers.choice(id) }, terms.uniq)
+        keyword_tokens = query.tokens.each_index.filter_map { |position| query.tokens[position] if roles[position] == "keyword" }
+        label_term_tokens = query.tokens.each_index.filter_map { |position| query.tokens[position] if roles[position] == "label_term" }
         Search::Encoding.new(filters: filters, boosts: boosts, intent_vector: intent, keyword_tokens: keyword_tokens,
           label_term_tokens: label_term_tokens)
       end
@@ -162,7 +178,7 @@ module Truffler
       end
 
       # The label's storage key the query names, or nil for a choice label
-      # whose option answer is `none`.
+      # whose option answer is NO_OPTION.
       def storage_key(label, answers, tenant_key)
         return label.key unless label.type == :choice
 
@@ -170,6 +186,49 @@ module Truffler
         "#{label.key}:#{option}" if option != NO_OPTION && label.options(tenant_key).key?(option)
       end
 
+      def vocabulary_state(labels, tenant_key)
+        labels.transform_values do |label|
+          entry = { "description" => label.description }
+          entry["options"] = label.options(tenant_key).keys if label.type == :choice
+          entry
+        end
+      end
+
+      # {position => role} for every token. Time phrase words are "time";
+      # exact tokens and unasked words are keywords; a keyword naming an
+      # applied label becomes a label term; stopwords become filler unless
+      # they are all that would be left of an encoding that applies nothing.
+      def reconcile(query, answered, terms)
+        roles = query.tokens.each_index.to_h do |position|
+          [ position, query.time_position?(position) ? "time" : answered.fetch(position, "keyword") ]
+        end
+        words = roles.keys.select { |position| roles[position] == "keyword" && !query.exact_tokens.include?(query.tokens[position]) }
+        words.each { |position| roles[position] = "label_term" if names_label?(query.tokens[position], terms) }
+        stopwords = words.select { |position| roles[position] == "keyword" && STOPWORDS.include?(query.tokens[position]) }
+        return roles if terms.empty? && roles.values.count("keyword") == stopwords.size
+
+        stopwords.each { |position| roles[position] = "filler" }
+        roles
+      end
+
+      # "category:billing" names "category" and "billing"; "needs_action"
+      # names "needs_action", "needs", and "action".
+      STEM = 3
+
+      def label_terms(storage_key)
+        label, option = Search::Encoding.split_key(storage_key)
+        [ label.split(":").last, option ].compact.flat_map { |name| [ name.downcase, *name.downcase.split(/[^\p{Alnum}]+/) ] }
+          .reject(&:empty?).map(&:singularize)
+      end
+
+      # "angry" names "anger": the same word ignoring plurals, or two words of
+      # four letters or more that share their first three letters. Only the
+      # applied labels' terms are compared, so the loose match stays safe.
+      def names_label?(word, terms)
+        word = word.singularize
+        terms.any? { |term| term == word || (word.length > STEM && term.length > STEM && word[0, STEM] == term[0, STEM]) }
+      end
+
       def intent_instructions(label)
         %(How does the search query use the label "#{label.key}" (#{label.description})?)
       end

data/lib/truffler/search/encoding.rb

@@ -12,7 +12,9 @@ module Truffler
     # - `keyword_tokens`: query tokens for the keyword source; nil means
     #   every token that is not a label term.
     # - `label_term_tokens`: tokens that named a label rather than a keyword.
-    Encoding = Data.define(:filters, :boosts, :intent_vector, :keyword_tokens, :label_term_tokens) do
+    # - `time`: the query's `TimeRange`, resolved at search time and never
+    #   cached, since "today" moves.
+    Encoding = Data.define(:filters, :boosts, :intent_vector, :keyword_tokens, :label_term_tokens, :time) do
       def self.load(value, query)
         return value if value.is_a?(self)
         return if value.nil?
@@ -23,12 +25,12 @@ module Truffler
           label_term_tokens: Array(value["label_term_positions"]).filter_map { |position| tokens[position] })
       end
 
-      def initialize(filters: {}, boosts: {}, intent_vector: nil, keyword_tokens: nil, label_term_tokens: [])
+      def initialize(filters: {}, boosts: {}, intent_vector: nil, keyword_tokens: nil, label_term_tokens: [], time: nil)
         filters = weights(filters)
         boosts = weights(boosts)
         intent_vector = weights(intent_vector || boosts).reject { |_, weight| weight.zero? }
         super(filters: filters.freeze, boosts: boosts.freeze, intent_vector: intent_vector.freeze,
-          keyword_tokens: keyword_tokens&.map(&:to_s)&.freeze, label_term_tokens: Array(label_term_tokens).map(&:to_s).freeze)
+          keyword_tokens: keyword_tokens&.map(&:to_s)&.freeze, label_term_tokens: Array(label_term_tokens).map(&:to_s).freeze, time: time)
       end
 
       def empty?
@@ -36,13 +38,14 @@ module Truffler
       end
 
       # The encoding minus the chips the searcher removed (R20), matched by
-      # storage key or by label key.
+      # storage key or by label key; "time" removes the time range.
       def without(suppressed)
         suppressed = Array(suppressed).map(&:to_s).to_set
         return self if suppressed.empty?
 
         keep = ->(key, _) { !suppressed.include?(key) && !suppressed.include?(self.class.split_key(key).first) }
-        with(filters: filters.select(&keep), boosts: boosts.select(&keep), intent_vector: intent_vector.select(&keep))
+        with(filters: filters.select(&keep), boosts: boosts.select(&keep), intent_vector: intent_vector.select(&keep),
+          time: (time unless suppressed.include?(TimeRange.key)))
       end
 
       # Splits a storage key into its label key and choice option. Lens keys
@@ -58,7 +61,7 @@ module Truffler
       end
 
       def keywords(query)
-        keyword_tokens || (query.tokens - label_term_tokens)
+        keyword_tokens || (query.search_tokens - label_term_tokens)
       end
 
       # The cache form: decisions plus token positions in the normalized

data/lib/truffler/search/keystroke.rb

@@ -10,9 +10,10 @@ module Truffler
       attr_reader :model, :query, :tenant_key, :scope, :user_key, :suppressed, :surface, :limit
 
       def initialize(model, query, tenant:, scope:, user: nil, suppressed: [], surface: nil, limit: DEFAULT_LIMIT, weights: {},
-        cache: EncodingCache.new)
+        cache: EncodingCache.new, clock: -> { Time.current })
         @model = model
         @definition = model.try(:truffler_definition) || raise(DefinitionError, "#{model.name} has no truffler declaration")
+        @definition.validate_columns!
         @query = Query.wrap(query)
         @tenant_key = tenant&.to_s
         @scope = scope.nil? && !@definition.scoped? ? model.all : scope
@@ -22,6 +23,7 @@ module Truffler
         @limit = limit
         @weights = @definition.ranking.merge(weights.to_h { |key, weight| [ key.to_sym, Float(weight) ] })
         @cache = cache
+        @clock = clock
         check_scope!
       end
 
@@ -39,7 +41,7 @@ module Truffler
         explicit_action = surface_action
         cached = read_encoding
         status = encoding_status(cached)
-        encoding = visible_lenses_only(cached&.without(suppressed), record_usage: true)
+        encoding = with_time(visible_lenses_only(cached&.without(suppressed), record_usage: true))
         sql = sql(encoding)
         records = sql.relation(scope, limit: limit).to_a
         result = Result.new(records: records, query: query, encoding: encoding, encoding_status: status, watermark: watermark,
@@ -52,7 +54,7 @@ module Truffler
       # How many records the same search would return that arrived after
       # `since` (R25). Reads the cache only and never prefetches.
       def count(since:)
-        sql(visible_lenses_only(read_encoding&.without(suppressed))).candidates(scope)
+        sql(with_time(visible_lenses_only(read_encoding&.without(suppressed)))).candidates(scope)
           .where(model.arel_table[@definition.arrived_at_column].gt(since)).count
       end
 
@@ -91,6 +93,15 @@ module Truffler
         encoding.without(hidden)
       end
 
+      # The query's time phrase, resolved on this search's clock, unless the
+      # searcher removed its chip.
+      def with_time(encoding)
+        phrase = query.time_phrase
+        return encoding if phrase.nil? || suppressed.include?(TimeRange.key)
+
+        (encoding || Encoding.new).with(time: phrase.range(@clock.call))
+      end
+
       def lens_id(key)
         prefix, id = key.split(":", 3)
         Integer(id, exception: false) if prefix == Lenses::KEY_PREFIX

data/lib/truffler/search/query.rb

@@ -3,14 +3,15 @@ module Truffler
     # A normalized, tokenized search query. Quoted phrases stay one token.
     # Exact-text signals (quoted phrases, digit-bearing tokens, emails, and
     # identifier shapes such as `INV-4471` or `order_id`) are detected
-    # locally and never asked of Jev (R18).
+    # locally and never asked of Jev (R18), and so is a time phrase ("this
+    # week"), whose words are neither exact tokens nor keywords.
     class Query
       QUOTED = /"([^"]*)"/
       EMAIL = /\A[^@\s]+@[^@\s]+\.[a-z]{2,}\z/
       IDENTIFIER = /\A[a-z0-9]+(?:[-_.][a-z0-9]+)+\z/
       EDGE_PUNCTUATION = /\A[^\p{Alnum}@]+|[^\p{Alnum}]+\z/
 
-      attr_reader :raw, :normalized, :tokens, :exact_tokens
+      attr_reader :raw, :normalized, :tokens, :exact_tokens, :time_phrase
 
       def self.wrap(query)
         query.is_a?(self) ? query : new(query)
@@ -23,7 +24,18 @@ module Truffler
       def initialize(raw)
         @raw = raw.to_s
         @normalized = self.class.normalize(raw)
-        @tokens, @exact_tokens = tokenize(normalized)
+        @tokens, exact = tokenize(normalized)
+        @time_phrase = TimePhrase.find(@tokens)
+        @exact_tokens = exact.reject { |position| time_position?(position) }.map { |position| @tokens[position] }.freeze
+      end
+
+      def time_position?(position)
+        time_phrase&.positions&.include?(position) || false
+      end
+
+      # The tokens a keyword source may match: all but the time phrase.
+      def search_tokens
+        tokens.each_index.reject { |position| time_position?(position) }.map { |position| tokens[position] }
       end
 
       def blank?
@@ -44,19 +56,19 @@ module Truffler
             phrase = part.squish
             next if phrase.empty?
 
+            exact << tokens.size
             tokens << phrase
-            exact << phrase
           else
             part.split.each do |word|
               word = word.gsub(EDGE_PUNCTUATION, "")
               next if word.empty?
 
+              exact << tokens.size if exact?(word)
               tokens << word
-              exact << word if exact?(word)
             end
           end
         end
-        [ tokens.freeze, exact.freeze ]
+        [ tokens.freeze, exact ]
       end
 
       def exact?(word)

data/lib/truffler/search/result.rb

@@ -29,12 +29,14 @@ module Truffler
         records.map(&:id)
       end
 
-      # Applied filters, then boosts, as `{key:, label:, kind:, name:}`.
+      # Applied filters, then boosts, then the time range, as `{key:, label:, kind:, name:}`.
       def chips
         return [] unless encoding
 
         filters = encoding.filters.keys.map { |key| chip(key, :filter) }
-        filters + (encoding.boosts.keys - encoding.filters.keys).map { |key| chip(key, :boost) }
+        chips = filters + (encoding.boosts.keys - encoding.filters.keys).map { |key| chip(key, :boost) }
+        time = encoding.time
+        time ? chips + [ { key: TimeRange.key, label: TimeRange.key, kind: :time, name: time.name } ] : chips
       end
 
       def score(record)

data/lib/truffler/search/sql.rb

@@ -24,16 +24,18 @@ module Truffler
         @store = store
       end
 
-      # The caller's relation, ANDed with the tenant and the hard filters.
+      # The caller's relation, ANDed with the tenant, the time range, and the hard filters.
       def base(scope)
         scope = scope.where(definition.tenant_column => tenant_key) if definition.scoped?
+        scope = within_time(scope, encoding.time) if encoding.time
         encoding.filters.reduce(scope) { |relation, (key, threshold)| relation.where(Arel.sql(label_filter_sql(key, threshold))) }
       end
 
-      # Every record the query can return, before ranking.
+      # Every record the query can return, before ranking. Under a label
+      # filter the filter decides membership and text matches only rank.
       def candidates(scope)
         base = base(scope)
-        return base if label_only?
+        return base if label_only? || encoding.filters.any?
 
         conditions = [ keyword_sql, exact_sql, (text_candidate_sql if text_score_sql) ].compact
         base.where(Arel.sql(conditions.any? ? conditions.map { |condition| "(#{condition})" }.join(" OR ") : "1 = 0"))
@@ -104,6 +106,12 @@ module Truffler
         "#{label_column('record_type')} = #{quote(model.polymorphic_name)}#{tenant} AND #{label_column('record_id')} = #{primary_key}"
       end
 
+      def within_time(scope, range)
+        arrived_at = model.arel_table[definition.arrived_at_column]
+        scope = scope.where(arrived_at.gteq(range.from))
+        range.to ? scope.where(arrived_at.lt(range.to)) : scope
+      end
+
       def label_filter_sql(key, threshold)
         "EXISTS (SELECT 1 FROM #{quoted_labels} WHERE #{label_scope_sql} AND #{label_column('label_key')} = #{quote(key)} " \
           "AND #{label_column('value')} >= #{Float(threshold)})"

data/lib/truffler/search/time_phrase.rb

@@ -0,0 +1,70 @@
+module Truffler
+  module Search
+    # A time window named in a search query (R16, R18): computed here from
+    # the query tokens and a clock, never asked of Jev. Matching needs no
+    # clock, so the phrase is part of the query; `window(now)` resolves it
+    # to `[from, to]`, where `to` is nil for "until now".
+    #
+    #   today, yesterday, this week, last week, this month, last month,
+    #   past|last N day(s)|week(s), since monday..sunday
+    #
+    # Weeks start on `Date.beginning_of_week` (Monday by default). Only the
+    # first phrase in a query counts; a quoted phrase is exact text.
+    TimePhrase = Data.define(:name, :positions, :resolver) do
+      def self.find(tokens)
+        tokens.each_index do |start|
+          PATTERNS.each do |pattern|
+            phrase = pattern.call(tokens, start)
+            return phrase if phrase
+          end
+        end
+        nil
+      end
+
+      def window(now)
+        resolver.call(now)
+      end
+
+      def range(now)
+        from, to = window(now)
+        TimeRange.new(name: name, from: from, to: to)
+      end
+
+      def self.fixed(words, name, &resolver)
+        lambda do |tokens, start|
+          new(name: name, positions: (start...start + words.size).to_a, resolver: resolver) if tokens[start, words.size] == words
+        end
+      end
+
+      def self.rolling(tokens, start)
+        return unless %w[past last].include?(tokens[start]) && tokens[start + 1].to_s.match?(/\A\d{1,3}\z/)
+
+        count = Integer(tokens[start + 1], 10)
+        unit = { "day" => :days, "days" => :days, "week" => :weeks, "weeks" => :weeks }[tokens[start + 2]]
+        return unless unit && count.positive?
+
+        name = "Last #{count} #{count == 1 ? unit.to_s.singularize : unit}"
+        new(name: name, positions: [ start, start + 1, start + 2 ], resolver: ->(now) { [ now - count.public_send(unit), nil ] })
+      end
+
+      def self.since(tokens, start)
+        wday = Date::DAYNAMES.map(&:downcase).index(tokens[start + 1]) if tokens[start] == "since"
+        return unless wday
+
+        new(name: "Since #{Date::DAYNAMES[wday]}", positions: [ start, start + 1 ],
+          resolver: ->(now) { [ (now - ((now.wday - wday) % 7).days).beginning_of_day, nil ] })
+      end
+
+      PATTERNS = [
+        method(:rolling),
+        method(:since),
+        fixed(%w[today], "Today") { |now| [ now.beginning_of_day, nil ] },
+        fixed(%w[yesterday], "Yesterday") { |now| [ now.yesterday.beginning_of_day, now.beginning_of_day ] },
+        fixed(%w[this week], "This week") { |now| [ now.beginning_of_week, nil ] },
+        fixed(%w[last week], "Last week") { |now| [ now.prev_week.beginning_of_week, now.beginning_of_week ] },
+        fixed(%w[this month], "This month") { |now| [ now.beginning_of_month, nil ] },
+        fixed(%w[last month], "Last month") { |now| [ now.prev_month.beginning_of_month, now.beginning_of_month ] }
+      ].freeze
+    end
+  end
+end

data/lib/truffler/search/time_range.rb

@@ -0,0 +1,10 @@
+module Truffler
+  module Search
+    # A resolved time phrase: records whose `arrived_at` column is at or
+    # after `from` and, when `to` is set, before it. Shown as the chip
+    # `{key: "time", kind: :time}` and dropped by suppressing "time".
+    TimeRange = Data.define(:name, :from, :to) do
+      def self.key = "time"
+    end
+  end
+end

data/lib/truffler/smart_search/starter.rb

@@ -55,13 +55,13 @@ module Truffler
         pool = local_ids.first(depth)
         return pool if result.encoding_status == :cached || definition.labels.empty? || pool.size >= depth
 
-        pool + newest_in_scope(exclude: pool, limit: [ @config.smart_candidate_pool - pool.size, 0 ].max)
+        pool + newest_in_scope(exclude: pool, limit: [ @config.smart_candidate_pool - pool.size, 0 ].max, time: result.encoding&.time)
       end
 
-      def newest_in_scope(exclude:, limit:)
+      def newest_in_scope(exclude:, limit:, time:)
         return [] if limit.zero?
 
-        sql = Search::Sql.new(@model, tenant_key: keystroke.tenant_key, query: @query)
+        sql = Search::Sql.new(@model, tenant_key: keystroke.tenant_key, query: @query, encoding: Search::Encoding.new(time: time))
         relation = sql.base(keystroke.scope).where.not(@model.primary_key => exclude)
         column, direction = definition.order
         relation = column ? relation.reorder(column => direction) : relation.unscope(:order)

data/lib/truffler/version.rb

@@ -1,3 +1,3 @@
 module Truffler
-  VERSION = "0.1.0"
+  VERSION = "0.1.1"
 end

data/lib/truffler/vocabulary.rb

@@ -17,11 +17,13 @@ module Truffler
       @model = model
     end
 
-    # {key => LabelDefinition or Lenses::LensLabel}: the declared labels,
-    # then the lens labels keyed "lens:<id>:<label>".
+    # {key => LabelDefinition or Lenses::LensLabel}: the declared labels
+    # the tenant has (a per-tenant choice with no options for it is left
+    # out), then the lens labels keyed "lens:<id>:<label>".
     def labels_for(tenant_key: nil, user_key: nil, all_users: false)
+      declared = definition.per_tenant_vocabulary? ? definition.labels.select { |_, label| label.available?(tenant_key) } : definition.labels
       lenses = Lenses.labels(definition.model, tenant_key: tenant_key, user_key: user_key, all_users: all_users)
-      lenses.empty? ? definition.labels : definition.labels.merge(lenses)
+      lenses.empty? ? declared : declared.merge(lenses)
     end
 
     def fingerprint(label_key, tenant_key: nil)

data/lib/truffler.rb

@@ -19,6 +19,10 @@ loader.do_not_eager_load("#{__dir__}/truffler/lenses/ruby_llm_generator.rb")
 loader.setup
 
 module Truffler
+  # The answer to a choice question in query encoding that means "no
+  # option". The colon keeps it out of host option names, which reject it.
+  NO_OPTION = "truffler:none".freeze
+
   class << self
     def config
       @config ||= Configuration.new

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: truffler
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - Kieran Klaassen
@@ -295,6 +295,8 @@ files:
 - lib/truffler/search/query.rb
 - lib/truffler/search/result.rb
 - lib/truffler/search/sql.rb
+- lib/truffler/search/time_phrase.rb
+- lib/truffler/search/time_range.rb
 - lib/truffler/smart_search.rb
 - lib/truffler/smart_search/dispatcher.rb
 - lib/truffler/smart_search/reranker.rb