truffler 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 99d7f61714bf653e012b5cea1f9d57cfbde9d390aae048ba0d52387292a37159
-  data.tar.gz: b9826342d2552dae2b1cd9d671ac4d4d448f5fcbe3822b21e5788b7d272365b0
+  metadata.gz: bc0514a875fdc157249448de1514bb62560348ab4dfba22c75e3d0b0137b9072
+  data.tar.gz: 93b65a47fb89826536cdf03bd68efd824193488d20675789776400de2f19fb7a
 SHA512:
-  metadata.gz: 414ddacdd2e195a077792a36b3c1e8050a8e086a99ccb7ffed065776826f6af2c83da47835ce36d663db166b664e2559a01f2f92dcc6c493e120bbcd60ae89f8
-  data.tar.gz: 272d2d762fdcd5e0f6a6d0cec08ca332619a14e6ab8b3b94a5f2670e16091bf0c75395866a5a77ede42e7d22483a95252cbf6b0cee3565cacfb91d2ae5c17569
+  metadata.gz: 4cbcba6f0deaee8afecbbe3e72988a175422b6ce42ba0f3d42b6d30b77f4736f28f485c21e15aac956fb31c4853724c30402872c52f8e5aff44acec36c0850a1
+  data.tar.gz: df78717442f56984544151b2962f3b78d724e001b6111d2ea92c9fd41903cb53c57117f85bac698bac4726b66000608b6a5132352eaa3c0fa30c688c893c1d04

data/CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## [0.1.3]
+
+- Display-name and description words of an applied choice option now match query words exactly (ignoring case and plurals) instead of by shared three-letter prefix. Under a filter, ordinary search words such as "email", "inbox" and "summary" (with Cora applied) or "chat" and "change" (with billing applied) no longer become label terms and keep ranking results. Label keys and option keys still match by shared prefix, so "angry" names `anger`.
+
 ## [0.1.2]
 
 Fixes from happyhappy's production `churn_risk` backfill and its 0.1.1 upgrade.

data/README.md

@@ -183,7 +183,7 @@ result = Email.truffler(params[:q], tenant: Current.account.id, scope: Current.a
 
 A keystroke search makes no network call. It reads the query encoding and query vector from the cache. When the cache misses, it enqueues `EncodeQueryJob`, and the next keystroke or reload picks up the result.
 
-Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, the chosen option, or a word of that option's display name or description, ignoring case and plurals, or a word of four letters or more sharing its first three letters, so "angry" names `anger`) counts as naming the label, and common stopwords are dropped. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
+Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, or the chosen option key, ignoring case and plurals or sharing their first three letters when both words have four letters or more, so "angry" names `anger`; or, matched exactly, a word of that option's display name or description) counts as naming the label, and common stopwords are dropped. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
 
 Time phrases are handled in Ruby and never asked of Jev: `today`, `yesterday`, `this week`, `last week`, `this month`, `last month`, `past|last N hours|days|weeks|months`, `last hour`, `past hour`, `past week`, `past month`, and `since monday` through `since sunday`. "Past/last N units" is a rolling window ending now; "this/last week" and "this/last month" are calendar windows. The first one in a query limits results to records whose `arrived_at` column falls in that window, its words are not keywords, and it shows as a chip `{key: "time", label: "time", kind: :time, name: "This week"}`. Pass `"time"` in `suppressed:` to drop it. Weeks start on `Date.beginning_of_week`, and the window is computed from `Time.current` (or a `clock:` callable passed to the search, for tests).
 

data/lib/truffler/query_encoding/encoder.rb

@@ -120,6 +120,7 @@ module Truffler
         filters = {}
         boosts = {}
         intent = {}
+        stems = []
         terms = []
         labels(model, tenant_key, user_key).each_value do |label|
           key = storage_key(label, answers, tenant_key)
@@ -133,11 +134,12 @@ module Truffler
             boosts[key] = intent[key] = label.boost || DEFAULT_BOOST
           else next
           end
-          terms.concat(label_terms(key, option_name(label, key, tenant_key)))
+          stems.concat(label_terms(key))
+          terms.concat(name_terms(option_name(label, key, tenant_key)))
         end
 
         query = Search::Query.new(request.state["query"])
-        roles = reconcile(query, request.token_ids.transform_values { |id| answers.choice(id) }, terms.uniq)
+        roles = reconcile(query, request.token_ids.transform_values { |id| answers.choice(id) }, stems.uniq, terms.uniq)
         keyword_tokens = query.tokens.each_index.filter_map { |position| query.tokens[position] if roles[position] == "keyword" }
         label_term_tokens = query.tokens.each_index.filter_map { |position| query.tokens[position] if roles[position] == "label_term" }
         Search::Encoding.new(filters: filters, boosts: boosts, intent_vector: intent, keyword_tokens: keyword_tokens,
@@ -205,30 +207,36 @@ module Truffler
       # exact tokens and unasked words are keywords; a keyword naming an
       # applied label becomes a label term; stopwords become filler unless
       # they are all that would be left of an encoding that applies nothing.
-      def reconcile(query, answered, terms)
+      def reconcile(query, answered, stems, terms)
         roles = query.tokens.each_index.to_h do |position|
           [ position, query.time_position?(position) ? "time" : answered.fetch(position, "keyword") ]
         end
         words = roles.keys.select { |position| roles[position] == "keyword" && !query.exact_tokens.include?(query.tokens[position]) }
-        words.each { |position| roles[position] = "label_term" if names_label?(query.tokens[position], terms) }
+        words.each { |position| roles[position] = "label_term" if names_label?(query.tokens[position], stems, terms) }
         stopwords = words.select { |position| roles[position] == "keyword" && STOPWORDS.include?(query.tokens[position]) }
-        return roles if terms.empty? && roles.values.count("keyword") == stopwords.size
+        return roles if stems.empty? && roles.values.count("keyword") == stopwords.size
 
         stopwords.each { |position| roles[position] = "filler" }
         roles
       end
 
       # "category:billing" names "category" and "billing"; "needs_action"
-      # names "needs_action", "needs", and "action". An option's display
-      # name adds its words minus stopwords: "p_17" shown as "Spiral writing
-      # tool" also names "spiral", "writing", and "tool".
+      # names "needs_action", "needs", and "action". These key terms also
+      # match by shared stem (see names_label?).
       STEM = 3
 
-      def label_terms(storage_key, option_name = nil)
+      def label_terms(storage_key)
         label, option = Search::Encoding.split_key(storage_key)
-        keys = [ label.split(":").last, option ].compact.flat_map { |name| [ name.downcase, *name.downcase.split(/[^\p{Alnum}]+/) ] }
-        words = option_name.to_s.downcase.split(/[^\p{Alnum}]+/).reject { |word| STOPWORDS.include?(word) }
-        (keys + words).reject(&:empty?).map(&:singularize)
+        [ label.split(":").last, option ].compact.flat_map { |name| [ name.downcase, *name.downcase.split(/[^\p{Alnum}]+/) ] }
+          .reject(&:empty?).map(&:singularize)
+      end
+
+      # An option's display name adds its words minus stopwords: "p_17" shown
+      # as "Spiral writing tool" names "spiral", "writing", and "tool". These
+      # match exactly only: descriptions are prose, and a stem match on them
+      # would swallow ordinary search words ("chat" against "charge").
+      def name_terms(option_name)
+        option_name.to_s.downcase.split(/[^\p{Alnum}]+/).reject { |word| word.empty? || STOPWORDS.include?(word) }.map(&:singularize)
       end
 
       def option_name(label, storage_key, tenant_key)
@@ -237,12 +245,13 @@ module Truffler
         label.options(tenant_key)[Search::Encoding.split_key(storage_key).last]
       end
 
-      # "angry" names "anger": the same word ignoring plurals, or two words of
-      # four letters or more that share their first three letters. Only the
-      # applied labels' terms are compared, so the loose match stays safe.
-      def names_label?(word, terms)
+      # "angry" names "anger": the same word ignoring plurals, or, against a
+      # label or option key, two words of four letters or more that share
+      # their first three letters. Display-name words match exactly.
+      def names_label?(word, stems, terms)
         word = word.singularize
-        terms.any? { |term| term == word || (word.length > STEM && term.length > STEM && word[0, STEM] == term[0, STEM]) }
+        terms.include?(word) ||
+          stems.any? { |term| term == word || (word.length > STEM && term.length > STEM && word[0, STEM] == term[0, STEM]) }
       end
 
       def intent_instructions(label)

data/lib/truffler/version.rb

@@ -1,3 +1,3 @@
 module Truffler
-  VERSION = "0.1.2"
+  VERSION = "0.1.3"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: truffler
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.3
 platform: ruby
 authors:
 - Kieran Klaassen