huginn_datatable 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a7014d31503c76a5637c2bd15b82b8cf2be1c173ef11575a9325ec8457d7c65e
4
- data.tar.gz: 175542cecf2d1378d0fa571ef538c0f2cccf24d6acab94bb3e0b398e97926261
3
+ metadata.gz: e68c742da97818c79b41681099972a3e4c3cc4613b8500831226abe09b2a87ed
4
+ data.tar.gz: 855168e5b6bbc9df1bceb381484bf20f550fcf57c0d0a35d6aba15c50b20a775
5
5
  SHA512:
6
- metadata.gz: 951dcb3aeb617bc98b23fc8e883e0fa06d572ff312e0888171b17a26b70454eddbf84972bc6dfd91589a3788ac188266d1282d9122f53522923094ebdec0017c
7
- data.tar.gz: 21ed148247e3e9d3a9cff2682d4d19c15c4bdf7806fd0222c4018f829a98fde7f52720e8219e2caaaa929d8c778f5ccc6b7933de9b2245bc6b597c89f27a7745
6
+ metadata.gz: f345b93074fec47d95589bdc06078b5c2585a79e0fd2a7c02c519d3ca35baa286a885f94278a64048004891a139f4c97a293848cf094523cbcff85ccc2276ce5
7
+ data.tar.gz: d2bfa72d92cc096dba420c20e2a9bc0b26f466e0b981b5d9e653d6f5f2dc33c105c5fd12a1c3e3269691ecafd0704efabaec80943f166b610798066420c3fd81
data/CHANGELOG.md CHANGED
@@ -2,6 +2,19 @@
2
2
 
3
3
  ### Unreleased
4
4
 
5
+ #### 0.3.0
6
+
7
+ - **New `:full_text` search strategy** — PostgreSQL full text search over lexemes (`public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, 'term')`), so morphological variants match (e.g. "compras" finds "compra"). Blind to typos, unlike `:pg_trgm`. Uses a GIN tsvector index when the `fts_function` wrapper (IMMUTABLE, dictionary pinned) exists; degrades gracefully to unaccent+ILIKE otherwise.
8
+ - `search_strategy:` now accepts **a single symbol or an Array of them**; multiple strategies are combined with **OR** (a row matches if any strategy matches), e.g. `[ :pg_trgm, :full_text ]` to get typo tolerance plus stemming.
9
+ - New `fts_dictionary:` (default `"portuguese"`) and `fts_function:` (default `"public.f_tsvector"`) configurations, mirroring `unaccent_function`.
10
+ - New generator **`rails g huginn:fts_indexes [Model...]`** scaffolds a migration that creates the IMMUTABLE `public.f_tsvector` wrapper (with `--dictionary`) and GIN tsvector indexes on every `:string`/`:text` column of the Searchable models, including association-scoped ones.
11
+ - `searchable_columns columns: []` is now honored — an empty array means "search only the declared associations", no longer falling back to the model's own string/text columns.
12
+ - Specs for the `Configuration` normalization surface (defaults + `search_strategy` symbol/array shapes).
13
+
14
+ #### 0.2.1
15
+
16
+ - Association-scoped `search` now matches through **primary-key semi-join subqueries** (`pk IN (SELECT DISTINCT pk FROM <base> JOIN <chain> WHERE <fuzzy>)`) instead of `left_joins` — one subquery per distinct association chain, combined with OR. The base relation never joins, so the count is a plain `COUNT(*)`.
17
+
5
18
  #### 0.2.0
6
19
 
7
20
  - **Breaking:** `:pg_trgm` search now uses the PostgreSQL `%` operator (`UNACCENT(col) % UNACCENT('term')`), driven by the **indexable** operator instead of a bare `similarity()` comparison. A `gin_trgm_ops` GIN index on `UNACCENT(col)` is used when present (BitmapOr over the same index with the ILIKE branch); without it, results stay correct via sequential scan.
data/README.md CHANGED
@@ -18,7 +18,7 @@ Huginn is a lightweight query layer for Rails that turns a raw datatable request
18
18
  ## Highlights
19
19
 
20
20
  - **Two-phase execution** — association filters/orders/range become reflection-secured subqueries, then a lean count and `preload` **only on the paginated subset**.
21
- - **Lean counts** — `COUNT(DISTINCT pk)` through a stripped relation; no JOIN materialization.
21
+ - **Lean counts** — the base relation never joins (`COUNT(*)` over a stripped relation); association matches run as pk subqueries instead.
22
22
  - **SQL injection safe ordering/filtering** — every column reference is resolved through Arel reflection, never string-interpolated.
23
23
  - **Accent/typo tolerant search** — `pg_trgm` similarity (`%` operator) OR `unaccent+ILIKE`, with a configurable fallback chain; uses a `gin_trgm_ops` GIN index when one exists.
24
24
  - **Rails conventions** — works with `ActionController::Parameters`, Railtie auto-includes both concerns (toggleable), zero boilerplate.
@@ -66,9 +66,13 @@ gem "huginn"
66
66
  ```ruby
67
67
  # config/initializers/huginn.rb
68
68
  Huginn.configure do |config|
69
- # :pg_trgm (recommended) — trigram similarity (%) OR unaccent+ILIKE
69
+ # :pg_trgm (recommended) — trigram similarity (%) OR unaccent+ILIKE
70
+ # :full_text — PostgreSQL full text search over lexemes
71
+ # (to_tsvector @@ plainto_tsquery)
70
72
  # :unaccent — unaccent + ILIKE only
71
73
  # :simple — plain LIKE
74
+ # Pass an Array to combine strategies with OR, e.g.
75
+ # config.search_strategy = [:pg_trgm, :full_text]
72
76
  config.search_strategy = :pg_trgm
73
77
 
74
78
  # Unaccent function used around search terms/columns. Defaults to the pg
@@ -77,6 +81,13 @@ Huginn.configure do |config|
77
81
  # indexes can actually be used.
78
82
  config.unaccent_function = "unaccent"
79
83
 
84
+ # Full text search (strategy :full_text). The dictionary and the IMMUTABLE
85
+ # tsvector wrapper (`rails g huginn:fts_indexes`) pinned to that dictionary;
86
+ # they only apply when the wrapper exists, otherwise search degrades to the
87
+ # unaccent fallback.
88
+ # config.fts_dictionary = "portuguese"
89
+ # config.fts_function = "public.f_tsvector"
90
+
80
91
  config.pagy_items = 10 # default page size
81
92
  config.pagy_max_items = 500 # hard cap for per_page
82
93
  end
@@ -84,6 +95,8 @@ end
84
95
 
85
96
  > **Threshold:** under `:pg_trgm` the similarity cutoff is the PostgreSQL GUC `pg_trgm.similarity_threshold` (default `0.3`), **not** a gem-level setting. Tune it with `SET pg_trgm.similarity_threshold = 0.4` / `SELECT set_limit(0.4)` on the database.
86
97
 
98
+ > **Strategy tradeoffs:** `:pg_trgm` tolerates typos and substrings but has no notion of morphology. `:full_text` matches morphological variants (e.g. "correndo"/"correr" → same stem) but is blind to typos, and shines on longer text columns — for name/keyword lookups it mostly overlaps with trigram. Combining both (`[:pg_trgm, :full_text]`) unions the two result sets and requires both index sets (`rails g huginn:trigram_indexes` + `rails g huginn:fts_indexes`).
99
+
87
100
  ## Railtie (automatic include)
88
101
 
89
102
  By default the Railtie includes `Huginn::Datatable` and `Huginn::Searchable` into every `ActiveRecord::Base` model. You do **not** need `include` statements unless you opt in selectively:
@@ -129,7 +142,7 @@ Any `column` **or** `association.column` reference is validated and mapped to it
129
142
  Plano.datatable({ orders: [{ "operadora.pessoa.nome" => "asc" }] }, allowed_paths: [{ operadora: :pessoa }])
130
143
  ```
131
144
 
132
- > Association filters/range and ordering use reflection-resolved subqueries (see the "allowlist of associations" section below). The main relation stays singular and the count is `COUNT(DISTINCT pk)`.
145
+ > Association filters/range, ordering and search use reflection-resolved subqueries (see the "allowlist of associations" section below). The base relation stays singular and join-free, so the count is a plain `COUNT(*)`.
133
146
 
134
147
  ## Association allowlist (`allowed_paths`)
135
148
 
@@ -185,7 +198,7 @@ class Person < ApplicationRecord
185
198
  searchable_columns :name, company: [:name, :cnpj]
186
199
  end
187
200
 
188
- Person.search("globex") # matches company.name via a left_join
201
+ Person.search("globex") # matches company.name via a pk subquery
189
202
  Person.search("kayky", distinct: false) # disable the implicit DISTINCT
190
203
  ```
191
204
 
@@ -235,7 +248,7 @@ Notes:
235
248
 
236
249
  ```
237
250
  phase 1 build the relation subqueries (pk IN … / ORDER BY (SELECT …)) + search + filters + order (no data in memory)
238
- phase 2 count SELECT COUNT(DISTINCT "<pk column>") ... (subquery, pk-indexed)
251
+ phase 2 count SELECT COUNT(*) ... (base relation is join-free)
239
252
  paginate offset / limit
240
253
  preload SELECT ... WHERE id IN (subset) (2nd lightweight query)
241
254
  ```
@@ -257,7 +270,7 @@ lib/huginn/datatable/filter_normalizer.rb functional param normalization
257
270
  lib/huginn/datatable/paginator.rb lean count, pagination, isolated preload
258
271
  lib/huginn/searchable.rb Huginn::Searchable (aggregator)
259
272
  lib/huginn/searchable/searchable.rb the search Concern + DSL
260
- lib/huginn/searchable/query.rb tolerant search builder (joins + OR)
273
+ lib/huginn/searchable/query.rb tolerant search builder (subqueries + OR)
261
274
  lib/huginn/searchable/fuzzy.rb pg_trgm / unaccent / simple predicates
262
275
  lib/generators/... `huginn:trigram_indexes` (GIN index migrations)
263
276
  ```
data/README.pt-BR.md CHANGED
@@ -18,7 +18,7 @@ O Huginn é uma camada de consulta leve para Rails que transforma uma requisiç
18
18
  ## Destaques
19
19
 
20
20
  - **Execução em duas fases** — filtros/orders/ranges de associação se tornam subqueries resolvidas por reflexão, e o `preload` é feito **somente no subconjunto paginado**.
21
- - **Counts enxutos** — `COUNT(DISTINCT pk)` através de uma relation restrita, sem materialização de JOINs.
21
+ - **Counts enxutos** — a relation base nunca faz join (`COUNT(*)` sobre uma relation restrita); as buscas de associação acontecem via subqueries no pk.
22
22
  - **Ordenação/filtro seguros contra SQL injection** — toda referência de coluna é resolvida via reflexão do Arel, nunca interpolada como string.
23
23
  - **Busca tolerante a acentos/typos** — similaridade `pg_trgm` (operador `%`) OU `unaccent+ILIKE`, com cadeia de fallback configurável; usa índice GIN `gin_trgm_ops` quando existente.
24
24
  - **Convenções do Rails** — funciona com `ActionController::Parameters`, Railtie inclui ambos os concerns automaticamente (desligável), zero boilerplate.
@@ -66,9 +66,13 @@ gem "huginn"
66
66
  ```ruby
67
67
  # config/initializers/huginn.rb
68
68
  Huginn.configure do |config|
69
- # :pg_trgm (recomendado) — similaridade trigram (%) OU unaccent+ILIKE
69
+ # :pg_trgm (recomendado) — similaridade trigram (%) OU unaccent+ILIKE
70
+ # :full_text — full text search do PostgreSQL sobre lexemas
71
+ # (to_tsvector @@ plainto_tsquery)
70
72
  # :unaccent — somente unaccent + ILIKE
71
73
  # :simple — LIKE simples
74
+ # Para combinar estratégias com OR, passe um Array:
75
+ # config.search_strategy = [:pg_trgm, :full_text]
72
76
  config.search_strategy = :pg_trgm
73
77
 
74
78
  # Função unaccent usada em torno de termos/colunas. O default é o UNACCENT()
@@ -77,6 +81,13 @@ Huginn.configure do |config|
77
81
  # GIN trigram possam ser usados de fato.
78
82
  config.unaccent_function = "unaccent"
79
83
 
84
+ # Full text search (estratégia :full_text). O dicionário e o wrapper
85
+ # tsvector IMMUTABLE (`rails g huginn:fts_indexes`) fixado nesse dicionário;
86
+ # só têm efeito quando o wrapper existe — caso contrário a busca degrada
87
+ # para o fallback unaccent.
88
+ # config.fts_dictionary = "portuguese"
89
+ # config.fts_function = "public.f_tsvector"
90
+
80
91
  config.pagy_items = 10 # tamanho de página padrão
81
92
  config.pagy_max_items = 500 # teto máximo de per_page
82
93
  end
@@ -84,6 +95,8 @@ end
84
95
 
85
96
  > **Limiar:** sob `:pg_trgm`, o corte de similaridade é o GUC do PostgreSQL `pg_trgm.similarity_threshold` (default `0.3`), **não** uma configuração da gem. Ajuste com `SET pg_trgm.similarity_threshold = 0.4` / `SELECT set_limit(0.4)` no banco.
86
97
 
98
+ > **Tradeoffs de estratégia:** `:pg_trgm` tolera typos e substrings, mas não tem noção de morfologia. `:full_text` captura variações morfológicas (ex. "correndo"/"correr" → mesmo lexema), mas é cego a typos, e brilha em colunas de texto mais longo — para busca por nome/palavra-chave ele em grande parte se sobrepõe ao trigram. Combinar os dois (`[:pg_trgm, :full_text]`) une os dois conjuntos de resultado e exige os dois conjuntos de índices (`rails g huginn:trigram_indexes` + `rails g huginn:fts_indexes`).
99
+
87
100
  ## Railtie (include automático)
88
101
 
89
102
  Por padrão, o Railtie inclui `Huginn::Datatable` e `Huginn::Searchable` em toda `ActiveRecord::Base`. Você **não** precisa de `include`, a menos que opte seletivamente:
@@ -129,7 +142,7 @@ Qualquer referência `column` **ou** `associacao.column` é validada e mapeada p
129
142
  Plano.datatable({ orders: [{ "operadora.pessoa.nome" => "asc" }] }, allowed_paths: [{ operadora: :pessoa }])
130
143
  ```
131
144
 
132
- > Filtros/ranges e order de associação usam subqueries resolvidas por reflexão (veja a seção "Allowlist de associações" abaixo). A relation principal permanece única e o count é `COUNT(DISTINCT pk)`.
145
+ > Filtros/ranges, order e busca de associação usam subqueries resolvidas por reflexão (veja a seção "Allowlist de associações" abaixo). A relation base permanece única e sem joins, então o count é um `COUNT(*)` simples.
133
146
 
134
147
  ## Allowlist de associações (`allowed_paths`)
135
148
 
@@ -185,7 +198,7 @@ class Person < ApplicationRecord
185
198
  searchable_columns :name, company: [:name, :cnpj]
186
199
  end
187
200
 
188
- Person.search("globex") # encontra company.name via left_join
201
+ Person.search("globex") # encontra company.name via subquery no pk
189
202
  Person.search("kayky", distinct: false) # desativa o DISTINCT implícito
190
203
  ```
191
204
 
@@ -235,7 +248,7 @@ Observações:
235
248
 
236
249
  ```
237
250
  fase 1 construir a relation subqueries (pk IN … / ORDER BY (SELECT …)) + search + filters + order (sem dados em memória)
238
- fase 2 count SELECT COUNT(DISTINCT "<pk>") ... (subquery, indexada por pk)
251
+ fase 2 count SELECT COUNT(*) ... (relation base sem joins)
239
252
  paginate offset / limit
240
253
  preload SELECT ... WHERE id IN (subset) (2ª query leve)
241
254
  ```
@@ -257,7 +270,7 @@ lib/huginn/datatable/filter_normalizer.rb normalização funcional de params
257
270
  lib/huginn/datatable/paginator.rb count enxuto, paginação, preload isolado
258
271
  lib/huginn/searchable.rb Huginn::Searchable (agregador)
259
272
  lib/huginn/searchable/searchable.rb o Concern do search + DSL
260
- lib/huginn/searchable/query.rb construtor de busca tolerante (joins + OR)
273
+ lib/huginn/searchable/query.rb construtor de busca tolerante (subqueries + OR)
261
274
  lib/huginn/searchable/fuzzy.rb predicados pg_trgm / unaccent / simple
262
275
  lib/generators/... gerador `huginn:trigram_indexes` (migrations de índices GIN)
263
276
  ```
@@ -0,0 +1,49 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "rails/generators"
4
+ require "rails/generators/migration"
5
+ require "generators/huginn/searchable_columns"
6
+
7
+ module Huginn
8
+ module Generators
9
+ # Scaffolds a migration that creates the PostgreSQL GIN tsvector indexes
10
+ # (on an IMMUTABLE wrapper around to_tsvector) backing Huginn::Searchable's
11
+ # :full_text strategy for every model that includes Huginn::Searchable.
12
+ #
13
+ # rails g huginn:fts_indexes # all Searchable models
14
+ # rails g huginn:fts_indexes Person Product # specific models
15
+ # rails g huginn:fts_indexes --dictionary english # pick the dictionary
16
+ #
17
+ # The migration creates an IMMUTABLE public.f_tsvector(text) wrapper that
18
+ # pins the configured dictionary and the GIN indexes. Point the gem at the
19
+ # wrapper (to_tsvector itself is STABLE and cannot be indexed directly):
20
+ #
21
+ # Huginn.configure { |c| c.search_strategy = :full_text;
22
+ # c.fts_dictionary = "portuguese";
23
+ # c.fts_function = "public.f_tsvector" }
24
+ #
25
+ # Apply the same searchable model/association resolution as the trigram
26
+ # generator, so both index sets cover identical columns.
27
+ class FtsIndexesGenerator < Rails::Generators::Base
28
+ include SearchableColumns
29
+
30
+ source_root File.expand_path("templates", __dir__)
31
+
32
+ argument :model_names, type: :array, default: [], desc: "Models to index (default: all Searchable models)"
33
+
34
+ class_option :dictionary, type: :string, default: "portuguese",
35
+ desc: "tsvector dictionary pinned by the IMMUTABLE wrapper"
36
+
37
+ def copy_migration
38
+ @entries = resolve_entries
39
+ @dictionary = options[:dictionary]
40
+ return if @entries.empty?
41
+
42
+ migration_template(
43
+ "add_huginn_fts_indexes.rb.tt",
44
+ "db/migrate/add_huginn_fts_indexes.rb"
45
+ )
46
+ end
47
+ end
48
+ end
49
+ end
@@ -0,0 +1,24 @@
1
+ class AddHuginnFtsIndexes < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
2
+ def up
3
+ # to_tsvector(regconfig, text) is STABLE (not IMMUTABLE), so it cannot be
4
+ # used in an index expression as-is. Wrap it in an IMMUTABLE function that
5
+ # always pins the configured dictionary.
6
+ execute <<~SQL
7
+ CREATE OR REPLACE FUNCTION public.f_tsvector(text)
8
+ RETURNS tsvector AS $$
9
+ SELECT to_tsvector('<%= @dictionary %>', $1);
10
+ $$
11
+ LANGUAGE sql IMMUTABLE PARALLEL SAFE;
12
+ SQL
13
+ <% @entries.each do |entry| %>
14
+ add_index :<%= entry[:table] %>, "public.f_tsvector(<%= entry[:column] %>)", using: :gin, name: "index_<%= entry[:table] %>_<%= entry[:column] %>_fts"
15
+ <% end %>
16
+ end
17
+
18
+ def down
19
+ <% @entries.each do |entry| %>
20
+ remove_index :<%= entry[:table] %>, name: "index_<%= entry[:table] %>_<%= entry[:column] %>_fts"
21
+ <% end %>
22
+ execute "DROP FUNCTION IF EXISTS public.f_tsvector(text)"
23
+ end
24
+ end
@@ -0,0 +1,69 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "rails/generators"
4
+ require "rails/generators/migration"
5
+
6
+ module Huginn
7
+ module Generators
8
+ # Shared logic between the index-scaffold generators
9
+ # (huginn:trigram_indexes / huginn:fts_indexes): resolving the Searchable
10
+ # models (or the explicit ones given as arguments) into the table/column
11
+ # pairs that need an index, and the migration filename machinery.
12
+ module SearchableColumns
13
+ extend ActiveSupport::Concern
14
+
15
+ included do
16
+ include Rails::Generators::Migration
17
+
18
+ # Rails::Generators::Migration leaves this as NotImplementedError; it
19
+ # is expected to be overridden to produce the next migration number.
20
+ def self.next_migration_number(dirname)
21
+ number = current_migration_number(dirname) + 1
22
+ ActiveRecord::Migration.next_migration_number(number)
23
+ end
24
+ end
25
+
26
+ private
27
+
28
+ def resolve_entries
29
+ models = if model_names.any?
30
+ model_names.map { |name| constantize(name) }
31
+ else
32
+ ActiveRecord::Base.descendants
33
+ end
34
+
35
+ models.filter_map do |model|
36
+ next unless model.respond_to?(:searchable_columns_config_resolved)
37
+
38
+ columns_for(model)
39
+ end.reject(&:empty?).flatten.uniq
40
+ end
41
+
42
+ def columns_for(model)
43
+ scope = []
44
+ resolved = model.searchable_columns_config_resolved
45
+
46
+ Array(resolved[:columns]).map(&:to_s).reject(&:blank?).each do |field|
47
+ scope << { table: model.table_name, column: field } if model.columns_hash.key?(field)
48
+ end
49
+
50
+ resolved[:associations].each do |assoc, cols|
51
+ reflection = model.reflect_on_association(assoc.to_sym)
52
+ next unless reflection
53
+
54
+ Array(cols).map(&:to_s).each do |col|
55
+ scope << { table: reflection.klass.table_name, column: col } if reflection.klass.columns_hash.key?(col)
56
+ end
57
+ end
58
+
59
+ scope
60
+ end
61
+
62
+ def constantize(klass_name)
63
+ klass_name.constantize
64
+ rescue NameError
65
+ raise Thor::Error, "Could not find model #{klass_name.inspect}"
66
+ end
67
+ end
68
+ end
69
+ end
@@ -2,6 +2,7 @@
2
2
 
3
3
  require "rails/generators"
4
4
  require "rails/generators/migration"
5
+ require "generators/huginn/searchable_columns"
5
6
 
6
7
  module Huginn
7
8
  module Generators
@@ -23,17 +24,12 @@ module Huginn
23
24
  # Indexes are only used when the :pg_trgm strategy is configured and the
24
25
  # pg_trgm/unaccent extensions are installed.
25
26
  class TrigramIndexesGenerator < Rails::Generators::Base
26
- include Rails::Generators::Migration
27
+ include SearchableColumns
27
28
 
28
29
  source_root File.expand_path("templates", __dir__)
29
30
 
30
31
  argument :model_names, type: :array, default: [], desc: "Models to index (default: all Searchable models)"
31
32
 
32
- def self.next_migration_number(dirname)
33
- next_migration_number = current_migration_number(dirname) + 1
34
- ActiveRecord::Migration.next_migration_number(next_migration_number)
35
- end
36
-
37
33
  def copy_migration
38
34
  @entries = resolve_entries
39
35
  return if @entries.empty?
@@ -43,48 +39,6 @@ module Huginn
43
39
  "db/migrate/add_huginn_trigram_indexes.rb"
44
40
  )
45
41
  end
46
-
47
- private
48
-
49
- def resolve_entries
50
- models = if model_names.any?
51
- model_names.map { |name| constantize(name) }
52
- else
53
- ActiveRecord::Base.descendants
54
- end
55
-
56
- models.filter_map do |model|
57
- next unless model.respond_to?(:searchable_columns_config_resolved)
58
-
59
- columns_for(model)
60
- end.reject(&:empty?).flatten.uniq
61
- end
62
-
63
- def columns_for(model)
64
- scope = []
65
- resolved = model.searchable_columns_config_resolved
66
-
67
- Array(resolved[:columns]).map(&:to_s).reject(&:blank?).each do |field|
68
- scope << { table: model.table_name, column: field } if model.columns_hash.key?(field)
69
- end
70
-
71
- resolved[:associations].each do |assoc, cols|
72
- reflection = model.reflect_on_association(assoc.to_sym)
73
- next unless reflection
74
-
75
- Array(cols).map(&:to_s).each do |col|
76
- scope << { table: reflection.klass.table_name, column: col } if reflection.klass.columns_hash.key?(col)
77
- end
78
- end
79
-
80
- scope
81
- end
82
-
83
- def constantize(klass_name)
84
- klass_name.constantize
85
- rescue NameError
86
- raise Thor::Error, "Could not find model #{klass_name.inspect}"
87
- end
88
42
  end
89
43
  end
90
44
  end
@@ -2,7 +2,8 @@
2
2
 
3
3
  module Huginn
4
4
  class Configuration
5
- attr_accessor :pagy_items, :pagy_max_items, :unaccent_function
5
+ attr_accessor :pagy_items, :pagy_max_items, :unaccent_function,
6
+ :fts_dictionary, :fts_function
6
7
  attr_writer :search_strategy, :auto_include_datatable, :auto_include_searchable
7
8
 
8
9
  def initialize
@@ -13,19 +14,37 @@ module Huginn
13
14
  # this at an IMMUTABLE wrapper (see `rails g huginn:trigram_indexes`),
14
15
  # e.g. "public.f_unaccent".
15
16
  @unaccent_function = "unaccent"
17
+ # The tsvector wrapper and dictionary used by the :full_text strategy. The
18
+ # wrapper (see `rails g huginn:fts_indexes`) pins the dictionary and is
19
+ # IMMUTABLE so GIN tsvector indexes are usable.
20
+ @fts_dictionary = "portuguese"
21
+ @fts_function = "public.f_tsvector"
16
22
  @search_strategy = :pg_trgm
17
23
  @auto_include_datatable = true
18
24
  @auto_include_searchable = true
19
25
  end
20
26
 
21
- # :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE. The similarity
22
- # cutoff is the PostgreSQL GUC pg_trgm.similarity_threshold
23
- # (tune with SELECT set_limit(...)); a gin_trgm_ops GIN index
24
- # on UNACCENT(col) is used when present.
25
- # :unaccent -> unaccent + ILIKE only
26
- # :simple -> plain LIKE
27
+ # :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE. The
28
+ # similarity cutoff is the PostgreSQL GUC
29
+ # pg_trgm.similarity_threshold (tune with
30
+ # SELECT set_limit(...)); a gin_trgm_ops GIN index on
31
+ # UNACCENT(col) is used when present.
32
+ # :full_text -> PostgreSQL full text search (lexemes): the column is
33
+ # matched as to_tsvector(DICTIONARY, col) vs
34
+ # plainto_tsquery(DICTIONARY, term). Needs the
35
+ # fts_function wrapper and a GIN tsvector index.
36
+ # :unaccent -> unaccent + ILIKE only
37
+ # :simple -> plain LIKE
38
+ #
39
+ # Accepts a single symbol or an Array of symbols; when more than one is
40
+ # given the generated predicates are combined with OR (a row matches if
41
+ # any strategy matches).
27
42
  def search_strategy
28
- @search_strategy.to_sym
43
+ if @search_strategy.is_a?(Array)
44
+ @search_strategy.map(&:to_sym)
45
+ else
46
+ @search_strategy.to_sym
47
+ end
29
48
  end
30
49
 
31
50
  def auto_include_datatable?
@@ -77,6 +77,12 @@ module Huginn
77
77
  nil
78
78
  end
79
79
 
80
+ # The Arel predicates folded in by the association scopes of every step.
81
+ # Public so subquery builders can replicate ActiveRecord's scoped joins.
82
+ def scope_constraints
83
+ scope_predicates
84
+ end
85
+
80
86
  # Reverse engineers a join spec usable by `ActiveRecord::Relation#joins`
81
87
  # for filter subqueries: `.joins(:people: { company: ... })`.
82
88
  def self.join_spec_for(names)
@@ -10,10 +10,9 @@ module Huginn
10
10
  # 2. count lean, paginate with Pagy, then preload associations only
11
11
  # on the small paginated subset.
12
12
  #
13
- # The count runs against a stripped relation (no select/includes/order/
14
- # offset/limit) selecting only the primary key with DISTINCT, so that
15
- # PostgreSQL answers it through the PK index as a subquery instead of a
16
- # massive COUNT(DISTINCT ...) over the joins.
13
+ # The base relation never carries joins (association filters, ranges,
14
+ # orders and the search are all resolved through primary-key subqueries),
15
+ # so the count stays a plain COUNT(*) over the stripped relation.
17
16
  class Paginator
18
17
  attr_reader :relation, :params, :includes
19
18
 
@@ -77,10 +76,7 @@ module Huginn
77
76
  end
78
77
 
79
78
  def total_count
80
- stripped = relation.except(:select, :includes, :order, :offset, :limit)
81
- return stripped.count unless stripped.left_outer_joins_values.any? || stripped.joins_values.any?
82
-
83
- stripped.select(stripped.arel_table[stripped.primary_key]).distinct.count
79
+ relation.except(:select, :includes, :order, :offset, :limit).count
84
80
  end
85
81
 
86
82
  def config
@@ -9,6 +9,11 @@ module Huginn
9
9
  # :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE (best for typos).
10
10
  # Both branches are supported by a gin_trgm_ops GIN index
11
11
  # on UNACCENT(col); the index is used only when it exists.
12
+ # :full_text -> PostgreSQL full text search: to_tsvector(col) @@
13
+ # plainto_tsquery(term), so morphological variants match;
14
+ # needs the IMMUTABLE fts_function wrapper and a GIN
15
+ # tsvector index. Effective with a single strategy or
16
+ # combined with :pg_trgm (OR).
12
17
  # :unaccent -> unaccent + ILIKE (case/accents insensitive)
13
18
  # :simple -> plain LIKE
14
19
  module Fuzzy
@@ -90,6 +95,45 @@ module Huginn
90
95
  Unaccent.new(@column, @value).predicate
91
96
  end
92
97
  end
98
+
99
+ # PostgreSQL full text search over lexemes:
100
+ #
101
+ # public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, term)
102
+ #
103
+ # The column side runs through the configured fts_function (an IMMUTABLE
104
+ # wrapper around to_tsvector that pins the dictionary, created by
105
+ # `rails g huginn:fts_indexes`) so a GIN tsvector index is usable. The
106
+ # query side uses plainto_tsquery, which ANDs the lexemes of the term and
107
+ # applies stemming — tolerant to morphological variants, blind to typos.
108
+ class FullText
109
+ def initialize(column, value, dictionary: nil)
110
+ @column = column
111
+ @value = value.to_s
112
+ @dictionary = dictionary || Huginn.configuration.fts_dictionary
113
+ end
114
+
115
+ def predicate
116
+ Arel::Nodes::InfixOperation.new("@@", tsvector, query)
117
+ end
118
+
119
+ private
120
+
121
+ def tsvector
122
+ Arel::Nodes::NamedFunction.new(Huginn.configuration.fts_function, [@column])
123
+ end
124
+
125
+ def query
126
+ Arel::Nodes::NamedFunction.new("plainto_tsquery", [regconfig, quoted_value])
127
+ end
128
+
129
+ def regconfig
130
+ Arel.sql("'#{@dictionary}'::regconfig")
131
+ end
132
+
133
+ def quoted_value
134
+ Arel::Nodes.build_quoted(@value, @column)
135
+ end
136
+ end
93
137
  end
94
138
  end
95
- end
139
+ end
@@ -1,13 +1,31 @@
1
1
  # frozen_string_literal: true
2
2
 
3
+ require_relative "../datatable/association_path"
4
+
3
5
  module Huginn
4
6
  module Searchable
7
+ AssociationPath = Datatable::AssociationPath unless const_defined?(:AssociationPath)
5
8
  # Builds a tolerant full-text search relation over a model's searchable
6
- # columns, optionally crossing associations through left_joins.
9
+ # columns. The matching predicate per column is driven by
10
+ # Huginn.configuration.search_strategy (:pg_trgm, :full_text, :unaccent,
11
+ # :simple — or an Array of them combined with OR).
12
+ #
13
+ # Columns are split in two groups following the datatable pattern:
14
+ #
15
+ # * the model's own columns -> direct fuzzy predicates on the base WHERE
16
+ # * association columns -> reverse semi-join subqueries anchored on
17
+ # the first FK/has_many hop, so the base
18
+ # table is never re-scanned:
7
19
  #
8
- # All matched columns are fused into a single Arel OR predicate, so the
9
- # whole search stays one query regardless of how many columns (or
10
- # associations) are involved.
20
+ # belongs_to -> base[fk] IN (SELECT DISTINCT t1[pk] FROM t1 ...)
21
+ # has_many -> base[pk] IN (SELECT DISTINCT t1[fk] FROM t1 ...)
22
+ #
23
+ # Through/hand-joined chains keep the generic fallback:
24
+ # base[pk] IN (SELECT DISTINCT base[pk] FROM base JOIN <chain> ...)
25
+ #
26
+ # The base relation never carries joins, so every match group stays one
27
+ # query and the count remains a plain COUNT(*). Multiple association
28
+ # chains produce one subquery each, combined with OR.
11
29
  class Query
12
30
  def self.call(model, value, options = {})
13
31
  new(model, value, options).call
@@ -25,58 +43,152 @@ module Huginn
25
43
  return apply_distinct(relation) if @value.strip.empty?
26
44
 
27
45
  resolution = resolve_columns
28
- return apply_distinct(relation) if resolution.fetch(:attrs).empty?
46
+ predicates = fuzzy_predicates(resolution.fetch(:own))
47
+ resolution.fetch(:associations).each do |_chain, group|
48
+ predicate = association_predicate(group)
49
+ predicates << predicate if predicate
50
+ end
29
51
 
30
- joins = resolution.fetch(:joins)
31
- relation = relation.left_joins(*joins) if joins.any?
52
+ return apply_distinct(relation) if predicates.empty?
32
53
 
33
- predicate = resolution.fetch(:attrs).map { |attr| fuzzy_predicate(attr) }.reduce(&:or)
34
- apply_distinct(relation.where(predicate))
54
+ apply_distinct(relation.where(predicates.reduce(&:or)))
35
55
  end
36
56
 
37
57
  private
38
58
 
59
+ def fuzzy_predicates(attrs)
60
+ attrs.map { |attr| fuzzy_predicate(attr) }.compact
61
+ end
62
+
39
63
  def resolve_columns
40
64
  columns = @model.searchable_columns_config_resolved
41
- joins = []
42
- attrs = []
65
+ own = []
66
+ associations = Hash.new { |hash, key| hash[key] = { path: nil, attrs: [] } }
43
67
 
44
68
  Array(columns[:columns]).map(&:to_s).reject(&:blank?).each do |field|
45
69
  if field.include?(".")
46
- name, col = field.split(".", 2)
47
- push_association(joins, attrs, name, col)
70
+ chain, column = field.split(".", 2)
71
+ push_association(associations, chain, column)
48
72
  else
49
- attrs << @model.arel_table[field] if @model.columns_hash.key?(field)
73
+ own << @model.arel_table[field] if @model.columns_hash.key?(field)
50
74
  end
51
75
  end
52
76
 
53
77
  columns[:associations].each do |name, cols|
54
- Array(cols).each { |col| push_association(joins, attrs, name.to_s, col.to_s) }
78
+ Array(cols).each { |column| push_association(associations, name.to_s, column.to_s) }
55
79
  end
56
80
 
57
- { joins: joins.uniq, attrs: attrs }
81
+ { own: own, associations: associations }
82
+ end
83
+
84
+ def push_association(groups, chain, column)
85
+ path = AssociationPath.call(@model, "#{chain}.#{column}")
86
+ return unless path.valid?
87
+ return unless path.target_klass.columns_hash.key?(path.column)
88
+
89
+ group = groups[path.association_names]
90
+ group[:path] ||= path
91
+ group[:attrs] << path.target_table[path.column]
92
+ end
93
+
94
+ # Prefers the FK-anchored reverse semi-join; falls back to the generic
95
+ # pk subquery for through chains or anything the reverse plan rejects.
96
+ def association_predicate(group)
97
+ path = group[:path]
98
+ attrs = group[:attrs]
99
+ return nil if path.nil? || attrs.empty?
100
+
101
+ anchored_predicate(path, attrs)
102
+ rescue StandardError
103
+ generic_predicate(path, attrs)
104
+ end
105
+
106
+ # Reverse semi-join:
107
+ #
108
+ # belongs_to -> base[fk] IN (SELECT DISTINCT t1[join_pk] FROM t1 ...)
109
+ # has_many -> base[pk] IN (SELECT DISTINCT t1[fk] FROM t1 ...)
110
+ #
111
+ # Walks the chain from the first target table outwards so the base table
112
+ # is never scanned inside the subquery.
113
+ def anchored_predicate(path, attrs)
114
+ if through_association?(path)
115
+ return generic_predicate(path, attrs)
116
+ end
117
+
118
+ steps = path.resolved_steps
119
+ first = steps.first
120
+ reflection = first.reflection
121
+
122
+ outer = if reflection.belongs_to?
123
+ arel_table[reflection.foreign_key]
124
+ else
125
+ arel_table[reflection.active_record_primary_key]
126
+ end
127
+
128
+ projection = if reflection.belongs_to?
129
+ first.table[reflection.join_primary_key]
130
+ else
131
+ first.table[reflection.foreign_key]
132
+ end
133
+
134
+ manager = Arel::SelectManager.new
135
+ manager.from(first.table)
136
+ steps.drop(1).each { |step| manager.join(step.table).on(step.link_to_previous) }
137
+ manager.where(fuzzy_predicates(attrs).reduce(&:or))
138
+ path.scope_constraints.each { |constraint| manager.where(constraint) }
139
+ manager.project(projection)
140
+ manager.distinct
141
+
142
+ outer.in(manager)
143
+ end
144
+
145
+ # Whether the first hop goes through a join model; the FK anchoring is
146
+ # ambiguous there so the generic pk subquery is used instead.
147
+ def through_association?(path)
148
+ reflection = @model.reflect_on_association(path.segments.first.to_sym)
149
+ reflection.is_a?(ActiveRecord::Reflection::ThroughReflection)
58
150
  end
59
151
 
60
- def push_association(joins, attrs, name, col)
61
- association = @model.reflect_on_association(name.to_sym)
62
- return unless association
152
+ # Generic fallback semi-join subquery over the base table, mirroring
153
+ # datatable's association_ids_subquery:
154
+ # base[pk] IN (SELECT DISTINCT base[pk] FROM base JOIN <chain> ...).
155
+ def generic_predicate(path, attrs)
156
+ return nil unless path
63
157
 
64
- table = association.klass.arel_table
65
- attrs << table[col] if association.klass.columns_hash.key?(col)
66
- joins << name.to_sym unless joins.include?(name.to_sym)
158
+ spec = AssociationPath.join_spec_for(path.association_names.map(&:to_sym))
159
+ base = @model.all.except(:select, :order, :offset, :limit)
160
+ .select(arel_table[primary_key])
161
+ .distinct
162
+
163
+ predicate = fuzzy_predicates(attrs).reduce(&:or)
164
+ arel_table[primary_key].in(base.joins(spec).where(predicate).arel)
67
165
  end
68
166
 
69
167
  def fuzzy_predicate(attr)
70
- case strategy
168
+ strategies = Array(strategy)
169
+ predicates = strategies.map { |strat| predicate_for_strategy(attr, strat) }.compact
170
+ return nil if predicates.empty?
171
+
172
+ predicates.reduce(&:or)
173
+ end
174
+
175
+ def predicate_for_strategy(attr, strat)
176
+ case strat
71
177
  when :pg_trgm
72
178
  if pg_trgm_available?
73
179
  Fuzzy::Trigram.new(attr, @value).predicate
74
180
  else
75
181
  fallback_fuzzy(attr)
76
182
  end
183
+ when :full_text
184
+ if fts_available?
185
+ Fuzzy::FullText.new(attr, @value).predicate
186
+ else
187
+ fallback_fuzzy(attr)
188
+ end
77
189
  when :unaccent
78
190
  Fuzzy::Unaccent.new(attr, @value).predicate
79
- else
191
+ when :simple
80
192
  Fuzzy::Simple.new(attr, @value).predicate
81
193
  end
82
194
  end
@@ -105,6 +217,27 @@ module Huginn
105
217
  @unaccent_available ||= extension_installed?("unaccent")
106
218
  end
107
219
 
220
+ # Whether the IMMUTABLE tsvector wrapper behind the :full_text strategy
221
+ # exists (created by `rails g huginn:fts_indexes`). FTS is core
222
+ # PostgreSQL, but without the wrapper the predicate would raise, so we
223
+ # degrade to the unaccent fallback instead.
224
+ def fts_available?
225
+ return false unless postgresql?
226
+ @fts_available ||= begin
227
+ schema, name = split_function_name(Huginn.configuration.fts_function)
228
+ @model.connection.select_value(
229
+ ActiveRecord::Base.sanitize_sql_array(
230
+ ["SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname = ? AND p.proname = ?", schema, name]
231
+ )
232
+ ).present?
233
+ end
234
+ end
235
+
236
+ def split_function_name(function)
237
+ schema, _, name = function.to_s.rpartition(".")
238
+ [schema.presence || "public", name]
239
+ end
240
+
108
241
  def extension_installed?(name)
109
242
  @model.connection
110
243
  .select_all(sanitize_extension_query(name))
@@ -121,6 +254,14 @@ module Huginn
121
254
  @model.connection.adapter_name.downcase.include?("postgres")
122
255
  end
123
256
 
257
+ def arel_table
258
+ @model.arel_table
259
+ end
260
+
261
+ def primary_key
262
+ @model.primary_key
263
+ end
264
+
124
265
  def apply_distinct(relation)
125
266
  @distinct ? relation.distinct : relation
126
267
  end
@@ -17,8 +17,8 @@ module Huginn
17
17
  #
18
18
  # When nothing is declared, every :string/:text column of the model is
19
19
  # searched automatically. Columns may be association-scoped ("person.name")
20
- # or declared via keyword arguments; scoped columns are reached through a
21
- # left_join.
20
+ # or declared via keyword arguments; scoped columns are resolved through
21
+ # semi-join subqueries on the primary key (no joins on the base relation).
22
22
  def searchable_columns(*columns, **associations)
23
23
  self.searchable_columns_config = {
24
24
  columns: columns.flatten.compact.map(&:to_s),
@@ -36,11 +36,15 @@ module Huginn
36
36
  end
37
37
 
38
38
  def searchable_columns_config_resolved
39
- config = searchable_columns_config || {}
40
- {
41
- columns: config[:columns].presence || default_searchable_columns,
42
- associations: config[:associations] || {}
43
- }
39
+ config = searchable_columns_config
40
+ if config.nil?
41
+ { columns: default_searchable_columns, associations: {} }
42
+ else
43
+ {
44
+ columns: Array(config[:columns]),
45
+ associations: config[:associations] || {}
46
+ }
47
+ end
44
48
  end
45
49
 
46
50
  private
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Huginn
4
- VERSION = "0.2.0"
4
+ VERSION = "0.3.0"
5
5
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: huginn_datatable
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Kayky Marcelo
@@ -139,6 +139,9 @@ files:
139
139
  - LICENSE.txt
140
140
  - README.md
141
141
  - README.pt-BR.md
142
+ - lib/generators/huginn/fts_indexes/fts_indexes_generator.rb
143
+ - lib/generators/huginn/fts_indexes/templates/add_huginn_fts_indexes.rb.tt
144
+ - lib/generators/huginn/searchable_columns.rb
142
145
  - lib/generators/huginn/trigram_indexes/templates/add_huginn_trigram_indexes.rb.tt
143
146
  - lib/generators/huginn/trigram_indexes/trigram_indexes_generator.rb
144
147
  - lib/huginn.rb