huginn_datatable 0.2.1 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: '05719914312baa1225e26059bd5154312ee5df8bc369df6f5f9676d423ed9d0e'
4
- data.tar.gz: b876f81b7add7781a4bd6b7424e3674f251c09ecfe38e0a70e8f891bb2286c76
3
+ metadata.gz: 82a13098551b25e1b656beb853926f7dcbc964c420a64dde0e22182f735a02bb
4
+ data.tar.gz: abdcf5e8f489ad2b6bff0870bde97d443247f6d31b0efc93274a47da81837916
5
5
  SHA512:
6
- metadata.gz: bed2c0bc55d70a6ceaa57a6a84fb264360df75cbab4f42a2a82031073fdd0f15778b1681afcdc35aba31a478e4e8c18a6acfc960b725a1018306211845e72c72
7
- data.tar.gz: 8f58d9e3fec2d2a38350dc010f9b68d6ab60449e27338291dfae8ae7bcaba2b96cbb64f945067487c1ed97889c6642bbb451dfde21644b0170e708ae109aa30a
6
+ metadata.gz: 3a03a5dc06c8f00d4a66a9bd9af8d06ddc2a9a910dbbb5c8a392985f4b0a21a89439d40829d20559232e1a04d2978444fdcf1c39ac86ce20c176af3fbb98a1de
7
+ data.tar.gz: b229398a323d43253aba96f667ca27dc5d76eacdf3bec68d84cb51c482e8838914de5289fdc9093a293016743f5e14c69acdc5b60705ba9e57306d639b86ca74
data/CHANGELOG.md CHANGED
@@ -2,6 +2,26 @@
2
2
 
3
3
  ### Unreleased
4
4
 
5
+ #### 0.4.0
6
+
7
+ - **PostgreSQL array columns in `filters`** (`uuid[]`, `text[]`, `string[]`...), for own columns, `huginn_attributes` aliases and association-scoped columns. The payload is unchanged:
8
+ - one value → the row contains it (`col @> ARRAY[v]`);
9
+ - several values → **any** of them (`col && ARRAY[...]`, the same semantics as `IN`), or **all** of them (`col @> ARRAY[...]`) when the field is declared with `array: :all`;
10
+ - `"null"` → empty (or NULL) array, combinable with values.
11
+ - New `huginn_array_match :field, :all` and `huginn_attributes alias: { column: "table.col", array: :all }` to choose the match mode (default `:any`).
12
+ - Array filter values are cast by the element type (invalid values — e.g. a malformed UUID — are dropped, never reaching SQL) and the literal is built from quoted values only.
13
+ - `filters` accept a Hash whose value is an Array (`filters[col][]=a&filters[col][]=b` → `{ "col" => [a, b] }`) as several values (IN / array match), the same as repeating the key. Before, the Array was taken as one value and never matched.
14
+ - **Fix:** default searchable columns no longer include array columns (`string[]`/`text[]`), which broke `search` with `unaccent(character varying[]) does not exist`.
15
+
16
+ #### 0.3.0
17
+
18
+ - **New `:full_text` search strategy** — PostgreSQL full text search over lexemes (`public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, 'term')`), so morphological variants match (e.g. "compras" finds "compra"). Blind to typos, unlike `:pg_trgm`. Uses a GIN tsvector index when the `fts_function` wrapper (IMMUTABLE, dictionary pinned) exists; degrades gracefully to unaccent+ILIKE otherwise.
19
+ - `search_strategy:` now accepts **a single symbol or an Array of them**; multiple strategies are combined with **OR** (a row matches if any strategy matches), e.g. `[ :pg_trgm, :full_text ]` to get typo tolerance plus stemming.
20
+ - New `fts_dictionary:` (default `"portuguese"`) and `fts_function:` (default `"public.f_tsvector"`) configurations, mirroring `unaccent_function`.
21
+ - New generator **`rails g huginn:fts_indexes [Model...]`** scaffolds a migration that creates the IMMUTABLE `public.f_tsvector` wrapper (with `--dictionary`) and GIN tsvector indexes on every `:string`/`:text` column of the Searchable models, including association-scoped ones.
22
+ - `searchable_columns columns: []` is now honored — an empty array means "search only the declared associations", no longer falling back to the model's own string/text columns.
23
+ - Specs for the `Configuration` normalization surface (defaults + `search_strategy` symbol/array shapes).
24
+
5
25
  #### 0.2.1
6
26
 
7
27
  - Association-scoped `search` now matches through **primary-key semi-join subqueries** (`pk IN (SELECT DISTINCT pk FROM <base> JOIN <chain> WHERE <fuzzy>)`) instead of `left_joins` — one subquery per distinct association chain, combined with OR. The base relation never joins, so the count is a plain `COUNT(*)`.
data/README.md CHANGED
@@ -13,14 +13,14 @@
13
13
  🇺🇸 English · <a href="./README.pt-BR.md">🇧🇷 Português</a>
14
14
  </p>
15
15
 
16
- Huginn is a lightweight query layer for Rails that turns a raw datatable request into a **lean count, a paginated subset and one preload** — instead of a massive JOIN materialized in memory. It also ships a PostgreSQL fuzzy-search builder (`pg_trgm` similarity via the `%` operator with `unaccent` and `ILIKE` fallback) that is tolerant to typos and accents — and that can lean on a GIN index when present.
16
+ Huginn is a lightweight query layer for Rails that turns a raw datatable request into a **lean count, a paginated subset and one preload** — instead of a massive JOIN materialized in memory. It also ships a PostgreSQL tolerant-search builder (`pg_trgm` similarity via the `%` operator with `unaccent` and `ILIKE` fallback, plus a `:full_text` lexeme strategy) that is tolerant to typos and accents — and that can lean on GIN indexes when present.
17
17
 
18
18
  ## Highlights
19
19
 
20
20
  - **Two-phase execution** — association filters/orders/range become reflection-secured subqueries, then a lean count and `preload` **only on the paginated subset**.
21
21
  - **Lean counts** — the base relation never joins (`COUNT(*)` over a stripped relation); association matches run as pk subqueries instead.
22
22
  - **SQL injection safe ordering/filtering** — every column reference is resolved through Arel reflection, never string-interpolated.
23
- - **Accent/typo tolerant search** — `pg_trgm` similarity (`%` operator) OR `unaccent+ILIKE`, with a configurable fallback chain; uses a `gin_trgm_ops` GIN index when one exists.
23
+ - **Tolerant search** — `pg_trgm` similarity (`%` operator) OR `unaccent+ILIKE` (typo/acent tolerant) and `:full_text` over lexemes (morphology-aware); strategies are `search_strategy`-selectable or combined with OR, backed by GIN indexes when present.
24
24
  - **Rails conventions** — works with `ActionController::Parameters`, Railtie auto-includes both concerns (toggleable), zero boilerplate.
25
25
 
26
26
  ## Development
@@ -58,7 +58,7 @@ The `gemfiles/*.gemfile` are generated by Appraisal (committed); their `.lock` f
58
58
  ## Installation
59
59
 
60
60
  ```ruby
61
- gem "huginn"
61
+ gem "huginn_datatable", require: "huginn"
62
62
  ```
63
63
 
64
64
  ## Configuration
@@ -66,9 +66,13 @@ gem "huginn"
66
66
  ```ruby
67
67
  # config/initializers/huginn.rb
68
68
  Huginn.configure do |config|
69
- # :pg_trgm (recommended) — trigram similarity (%) OR unaccent+ILIKE
69
+ # :pg_trgm (recommended) — trigram similarity (%) OR unaccent+ILIKE
70
+ # :full_text — PostgreSQL full text search over lexemes
71
+ # (to_tsvector @@ plainto_tsquery)
70
72
  # :unaccent — unaccent + ILIKE only
71
73
  # :simple — plain LIKE
74
+ # Pass an Array to combine strategies with OR, e.g.
75
+ # config.search_strategy = [:pg_trgm, :full_text]
72
76
  config.search_strategy = :pg_trgm
73
77
 
74
78
  # Unaccent function used around search terms/columns. Defaults to the pg
@@ -77,6 +81,13 @@ Huginn.configure do |config|
77
81
  # indexes can actually be used.
78
82
  config.unaccent_function = "unaccent"
79
83
 
84
+ # Full text search (strategy :full_text). The dictionary and the IMMUTABLE
85
+ # tsvector wrapper (`rails g huginn:fts_indexes`) pinned to that dictionary;
86
+ # they only apply when the wrapper exists, otherwise search degrades to the
87
+ # unaccent fallback.
88
+ # config.fts_dictionary = "portuguese"
89
+ # config.fts_function = "public.f_tsvector"
90
+
80
91
  config.pagy_items = 10 # default page size
81
92
  config.pagy_max_items = 500 # hard cap for per_page
82
93
  end
@@ -84,6 +95,8 @@ end
84
95
 
85
96
  > **Threshold:** under `:pg_trgm` the similarity cutoff is the PostgreSQL GUC `pg_trgm.similarity_threshold` (default `0.3`), **not** a gem-level setting. Tune it with `SET pg_trgm.similarity_threshold = 0.4` / `SELECT set_limit(0.4)` on the database.
86
97
 
98
+ > **Strategy tradeoffs:** `:pg_trgm` tolerates typos and substrings but has no notion of morphology. `:full_text` matches morphological variants (e.g. "correndo"/"correr" → same stem) but is blind to typos, and shines on longer text columns — for name/keyword lookups it mostly overlaps with trigram. Combining both (`[:pg_trgm, :full_text]`) unions the two result sets and requires both index sets (`rails g huginn:trigram_indexes` + `rails g huginn:fts_indexes`).
99
+
87
100
  ## Railtie (automatic include)
88
101
 
89
102
  By default the Railtie includes `Huginn::Datatable` and `Huginn::Searchable` into every `ActiveRecord::Base` model. You do **not** need `include` statements unless you opt in selectively:
@@ -107,7 +120,7 @@ result = Plano.datatable(
107
120
  includes: [{ operadora: { pessoa: [:endereco, :contatos] } }] # preloaded on the page only
108
121
  )
109
122
 
110
- result[:total_count] # Integer (lean COUNT DISTINCT pk)
123
+ result[:total_count] # Integer (lean COUNT(*))
111
124
  result[:data] # ActiveRecord::Relation (paged + preloaded)
112
125
  ```
113
126
 
@@ -117,10 +130,32 @@ Supported params:
117
130
  |---|---|
118
131
  | `page`, `per_page` | Pagination (clamped to `pagy_max_items`) |
119
132
  | `search` | Delegates to `Huginn::Searchable.search` |
120
- | `filters` | Hash / Array of hashes / pairs -> exact or `IN` conditions (`"col" => "null"` → `IS NULL`) |
133
+ | `filters` | Hash (`{ "col" => v }` or `{ "col" => [a, b] }`) / Array of hashes / pairs -> exact or `IN` conditions (`"col" => "null"` → `IS NULL`) |
121
134
  | `range_data` | `{ "created_at" => ["2024-01-01", "2024-12-31"] }` — date or numeric ranges |
122
135
  | `orders` | `[{ "pessoa.nome" => "desc" }]` — plain or association-scoped columns |
123
136
 
137
+ ### Array columns (PostgreSQL)
138
+
139
+ `filters` also work on array columns (`uuid[]`, `text[]`, `string[]`...) — own, aliased or association-scoped — with the same payload:
140
+
141
+ ```ruby
142
+ Person.datatable({ filters: { tag_ids: id } }) # contains id (tag_ids @> ARRAY[id])
143
+ Person.datatable({ filters: [{ tag_ids: a }, { tag_ids: b }] }) # contains a OR b (tag_ids && ARRAY[a, b])
144
+ Person.datatable({ filters: { tag_ids: "null" } }) # empty array
145
+ ```
146
+
147
+ Several values match **any** of them by default (like `IN`). To require **all** of them:
148
+
149
+ ```ruby
150
+ class Person < ApplicationRecord
151
+ huginn_array_match :tag_ids, :all
152
+ # or, with an alias:
153
+ huginn_attributes tags: { column: "people.tag_ids", array: :all }
154
+ end
155
+ ```
156
+
157
+ Values are cast by the element type (an invalid UUID is dropped, never sent to the database) and the array literal is built from quoted values only. Array columns are left out of the default `search` columns.
158
+
124
159
  ### Scoped ordering / filtering
125
160
 
126
161
  Any `column` **or** `association.column` reference is validated and mapped to its real reflected table:
@@ -189,6 +224,14 @@ Person.search("globex") # matches company.name via a
189
224
  Person.search("kayky", distinct: false) # disable the implicit DISTINCT
190
225
  ```
191
226
 
227
+ `searchable_columns columns: []` is honored: an empty array means "search only the declared associations" — it no longer falls back to the model's own `:string`/`:text` columns:
228
+
229
+ ```ruby
230
+ class Person < ApplicationRecord
231
+ searchable_columns columns: [], company: [:name, :cnpj] # only company.name/cnpj are searched
232
+ end
233
+ ```
234
+
192
235
  `Huginn::Datatable` reuses `Huginn::Searchable.search` automatically when the model responds to `search`.
193
236
 
194
237
  ## Trigram indexes (performance)
@@ -231,6 +274,51 @@ Notes:
231
274
  - Only helpful for search terms of **3 or more characters** — trigrams need that many to match. Shorter terms always scan.
232
275
  - To see it working, run a `Person.search("term").explain` with the wrapper configured and a `gin_trgm_ops` index present.
233
276
 
277
+ ## Full-text indexes (performance)
278
+
279
+ Under `:full_text`, each searchable column produces a lexeme predicate backed by a GIN tsvector index:
280
+
281
+ ```sql
282
+ public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, 'term')
283
+ ```
284
+
285
+ Because `to_tsvector()` is `STABLE` (not `IMMUTABLE`), it cannot be indexed directly — you need an `IMMUTABLE` wrapper that pins the dictionary plus an expression index on it:
286
+
287
+ ```sql
288
+ CREATE OR REPLACE FUNCTION public.f_tsvector(text)
289
+ RETURNS tsvector AS $$
290
+ SELECT to_tsvector('portuguese'::regconfig, $1);
291
+ $$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;
292
+
293
+ CREATE INDEX index_people_name_fts ON people USING gin (public.f_tsvector(name));
294
+ ```
295
+
296
+ Then point the gem at the wrapper and dictionary:
297
+
298
+ ```ruby
299
+ Huginn.configure do |c|
300
+ c.search_strategy = :full_text
301
+ c.fts_dictionary = "portuguese"
302
+ c.fts_function = "public.f_tsvector"
303
+ end
304
+ ```
305
+
306
+ Or scaffold the migration for every Searchable model (wrapper + GIN indexes included), picking a dictionary via `--dictionary`:
307
+
308
+ ```sh
309
+ rails g huginn:fts_indexes # all Searchable models
310
+ rails g huginn:fts_indexes Person Product # specific models
311
+ rails g huginn:fts_indexes --dictionary english # pin a different dictionary
312
+ ```
313
+
314
+ The index is **optional** — without the wrapper/index the search degrades gracefully to the unaccent+ILIKE fallback and stays correct (via sequential scan).
315
+
316
+ Notes:
317
+
318
+ - Requires the full text search support bundled with PostgreSQL (no extra extension).
319
+ - `:full_text` has no typo tolerance — combine it with `:pg_trgm` (`search_strategy: [:pg_trgm, :full_text]`) and generate **both** index sets (`huginn:trigram_indexes` + `huginn:fts_indexes`).
320
+ - To see it working, run a `Person.search("term").explain` with the wrapper configured and a GIN tsvector index present.
321
+
234
322
  ## Query efficiency
235
323
 
236
324
  ```
@@ -246,7 +334,7 @@ For a `Plano` datatable with deep `includes:`, this is exactly **2 extra queries
246
334
 
247
335
  ```
248
336
  lib/huginn.rb entry, Huginn.configure, Huginn.instrument
249
- lib/huginn/configuration.rb search_strategy, unaccent_function, pagy_*
337
+ lib/huginn/configuration.rb search_strategy, unaccent_function, fts_*, pagy_*
250
338
  lib/huginn/railtie.rb auto-includes concerns into ActiveRecord
251
339
  lib/huginn/datatable.rb Huginn::Datatable (aggregator)
252
340
  lib/huginn/datatable/datatable.rb the datatable Concern
@@ -258,8 +346,8 @@ lib/huginn/datatable/paginator.rb lean count, pagination, isolated preload
258
346
  lib/huginn/searchable.rb Huginn::Searchable (aggregator)
259
347
  lib/huginn/searchable/searchable.rb the search Concern + DSL
260
348
  lib/huginn/searchable/query.rb tolerant search builder (subqueries + OR)
261
- lib/huginn/searchable/fuzzy.rb pg_trgm / unaccent / simple predicates
262
- lib/generators/... `huginn:trigram_indexes` (GIN index migrations)
349
+ lib/huginn/searchable/fuzzy.rb pg_trgm / full_text / unaccent / simple predicates
350
+ lib/generators/... `huginn:trigram_indexes` + `huginn:fts_indexes` (GIN index migrations)
263
351
  ```
264
352
 
265
353
  ## Instrumentation
data/README.pt-BR.md CHANGED
@@ -13,14 +13,14 @@
13
13
  <a href="./README.md">🇺🇸 English</a> · 🇧🇷 Português
14
14
  </p>
15
15
 
16
- O Huginn é uma camada de consulta leve para Rails que transforma uma requisição de datatable bruta em um **count enxuto, um subconjunto paginado e um único preload** — em vez de um JOIN enorme materializado em memória. Também traz um construtor de busca *fuzzy* para PostgreSQL (similaridade `pg_trgm` via operador `%` com `unaccent` e fallback `ILIKE`) tolerante a erros de digitação e acentuação — e que pode usar um índice GIN quando presente.
16
+ O Huginn é uma camada de consulta leve para Rails que transforma uma requisição de datatable bruta em um **count enxuto, um subconjunto paginado e um único preload** — em vez de um JOIN enorme materializado em memória. Também traz um construtor de busca tolerante a erros para PostgreSQL (similaridade `pg_trgm` via operador `%` com `unaccent` e fallback `ILIKE`, além da estratégia `:full_text` sobre lexemas) que tolera erros de digitação e acentuação — e que pode usar índices GIN quando presentes.
17
17
 
18
18
  ## Destaques
19
19
 
20
20
  - **Execução em duas fases** — filtros/orders/ranges de associação se tornam subqueries resolvidas por reflexão, e o `preload` é feito **somente no subconjunto paginado**.
21
21
  - **Counts enxutos** — a relation base nunca faz join (`COUNT(*)` sobre uma relation restrita); as buscas de associação acontecem via subqueries no pk.
22
22
  - **Ordenação/filtro seguros contra SQL injection** — toda referência de coluna é resolvida via reflexão do Arel, nunca interpolada como string.
23
- - **Busca tolerante a acentos/typos** — similaridade `pg_trgm` (operador `%`) OU `unaccent+ILIKE`, com cadeia de fallback configurável; usa índice GIN `gin_trgm_ops` quando existente.
23
+ - **Busca tolerante** — similaridade `pg_trgm` (operador `%`) OU `unaccent+ILIKE` (tolerante a typos/acentos) e `:full_text` sobre lexemas (ciente de morfologia); as estratégias são selecionáveis via `search_strategy` ou combinadas com OR, respaldadas por índices GIN quando presentes.
24
24
  - **Convenções do Rails** — funciona com `ActionController::Parameters`, Railtie inclui ambos os concerns automaticamente (desligável), zero boilerplate.
25
25
 
26
26
  ## Desenvolvimento
@@ -58,7 +58,7 @@ Os `gemfiles/*.gemfile` são gerados pelo Appraisal (commitados); os arquivos `.
58
58
  ## Instalação
59
59
 
60
60
  ```ruby
61
- gem "huginn"
61
+ gem "huginn_datatable", require: "huginn"
62
62
  ```
63
63
 
64
64
  ## Configuração
@@ -66,9 +66,13 @@ gem "huginn"
66
66
  ```ruby
67
67
  # config/initializers/huginn.rb
68
68
  Huginn.configure do |config|
69
- # :pg_trgm (recomendado) — similaridade trigram (%) OU unaccent+ILIKE
69
+ # :pg_trgm (recomendado) — similaridade trigram (%) OU unaccent+ILIKE
70
+ # :full_text — full text search do PostgreSQL sobre lexemas
71
+ # (to_tsvector @@ plainto_tsquery)
70
72
  # :unaccent — somente unaccent + ILIKE
71
73
  # :simple — LIKE simples
74
+ # Para combinar estratégias com OR, passe um Array:
75
+ # config.search_strategy = [:pg_trgm, :full_text]
72
76
  config.search_strategy = :pg_trgm
73
77
 
74
78
  # Função unaccent usada em torno de termos/colunas. O default é o UNACCENT()
@@ -77,6 +81,13 @@ Huginn.configure do |config|
77
81
  # GIN trigram possam ser usados de fato.
78
82
  config.unaccent_function = "unaccent"
79
83
 
84
+ # Full text search (estratégia :full_text). O dicionário e o wrapper
85
+ # tsvector IMMUTABLE (`rails g huginn:fts_indexes`) fixado nesse dicionário;
86
+ # só têm efeito quando o wrapper existe — caso contrário a busca degrada
87
+ # para o fallback unaccent.
88
+ # config.fts_dictionary = "portuguese"
89
+ # config.fts_function = "public.f_tsvector"
90
+
80
91
  config.pagy_items = 10 # tamanho de página padrão
81
92
  config.pagy_max_items = 500 # teto máximo de per_page
82
93
  end
@@ -84,6 +95,8 @@ end
84
95
 
85
96
  > **Limiar:** sob `:pg_trgm`, o corte de similaridade é o GUC do PostgreSQL `pg_trgm.similarity_threshold` (default `0.3`), **não** uma configuração da gem. Ajuste com `SET pg_trgm.similarity_threshold = 0.4` / `SELECT set_limit(0.4)` no banco.
86
97
 
98
+ > **Tradeoffs de estratégia:** `:pg_trgm` tolera typos e substrings, mas não tem noção de morfologia. `:full_text` captura variações morfológicas (ex. "correndo"/"correr" → mesmo lexema), mas é cego a typos, e brilha em colunas de texto mais longo — para busca por nome/palavra-chave ele em grande parte se sobrepõe ao trigram. Combinar os dois (`[:pg_trgm, :full_text]`) une os dois conjuntos de resultado e exige os dois conjuntos de índices (`rails g huginn:trigram_indexes` + `rails g huginn:fts_indexes`).
99
+
87
100
  ## Railtie (include automático)
88
101
 
89
102
  Por padrão, o Railtie inclui `Huginn::Datatable` e `Huginn::Searchable` em toda `ActiveRecord::Base`. Você **não** precisa de `include`, a menos que opte seletivamente:
@@ -107,7 +120,7 @@ result = Plano.datatable(
107
120
  includes: [{ operadora: { pessoa: [:endereco, :contatos] } }] # preload somente na página
108
121
  )
109
122
 
110
- result[:total_count] # Integer (count enxuto COUNT DISTINCT pk)
123
+ result[:total_count] # Integer (count enxuto COUNT(*))
111
124
  result[:data] # ActiveRecord::Relation (paginada + preloaded)
112
125
  ```
113
126
 
@@ -117,10 +130,32 @@ Parâmetros suportados:
117
130
  |---|---|
118
131
  | `page`, `per_page` | Paginação (limitada a `pagy_max_items`) |
119
132
  | `search` | Delega para `Huginn::Searchable.search` |
120
- | `filters` | Hash / Array de hashes / pares -> condições exatas ou `IN` (`"col" => "null"` → `IS NULL`) |
133
+ | `filters` | Hash (`{ "col" => v }` ou `{ "col" => [a, b] }`) / Array de hashes / pares -> condições exatas ou `IN` (`"col" => "null"` → `IS NULL`) |
121
134
  | `range_data` | `{ "created_at" => ["2024-01-01", "2024-12-31"] }` — ranges de datas ou numéricos |
122
135
  | `orders` | `[{ "pessoa.nome" => "desc" }]` — colunas simples ou de associação |
123
136
 
137
+ ### Colunas array (PostgreSQL)
138
+
139
+ Os `filters` também funcionam em colunas array (`uuid[]`, `text[]`, `string[]`...) — próprias, com alias ou de associação — com o mesmo payload:
140
+
141
+ ```ruby
142
+ Person.datatable({ filters: { tag_ids: id } }) # contém id (tag_ids @> ARRAY[id])
143
+ Person.datatable({ filters: [{ tag_ids: a }, { tag_ids: b }] }) # contém a OU b (tag_ids && ARRAY[a, b])
144
+ Person.datatable({ filters: { tag_ids: "null" } }) # array vazio
145
+ ```
146
+
147
+ Vários valores casam com **qualquer** um por padrão (como o `IN`). Para exigir **todos**:
148
+
149
+ ```ruby
150
+ class Person < ApplicationRecord
151
+ huginn_array_match :tag_ids, :all
152
+ # ou, com alias:
153
+ huginn_attributes tags: { column: "people.tag_ids", array: :all }
154
+ end
155
+ ```
156
+
157
+ Os valores são convertidos pelo tipo do elemento (um UUID inválido é descartado, nunca vai ao banco) e o literal do array é montado só com valores escapados. Colunas array ficam fora das colunas padrão do `search`.
158
+
124
159
  ### Ordenação/filtro com associação
125
160
 
126
161
  Qualquer referência `column` **ou** `associacao.column` é validada e mapeada para a tabela refletida real:
@@ -189,6 +224,14 @@ Person.search("globex") # encontra company.name via s
189
224
  Person.search("kayky", distinct: false) # desativa o DISTINCT implícito
190
225
  ```
191
226
 
227
+ `searchable_columns columns: []` é respeitado: um array vazio significa "pesquisar somente as associações declaradas" — não cai mais para as colunas `:string`/`:text` do próprio modelo:
228
+
229
+ ```ruby
230
+ class Person < ApplicationRecord
231
+ searchable_columns columns: [], company: [:name, :cnpj] # só company.name/cnpj são pesquisados
232
+ end
233
+ ```
234
+
192
235
  `Huginn::Datatable` reutiliza `Huginn::Searchable.search` automaticamente quando o modelo responde a `search`.
193
236
 
194
237
  ## Índices trigram (performance)
@@ -231,6 +274,51 @@ Observações:
231
274
  - Só ajuda em termos de busca de **3 caracteres ou mais** — trigramas precisam disso para casar. Termos menores sempre fazem scan.
232
275
  - Para conferir o uso, rode `Person.search("termo").explain` com o wrapper configurado e um índice `gin_trgm_ops` presente.
233
276
 
277
+ ## Índices full-text (performance)
278
+
279
+ Sob `:full_text`, cada coluna pesquisável gera um predicado por lexema respaldado por um índice GIN tsvector:
280
+
281
+ ```sql
282
+ public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, 'termo')
283
+ ```
284
+
285
+ Como o `to_tsvector()` é `STABLE` (não `IMMUTABLE`), ele não pode ser indexado diretamente — você precisa de um wrapper `IMMUTABLE` que fixe o dicionário além do índice de expressão:
286
+
287
+ ```sql
288
+ CREATE OR REPLACE FUNCTION public.f_tsvector(text)
289
+ RETURNS tsvector AS $$
290
+ SELECT to_tsvector('portuguese'::regconfig, $1);
291
+ $$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;
292
+
293
+ CREATE INDEX index_people_name_fts ON people USING gin (public.f_tsvector(name));
294
+ ```
295
+
296
+ E aponte a gem para o wrapper e o dicionário:
297
+
298
+ ```ruby
299
+ Huginn.configure do |c|
300
+ c.search_strategy = :full_text
301
+ c.fts_dictionary = "portuguese"
302
+ c.fts_function = "public.f_tsvector"
303
+ end
304
+ ```
305
+
306
+ Ou gere a migration para todos os modelos Searchable (wrapper + índices GIN inclusos), escolhendo um dicionário com `--dictionary`:
307
+
308
+ ```sh
309
+ rails g huginn:fts_indexes # todos os modelos Searchable
310
+ rails g huginn:fts_indexes Person Product # modelos específicos
311
+ rails g huginn:fts_indexes --dictionary english # fixar outro dicionário
312
+ ```
313
+
314
+ O índice é **opcional** — sem o wrapper/índice a busca degrada graciosamente para o fallback unaccent+ILIKE e permanece correta (via sequential scan).
315
+
316
+ Observações:
317
+
318
+ - Requer o suporte a full text search embutido no PostgreSQL (sem extensão extra).
319
+ - `:full_text` não tem tolerância a typos — combine com `:pg_trgm` (`search_strategy: [:pg_trgm, :full_text]`) e gere **os dois** conjuntos de índices (`huginn:trigram_indexes` + `huginn:fts_indexes`).
320
+ - Para conferir o uso, rode `Person.search("termo").explain` com o wrapper configurado e um índice GIN tsvector presente.
321
+
234
322
  ## Eficiência da query
235
323
 
236
324
  ```
@@ -246,7 +334,7 @@ Para um datatable de `Plano` com `includes:` profundos, isso são exatamente **2
246
334
 
247
335
  ```
248
336
  lib/huginn.rb entry, Huginn.configure, Huginn.instrument
249
- lib/huginn/configuration.rb search_strategy, unaccent_function, pagy_*
337
+ lib/huginn/configuration.rb search_strategy, unaccent_function, fts_*, pagy_*
250
338
  lib/huginn/railtie.rb auto-inclui os concerns no ActiveRecord
251
339
  lib/huginn/datatable.rb Huginn::Datatable (agregador)
252
340
  lib/huginn/datatable/datatable.rb o Concern do datatable
@@ -258,8 +346,8 @@ lib/huginn/datatable/paginator.rb count enxuto, paginação, preload isolado
258
346
  lib/huginn/searchable.rb Huginn::Searchable (agregador)
259
347
  lib/huginn/searchable/searchable.rb o Concern do search + DSL
260
348
  lib/huginn/searchable/query.rb construtor de busca tolerante (subqueries + OR)
261
- lib/huginn/searchable/fuzzy.rb predicados pg_trgm / unaccent / simple
262
- lib/generators/... gerador `huginn:trigram_indexes` (migrations de índices GIN)
349
+ lib/huginn/searchable/fuzzy.rb predicados pg_trgm / full_text / unaccent / simple
350
+ lib/generators/... geradores `huginn:trigram_indexes` + `huginn:fts_indexes` (migrations de índices GIN)
263
351
  ```
264
352
 
265
353
  ## Instrumentação
@@ -0,0 +1,49 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "rails/generators"
4
+ require "rails/generators/migration"
5
+ require "generators/huginn/searchable_columns"
6
+
7
+ module Huginn
8
+ module Generators
9
+ # Scaffolds a migration that creates the PostgreSQL GIN tsvector indexes
10
+ # (on an IMMUTABLE wrapper around to_tsvector) backing Huginn::Searchable's
11
+ # :full_text strategy for every model that includes Huginn::Searchable.
12
+ #
13
+ # rails g huginn:fts_indexes # all Searchable models
14
+ # rails g huginn:fts_indexes Person Product # specific models
15
+ # rails g huginn:fts_indexes --dictionary english # pick the dictionary
16
+ #
17
+ # The migration creates an IMMUTABLE public.f_tsvector(text) wrapper that
18
+ # pins the configured dictionary and the GIN indexes. Point the gem at the
19
+ # wrapper (to_tsvector itself is STABLE and cannot be indexed directly):
20
+ #
21
+ # Huginn.configure { |c| c.search_strategy = :full_text;
22
+ # c.fts_dictionary = "portuguese";
23
+ # c.fts_function = "public.f_tsvector" }
24
+ #
25
+ # Apply the same searchable model/association resolution as the trigram
26
+ # generator, so both index sets cover identical columns.
27
+ class FtsIndexesGenerator < Rails::Generators::Base
28
+ include SearchableColumns
29
+
30
+ source_root File.expand_path("templates", __dir__)
31
+
32
+ argument :model_names, type: :array, default: [], desc: "Models to index (default: all Searchable models)"
33
+
34
+ class_option :dictionary, type: :string, default: "portuguese",
35
+ desc: "tsvector dictionary pinned by the IMMUTABLE wrapper"
36
+
37
+ def copy_migration
38
+ @entries = resolve_entries
39
+ @dictionary = options[:dictionary]
40
+ return if @entries.empty?
41
+
42
+ migration_template(
43
+ "add_huginn_fts_indexes.rb.tt",
44
+ "db/migrate/add_huginn_fts_indexes.rb"
45
+ )
46
+ end
47
+ end
48
+ end
49
+ end
@@ -0,0 +1,24 @@
1
+ class AddHuginnFtsIndexes < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
2
+ def up
3
+ # to_tsvector(regconfig, text) is STABLE (not IMMUTABLE), so it cannot be
4
+ # used in an index expression as-is. Wrap it in an IMMUTABLE function that
5
+ # always pins the configured dictionary.
6
+ execute <<~SQL
7
+ CREATE OR REPLACE FUNCTION public.f_tsvector(text)
8
+ RETURNS tsvector AS $$
9
+ SELECT to_tsvector('<%= @dictionary %>', $1);
10
+ $$
11
+ LANGUAGE sql IMMUTABLE PARALLEL SAFE;
12
+ SQL
13
+ <% @entries.each do |entry| %>
14
+ add_index :<%= entry[:table] %>, "public.f_tsvector(<%= entry[:column] %>)", using: :gin, name: "index_<%= entry[:table] %>_<%= entry[:column] %>_fts"
15
+ <% end %>
16
+ end
17
+
18
+ def down
19
+ <% @entries.each do |entry| %>
20
+ remove_index :<%= entry[:table] %>, name: "index_<%= entry[:table] %>_<%= entry[:column] %>_fts"
21
+ <% end %>
22
+ execute "DROP FUNCTION IF EXISTS public.f_tsvector(text)"
23
+ end
24
+ end
@@ -0,0 +1,69 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "rails/generators"
4
+ require "rails/generators/migration"
5
+
6
+ module Huginn
7
+ module Generators
8
+ # Shared logic between the index-scaffold generators
9
+ # (huginn:trigram_indexes / huginn:fts_indexes): resolving the Searchable
10
+ # models (or the explicit ones given as arguments) into the table/column
11
+ # pairs that need an index, and the migration filename machinery.
12
+ module SearchableColumns
13
+ extend ActiveSupport::Concern
14
+
15
+ included do
16
+ include Rails::Generators::Migration
17
+
18
+ # Rails::Generators::Migration leaves this as NotImplementedError; it
19
+ # is expected to be overridden to produce the next migration number.
20
+ def self.next_migration_number(dirname)
21
+ number = current_migration_number(dirname) + 1
22
+ ActiveRecord::Migration.next_migration_number(number)
23
+ end
24
+ end
25
+
26
+ private
27
+
28
+ def resolve_entries
29
+ models = if model_names.any?
30
+ model_names.map { |name| constantize(name) }
31
+ else
32
+ ActiveRecord::Base.descendants
33
+ end
34
+
35
+ models.filter_map do |model|
36
+ next unless model.respond_to?(:searchable_columns_config_resolved)
37
+
38
+ columns_for(model)
39
+ end.reject(&:empty?).flatten.uniq
40
+ end
41
+
42
+ def columns_for(model)
43
+ scope = []
44
+ resolved = model.searchable_columns_config_resolved
45
+
46
+ Array(resolved[:columns]).map(&:to_s).reject(&:blank?).each do |field|
47
+ scope << { table: model.table_name, column: field } if model.columns_hash.key?(field)
48
+ end
49
+
50
+ resolved[:associations].each do |assoc, cols|
51
+ reflection = model.reflect_on_association(assoc.to_sym)
52
+ next unless reflection
53
+
54
+ Array(cols).map(&:to_s).each do |col|
55
+ scope << { table: reflection.klass.table_name, column: col } if reflection.klass.columns_hash.key?(col)
56
+ end
57
+ end
58
+
59
+ scope
60
+ end
61
+
62
+ def constantize(klass_name)
63
+ klass_name.constantize
64
+ rescue NameError
65
+ raise Thor::Error, "Could not find model #{klass_name.inspect}"
66
+ end
67
+ end
68
+ end
69
+ end
@@ -2,6 +2,7 @@
2
2
 
3
3
  require "rails/generators"
4
4
  require "rails/generators/migration"
5
+ require "generators/huginn/searchable_columns"
5
6
 
6
7
  module Huginn
7
8
  module Generators
@@ -23,17 +24,12 @@ module Huginn
23
24
  # Indexes are only used when the :pg_trgm strategy is configured and the
24
25
  # pg_trgm/unaccent extensions are installed.
25
26
  class TrigramIndexesGenerator < Rails::Generators::Base
26
- include Rails::Generators::Migration
27
+ include SearchableColumns
27
28
 
28
29
  source_root File.expand_path("templates", __dir__)
29
30
 
30
31
  argument :model_names, type: :array, default: [], desc: "Models to index (default: all Searchable models)"
31
32
 
32
- def self.next_migration_number(dirname)
33
- next_migration_number = current_migration_number(dirname) + 1
34
- ActiveRecord::Migration.next_migration_number(next_migration_number)
35
- end
36
-
37
33
  def copy_migration
38
34
  @entries = resolve_entries
39
35
  return if @entries.empty?
@@ -43,48 +39,6 @@ module Huginn
43
39
  "db/migrate/add_huginn_trigram_indexes.rb"
44
40
  )
45
41
  end
46
-
47
- private
48
-
49
- def resolve_entries
50
- models = if model_names.any?
51
- model_names.map { |name| constantize(name) }
52
- else
53
- ActiveRecord::Base.descendants
54
- end
55
-
56
- models.filter_map do |model|
57
- next unless model.respond_to?(:searchable_columns_config_resolved)
58
-
59
- columns_for(model)
60
- end.reject(&:empty?).flatten.uniq
61
- end
62
-
63
- def columns_for(model)
64
- scope = []
65
- resolved = model.searchable_columns_config_resolved
66
-
67
- Array(resolved[:columns]).map(&:to_s).reject(&:blank?).each do |field|
68
- scope << { table: model.table_name, column: field } if model.columns_hash.key?(field)
69
- end
70
-
71
- resolved[:associations].each do |assoc, cols|
72
- reflection = model.reflect_on_association(assoc.to_sym)
73
- next unless reflection
74
-
75
- Array(cols).map(&:to_s).each do |col|
76
- scope << { table: reflection.klass.table_name, column: col } if reflection.klass.columns_hash.key?(col)
77
- end
78
- end
79
-
80
- scope
81
- end
82
-
83
- def constantize(klass_name)
84
- klass_name.constantize
85
- rescue NameError
86
- raise Thor::Error, "Could not find model #{klass_name.inspect}"
87
- end
88
42
  end
89
43
  end
90
44
  end
@@ -2,7 +2,8 @@
2
2
 
3
3
  module Huginn
4
4
  class Configuration
5
- attr_accessor :pagy_items, :pagy_max_items, :unaccent_function
5
+ attr_accessor :pagy_items, :pagy_max_items, :unaccent_function,
6
+ :fts_dictionary, :fts_function
6
7
  attr_writer :search_strategy, :auto_include_datatable, :auto_include_searchable
7
8
 
8
9
  def initialize
@@ -13,19 +14,37 @@ module Huginn
13
14
  # this at an IMMUTABLE wrapper (see `rails g huginn:trigram_indexes`),
14
15
  # e.g. "public.f_unaccent".
15
16
  @unaccent_function = "unaccent"
17
+ # The tsvector wrapper and dictionary used by the :full_text strategy. The
18
+ # wrapper (see `rails g huginn:fts_indexes`) pins the dictionary and is
19
+ # IMMUTABLE so GIN tsvector indexes are usable.
20
+ @fts_dictionary = "portuguese"
21
+ @fts_function = "public.f_tsvector"
16
22
  @search_strategy = :pg_trgm
17
23
  @auto_include_datatable = true
18
24
  @auto_include_searchable = true
19
25
  end
20
26
 
21
- # :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE. The similarity
22
- # cutoff is the PostgreSQL GUC pg_trgm.similarity_threshold
23
- # (tune with SELECT set_limit(...)); a gin_trgm_ops GIN index
24
- # on UNACCENT(col) is used when present.
25
- # :unaccent -> unaccent + ILIKE only
26
- # :simple -> plain LIKE
27
+ # :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE. The
28
+ # similarity cutoff is the PostgreSQL GUC
29
+ # pg_trgm.similarity_threshold (tune with
30
+ # SELECT set_limit(...)); a gin_trgm_ops GIN index on
31
+ # UNACCENT(col) is used when present.
32
+ # :full_text -> PostgreSQL full text search (lexemes): the column is
33
+ # matched as to_tsvector(DICTIONARY, col) vs
34
+ # plainto_tsquery(DICTIONARY, term). Needs the
35
+ # fts_function wrapper and a GIN tsvector index.
36
+ # :unaccent -> unaccent + ILIKE only
37
+ # :simple -> plain LIKE
38
+ #
39
+ # Accepts a single symbol or an Array of symbols; when more than one is
40
+ # given the generated predicates are combined with OR (a row matches if
41
+ # any strategy matches).
27
42
  def search_strategy
28
- @search_strategy.to_sym
43
+ if @search_strategy.is_a?(Array)
44
+ @search_strategy.map(&:to_sym)
45
+ else
46
+ @search_strategy.to_sym
47
+ end
29
48
  end
30
49
 
31
50
  def auto_include_datatable?
@@ -77,6 +77,12 @@ module Huginn
77
77
  nil
78
78
  end
79
79
 
80
+ # The Arel predicates folded in by the association scopes of every step.
81
+ # Public so subquery builders can replicate ActiveRecord's scoped joins.
82
+ def scope_constraints
83
+ scope_predicates
84
+ end
85
+
80
86
  # Reverse engineers a join spec usable by `ActiveRecord::Relation#joins`
81
87
  # for filter subqueries: `.joins(:people: { company: ... })`.
82
88
  def self.join_spec_for(names)
@@ -7,6 +7,7 @@ module Huginn
7
7
  included do
8
8
  class_attribute :huginn_attribute_mapping, default: nil
9
9
  class_attribute :huginn_strict_mapping, default: false
10
+ class_attribute :huginn_array_modes, default: {}
10
11
  end
11
12
 
12
13
  class_methods do
@@ -60,15 +61,49 @@ module Huginn
60
61
  # are rejected — API callers can only filter/order by the aliases.
61
62
  # Without a mapping the model falls back to plain/reflected columns.
62
63
  # Accepts a positional Hash or keyword arguments.
64
+ #
65
+ # An alias over a PostgreSQL array column may set how several filter
66
+ # values match (see `huginn_array_match`):
67
+ #
68
+ # huginn_attributes tags: { column: "people.tags", array: :all }
63
69
  def huginn_attributes(mapping = nil, strict: true, **aliases)
64
70
  merged = (mapping || {}).merge(aliases)
65
- self.huginn_attribute_mapping = merged.map { |key, value| [key.to_s, value.to_s] }.to_h
71
+ modes = {}
72
+ self.huginn_attribute_mapping = merged.to_h do |key, value|
73
+ if value.is_a?(Hash)
74
+ options = value.transform_keys(&:to_sym)
75
+ modes[key.to_s] = normalized_array_mode(options[:array]) if options.key?(:array)
76
+ [key.to_s, options.fetch(:column).to_s]
77
+ else
78
+ [key.to_s, value.to_s]
79
+ end
80
+ end
81
+ self.huginn_array_modes = huginn_array_modes.merge(modes)
66
82
  self.huginn_strict_mapping = strict
67
83
  huginn_attribute_mapping
68
84
  end
69
85
 
86
+ # How several values filter a PostgreSQL array column:
87
+ #
88
+ # :any (default) row contains at least one of the values (`&&`),
89
+ # the same semantics as `IN` on a plain column
90
+ # :all row contains every value (`@>`)
91
+ #
92
+ # huginn_array_match :tag_ids, :all
93
+ def huginn_array_match(*fields, mode)
94
+ mode = normalized_array_mode(mode)
95
+ self.huginn_array_modes = huginn_array_modes.merge(fields.flatten.to_h { |field| [field.to_s, mode] })
96
+ end
97
+
70
98
  private
71
99
 
100
+ def normalized_array_mode(mode)
101
+ mode = mode.to_s.to_sym
102
+ raise ArgumentError, "array mode must be :any or :all, got #{mode.inspect}" unless %i[any all].include?(mode)
103
+
104
+ mode
105
+ end
106
+
72
107
  def apply_datatable_search(relation, value)
73
108
  return relation if value.blank? || !respond_to?(:search)
74
109
 
@@ -102,6 +137,7 @@ module Huginn
102
137
  def apply_plain_filter(relation, validator, values)
103
138
  attr = validator.arel_attribute
104
139
  return relation if attr.nil?
140
+ return relation.where(array_predicate(attr, validator, values)) if validator.array?
105
141
 
106
142
  if values.include?("null")
107
143
  apply_nullable_predicate(relation, attr, values)
@@ -119,7 +155,9 @@ module Huginn
119
155
  return relation unless subquery
120
156
 
121
157
  attr = validator.arel_attribute
122
- target = if values.include?("null")
158
+ target = if validator.array?
159
+ subquery.where(array_predicate(attr, validator, values))
160
+ elsif values.include?("null")
123
161
  apply_nullable_predicate(subquery, attr, values)
124
162
  elsif values.size == 1
125
163
  subquery.where(attr.eq(values.first))
@@ -207,6 +245,35 @@ module Huginn
207
245
  nil
208
246
  end
209
247
 
248
+ # PostgreSQL array columns: one value means "contains it" (`@>`); several
249
+ # values follow the field's array mode — :any (`&&`, like IN) or :all
250
+ # (`@>`). "null" matches an empty (or NULL) array. Values are cast by the
251
+ # element type, so invalid ones are dropped before reaching SQL, and the
252
+ # literal is built from quoted values only.
253
+ def array_predicate(attr, validator, values)
254
+ wants_empty = values.include?("null")
255
+ cast = values.reject { |value| value == "null" }.filter_map { |value| validator.cast_array_element(value) }.uniq
256
+ predicates = []
257
+ predicates << empty_array_predicate(attr) if wants_empty
258
+ unless cast.empty?
259
+ operator = cast.size == 1 || validator.array_mode == :all ? "@>" : "&&"
260
+ predicates << Arel::Nodes::InfixOperation.new(operator, attr, array_literal(cast, validator.array_element_sql_type))
261
+ end
262
+ return Arel::Nodes::SqlLiteral.new("1=0") if predicates.empty?
263
+
264
+ predicates.reduce { |left, right| Arel::Nodes::Grouping.new(left.or(right)) }
265
+ end
266
+
267
+ def array_literal(values, sql_type)
268
+ quoted = values.map { |value| connection.quote(value) }.join(", ")
269
+ Arel::Nodes::SqlLiteral.new("ARRAY[#{quoted}]::#{sql_type}[]")
270
+ end
271
+
272
+ def empty_array_predicate(attr)
273
+ cardinality = Arel::Nodes::NamedFunction.new("cardinality", [attr])
274
+ Arel::Nodes::NamedFunction.new("COALESCE", [cardinality, Arel::Nodes.build_quoted(0)]).eq(0)
275
+ end
276
+
210
277
  def apply_nullable_predicate(relation, attr, values)
211
278
  non_null = values.reject { |value| value == "null" }
212
279
  predicate = if non_null.empty?
@@ -6,8 +6,9 @@ module Huginn
6
6
  # (ActionController::Parameters, Hash, Array of hashes, [key, value]
7
7
  # pairs, ...) into a single Hash of { column => Array(values) }.
8
8
  #
9
- # Values are accumulated in arrays so that repeated keys produce an
10
- # IN condition when applied as a datatable filter. Invalid payload
9
+ # Values are accumulated in arrays so that repeated keys — or a key with
10
+ # an Array of values — produce an IN condition when applied as a
11
+ # datatable filter. Invalid payload
11
12
  # shapes are simply ignored, and keys are normalized to strings.
12
13
  class FilterNormalizer
13
14
  def self.call(raw)
@@ -20,7 +21,9 @@ module Huginn
20
21
 
21
22
  def call
22
23
  each_pair(@raw).each_with_object({}) do |(key, value), acc|
23
- (acc[key.to_s] ||= []) << value
24
+ # { "col" => [a, b] } (e.g. filters[col][]=a&filters[col][]=b) is the
25
+ # same as repeating the key: several values, IN / array match.
26
+ (acc[key.to_s] ||= []).concat(value.is_a?(Array) ? value : [value])
24
27
  end
25
28
  end
26
29
 
@@ -24,6 +24,7 @@ module Huginn
24
24
 
25
25
  def initialize(model, field)
26
26
  @model = model
27
+ @public_name = field.to_s
27
28
  @strict_denied = false
28
29
  @field = resolve_alias(field)
29
30
  @field = @field.split(".").drop(1).join(".") if own_table_prefix?(@field)
@@ -74,6 +75,35 @@ module Huginn
74
75
  klass.columns_hash[column].type
75
76
  end
76
77
 
78
+ # True for PostgreSQL array columns (e.g. `uuid[]`, `text[]`).
79
+ def array?
80
+ definition = column_definition
81
+ !!(definition.respond_to?(:array?) ? definition.array? : definition&.try(:array))
82
+ end
83
+
84
+ # SQL type of the array elements ("uuid", "character varying"...),
85
+ # used to cast the filter literal.
86
+ def array_element_sql_type
87
+ column_definition.sql_type.delete_suffix("[]")
88
+ end
89
+
90
+ # Casts a filter value with the array element type; nil when invalid.
91
+ def cast_array_element(value)
92
+ type = target_klass.type_for_attribute(column)
93
+ element = type.respond_to?(:subtype) ? type.subtype : type
94
+ element.serialize(element.cast(value))
95
+ rescue ArgumentError, TypeError
96
+ nil
97
+ end
98
+
99
+ # :any or :all, declared with huginn_attributes(array:) or
100
+ # huginn_array_match (by public alias or by column name).
101
+ def array_mode
102
+ modes = target_klass.respond_to?(:huginn_array_modes) ? target_klass.huginn_array_modes : {}
103
+ own = model.respond_to?(:huginn_array_modes) ? model.huginn_array_modes : {}
104
+ own[@public_name] || own[@field] || modes[column] || :any
105
+ end
106
+
77
107
  # The AssociationPath for scoped fields (builds filter/order
78
108
  # subqueries), or nil for plain columns.
79
109
  def association_path
@@ -111,6 +141,16 @@ module Huginn
111
141
  segments.size >= 2 && segments.first == model.table_name
112
142
  end
113
143
 
144
+ def target_klass
145
+ plain? ? model : association_path.target_klass
146
+ end
147
+
148
+ def column_definition
149
+ return nil unless valid?
150
+
151
+ target_klass.columns_hash[column]
152
+ end
153
+
114
154
  def strict_denied?
115
155
  @strict_denied
116
156
  end
@@ -9,6 +9,11 @@ module Huginn
9
9
  # :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE (best for typos).
10
10
  # Both branches are supported by a gin_trgm_ops GIN index
11
11
  # on UNACCENT(col); the index is used only when it exists.
12
+ # :full_text -> PostgreSQL full text search: to_tsvector(col) @@
13
+ # plainto_tsquery(term), so morphological variants match;
14
+ # needs the IMMUTABLE fts_function wrapper and a GIN
15
+ # tsvector index. Effective with a single strategy or
16
+ # combined with :pg_trgm (OR).
12
17
  # :unaccent -> unaccent + ILIKE (case/accents insensitive)
13
18
  # :simple -> plain LIKE
14
19
  module Fuzzy
@@ -90,6 +95,45 @@ module Huginn
90
95
  Unaccent.new(@column, @value).predicate
91
96
  end
92
97
  end
98
+
99
+ # PostgreSQL full text search over lexemes:
100
+ #
101
+ # public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, term)
102
+ #
103
+ # The column side runs through the configured fts_function (an IMMUTABLE
104
+ # wrapper around to_tsvector that pins the dictionary, created by
105
+ # `rails g huginn:fts_indexes`) so a GIN tsvector index is usable. The
106
+ # query side uses plainto_tsquery, which ANDs the lexemes of the term and
107
+ # applies stemming — tolerant to morphological variants, blind to typos.
108
+ class FullText
109
+ def initialize(column, value, dictionary: nil)
110
+ @column = column
111
+ @value = value.to_s
112
+ @dictionary = dictionary || Huginn.configuration.fts_dictionary
113
+ end
114
+
115
+ def predicate
116
+ Arel::Nodes::InfixOperation.new("@@", tsvector, query)
117
+ end
118
+
119
+ private
120
+
121
+ def tsvector
122
+ Arel::Nodes::NamedFunction.new(Huginn.configuration.fts_function, [@column])
123
+ end
124
+
125
+ def query
126
+ Arel::Nodes::NamedFunction.new("plainto_tsquery", [regconfig, quoted_value])
127
+ end
128
+
129
+ def regconfig
130
+ Arel.sql("'#{@dictionary}'::regconfig")
131
+ end
132
+
133
+ def quoted_value
134
+ Arel::Nodes.build_quoted(@value, @column)
135
+ end
136
+ end
93
137
  end
94
138
  end
95
- end
139
+ end
@@ -6,14 +6,22 @@ module Huginn
6
6
  module Searchable
7
7
  AssociationPath = Datatable::AssociationPath unless const_defined?(:AssociationPath)
8
8
  # Builds a tolerant full-text search relation over a model's searchable
9
- # columns.
9
+ # columns. The matching predicate per column is driven by
10
+ # Huginn.configuration.search_strategy (:pg_trgm, :full_text, :unaccent,
11
+ # :simple — or an Array of them combined with OR).
10
12
  #
11
13
  # Columns are split in two groups following the datatable pattern:
12
14
  #
13
15
  # * the model's own columns -> direct fuzzy predicates on the base WHERE
14
- # * association columns -> semi-join subqueries on the primary key
15
- # pk IN (SELECT DISTINCT pk FROM base
16
- # JOIN <chain> ... WHERE <fuzzy>)
16
+ # * association columns -> reverse semi-join subqueries anchored on
17
+ # the first FK/has_many hop, so the base
18
+ # table is never re-scanned:
19
+ #
20
+ # belongs_to -> base[fk] IN (SELECT DISTINCT t1[pk] FROM t1 ...)
21
+ # has_many -> base[pk] IN (SELECT DISTINCT t1[fk] FROM t1 ...)
22
+ #
23
+ # Through/hand-joined chains keep the generic fallback:
24
+ # base[pk] IN (SELECT DISTINCT base[pk] FROM base JOIN <chain> ...)
17
25
  #
18
26
  # The base relation never carries joins, so every match group stays one
19
27
  # query and the count remains a plain COUNT(*). Multiple association
@@ -36,9 +44,9 @@ module Huginn
36
44
 
37
45
  resolution = resolve_columns
38
46
  predicates = fuzzy_predicates(resolution.fetch(:own))
39
- resolution.fetch(:associations).each do |chain, attrs|
40
- subquery = association_subquery(chain, attrs)
41
- predicates << arel_table[primary_key].in(subquery.arel) if subquery
47
+ resolution.fetch(:associations).each do |_chain, group|
48
+ predicate = association_predicate(group)
49
+ predicates << predicate if predicate
42
50
  end
43
51
 
44
52
  return apply_distinct(relation) if predicates.empty?
@@ -49,13 +57,13 @@ module Huginn
49
57
  private
50
58
 
51
59
  def fuzzy_predicates(attrs)
52
- attrs.map { |attr| fuzzy_predicate(attr) }
60
+ attrs.map { |attr| fuzzy_predicate(attr) }.compact
53
61
  end
54
62
 
55
63
  def resolve_columns
56
64
  columns = @model.searchable_columns_config_resolved
57
65
  own = []
58
- associations = Hash.new { |hash, key| hash[key] = [] }
66
+ associations = Hash.new { |hash, key| hash[key] = { path: nil, attrs: [] } }
59
67
 
60
68
  Array(columns[:columns]).map(&:to_s).reject(&:blank?).each do |field|
61
69
  if field.include?(".")
@@ -78,35 +86,109 @@ module Huginn
78
86
  return unless path.valid?
79
87
  return unless path.target_klass.columns_hash.key?(path.column)
80
88
 
81
- groups[path.association_names] << path.target_table[path.column]
89
+ group = groups[path.association_names]
90
+ group[:path] ||= path
91
+ group[:attrs] << path.target_table[path.column]
92
+ end
93
+
94
+ # Prefers the FK-anchored reverse semi-join; falls back to the generic
95
+ # pk subquery for through chains or anything the reverse plan rejects.
96
+ def association_predicate(group)
97
+ path = group[:path]
98
+ attrs = group[:attrs]
99
+ return nil if path.nil? || attrs.empty?
100
+
101
+ anchored_predicate(path, attrs)
102
+ rescue StandardError
103
+ generic_predicate(path, attrs)
104
+ end
105
+
106
+ # Reverse semi-join:
107
+ #
108
+ # belongs_to -> base[fk] IN (SELECT DISTINCT t1[join_pk] FROM t1 ...)
109
+ # has_many -> base[pk] IN (SELECT DISTINCT t1[fk] FROM t1 ...)
110
+ #
111
+ # Walks the chain from the first target table outwards so the base table
112
+ # is never scanned inside the subquery.
113
+ def anchored_predicate(path, attrs)
114
+ if through_association?(path)
115
+ return generic_predicate(path, attrs)
116
+ end
117
+
118
+ steps = path.resolved_steps
119
+ first = steps.first
120
+ reflection = first.reflection
121
+
122
+ outer = if reflection.belongs_to?
123
+ arel_table[reflection.foreign_key]
124
+ else
125
+ arel_table[reflection.active_record_primary_key]
126
+ end
127
+
128
+ projection = if reflection.belongs_to?
129
+ first.table[reflection.join_primary_key]
130
+ else
131
+ first.table[reflection.foreign_key]
132
+ end
133
+
134
+ manager = Arel::SelectManager.new
135
+ manager.from(first.table)
136
+ steps.drop(1).each { |step| manager.join(step.table).on(step.link_to_previous) }
137
+ manager.where(fuzzy_predicates(attrs).reduce(&:or))
138
+ path.scope_constraints.each { |constraint| manager.where(constraint) }
139
+ manager.project(projection)
140
+ manager.distinct
141
+
142
+ outer.in(manager)
143
+ end
144
+
145
+ # Whether the first hop goes through a join model; the FK anchoring is
146
+ # ambiguous there so the generic pk subquery is used instead.
147
+ def through_association?(path)
148
+ reflection = @model.reflect_on_association(path.segments.first.to_sym)
149
+ reflection.is_a?(ActiveRecord::Reflection::ThroughReflection)
82
150
  end
83
151
 
84
- # Semi-join subquery over the base table: DISTINCT pk while joining the
85
- # association chain of the group and applying the fuzzy predicates on the
86
- # target columns. Mirrors datatable's association_ids_subquery.
87
- def association_subquery(chain, attrs)
88
- return nil if chain.empty?
152
+ # Generic fallback semi-join subquery over the base table, mirroring
153
+ # datatable's association_ids_subquery:
154
+ # base[pk] IN (SELECT DISTINCT base[pk] FROM base JOIN <chain> ...).
155
+ def generic_predicate(path, attrs)
156
+ return nil unless path
89
157
 
90
- spec = AssociationPath.join_spec_for(chain.map(&:to_sym))
158
+ spec = AssociationPath.join_spec_for(path.association_names.map(&:to_sym))
91
159
  base = @model.all.except(:select, :order, :offset, :limit)
92
160
  .select(arel_table[primary_key])
93
161
  .distinct
94
162
 
95
163
  predicate = fuzzy_predicates(attrs).reduce(&:or)
96
- base.joins(spec).where(predicate)
164
+ arel_table[primary_key].in(base.joins(spec).where(predicate).arel)
97
165
  end
98
166
 
99
167
  def fuzzy_predicate(attr)
100
- case strategy
168
+ strategies = Array(strategy)
169
+ predicates = strategies.map { |strat| predicate_for_strategy(attr, strat) }.compact
170
+ return nil if predicates.empty?
171
+
172
+ predicates.reduce(&:or)
173
+ end
174
+
175
+ def predicate_for_strategy(attr, strat)
176
+ case strat
101
177
  when :pg_trgm
102
178
  if pg_trgm_available?
103
179
  Fuzzy::Trigram.new(attr, @value).predicate
104
180
  else
105
181
  fallback_fuzzy(attr)
106
182
  end
183
+ when :full_text
184
+ if fts_available?
185
+ Fuzzy::FullText.new(attr, @value).predicate
186
+ else
187
+ fallback_fuzzy(attr)
188
+ end
107
189
  when :unaccent
108
190
  Fuzzy::Unaccent.new(attr, @value).predicate
109
- else
191
+ when :simple
110
192
  Fuzzy::Simple.new(attr, @value).predicate
111
193
  end
112
194
  end
@@ -135,6 +217,27 @@ module Huginn
135
217
  @unaccent_available ||= extension_installed?("unaccent")
136
218
  end
137
219
 
220
+ # Whether the IMMUTABLE tsvector wrapper behind the :full_text strategy
221
+ # exists (created by `rails g huginn:fts_indexes`). FTS is core
222
+ # PostgreSQL, but without the wrapper the predicate would raise, so we
223
+ # degrade to the unaccent fallback instead.
224
+ def fts_available?
225
+ return false unless postgresql?
226
+ @fts_available ||= begin
227
+ schema, name = split_function_name(Huginn.configuration.fts_function)
228
+ @model.connection.select_value(
229
+ ActiveRecord::Base.sanitize_sql_array(
230
+ ["SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname = ? AND p.proname = ?", schema, name]
231
+ )
232
+ ).present?
233
+ end
234
+ end
235
+
236
+ def split_function_name(function)
237
+ schema, _, name = function.to_s.rpartition(".")
238
+ [schema.presence || "public", name]
239
+ end
240
+
138
241
  def extension_installed?(name)
139
242
  @model.connection
140
243
  .select_all(sanitize_extension_query(name))
@@ -36,18 +36,23 @@ module Huginn
36
36
  end
37
37
 
38
38
  def searchable_columns_config_resolved
39
- config = searchable_columns_config || {}
40
- {
41
- columns: config[:columns].presence || default_searchable_columns,
42
- associations: config[:associations] || {}
43
- }
39
+ config = searchable_columns_config
40
+ if config.nil?
41
+ { columns: default_searchable_columns, associations: {} }
42
+ else
43
+ {
44
+ columns: Array(config[:columns]),
45
+ associations: config[:associations] || {}
46
+ }
47
+ end
44
48
  end
45
49
 
46
50
  private
47
51
 
48
52
  def default_searchable_columns
49
53
  columns_hash.each_with_object([]) do |(name, column), acc|
50
- acc << name if %i[string text].include?(column.type)
54
+ # Array columns (string[]/text[]) are not searchable text: unaccent/ILIKE need a scalar.
55
+ acc << name if %i[string text].include?(column.type) && !column.try(:array)
51
56
  end
52
57
  end
53
58
  end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Huginn
4
- VERSION = "0.2.1"
4
+ VERSION = "0.4.0"
5
5
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: huginn_datatable
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.1
4
+ version: 0.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Kayky Marcelo
@@ -139,6 +139,9 @@ files:
139
139
  - LICENSE.txt
140
140
  - README.md
141
141
  - README.pt-BR.md
142
+ - lib/generators/huginn/fts_indexes/fts_indexes_generator.rb
143
+ - lib/generators/huginn/fts_indexes/templates/add_huginn_fts_indexes.rb.tt
144
+ - lib/generators/huginn/searchable_columns.rb
142
145
  - lib/generators/huginn/trigram_indexes/templates/add_huginn_trigram_indexes.rb.tt
143
146
  - lib/generators/huginn/trigram_indexes/trigram_indexes_generator.rb
144
147
  - lib/huginn.rb
@@ -178,7 +181,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
178
181
  - !ruby/object:Gem::Version
179
182
  version: '0'
180
183
  requirements: []
181
- rubygems_version: 4.0.6
184
+ rubygems_version: 4.0.16
182
185
  specification_version: 4
183
186
  summary: Performant and elegant ActiveRecord datatables/data-grids with tolerant search
184
187
  test_files: []