huginn_datatable 0.2.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +20 -0
- data/README.md +97 -9
- data/README.pt-BR.md +97 -9
- data/lib/generators/huginn/fts_indexes/fts_indexes_generator.rb +49 -0
- data/lib/generators/huginn/fts_indexes/templates/add_huginn_fts_indexes.rb.tt +24 -0
- data/lib/generators/huginn/searchable_columns.rb +69 -0
- data/lib/generators/huginn/trigram_indexes/trigram_indexes_generator.rb +2 -48
- data/lib/huginn/configuration.rb +27 -8
- data/lib/huginn/datatable/association_path.rb +6 -0
- data/lib/huginn/datatable/datatable.rb +69 -2
- data/lib/huginn/datatable/filter_normalizer.rb +6 -3
- data/lib/huginn/datatable/validator.rb +40 -0
- data/lib/huginn/searchable/fuzzy.rb +45 -1
- data/lib/huginn/searchable/query.rb +122 -19
- data/lib/huginn/searchable/searchable.rb +11 -6
- data/lib/huginn/version.rb +1 -1
- metadata +5 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 82a13098551b25e1b656beb853926f7dcbc964c420a64dde0e22182f735a02bb
|
|
4
|
+
data.tar.gz: abdcf5e8f489ad2b6bff0870bde97d443247f6d31b0efc93274a47da81837916
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3a03a5dc06c8f00d4a66a9bd9af8d06ddc2a9a910dbbb5c8a392985f4b0a21a89439d40829d20559232e1a04d2978444fdcf1c39ac86ce20c176af3fbb98a1de
|
|
7
|
+
data.tar.gz: b229398a323d43253aba96f667ca27dc5d76eacdf3bec68d84cb51c482e8838914de5289fdc9093a293016743f5e14c69acdc5b60705ba9e57306d639b86ca74
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,26 @@
|
|
|
2
2
|
|
|
3
3
|
### Unreleased
|
|
4
4
|
|
|
5
|
+
#### 0.4.0
|
|
6
|
+
|
|
7
|
+
- **PostgreSQL array columns in `filters`** (`uuid[]`, `text[]`, `string[]`...), for own columns, `huginn_attributes` aliases and association-scoped columns. The payload is unchanged:
|
|
8
|
+
- one value → the row contains it (`col @> ARRAY[v]`);
|
|
9
|
+
- several values → **any** of them (`col && ARRAY[...]`, the same semantics as `IN`), or **all** of them (`col @> ARRAY[...]`) when the field is declared with `array: :all`;
|
|
10
|
+
- `"null"` → empty (or NULL) array, combinable with values.
|
|
11
|
+
- New `huginn_array_match :field, :all` and `huginn_attributes alias: { column: "table.col", array: :all }` to choose the match mode (default `:any`).
|
|
12
|
+
- Array filter values are cast by the element type (invalid values — e.g. a malformed UUID — are dropped, never reaching SQL) and the literal is built from quoted values only.
|
|
13
|
+
- `filters` accept a Hash whose value is an Array (`filters[col][]=a&filters[col][]=b` → `{ "col" => [a, b] }`) as several values (IN / array match), the same as repeating the key. Before, the Array was taken as one value and never matched.
|
|
14
|
+
- **Fix:** default searchable columns no longer include array columns (`string[]`/`text[]`), which broke `search` with `unaccent(character varying[]) does not exist`.
|
|
15
|
+
|
|
16
|
+
#### 0.3.0
|
|
17
|
+
|
|
18
|
+
- **New `:full_text` search strategy** — PostgreSQL full text search over lexemes (`public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, 'term')`), so morphological variants match (e.g. "compras" finds "compra"). Blind to typos, unlike `:pg_trgm`. Uses a GIN tsvector index when the `fts_function` wrapper (IMMUTABLE, dictionary pinned) exists; degrades gracefully to unaccent+ILIKE otherwise.
|
|
19
|
+
- `search_strategy:` now accepts **a single symbol or an Array of them**; multiple strategies are combined with **OR** (a row matches if any strategy matches), e.g. `[ :pg_trgm, :full_text ]` to get typo tolerance plus stemming.
|
|
20
|
+
- New `fts_dictionary:` (default `"portuguese"`) and `fts_function:` (default `"public.f_tsvector"`) configurations, mirroring `unaccent_function`.
|
|
21
|
+
- New generator **`rails g huginn:fts_indexes [Model...]`** scaffolds a migration that creates the IMMUTABLE `public.f_tsvector` wrapper (with `--dictionary`) and GIN tsvector indexes on every `:string`/`:text` column of the Searchable models, including association-scoped ones.
|
|
22
|
+
- `searchable_columns columns: []` is now honored — an empty array means "search only the declared associations", no longer falling back to the model's own string/text columns.
|
|
23
|
+
- Specs for the `Configuration` normalization surface (defaults + `search_strategy` symbol/array shapes).
|
|
24
|
+
|
|
5
25
|
#### 0.2.1
|
|
6
26
|
|
|
7
27
|
- Association-scoped `search` now matches through **primary-key semi-join subqueries** (`pk IN (SELECT DISTINCT pk FROM <base> JOIN <chain> WHERE <fuzzy>)`) instead of `left_joins` — one subquery per distinct association chain, combined with OR. The base relation never joins, so the count is a plain `COUNT(*)`.
|
data/README.md
CHANGED
|
@@ -13,14 +13,14 @@
|
|
|
13
13
|
🇺🇸 English · <a href="./README.pt-BR.md">🇧🇷 Português</a>
|
|
14
14
|
</p>
|
|
15
15
|
|
|
16
|
-
Huginn is a lightweight query layer for Rails that turns a raw datatable request into a **lean count, a paginated subset and one preload** — instead of a massive JOIN materialized in memory. It also ships a PostgreSQL
|
|
16
|
+
Huginn is a lightweight query layer for Rails that turns a raw datatable request into a **lean count, a paginated subset and one preload** — instead of a massive JOIN materialized in memory. It also ships a PostgreSQL tolerant-search builder (`pg_trgm` similarity via the `%` operator with `unaccent` and `ILIKE` fallback, plus a `:full_text` lexeme strategy) that is tolerant to typos and accents — and that can lean on GIN indexes when present.
|
|
17
17
|
|
|
18
18
|
## Highlights
|
|
19
19
|
|
|
20
20
|
- **Two-phase execution** — association filters/orders/range become reflection-secured subqueries, then a lean count and `preload` **only on the paginated subset**.
|
|
21
21
|
- **Lean counts** — the base relation never joins (`COUNT(*)` over a stripped relation); association matches run as pk subqueries instead.
|
|
22
22
|
- **SQL injection safe ordering/filtering** — every column reference is resolved through Arel reflection, never string-interpolated.
|
|
23
|
-
- **
|
|
23
|
+
- **Tolerant search** — `pg_trgm` similarity (`%` operator) OR `unaccent+ILIKE` (typo/acent tolerant) and `:full_text` over lexemes (morphology-aware); strategies are `search_strategy`-selectable or combined with OR, backed by GIN indexes when present.
|
|
24
24
|
- **Rails conventions** — works with `ActionController::Parameters`, Railtie auto-includes both concerns (toggleable), zero boilerplate.
|
|
25
25
|
|
|
26
26
|
## Development
|
|
@@ -58,7 +58,7 @@ The `gemfiles/*.gemfile` are generated by Appraisal (committed); their `.lock` f
|
|
|
58
58
|
## Installation
|
|
59
59
|
|
|
60
60
|
```ruby
|
|
61
|
-
gem "huginn"
|
|
61
|
+
gem "huginn_datatable", require: "huginn"
|
|
62
62
|
```
|
|
63
63
|
|
|
64
64
|
## Configuration
|
|
@@ -66,9 +66,13 @@ gem "huginn"
|
|
|
66
66
|
```ruby
|
|
67
67
|
# config/initializers/huginn.rb
|
|
68
68
|
Huginn.configure do |config|
|
|
69
|
-
# :pg_trgm
|
|
69
|
+
# :pg_trgm (recommended) — trigram similarity (%) OR unaccent+ILIKE
|
|
70
|
+
# :full_text — PostgreSQL full text search over lexemes
|
|
71
|
+
# (to_tsvector @@ plainto_tsquery)
|
|
70
72
|
# :unaccent — unaccent + ILIKE only
|
|
71
73
|
# :simple — plain LIKE
|
|
74
|
+
# Pass an Array to combine strategies with OR, e.g.
|
|
75
|
+
# config.search_strategy = [:pg_trgm, :full_text]
|
|
72
76
|
config.search_strategy = :pg_trgm
|
|
73
77
|
|
|
74
78
|
# Unaccent function used around search terms/columns. Defaults to the pg
|
|
@@ -77,6 +81,13 @@ Huginn.configure do |config|
|
|
|
77
81
|
# indexes can actually be used.
|
|
78
82
|
config.unaccent_function = "unaccent"
|
|
79
83
|
|
|
84
|
+
# Full text search (strategy :full_text). The dictionary and the IMMUTABLE
|
|
85
|
+
# tsvector wrapper (`rails g huginn:fts_indexes`) pinned to that dictionary;
|
|
86
|
+
# they only apply when the wrapper exists, otherwise search degrades to the
|
|
87
|
+
# unaccent fallback.
|
|
88
|
+
# config.fts_dictionary = "portuguese"
|
|
89
|
+
# config.fts_function = "public.f_tsvector"
|
|
90
|
+
|
|
80
91
|
config.pagy_items = 10 # default page size
|
|
81
92
|
config.pagy_max_items = 500 # hard cap for per_page
|
|
82
93
|
end
|
|
@@ -84,6 +95,8 @@ end
|
|
|
84
95
|
|
|
85
96
|
> **Threshold:** under `:pg_trgm` the similarity cutoff is the PostgreSQL GUC `pg_trgm.similarity_threshold` (default `0.3`), **not** a gem-level setting. Tune it with `SET pg_trgm.similarity_threshold = 0.4` / `SELECT set_limit(0.4)` on the database.
|
|
86
97
|
|
|
98
|
+
> **Strategy tradeoffs:** `:pg_trgm` tolerates typos and substrings but has no notion of morphology. `:full_text` matches morphological variants (e.g. "correndo"/"correr" → same stem) but is blind to typos, and shines on longer text columns — for name/keyword lookups it mostly overlaps with trigram. Combining both (`[:pg_trgm, :full_text]`) unions the two result sets and requires both index sets (`rails g huginn:trigram_indexes` + `rails g huginn:fts_indexes`).
|
|
99
|
+
|
|
87
100
|
## Railtie (automatic include)
|
|
88
101
|
|
|
89
102
|
By default the Railtie includes `Huginn::Datatable` and `Huginn::Searchable` into every `ActiveRecord::Base` model. You do **not** need `include` statements unless you opt in selectively:
|
|
@@ -107,7 +120,7 @@ result = Plano.datatable(
|
|
|
107
120
|
includes: [{ operadora: { pessoa: [:endereco, :contatos] } }] # preloaded on the page only
|
|
108
121
|
)
|
|
109
122
|
|
|
110
|
-
result[:total_count] # Integer (lean COUNT
|
|
123
|
+
result[:total_count] # Integer (lean COUNT(*))
|
|
111
124
|
result[:data] # ActiveRecord::Relation (paged + preloaded)
|
|
112
125
|
```
|
|
113
126
|
|
|
@@ -117,10 +130,32 @@ Supported params:
|
|
|
117
130
|
|---|---|
|
|
118
131
|
| `page`, `per_page` | Pagination (clamped to `pagy_max_items`) |
|
|
119
132
|
| `search` | Delegates to `Huginn::Searchable.search` |
|
|
120
|
-
| `filters` | Hash / Array of hashes / pairs -> exact or `IN` conditions (`"col" => "null"` → `IS NULL`) |
|
|
133
|
+
| `filters` | Hash (`{ "col" => v }` or `{ "col" => [a, b] }`) / Array of hashes / pairs -> exact or `IN` conditions (`"col" => "null"` → `IS NULL`) |
|
|
121
134
|
| `range_data` | `{ "created_at" => ["2024-01-01", "2024-12-31"] }` — date or numeric ranges |
|
|
122
135
|
| `orders` | `[{ "pessoa.nome" => "desc" }]` — plain or association-scoped columns |
|
|
123
136
|
|
|
137
|
+
### Array columns (PostgreSQL)
|
|
138
|
+
|
|
139
|
+
`filters` also work on array columns (`uuid[]`, `text[]`, `string[]`...) — own, aliased or association-scoped — with the same payload:
|
|
140
|
+
|
|
141
|
+
```ruby
|
|
142
|
+
Person.datatable({ filters: { tag_ids: id } }) # contains id (tag_ids @> ARRAY[id])
|
|
143
|
+
Person.datatable({ filters: [{ tag_ids: a }, { tag_ids: b }] }) # contains a OR b (tag_ids && ARRAY[a, b])
|
|
144
|
+
Person.datatable({ filters: { tag_ids: "null" } }) # empty array
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Several values match **any** of them by default (like `IN`). To require **all** of them:
|
|
148
|
+
|
|
149
|
+
```ruby
|
|
150
|
+
class Person < ApplicationRecord
|
|
151
|
+
huginn_array_match :tag_ids, :all
|
|
152
|
+
# or, with an alias:
|
|
153
|
+
huginn_attributes tags: { column: "people.tag_ids", array: :all }
|
|
154
|
+
end
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Values are cast by the element type (an invalid UUID is dropped, never sent to the database) and the array literal is built from quoted values only. Array columns are left out of the default `search` columns.
|
|
158
|
+
|
|
124
159
|
### Scoped ordering / filtering
|
|
125
160
|
|
|
126
161
|
Any `column` **or** `association.column` reference is validated and mapped to its real reflected table:
|
|
@@ -189,6 +224,14 @@ Person.search("globex") # matches company.name via a
|
|
|
189
224
|
Person.search("kayky", distinct: false) # disable the implicit DISTINCT
|
|
190
225
|
```
|
|
191
226
|
|
|
227
|
+
`searchable_columns columns: []` is honored: an empty array means "search only the declared associations" — it no longer falls back to the model's own `:string`/`:text` columns:
|
|
228
|
+
|
|
229
|
+
```ruby
|
|
230
|
+
class Person < ApplicationRecord
|
|
231
|
+
searchable_columns columns: [], company: [:name, :cnpj] # only company.name/cnpj are searched
|
|
232
|
+
end
|
|
233
|
+
```
|
|
234
|
+
|
|
192
235
|
`Huginn::Datatable` reuses `Huginn::Searchable.search` automatically when the model responds to `search`.
|
|
193
236
|
|
|
194
237
|
## Trigram indexes (performance)
|
|
@@ -231,6 +274,51 @@ Notes:
|
|
|
231
274
|
- Only helpful for search terms of **3 or more characters** — trigrams need that many to match. Shorter terms always scan.
|
|
232
275
|
- To see it working, run a `Person.search("term").explain` with the wrapper configured and a `gin_trgm_ops` index present.
|
|
233
276
|
|
|
277
|
+
## Full-text indexes (performance)
|
|
278
|
+
|
|
279
|
+
Under `:full_text`, each searchable column produces a lexeme predicate backed by a GIN tsvector index:
|
|
280
|
+
|
|
281
|
+
```sql
|
|
282
|
+
public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, 'term')
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
Because `to_tsvector()` is `STABLE` (not `IMMUTABLE`), it cannot be indexed directly — you need an `IMMUTABLE` wrapper that pins the dictionary plus an expression index on it:
|
|
286
|
+
|
|
287
|
+
```sql
|
|
288
|
+
CREATE OR REPLACE FUNCTION public.f_tsvector(text)
|
|
289
|
+
RETURNS tsvector AS $$
|
|
290
|
+
SELECT to_tsvector('portuguese'::regconfig, $1);
|
|
291
|
+
$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;
|
|
292
|
+
|
|
293
|
+
CREATE INDEX index_people_name_fts ON people USING gin (public.f_tsvector(name));
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
Then point the gem at the wrapper and dictionary:
|
|
297
|
+
|
|
298
|
+
```ruby
|
|
299
|
+
Huginn.configure do |c|
|
|
300
|
+
c.search_strategy = :full_text
|
|
301
|
+
c.fts_dictionary = "portuguese"
|
|
302
|
+
c.fts_function = "public.f_tsvector"
|
|
303
|
+
end
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
Or scaffold the migration for every Searchable model (wrapper + GIN indexes included), picking a dictionary via `--dictionary`:
|
|
307
|
+
|
|
308
|
+
```sh
|
|
309
|
+
rails g huginn:fts_indexes # all Searchable models
|
|
310
|
+
rails g huginn:fts_indexes Person Product # specific models
|
|
311
|
+
rails g huginn:fts_indexes --dictionary english # pin a different dictionary
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
The index is **optional** — without the wrapper/index the search degrades gracefully to the unaccent+ILIKE fallback and stays correct (via sequential scan).
|
|
315
|
+
|
|
316
|
+
Notes:
|
|
317
|
+
|
|
318
|
+
- Requires the full text search support bundled with PostgreSQL (no extra extension).
|
|
319
|
+
- `:full_text` has no typo tolerance — combine it with `:pg_trgm` (`search_strategy: [:pg_trgm, :full_text]`) and generate **both** index sets (`huginn:trigram_indexes` + `huginn:fts_indexes`).
|
|
320
|
+
- To see it working, run a `Person.search("term").explain` with the wrapper configured and a GIN tsvector index present.
|
|
321
|
+
|
|
234
322
|
## Query efficiency
|
|
235
323
|
|
|
236
324
|
```
|
|
@@ -246,7 +334,7 @@ For a `Plano` datatable with deep `includes:`, this is exactly **2 extra queries
|
|
|
246
334
|
|
|
247
335
|
```
|
|
248
336
|
lib/huginn.rb entry, Huginn.configure, Huginn.instrument
|
|
249
|
-
lib/huginn/configuration.rb search_strategy, unaccent_function, pagy_*
|
|
337
|
+
lib/huginn/configuration.rb search_strategy, unaccent_function, fts_*, pagy_*
|
|
250
338
|
lib/huginn/railtie.rb auto-includes concerns into ActiveRecord
|
|
251
339
|
lib/huginn/datatable.rb Huginn::Datatable (aggregator)
|
|
252
340
|
lib/huginn/datatable/datatable.rb the datatable Concern
|
|
@@ -258,8 +346,8 @@ lib/huginn/datatable/paginator.rb lean count, pagination, isolated preload
|
|
|
258
346
|
lib/huginn/searchable.rb Huginn::Searchable (aggregator)
|
|
259
347
|
lib/huginn/searchable/searchable.rb the search Concern + DSL
|
|
260
348
|
lib/huginn/searchable/query.rb tolerant search builder (subqueries + OR)
|
|
261
|
-
lib/huginn/searchable/fuzzy.rb pg_trgm / unaccent / simple predicates
|
|
262
|
-
lib/generators/... `huginn:trigram_indexes` (GIN index migrations)
|
|
349
|
+
lib/huginn/searchable/fuzzy.rb pg_trgm / full_text / unaccent / simple predicates
|
|
350
|
+
lib/generators/... `huginn:trigram_indexes` + `huginn:fts_indexes` (GIN index migrations)
|
|
263
351
|
```
|
|
264
352
|
|
|
265
353
|
## Instrumentation
|
data/README.pt-BR.md
CHANGED
|
@@ -13,14 +13,14 @@
|
|
|
13
13
|
<a href="./README.md">🇺🇸 English</a> · 🇧🇷 Português
|
|
14
14
|
</p>
|
|
15
15
|
|
|
16
|
-
O Huginn é uma camada de consulta leve para Rails que transforma uma requisição de datatable bruta em um **count enxuto, um subconjunto paginado e um único preload** — em vez de um JOIN enorme materializado em memória. Também traz um construtor de busca
|
|
16
|
+
O Huginn é uma camada de consulta leve para Rails que transforma uma requisição de datatable bruta em um **count enxuto, um subconjunto paginado e um único preload** — em vez de um JOIN enorme materializado em memória. Também traz um construtor de busca tolerante a erros para PostgreSQL (similaridade `pg_trgm` via operador `%` com `unaccent` e fallback `ILIKE`, além da estratégia `:full_text` sobre lexemas) que tolera erros de digitação e acentuação — e que pode usar índices GIN quando presentes.
|
|
17
17
|
|
|
18
18
|
## Destaques
|
|
19
19
|
|
|
20
20
|
- **Execução em duas fases** — filtros/orders/ranges de associação se tornam subqueries resolvidas por reflexão, e o `preload` é feito **somente no subconjunto paginado**.
|
|
21
21
|
- **Counts enxutos** — a relation base nunca faz join (`COUNT(*)` sobre uma relation restrita); as buscas de associação acontecem via subqueries no pk.
|
|
22
22
|
- **Ordenação/filtro seguros contra SQL injection** — toda referência de coluna é resolvida via reflexão do Arel, nunca interpolada como string.
|
|
23
|
-
- **Busca tolerante
|
|
23
|
+
- **Busca tolerante** — similaridade `pg_trgm` (operador `%`) OU `unaccent+ILIKE` (tolerante a typos/acentos) e `:full_text` sobre lexemas (ciente de morfologia); as estratégias são selecionáveis via `search_strategy` ou combinadas com OR, respaldadas por índices GIN quando presentes.
|
|
24
24
|
- **Convenções do Rails** — funciona com `ActionController::Parameters`, Railtie inclui ambos os concerns automaticamente (desligável), zero boilerplate.
|
|
25
25
|
|
|
26
26
|
## Desenvolvimento
|
|
@@ -58,7 +58,7 @@ Os `gemfiles/*.gemfile` são gerados pelo Appraisal (commitados); os arquivos `.
|
|
|
58
58
|
## Instalação
|
|
59
59
|
|
|
60
60
|
```ruby
|
|
61
|
-
gem "huginn"
|
|
61
|
+
gem "huginn_datatable", require: "huginn"
|
|
62
62
|
```
|
|
63
63
|
|
|
64
64
|
## Configuração
|
|
@@ -66,9 +66,13 @@ gem "huginn"
|
|
|
66
66
|
```ruby
|
|
67
67
|
# config/initializers/huginn.rb
|
|
68
68
|
Huginn.configure do |config|
|
|
69
|
-
# :pg_trgm
|
|
69
|
+
# :pg_trgm (recomendado) — similaridade trigram (%) OU unaccent+ILIKE
|
|
70
|
+
# :full_text — full text search do PostgreSQL sobre lexemas
|
|
71
|
+
# (to_tsvector @@ plainto_tsquery)
|
|
70
72
|
# :unaccent — somente unaccent + ILIKE
|
|
71
73
|
# :simple — LIKE simples
|
|
74
|
+
# Para combinar estratégias com OR, passe um Array:
|
|
75
|
+
# config.search_strategy = [:pg_trgm, :full_text]
|
|
72
76
|
config.search_strategy = :pg_trgm
|
|
73
77
|
|
|
74
78
|
# Função unaccent usada em torno de termos/colunas. O default é o UNACCENT()
|
|
@@ -77,6 +81,13 @@ Huginn.configure do |config|
|
|
|
77
81
|
# GIN trigram possam ser usados de fato.
|
|
78
82
|
config.unaccent_function = "unaccent"
|
|
79
83
|
|
|
84
|
+
# Full text search (estratégia :full_text). O dicionário e o wrapper
|
|
85
|
+
# tsvector IMMUTABLE (`rails g huginn:fts_indexes`) fixado nesse dicionário;
|
|
86
|
+
# só têm efeito quando o wrapper existe — caso contrário a busca degrada
|
|
87
|
+
# para o fallback unaccent.
|
|
88
|
+
# config.fts_dictionary = "portuguese"
|
|
89
|
+
# config.fts_function = "public.f_tsvector"
|
|
90
|
+
|
|
80
91
|
config.pagy_items = 10 # tamanho de página padrão
|
|
81
92
|
config.pagy_max_items = 500 # teto máximo de per_page
|
|
82
93
|
end
|
|
@@ -84,6 +95,8 @@ end
|
|
|
84
95
|
|
|
85
96
|
> **Limiar:** sob `:pg_trgm`, o corte de similaridade é o GUC do PostgreSQL `pg_trgm.similarity_threshold` (default `0.3`), **não** uma configuração da gem. Ajuste com `SET pg_trgm.similarity_threshold = 0.4` / `SELECT set_limit(0.4)` no banco.
|
|
86
97
|
|
|
98
|
+
> **Tradeoffs de estratégia:** `:pg_trgm` tolera typos e substrings, mas não tem noção de morfologia. `:full_text` captura variações morfológicas (ex. "correndo"/"correr" → mesmo lexema), mas é cego a typos, e brilha em colunas de texto mais longo — para busca por nome/palavra-chave ele em grande parte se sobrepõe ao trigram. Combinar os dois (`[:pg_trgm, :full_text]`) une os dois conjuntos de resultado e exige os dois conjuntos de índices (`rails g huginn:trigram_indexes` + `rails g huginn:fts_indexes`).
|
|
99
|
+
|
|
87
100
|
## Railtie (include automático)
|
|
88
101
|
|
|
89
102
|
Por padrão, o Railtie inclui `Huginn::Datatable` e `Huginn::Searchable` em toda `ActiveRecord::Base`. Você **não** precisa de `include`, a menos que opte seletivamente:
|
|
@@ -107,7 +120,7 @@ result = Plano.datatable(
|
|
|
107
120
|
includes: [{ operadora: { pessoa: [:endereco, :contatos] } }] # preload somente na página
|
|
108
121
|
)
|
|
109
122
|
|
|
110
|
-
result[:total_count] # Integer (count enxuto COUNT
|
|
123
|
+
result[:total_count] # Integer (count enxuto COUNT(*))
|
|
111
124
|
result[:data] # ActiveRecord::Relation (paginada + preloaded)
|
|
112
125
|
```
|
|
113
126
|
|
|
@@ -117,10 +130,32 @@ Parâmetros suportados:
|
|
|
117
130
|
|---|---|
|
|
118
131
|
| `page`, `per_page` | Paginação (limitada a `pagy_max_items`) |
|
|
119
132
|
| `search` | Delega para `Huginn::Searchable.search` |
|
|
120
|
-
| `filters` | Hash / Array de hashes / pares -> condições exatas ou `IN` (`"col" => "null"` → `IS NULL`) |
|
|
133
|
+
| `filters` | Hash (`{ "col" => v }` ou `{ "col" => [a, b] }`) / Array de hashes / pares -> condições exatas ou `IN` (`"col" => "null"` → `IS NULL`) |
|
|
121
134
|
| `range_data` | `{ "created_at" => ["2024-01-01", "2024-12-31"] }` — ranges de datas ou numéricos |
|
|
122
135
|
| `orders` | `[{ "pessoa.nome" => "desc" }]` — colunas simples ou de associação |
|
|
123
136
|
|
|
137
|
+
### Colunas array (PostgreSQL)
|
|
138
|
+
|
|
139
|
+
Os `filters` também funcionam em colunas array (`uuid[]`, `text[]`, `string[]`...) — próprias, com alias ou de associação — com o mesmo payload:
|
|
140
|
+
|
|
141
|
+
```ruby
|
|
142
|
+
Person.datatable({ filters: { tag_ids: id } }) # contém id (tag_ids @> ARRAY[id])
|
|
143
|
+
Person.datatable({ filters: [{ tag_ids: a }, { tag_ids: b }] }) # contém a OU b (tag_ids && ARRAY[a, b])
|
|
144
|
+
Person.datatable({ filters: { tag_ids: "null" } }) # array vazio
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Vários valores casam com **qualquer** um por padrão (como o `IN`). Para exigir **todos**:
|
|
148
|
+
|
|
149
|
+
```ruby
|
|
150
|
+
class Person < ApplicationRecord
|
|
151
|
+
huginn_array_match :tag_ids, :all
|
|
152
|
+
# ou, com alias:
|
|
153
|
+
huginn_attributes tags: { column: "people.tag_ids", array: :all }
|
|
154
|
+
end
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Os valores são convertidos pelo tipo do elemento (um UUID inválido é descartado, nunca vai ao banco) e o literal do array é montado só com valores escapados. Colunas array ficam fora das colunas padrão do `search`.
|
|
158
|
+
|
|
124
159
|
### Ordenação/filtro com associação
|
|
125
160
|
|
|
126
161
|
Qualquer referência `column` **ou** `associacao.column` é validada e mapeada para a tabela refletida real:
|
|
@@ -189,6 +224,14 @@ Person.search("globex") # encontra company.name via s
|
|
|
189
224
|
Person.search("kayky", distinct: false) # desativa o DISTINCT implícito
|
|
190
225
|
```
|
|
191
226
|
|
|
227
|
+
`searchable_columns columns: []` é respeitado: um array vazio significa "pesquisar somente as associações declaradas" — não cai mais para as colunas `:string`/`:text` do próprio modelo:
|
|
228
|
+
|
|
229
|
+
```ruby
|
|
230
|
+
class Person < ApplicationRecord
|
|
231
|
+
searchable_columns columns: [], company: [:name, :cnpj] # só company.name/cnpj são pesquisados
|
|
232
|
+
end
|
|
233
|
+
```
|
|
234
|
+
|
|
192
235
|
`Huginn::Datatable` reutiliza `Huginn::Searchable.search` automaticamente quando o modelo responde a `search`.
|
|
193
236
|
|
|
194
237
|
## Índices trigram (performance)
|
|
@@ -231,6 +274,51 @@ Observações:
|
|
|
231
274
|
- Só ajuda em termos de busca de **3 caracteres ou mais** — trigramas precisam disso para casar. Termos menores sempre fazem scan.
|
|
232
275
|
- Para conferir o uso, rode `Person.search("termo").explain` com o wrapper configurado e um índice `gin_trgm_ops` presente.
|
|
233
276
|
|
|
277
|
+
## Índices full-text (performance)
|
|
278
|
+
|
|
279
|
+
Sob `:full_text`, cada coluna pesquisável gera um predicado por lexema respaldado por um índice GIN tsvector:
|
|
280
|
+
|
|
281
|
+
```sql
|
|
282
|
+
public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, 'termo')
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
Como o `to_tsvector()` é `STABLE` (não `IMMUTABLE`), ele não pode ser indexado diretamente — você precisa de um wrapper `IMMUTABLE` que fixe o dicionário além do índice de expressão:
|
|
286
|
+
|
|
287
|
+
```sql
|
|
288
|
+
CREATE OR REPLACE FUNCTION public.f_tsvector(text)
|
|
289
|
+
RETURNS tsvector AS $$
|
|
290
|
+
SELECT to_tsvector('portuguese'::regconfig, $1);
|
|
291
|
+
$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;
|
|
292
|
+
|
|
293
|
+
CREATE INDEX index_people_name_fts ON people USING gin (public.f_tsvector(name));
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
E aponte a gem para o wrapper e o dicionário:
|
|
297
|
+
|
|
298
|
+
```ruby
|
|
299
|
+
Huginn.configure do |c|
|
|
300
|
+
c.search_strategy = :full_text
|
|
301
|
+
c.fts_dictionary = "portuguese"
|
|
302
|
+
c.fts_function = "public.f_tsvector"
|
|
303
|
+
end
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
Ou gere a migration para todos os modelos Searchable (wrapper + índices GIN inclusos), escolhendo um dicionário com `--dictionary`:
|
|
307
|
+
|
|
308
|
+
```sh
|
|
309
|
+
rails g huginn:fts_indexes # todos os modelos Searchable
|
|
310
|
+
rails g huginn:fts_indexes Person Product # modelos específicos
|
|
311
|
+
rails g huginn:fts_indexes --dictionary english # fixar outro dicionário
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
O índice é **opcional** — sem o wrapper/índice a busca degrada graciosamente para o fallback unaccent+ILIKE e permanece correta (via sequential scan).
|
|
315
|
+
|
|
316
|
+
Observações:
|
|
317
|
+
|
|
318
|
+
- Requer o suporte a full text search embutido no PostgreSQL (sem extensão extra).
|
|
319
|
+
- `:full_text` não tem tolerância a typos — combine com `:pg_trgm` (`search_strategy: [:pg_trgm, :full_text]`) e gere **os dois** conjuntos de índices (`huginn:trigram_indexes` + `huginn:fts_indexes`).
|
|
320
|
+
- Para conferir o uso, rode `Person.search("termo").explain` com o wrapper configurado e um índice GIN tsvector presente.
|
|
321
|
+
|
|
234
322
|
## Eficiência da query
|
|
235
323
|
|
|
236
324
|
```
|
|
@@ -246,7 +334,7 @@ Para um datatable de `Plano` com `includes:` profundos, isso são exatamente **2
|
|
|
246
334
|
|
|
247
335
|
```
|
|
248
336
|
lib/huginn.rb entry, Huginn.configure, Huginn.instrument
|
|
249
|
-
lib/huginn/configuration.rb search_strategy, unaccent_function, pagy_*
|
|
337
|
+
lib/huginn/configuration.rb search_strategy, unaccent_function, fts_*, pagy_*
|
|
250
338
|
lib/huginn/railtie.rb auto-inclui os concerns no ActiveRecord
|
|
251
339
|
lib/huginn/datatable.rb Huginn::Datatable (agregador)
|
|
252
340
|
lib/huginn/datatable/datatable.rb o Concern do datatable
|
|
@@ -258,8 +346,8 @@ lib/huginn/datatable/paginator.rb count enxuto, paginação, preload isolado
|
|
|
258
346
|
lib/huginn/searchable.rb Huginn::Searchable (agregador)
|
|
259
347
|
lib/huginn/searchable/searchable.rb o Concern do search + DSL
|
|
260
348
|
lib/huginn/searchable/query.rb construtor de busca tolerante (subqueries + OR)
|
|
261
|
-
lib/huginn/searchable/fuzzy.rb predicados pg_trgm / unaccent / simple
|
|
262
|
-
lib/generators/...
|
|
349
|
+
lib/huginn/searchable/fuzzy.rb predicados pg_trgm / full_text / unaccent / simple
|
|
350
|
+
lib/generators/... geradores `huginn:trigram_indexes` + `huginn:fts_indexes` (migrations de índices GIN)
|
|
263
351
|
```
|
|
264
352
|
|
|
265
353
|
## Instrumentação
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rails/generators"
|
|
4
|
+
require "rails/generators/migration"
|
|
5
|
+
require "generators/huginn/searchable_columns"
|
|
6
|
+
|
|
7
|
+
module Huginn
|
|
8
|
+
module Generators
|
|
9
|
+
# Scaffolds a migration that creates the PostgreSQL GIN tsvector indexes
|
|
10
|
+
# (on an IMMUTABLE wrapper around to_tsvector) backing Huginn::Searchable's
|
|
11
|
+
# :full_text strategy for every model that includes Huginn::Searchable.
|
|
12
|
+
#
|
|
13
|
+
# rails g huginn:fts_indexes # all Searchable models
|
|
14
|
+
# rails g huginn:fts_indexes Person Product # specific models
|
|
15
|
+
# rails g huginn:fts_indexes --dictionary english # pick the dictionary
|
|
16
|
+
#
|
|
17
|
+
# The migration creates an IMMUTABLE public.f_tsvector(text) wrapper that
|
|
18
|
+
# pins the configured dictionary and the GIN indexes. Point the gem at the
|
|
19
|
+
# wrapper (to_tsvector itself is STABLE and cannot be indexed directly):
|
|
20
|
+
#
|
|
21
|
+
# Huginn.configure { |c| c.search_strategy = :full_text;
|
|
22
|
+
# c.fts_dictionary = "portuguese";
|
|
23
|
+
# c.fts_function = "public.f_tsvector" }
|
|
24
|
+
#
|
|
25
|
+
# Apply the same searchable model/association resolution as the trigram
|
|
26
|
+
# generator, so both index sets cover identical columns.
|
|
27
|
+
class FtsIndexesGenerator < Rails::Generators::Base
|
|
28
|
+
include SearchableColumns
|
|
29
|
+
|
|
30
|
+
source_root File.expand_path("templates", __dir__)
|
|
31
|
+
|
|
32
|
+
argument :model_names, type: :array, default: [], desc: "Models to index (default: all Searchable models)"
|
|
33
|
+
|
|
34
|
+
class_option :dictionary, type: :string, default: "portuguese",
|
|
35
|
+
desc: "tsvector dictionary pinned by the IMMUTABLE wrapper"
|
|
36
|
+
|
|
37
|
+
def copy_migration
|
|
38
|
+
@entries = resolve_entries
|
|
39
|
+
@dictionary = options[:dictionary]
|
|
40
|
+
return if @entries.empty?
|
|
41
|
+
|
|
42
|
+
migration_template(
|
|
43
|
+
"add_huginn_fts_indexes.rb.tt",
|
|
44
|
+
"db/migrate/add_huginn_fts_indexes.rb"
|
|
45
|
+
)
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
end
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
class AddHuginnFtsIndexes < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
|
|
2
|
+
def up
|
|
3
|
+
# to_tsvector(regconfig, text) is STABLE (not IMMUTABLE), so it cannot be
|
|
4
|
+
# used in an index expression as-is. Wrap it in an IMMUTABLE function that
|
|
5
|
+
# always pins the configured dictionary.
|
|
6
|
+
execute <<~SQL
|
|
7
|
+
CREATE OR REPLACE FUNCTION public.f_tsvector(text)
|
|
8
|
+
RETURNS tsvector AS $$
|
|
9
|
+
SELECT to_tsvector('<%= @dictionary %>', $1);
|
|
10
|
+
$$
|
|
11
|
+
LANGUAGE sql IMMUTABLE PARALLEL SAFE;
|
|
12
|
+
SQL
|
|
13
|
+
<% @entries.each do |entry| %>
|
|
14
|
+
add_index :<%= entry[:table] %>, "public.f_tsvector(<%= entry[:column] %>)", using: :gin, name: "index_<%= entry[:table] %>_<%= entry[:column] %>_fts"
|
|
15
|
+
<% end %>
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def down
|
|
19
|
+
<% @entries.each do |entry| %>
|
|
20
|
+
remove_index :<%= entry[:table] %>, name: "index_<%= entry[:table] %>_<%= entry[:column] %>_fts"
|
|
21
|
+
<% end %>
|
|
22
|
+
execute "DROP FUNCTION IF EXISTS public.f_tsvector(text)"
|
|
23
|
+
end
|
|
24
|
+
end
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rails/generators"
|
|
4
|
+
require "rails/generators/migration"
|
|
5
|
+
|
|
6
|
+
module Huginn
|
|
7
|
+
module Generators
|
|
8
|
+
# Shared logic between the index-scaffold generators
|
|
9
|
+
# (huginn:trigram_indexes / huginn:fts_indexes): resolving the Searchable
|
|
10
|
+
# models (or the explicit ones given as arguments) into the table/column
|
|
11
|
+
# pairs that need an index, and the migration filename machinery.
|
|
12
|
+
module SearchableColumns
|
|
13
|
+
extend ActiveSupport::Concern
|
|
14
|
+
|
|
15
|
+
included do
|
|
16
|
+
include Rails::Generators::Migration
|
|
17
|
+
|
|
18
|
+
# Rails::Generators::Migration leaves this as NotImplementedError; it
|
|
19
|
+
# is expected to be overridden to produce the next migration number.
|
|
20
|
+
def self.next_migration_number(dirname)
|
|
21
|
+
number = current_migration_number(dirname) + 1
|
|
22
|
+
ActiveRecord::Migration.next_migration_number(number)
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
private
|
|
27
|
+
|
|
28
|
+
def resolve_entries
|
|
29
|
+
models = if model_names.any?
|
|
30
|
+
model_names.map { |name| constantize(name) }
|
|
31
|
+
else
|
|
32
|
+
ActiveRecord::Base.descendants
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
models.filter_map do |model|
|
|
36
|
+
next unless model.respond_to?(:searchable_columns_config_resolved)
|
|
37
|
+
|
|
38
|
+
columns_for(model)
|
|
39
|
+
end.reject(&:empty?).flatten.uniq
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def columns_for(model)
|
|
43
|
+
scope = []
|
|
44
|
+
resolved = model.searchable_columns_config_resolved
|
|
45
|
+
|
|
46
|
+
Array(resolved[:columns]).map(&:to_s).reject(&:blank?).each do |field|
|
|
47
|
+
scope << { table: model.table_name, column: field } if model.columns_hash.key?(field)
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
resolved[:associations].each do |assoc, cols|
|
|
51
|
+
reflection = model.reflect_on_association(assoc.to_sym)
|
|
52
|
+
next unless reflection
|
|
53
|
+
|
|
54
|
+
Array(cols).map(&:to_s).each do |col|
|
|
55
|
+
scope << { table: reflection.klass.table_name, column: col } if reflection.klass.columns_hash.key?(col)
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
scope
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
def constantize(klass_name)
|
|
63
|
+
klass_name.constantize
|
|
64
|
+
rescue NameError
|
|
65
|
+
raise Thor::Error, "Could not find model #{klass_name.inspect}"
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
end
|
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
require "rails/generators"
|
|
4
4
|
require "rails/generators/migration"
|
|
5
|
+
require "generators/huginn/searchable_columns"
|
|
5
6
|
|
|
6
7
|
module Huginn
|
|
7
8
|
module Generators
|
|
@@ -23,17 +24,12 @@ module Huginn
|
|
|
23
24
|
# Indexes are only used when the :pg_trgm strategy is configured and the
|
|
24
25
|
# pg_trgm/unaccent extensions are installed.
|
|
25
26
|
class TrigramIndexesGenerator < Rails::Generators::Base
|
|
26
|
-
include
|
|
27
|
+
include SearchableColumns
|
|
27
28
|
|
|
28
29
|
source_root File.expand_path("templates", __dir__)
|
|
29
30
|
|
|
30
31
|
argument :model_names, type: :array, default: [], desc: "Models to index (default: all Searchable models)"
|
|
31
32
|
|
|
32
|
-
def self.next_migration_number(dirname)
|
|
33
|
-
next_migration_number = current_migration_number(dirname) + 1
|
|
34
|
-
ActiveRecord::Migration.next_migration_number(next_migration_number)
|
|
35
|
-
end
|
|
36
|
-
|
|
37
33
|
def copy_migration
|
|
38
34
|
@entries = resolve_entries
|
|
39
35
|
return if @entries.empty?
|
|
@@ -43,48 +39,6 @@ module Huginn
|
|
|
43
39
|
"db/migrate/add_huginn_trigram_indexes.rb"
|
|
44
40
|
)
|
|
45
41
|
end
|
|
46
|
-
|
|
47
|
-
private
|
|
48
|
-
|
|
49
|
-
def resolve_entries
|
|
50
|
-
models = if model_names.any?
|
|
51
|
-
model_names.map { |name| constantize(name) }
|
|
52
|
-
else
|
|
53
|
-
ActiveRecord::Base.descendants
|
|
54
|
-
end
|
|
55
|
-
|
|
56
|
-
models.filter_map do |model|
|
|
57
|
-
next unless model.respond_to?(:searchable_columns_config_resolved)
|
|
58
|
-
|
|
59
|
-
columns_for(model)
|
|
60
|
-
end.reject(&:empty?).flatten.uniq
|
|
61
|
-
end
|
|
62
|
-
|
|
63
|
-
def columns_for(model)
|
|
64
|
-
scope = []
|
|
65
|
-
resolved = model.searchable_columns_config_resolved
|
|
66
|
-
|
|
67
|
-
Array(resolved[:columns]).map(&:to_s).reject(&:blank?).each do |field|
|
|
68
|
-
scope << { table: model.table_name, column: field } if model.columns_hash.key?(field)
|
|
69
|
-
end
|
|
70
|
-
|
|
71
|
-
resolved[:associations].each do |assoc, cols|
|
|
72
|
-
reflection = model.reflect_on_association(assoc.to_sym)
|
|
73
|
-
next unless reflection
|
|
74
|
-
|
|
75
|
-
Array(cols).map(&:to_s).each do |col|
|
|
76
|
-
scope << { table: reflection.klass.table_name, column: col } if reflection.klass.columns_hash.key?(col)
|
|
77
|
-
end
|
|
78
|
-
end
|
|
79
|
-
|
|
80
|
-
scope
|
|
81
|
-
end
|
|
82
|
-
|
|
83
|
-
def constantize(klass_name)
|
|
84
|
-
klass_name.constantize
|
|
85
|
-
rescue NameError
|
|
86
|
-
raise Thor::Error, "Could not find model #{klass_name.inspect}"
|
|
87
|
-
end
|
|
88
42
|
end
|
|
89
43
|
end
|
|
90
44
|
end
|
data/lib/huginn/configuration.rb
CHANGED
|
@@ -2,7 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
module Huginn
|
|
4
4
|
class Configuration
|
|
5
|
-
attr_accessor :pagy_items, :pagy_max_items, :unaccent_function
|
|
5
|
+
attr_accessor :pagy_items, :pagy_max_items, :unaccent_function,
|
|
6
|
+
:fts_dictionary, :fts_function
|
|
6
7
|
attr_writer :search_strategy, :auto_include_datatable, :auto_include_searchable
|
|
7
8
|
|
|
8
9
|
def initialize
|
|
@@ -13,19 +14,37 @@ module Huginn
|
|
|
13
14
|
# this at an IMMUTABLE wrapper (see `rails g huginn:trigram_indexes`),
|
|
14
15
|
# e.g. "public.f_unaccent".
|
|
15
16
|
@unaccent_function = "unaccent"
|
|
17
|
+
# The tsvector wrapper and dictionary used by the :full_text strategy. The
|
|
18
|
+
# wrapper (see `rails g huginn:fts_indexes`) pins the dictionary and is
|
|
19
|
+
# IMMUTABLE so GIN tsvector indexes are usable.
|
|
20
|
+
@fts_dictionary = "portuguese"
|
|
21
|
+
@fts_function = "public.f_tsvector"
|
|
16
22
|
@search_strategy = :pg_trgm
|
|
17
23
|
@auto_include_datatable = true
|
|
18
24
|
@auto_include_searchable = true
|
|
19
25
|
end
|
|
20
26
|
|
|
21
|
-
# :pg_trgm
|
|
22
|
-
#
|
|
23
|
-
#
|
|
24
|
-
#
|
|
25
|
-
#
|
|
26
|
-
# :
|
|
27
|
+
# :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE. The
|
|
28
|
+
# similarity cutoff is the PostgreSQL GUC
|
|
29
|
+
# pg_trgm.similarity_threshold (tune with
|
|
30
|
+
# SELECT set_limit(...)); a gin_trgm_ops GIN index on
|
|
31
|
+
# UNACCENT(col) is used when present.
|
|
32
|
+
# :full_text -> PostgreSQL full text search (lexemes): the column is
|
|
33
|
+
# matched as to_tsvector(DICTIONARY, col) vs
|
|
34
|
+
# plainto_tsquery(DICTIONARY, term). Needs the
|
|
35
|
+
# fts_function wrapper and a GIN tsvector index.
|
|
36
|
+
# :unaccent -> unaccent + ILIKE only
|
|
37
|
+
# :simple -> plain LIKE
|
|
38
|
+
#
|
|
39
|
+
# Accepts a single symbol or an Array of symbols; when more than one is
|
|
40
|
+
# given the generated predicates are combined with OR (a row matches if
|
|
41
|
+
# any strategy matches).
|
|
27
42
|
def search_strategy
|
|
28
|
-
@search_strategy.
|
|
43
|
+
if @search_strategy.is_a?(Array)
|
|
44
|
+
@search_strategy.map(&:to_sym)
|
|
45
|
+
else
|
|
46
|
+
@search_strategy.to_sym
|
|
47
|
+
end
|
|
29
48
|
end
|
|
30
49
|
|
|
31
50
|
def auto_include_datatable?
|
|
@@ -77,6 +77,12 @@ module Huginn
|
|
|
77
77
|
nil
|
|
78
78
|
end
|
|
79
79
|
|
|
80
|
+
# The Arel predicates folded in by the association scopes of every step.
|
|
81
|
+
# Public so subquery builders can replicate ActiveRecord's scoped joins.
|
|
82
|
+
def scope_constraints
|
|
83
|
+
scope_predicates
|
|
84
|
+
end
|
|
85
|
+
|
|
80
86
|
# Reverse engineers a join spec usable by `ActiveRecord::Relation#joins`
|
|
81
87
|
# for filter subqueries: `.joins(:people: { company: ... })`.
|
|
82
88
|
def self.join_spec_for(names)
|
|
@@ -7,6 +7,7 @@ module Huginn
|
|
|
7
7
|
included do
|
|
8
8
|
class_attribute :huginn_attribute_mapping, default: nil
|
|
9
9
|
class_attribute :huginn_strict_mapping, default: false
|
|
10
|
+
class_attribute :huginn_array_modes, default: {}
|
|
10
11
|
end
|
|
11
12
|
|
|
12
13
|
class_methods do
|
|
@@ -60,15 +61,49 @@ module Huginn
|
|
|
60
61
|
# are rejected — API callers can only filter/order by the aliases.
|
|
61
62
|
# Without a mapping the model falls back to plain/reflected columns.
|
|
62
63
|
# Accepts a positional Hash or keyword arguments.
|
|
64
|
+
#
|
|
65
|
+
# An alias over a PostgreSQL array column may set how several filter
|
|
66
|
+
# values match (see `huginn_array_match`):
|
|
67
|
+
#
|
|
68
|
+
# huginn_attributes tags: { column: "people.tags", array: :all }
|
|
63
69
|
def huginn_attributes(mapping = nil, strict: true, **aliases)
|
|
64
70
|
merged = (mapping || {}).merge(aliases)
|
|
65
|
-
|
|
71
|
+
modes = {}
|
|
72
|
+
self.huginn_attribute_mapping = merged.to_h do |key, value|
|
|
73
|
+
if value.is_a?(Hash)
|
|
74
|
+
options = value.transform_keys(&:to_sym)
|
|
75
|
+
modes[key.to_s] = normalized_array_mode(options[:array]) if options.key?(:array)
|
|
76
|
+
[key.to_s, options.fetch(:column).to_s]
|
|
77
|
+
else
|
|
78
|
+
[key.to_s, value.to_s]
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
self.huginn_array_modes = huginn_array_modes.merge(modes)
|
|
66
82
|
self.huginn_strict_mapping = strict
|
|
67
83
|
huginn_attribute_mapping
|
|
68
84
|
end
|
|
69
85
|
|
|
86
|
+
# How several values filter a PostgreSQL array column:
|
|
87
|
+
#
|
|
88
|
+
# :any (default) row contains at least one of the values (`&&`),
|
|
89
|
+
# the same semantics as `IN` on a plain column
|
|
90
|
+
# :all row contains every value (`@>`)
|
|
91
|
+
#
|
|
92
|
+
# huginn_array_match :tag_ids, :all
|
|
93
|
+
def huginn_array_match(*fields, mode)
|
|
94
|
+
mode = normalized_array_mode(mode)
|
|
95
|
+
self.huginn_array_modes = huginn_array_modes.merge(fields.flatten.to_h { |field| [field.to_s, mode] })
|
|
96
|
+
end
|
|
97
|
+
|
|
70
98
|
private
|
|
71
99
|
|
|
100
|
+
def normalized_array_mode(mode)
|
|
101
|
+
mode = mode.to_s.to_sym
|
|
102
|
+
raise ArgumentError, "array mode must be :any or :all, got #{mode.inspect}" unless %i[any all].include?(mode)
|
|
103
|
+
|
|
104
|
+
mode
|
|
105
|
+
end
|
|
106
|
+
|
|
72
107
|
def apply_datatable_search(relation, value)
|
|
73
108
|
return relation if value.blank? || !respond_to?(:search)
|
|
74
109
|
|
|
@@ -102,6 +137,7 @@ module Huginn
|
|
|
102
137
|
def apply_plain_filter(relation, validator, values)
|
|
103
138
|
attr = validator.arel_attribute
|
|
104
139
|
return relation if attr.nil?
|
|
140
|
+
return relation.where(array_predicate(attr, validator, values)) if validator.array?
|
|
105
141
|
|
|
106
142
|
if values.include?("null")
|
|
107
143
|
apply_nullable_predicate(relation, attr, values)
|
|
@@ -119,7 +155,9 @@ module Huginn
|
|
|
119
155
|
return relation unless subquery
|
|
120
156
|
|
|
121
157
|
attr = validator.arel_attribute
|
|
122
|
-
target = if
|
|
158
|
+
target = if validator.array?
|
|
159
|
+
subquery.where(array_predicate(attr, validator, values))
|
|
160
|
+
elsif values.include?("null")
|
|
123
161
|
apply_nullable_predicate(subquery, attr, values)
|
|
124
162
|
elsif values.size == 1
|
|
125
163
|
subquery.where(attr.eq(values.first))
|
|
@@ -207,6 +245,35 @@ module Huginn
|
|
|
207
245
|
nil
|
|
208
246
|
end
|
|
209
247
|
|
|
248
|
+
# PostgreSQL array columns: one value means "contains it" (`@>`); several
|
|
249
|
+
# values follow the field's array mode — :any (`&&`, like IN) or :all
|
|
250
|
+
# (`@>`). "null" matches an empty (or NULL) array. Values are cast by the
|
|
251
|
+
# element type, so invalid ones are dropped before reaching SQL, and the
|
|
252
|
+
# literal is built from quoted values only.
|
|
253
|
+
def array_predicate(attr, validator, values)
|
|
254
|
+
wants_empty = values.include?("null")
|
|
255
|
+
cast = values.reject { |value| value == "null" }.filter_map { |value| validator.cast_array_element(value) }.uniq
|
|
256
|
+
predicates = []
|
|
257
|
+
predicates << empty_array_predicate(attr) if wants_empty
|
|
258
|
+
unless cast.empty?
|
|
259
|
+
operator = cast.size == 1 || validator.array_mode == :all ? "@>" : "&&"
|
|
260
|
+
predicates << Arel::Nodes::InfixOperation.new(operator, attr, array_literal(cast, validator.array_element_sql_type))
|
|
261
|
+
end
|
|
262
|
+
return Arel::Nodes::SqlLiteral.new("1=0") if predicates.empty?
|
|
263
|
+
|
|
264
|
+
predicates.reduce { |left, right| Arel::Nodes::Grouping.new(left.or(right)) }
|
|
265
|
+
end
|
|
266
|
+
|
|
267
|
+
def array_literal(values, sql_type)
|
|
268
|
+
quoted = values.map { |value| connection.quote(value) }.join(", ")
|
|
269
|
+
Arel::Nodes::SqlLiteral.new("ARRAY[#{quoted}]::#{sql_type}[]")
|
|
270
|
+
end
|
|
271
|
+
|
|
272
|
+
def empty_array_predicate(attr)
|
|
273
|
+
cardinality = Arel::Nodes::NamedFunction.new("cardinality", [attr])
|
|
274
|
+
Arel::Nodes::NamedFunction.new("COALESCE", [cardinality, Arel::Nodes.build_quoted(0)]).eq(0)
|
|
275
|
+
end
|
|
276
|
+
|
|
210
277
|
def apply_nullable_predicate(relation, attr, values)
|
|
211
278
|
non_null = values.reject { |value| value == "null" }
|
|
212
279
|
predicate = if non_null.empty?
|
|
@@ -6,8 +6,9 @@ module Huginn
|
|
|
6
6
|
# (ActionController::Parameters, Hash, Array of hashes, [key, value]
|
|
7
7
|
# pairs, ...) into a single Hash of { column => Array(values) }.
|
|
8
8
|
#
|
|
9
|
-
# Values are accumulated in arrays so that repeated keys
|
|
10
|
-
# IN condition when applied as a
|
|
9
|
+
# Values are accumulated in arrays so that repeated keys — or a key with
|
|
10
|
+
# an Array of values — produce an IN condition when applied as a
|
|
11
|
+
# datatable filter. Invalid payload
|
|
11
12
|
# shapes are simply ignored, and keys are normalized to strings.
|
|
12
13
|
class FilterNormalizer
|
|
13
14
|
def self.call(raw)
|
|
@@ -20,7 +21,9 @@ module Huginn
|
|
|
20
21
|
|
|
21
22
|
def call
|
|
22
23
|
each_pair(@raw).each_with_object({}) do |(key, value), acc|
|
|
23
|
-
|
|
24
|
+
# { "col" => [a, b] } (e.g. filters[col][]=a&filters[col][]=b) is the
|
|
25
|
+
# same as repeating the key: several values, IN / array match.
|
|
26
|
+
(acc[key.to_s] ||= []).concat(value.is_a?(Array) ? value : [value])
|
|
24
27
|
end
|
|
25
28
|
end
|
|
26
29
|
|
|
@@ -24,6 +24,7 @@ module Huginn
|
|
|
24
24
|
|
|
25
25
|
def initialize(model, field)
|
|
26
26
|
@model = model
|
|
27
|
+
@public_name = field.to_s
|
|
27
28
|
@strict_denied = false
|
|
28
29
|
@field = resolve_alias(field)
|
|
29
30
|
@field = @field.split(".").drop(1).join(".") if own_table_prefix?(@field)
|
|
@@ -74,6 +75,35 @@ module Huginn
|
|
|
74
75
|
klass.columns_hash[column].type
|
|
75
76
|
end
|
|
76
77
|
|
|
78
|
+
# True for PostgreSQL array columns (e.g. `uuid[]`, `text[]`).
|
|
79
|
+
def array?
|
|
80
|
+
definition = column_definition
|
|
81
|
+
!!(definition.respond_to?(:array?) ? definition.array? : definition&.try(:array))
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
# SQL type of the array elements ("uuid", "character varying"...),
|
|
85
|
+
# used to cast the filter literal.
|
|
86
|
+
def array_element_sql_type
|
|
87
|
+
column_definition.sql_type.delete_suffix("[]")
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
# Casts a filter value with the array element type; nil when invalid.
|
|
91
|
+
def cast_array_element(value)
|
|
92
|
+
type = target_klass.type_for_attribute(column)
|
|
93
|
+
element = type.respond_to?(:subtype) ? type.subtype : type
|
|
94
|
+
element.serialize(element.cast(value))
|
|
95
|
+
rescue ArgumentError, TypeError
|
|
96
|
+
nil
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
# :any or :all, declared with huginn_attributes(array:) or
|
|
100
|
+
# huginn_array_match (by public alias or by column name).
|
|
101
|
+
def array_mode
|
|
102
|
+
modes = target_klass.respond_to?(:huginn_array_modes) ? target_klass.huginn_array_modes : {}
|
|
103
|
+
own = model.respond_to?(:huginn_array_modes) ? model.huginn_array_modes : {}
|
|
104
|
+
own[@public_name] || own[@field] || modes[column] || :any
|
|
105
|
+
end
|
|
106
|
+
|
|
77
107
|
# The AssociationPath for scoped fields (builds filter/order
|
|
78
108
|
# subqueries), or nil for plain columns.
|
|
79
109
|
def association_path
|
|
@@ -111,6 +141,16 @@ module Huginn
|
|
|
111
141
|
segments.size >= 2 && segments.first == model.table_name
|
|
112
142
|
end
|
|
113
143
|
|
|
144
|
+
def target_klass
|
|
145
|
+
plain? ? model : association_path.target_klass
|
|
146
|
+
end
|
|
147
|
+
|
|
148
|
+
def column_definition
|
|
149
|
+
return nil unless valid?
|
|
150
|
+
|
|
151
|
+
target_klass.columns_hash[column]
|
|
152
|
+
end
|
|
153
|
+
|
|
114
154
|
def strict_denied?
|
|
115
155
|
@strict_denied
|
|
116
156
|
end
|
|
@@ -9,6 +9,11 @@ module Huginn
|
|
|
9
9
|
# :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE (best for typos).
|
|
10
10
|
# Both branches are supported by a gin_trgm_ops GIN index
|
|
11
11
|
# on UNACCENT(col); the index is used only when it exists.
|
|
12
|
+
# :full_text -> PostgreSQL full text search: to_tsvector(col) @@
|
|
13
|
+
# plainto_tsquery(term), so morphological variants match;
|
|
14
|
+
# needs the IMMUTABLE fts_function wrapper and a GIN
|
|
15
|
+
# tsvector index. Effective with a single strategy or
|
|
16
|
+
# combined with :pg_trgm (OR).
|
|
12
17
|
# :unaccent -> unaccent + ILIKE (case/accents insensitive)
|
|
13
18
|
# :simple -> plain LIKE
|
|
14
19
|
module Fuzzy
|
|
@@ -90,6 +95,45 @@ module Huginn
|
|
|
90
95
|
Unaccent.new(@column, @value).predicate
|
|
91
96
|
end
|
|
92
97
|
end
|
|
98
|
+
|
|
99
|
+
# PostgreSQL full text search over lexemes:
|
|
100
|
+
#
|
|
101
|
+
# public.f_tsvector(col) @@ plainto_tsquery('portuguese'::regconfig, term)
|
|
102
|
+
#
|
|
103
|
+
# The column side runs through the configured fts_function (an IMMUTABLE
|
|
104
|
+
# wrapper around to_tsvector that pins the dictionary, created by
|
|
105
|
+
# `rails g huginn:fts_indexes`) so a GIN tsvector index is usable. The
|
|
106
|
+
# query side uses plainto_tsquery, which ANDs the lexemes of the term and
|
|
107
|
+
# applies stemming — tolerant to morphological variants, blind to typos.
|
|
108
|
+
class FullText
|
|
109
|
+
def initialize(column, value, dictionary: nil)
|
|
110
|
+
@column = column
|
|
111
|
+
@value = value.to_s
|
|
112
|
+
@dictionary = dictionary || Huginn.configuration.fts_dictionary
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
def predicate
|
|
116
|
+
Arel::Nodes::InfixOperation.new("@@", tsvector, query)
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
private
|
|
120
|
+
|
|
121
|
+
def tsvector
|
|
122
|
+
Arel::Nodes::NamedFunction.new(Huginn.configuration.fts_function, [@column])
|
|
123
|
+
end
|
|
124
|
+
|
|
125
|
+
def query
|
|
126
|
+
Arel::Nodes::NamedFunction.new("plainto_tsquery", [regconfig, quoted_value])
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
def regconfig
|
|
130
|
+
Arel.sql("'#{@dictionary}'::regconfig")
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
def quoted_value
|
|
134
|
+
Arel::Nodes.build_quoted(@value, @column)
|
|
135
|
+
end
|
|
136
|
+
end
|
|
93
137
|
end
|
|
94
138
|
end
|
|
95
|
-
end
|
|
139
|
+
end
|
|
@@ -6,14 +6,22 @@ module Huginn
|
|
|
6
6
|
module Searchable
|
|
7
7
|
AssociationPath = Datatable::AssociationPath unless const_defined?(:AssociationPath)
|
|
8
8
|
# Builds a tolerant full-text search relation over a model's searchable
|
|
9
|
-
# columns.
|
|
9
|
+
# columns. The matching predicate per column is driven by
|
|
10
|
+
# Huginn.configuration.search_strategy (:pg_trgm, :full_text, :unaccent,
|
|
11
|
+
# :simple — or an Array of them combined with OR).
|
|
10
12
|
#
|
|
11
13
|
# Columns are split in two groups following the datatable pattern:
|
|
12
14
|
#
|
|
13
15
|
# * the model's own columns -> direct fuzzy predicates on the base WHERE
|
|
14
|
-
# * association columns -> semi-join subqueries on
|
|
15
|
-
#
|
|
16
|
-
#
|
|
16
|
+
# * association columns -> reverse semi-join subqueries anchored on
|
|
17
|
+
# the first FK/has_many hop, so the base
|
|
18
|
+
# table is never re-scanned:
|
|
19
|
+
#
|
|
20
|
+
# belongs_to -> base[fk] IN (SELECT DISTINCT t1[pk] FROM t1 ...)
|
|
21
|
+
# has_many -> base[pk] IN (SELECT DISTINCT t1[fk] FROM t1 ...)
|
|
22
|
+
#
|
|
23
|
+
# Through/hand-joined chains keep the generic fallback:
|
|
24
|
+
# base[pk] IN (SELECT DISTINCT base[pk] FROM base JOIN <chain> ...)
|
|
17
25
|
#
|
|
18
26
|
# The base relation never carries joins, so every match group stays one
|
|
19
27
|
# query and the count remains a plain COUNT(*). Multiple association
|
|
@@ -36,9 +44,9 @@ module Huginn
|
|
|
36
44
|
|
|
37
45
|
resolution = resolve_columns
|
|
38
46
|
predicates = fuzzy_predicates(resolution.fetch(:own))
|
|
39
|
-
resolution.fetch(:associations).each do |
|
|
40
|
-
|
|
41
|
-
predicates <<
|
|
47
|
+
resolution.fetch(:associations).each do |_chain, group|
|
|
48
|
+
predicate = association_predicate(group)
|
|
49
|
+
predicates << predicate if predicate
|
|
42
50
|
end
|
|
43
51
|
|
|
44
52
|
return apply_distinct(relation) if predicates.empty?
|
|
@@ -49,13 +57,13 @@ module Huginn
|
|
|
49
57
|
private
|
|
50
58
|
|
|
51
59
|
def fuzzy_predicates(attrs)
|
|
52
|
-
attrs.map { |attr| fuzzy_predicate(attr) }
|
|
60
|
+
attrs.map { |attr| fuzzy_predicate(attr) }.compact
|
|
53
61
|
end
|
|
54
62
|
|
|
55
63
|
def resolve_columns
|
|
56
64
|
columns = @model.searchable_columns_config_resolved
|
|
57
65
|
own = []
|
|
58
|
-
associations = Hash.new { |hash, key| hash[key] = [] }
|
|
66
|
+
associations = Hash.new { |hash, key| hash[key] = { path: nil, attrs: [] } }
|
|
59
67
|
|
|
60
68
|
Array(columns[:columns]).map(&:to_s).reject(&:blank?).each do |field|
|
|
61
69
|
if field.include?(".")
|
|
@@ -78,35 +86,109 @@ module Huginn
|
|
|
78
86
|
return unless path.valid?
|
|
79
87
|
return unless path.target_klass.columns_hash.key?(path.column)
|
|
80
88
|
|
|
81
|
-
groups[path.association_names]
|
|
89
|
+
group = groups[path.association_names]
|
|
90
|
+
group[:path] ||= path
|
|
91
|
+
group[:attrs] << path.target_table[path.column]
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
# Prefers the FK-anchored reverse semi-join; falls back to the generic
|
|
95
|
+
# pk subquery for through chains or anything the reverse plan rejects.
|
|
96
|
+
def association_predicate(group)
|
|
97
|
+
path = group[:path]
|
|
98
|
+
attrs = group[:attrs]
|
|
99
|
+
return nil if path.nil? || attrs.empty?
|
|
100
|
+
|
|
101
|
+
anchored_predicate(path, attrs)
|
|
102
|
+
rescue StandardError
|
|
103
|
+
generic_predicate(path, attrs)
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
# Reverse semi-join:
|
|
107
|
+
#
|
|
108
|
+
# belongs_to -> base[fk] IN (SELECT DISTINCT t1[join_pk] FROM t1 ...)
|
|
109
|
+
# has_many -> base[pk] IN (SELECT DISTINCT t1[fk] FROM t1 ...)
|
|
110
|
+
#
|
|
111
|
+
# Walks the chain from the first target table outwards so the base table
|
|
112
|
+
# is never scanned inside the subquery.
|
|
113
|
+
def anchored_predicate(path, attrs)
|
|
114
|
+
if through_association?(path)
|
|
115
|
+
return generic_predicate(path, attrs)
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
steps = path.resolved_steps
|
|
119
|
+
first = steps.first
|
|
120
|
+
reflection = first.reflection
|
|
121
|
+
|
|
122
|
+
outer = if reflection.belongs_to?
|
|
123
|
+
arel_table[reflection.foreign_key]
|
|
124
|
+
else
|
|
125
|
+
arel_table[reflection.active_record_primary_key]
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
projection = if reflection.belongs_to?
|
|
129
|
+
first.table[reflection.join_primary_key]
|
|
130
|
+
else
|
|
131
|
+
first.table[reflection.foreign_key]
|
|
132
|
+
end
|
|
133
|
+
|
|
134
|
+
manager = Arel::SelectManager.new
|
|
135
|
+
manager.from(first.table)
|
|
136
|
+
steps.drop(1).each { |step| manager.join(step.table).on(step.link_to_previous) }
|
|
137
|
+
manager.where(fuzzy_predicates(attrs).reduce(&:or))
|
|
138
|
+
path.scope_constraints.each { |constraint| manager.where(constraint) }
|
|
139
|
+
manager.project(projection)
|
|
140
|
+
manager.distinct
|
|
141
|
+
|
|
142
|
+
outer.in(manager)
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
# Whether the first hop goes through a join model; the FK anchoring is
|
|
146
|
+
# ambiguous there so the generic pk subquery is used instead.
|
|
147
|
+
def through_association?(path)
|
|
148
|
+
reflection = @model.reflect_on_association(path.segments.first.to_sym)
|
|
149
|
+
reflection.is_a?(ActiveRecord::Reflection::ThroughReflection)
|
|
82
150
|
end
|
|
83
151
|
|
|
84
|
-
#
|
|
85
|
-
#
|
|
86
|
-
#
|
|
87
|
-
def
|
|
88
|
-
return nil
|
|
152
|
+
# Generic fallback semi-join subquery over the base table, mirroring
|
|
153
|
+
# datatable's association_ids_subquery:
|
|
154
|
+
# base[pk] IN (SELECT DISTINCT base[pk] FROM base JOIN <chain> ...).
|
|
155
|
+
def generic_predicate(path, attrs)
|
|
156
|
+
return nil unless path
|
|
89
157
|
|
|
90
|
-
spec = AssociationPath.join_spec_for(
|
|
158
|
+
spec = AssociationPath.join_spec_for(path.association_names.map(&:to_sym))
|
|
91
159
|
base = @model.all.except(:select, :order, :offset, :limit)
|
|
92
160
|
.select(arel_table[primary_key])
|
|
93
161
|
.distinct
|
|
94
162
|
|
|
95
163
|
predicate = fuzzy_predicates(attrs).reduce(&:or)
|
|
96
|
-
base.joins(spec).where(predicate)
|
|
164
|
+
arel_table[primary_key].in(base.joins(spec).where(predicate).arel)
|
|
97
165
|
end
|
|
98
166
|
|
|
99
167
|
def fuzzy_predicate(attr)
|
|
100
|
-
|
|
168
|
+
strategies = Array(strategy)
|
|
169
|
+
predicates = strategies.map { |strat| predicate_for_strategy(attr, strat) }.compact
|
|
170
|
+
return nil if predicates.empty?
|
|
171
|
+
|
|
172
|
+
predicates.reduce(&:or)
|
|
173
|
+
end
|
|
174
|
+
|
|
175
|
+
def predicate_for_strategy(attr, strat)
|
|
176
|
+
case strat
|
|
101
177
|
when :pg_trgm
|
|
102
178
|
if pg_trgm_available?
|
|
103
179
|
Fuzzy::Trigram.new(attr, @value).predicate
|
|
104
180
|
else
|
|
105
181
|
fallback_fuzzy(attr)
|
|
106
182
|
end
|
|
183
|
+
when :full_text
|
|
184
|
+
if fts_available?
|
|
185
|
+
Fuzzy::FullText.new(attr, @value).predicate
|
|
186
|
+
else
|
|
187
|
+
fallback_fuzzy(attr)
|
|
188
|
+
end
|
|
107
189
|
when :unaccent
|
|
108
190
|
Fuzzy::Unaccent.new(attr, @value).predicate
|
|
109
|
-
|
|
191
|
+
when :simple
|
|
110
192
|
Fuzzy::Simple.new(attr, @value).predicate
|
|
111
193
|
end
|
|
112
194
|
end
|
|
@@ -135,6 +217,27 @@ module Huginn
|
|
|
135
217
|
@unaccent_available ||= extension_installed?("unaccent")
|
|
136
218
|
end
|
|
137
219
|
|
|
220
|
+
# Whether the IMMUTABLE tsvector wrapper behind the :full_text strategy
|
|
221
|
+
# exists (created by `rails g huginn:fts_indexes`). FTS is core
|
|
222
|
+
# PostgreSQL, but without the wrapper the predicate would raise, so we
|
|
223
|
+
# degrade to the unaccent fallback instead.
|
|
224
|
+
def fts_available?
|
|
225
|
+
return false unless postgresql?
|
|
226
|
+
@fts_available ||= begin
|
|
227
|
+
schema, name = split_function_name(Huginn.configuration.fts_function)
|
|
228
|
+
@model.connection.select_value(
|
|
229
|
+
ActiveRecord::Base.sanitize_sql_array(
|
|
230
|
+
["SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname = ? AND p.proname = ?", schema, name]
|
|
231
|
+
)
|
|
232
|
+
).present?
|
|
233
|
+
end
|
|
234
|
+
end
|
|
235
|
+
|
|
236
|
+
def split_function_name(function)
|
|
237
|
+
schema, _, name = function.to_s.rpartition(".")
|
|
238
|
+
[schema.presence || "public", name]
|
|
239
|
+
end
|
|
240
|
+
|
|
138
241
|
def extension_installed?(name)
|
|
139
242
|
@model.connection
|
|
140
243
|
.select_all(sanitize_extension_query(name))
|
|
@@ -36,18 +36,23 @@ module Huginn
|
|
|
36
36
|
end
|
|
37
37
|
|
|
38
38
|
def searchable_columns_config_resolved
|
|
39
|
-
config = searchable_columns_config
|
|
40
|
-
|
|
41
|
-
columns:
|
|
42
|
-
|
|
43
|
-
|
|
39
|
+
config = searchable_columns_config
|
|
40
|
+
if config.nil?
|
|
41
|
+
{ columns: default_searchable_columns, associations: {} }
|
|
42
|
+
else
|
|
43
|
+
{
|
|
44
|
+
columns: Array(config[:columns]),
|
|
45
|
+
associations: config[:associations] || {}
|
|
46
|
+
}
|
|
47
|
+
end
|
|
44
48
|
end
|
|
45
49
|
|
|
46
50
|
private
|
|
47
51
|
|
|
48
52
|
def default_searchable_columns
|
|
49
53
|
columns_hash.each_with_object([]) do |(name, column), acc|
|
|
50
|
-
|
|
54
|
+
# Array columns (string[]/text[]) are not searchable text: unaccent/ILIKE need a scalar.
|
|
55
|
+
acc << name if %i[string text].include?(column.type) && !column.try(:array)
|
|
51
56
|
end
|
|
52
57
|
end
|
|
53
58
|
end
|
data/lib/huginn/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: huginn_datatable
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Kayky Marcelo
|
|
@@ -139,6 +139,9 @@ files:
|
|
|
139
139
|
- LICENSE.txt
|
|
140
140
|
- README.md
|
|
141
141
|
- README.pt-BR.md
|
|
142
|
+
- lib/generators/huginn/fts_indexes/fts_indexes_generator.rb
|
|
143
|
+
- lib/generators/huginn/fts_indexes/templates/add_huginn_fts_indexes.rb.tt
|
|
144
|
+
- lib/generators/huginn/searchable_columns.rb
|
|
142
145
|
- lib/generators/huginn/trigram_indexes/templates/add_huginn_trigram_indexes.rb.tt
|
|
143
146
|
- lib/generators/huginn/trigram_indexes/trigram_indexes_generator.rb
|
|
144
147
|
- lib/huginn.rb
|
|
@@ -178,7 +181,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
178
181
|
- !ruby/object:Gem::Version
|
|
179
182
|
version: '0'
|
|
180
183
|
requirements: []
|
|
181
|
-
rubygems_version: 4.0.
|
|
184
|
+
rubygems_version: 4.0.16
|
|
182
185
|
specification_version: 4
|
|
183
186
|
summary: Performant and elegant ActiveRecord datatables/data-grids with tolerant search
|
|
184
187
|
test_files: []
|