huginn_datatable 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -0
- data/README.md +57 -5
- data/README.pt-BR.md +57 -5
- data/lib/generators/huginn/trigram_indexes/templates/add_huginn_trigram_indexes.rb.tt +26 -0
- data/lib/generators/huginn/trigram_indexes/trigram_indexes_generator.rb +90 -0
- data/lib/huginn/configuration.rb +10 -3
- data/lib/huginn/searchable/fuzzy.rb +18 -11
- data/lib/huginn/searchable/query.rb +1 -1
- data/lib/huginn/version.rb +1 -1
- metadata +3 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a7014d31503c76a5637c2bd15b82b8cf2be1c173ef11575a9325ec8457d7c65e
|
|
4
|
+
data.tar.gz: 175542cecf2d1378d0fa571ef538c0f2cccf24d6acab94bb3e0b398e97926261
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 951dcb3aeb617bc98b23fc8e883e0fa06d572ff312e0888171b17a26b70454eddbf84972bc6dfd91589a3788ac188266d1282d9122f53522923094ebdec0017c
|
|
7
|
+
data.tar.gz: 21ed148247e3e9d3a9cff2682d4d19c15c4bdf7806fd0222c4018f829a98fde7f52720e8219e2caaaa929d8c778f5ccc6b7933de9b2245bc6b597c89f27a7745
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,12 @@
|
|
|
2
2
|
|
|
3
3
|
### Unreleased
|
|
4
4
|
|
|
5
|
+
#### 0.2.0
|
|
6
|
+
|
|
7
|
+
- **Breaking:** `:pg_trgm` search now uses the PostgreSQL `%` operator (`UNACCENT(col) % UNACCENT('term')`), driven by the **indexable** operator instead of a bare `similarity()` comparison. A `gin_trgm_ops` GIN index on `UNACCENT(col)` is used when present (BitmapOr over the same index with the ILIKE branch); without it, results stay correct via sequential scan.
|
|
8
|
+
- **Breaking:** removed the `fuzzy_threshold` configuration. The similarity cutoff is now the PostgreSQL GUC `pg_trgm.similarity_threshold` (default `0.3`, same as the old gem default) — tune with `SELECT set_limit(...)` / `SET pg_trgm.similarity_threshold`.
|
|
9
|
+
- New `unaccent_function` configuration (default `"unaccent"`): the unaccent function used in search SQL. Point it at an `IMMUTABLE` wrapper to make GIN trigram indexes usable (the pg builtin `unaccent()` is `STABLE` on PG 13+ and cannot be indexed directly).
|
|
10
|
+
- New generator `rails g huginn:trigram_indexes [Model...]` scaffolds a migration that creates an `IMMUTABLE public.f_unaccent` wrapper and `CREATE INDEX ... USING gin (public.f_unaccent(col) gin_trgm_ops)` for Searchable models (all `:string`/`:text` columns, including association-scoped ones).
|
|
5
11
|
- Runtime dependency on Pagy declared in the gemspec (`pagy >= 6`) — consumers install it automatically (datatable pagination is not test-only anymore).
|
|
6
12
|
- Supported Rails floor bumped to **7.1** (`>= 7.1, < 9`); Ruby `>= 3.0` without an upper bound (Rails 8 + Ruby 4 supported).
|
|
7
13
|
- Appraisals now cover **Rails 7.1 / 7.2 / 8.0** (was 6.1/7.0/7.1) with generated `gemfiles/*.gemfile`; CI matrix runs every Rails over supported Rubies (3.0 up to 4.0, including Rails 8 on Ruby 4.0).
|
data/README.md
CHANGED
|
@@ -1,3 +1,7 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="./logo.png" width="140" alt="Huginn logo">
|
|
3
|
+
</p>
|
|
4
|
+
|
|
1
5
|
<h1 align="center">Huginn</h1>
|
|
2
6
|
|
|
3
7
|
<p align="center">
|
|
@@ -9,14 +13,14 @@
|
|
|
9
13
|
🇺🇸 English · <a href="./README.pt-BR.md">🇧🇷 Português</a>
|
|
10
14
|
</p>
|
|
11
15
|
|
|
12
|
-
Huginn is a lightweight query layer for Rails that turns a raw datatable request into a **lean count, a paginated subset and one preload** — instead of a massive JOIN materialized in memory. It also ships a PostgreSQL fuzzy-search builder (`pg_trgm` similarity with `unaccent` and `ILIKE` fallback) that is tolerant to typos and accents.
|
|
16
|
+
Huginn is a lightweight query layer for Rails that turns a raw datatable request into a **lean count, a paginated subset and one preload** — instead of a massive JOIN materialized in memory. It also ships a PostgreSQL fuzzy-search builder (`pg_trgm` similarity via the `%` operator with `unaccent` and `ILIKE` fallback) that is tolerant to typos and accents — and that can lean on a GIN index when present.
|
|
13
17
|
|
|
14
18
|
## Highlights
|
|
15
19
|
|
|
16
20
|
- **Two-phase execution** — association filters/orders/range become reflection-secured subqueries, then a lean count and `preload` **only on the paginated subset**.
|
|
17
21
|
- **Lean counts** — `COUNT(DISTINCT pk)` through a stripped relation; no JOIN materialization.
|
|
18
22
|
- **SQL injection safe ordering/filtering** — every column reference is resolved through Arel reflection, never string-interpolated.
|
|
19
|
-
- **Accent/typo tolerant search** — `pg_trgm` similarity OR `unaccent+ILIKE`, with a configurable fallback chain.
|
|
23
|
+
- **Accent/typo tolerant search** — `pg_trgm` similarity (`%` operator) OR `unaccent+ILIKE`, with a configurable fallback chain; uses a `gin_trgm_ops` GIN index when one exists.
|
|
20
24
|
- **Rails conventions** — works with `ActionController::Parameters`, Railtie auto-includes both concerns (toggleable), zero boilerplate.
|
|
21
25
|
|
|
22
26
|
## Development
|
|
@@ -62,17 +66,24 @@ gem "huginn"
|
|
|
62
66
|
```ruby
|
|
63
67
|
# config/initializers/huginn.rb
|
|
64
68
|
Huginn.configure do |config|
|
|
65
|
-
# :pg_trgm (recommended) — trigram similarity OR unaccent+ILIKE
|
|
69
|
+
# :pg_trgm (recommended) — trigram similarity (%) OR unaccent+ILIKE
|
|
66
70
|
# :unaccent — unaccent + ILIKE only
|
|
67
71
|
# :simple — plain LIKE
|
|
68
72
|
config.search_strategy = :pg_trgm
|
|
69
73
|
|
|
70
|
-
|
|
74
|
+
# Unaccent function used around search terms/columns. Defaults to the pg
|
|
75
|
+
# builtin UNACCENT(). Point it at the IMMUTABLE wrapper created by
|
|
76
|
+
# `rails g huginn:trigram_indexes` ("public.f_unaccent") so GIN trigram
|
|
77
|
+
# indexes can actually be used.
|
|
78
|
+
config.unaccent_function = "unaccent"
|
|
79
|
+
|
|
71
80
|
config.pagy_items = 10 # default page size
|
|
72
81
|
config.pagy_max_items = 500 # hard cap for per_page
|
|
73
82
|
end
|
|
74
83
|
```
|
|
75
84
|
|
|
85
|
+
> **Threshold:** under `:pg_trgm` the similarity cutoff is the PostgreSQL GUC `pg_trgm.similarity_threshold` (default `0.3`), **not** a gem-level setting. Tune it with `SET pg_trgm.similarity_threshold = 0.4` / `SELECT set_limit(0.4)` on the database.
|
|
86
|
+
|
|
76
87
|
## Railtie (automatic include)
|
|
77
88
|
|
|
78
89
|
By default the Railtie includes `Huginn::Datatable` and `Huginn::Searchable` into every `ActiveRecord::Base` model. You do **not** need `include` statements unless you opt in selectively:
|
|
@@ -180,6 +191,46 @@ Person.search("kayky", distinct: false) # disable the implicit DISTIN
|
|
|
180
191
|
|
|
181
192
|
`Huginn::Datatable` reuses `Huginn::Searchable.search` automatically when the model responds to `search`.
|
|
182
193
|
|
|
194
|
+
## Trigram indexes (performance)
|
|
195
|
+
|
|
196
|
+
Under `:pg_trgm`, each searchable column produces an indexable predicate:
|
|
197
|
+
|
|
198
|
+
```sql
|
|
199
|
+
(UNACCENT(col) % UNACCENT('term')) OR (UNACCENT(col) ILIKE UNACCENT('%term%'))
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
Both branches are supported by a **GIN trigram index**, so the planner can run a `BitmapOr` on it. Because the pg builtin `unaccent()` is `STABLE` (not `IMMUTABLE`) on PostgreSQL 13+, it cannot be used directly in an index expression — you need an `IMMUTABLE` wrapper plus an expression index on it:
|
|
203
|
+
|
|
204
|
+
```sql
|
|
205
|
+
CREATE OR REPLACE FUNCTION public.f_unaccent(text)
|
|
206
|
+
RETURNS text AS $$
|
|
207
|
+
SELECT public.unaccent('public.unaccent', $1);
|
|
208
|
+
$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;
|
|
209
|
+
|
|
210
|
+
CREATE INDEX index_people_name_trgm ON people USING gin (public.f_unaccent(name) gin_trgm_ops);
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
Then point the gem at the wrapper:
|
|
214
|
+
|
|
215
|
+
```ruby
|
|
216
|
+
Huginn.configure { |c| c.unaccent_function = "public.f_unaccent" }
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
Or scaffold the migration for every Searchable model (wrapper + indexes included):
|
|
220
|
+
|
|
221
|
+
```sh
|
|
222
|
+
rails g huginn:trigram_indexes # all Searchable models
|
|
223
|
+
rails g huginn:trigram_indexes Person Product # specific models
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
The index is **optional** — without it the search still returns correct results (via a sequential scan), and the same indexes also accelerate the `:unaccent` (ILIKE) and `:simple` (LIKE) strategies.
|
|
227
|
+
|
|
228
|
+
Notes:
|
|
229
|
+
|
|
230
|
+
- Requires the `pg_trgm` and `unaccent` extensions.
|
|
231
|
+
- Only helpful for search terms of **3 or more characters** — trigrams need that many to match. Shorter terms always scan.
|
|
232
|
+
- To see it working, run a `Person.search("term").explain` with the wrapper configured and a `gin_trgm_ops` index present.
|
|
233
|
+
|
|
183
234
|
## Query efficiency
|
|
184
235
|
|
|
185
236
|
```
|
|
@@ -195,7 +246,7 @@ For a `Plano` datatable with deep `includes:`, this is exactly **2 extra queries
|
|
|
195
246
|
|
|
196
247
|
```
|
|
197
248
|
lib/huginn.rb entry, Huginn.configure, Huginn.instrument
|
|
198
|
-
lib/huginn/configuration.rb search_strategy,
|
|
249
|
+
lib/huginn/configuration.rb search_strategy, unaccent_function, pagy_*
|
|
199
250
|
lib/huginn/railtie.rb auto-includes concerns into ActiveRecord
|
|
200
251
|
lib/huginn/datatable.rb Huginn::Datatable (aggregator)
|
|
201
252
|
lib/huginn/datatable/datatable.rb the datatable Concern
|
|
@@ -208,6 +259,7 @@ lib/huginn/searchable.rb Huginn::Searchable (aggregator)
|
|
|
208
259
|
lib/huginn/searchable/searchable.rb the search Concern + DSL
|
|
209
260
|
lib/huginn/searchable/query.rb tolerant search builder (joins + OR)
|
|
210
261
|
lib/huginn/searchable/fuzzy.rb pg_trgm / unaccent / simple predicates
|
|
262
|
+
lib/generators/... `huginn:trigram_indexes` (GIN index migrations)
|
|
211
263
|
```
|
|
212
264
|
|
|
213
265
|
## Instrumentation
|
data/README.pt-BR.md
CHANGED
|
@@ -1,3 +1,7 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="./logo.png" width="140" alt="Logo do Huginn">
|
|
3
|
+
</p>
|
|
4
|
+
|
|
1
5
|
<h1 align="center">Huginn</h1>
|
|
2
6
|
|
|
3
7
|
<p align="center">
|
|
@@ -9,14 +13,14 @@
|
|
|
9
13
|
<a href="./README.md">🇺🇸 English</a> · 🇧🇷 Português
|
|
10
14
|
</p>
|
|
11
15
|
|
|
12
|
-
O Huginn é uma camada de consulta leve para Rails que transforma uma requisição de datatable bruta em um **count enxuto, um subconjunto paginado e um único preload** — em vez de um JOIN enorme materializado em memória. Também traz um construtor de busca *fuzzy* para PostgreSQL (similaridade `pg_trgm` com `unaccent` e fallback `ILIKE`) tolerante a erros de digitação e acentuação.
|
|
16
|
+
O Huginn é uma camada de consulta leve para Rails que transforma uma requisição de datatable bruta em um **count enxuto, um subconjunto paginado e um único preload** — em vez de um JOIN enorme materializado em memória. Também traz um construtor de busca *fuzzy* para PostgreSQL (similaridade `pg_trgm` via operador `%` com `unaccent` e fallback `ILIKE`) tolerante a erros de digitação e acentuação — e que pode usar um índice GIN quando presente.
|
|
13
17
|
|
|
14
18
|
## Destaques
|
|
15
19
|
|
|
16
20
|
- **Execução em duas fases** — filtros/orders/ranges de associação se tornam subqueries resolvidas por reflexão, e o `preload` é feito **somente no subconjunto paginado**.
|
|
17
21
|
- **Counts enxutos** — `COUNT(DISTINCT pk)` através de uma relation restrita, sem materialização de JOINs.
|
|
18
22
|
- **Ordenação/filtro seguros contra SQL injection** — toda referência de coluna é resolvida via reflexão do Arel, nunca interpolada como string.
|
|
19
|
-
- **Busca tolerante a acentos/typos** — similaridade `pg_trgm` OU `unaccent+ILIKE`, com cadeia de fallback configurável.
|
|
23
|
+
- **Busca tolerante a acentos/typos** — similaridade `pg_trgm` (operador `%`) OU `unaccent+ILIKE`, com cadeia de fallback configurável; usa índice GIN `gin_trgm_ops` quando existente.
|
|
20
24
|
- **Convenções do Rails** — funciona com `ActionController::Parameters`, Railtie inclui ambos os concerns automaticamente (desligável), zero boilerplate.
|
|
21
25
|
|
|
22
26
|
## Desenvolvimento
|
|
@@ -62,17 +66,24 @@ gem "huginn"
|
|
|
62
66
|
```ruby
|
|
63
67
|
# config/initializers/huginn.rb
|
|
64
68
|
Huginn.configure do |config|
|
|
65
|
-
# :pg_trgm (recomendado) — similaridade trigram OU unaccent+ILIKE
|
|
69
|
+
# :pg_trgm (recomendado) — similaridade trigram (%) OU unaccent+ILIKE
|
|
66
70
|
# :unaccent — somente unaccent + ILIKE
|
|
67
71
|
# :simple — LIKE simples
|
|
68
72
|
config.search_strategy = :pg_trgm
|
|
69
73
|
|
|
70
|
-
|
|
74
|
+
# Função unaccent usada em torno de termos/colunas. O default é o UNACCENT()
|
|
75
|
+
# built-in do PG. Aponte para o wrapper IMMUTABLE criado por
|
|
76
|
+
# `rails g huginn:trigram_indexes` ("public.f_unaccent") para que índices
|
|
77
|
+
# GIN trigram possam ser usados de fato.
|
|
78
|
+
config.unaccent_function = "unaccent"
|
|
79
|
+
|
|
71
80
|
config.pagy_items = 10 # tamanho de página padrão
|
|
72
81
|
config.pagy_max_items = 500 # teto máximo de per_page
|
|
73
82
|
end
|
|
74
83
|
```
|
|
75
84
|
|
|
85
|
+
> **Limiar:** sob `:pg_trgm`, o corte de similaridade é o GUC do PostgreSQL `pg_trgm.similarity_threshold` (default `0.3`), **não** uma configuração da gem. Ajuste com `SET pg_trgm.similarity_threshold = 0.4` / `SELECT set_limit(0.4)` no banco.
|
|
86
|
+
|
|
76
87
|
## Railtie (include automático)
|
|
77
88
|
|
|
78
89
|
Por padrão, o Railtie inclui `Huginn::Datatable` e `Huginn::Searchable` em toda `ActiveRecord::Base`. Você **não** precisa de `include`, a menos que opte seletivamente:
|
|
@@ -180,6 +191,46 @@ Person.search("kayky", distinct: false) # desativa o DISTINCT implíc
|
|
|
180
191
|
|
|
181
192
|
`Huginn::Datatable` reutiliza `Huginn::Searchable.search` automaticamente quando o modelo responde a `search`.
|
|
182
193
|
|
|
194
|
+
## Índices trigram (performance)
|
|
195
|
+
|
|
196
|
+
Sob `:pg_trgm`, cada coluna pesquisável gera um predicado indexável:
|
|
197
|
+
|
|
198
|
+
```sql
|
|
199
|
+
(UNACCENT(col) % UNACCENT('termo')) OR (UNACCENT(col) ILIKE UNACCENT('%termo%'))
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
Ambos os ramos são suportados por um **índice GIN trigram**, então o planner pode executar um `BitmapOr` sobre ele. Como o `unaccent()` built-in do PG é `STABLE` (não `IMMUTABLE`) desde o PostgreSQL 13+, ele não pode ser usado diretamente numa expressão de índice — você precisa de um wrapper `IMMUTABLE` + índice de expressão:
|
|
203
|
+
|
|
204
|
+
```sql
|
|
205
|
+
CREATE OR REPLACE FUNCTION public.f_unaccent(text)
|
|
206
|
+
RETURNS text AS $$
|
|
207
|
+
SELECT public.unaccent('public.unaccent', $1);
|
|
208
|
+
$$ LANGUAGE sql IMMUTABLE PARALLEL SAFE;
|
|
209
|
+
|
|
210
|
+
CREATE INDEX index_people_name_trgm ON people USING gin (public.f_unaccent(name) gin_trgm_ops);
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
E aponte a gem para o wrapper:
|
|
214
|
+
|
|
215
|
+
```ruby
|
|
216
|
+
Huginn.configure { |c| c.unaccent_function = "public.f_unaccent" }
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
Ou gere a migration para todos os modelos Searchable (wrapper + índices inclusos):
|
|
220
|
+
|
|
221
|
+
```sh
|
|
222
|
+
rails g huginn:trigram_indexes # todos os modelos Searchable
|
|
223
|
+
rails g huginn:trigram_indexes Person Product # modelos específicos
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
O índice é **opcional** — sem ele a busca ainda retorna resultados corretos (via sequential scan), e os mesmos índices também aceleram as estratégias `:unaccent` (ILIKE) e `:simple` (LIKE).
|
|
227
|
+
|
|
228
|
+
Observações:
|
|
229
|
+
|
|
230
|
+
- Requer as extensões `pg_trgm` e `unaccent`.
|
|
231
|
+
- Só ajuda em termos de busca de **3 caracteres ou mais** — trigramas precisam disso para casar. Termos menores sempre fazem scan.
|
|
232
|
+
- Para conferir o uso, rode `Person.search("termo").explain` com o wrapper configurado e um índice `gin_trgm_ops` presente.
|
|
233
|
+
|
|
183
234
|
## Eficiência da query
|
|
184
235
|
|
|
185
236
|
```
|
|
@@ -195,7 +246,7 @@ Para um datatable de `Plano` com `includes:` profundos, isso são exatamente **2
|
|
|
195
246
|
|
|
196
247
|
```
|
|
197
248
|
lib/huginn.rb entry, Huginn.configure, Huginn.instrument
|
|
198
|
-
lib/huginn/configuration.rb search_strategy,
|
|
249
|
+
lib/huginn/configuration.rb search_strategy, unaccent_function, pagy_*
|
|
199
250
|
lib/huginn/railtie.rb auto-inclui os concerns no ActiveRecord
|
|
200
251
|
lib/huginn/datatable.rb Huginn::Datatable (agregador)
|
|
201
252
|
lib/huginn/datatable/datatable.rb o Concern do datatable
|
|
@@ -208,6 +259,7 @@ lib/huginn/searchable.rb Huginn::Searchable (agregador)
|
|
|
208
259
|
lib/huginn/searchable/searchable.rb o Concern do search + DSL
|
|
209
260
|
lib/huginn/searchable/query.rb construtor de busca tolerante (joins + OR)
|
|
210
261
|
lib/huginn/searchable/fuzzy.rb predicados pg_trgm / unaccent / simple
|
|
262
|
+
lib/generators/... gerador `huginn:trigram_indexes` (migrations de índices GIN)
|
|
211
263
|
```
|
|
212
264
|
|
|
213
265
|
## Instrumentação
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
class AddHuginnTrigramIndexes < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
|
|
2
|
+
def up
|
|
3
|
+
enable_extension "unaccent" unless connection.extension_enabled?("unaccent")
|
|
4
|
+
|
|
5
|
+
# The pg builtin unaccent() is STABLE (not IMMUTABLE) since PG 13+, so it
|
|
6
|
+
# cannot be used in an index expression. Wrap it in an IMMUTABLE function
|
|
7
|
+
# that always pins the default dictionary.
|
|
8
|
+
execute <<~SQL
|
|
9
|
+
CREATE OR REPLACE FUNCTION public.f_unaccent(text)
|
|
10
|
+
RETURNS text AS $$
|
|
11
|
+
SELECT public.unaccent('public.unaccent', $1);
|
|
12
|
+
$$
|
|
13
|
+
LANGUAGE sql IMMUTABLE PARALLEL SAFE;
|
|
14
|
+
SQL
|
|
15
|
+
<% @entries.each do |entry| %>
|
|
16
|
+
add_index :<%= entry[:table] %>, "public.f_unaccent(<%= entry[:column] %>) gin_trgm_ops", using: :gin, name: "index_<%= entry[:table] %>_<%= entry[:column] %>_trgm"
|
|
17
|
+
<% end %>
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def down
|
|
21
|
+
<% @entries.each do |entry| %>
|
|
22
|
+
remove_index :<%= entry[:table] %>, name: "index_<%= entry[:table] %>_<%= entry[:column] %>_trgm"
|
|
23
|
+
<% end %>
|
|
24
|
+
execute "DROP FUNCTION IF EXISTS public.f_unaccent(text)"
|
|
25
|
+
end
|
|
26
|
+
end
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rails/generators"
|
|
4
|
+
require "rails/generators/migration"
|
|
5
|
+
|
|
6
|
+
module Huginn
|
|
7
|
+
module Generators
|
|
8
|
+
# Scaffolds a migration that creates the PostgreSQL expression GIN indexes
|
|
9
|
+
# (gin_trgm_ops on an IMMUTABLE unaccent wrapper of the column) backing
|
|
10
|
+
# Huginn::Searchable's :pg_trgm strategy for every model that includes
|
|
11
|
+
# Huginn::Searchable.
|
|
12
|
+
#
|
|
13
|
+
# rails g huginn:trigram_indexes # all Searchable models
|
|
14
|
+
# rails g huginn:trigram_indexes Person Product # specific models
|
|
15
|
+
#
|
|
16
|
+
# The migration creates an IMMUTABLE public.f_unaccent(text) wrapper and
|
|
17
|
+
# the GIN indexes. To make the search actually use them, point the gem at
|
|
18
|
+
# the wrapper (the pg builtin unaccent() is STABLE on PG 13+ and cannot be
|
|
19
|
+
# indexed directly):
|
|
20
|
+
#
|
|
21
|
+
# Huginn.configure { |c| c.unaccent_function = "public.f_unaccent" }
|
|
22
|
+
#
|
|
23
|
+
# Indexes are only used when the :pg_trgm strategy is configured and the
|
|
24
|
+
# pg_trgm/unaccent extensions are installed.
|
|
25
|
+
class TrigramIndexesGenerator < Rails::Generators::Base
|
|
26
|
+
include Rails::Generators::Migration
|
|
27
|
+
|
|
28
|
+
source_root File.expand_path("templates", __dir__)
|
|
29
|
+
|
|
30
|
+
argument :model_names, type: :array, default: [], desc: "Models to index (default: all Searchable models)"
|
|
31
|
+
|
|
32
|
+
def self.next_migration_number(dirname)
|
|
33
|
+
next_migration_number = current_migration_number(dirname) + 1
|
|
34
|
+
ActiveRecord::Migration.next_migration_number(next_migration_number)
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def copy_migration
|
|
38
|
+
@entries = resolve_entries
|
|
39
|
+
return if @entries.empty?
|
|
40
|
+
|
|
41
|
+
migration_template(
|
|
42
|
+
"add_huginn_trigram_indexes.rb.tt",
|
|
43
|
+
"db/migrate/add_huginn_trigram_indexes.rb"
|
|
44
|
+
)
|
|
45
|
+
end
|
|
46
|
+
|
|
47
|
+
private
|
|
48
|
+
|
|
49
|
+
def resolve_entries
|
|
50
|
+
models = if model_names.any?
|
|
51
|
+
model_names.map { |name| constantize(name) }
|
|
52
|
+
else
|
|
53
|
+
ActiveRecord::Base.descendants
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
models.filter_map do |model|
|
|
57
|
+
next unless model.respond_to?(:searchable_columns_config_resolved)
|
|
58
|
+
|
|
59
|
+
columns_for(model)
|
|
60
|
+
end.reject(&:empty?).flatten.uniq
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def columns_for(model)
|
|
64
|
+
scope = []
|
|
65
|
+
resolved = model.searchable_columns_config_resolved
|
|
66
|
+
|
|
67
|
+
Array(resolved[:columns]).map(&:to_s).reject(&:blank?).each do |field|
|
|
68
|
+
scope << { table: model.table_name, column: field } if model.columns_hash.key?(field)
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
resolved[:associations].each do |assoc, cols|
|
|
72
|
+
reflection = model.reflect_on_association(assoc.to_sym)
|
|
73
|
+
next unless reflection
|
|
74
|
+
|
|
75
|
+
Array(cols).map(&:to_s).each do |col|
|
|
76
|
+
scope << { table: reflection.klass.table_name, column: col } if reflection.klass.columns_hash.key?(col)
|
|
77
|
+
end
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
scope
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
def constantize(klass_name)
|
|
84
|
+
klass_name.constantize
|
|
85
|
+
rescue NameError
|
|
86
|
+
raise Thor::Error, "Could not find model #{klass_name.inspect}"
|
|
87
|
+
end
|
|
88
|
+
end
|
|
89
|
+
end
|
|
90
|
+
end
|
data/lib/huginn/configuration.rb
CHANGED
|
@@ -2,19 +2,26 @@
|
|
|
2
2
|
|
|
3
3
|
module Huginn
|
|
4
4
|
class Configuration
|
|
5
|
-
attr_accessor :
|
|
5
|
+
attr_accessor :pagy_items, :pagy_max_items, :unaccent_function
|
|
6
6
|
attr_writer :search_strategy, :auto_include_datatable, :auto_include_searchable
|
|
7
7
|
|
|
8
8
|
def initialize
|
|
9
|
-
@fuzzy_threshold = 0.3
|
|
10
9
|
@pagy_items = 10
|
|
11
10
|
@pagy_max_items = 500
|
|
11
|
+
# The unaccent function SQL emits around searchable columns/terms. Defaults
|
|
12
|
+
# to the pg builtin UNACCENT(). To let GIN trigram indexes be used, point
|
|
13
|
+
# this at an IMMUTABLE wrapper (see `rails g huginn:trigram_indexes`),
|
|
14
|
+
# e.g. "public.f_unaccent".
|
|
15
|
+
@unaccent_function = "unaccent"
|
|
12
16
|
@search_strategy = :pg_trgm
|
|
13
17
|
@auto_include_datatable = true
|
|
14
18
|
@auto_include_searchable = true
|
|
15
19
|
end
|
|
16
20
|
|
|
17
|
-
# :pg_trgm -> similarity(
|
|
21
|
+
# :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE. The similarity
|
|
22
|
+
# cutoff is the PostgreSQL GUC pg_trgm.similarity_threshold
|
|
23
|
+
# (tune with SELECT set_limit(...)); a gin_trgm_ops GIN index
|
|
24
|
+
# on UNACCENT(col) is used when present.
|
|
18
25
|
# :unaccent -> unaccent + ILIKE only
|
|
19
26
|
# :simple -> plain LIKE
|
|
20
27
|
def search_strategy
|
|
@@ -6,16 +6,24 @@ module Huginn
|
|
|
6
6
|
#
|
|
7
7
|
# Strategy chain (driven by Huginn.configuration.search_strategy):
|
|
8
8
|
#
|
|
9
|
-
# :pg_trgm -> trigram similarity OR unaccent+ILIKE (best for typos)
|
|
9
|
+
# :pg_trgm -> trigram similarity (%) OR unaccent+ILIKE (best for typos).
|
|
10
|
+
# Both branches are supported by a gin_trgm_ops GIN index
|
|
11
|
+
# on UNACCENT(col); the index is used only when it exists.
|
|
10
12
|
# :unaccent -> unaccent + ILIKE (case/accents insensitive)
|
|
11
13
|
# :simple -> plain LIKE
|
|
12
14
|
module Fuzzy
|
|
13
|
-
# Gentle helpers to wrap a node in UNACCENT(...).
|
|
15
|
+
# Gentle helpers to wrap a node in UNACCENT(...). The function name comes
|
|
16
|
+
# from Huginn.configuration.unaccent_function so an IMMUTABLE wrapper can
|
|
17
|
+
# be swapped in to make GIN trigram indexes usable.
|
|
14
18
|
module Unaccentable
|
|
15
19
|
private
|
|
16
20
|
|
|
17
21
|
def unaccent(node)
|
|
18
|
-
Arel::Nodes::NamedFunction.new(
|
|
22
|
+
Arel::Nodes::NamedFunction.new(unaccent_function, [node])
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def unaccent_function
|
|
26
|
+
Huginn.configuration.unaccent_function
|
|
19
27
|
end
|
|
20
28
|
end
|
|
21
29
|
|
|
@@ -62,23 +70,22 @@ module Huginn
|
|
|
62
70
|
class Trigram
|
|
63
71
|
include Unaccentable
|
|
64
72
|
|
|
65
|
-
def initialize(column, value
|
|
73
|
+
def initialize(column, value)
|
|
66
74
|
@column = column
|
|
67
75
|
@value = value
|
|
68
|
-
@threshold = threshold
|
|
69
76
|
end
|
|
70
77
|
|
|
71
78
|
def predicate
|
|
72
|
-
similarity = Arel::Nodes::
|
|
73
|
-
|
|
74
|
-
unaccent(Arel::Nodes.build_quoted(@value, @column))
|
|
75
|
-
])
|
|
76
|
-
threshold = Arel::Nodes.build_quoted(@threshold)
|
|
77
|
-
Arel::Nodes::InfixOperation.new(">", similarity, threshold).or(fallback)
|
|
79
|
+
similarity = Arel::Nodes::InfixOperation.new("%", unaccent(@column), unaccent(value))
|
|
80
|
+
similarity.or(fallback)
|
|
78
81
|
end
|
|
79
82
|
|
|
80
83
|
private
|
|
81
84
|
|
|
85
|
+
def value
|
|
86
|
+
Arel::Nodes.build_quoted(@value, @column)
|
|
87
|
+
end
|
|
88
|
+
|
|
82
89
|
def fallback
|
|
83
90
|
Unaccent.new(@column, @value).predicate
|
|
84
91
|
end
|
data/lib/huginn/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: huginn_datatable
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Kayky Marcelo
|
|
@@ -139,6 +139,8 @@ files:
|
|
|
139
139
|
- LICENSE.txt
|
|
140
140
|
- README.md
|
|
141
141
|
- README.pt-BR.md
|
|
142
|
+
- lib/generators/huginn/trigram_indexes/templates/add_huginn_trigram_indexes.rb.tt
|
|
143
|
+
- lib/generators/huginn/trigram_indexes/trigram_indexes_generator.rb
|
|
142
144
|
- lib/huginn.rb
|
|
143
145
|
- lib/huginn/configuration.rb
|
|
144
146
|
- lib/huginn/datatable.rb
|