rails-paradedb 0.11.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +28 -2
- data/README.md +13 -35
- data/lib/generators/parade_db/index/templates/index.rb.tt +0 -1
- data/lib/parade_db/aggregations.rb +5 -37
- data/lib/parade_db/diagnostics.rb +19 -0
- data/lib/parade_db/index.rb +23 -32
- data/lib/parade_db/migration_helpers.rb +25 -29
- data/lib/parade_db/model.rb +18 -12
- data/lib/parade_db/query_builder.rb +422 -0
- data/lib/parade_db/search_methods.rb +81 -153
- data/lib/parade_db/tokenizer.rb +6 -0
- data/lib/parade_db/version.rb +1 -1
- data/lib/parade_db.rb +23 -2
- metadata +6 -11
- data/lib/parade_db/arel/builder.rb +0 -654
- data/lib/parade_db/arel/nodes.rb +0 -191
- data/lib/parade_db/arel/predications.rb +0 -261
- data/lib/parade_db/arel/visitor.rb +0 -99
- data/lib/parade_db/arel.rb +0 -26
- data/lib/parade_db/railtie.rb +0 -14
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 87745906c0e84ce264fcb243665c8edaf756218568745eecbe2781b0aa35f52f
|
|
4
|
+
data.tar.gz: 499f1837259b7a2c2a56ff6c55a98ff8dc6ef72e9de08517631215d9a7c091c2
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ffcfe007ab852d8b36f69e9af1853dc1b0df485bfdbb0599749aa6ee27e1dd43707ea8958733c6b4c112fdd3d2ba6038d460dfb8c3062475e53826164d54a3f7
|
|
7
|
+
data.tar.gz: 236e0df31d2f705f3b7f015e5c051951e4aec16e3438961e5ff2a2757ab007f3a21565c312847aaa26fc844d3fd9c16e3c732634ba2d1d9b3c4a00f9a30263a9
|
data/CHANGELOG.md
CHANGED
|
@@ -2,13 +2,37 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
4
4
|
|
|
5
|
+
## [Unreleased]
|
|
6
|
+
|
|
7
|
+
## [0.13.0] - 2026-10-07
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- Index partitioning, target segment count, and vector quantization configuration.
|
|
12
|
+
- Vector storage, configuration, and estimator diagnostics, plus aggregation visibility modes.
|
|
13
|
+
- Snippet-position pagination with `limit` and `offset`.
|
|
14
|
+
- Index layer sizes, background layer sizes, and mutable segment row limits.
|
|
15
|
+
- Index search tokenizer configuration.
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
|
|
19
|
+
- **Breaking:** Remove the key-field setting from index definitions, migration helpers, and schema dumps. Query helpers use the model primary key. Require ParadeDB 0.26.0 or newer.
|
|
20
|
+
- **Breaking:** Update vector index build options to `training_sample_ratio` and `max_leaf_size` for ParadeDB 0.26.0 and remove the obsolete cluster replication option.
|
|
21
|
+
|
|
22
|
+
## [0.12.0] - 2026-08-13
|
|
23
|
+
|
|
24
|
+
### Changed
|
|
25
|
+
|
|
26
|
+
- **BREAKING**: Search modifiers are now composable functions: `ParadeDB.boost`, `ParadeDB.constant`, `ParadeDB.fuzzy`, `ParadeDB.slop`, and `ParadeDB.tokenize`. Pass the wrapped value to search methods instead of using modifier keyword arguments.
|
|
27
|
+
- **BREAKING**: Removed the public `ParadeDB::Arel` builder, Arel attribute predications, and `Model.paradedb_arel`. ParadeDB queries now use a private builder composed from standard Active Record nodes.
|
|
28
|
+
|
|
5
29
|
## [0.11.0] - 2026-08-04
|
|
6
30
|
|
|
7
31
|
### Added
|
|
8
32
|
|
|
9
33
|
- Vector index build options `centroid_ratio`, `training_samples_per_centroid`, and `cluster_replication` in `ParadeDB::Index` `index_options` and `add_paradedb_index` (pg_search 0.25.0+). They are emitted in the `WITH (...)` clause and round-tripped through the schema dumper.
|
|
10
34
|
|
|
11
|
-
[0.10.0] - 2026-08-04
|
|
35
|
+
## [0.10.0] - 2026-08-04
|
|
12
36
|
|
|
13
37
|
### Added
|
|
14
38
|
|
|
@@ -24,7 +48,7 @@ All notable changes to this project will be documented in this file. The format
|
|
|
24
48
|
- **BREAKING**: Index creation always emits `USING paradedb`, which requires pg_search 0.25.0+. There is no option to select the legacy `bm25` access method.
|
|
25
49
|
- **BREAKING**: The default index name is now `<table>_search_idx` (previously `<table>_bm25_idx`), and the index generator emits `Create<Model>SearchIndex` migrations named `create_<table>_search_index.rb`.
|
|
26
50
|
|
|
27
|
-
[0.9.0] - 2026-07-14
|
|
51
|
+
## [0.9.0] - 2026-07-14
|
|
28
52
|
|
|
29
53
|
### Added
|
|
30
54
|
|
|
@@ -174,6 +198,8 @@ All notable changes to this project will be documented in this file. The format
|
|
|
174
198
|
- Schema dump/load round-trip for tokenizer configuration and index options
|
|
175
199
|
(including `target_segment_count`)
|
|
176
200
|
|
|
201
|
+
[0.13.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.13.0
|
|
202
|
+
[0.12.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.12.0
|
|
177
203
|
[0.11.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.11.0
|
|
178
204
|
[0.10.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.10.0
|
|
179
205
|
[0.9.0]: https://github.com/paradedb/rails-paradedb/releases/tag/v0.9.0
|
data/README.md
CHANGED
|
@@ -10,16 +10,16 @@
|
|
|
10
10
|
</h1>
|
|
11
11
|
|
|
12
12
|
<p align="center">
|
|
13
|
-
<b>
|
|
13
|
+
<b>Just use Postgres.</b><br/>
|
|
14
14
|
One Postgres for your application data, full-text search, vector retrieval, and aggregations.
|
|
15
15
|
</p>
|
|
16
16
|
|
|
17
17
|
<h3 align="center">
|
|
18
18
|
<a href="https://paradedb.com">Website</a> •
|
|
19
|
-
<a href="https://
|
|
19
|
+
<a href="https://www.paradedb.com/docs/start/introduction">Docs</a> •
|
|
20
20
|
<a href="https://paradedb.com/slack/">Community</a> •
|
|
21
21
|
<a href="https://paradedb.com/blog/">Blog</a> •
|
|
22
|
-
<a href="https://
|
|
22
|
+
<a href="https://www.paradedb.com/docs/project/changelog">Changelog</a>
|
|
23
23
|
</h3>
|
|
24
24
|
|
|
25
25
|
<p align="center">
|
|
@@ -36,47 +36,25 @@
|
|
|
36
36
|
|
|
37
37
|
## ParadeDB for Rails
|
|
38
38
|
|
|
39
|
-
The official ActiveRecord
|
|
39
|
+
The official [ActiveRecord](https://guides.rubyonrails.org/active_record_basics.html) integration for ParadeDB. Get started with the [setup guide](https://www.paradedb.com/docs/start/connect-your-app#rails).
|
|
40
40
|
|
|
41
41
|
## Requirements & Compatibility
|
|
42
42
|
|
|
43
|
-
| Component
|
|
44
|
-
|
|
|
45
|
-
| Ruby
|
|
46
|
-
| Rails
|
|
47
|
-
|
|
|
48
|
-
|
|
|
49
|
-
|
|
|
50
|
-
|
|
51
|
-
## Vector Search
|
|
52
|
-
|
|
53
|
-
rails-paradedb supports full-text search and vector search over pgvector `vector(n)` columns. See the [vector search documentation](https://docs.paradedb.com/documentation/vector/overview) for details.
|
|
54
|
-
|
|
55
|
-
## Examples
|
|
56
|
-
|
|
57
|
-
- [Quickstart](examples/quickstart/quickstart.rb)
|
|
58
|
-
- [Vector Search](examples/vector_search/vector_search.rb)
|
|
59
|
-
- [Faceted Search](examples/faceted_search/faceted_search.rb)
|
|
60
|
-
- [Autocomplete](examples/autocomplete/autocomplete.rb)
|
|
61
|
-
- [More Like This](examples/more_like_this/more_like_this.rb)
|
|
62
|
-
- [Hybrid Search (RRF)](examples/hybrid_rrf/hybrid_rrf.rb)
|
|
63
|
-
- [RAG](examples/rag/rag.rb)
|
|
43
|
+
| Component | Supported |
|
|
44
|
+
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
|
|
45
|
+
| Ruby | 3.2+ |
|
|
46
|
+
| Rails | 7.2+ |
|
|
47
|
+
| PostgreSQL | 15+ |
|
|
48
|
+
| pgvector | 0.7.0+ (provides vector types for ParadeDB’s [native vector search](https://www.paradedb.com/docs/reference/vector/overview)) |
|
|
49
|
+
| ParadeDB / pg_search | 0.26.0+ |
|
|
64
50
|
|
|
65
51
|
## Contributing
|
|
66
52
|
|
|
67
|
-
See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup,
|
|
53
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, running tests, linting, and the PR workflow.
|
|
68
54
|
|
|
69
55
|
## Support
|
|
70
56
|
|
|
71
|
-
If you're missing a feature or have found a bug, open a
|
|
72
|
-
[GitHub Issue](https://github.com/paradedb/rails-paradedb/issues/new/choose).
|
|
73
|
-
|
|
74
|
-
For community support:
|
|
75
|
-
|
|
76
|
-
- Join the [ParadeDB Slack Community](https://paradedb.com/slack)
|
|
77
|
-
- Ask in [ParadeDB Discussions](https://github.com/paradedb/paradedb/discussions)
|
|
78
|
-
|
|
79
|
-
For commercial support, contact [sales@paradedb.com](mailto:sales@paradedb.com).
|
|
57
|
+
If you're missing a feature or have found a bug, please open a [GitHub Issue](https://github.com/paradedb/rails-paradedb/issues/new/choose). For community support, join the [ParadeDB Slack Community](https://paradedb.com/slack).
|
|
80
58
|
|
|
81
59
|
## Acknowledgments
|
|
82
60
|
|
|
@@ -7,14 +7,7 @@ module ParadeDB
|
|
|
7
7
|
# Backward-compatible reader for code that accessed `filtered_spec.filter`.
|
|
8
8
|
alias filter agg_filter
|
|
9
9
|
end
|
|
10
|
-
FieldTermFilter = Struct.new(
|
|
11
|
-
:field,
|
|
12
|
-
:term,
|
|
13
|
-
:distance,
|
|
14
|
-
:prefix,
|
|
15
|
-
:transposition_cost_one,
|
|
16
|
-
keyword_init: true
|
|
17
|
-
)
|
|
10
|
+
FieldTermFilter = Struct.new(:field, :term, keyword_init: true)
|
|
18
11
|
|
|
19
12
|
TERMS_ORDER = {
|
|
20
13
|
count_desc: { "_count" => "desc" },
|
|
@@ -154,16 +147,9 @@ module ParadeDB
|
|
|
154
147
|
{ "top_hits" => payload }
|
|
155
148
|
end
|
|
156
149
|
|
|
157
|
-
def filtered(spec, filter: nil, field: nil, term: nil
|
|
150
|
+
def filtered(spec, filter: nil, field: nil, term: nil)
|
|
158
151
|
normalized_spec = normalize_spec(spec)
|
|
159
|
-
normalized_filter = normalize_filter(
|
|
160
|
-
filter: filter,
|
|
161
|
-
field: field,
|
|
162
|
-
term: term,
|
|
163
|
-
distance: distance,
|
|
164
|
-
prefix: prefix,
|
|
165
|
-
transposition_cost_one: transposition_cost_one
|
|
166
|
-
)
|
|
152
|
+
normalized_filter = normalize_filter(filter: filter, field: field, term: term)
|
|
167
153
|
FilteredSpec.new(spec: normalized_spec, agg_filter: normalized_filter)
|
|
168
154
|
end
|
|
169
155
|
|
|
@@ -211,7 +197,7 @@ module ParadeDB
|
|
|
211
197
|
end
|
|
212
198
|
private_class_method :normalize_spec
|
|
213
199
|
|
|
214
|
-
def normalize_filter(filter:, field:, term
|
|
200
|
+
def normalize_filter(filter:, field:, term:)
|
|
215
201
|
if filter
|
|
216
202
|
if !field.nil? || !term.nil?
|
|
217
203
|
raise ArgumentError, "filtered aggregation accepts either filter: or field/term arguments, not both"
|
|
@@ -223,17 +209,7 @@ module ParadeDB
|
|
|
223
209
|
raise ArgumentError, "filtered aggregation requires filter: or both field: and term:"
|
|
224
210
|
end
|
|
225
211
|
|
|
226
|
-
|
|
227
|
-
normalized_prefix = normalize_boolean_option(prefix, "prefix")
|
|
228
|
-
normalized_transposition = normalize_boolean_option(transposition_cost_one, "transposition_cost_one")
|
|
229
|
-
|
|
230
|
-
FieldTermFilter.new(
|
|
231
|
-
field: normalize_field(field),
|
|
232
|
-
term: term,
|
|
233
|
-
distance: normalized_distance,
|
|
234
|
-
prefix: normalized_prefix,
|
|
235
|
-
transposition_cost_one: normalized_transposition
|
|
236
|
-
)
|
|
212
|
+
FieldTermFilter.new(field: normalize_field(field), term: term)
|
|
237
213
|
end
|
|
238
214
|
private_class_method :normalize_filter
|
|
239
215
|
|
|
@@ -318,14 +294,6 @@ module ParadeDB
|
|
|
318
294
|
end
|
|
319
295
|
private_class_method :normalize_sort_direction
|
|
320
296
|
|
|
321
|
-
def normalize_boolean_option(value, name)
|
|
322
|
-
return nil if value.nil?
|
|
323
|
-
return value if value == true || value == false
|
|
324
|
-
|
|
325
|
-
raise ArgumentError, "#{name} must be true, false, or nil"
|
|
326
|
-
end
|
|
327
|
-
private_class_method :normalize_boolean_option
|
|
328
|
-
|
|
329
297
|
def deep_stringify(value)
|
|
330
298
|
case value
|
|
331
299
|
when Hash
|
|
@@ -13,6 +13,25 @@ module ParadeDB
|
|
|
13
13
|
execute_table_function(connection, sql)
|
|
14
14
|
end
|
|
15
15
|
|
|
16
|
+
def vector_info(index, field, connection: ActiveRecord::Base.connection)
|
|
17
|
+
args = "#{connection.quote(index.to_s)}::regclass, #{connection.quote(field.to_s)}::text"
|
|
18
|
+
execute_table_function(connection, "SELECT * FROM paradedb.vector_info(#{args})")
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def vector_config(index, field, connection: ActiveRecord::Base.connection)
|
|
22
|
+
args = "#{connection.quote(index.to_s)}::regclass, #{connection.quote(field.to_s)}::text"
|
|
23
|
+
execute_table_function(connection, "SELECT * FROM paradedb.vector_config(#{args})")
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def vector_estimator_info(index, field, queries: nil, connection: ActiveRecord::Base.connection)
|
|
27
|
+
args = "#{connection.quote(index.to_s)}::regclass, #{connection.quote(field.to_s)}::text"
|
|
28
|
+
unless queries.nil?
|
|
29
|
+
vectors = queries.map { |query| connection.quote("[#{query.map { |value| Float(value) }.join(',')}]") }
|
|
30
|
+
args += ", ARRAY[#{vectors.join(', ')}]::vector[]"
|
|
31
|
+
end
|
|
32
|
+
execute_table_function(connection, "SELECT * FROM paradedb.vector_estimator_info(#{args})")
|
|
33
|
+
end
|
|
34
|
+
|
|
16
35
|
def verify_index(
|
|
17
36
|
index,
|
|
18
37
|
heapallindexed: false,
|
data/lib/parade_db/index.rb
CHANGED
|
@@ -6,16 +6,12 @@ require_relative "vector"
|
|
|
6
6
|
module ParadeDB
|
|
7
7
|
class Index
|
|
8
8
|
class << self
|
|
9
|
-
attr_writer :table_name, :
|
|
9
|
+
attr_writer :table_name, :index_name, :fields, :index_options, :where
|
|
10
10
|
|
|
11
11
|
def table_name
|
|
12
12
|
@table_name
|
|
13
13
|
end
|
|
14
14
|
|
|
15
|
-
def key_field
|
|
16
|
-
@key_field
|
|
17
|
-
end
|
|
18
|
-
|
|
19
15
|
def index_name
|
|
20
16
|
@index_name || default_index_name
|
|
21
17
|
end
|
|
@@ -83,15 +79,14 @@ module ParadeDB
|
|
|
83
79
|
# Consumed by migration helpers; validates and normalizes the DSL class
|
|
84
80
|
class DefinitionCompiler
|
|
85
81
|
FIELD_OPTION_KEYS = %i[fast record normalizer expand_dots].freeze
|
|
86
|
-
INDEX_OPTION_KEYS = %i[target_segment_count
|
|
87
|
-
POSITIVE_INTEGER_INDEX_OPTION_KEYS = %i[target_segment_count
|
|
82
|
+
INDEX_OPTION_KEYS = %i[target_segment_count training_sample_ratio max_leaf_size partition_by vector_fields search_tokenizer layer_sizes background_layer_sizes mutable_segment_rows].freeze
|
|
83
|
+
POSITIVE_INTEGER_INDEX_OPTION_KEYS = %i[target_segment_count max_leaf_size].freeze
|
|
88
84
|
|
|
89
85
|
class Compiled
|
|
90
|
-
attr_reader :table_name, :
|
|
86
|
+
attr_reader :table_name, :index_name, :entries, :index_options, :field_options, :where
|
|
91
87
|
|
|
92
|
-
def initialize(table_name:,
|
|
88
|
+
def initialize(table_name:, index_name:, entries:, index_options:, field_options:, where:)
|
|
93
89
|
@table_name = table_name
|
|
94
|
-
@key_field = key_field
|
|
95
90
|
@index_name = index_name
|
|
96
91
|
@entries = entries
|
|
97
92
|
@index_options = index_options
|
|
@@ -104,7 +99,6 @@ module ParadeDB
|
|
|
104
99
|
class << self
|
|
105
100
|
def compile!(klass)
|
|
106
101
|
table_name = require_symbol!(klass.table_name, "table_name")
|
|
107
|
-
key_field = require_symbol!(klass.key_field, "key_field")
|
|
108
102
|
index_name = klass.index_name.to_s
|
|
109
103
|
raise InvalidIndexDefinition, "index_name must be present" if index_name.strip.empty?
|
|
110
104
|
|
|
@@ -116,12 +110,10 @@ module ParadeDB
|
|
|
116
110
|
|
|
117
111
|
index_options = normalize_index_options(klass.index_options)
|
|
118
112
|
|
|
119
|
-
validate_key_field_shape!(key_field.to_s, entries)
|
|
120
113
|
validate_query_key_collisions!(entries)
|
|
121
114
|
|
|
122
115
|
Compiled.new(
|
|
123
116
|
table_name: table_name,
|
|
124
|
-
key_field: key_field,
|
|
125
117
|
index_name: index_name,
|
|
126
118
|
entries: entries,
|
|
127
119
|
index_options: index_options,
|
|
@@ -255,14 +247,29 @@ module ParadeDB
|
|
|
255
247
|
end
|
|
256
248
|
end
|
|
257
249
|
|
|
258
|
-
if normalized.
|
|
259
|
-
|
|
250
|
+
if normalized[:search_tokenizer].is_a?(ParadeDB::Tokenizer)
|
|
251
|
+
normalized[:search_tokenizer] = normalized[:search_tokenizer].render_search
|
|
252
|
+
end
|
|
253
|
+
%i[search_tokenizer layer_sizes background_layer_sizes].each do |key|
|
|
254
|
+
if normalized.key?(key) && (!normalized[key].is_a?(String) || normalized[key].strip.empty?)
|
|
255
|
+
raise InvalidIndexDefinition, "index_options[#{key.inspect}] must be a non-empty string"
|
|
256
|
+
end
|
|
257
|
+
end
|
|
258
|
+
if normalized.key?(:mutable_segment_rows) && (!normalized[:mutable_segment_rows].is_a?(Integer) || !(0..10000).cover?(normalized[:mutable_segment_rows]))
|
|
259
|
+
raise InvalidIndexDefinition, "mutable_segment_rows must be an integer between 0 and 10000"
|
|
260
|
+
end
|
|
261
|
+
if normalized.key?(:training_sample_ratio)
|
|
262
|
+
ratio = normalized[:training_sample_ratio]
|
|
260
263
|
unless ratio.is_a?(Numeric) && ratio >= 0.000001 && ratio <= 1.0
|
|
261
264
|
raise InvalidIndexDefinition,
|
|
262
|
-
"index_options[:
|
|
265
|
+
"index_options[:training_sample_ratio] must be a Numeric between 0.000001 and 1.0"
|
|
263
266
|
end
|
|
264
267
|
end
|
|
265
268
|
|
|
269
|
+
if normalized.key?(:vector_fields) && !normalized[:vector_fields].is_a?(Hash)
|
|
270
|
+
raise InvalidIndexDefinition, "index_options[:vector_fields] must be a Hash"
|
|
271
|
+
end
|
|
272
|
+
|
|
266
273
|
normalized
|
|
267
274
|
end
|
|
268
275
|
|
|
@@ -288,23 +295,7 @@ module ParadeDB
|
|
|
288
295
|
end
|
|
289
296
|
end
|
|
290
297
|
|
|
291
|
-
def validate_key_field_shape!(key_field_name, entries)
|
|
292
|
-
unless entries.any? { |entry| entry.source == key_field_name }
|
|
293
|
-
raise InvalidIndexDefinition,
|
|
294
|
-
"key_field #{key_field_name.inspect} must be present in fields."
|
|
295
|
-
end
|
|
296
298
|
|
|
297
|
-
first_entry = entries.first
|
|
298
|
-
unless first_entry.source == key_field_name
|
|
299
|
-
raise InvalidIndexDefinition,
|
|
300
|
-
"key_field #{key_field_name.inspect} must be first in fields."
|
|
301
|
-
end
|
|
302
|
-
|
|
303
|
-
return if first_entry.tokenizer.nil?
|
|
304
|
-
|
|
305
|
-
raise InvalidIndexDefinition,
|
|
306
|
-
"key_field #{key_field_name.inspect} must not be tokenized."
|
|
307
|
-
end
|
|
308
299
|
end
|
|
309
300
|
end
|
|
310
301
|
end
|
|
@@ -25,11 +25,10 @@ module ParadeDB
|
|
|
25
25
|
remember_schema_index_reference(resolved)
|
|
26
26
|
end
|
|
27
27
|
|
|
28
|
-
def add_paradedb_index(table, fields:,
|
|
28
|
+
def add_paradedb_index(table, fields:, name: nil, index_options: nil, where: nil, if_not_exists: false, concurrently: false)
|
|
29
29
|
ensure_postgresql_adapter!
|
|
30
30
|
anonymous = Class.new(ParadeDB::Index)
|
|
31
31
|
anonymous.table_name = table
|
|
32
|
-
anonymous.key_field = key_field
|
|
33
32
|
anonymous.index_name = name unless name.nil?
|
|
34
33
|
anonymous.fields = fields
|
|
35
34
|
anonymous.index_options = index_options unless index_options.nil?
|
|
@@ -86,26 +85,30 @@ module ParadeDB
|
|
|
86
85
|
prefix = if_not_exists ? "IF NOT EXISTS " : ""
|
|
87
86
|
fields_sql = compiled.entries.map { |entry| paradedb_entry_sql(entry) }.join(", ")
|
|
88
87
|
with_options_sql = paradedb_with_options_sql(compiled)
|
|
88
|
+
with_clause = with_options_sql.empty? ? "" : "\nWITH (#{with_options_sql})"
|
|
89
89
|
where_sql = compiled.where ? "\nWHERE #{compiled.where}" : ""
|
|
90
90
|
|
|
91
91
|
<<~SQL.strip.gsub(/\s+/, " ")
|
|
92
92
|
CREATE INDEX#{modifier} #{prefix}#{quote_table_name(compiled.index_name)} ON #{quote_table_name(compiled.table_name)}
|
|
93
|
-
USING paradedb (#{fields_sql})
|
|
94
|
-
WITH (#{with_options_sql})#{where_sql}
|
|
93
|
+
USING paradedb (#{fields_sql})#{with_clause}#{where_sql}
|
|
95
94
|
SQL
|
|
96
95
|
end
|
|
97
96
|
|
|
98
97
|
def paradedb_with_options_sql(compiled)
|
|
99
98
|
options = []
|
|
100
|
-
options << "key_field=#{quote(compiled.key_field.to_s)}"
|
|
101
|
-
|
|
102
99
|
compiled.index_options.each do |key, value|
|
|
103
100
|
name = key.to_sym
|
|
104
101
|
case name
|
|
105
|
-
when :target_segment_count, :
|
|
102
|
+
when :target_segment_count, :max_leaf_size, :mutable_segment_rows
|
|
106
103
|
options << "#{name}=#{Integer(value)}"
|
|
107
|
-
when :
|
|
108
|
-
options << "
|
|
104
|
+
when :partition_by
|
|
105
|
+
options << "#{name}=#{quote(value.join(","))}"
|
|
106
|
+
when :search_tokenizer, :layer_sizes, :background_layer_sizes
|
|
107
|
+
options << "#{name}=#{quote(value.to_s)}"
|
|
108
|
+
when :vector_fields
|
|
109
|
+
options << "vector_fields=#{quote(JSON.generate(value))}"
|
|
110
|
+
when :training_sample_ratio
|
|
111
|
+
options << "training_sample_ratio=#{Float(value)}"
|
|
109
112
|
else
|
|
110
113
|
raise ParadeDB::InvalidIndexDefinition, "unsupported index option #{key.inspect}"
|
|
111
114
|
end
|
|
@@ -322,12 +325,11 @@ module ParadeDB
|
|
|
322
325
|
table = row["table_name"]
|
|
323
326
|
name = row["index_name"]
|
|
324
327
|
|
|
325
|
-
key_field = extract_paradedb_key_field(indexdef)
|
|
326
328
|
index_options = extract_paradedb_index_options(row["reloptions"])
|
|
327
329
|
fields_sql = extract_paradedb_fields_sql(indexdef)
|
|
328
330
|
where = normalize_paradedb_where_clause(row["where_clause"])
|
|
329
331
|
|
|
330
|
-
if
|
|
332
|
+
if fields_sql
|
|
331
333
|
field_sqls = split_paradedb_top_level(fields_sql).map(&:strip)
|
|
332
334
|
parsed = field_sqls.map { |f| paradedb_parse_column_entry(f) }
|
|
333
335
|
|
|
@@ -350,9 +352,8 @@ module ParadeDB
|
|
|
350
352
|
end
|
|
351
353
|
|
|
352
354
|
statement = "add_paradedb_index #{table.to_sym.inspect}, " \
|
|
353
|
-
"fields: { #{fields_pairs.join(', ')} }, "
|
|
354
|
-
|
|
355
|
-
"name: #{name.inspect}"
|
|
355
|
+
"fields: { #{fields_pairs.join(', ')} }, "
|
|
356
|
+
statement += "name: #{name.inspect}"
|
|
356
357
|
unless index_options.empty?
|
|
357
358
|
statement += ", index_options: #{ruby_hash_literal(index_options)}"
|
|
358
359
|
end
|
|
@@ -363,28 +364,24 @@ module ParadeDB
|
|
|
363
364
|
end
|
|
364
365
|
end
|
|
365
366
|
|
|
366
|
-
def extract_paradedb_key_field(indexdef)
|
|
367
|
-
quoted = indexdef.match(/WITH\s*\([^)]*key_field\s*=\s*'((?:[^']|'')*)'/i)
|
|
368
|
-
return quoted[1].gsub("''", "'") if quoted
|
|
369
|
-
|
|
370
|
-
unquoted = indexdef.match(/WITH\s*\([^)]*key_field\s*=\s*([a-zA-Z_][a-zA-Z0-9_]*)/i)
|
|
371
|
-
return unquoted[1] if unquoted
|
|
372
|
-
|
|
373
|
-
nil
|
|
374
|
-
end
|
|
375
|
-
|
|
376
367
|
def extract_paradedb_index_options(reloptions)
|
|
377
368
|
paradedb_reloption_entries(reloptions).each_with_object({}) do |entry, options|
|
|
378
369
|
key, separator, value = entry.to_s.partition("=")
|
|
379
370
|
next if separator.empty?
|
|
380
371
|
|
|
381
372
|
case key
|
|
382
|
-
when "target_segment_count", "
|
|
373
|
+
when "target_segment_count", "max_leaf_size", "mutable_segment_rows"
|
|
383
374
|
parsed = Integer(value, 10, exception: false)
|
|
384
375
|
options[key.to_sym] = parsed if parsed
|
|
385
|
-
when "
|
|
376
|
+
when "partition_by"
|
|
377
|
+
options[key.to_sym] = value.split(",")
|
|
378
|
+
when "search_tokenizer", "layer_sizes", "background_layer_sizes"
|
|
379
|
+
options[key.to_sym] = value
|
|
380
|
+
when "vector_fields"
|
|
381
|
+
options[:vector_fields] = JSON.parse(value)
|
|
382
|
+
when "training_sample_ratio"
|
|
386
383
|
parsed = Float(value, exception: false)
|
|
387
|
-
options[:
|
|
384
|
+
options[:training_sample_ratio] = parsed if parsed
|
|
388
385
|
end
|
|
389
386
|
end
|
|
390
387
|
end
|
|
@@ -868,11 +865,10 @@ if defined?(ActiveRecord::Migration)
|
|
|
868
865
|
connection.replace_paradedb_index(index_klass)
|
|
869
866
|
end
|
|
870
867
|
|
|
871
|
-
def add_paradedb_index(table, fields:,
|
|
868
|
+
def add_paradedb_index(table, fields:, name: nil, index_options: nil, where: nil, if_not_exists: false, concurrently: false)
|
|
872
869
|
connection.add_paradedb_index(
|
|
873
870
|
table,
|
|
874
871
|
fields: fields,
|
|
875
|
-
key_field: key_field,
|
|
876
872
|
name: name,
|
|
877
873
|
index_options: index_options,
|
|
878
874
|
where: where,
|
data/lib/parade_db/model.rb
CHANGED
|
@@ -11,17 +11,18 @@ module ParadeDB
|
|
|
11
11
|
:paradedb_search,
|
|
12
12
|
:more_like_this,
|
|
13
13
|
:nearest,
|
|
14
|
+
:l2_distance,
|
|
15
|
+
:cosine_distance,
|
|
16
|
+
:inner_product,
|
|
14
17
|
:with_facets,
|
|
15
18
|
:facets,
|
|
16
19
|
:with_agg,
|
|
17
20
|
:facets_agg,
|
|
18
21
|
:aggregate_by,
|
|
19
|
-
:paradedb_arel,
|
|
20
22
|
:paradedb_index,
|
|
21
23
|
:paradedb_index_class,
|
|
22
24
|
:paradedb_index_classes,
|
|
23
25
|
:paradedb_indexed_fields,
|
|
24
|
-
:paradedb_key_field,
|
|
25
26
|
:paradedb_index_name,
|
|
26
27
|
:paradedb_validate_index!
|
|
27
28
|
].freeze
|
|
@@ -74,6 +75,21 @@ module ParadeDB
|
|
|
74
75
|
all.extending(SearchMethods).nearest(column, vector, metric: metric)
|
|
75
76
|
end
|
|
76
77
|
|
|
78
|
+
def l2_distance(column, vector)
|
|
79
|
+
ensure_postgres!
|
|
80
|
+
QueryBuilder.new(table_name).vector_distance(column, vector, metric: :l2)
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
def cosine_distance(column, vector)
|
|
84
|
+
ensure_postgres!
|
|
85
|
+
QueryBuilder.new(table_name).vector_distance(column, vector, metric: :cosine)
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
def inner_product(column, vector)
|
|
89
|
+
ensure_postgres!
|
|
90
|
+
QueryBuilder.new(table_name).vector_distance(column, vector, metric: :ip)
|
|
91
|
+
end
|
|
92
|
+
|
|
77
93
|
def with_facets(*fields, **opts)
|
|
78
94
|
ensure_postgres!
|
|
79
95
|
paradedb_validate_index!
|
|
@@ -108,11 +124,6 @@ module ParadeDB
|
|
|
108
124
|
)
|
|
109
125
|
end
|
|
110
126
|
|
|
111
|
-
def paradedb_arel
|
|
112
|
-
ensure_postgres!
|
|
113
|
-
@paradedb_arel ||= ParadeDB::Arel::Builder.new(table_name)
|
|
114
|
-
end
|
|
115
|
-
|
|
116
127
|
def paradedb_index(index_class)
|
|
117
128
|
@paradedb_explicit_index_classes ||= []
|
|
118
129
|
@paradedb_explicit_index_classes << index_class unless @paradedb_explicit_index_classes.include?(index_class)
|
|
@@ -143,11 +154,6 @@ module ParadeDB
|
|
|
143
154
|
definition.entries.map(&:query_key).uniq
|
|
144
155
|
end
|
|
145
156
|
|
|
146
|
-
def paradedb_key_field
|
|
147
|
-
definition = paradedb_index_definition
|
|
148
|
-
definition&.key_field
|
|
149
|
-
end
|
|
150
|
-
|
|
151
157
|
def paradedb_index_name
|
|
152
158
|
definition = paradedb_index_definition
|
|
153
159
|
definition&.index_name
|