drtom-textacular 4.0.0.alpha.20160302
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/CHANGELOG.md +199 -0
- data/Gemfile +3 -0
- data/README.md +196 -0
- data/Rakefile +61 -0
- data/lib/textacular.rb +160 -0
- data/lib/textacular/full_text_indexer.rb +66 -0
- data/lib/textacular/migration_generator.rb +31 -0
- data/lib/textacular/postgres_module_installer.rb +57 -0
- data/lib/textacular/rails.rb +14 -0
- data/lib/textacular/searchable.rb +20 -0
- data/lib/textacular/tasks.rb +23 -0
- data/lib/textacular/trigram_installer.rb +18 -0
- data/lib/textacular/version.rb +7 -0
- data/spec/config.travis.yml +8 -0
- data/spec/config.yml.example +5 -0
- data/spec/spec_helper.rb +104 -0
- data/spec/support/ar_stand_in.rb +4 -0
- data/spec/support/character.rb +7 -0
- data/spec/support/game.rb +5 -0
- data/spec/support/game_extended_with_textacular.rb +5 -0
- data/spec/support/game_extended_with_textacular_and_custom_language.rb +7 -0
- data/spec/support/game_fail.rb +3 -0
- data/spec/support/game_fail_extended_with_textacular.rb +5 -0
- data/spec/support/not_there.rb +3 -0
- data/spec/support/textacular_web_comic.rb +7 -0
- data/spec/support/web_comic.rb +7 -0
- data/spec/support/web_comic_with_searchable.rb +6 -0
- data/spec/support/web_comic_with_searchable_name.rb +6 -0
- data/spec/support/web_comic_with_searchable_name_and_author.rb +6 -0
- data/spec/textacular/full_text_indexer_spec.rb +69 -0
- data/spec/textacular/migration_generator_spec.rb +67 -0
- data/spec/textacular/searchable_spec.rb +194 -0
- data/spec/textacular/trigram_installer_spec.rb +24 -0
- data/spec/textacular_spec.rb +287 -0
- metadata +210 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 42f77ab66144c7716a06bc4e88e1810752077744
|
4
|
+
data.tar.gz: 892309c1127a3ad16b84661c5b16f597d49950b7
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 96c3abd3b0e03a58450674c632f933df4cde4cf42f4ca38053ddf184c66a846315e322b09568c2f97ef4af3d42b291e4e770b602db0d9619880f54117b2a9f71
|
7
|
+
data.tar.gz: d53292f2c1777ec2322fa6225c715bfdec0ee1aba10d6903647a12aec308dec9e1d0e71099dc05c512125c7c116b93cec24d5f2c6dc3696df2b4068634b48cda
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,199 @@
|
|
1
|
+
# Change Log
|
2
|
+
|
3
|
+
## Unreleased
|
4
|
+
|
5
|
+
## 3.2.2
|
6
|
+
|
7
|
+
* Fallback to zero similarity when trying to match null values
|
8
|
+
* Fix search term escaping
|
9
|
+
* Trigram installation migration only install extension if not installed yet.
|
10
|
+
* Update the README to illustrate how to change the similarity threshold.
|
11
|
+
* Remove deprecated dynamic search methods.
|
12
|
+
|
13
|
+
## 3.2.1
|
14
|
+
|
15
|
+
* Fix trigram installation migration reversed filename and content.
|
16
|
+
* Rewrite all tests using RSpec.
|
17
|
+
* We're ActiveRecord 4.2+ compatible until tests prove otherwise.
|
18
|
+
|
19
|
+
## 3.2.0
|
20
|
+
|
21
|
+
* Add generator for trigram installation migration.
|
22
|
+
* Expand gemspec to allow newest version of ActiveRecord.
|
23
|
+
|
24
|
+
## 3.1.0
|
25
|
+
|
26
|
+
* Avoid Deprecation warnings from ActiveRecord 4.0.0.rc2.
|
27
|
+
* Fix `method_missing` in ActiveRecord 4.0.0.rc2.
|
28
|
+
* Remove unused `Textacular#normalize` method.
|
29
|
+
* Add `OR` example to the README.
|
30
|
+
* Fix tests for Ruby 2.0.0 & related improvements.
|
31
|
+
* Improve Rails integration.
|
32
|
+
* Fix dependency loading for textacular rake tasks.
|
33
|
+
* Fix ranking failures when rows had `NULL` column values.
|
34
|
+
* Clean up Rakefile; should make developing the gem nicer.
|
35
|
+
* DEPRECATION: The dynamic search helpers will be removed at the next major
|
36
|
+
release.
|
37
|
+
|
38
|
+
## 3.0.0
|
39
|
+
|
40
|
+
* All deprecations have been resolved. This breaks backwards compatibility.
|
41
|
+
* Rename gem to Textacular.
|
42
|
+
|
43
|
+
## 2.2.0
|
44
|
+
|
45
|
+
* 1 DEPRECATION
|
46
|
+
* The whole gem is being renamed and will no longer be maintained as Texticle.
|
47
|
+
The new name it Textacular.
|
48
|
+
* 1 new feature
|
49
|
+
* Expand gemspec to allow Rails 4.
|
50
|
+
|
51
|
+
## 2.1.1
|
52
|
+
|
53
|
+
* 1 bugfix
|
54
|
+
* Include `lib/textacular/version.rb` in the gemspec so the gem will load. Sorry!
|
55
|
+
|
56
|
+
|
57
|
+
## 2.1.0
|
58
|
+
|
59
|
+
* 1 DEPRECATION
|
60
|
+
* `search` aliases new `advanced_search` method (same functionality as before), but will
|
61
|
+
alias `basic_search` in 3.0! Should print warnings.
|
62
|
+
* 3 new features
|
63
|
+
* Generate full text search indexes from a rake task (sort of like in 1.x). Supply a specific
|
64
|
+
model name.
|
65
|
+
* New search methods: `basic_search`, `advanced_search` and `fuzzy_search`. Basic allows special
|
66
|
+
characters like &, and % in search terms. Fuzzy is based on Postgres's trigram matching extension
|
67
|
+
pg_trgm. Advanced is the same functionality from `search` previously.
|
68
|
+
* Rake task that installs pg_trgm now works on Postgres 9.1 and up.
|
69
|
+
* 2 dev improvements
|
70
|
+
* Test database configuration not automatically generated from a rake task and ignored by git.
|
71
|
+
* New interactive developer console (powered by pry).
|
72
|
+
|
73
|
+
|
74
|
+
## 2.0.3
|
75
|
+
|
76
|
+
* 1 new feature
|
77
|
+
* Allow searching through relations. Model.join(:relation).search(:relation => {:column => "query"})
|
78
|
+
works, and reduces the need for multi-model tables. Huge thanks to Ben Hamill for the pull request.
|
79
|
+
* Allow searching through all model columns irrespective of the column's type; we cast all columns to text
|
80
|
+
in the search query. Performance may degrade when searching through anything but a string column.
|
81
|
+
* 2 bugfixes
|
82
|
+
* Fix exceptions when adding Textacular to a table-less model.
|
83
|
+
* Column names in a search query are now scoped to the current table.
|
84
|
+
* 1 dev improvement
|
85
|
+
* Running `rake` from the project root will setup the test environment by creating a test database
|
86
|
+
and running the necessary migrations. `rake` can also be used to run all the project tests.
|
87
|
+
|
88
|
+
|
89
|
+
## 2.0.2
|
90
|
+
|
91
|
+
* 1 bugfix
|
92
|
+
* Our #respond_to? overwritten method was causing failures when a model didn't have
|
93
|
+
a table (e.g. if migrations hadn't been run yet). Not the case anymore.
|
94
|
+
|
95
|
+
|
96
|
+
## 2.0.1
|
97
|
+
|
98
|
+
* 1 new feature
|
99
|
+
* Can now define #searchable_language to specify the language used for the query. This changes
|
100
|
+
what's considered a stop word on Postgres' side. 'english' is the default language.
|
101
|
+
* 1 bugfix
|
102
|
+
* We were only specifying a language in to_tsvector() and not in to_tsquery(), which could
|
103
|
+
cause queries to fail if the default database language wasn't set to 'english'.
|
104
|
+
|
105
|
+
|
106
|
+
## 2.0.pre4
|
107
|
+
|
108
|
+
* 1 new feature
|
109
|
+
* Searchable is now available to specify which columns you want searched:
|
110
|
+
|
111
|
+
```ruby
|
112
|
+
require 'textacular/searchable'
|
113
|
+
class Game
|
114
|
+
extend Searchable(:title)
|
115
|
+
end
|
116
|
+
```
|
117
|
+
|
118
|
+
This also allows Textacular use in Rails without having #search available to all models:
|
119
|
+
|
120
|
+
```
|
121
|
+
gem 'textacular', '~> 2.0.pre4', :require => 'textacular/searchable'
|
122
|
+
```
|
123
|
+
* 1 bugfix
|
124
|
+
* ActiveRecord::Base.extend(Textacular) doesn't break #method_missing and #respond_to? anymore
|
125
|
+
|
126
|
+
|
127
|
+
## 2.0.pre3
|
128
|
+
|
129
|
+
* 1 new feature
|
130
|
+
* #select calls now limit the columns that are searched
|
131
|
+
* 1 bugfix
|
132
|
+
* #search calls without an argument assume an empty string as a search term (it errored out previously)
|
133
|
+
|
134
|
+
|
135
|
+
## 2.0.pre2
|
136
|
+
|
137
|
+
* 1 bugfix
|
138
|
+
* #respond_to? wasn't overwritten correctly
|
139
|
+
|
140
|
+
## 2.0.pre
|
141
|
+
|
142
|
+
* Complete refactoring of Textacular
|
143
|
+
* For users:
|
144
|
+
* Textacular should only be used for its simplicity; if you need to deeply configure your text search, please give `gem install pg_search` a try.
|
145
|
+
* `#search` method is now included in all ActiveRecord models by default, and searches across a model's :string columns.
|
146
|
+
* `#search_by_<column>` dynamic methods are now available.
|
147
|
+
* `#search` can now be chained; `Game.search_by_title("Street Fighter").search_by_system("PS3")` works.
|
148
|
+
* `#search` now accepts a hash to specify columns to be searched, e.g. `Game.search(:name => "Mario")`
|
149
|
+
* No more access to `#rank` values for results (though they're still ordered by rank).
|
150
|
+
* No way to give different weights to different columns in this release.
|
151
|
+
* For devs:
|
152
|
+
* We now have actual tests to run against; this will make accepting pull requests much more enjoyable.
|
153
|
+
|
154
|
+
|
155
|
+
## HEAD (unreleased)
|
156
|
+
|
157
|
+
* 1 minor bugfix
|
158
|
+
* Multiple named indices are now supported.
|
159
|
+
|
160
|
+
|
161
|
+
## 1.0.4 / 2010-08-19
|
162
|
+
|
163
|
+
* 2 major enhancements
|
164
|
+
* use Rails.root instead of RAILS_ROOT
|
165
|
+
* refactored tasks to ease maintainance and patchability
|
166
|
+
* 3 minor enhancements
|
167
|
+
* fix timestamp for migrationfile
|
168
|
+
* fixed deprecation warning for rails3 (dropping rails2-support)
|
169
|
+
* prevented warning about defined constant
|
170
|
+
|
171
|
+
|
172
|
+
## 1.0.3 / 2010-07-07
|
173
|
+
|
174
|
+
* 1 major enhancement
|
175
|
+
* Added Rails 3 support.
|
176
|
+
* 1 bugfix
|
177
|
+
* Model names that end in double 's's (like Address) don't choke the rake tasks anymore.
|
178
|
+
|
179
|
+
|
180
|
+
## 1.0.2 / 2009-10-17
|
181
|
+
|
182
|
+
* 1 bugfix
|
183
|
+
* Generated migration now uses UTC time rather than local time.
|
184
|
+
|
185
|
+
|
186
|
+
## 1.0.1 / 2009-04-14
|
187
|
+
|
188
|
+
* 1 minor enhancement
|
189
|
+
* Textical adds a rake task to generate FTS index migrations. Just run:
|
190
|
+
|
191
|
+
```
|
192
|
+
rake textical:migration
|
193
|
+
```
|
194
|
+
|
195
|
+
|
196
|
+
## 1.0.0 / 2009-04-14
|
197
|
+
|
198
|
+
* 1 major enhancement
|
199
|
+
* Birthday!
|
data/Gemfile
ADDED
data/README.md
ADDED
@@ -0,0 +1,196 @@
|
|
1
|
+
# textacular
|
2
|
+
[![Gem Version](http://img.shields.io/gem/v/textacular.svg)][rubygems]
|
3
|
+
[![Build Status](https://img.shields.io/travis/textacular/textacular/master.svg)][travis]
|
4
|
+
[![Code Climate](https://img.shields.io/codeclimate/github/textacular/textacular.svg)][codeclimate]
|
5
|
+
|
6
|
+
[rubygems]: http://rubygems.org/gems/textacular
|
7
|
+
[travis]: https://travis-ci.org/textacular/textacular
|
8
|
+
[codeclimate]: https://codeclimate.com/github/textacular/textacular
|
9
|
+
|
10
|
+
Further documentation available at http://textacular.github.com/textacular.
|
11
|
+
|
12
|
+
|
13
|
+
## DESCRIPTION:
|
14
|
+
|
15
|
+
Textacular exposes full text search capabilities from PostgreSQL,
|
16
|
+
extending ActiveRecord with scopes making search easy and fun!
|
17
|
+
|
18
|
+
|
19
|
+
## FEATURES/PROBLEMS:
|
20
|
+
|
21
|
+
* Only works with PostgreSQL
|
22
|
+
|
23
|
+
|
24
|
+
## SYNOPSIS:
|
25
|
+
|
26
|
+
### Quick Start
|
27
|
+
|
28
|
+
#### Rails 3 (or 4!)
|
29
|
+
|
30
|
+
In the project's Gemfile add
|
31
|
+
|
32
|
+
```ruby
|
33
|
+
gem 'textacular', '~> 3.0'
|
34
|
+
```
|
35
|
+
|
36
|
+
#### ActiveRecord outside of Rails
|
37
|
+
|
38
|
+
```ruby
|
39
|
+
require 'textacular'
|
40
|
+
|
41
|
+
ActiveRecord::Base.extend(Textacular)
|
42
|
+
```
|
43
|
+
|
44
|
+
|
45
|
+
### Usage
|
46
|
+
|
47
|
+
Your models now have access to search methods:
|
48
|
+
|
49
|
+
The `#basic_search` method is what you might expect: it looks literally for what
|
50
|
+
you send to it, doing nothing fancy with the input:
|
51
|
+
|
52
|
+
```ruby
|
53
|
+
Game.basic_search('Sonic') # will search through the model's :string columns
|
54
|
+
Game.basic_search(title: 'Mario', system: 'Nintendo')
|
55
|
+
```
|
56
|
+
|
57
|
+
The `#advanced_search` method lets you use Postgres's search syntax like '|',
|
58
|
+
'&' and '!' ('or', 'and', and 'not') as well as some other craziness. The ideal
|
59
|
+
use for advanced_search is to take a search DSL you make up for your users and
|
60
|
+
translate it to PG's syntax. If for some reason you want to put user input
|
61
|
+
directly into an advanced search, you should be sure to catch exceptions from
|
62
|
+
syntax errors. Check [the Postgres docs]
|
63
|
+
(http://www.postgresql.org/docs/9.2/static/datatype-textsearch.html) for more:
|
64
|
+
|
65
|
+
```ruby
|
66
|
+
Game.advanced_search(title: 'Street|Fantasy')
|
67
|
+
Game.advanced_search(system: '!PS2')
|
68
|
+
```
|
69
|
+
|
70
|
+
Finally, the `#fuzzy_search` method lets you use Postgres's trigram search
|
71
|
+
functionality.
|
72
|
+
|
73
|
+
In order to use this, you'll need to make sure your database has the `pg_trgm`
|
74
|
+
module installed. Create and run a migration to install the module:
|
75
|
+
|
76
|
+
```
|
77
|
+
rake textacular:create_trigram_migration
|
78
|
+
rake db:migrate
|
79
|
+
```
|
80
|
+
|
81
|
+
Once that's installed, you can use it like this:
|
82
|
+
|
83
|
+
```ruby
|
84
|
+
Comic.fuzzy_search(title: 'Questio') # matches Questionable Content
|
85
|
+
```
|
86
|
+
|
87
|
+
Note that fuzzy searches are subject to a similarity threshold imposed by the `pg_trgm` module. The default is 0.3, meaning that at least 30% of the total string must match your search content. For example:
|
88
|
+
|
89
|
+
```ruby
|
90
|
+
Comic.fuzzy_search(title: 'Pearls') # matches Pearls Before Swine
|
91
|
+
Comic.fuzzy_search(title: 'Pear') # does not match Pearls Before Swine
|
92
|
+
```
|
93
|
+
|
94
|
+
The similarity threshold is hardcoded in PostgreSQL and can be modified on a per-connection basis, for example:
|
95
|
+
|
96
|
+
```ruby
|
97
|
+
ActiveRecord::Base.connection.execute("SELECT set_limit(0.9);")
|
98
|
+
```
|
99
|
+
|
100
|
+
For more info, view the `pg_trgm` documentation, specifically [F.35.2. Functions and Operators](http://www.postgresql.org/docs/9.1/static/pgtrgm.html).
|
101
|
+
|
102
|
+
Searches are also chainable:
|
103
|
+
|
104
|
+
```ruby
|
105
|
+
Game.fuzzy_search(title: 'tree').basic_search(system: 'SNES')
|
106
|
+
```
|
107
|
+
|
108
|
+
If you want to search on two or more fields with the OR operator use a hash for
|
109
|
+
the conditions and pass false as the second parameter:
|
110
|
+
|
111
|
+
```ruby
|
112
|
+
Game.basic_search({name: 'Mario', nickname: 'Mario'}, false)
|
113
|
+
```
|
114
|
+
|
115
|
+
|
116
|
+
### Setting Language
|
117
|
+
|
118
|
+
To set proper searching dictionary just override class method on your model:
|
119
|
+
|
120
|
+
```ruby
|
121
|
+
def self.searchable_language
|
122
|
+
'russian'
|
123
|
+
end
|
124
|
+
```
|
125
|
+
|
126
|
+
And all your queries would go right! And don`t forget to change the migration for indexes, like shown below.
|
127
|
+
|
128
|
+
|
129
|
+
### Creating Indexes for Super Speed
|
130
|
+
You can have Postgresql use an index for the full-text search. To declare a full-text index, in a
|
131
|
+
migration add code like the following:
|
132
|
+
|
133
|
+
```ruby
|
134
|
+
execute "
|
135
|
+
create index on email_logs using gin(to_tsvector('english', subject));
|
136
|
+
create index on email_logs using gin(to_tsvector('english', email_address));"
|
137
|
+
```
|
138
|
+
|
139
|
+
In the above example, the table email_logs has two text columns that we search against, subject and email_address.
|
140
|
+
You will need to add an index for every text/string column you query against, or else Postgresql will revert to a
|
141
|
+
full table scan instead of using the indexes.
|
142
|
+
|
143
|
+
If you create these indexes, you should also switch to sql for your schema_format in `config/application.rb`:
|
144
|
+
|
145
|
+
```ruby
|
146
|
+
config.active_record.schema_format = :sql
|
147
|
+
```
|
148
|
+
|
149
|
+
|
150
|
+
## REQUIREMENTS:
|
151
|
+
|
152
|
+
* ActiveRecord
|
153
|
+
* Ruby 1.9.2
|
154
|
+
|
155
|
+
|
156
|
+
## INSTALL:
|
157
|
+
|
158
|
+
```
|
159
|
+
$ gem install textacular
|
160
|
+
```
|
161
|
+
|
162
|
+
## Contributing
|
163
|
+
|
164
|
+
If you'd like to contribute, please see the [contribution guidelines](CONTRIBUTING.md).
|
165
|
+
|
166
|
+
|
167
|
+
## Releasing
|
168
|
+
|
169
|
+
Maintainers: Please make sure to follow the [release steps](RELEASING.md) when
|
170
|
+
it's time to cut a new release.
|
171
|
+
|
172
|
+
|
173
|
+
## LICENSE:
|
174
|
+
|
175
|
+
(The MIT License)
|
176
|
+
|
177
|
+
Copyright (c) 2011 Aaron Patterson
|
178
|
+
|
179
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
180
|
+
a copy of this software and associated documentation files (the
|
181
|
+
'Software'), to deal in the Software without restriction, including
|
182
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
183
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
184
|
+
permit persons to whom the Software is furnished to do so, subject to
|
185
|
+
the following conditions:
|
186
|
+
|
187
|
+
The above copyright notice and this permission notice shall be
|
188
|
+
included in all copies or substantial portions of the Software.
|
189
|
+
|
190
|
+
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
|
191
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
192
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
|
193
|
+
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
|
194
|
+
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
|
195
|
+
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
|
196
|
+
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/Rakefile
ADDED
@@ -0,0 +1,61 @@
|
|
1
|
+
require "bundler/gem_tasks"
|
2
|
+
|
3
|
+
require 'active_record'
|
4
|
+
require 'pry'
|
5
|
+
|
6
|
+
require 'rspec/core/rake_task'
|
7
|
+
|
8
|
+
RSpec::Core::RakeTask.new(:spec)
|
9
|
+
|
10
|
+
task :default => :spec
|
11
|
+
|
12
|
+
file 'spec/config.yml' do |t|
|
13
|
+
sh 'erb spec/config.yml.example > spec/config.yml'
|
14
|
+
end
|
15
|
+
|
16
|
+
desc 'Fire up an interactive terminal to play with'
|
17
|
+
task :console => :'db:connect' do
|
18
|
+
Pry.start
|
19
|
+
end
|
20
|
+
|
21
|
+
namespace :db do
|
22
|
+
|
23
|
+
task :connect => 'spec/config.yml' do |t|
|
24
|
+
ActiveRecord::Base.establish_connection \
|
25
|
+
YAML.load_file 'spec/config.yml'
|
26
|
+
end
|
27
|
+
|
28
|
+
task :disconnect do
|
29
|
+
ActiveRecord::Base.clear_all_connections!
|
30
|
+
end
|
31
|
+
|
32
|
+
desc 'Create the test database'
|
33
|
+
task :create do
|
34
|
+
sh 'createdb textacular'
|
35
|
+
end
|
36
|
+
|
37
|
+
desc 'Drop the test database'
|
38
|
+
task :drop => :disconnect do
|
39
|
+
sh 'dropdb textacular'
|
40
|
+
end
|
41
|
+
|
42
|
+
namespace :migrate do
|
43
|
+
|
44
|
+
desc 'Run the test database migrations'
|
45
|
+
task :up => :'db:connect' do
|
46
|
+
ActiveRecord::Migrator.up 'db/migrate'
|
47
|
+
end
|
48
|
+
|
49
|
+
desc 'Reverse the test database migrations'
|
50
|
+
task :down => :'db:connect' do
|
51
|
+
ActiveRecord::Migrator.down 'db/migrate'
|
52
|
+
end
|
53
|
+
end
|
54
|
+
task :migrate => :'migrate:up'
|
55
|
+
|
56
|
+
desc 'Create and configure the test database'
|
57
|
+
task :setup => [ :create, :migrate ]
|
58
|
+
|
59
|
+
desc 'Drop the test tables and database'
|
60
|
+
task :teardown => [ :'migrate:down', :drop ]
|
61
|
+
end
|