slingshot-rb 0.0.8 → 0.0.9

data/.gitignore

@@ -6,3 +6,4 @@ rdoc/
 coverage/
 scratch/
 examples/*.html
+*.log

data/README.markdown

@@ -4,26 +4,32 @@ Slingshot
 ![Slingshot](https://github.com/karmi/slingshot/raw/master/slingshot.png)
 
 _Slingshot_ is a Ruby client for the [ElasticSearch](http://www.elasticsearch.org/) search engine/database.
-It aims to provide rich and comfortable Ruby API in the form of a simple domain-specific language.
 
 _ElasticSearch_ is a scalable, distributed, cloud-ready, highly-available,
-RESTful database communicating by JSON over HTTP, based on [Lucene](http://lucene.apache.org/),
-written in Java. It manages to be very simple to use and very powerful at the same time.
+full-text search engine and database, communicating by JSON over RESTful HTTP,
+based on [Lucene](http://lucene.apache.org/), written in Java.
+
+This document provides just a brief overview of _Slingshot's_ features. Be sure to check out also
+the extensive documentation at <http://karmi.github.com/slingshot/> if you're interested.
 
 Installation
 ------------
 
 First, you need a running _ElasticSearch_ server. Thankfully, it's easy. Let's define easy:
 
-    $ curl -k -L -o elasticsearch-0.15.2.tar.gz http://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.15.2.tar.gz
-    $ tar -zxvf elasticsearch-0.15.2.tar.gz
-    $ ./elasticsearch-0.15.2/bin/elasticsearch -f
+    $ curl -k -L -o elasticsearch-0.16.0.tar.gz http://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.16.0.tar.gz
+    $ tar -zxvf elasticsearch-0.16.0.tar.gz
+    $ ./elasticsearch-0.16.0/bin/elasticsearch -f
+
+OK. Easy. On a Mac, you can also use _Homebrew_:
+
+    $ brew install elasticsearch
 
-OK, easy. Now, install the gem via Rubygems:
+OK. Let's install the gem via Rubygems:
 
     $ gem install slingshot-rb
 
-or from source:
+Of course, you can install it from the source as well:
 
     $ git clone git://github.com/karmi/slingshot.git
     $ cd slingshot
@@ -33,17 +39,20 @@ or from source:
 Usage
 -----
 
-Currently, you can use _Slingshot_ via the DSL (eg. by extending your class with it).
-Plans for full ActiveModel integration (and other convenience layers) are in progress
-(see the [`activemodel`](https://github.com/karmi/slingshot/compare/activemodel) branch).
+_Slingshot_ exposes easy-to-use domain specific language for fluent communication with _ElasticSearch_.
 
-To kick the tires, require the gem in an IRB session or a Ruby script
-(note that you can just run the full example from [`examples/dsl.rb`](https://github.com/karmi/slingshot/blob/master/examples/dsl.rb)):
+It also blends with your [ActiveModel](https://github.com/rails/rails/tree/master/activemodel)
+classes for convenient usage in Rails applications.
+
+To test-drive the core _ElasticSearch_ functionality, let's require the gem:
 
     require 'rubygems'
     require 'slingshot'
 
-First, let's create an index named `articles` and store/index some documents:
+Please note that you can copy these snippets from the much more extensive and heavily annotated file
+in [examples/slingshot-dsl.rb](http://karmi.github.com/slingshot/).
+
+OK. Let's create an index named `articles` and store/index some documents:
 
     Slingshot.index 'articles' do
       delete
@@ -57,8 +66,9 @@ First, let's create an index named `articles` and store/index some documents:
       refresh
     end
 
-We can also create the
-index with specific [mapping](http://www.elasticsearch.org/guide/reference/api/admin-indices-create-index.html):
+We can also create the index with custom
+[mapping](http://www.elasticsearch.org/guide/reference/api/admin-indices-create-index.html)
+for a specific document type:
 
     Slingshot.index 'articles' do
       create :mappings => {
@@ -73,9 +83,32 @@ index with specific [mapping](http://www.elasticsearch.org/guide/reference/api/a
       }
     end
 
-Now, let's query the database.
+Of course, we may have large amounts of data, and it may be impossible or impractical to add them to the index
+one by one. We can use _ElasticSearch's_ [bulk storage](http://www.elasticsearch.org/guide/reference/api/bulk.html):
+
+    articles = [
+      { :id => '1', :title => 'one'   },
+      { :id => '2', :title => 'two'   },
+      { :id => '3', :title => 'three' }
+    ]
+
+    Slingshot.index 'bulk' do
+      import articles
+    end
+
+We can also easily manipulate the documents before storing them in the index, by passing a block to the
+`import` method:
+
+    Slingshot.index 'bulk' do
+      import articles do |documents|
+
+        documents.each { |document| document[:title].capitalize! }
+      end
+    end
+
+OK. Now, let's go search all the data.
 
-We are searching for articles whose `title` begins with letter “T”, sorted by `title` in `descending` order,
+We will be searching for articles whose `title` begins with letter “T”, sorted by `title` in `descending` order,
 filtering them for ones tagged “ruby”, and also retrieving some [_facets_](http://www.elasticsearch.org/guide/reference/api/search/facets/)
 from the database:
 
@@ -97,6 +130,9 @@ from the database:
       end
     end
 
+(Of course, we may also page the results with `from` and `size` query options, retrieve only specific fields
+or highlight content matching our query, etc.)
+
 Let's display the results:
 
     s.results.each do |document|
@@ -128,70 +164,260 @@ count for articles tagged 'php' is excluded, since they don't match the current
     # python     1
     # java       1
 
-We can display the full query JSON:
+If configuring the search payload with a block somehow feels too weak for you, you can simply pass
+a Ruby `Hash` (or JSON string) with the query declaration to the `search` method:
+
+    Slingshot.search 'articles', :query => { :fuzzy => { :title => 'Sour' } }
+
+If this sounds like a great idea to you, you are probably able to write your application
+using just `curl`, `sed` and `awk`.
+
+We can display the full query JSON for close inspection:
 
     puts s.to_json
     # {"facets":{"current-tags":{"terms":{"field":"tags"}},"global-tags":{"global":true,"terms":{"field":"tags"}}},"query":{"query_string":{"query":"title:T*"}},"filter":{"terms":{"tags":["ruby"]}},"sort":[{"title":"desc"}]}
 
-Or, better, we can display the corresponding `curl` command for easy debugging:
+Or, better, we can display the corresponding `curl` command to recreate and debug the request in the terminal:
 
     puts s.to_curl
     # curl -X POST "http://localhost:9200/articles/_search?pretty=true" -d '{"facets":{"current-tags":{"terms":{"field":"tags"}},"global-tags":{"global":true,"terms":{"field":"tags"}}},"query":{"query_string":{"query":"title:T*"}},"filter":{"terms":{"tags":["ruby"]}},"sort":[{"title":"desc"}]}'
 
-Since `curl` is the crucial debugging tool in _ElasticSearch_ land, we can log every search query in `curl` format:
+However, we can simply log every search query (and other requests) in this `curl`-friendly format:
 
     Slingshot.configure { logger 'elasticsearch.log' }
 
+When you set the log level to _debug_:
 
-Features
---------
+    Slingshot.configure { logger 'elasticsearch.log', :level => 'debug' }
+
+the JSON responses are logged as well. This is not a great idea for production environment,
+but it's priceless when you want to paste a complicated transaction to the mailing list or IRC channel.
+
+The _Slingshot_ DSL tries hard to provide a strong Ruby-like API for the main _ElasticSearch_ features.
+
+By default, _Slingshot_ wraps the results collection in a enumerable `Results::Collection` class,
+and result items in a `Results::Item` class, which looks like a child of `Hash` and `Openstruct`,
+for smooth iterating and displaying the results.
+
+You may wrap the result items in your own class by setting the `Slingshot.configuration.wrapper`
+property. Your class must take a `Hash` of attributes on initialization.
+
+If that seems like a great idea to you, there's a big chance you already have such class, and one would bet
+it's an `ActiveRecord` or `ActiveModel` class, containing model of your Rails application.
+
+Fortunately, _Slingshot_ makes blending _ElasticSearch_ features into your models trivially possible.
+
+
+ActiveModel Integration
+-----------------------
+
+Let's suppose you have an `Article` class in your Rails application. To make it searchable with
+_Slingshot_, you just `include` it:
+
+    class Article < ActiveRecord::Base
+      include Slingshot::Model::Search
+      include Slingshot::Model::Callbacks
+    end
+
+When you now save a record:
+
+    Article.create :title =>   "I Love ElasticSearch",
+                   :content => "...",
+                   :author =>  "Captain Nemo",
+                   :published_on => Time.now
+
+it is automatically added into the index, because of the included callbacks. The document attributes
+are indexed exactly as when you call the `Article#to_json` method.
+
+Now you can search the records:
+
+    Article.search 'love'
+
+OK. Often, this is where the game stops. Not here.
+
+First of all, you may use the full query DSL, as explained above, with filters, sorting,
+advanced facet aggregation, highlighting, etc:
+
+    q = 'love'
+    Article.search do
+      query { string q }
+      facet('timeline') { date :published_on, :interval => 'month' }
+      sort  { published_on 'desc' }
+    end
+
+Dynamic mapping is a godsend when you're prototyping.
+For serious usage, though, you'll definitely want to define a custom mapping for your model:
+
+    class Article < ActiveRecord::Base
+      include Slingshot::Model::Search
+      include Slingshot::Model::Callbacks
+
+      mapping do
+        indexes :id,           :type => 'string',  :analyzed => false
+        indexes :title,        :type => 'string',  :analyzer => 'snowball', :boost => 100
+        indexes :content,      :type => 'string',  :analyzer => 'snowball'
+        indexes :author,       :type => 'string',  :analyzer => 'keyword'
+        indexes :published_on, :type => 'date',    :include_in_all => false
+      end
+    end
 
-Currently, _Slingshot_ supports main features of the _ElasticSearch_ [Search API](http://www.elasticsearch.org/guide/reference/api/search/request-body.html) and it's [Query DSL](http://www.elasticsearch.org/guide/reference/query-dsl/). In present, it allows you to:
+In this case, _only_ the defined model attributes are indexed when adding to the index.
 
-* Create, delete and refresh the index
-* Create the index with specific [mapping](http://www.elasticsearch.org/guide/reference/api/admin-indices-create-index.html)
-* Store a document in the index
-* [Query](https://github.com/karmi/slingshot/blob/master/examples/dsl.rb) the index with the `query_string`, `term`, `terms` and `match_all` types of queries
-* [Sort](http://elasticsearch.org/guide/reference/api/search/sort.html) the results by `fields`
-* [Filter](http://elasticsearch.org/guide/reference/query-dsl/) the results
-* Retrieve the _terms_ and _date histogram_ types of [facets](http://www.elasticsearch.org/guide/reference/api/search/facets/index.html) (other types are high priority)
-* [Highlight](http://www.elasticsearch.org/guide/reference/api/search/highlighting.html) matching fields
-* Return just specific `fields` from documents
-* Page the results with `from` and `size` query options
-* Log the `curl`-equivalent of requests and response JSON
+When you want tight grip on how your model attributes are added to the index, just
+provide the `to_indexed_json` method yourself:
 
-See the [`examples/slingshot-dsl.rb`](blob/master/examples/slingshot-dsl.rb) file for the full, working examples.
+    class Article < ActiveRecord::Base
+      include Slingshot::Model::Search
+      include Slingshot::Model::Callbacks
 
-_Slingshot_ wraps the results in a enumerable `Results::Collection` class, and every result in a `Results::Item` class,
-which looks like a child of `Hash` and `Openstruct`, for smooth iterating and displaying the results.
+      def to_indexed_json
+        names      = author.split(/\W/)
+        last_name  = names.pop
+        first_name = names.join
 
-You may wrap the result items in your own class just by setting the `Configuration.wrapper` property,
-supposed your class takes a hash of attributes upon initialization, in ActiveModel/ActiveRecord manner.
-Please see the files `test/models/article.rb` and `test/unit/results_collection_test.rb` for details.
+        {
+          :title   => title,
+          :content => content,
+          :author  => {
+            :first_name => first_name,
+            :last_name  => last_name
+          }
+        }.to_json
+      end
+
+    end
+
+Note that _Slingshot_-enhanced models are fully compatible with [`will_paginate`](https://github.com/mislav/will_paginate),
+so you can pass any parameters to the `search` method in the controller, as usual:
+
+    @articles = Article.search params[:q], :page => (params[:page] || 1)
+
+OK. Chances are, you have lots of records stored in the underlying database. How will you get them to _ElasticSearch_? Easy:
+
+    Article.index.import Article.all
+
+However, this way, all your records are loaded into memory, serialized into JSON,
+and sent down the wire to _ElasticSearch_. Not practical, you say? You're right.
+
+Provided your model implements some sort of _pagination_ — and it probably does, for so much data —,
+you can just run:
+
+    Article.import
+
+In this case, the `Article.paginate` method is called, and your records are sent to the index
+in chunks of 1000. If that number doesn't suit you, just provide a better one:
+
+    Article.import :per_page => 100
+
+Any other parameters you provide to the `import` method are passed down to the `paginate` method.
+
+Are we saying you have to fiddle with this thing in a `rails console` or silly Ruby scripts? No.
+Just call the included _Rake_ task on the commandline:
+
+    $ rake environment slingshot:import CLASS='Article'
+
+You can also force-import the data by deleting the index first (and creating it with mapping
+provided by the `mapping` block in your model):
+
+    $ rake environment slingshot:import CLASS='Article' FORCE=true
+
+When you'll spend more time with _ElasticSearch_, you'll notice how
+[index aliases](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)
+are the best idea since the invention of inverted index.
+You can index your data into a fresh index (and possibly update an alias if everything's fine):
+
+    $ rake environment slingshot:import CLASS='Article' INDEX='articles-2011-05'
+
+If you're the type who has no time for long introductions, you can generate a fully working
+example Rails application, with an `ActiveRecord` model and a search form, to play with:
+
+    $ rails new searchapp -m https://github.com/karmi/slingshot/raw/master/examples/rails-application-template.rb
+
+OK. All this time we have been talking about `ActiveRecord` models, since
+it is a reasonable Rails' default for the storage layer.
+
+But what if you use another database such as [MongoDB](http://www.mongodb.org/),
+another object mapping library, such as [Mongoid](http://mongoid.org/)?
+
+Well, things stay mostly the same:
+
+    class Article
+      include Mongoid::Document
+      field :title, :type => String
+      field :content, :type => String
+
+      include Slingshot::Model::Search
+      include Slingshot::Model::Callbacks
+
+      # Let's use a different index name so stuff doesn't get mixed up
+      #
+      index_name 'mongo-articles'
+
+      # These Mongo guys sure do some funky stuff with their IDs
+      # in +serializable_hash+, let's fix it.
+      #
+      def to_indexed_json
+        self.to_json
+      end
+
+    end
+
+    Article.create :title => 'I Love ElasticSearch'
+
+    Article.search 'love'
+
+That's kinda nice. But there's more.
+
+_Slingshot_ implements not only _searchable_ features, but also _persistence_ features.
+
+This means that you can use a _Slingshot_ model **instead of** your database, not just
+for searching your database. Why would you like to do that?
+
+Well, because you're tired of database migrations and lots of hand-holding with your
+database to store stuff like `{ :name => 'Slingshot', :tags => [ 'ruby', 'search' ] }`.
+Because what you need is to just dump a JSON-representation of your data into a database and
+load it back when needed.
+Because you've noticed that _searching_ your data is a much more effective way of retrieval
+then constructing elaborate database query conditions.
+Because you have _lots_ of data and want to use _ElasticSearch's_
+advanced distributed features.
+
+To use the persistence features, you have to include the `Slingshot::Persistence` module
+in your class and define the properties (analogous to the way you do with CouchDB- or MongoDB-based models):
+
+    class Article
+      include Slingshot::Model::Persistence
+      include Slingshot::Model::Search
+      include Slingshot::Model::Callbacks
+
+      validates_presence_of :title, :author
+
+      property :title
+      property :author
+      property :content
+      property :published_on
+
+    end
 
+Of course, not all validations or `ActionPack` helpers will be available to your models,
+but if you can live with that, you've just got a schema-free, highly-scalable storage
+and retrieval engine for your data.
 
 Todo, Plans & Ideas
 -------------------
 
-_Slingshot_ is already used in production by its authors. Nevertheless, it's not finished yet.
+_Slingshot_ is already used in production by its authors. Nevertheless, it's not considered finished yet.
 
-The todos and plans are vast, and the most important are listed below, in the order of importance:
+There are todos, plans and ideas, some of which are listed below, in the order of importance:
 
-* Seamless _ActiveModel_ compatibility for easy usage in _Rails_ applications (this also means nearly full _ActiveRecord_ compatibility). See the ongoing work in the [`activemodel`](https://github.com/karmi/slingshot/compare/activemodel) branch
-* Seamless [will_paginate](https://github.com/mislav/will_paginate) compatibility for easy pagination. Already [implemented](https://github.com/karmi/slingshot/commit/e1351f6) on the `activemodel` branch
-* [Mapping](http://www.elasticsearch.org/guide/reference/mapping/) definition for models
 * Proper RDoc annotations for the source code
-* Dual interface: allow to simply pass queries/options for _ElasticSearch_ as a Hash in any method
 * [Histogram](http://www.elasticsearch.org/guide/reference/api/search/facets/histogram-facet.html) facets
 * [Statistical](http://www.elasticsearch.org/guide/reference/api/search/facets/statistical-facet.html) facets
 * [Geo Distance](http://www.elasticsearch.org/guide/reference/api/search/facets/geo-distance-facet.html) facets
 * [Index aliases](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html) management
 * [Analyze](http://www.elasticsearch.org/guide/reference/api/admin-indices-analyze.html) API support
-* [Bulk](http://www.elasticsearch.org/guide/reference/api/bulk.html) API
 * Embedded webserver to display statistics and to allow easy searches
-* Seamless support for [auto-updating _river_ index](http://www.elasticsearch.org/guide/reference/river/couchdb.html) for _CouchDB_ `_changes` feed
 
-The full ActiveModel integration is planned for the 1.0 release.
 
 Other Clients
 -------------

data/examples/rails-application-template.rb

@@ -0,0 +1,144 @@
+# ===================================================================================================================
+# Template for generating a no-frills Rails application with support for ElasticSearch full-text search via Slingshot
+# ===================================================================================================================
+#
+# This file creates a basic Rails application with support for ElasticSearch full-text via the Slingshot gem
+#
+# Run it like this:
+#
+#     rails new searchapp -m https://github.com/karmi/slingshot/raw/master/examples/rails-application-template.rb
+#
+
+run "rm public/index.html"
+run "rm public/images/rails.png"
+run "touch tmp/.gitignore log/.gitignore vendor/.gitignore"
+
+git :init
+git :add => '.'
+git :commit => "-m 'Initial commit: Clean application'"
+
+puts
+say_status  "Rubygems", "Adding Rubygems into Gemfile...\n", :yellow
+puts        '-'*80, ''
+
+gem 'slingshot-rb', :git => 'https://github.com/karmi/slingshot.git', :branch => 'activemodel'
+gem 'will_paginate', '~>3.0.pre'
+git :add => '.'
+git :commit => "-m 'Added gems'"
+
+puts
+say_status  "Rubygems", "Installing Rubygems...", :yellow
+
+puts
+puts "********************************************************************************"
+puts "                Running `bundle install`. Let's watch a movie!"
+puts "********************************************************************************", ""
+
+run "bundle install"
+
+puts
+say_status  "Model", "Adding search support into the Article model...", :yellow
+puts        '-'*80, ''
+
+generate :scaffold, "Article title:string content:text published_on:date"
+route "root :to => 'articles#index'"
+rake  "db:migrate"
+
+git :add => '.'
+git :commit => "-m 'Added the Article resource'"
+
+run "rm -f app/models/article.rb"
+file 'app/models/article.rb', <<-CODE
+class Article < ActiveRecord::Base
+  include Slingshot::Model::Search
+  include Slingshot::Model::Callbacks
+end
+CODE
+
+initializer 'slingshot.rb', <<-CODE
+Slingshot.configure do
+  logger STDERR
+end
+CODE
+
+git :commit => "-a -m 'Added Slingshot support into the Article class and an initializer'"
+
+puts
+say_status  "Controller", "Adding controller action, route, and neccessary HTML for search...", :yellow
+puts        '-'*80, ''
+
+gsub_file 'app/controllers/articles_controller.rb', %r{# GET /articles/1$}, <<-CODE
+  # GET /articles/search
+  def search
+    @articles = Article.search params[:q]
+
+    render :action => "index"
+  end
+
+  # GET /articles/1
+CODE
+
+gsub_file 'app/views/articles/index.html.erb', %r{<h1>Listing articles</h1>}, <<-CODE
+<h1>Listing articles</h1>
+
+<%= form_tag search_articles_path, :method => 'get' do %>
+  <%= label_tag :query %>
+  <%= text_field_tag :q, params[:q] %>
+  <%= submit_tag :search %>
+<% end %>
+
+<hr>
+CODE
+
+gsub_file 'app/views/articles/index.html.erb', %r{<%= link_to 'New Article', new_article_path %>}, <<-CODE
+<%= link_to 'New Article', new_article_path %>
+<%= link_to 'Back', articles_path if params[:q] %>
+CODE
+
+gsub_file 'config/routes.rb', %r{resources :articles}, <<-CODE
+  resources :articles do
+    collection { get :search }
+  end
+CODE
+
+git :commit => "-a -m 'Added Slingshot support into the frontend of application'"
+
+puts
+say_status  "Database", "Seeding the database with data...", :yellow
+puts        '-'*80, ''
+
+run "rm -f db/seeds.rb"
+file 'db/seeds.rb', <<-CODE
+contents = [
+'Lorem ipsum dolor sit amet.',
+'Consectetur adipisicing elit, sed do eiusmod tempor incididunt.',
+'Labore et dolore magna aliqua.',
+'Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris.',
+'Excepteur sint occaecat cupidatat non proident.'
+]
+
+puts "Deleting all articles..."
+Article.delete_all
+
+puts "Creating articles:"
+%w[ One Two Three Four Five ].each_with_index do |title, i|
+  Article.create :title => title, :content => contents[i], :published_on => i.days.ago.utc
+end
+CODE
+
+rake "db:seed"
+
+git :add    => "db/seeds.rb"
+git :commit => "-m 'Added database seeding script'"
+
+puts
+say_status  "Index", "Indexing database...", :yellow
+puts        '-'*80, ''
+
+rake "environment slingshot:import CLASS='Article' FORCE=true"
+
+puts  "", "="*80
+say_status  "DONE", "\e[1mStarting the application. Open http://localhost:3000 and search for something...\e[0m", :yellow
+puts  "="*80, ""
+
+run  "rails server"