tire 0.1.16 → 0.2.0

data/README.markdown

@@ -1,29 +1,33 @@
 Tire
 =========
 
-_Tire_ is a Ruby client for the [ElasticSearch](http://www.elasticsearch.org/) search engine/database.
+_Tire_ is a Ruby (1.8 or 1.9) client for the [ElasticSearch](http://www.elasticsearch.org/)
+search engine/database.
 
 _ElasticSearch_ is a scalable, distributed, cloud-ready, highly-available,
-full-text search engine and database, communicating by JSON over RESTful HTTP,
-based on [Lucene](http://lucene.apache.org/), written in Java.
+full-text search engine and database with
+[powerfull aggregation features](http://www.elasticsearch.org/guide/reference/api/search/facets/),
+communicating by JSON over RESTful HTTP, based on [Lucene](http://lucene.apache.org/), written in Java.
 
-This document provides just a brief overview of _Tire's_ features. Be sure to check out also
-the extensive documentation at <http://karmi.github.com/tire/> if you're interested.
+This Readme provides a brief overview of _Tire's_ features. The more detailed documentation is at <http://karmi.github.com/tire/>.
+
+Both of these documents contain a lot of information. Please set aside some time to read them thoroughly, before you blindly dive into „somehow making it work“. Just skimming through it **won't work** for you. For more information, please refer to the [integration test suite](https://github.com/karmi/tire/tree/master/test/integration)
+and [issues](https://github.com/karmi/tire/issues).
 
 Installation
 ------------
 
-First, you need a running _ElasticSearch_ server. Thankfully, it's easy. Let's define easy:
+OK. First, you need a running _ElasticSearch_ server. Thankfully, it's easy. Let's define easy:
 
-    $ curl -k -L -o elasticsearch-0.16.0.tar.gz http://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.16.0.tar.gz
-    $ tar -zxvf elasticsearch-0.16.0.tar.gz
-    $ ./elasticsearch-0.16.0/bin/elasticsearch -f
+    $ curl -k -L -o elasticsearch-0.17.2.tar.gz http://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.17.2.tar.gz
+    $ tar -zxvf elasticsearch-0.17.2.tar.gz
+    $ ./elasticsearch-0.17.2/bin/elasticsearch -f
 
-OK. Easy. On a Mac, you can also use _Homebrew_:
+See, easy. On a Mac, you can also use _Homebrew_:
 
     $ brew install elasticsearch
 
-OK. Let's install the gem via Rubygems:
+Now, let's install the gem via Rubygems:
 
     $ gem install tire
 
@@ -39,8 +43,7 @@ Usage
 
 _Tire_ exposes easy-to-use domain specific language for fluent communication with _ElasticSearch_.
 
-It also blends with your [ActiveModel](https://github.com/rails/rails/tree/master/activemodel)
-classes for convenient usage in Rails applications.
+It easily blends with your _ActiveModel_/_ActiveRecord_ classes for convenient usage in _Rails_ applications.
 
 To test-drive the core _ElasticSearch_ functionality, let's require the gem:
 
@@ -61,7 +64,7 @@ which allows you to use your preferred JSON library. We'll use the
     require 'yajl/json_gem'
 ```
 
-OK. Let's create an index named `articles` and store/index some documents:
+Let's create an index named `articles` and store/index some documents:
 
 ```ruby
     Tire.index 'articles' do
@@ -83,6 +86,8 @@ for a specific document type:
 
 ```ruby
     Tire.index 'articles' do
+      delete
+
       create :mappings => {
         :article => {
           :properties => {
@@ -97,29 +102,35 @@ for a specific document type:
 ```
 
 Of course, we may have large amounts of data, and it may be impossible or impractical to add them to the index
-one by one. We can use _ElasticSearch's_ [bulk storage](http://www.elasticsearch.org/guide/reference/api/bulk.html):
+one by one. We can use _ElasticSearch's_
+[bulk storage](http://www.elasticsearch.org/guide/reference/api/bulk.html).
+Notice, that collection items must have an `id` property or method,
+and should have a `type` property, if you've set any specific mapping for the index.
 
 ```ruby
     articles = [
-      { :id => '1', :title => 'one'   },
-      { :id => '2', :title => 'two'   },
-      { :id => '3', :title => 'three' }
+      { :id => '1', :type => 'article', :title => 'one',   :tags => ['ruby']           },
+      { :id => '2', :type => 'article', :title => 'two',   :tags => ['ruby', 'python'] },
+      { :id => '3', :type => 'article', :title => 'three', :tags => ['java']           },
+      { :id => '4', :type => 'article', :title => 'four',  :tags => ['ruby', 'php']    }
     ]
 
-    Tire.index 'bulk' do
+    Tire.index 'articles' do
       import articles
     end
 ```
 
-We can also easily manipulate the documents before storing them in the index, by passing a block to the
-`import` method:
+We can easily manipulate the documents before storing them in the index, by passing a block to the
+`import` method, like this:
 
 ```ruby
-    Tire.index 'bulk' do
+    Tire.index 'articles' do
       import articles do |documents|
 
         documents.each { |document| document[:title].capitalize! }
       end
+
+      refresh
     end
 ```
 
@@ -137,7 +148,7 @@ from the database:
 
       filter :terms, :tags => ['ruby']
 
-      sort { title 'desc' }
+      sort { by :title, 'desc' }
 
       facet 'global-tags' do
         terms :tags, :global => true
@@ -189,7 +200,7 @@ count for articles tagged 'php' is excluded, since they don't match the current
     # java       1
 ```
 
-Notice, that only variables from the enclosing scope are accesible.
+Notice, that only variables from the enclosing scope are accessible.
 If we want to access the variables or methods from outer scope,
 we have to use a slight variation of the DSL, by passing the
 `search` and `query` objects around.
@@ -210,7 +221,7 @@ we can use the [_bool_](http://www.elasticsearch.org/guide/reference/query-dsl/b
 query. In _Tire_, we build them declaratively.
 
 ```ruby
-    Tire.search('articles') do
+    Tire.search 'articles' do
       query do
         boolean do
           should   { string 'tags:ruby' }
@@ -242,7 +253,7 @@ And a query for the _published_on_ property:
 Now, we can combine these queries for different searches:
 
 ```ruby
-    Tire.search('articles') do
+    Tire.search 'articles' do
       query do
         boolean &tags_query
         boolean &published_on_query
@@ -250,7 +261,17 @@ Now, we can combine these queries for different searches:
     end
 ```
 
-If configuring the search payload with blocks somehow feels too weak for you, you can pass
+Note, that you can pass options for configuring queries, facets, etc. by passing a Hash as the last argument to the method call:
+
+```ruby
+    Tire.search 'articles' do
+      query do
+        string 'ruby python', :default_operator => 'AND', :use_dis_max => true
+      end
+    end
+```
+
+If configuring the search payload with blocks feels somehow too weak for you, you can pass
 a plain old Ruby `Hash` (or JSON string) with the query declaration to the `search` method:
 
 ```ruby
@@ -260,7 +281,7 @@ a plain old Ruby `Hash` (or JSON string) with the query declaration to the `sear
 If this sounds like a great idea to you, you are probably able to write your application
 using just `curl`, `sed` and `awk`.
 
-We can display the full query JSON for close inspection:
+For debugging purposes, we can display the full query JSON for close inspection:
 
 ```ruby
     puts s.to_json
@@ -293,13 +314,14 @@ The _Tire_ DSL tries hard to provide a strong Ruby-like API for the main _Elasti
 
 By default, _Tire_ wraps the results collection in a enumerable `Results::Collection` class,
 and result items in a `Results::Item` class, which looks like a child of `Hash` and `Openstruct`,
-for smooth iterating and displaying the results.
+for smooth iterating over and displaying the results.
 
 You may wrap the result items in your own class by setting the `Tire.configuration.wrapper`
 property. Your class must take a `Hash` of attributes on initialization.
 
-If that seems like a great idea to you, there's a big chance you already have such class, and one would bet
-it's an `ActiveRecord` or `ActiveModel` class, containing model of your Rails application.
+If that seems like a great idea to you, there's a big chance you already have such class.
+
+One would bet it's an `ActiveRecord` or `ActiveModel` class, containing model of your Rails application.
 
 Fortunately, _Tire_ makes blending _ElasticSearch_ features into your models trivially possible.
 
@@ -308,12 +330,15 @@ ActiveModel Integration
 -----------------------
 
 If you're the type with no time for lengthy introductions, you can generate a fully working
-example Rails application, with an `ActiveRecord` model and a search form, to play with:
+example Rails application, with an `ActiveRecord` model and a search form, to play with
+(it even downloads _ElasticSearch_ itself, generates the application skeleton and leaves you with
+a _Git_ repository to explore the steps and the code):
 
     $ rails new searchapp -m https://github.com/karmi/tire/raw/master/examples/rails-application-template.rb
 
-For the rest, let's suppose you have an `Article` class in your Rails application.
-To make it searchable with _Tire_, you just `include` it:
+For the rest of us, let's suppose you have an `Article` class in your _Rails_ application.
+
+To make it searchable with _Tire_, just `include` it:
 
 ```ruby
     class Article < ActiveRecord::Base
@@ -333,8 +358,8 @@ When you now save a record:
 
 it is automatically added into the index, because of the included callbacks.
 (You may want to skip them in special cases, like when your records are indexed via some external
-mechanism, let's say CouchDB or RabbitMQ [river](http://www.elasticsearch.org/blog/2010/09/28/the_river.html)
-for _ElasticSearch_.)
+mechanism, let's say a _CouchDB_ or _RabbitMQ_
+[river](http://www.elasticsearch.org/blog/2010/09/28/the_river.html).
 
 The document attributes are indexed exactly as when you call the `Article#to_json` method.
 
@@ -344,22 +369,21 @@ Now you can search the records:
     Article.search 'love'
 ```
 
-OK. This is where the game stops, often. Not here.
+OK. This is where the search game stops, often. Not here.
 
 First of all, you may use the full query DSL, as explained above, with filters, sorting,
 advanced facet aggregation, highlighting, etc:
 
 ```ruby
-    q = 'love'
     Article.search do
-      query { string q }
-      facet('timeline') { date :published_on, :interval => 'month' }
-      sort  { published_on 'desc' }
+      query             { string 'love' }
+      facet('timeline') { date   :published_on, :interval => 'month' }
+      sort              { by     :published_on, 'desc' }
     end
 ```
 
-Dynamic mapping is a godsend when you're prototyping.
-For serious usage, though, you'll definitely want to define a custom mapping for your model:
+Second, dynamic mapping is a godsend when you're prototyping.
+For serious usage, though, you'll definitely want to define a custom mapping for your models:
 
 ```ruby
     class Article < ActiveRecord::Base
@@ -367,7 +391,7 @@ For serious usage, though, you'll definitely want to define a custom mapping for
       include Tire::Model::Callbacks
 
       mapping do
-        indexes :id,           :type => 'string',  :analyzed => false
+        indexes :id,           :type => 'string',  :index    => :not_analyzed
         indexes :title,        :type => 'string',  :analyzer => 'snowball', :boost => 100
         indexes :content,      :type => 'string',  :analyzer => 'snowball'
         indexes :author,       :type => 'string',  :analyzer => 'keyword'
@@ -376,10 +400,13 @@ For serious usage, though, you'll definitely want to define a custom mapping for
     end
 ```
 
-In this case, _only_ the defined model attributes are indexed when adding to the index.
+In this case, _only_ the defined model attributes are indexed. The `mapping` declaration creates the
+index when the class is loaded or when the importing features are used, and _only_ when it does not yet exist.
+(It may well be reasonable to wrap the index creation logic in a class method of your model, so you
+have better control on index creation when bootstrapping your application or when setting up the test suite.)
 
-When you want tight grip on how your model attributes are added to the index, just
-provide the `to_indexed_json` method yourself:
+When you want a tight grip on how the attributes are added to the index, just
+implement the `to_indexed_json` method in your model:
 
 ```ruby
     class Article < ActiveRecord::Base
@@ -404,24 +431,66 @@ provide the `to_indexed_json` method yourself:
     end
 ```
 
-Note that _Tire_-enhanced models are fully compatible with [`will_paginate`](https://github.com/mislav/will_paginate),
-so you can pass any parameters to the `search` method in the controller, as usual:
+The results returned by `Article.search` are wrapped in the aforementioned `Item` class, by default.
+This way, we have a fast and flexible access to the properties returned from _ElasticSearch_ (via the
+`_source` or `fields` JSON properties). This way, we can index whatever JSON we like in _ElasticSearch_,
+and retrieve it, simply, via the dot notation:
+
+```ruby
+    articles = Article.search 'love'
+    articles.each do |article|
+      puts article.title
+      puts article.author.last_name
+    end
+```
+
+The `Item` instances masquerade themselves as instances of your model within a _Rails_ application
+(based on the `_type` property retrieved from _ElasticSearch_), so you can use them carefree;
+all the `url_for` or `dom_id` helpers work as expected.
+
+If you need to access the “real” model (eg. to access its assocations or methods not
+stored in _ElasticSearch_), just load it from the database:
+
+```ruby
+    puts article.load(:include => 'comments').comments.size
+```
+
+You can see that _Tire_ stays as far from the database as possible. That's because it believes
+you have most of the data you want to display stored in _ElasticSearch_. When you need
+to eagerly load the records from the database itself, for whatever reason,
+you can do it with the `:load` option when searching:
+
+```ruby
+    # Will call `Article.search [1, 2, 3]`
+    Article.search 'love', :load => true
+```
+
+Instead of simple `true`, you can pass any options for the model's find method:
+
+```ruby
+    # Will call `Article.search [1, 2, 3], :include => 'comments'`
+    Article.search :load => { :include => 'comments' } do
+      query { string 'love' }
+    end
+```
+
+Note that _Tire_ search results are fully compatible with [`will_paginate`](https://github.com/mislav/will_paginate),
+so you can pass all the usual parameters to the `search` method in the controller:
 
 ```ruby
     @articles = Article.search params[:q], :page => (params[:page] || 1)
 ```
 
-OK. Chances are, you have lots of records stored in the underlying database. How will you get them to _ElasticSearch_? Easy:
+OK. Chances are, you have lots of records stored in your database. How will you get them to _ElasticSearch_? Easy:
 
 ```ruby
     Article.elasticsearch_index.import Article.all
 ```
 
-However, this way, all your records are loaded into memory, serialized into JSON,
+This way, however, all your records are loaded into memory, serialized into JSON,
 and sent down the wire to _ElasticSearch_. Not practical, you say? You're right.
 
-Provided your model implements some sort of _pagination_ — and it probably does, for so much data —,
-you can just run:
+Provided your model implements some sort of _pagination_ — and it probably does —, you can just run:
 
 ```ruby
     Article.import
@@ -453,14 +522,14 @@ provided by the `mapping` block in your model):
 When you'll spend more time with _ElasticSearch_, you'll notice how
 [index aliases](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)
 are the best idea since the invention of inverted index.
-You can index your data into a fresh index (and possibly update an alias if everything's fine):
+You can index your data into a fresh index (and possibly update an alias once everything's fine):
 
 ```bash
     $ rake environment tire:import CLASS='Article' INDEX='articles-2011-05'
 ```
 
 OK. All this time we have been talking about `ActiveRecord` models, since
-it is a reasonable Rails' default for the storage layer.
+it is a reasonable _Rails_' default for the storage layer.
 
 But what if you use another database such as [MongoDB](http://www.mongodb.org/),
 another object mapping library, such as [Mongoid](http://mongoid.org/)?
@@ -494,12 +563,10 @@ Well, things stay mostly the same:
     Article.search 'love'
 ```
 
-That's kinda nice. But there's more.
+_Tire_ does not care what's your primary data storage solution, if it has an _ActiveModel_-compatible
+adapter. But there's more.
 
-_Tire_ implements not only _searchable_ features, but also _persistence_ features.
-
-This means that you can use a _Tire_ model **instead of** your database, not just
-for searching your database. Why would you like to do that?
+_Tire_ implements not only _searchable_ features, but also _persistence_ features. This means you can use a _Tire_ model **instead of your database**, not just for _searching_ your database. Why would you like to do that?
 
 Well, because you're tired of database migrations and lots of hand-holding with your
 database to store stuff like `{ :name => 'Tire', :tags => [ 'ruby', 'search' ] }`.
@@ -510,8 +577,7 @@ then constructing elaborate database query conditions.
 Because you have _lots_ of data and want to use _ElasticSearch's_
 advanced distributed features.
 
-To use the persistence features, you have to include the `Tire::Persistence` module
-in your class and define the properties (analogous to the way you do with CouchDB- or MongoDB-based models):
+To use the persistence features, just include the `Tire::Persistence` module in your class and define the properties (like with _CouchDB_- or _MongoDB_-based models):
 
 ```ruby
     class Article
@@ -532,6 +598,9 @@ Of course, not all validations or `ActionPack` helpers will be available to your
 but if you can live with that, you've just got a schema-free, highly-scalable storage
 and retrieval engine for your data.
 
+Please be sure to peruse the [integration test suite](https://github.com/karmi/tire/tree/master/test/integration)
+for examples of the API and _ActiveModel_ integration usage.
+
 Todo, Plans & Ideas
 -------------------
 

data/examples/rails-application-template.rb

@@ -106,7 +106,7 @@ say_status  "Rubygems", "Adding Rubygems into Gemfile...\n", :yellow
 puts        '-'*80, ''; sleep 1
 
 gem 'tire'
-gem 'will_paginate', '~>3.0.pre'
+gem 'will_paginate', '~> 3.0'
 
 git :add => '.'
 git :commit => "-m 'Added gems'"

data/examples/tire-dsl.rb

@@ -43,9 +43,9 @@ require 'tire'
 
  [ERROR] You don’t appear to have ElasticSearch installed. Please install and launch it with the following commands:
 
- curl -k -L -o elasticsearch-0.16.0.tar.gz http://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.16.0.tar.gz
- tar -zxvf elasticsearch-0.16.0.tar.gz
- ./elasticsearch-0.16.0/bin/elasticsearch -f
+ curl -k -L -o elasticsearch-0.17.2.tar.gz http://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.17.2.tar.gz
+ tar -zxvf elasticsearch-0.17.2.tar.gz
+ ./elasticsearch-0.17.2/bin/elasticsearch -f
 INSTALL
 
 ### Storing and indexing documents
@@ -667,7 +667,7 @@ s = Tire.search 'articles' do
 
    # ... but will sort them by their `title`, in descending order.
    #
-  sort { title 'desc' }
+  sort { by :title, 'desc' }
 end
 
 # The results:
@@ -692,11 +692,11 @@ s = Tire.search 'articles' do
 
     # We will sort the results by their `published_on` property in _ascending_ order (the default),
     #
-    published_on
+    by :published_on
 
     # and by their `title` property, in _descending_ order.
     #
-    title 'desc'
+    by :title 'desc'
   end
 end
 

data/lib/tire/index.rb

@@ -41,21 +41,8 @@ module Tire
     end
 
     def store(*args)
-      case
-        when ( args.size === 3 && (args.first.is_a?(String) || args.first.is_a?(Symbol)) )
-          Tire.warn "Passing the document type as argument in Index#store has been deprecated, " +
-                    "please pass a Hash with _type/type property, or " +
-                    "an object with _type/type/document_type method."
-          type, document, options = args
-        when ( args.size === 2 && (args.first.is_a?(String) || args.first.is_a?(Symbol)) )
-          Tire.warn "Passing the document type as argument in Index#store has been deprecated" +
-                    "please pass a Hash with _type/type property, or " +
-                    "an object with _type/type/document_type method."
-          type, document = args
-        else
-          document, options = args
-          type = get_type_from_document(document)
-      end
+      document, options = args
+      type = get_type_from_document(document)
 
       if options
         percolate = options[:percolate]
@@ -118,7 +105,7 @@ module Tire
         when method
           options = {:page => 1, :per_page => 1000}.merge options
           while documents = klass_or_collection.send(method.to_sym, options.merge(:page => options[:page])) \
-                            and documents.size > 0
+                            and not documents.empty?
             documents = yield documents if block_given?
 
             bulk_store documents
@@ -218,15 +205,8 @@ module Tire
     end
 
     def percolate(*args, &block)
-      if args.size > 1
-        Tire.warn "Passing the document type as argument in Index#percolate has been deprecated, " +
-                  "please pass a Hash with _type/type property, or " +
-                  "an object with _type/type/document_type method."
-        type, document = args
-      else
-        document = args.pop
-        type     = get_type_from_document(document)
-      end
+      document = args.pop
+      type     = get_type_from_document(document)
 
       document = MultiJson.decode convert_document_to_json(document)
 
@@ -273,14 +253,14 @@ module Tire
         when document.respond_to?(:document_type)
           document.document_type
         when document.is_a?(Hash)
-          document.delete(:_type) || document.delete('_type') || document.delete(:type) || document.delete('type')
+          document[:_type] || document['_type'] || document[:type] || document['type']
         when document.respond_to?(:_type)
           document._type
         when document.respond_to?(:type) && document.type != document.class
           document.type
         end
       $VERBOSE = old_verbose
-      type ||= :document
+      type || :document
     end
 
     def get_id_from_document(document)