waistband 0.8.5 → 0.9.0

data/Gemfile

@@ -5,5 +5,5 @@ gemspec
 
 gem 'rspec'
 gem 'debugger'
-gem 'kaminari'
-gem 'timecop'
+gem 'kaminari', require: false
+gem 'timecop'

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2013 David Jairala
+Copyright (c) 2014 David Jairala
 
 MIT License
 

data/README.md

@@ -1,8 +1,8 @@
 # Waistband
 
-Ruby interface to Elastic Search
+Configuration and sensible defaults for ElasticSearch on Ruby.  Handles configuration, index creation, quality of life, etc, of Elastic Search in Ruby.
 
-## Installation
+# Installation
 
 Install ElasticSearch:
 
@@ -20,7 +20,7 @@ And then execute:
 
 Or install it yourself as:
 
-    $ gem install waistband
+$ gem install waistband
 
 ## Configuration
 
@@ -29,11 +29,14 @@ Configuration is generally pretty simple.  First, create a folder where you'll s
 ```yml
 # #{APP_DIR}/config/waistband/waistband.yml
 development:
-  timeout: 2
-  servers:
-    server1:
-      host: http://localhost
-      port: 9200
+    retries: 5
+    timeout: 2
+    reload_on_failure: true
+    servers:
+        server1:
+            protocol: http
+            host: localhost
+            port: 9200
 ```
 
 You can name the servers whatever you want, and one of them is selected at random using `Array.sample`, excluding blacklisted servers, when conduction operations on the server.  Here's an example with two servers:
@@ -41,14 +44,18 @@ You can name the servers whatever you want, and one of them is selected at rando
 ```yml
 # #{APP_DIR}/config/waistband/waistband.yml
 development:
-  timeout: 2
-  servers:
-    server1:
-      host: http://173.247.192.214
-      port: 9200
-    server2:
-      host: http://173.247.192.215
-      port: 9200
+    retries: 5
+    timeout: 2
+    reload_on_failure: true
+    servers:
+        server1:
+            protocol: http
+            host: 173.247.192.214
+            port: 9200
+        server2:
+            protocol: http
+            host: 173.247.192.215
+            port: 9200
 ```
 
 You'll need a separate config file for each index you use, containing the index settings and mappings.  For example, for my search index, I use something akin to this:
@@ -56,31 +63,35 @@ You'll need a separate config file for each index you use, containing the index
 ```yml
 # #{APP_DIR}/config/waistband/waistband_search.yml
 development:
-  stringify: false
-  settings:
+    stringify: false
+    settings:
     index:
-      number_of_shards: 4
-  mappings:
-    event:
-      _source:
-        includes: ["*"]
+        number_of_shards: 4
+    mappings:
+        event:
+            _source:
+                includes: ["*"]
 ```
 
 ## List of config settings:
 
 * `settings`: settings for the Elastic Search index.  Refer to the ["admin indices update settings"](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings/) document for more info.
 * `mappings`: the index mappings.  More often than not you'll want to include all of the document attribute, so you'll do something like in the example above.  For more info, refer to the [mapping reference]("http://www.elasticsearch.org/guide/reference/mapping/").
+* `retries`: number of times to retry before moving on to the next server node.
+* `reload_on_failure`: should we reload the node list from the server on failure.
+* `timeout`: seconds till a timeout exception is raise when trying to connect to the node.
 * `name`: optional - name of the index.  You can (and probably should) have a different name for the index for your test environment.  If not specified, it defaults to the name of the yml file minus the `waistband_` portion, so in the above example, the index name would become `search_#{env}`, where env is your environment variable as defined in `Waistband::Configuration#setup` (determined by `RAILS_ENV` or `RACK_ENV`).
 * `stringify`: optional - determines wether whatever is stored into the index is going to be converted to a string before storage.  Usually false unless you need it to be true for specific cases, like if for some `key => value` pairs the value is of different types some times.
 
 ## Initializer
 
-After getting all the YML config files in place, you'll just need to hook up an initializer to these files:
+
+Waistband will look for config files by default in `File.join(Rails.root, 'config')`.  You can override the default location of the config folder. After getting all the YML config files in place, you'll just need to hook up an initializer to these files:
 
 ```ruby
 # #{APP_DIR}/config/initializers/waistband.rb
 Waistband.configure do |c|
-  c.config_dir = "#{APP_DIR}/spec/config/waistband"
+    c.config_dir = "#{APP_DIR}/spec/config/waistband"
 end
 ```
 
@@ -91,20 +102,20 @@ end
 
 #### Creating and destroying the indexes
 
-For each index you have, you'll probably want to make sure it's created on initialization, so either in the same waistband initializer or in another initializer, depending on your preferences, you'll have to create them.  For our search example:
+For each index you have, you'll probably want to make sure it's created on initialization, or in a Rake task, so either in the same waistband initializer or in another initializer, depending on your preferences, you'll have to create them.  For our search example:
 
 ```ruby
 # #{APP_DIR}/config/initializers/waistband.rb
 # ...
-Waistband::Index.new('search').create!
+Waistband::Index.new('search').create
 ```
 
-This will create the index if it's not been created already or return nil if it already exists.
+This will create the index if it's not been created already or return nil if it already exists.  If you want to raise an exception if it already exists, use the `#create!` method.
 
 Destroying an index is equally easy:
 
 ```ruby
-Waistband::Index.new('search').destroy!
+Waistband::Index.new('search').delete
 ```
 
 When writing tests, it would generally be advisable to destroy and create the indexes in a `before(:each)` or `before(:all)` depending in your circumstances.  Also, remember for testing that replication and data availability is not inmediate on the indexes, so if you create an immediate expectation for data to be there, you should refresh the index before it:
@@ -113,7 +124,7 @@ When writing tests, it would generally be advisable to destroy and create the in
 Waistband::Index.new('search').refresh
 ```
 
-Note: most index methods such as `create`, `destroy`, `read`, etc, have an equivalent bang method (`destroy!`) that will actually throw an exception if something goes wrong.  For example, `destroy` will return nil if the index doesn't exist, but will raise any other unrelated exceptions, whereas `destroy!` will raise even the Index Not Found exception.
+Note: most index methods such as `create`, `delete`, etc, have an equivalent bang method (`delete!`) that will actually throw an exception if something goes wrong.  For example, `delete` will return nil if the index doesn't exist, but will raise any other unrelated exceptions, whereas `delete!` will raise even the Index Not Found exception.
 
 #### Writing, reading and deleting from an index
 
@@ -121,55 +132,63 @@ Note: most index methods such as `create`, `destroy`, `read`, etc, have an equiv
 index = Waistband::Index.new('search')
 
 # writing
-index.store!('my_data', {'important' => true, 'valuable' => {'always' => true}})
-# => "{\"ok\":true,\"_index\":\"search\",\"_type\":\"search\",\"_id\":\"my_data\",\"_version\":1}"
+index.save('my_data', {'important' => true, 'valuable' => {'always' => true}}) # => true
 
 # reading
-index.read('my_data')
-# => {"important"=>true, "valuable"=>{"always"=>true}}
+index.find('my_data') # => {"important"=>true, "valuable"=>{"always"=>true}}
+
+# reading with all the internal data
+index.read('my_data') # => {'_id' => '123123', '_source' => {"important"=>true, "valuable"=>{"always"=>true}}, ...}
 
 # deleting
-index.delete!('my_data')
-# => "{\"ok\":true,\"found\":true,\"_index\":\"search\",\"_type\":\"search\",\"_id\":\"my_data\",\"_version\":2}"
+index.destroy('my_data') # => "{\"ok\":true,\"found\":true,\"_index\":\"search\",\"_type\":\"search\",\"_id\":\"my_data\",\"_version\":2}"
 
 # reading non-existent data
-index.read('my_data')
-# => nil
+index.find('my_data') # => nil
 ```
 
 ### Searching
 
-For searching, you construct a query from your index:
+For searching, you construct a search from your index:
 
 ```ruby
 index = Waistband::Index.new('search')
-query = index.query(page_size: 5).prepare({
-  query: {
-    term: { hidden: false }
-  },
-  sort: { created_at: {order: 'desc' } }
+results = index.search({
+    query: {
+        term: { hidden: false }
+    },
+    sort: { created_at: {order: 'desc' } },
+    page: 1,
+    page_size: 5
 })
 
-query.results # => returns an array of Waistband::QueryResult
+results.hits # => returns a search results object
+results.total_hits # => 28481
 
-query.total_hits
-# => 28481
+For paginating the results, you can use the `#paginated_results` method, which requires the [Kaminari](https://github.com/amatsuda/kaminari), gem.  If you use another gem, you can just override the method, etc.
 
-# get the second page of results:
-query.page = 2
-query.results
+For more information and extra methods, take a peek into the class docs.
 
-# change the page size:
-query.page_size = 50
-query.page = 1
-query.results
-```
+Also, for convenience, the gem provides the `Result` class, which just provides some quality-of-life methods for working with search result hashes or their inner `_source` hashes, for example:
 
-For paginating the results, you can use the `#paginated_results` method, which requires the [Kaminari](https://github.com/amatsuda/kaminari), gem.  If you use another gem, you can just override the method, etc.
+```ruby
+search = index.search({
+    query: {
+        term: { hidden: false }
+    }
+})
+results = search.results
+result = result.first
 
-For more information and extra methods, take a peek into the class docs.
+result._id # => '123123'
+result._type # => 'search_result'
+result._index # => 'search'
+result.task_id # => 991122 -- note that this is a method missing interface directly either to the search result hash, or to the _source sub-hash
+```
+
+The `Result` class is directly exposed via two methods in the `SearchResults` class: `#results` and `#paginated_results`.  You can use `#paginated_results` if you're using Kaminari for pagination and wish to use the awesomeness it provides.
 
-### Sub-Indexes
+### Index Aliasing
 
 Sometimes it can be useful to sub-divide your index into smaller indexes based on dates or other partitioning schemes.  To do this, the `Index` class exposes the `subs` option on instantiation:
 
@@ -187,13 +206,11 @@ Part of subbing is gonna be creating the correct aliases that group up your sub-
 ```ruby
 index = Waistband::Index.new('events', subs: %w(2013 01))
 index.create!
-index.alias!('my_super_events_alias')
-=> true
-index.fetch_alias('my_super_events_alias')
-=> {"events__2013_01"=>{"aliases"=>{"my_super_events_alias"=>{}}}}
+index.alias('my_super_events_alias') # => true
+index.alias_exists?('my_super_events_alias') # => true
 ```
 
-The `alias!` methods receives a param to define the alias name.
+The `alias` methods receives a param to define the alias name.
 
 ## Contributing
 
@@ -202,3 +219,4 @@ The `alias!` methods receives a param to define the alias name.
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
+

data/lib/waistband.rb

@@ -1,17 +1,14 @@
 require "waistband/version"
 
 module Waistband
-  
-  autoload :Configuration,    "waistband/configuration"
-  autoload :Connection,       "waistband/connection"
+
+  autoload :Errors,           "waistband/errors"
   autoload :StringifiedArray, "waistband/stringified_array"
   autoload :StringifiedHash,  "waistband/stringified_hash"
-  autoload :QueryResult,      "waistband/query_result"
-  autoload :QueryHelpers,     "waistband/query_helpers"
-  autoload :Query,            "waistband/query"
+  autoload :Configuration,    "waistband/configuration"
   autoload :Index,            "waistband/index"
-  autoload :QuickError,       "waistband/quick_error"
-  autoload :Model,            "waistband/model"
+  autoload :SearchResults,    "waistband/search_results"
+  autoload :Result,           "waistband/result"
 
   class << self
 
@@ -23,6 +20,10 @@ module Waistband
     end
     alias_method :config, :configure
 
+    def client
+      ::Waistband.config.client
+    end
+
   end
 
 end

data/lib/waistband/configuration.rb

@@ -17,6 +17,8 @@ module Waistband
     end
 
     def setup
+      self.config_dir = default_config_dir unless config_dir
+
       raise "Please define a valid `config_dir` configuration variable!"  unless config_dir
       raise "Couldn't find configuration directory #{config_dir}"         unless File.exist?(config_dir)
 
@@ -29,23 +31,32 @@ module Waistband
     end
 
     def method_missing(method_name, *args, &block)
-      return current_server[method_name]  if current_server[method_name]
-      return @yml_config[method_name]     if @yml_config[method_name]
+      return @yml_config[method_name] if @yml_config[method_name]
       super
     end
 
-    def servers
-      @servers ||= @yml_config['servers'].map do |server_name, config|
-        config.merge({
-          '_id' => Digest::SHA1.hexdigest("#{config['host']}:#{config['port']}")
-        })
+    def hosts
+      @hosts ||= @yml_config['servers'].map do |server_name, config|
+        config
       end
     end
 
+    def client
+      Elasticsearch::Client.new(
+              hosts: hosts,
+              randomize_hosts: true,
+              retry_on_failure: retries,
+              reload_on_failure: reload_on_failure
+            )
+    end
+
     private
 
-      def current_server
-        servers.sample
+      def default_config_dir
+        @default_config_dir ||= begin
+          return nil unless defined?(Rails)
+          File.join(Rails.root, 'config')
+        end
       end
 
     # /private

data/lib/waistband/errors.rb

@@ -0,0 +1,9 @@
+module Waistband
+  module Errors
+
+    class IndexExists < StandardError; end
+    class IndexNotFound < StandardError; end
+    class NoSearchHits < StandardError; end
+
+  end
+end

data/lib/waistband/index.rb

@@ -1,79 +1,199 @@
-require 'active_support/core_ext/object/blank'
 require 'active_support/core_ext/hash/keys'
+require 'active_support/core_ext/array/extract_options'
+require 'elasticsearch'
 
 module Waistband
   class Index
 
-    delegate  :create!, :create, :destroy!, :destroy,
-              :update_settings!, :update_settings,
-              :delete!, :delete, :read!, :read,
-              :alias, :alias!,
-              :fetch_alias, :mapping, :exists?,
-              :refresh!, :refresh,
-              :search_url,
-              to: :connection
-
-    attr_reader :base_name
-
-    def initialize(index, options = {})
+    def initialize(index_name, options = {})
       options = options.stringify_keys
 
-      @index          = index
-      @base_name      = index
-      @stringify      = config['stringify']
+      @index_name = index_name
+      @stringify = config['stringify']
 
+      # alias subs
       @subs = [options['subs']] if options['subs'].present?
       @subs = @subs.flatten     if @subs.is_a?(Array)
     end
 
-    def store!(key, data)
-      # map everything to strings
-      if @stringify
-        original_data = data
-        data = stringify_all data
-      end
+    def exists?
+      client.indices.exists index: config_name
+    end
+
+    def refresh
+      client.indices.refresh index: config_name
+    end
+
+    def update_mapping(type)
+      client.indices.put_mapping(
+        index: config_name,
+        type: type,
+        body: config['mappings'][type]
+      )
+    end
+
+    def update_settings
+      client.indices.put_settings(
+        index: config_name,
+        body: settings
+      )
+    end
+
+    def create
+      create!
+    rescue ::Waistband::Errors::IndexExists => ex
+      true
+    end
+
+    def create!
+      raise ::Waistband::Errors::IndexExists.new("Index already exists") if exists?
+      client.indices.create index: config_name, body: config.except('name', 'stringify')
+    end
+
+    def delete
+      delete!
+    rescue ::Waistband::Errors::IndexNotFound => ex
+      true
+    end
+
+    def delete!
+      raise ::Waistband::Errors::IndexNotFound.new("Index not found") unless exists?
+      client.indices.delete index: config_name
+    end
+
+    def save(*args)
+      body_hash = args.extract_options!
+      id = args.first
+      type = body_hash.delete(:type) || default_type_name
+
+      # map everything to strings if need be
+      body_hash = stringify_all(body_hash) if @stringify
+
+      saved = client.index(
+        index: config_name,
+        type: type,
+        id: id,
+        body: body_hash
+      )
+
+      saved['_id'].present?
+    end
+
+    def find(id, options = {})
+      find!(id, options)
+    rescue Elasticsearch::Transport::Transport::Errors::NotFound
+      nil
+    end
 
-      result = connection.put key, data
-      data = original_data if @stringify
+    def find!(id, options = {})
+      doc = read!(id, options)
+      doc['_source']
+    end
 
-      result
+    def read(id, options = {})
+      read!(id, options)
+    rescue Elasticsearch::Transport::Transport::Errors::NotFound
+      nil
     end
 
-    def query(options = {})
-      ::Waistband::Query.new self, options
+    def read!(id, options = {})
+      options = options.with_indifferent_access
+      type = options[:type] || default_type_name
+
+      client.get(
+        index: config_name,
+        type: type,
+        id: id
+      ).with_indifferent_access
     end
 
-    def name
-      @subs ? "#{@index}__#{@subs.join('_')}" : @index
+    def destroy(id, options = {})
+      destroy!(id, options)
+    rescue Elasticsearch::Transport::Transport::Errors::NotFound
+      nil
     end
 
-    def config_name
-      @subs ? "#{base_config_name}__#{@subs.join('_')}" : base_config_name
+    def destroy!(id, options = {})
+      options = options.with_indifferent_access
+      type = options[:type] || default_type_name
+
+      client.delete(
+        index: config_name,
+        id: id,
+        type: type
+      )
     end
 
-    def config_json
-      config.except('name', 'stringify').to_json
+    def search(body_hash)
+      page, page_size = get_page_info body_hash
+      body_hash       = parse_search_body(body_hash)
+
+      search_hash = client.search(
+        index: config_name,
+        body: body_hash
+      )
+
+      ::Waistband::SearchResults.new(search_hash, page: page, page_size: page_size)
     end
 
-    def base_config_name
-      return config['name'] if config['name']
-      "#{@base_name}_#{env}"
+    def alias(alias_name)
+      alias_name = full_alias_name alias_name
+      client.indices.put_alias(
+        index: config_name,
+        name: alias_name
+      )
     end
 
-    def custom_name?
-      !!config['name']
+    def alias_exists?(alias_name)
+      alias_name = full_alias_name alias_name
+      client.indices.alias_exists?(
+        index: config_name,
+        name: alias_name
+      )
     end
 
     def config
-      Waistband.config.index @index
+      ::Waistband.config.index @index_name
     end
 
-    def env
-      Waistband.config.env
+    def client
+      @client ||= ::Waistband.config.client
     end
 
     private
 
+      def get_page_info(body_hash)
+        page = body_hash[:page]
+        page_size = body_hash[:page_size]
+        [page, page_size]
+      end
+
+      def parse_search_body(body_hash)
+        body_hash = body_hash.with_indifferent_access
+
+        page = body_hash.delete(:page)
+        page_size = body_hash.delete(:page_size)
+
+        if page
+          page = page.to_i
+          page_size ||= 20
+          body_hash[:from] = page_size * (page - 1) unless body_hash[:from]
+          body_hash[:size] = page_size              unless body_hash[:size]
+        end
+
+        body_hash
+      end
+
+      def full_alias_name(alias_name)
+        name = alias_name
+        name << "_#{::Waistband.config.env}" unless custom_name?
+        name
+      end
+
+      def custom_name?
+        !!config['name']
+      end
+
       def stringify_all(data)
         data = if data.is_a? Array
           ::Waistband::StringifiedArray.new data
@@ -85,8 +205,22 @@ module Waistband
         data
       end
 
-      def connection
-        ::Waistband::Connection.new self
+      def default_type_name
+        @index_name.singularize
+      end
+
+      def settings
+        settings = config['settings']['index'].except('number_of_shards')
+        {index: settings}
+      end
+
+      def config_name
+        @subs ? "#{base_config_name}__#{@subs.join('_')}" : base_config_name
+      end
+
+      def base_config_name
+        return config['name'] if config['name']
+        "#{@index_name}_#{::Waistband.config.env}"
       end
 
     # /private