waistband 0.0.15

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+.DS_Store

data/.rspec

@@ -0,0 +1,2 @@
+--format nested
+--color

data/.rvmrc

@@ -0,0 +1,2 @@
+rvm use ruby-1.9.3-p374@waistband --create
+export PATH=./bin:$PATH

data/Gemfile

@@ -0,0 +1,9 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in waistband.gemspec
+gemspec
+
+gem 'rspec'
+gem 'debugger'
+gem 'kaminari'
+gem 'timecop'

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 David Jairala
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,193 @@
+# Waistband
+
+Ruby interface to Elastic Search
+
+## Installation
+
+Install ElasticSearch:
+
+```bash
+brew install elasticsearch
+```
+
+Add this line to your application's Gemfile:
+
+    gem 'waistband', git: 'git@github.com:taskrabbit/waistband.git', tag: 'v0.0.9'
+
+Be sure to check out the [releases page](https://github.com/taskrabbit/waistband/releases) to get the latest tag.
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install waistband
+
+## Configuration
+
+Configuration is generally pretty simple.  First, create a folder where you'll store your Waistband configuration docs, usually under `#{APP_DIR}/config/waistband/`, you can also just throw it under `#{APP_DIR}/config/` if you want.  The baseline config contains something like this:
+
+```yml
+# #{APP_DIR}/config/waistband/waistband.yml
+development:
+  servers:
+    server1:
+      host: http://localhost
+      port: 9200
+```
+
+You can name the servers whatever you want, and one of them is selected at random using `Array.sample` when initializing the configuration singleton.  Here's an example with two servers:
+
+```yml
+# #{APP_DIR}/config/waistband/waistband.yml
+development:
+  servers:
+    server1:
+      host: http://173.247.192.214
+      port: 9200
+    server2:
+      host: http://173.247.192.215
+      port: 9200
+```
+
+You'll need a separate config file for each index you use, containing the index settings and mappings.  For example, for my search index, I use something akin to this:
+
+```yml
+# #{APP_DIR}/config/waistband/waistband_search.yml
+development:
+  name: search
+  stringify: false
+  settings:
+    index:
+      number_of_shards: 4
+  mappings:
+    event:
+      _source:
+        includes: ["*"]
+```
+
+## List of config settings:
+
+* `name`: name of the index.  You can (and probably should) have a different name for the index for your test environment.
+* `stringify`: determines wether whatever is stored into the index is going to be converted to a string before storage.  Usually false unless you need it to be true for specific cases, like if for some `key => value` pairs the value is of different types some times.  If you do decide to stringify your values for an index, please read the **Stringify All** section of the docs further down.
+* `settings`: settings for the Elastic Search index.  Refer to the ["admin indices update settings"](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings/) document for more info.
+* `mappings`: the index mappings.  More often than not you'll want to include all of the document attribute, so you'll do something like in the example above.  For more info, refer to the [mapping reference]("http://www.elasticsearch.org/guide/reference/mapping/").
+
+## Initializer
+
+After getting all the YML config files in place, you'll just need to hook up an initializer to these files:
+
+```ruby
+# #{APP_DIR}/config/initializers/waistband.rb
+Waistband.configure do |c|
+  c.config_dir = "#{APP_DIR}/spec/config/waistband"
+end
+```
+
+## Usage
+
+### Indexes
+
+
+#### Creating and destroying the indexes
+
+For each index you have, you'll probably want to make sure it's created on initialization, so either in the same waistband initializer or in another initializer, depending on your preferences, you'll have to create them.  For our search example:
+
+```ruby
+# #{APP_DIR}/config/initializers/waistband.rb
+# ...
+Waistband::Index.new('search').create!
+```
+
+This will create the index if it's not been created already or return nil if it already exists.
+
+Destroying an index is equally easy:
+
+```ruby
+Waistband::Index.new('search').destroy!
+```
+
+When writing tests, it would generally be advisable to destroy and create the indexes in a `before(:each)` or `before(:all)` depending in your circumstances.  Also, remember for testing that replication and data availability is not inmediate on the indexes, so if you create an immediate expectation for data to be there, you should refresh the index before it:
+
+```ruby
+Waistband::Index.new('search').refresh
+```
+
+#### Writing, reading and deleting from an index
+
+```ruby
+index = Waistband::Index.new('search')
+
+# writing
+index.store!('my_data', {'important' => true, 'valuable' => {'always' => true}})
+# => "{\"ok\":true,\"_index\":\"search\",\"_type\":\"search\",\"_id\":\"my_data\",\"_version\":1}"
+
+# reading
+index.read('my_data')
+# => {"important"=>true, "valuable"=>{"always"=>true}}
+
+# deleting
+index.delete!('my_data')
+# => "{\"ok\":true,\"found\":true,\"_index\":\"search\",\"_type\":\"search\",\"_id\":\"my_data\",\"_version\":2}"
+
+# reading non-existent data
+index.read('my_data')
+# => nil
+```
+
+### Searching
+
+For searching, Waistband has the Query class available:
+
+```ruby
+index = Waistband::Index.new('search')
+query = index.query('shopping')
+query.add_fields('name', 'description') # look for the search term `shopping` in the attributes `name` and `description`
+query.add_term('task', 'true')          # only in documents where the attribute task is set to true
+
+query.results # => returns an array of Waistband::QueryResult
+
+query.total_hits
+# => 28481
+
+# get the second page of results:
+query.page = 2
+query.results
+
+# change the page size:
+query.page_size = 50
+query.page = 1
+query.results
+```
+
+For paginating the results, you can use the `#paginated_results` method, which requires the [Kaminari](https://github.com/amatsuda/kaminari), gem.  If you use another gem, you can just override the method, etc.
+
+### Stringify All
+
+For convenience, Waistband provides two mixins useful to stringify Arrays and Hashes recursively in the `Waistband::StringifyAll::Array` and `Waistband::StringifyAll::Hash` modules.  Feel free to include them into your Array or Hash classes as necessary in an initializer if needed.
+
+### Model
+
+The gem offers a `Waistband::Model` that you can use to store model type data into the Elastic Search index.  You just inherit from the class and define the following class methods:
+
+```ruby
+class Thing < Waistband::Model
+  with_index :my_index_name
+
+  columns   :user_id, :content, :admin_visible
+  defaults  admin_visible: false
+
+  validates :user_id, :content
+end
+```
+
+For more information and extra methods, take a peek into the class docs.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/lib/waistband.rb

@@ -0,0 +1,25 @@
+require "waistband/version"
+
+module Waistband
+  
+  autoload :Configuration,  "waistband/configuration"
+  autoload :StringifyAll,   "waistband/stringify_all"
+  autoload :QueryResult,    "waistband/query_result"
+  autoload :Query,          "waistband/query"
+  autoload :Index,          "waistband/index"
+  autoload :QuickError,     "waistband/quick_error"
+  autoload :Model,          "waistband/model"
+
+  class << self
+
+    def configure
+      yield ::Waistband::Configuration.instance if block_given?
+      config_instance = ::Waistband::Configuration.instance
+      config_instance.setup
+      config_instance
+    end
+    alias_method :config, :configure
+
+  end
+
+end

data/lib/waistband/configuration.rb

@@ -0,0 +1,51 @@
+require 'singleton'
+require 'active_support/core_ext/hash/indifferent_access'
+
+module Waistband
+  class Configuration
+
+    include Singleton
+
+    attr_accessor :config_dir
+
+    def initialize
+      @yml_config = {}
+      @indexes    = {}
+    end
+
+    def setup
+      raise "Please define a valid `config_dir` configuration variable!"  unless config_dir
+      raise "Couldn't find configuration directory #{config_dir}"         unless File.exist?(config_dir)
+
+      @env        ||= ENV['RAILS_ENV'] || ENV['RACK_ENV'] || 'development'
+      @yml_config   = YAML.load_file("#{config_dir}/waistband.yml")[@env].with_indifferent_access
+    end
+
+    def index(name)
+      @indexes[name] ||= YAML.load_file("#{config_dir}/waistband_#{name}.yml")[@env].with_indifferent_access
+    end
+
+    def hostname
+      "#{host}:#{port}"
+    end
+
+    def method_missing(method_name, *args, &block)
+      return current_server[method_name]  if current_server[method_name]
+      return @yml_config[method_name]     if @yml_config[method_name]
+      super
+    end
+
+    private
+
+      def current_server
+        servers.sample
+      end
+
+      def servers
+        @servers ||= @yml_config['servers'].map {|server_name, config| config}
+      end
+
+    # /private
+
+  end
+end

data/lib/waistband/index.rb

@@ -0,0 +1,91 @@
+require 'json'
+require 'rest-client'
+require 'active_support/core_ext/string/inflections'
+require 'active_support/core_ext/hash/indifferent_access'
+
+module Waistband
+  class Index
+
+    MAX_RETRIES = 10
+
+    def initialize(index)
+      @index        = index
+      @index_name   = config['name']
+      @stringify    = config['stringify']
+      @retries      = 0
+    end
+
+    # create the index
+    def create!
+      RestClient.post(url, index_json)
+    rescue RestClient::BadRequest => ex
+      nil
+    end
+
+    # destroy the index
+    def destroy!
+      RestClient.delete(url)
+    rescue RestClient::ResourceNotFound => ex
+      nil
+    end
+
+    def update_settings!
+      RestClient.put("#{url}/_settings", settings_json)
+    end
+
+    # refresh the index
+    def refresh
+      RestClient.post("#{url}/_refresh", {})
+    end
+
+    def store!(key, data)
+      # map everything to strings
+      data = data.stringify_all if @stringify && data.respond_to?(:stringify_all)
+
+      RestClient.put(url_for_key(key), data.to_json)
+    end
+
+    def delete!(key)
+      RestClient.delete(url_for_key(key))
+    end
+
+    def read(key)
+      fetched = RestClient.get(url_for_key(key))
+      JSON.parse(fetched)['_source'].with_indifferent_access
+    rescue RestClient::ResourceNotFound => ex
+      nil
+    end
+
+    def query(term, options = {})
+      Waistband::Query.new(@index_name, term, options)
+    end
+
+    private
+
+      def url_for_key(key)
+        "#{url}/#{@index.singularize}/#{key}"
+      end
+
+      def settings_json
+        @settings_json ||= begin
+          settings = config['settings']['index'].except('number_of_shards')
+          {index: settings}.to_json
+        end
+      end
+
+      def index_json
+        @index_json ||= config.except('name', 'stringify').to_json
+      end
+
+      def config
+        @config ||= Waistband.config.index(@index)
+      end
+
+      def url
+        "#{Waistband.config.hostname}/#{@index_name}"
+      end
+
+    # /private
+
+  end
+end

data/lib/waistband/model.rb

@@ -0,0 +1,215 @@
+require 'digest/sha1'
+require 'json'
+require 'active_support/core_ext/class/attribute'
+require 'active_support/values/time_zone'
+require 'active_support/time_with_zone'
+
+module Waistband
+  class Model
+
+    class_attribute :index_name, :column_names, :validate_columns,
+                    :default_values, :stringify_columns
+
+    class << self
+
+      def index
+        @index ||= Waistband::Index.new(index_name)
+      end
+
+      def find(id)
+        attrs = index.read(id)
+        raise ActiveRecord::RecordNotFound.new("#{self} not found!") unless attrs
+        new attrs
+      end
+
+      def create(attributes)
+        instance = new(attributes)
+        instance.save
+        instance
+      end
+
+      def query
+        q = index.query('')
+        q.add_term('model_type', name.underscore)
+        q
+      end
+
+      def query_desc
+        q = query
+        q.add_sort 'created_at', 'desc'
+        q
+      end
+
+      def query_asc
+        q = query
+        q.add_sort 'created_at', 'asc'
+        q
+      end
+
+      def first
+        attrs = query_asc.results.first.try(:source)
+        return new(attrs) if attrs
+        nil
+      end
+
+      def last
+        attrs = query_desc.results.first.try(:source)
+        return new(attrs) if attrs
+        nil
+      end
+
+      def with_index(name)
+        self.index_name = name.to_s
+      end
+
+      def create_index!
+        Waistband::Index.new(index_name).create!
+      end
+
+      def destroy_index!
+        Waistband::Index.new(index_name).destroy!
+      end
+
+      def validates(*cols)
+        self.validate_columns ||= []
+        self.validate_columns |= cols.map(&:to_sym)
+      end
+
+      def defaults(values = {})
+        self.default_values ||= {}
+
+        values.each do |k, v|
+          self.default_values = self.default_values.merge({k.to_sym => v})
+        end
+      end
+
+      def stringify(*cols)
+        self.stringify_columns ||= []
+        self.stringify_columns |= cols.map(&:to_sym)
+      end
+
+      def columns(*cols)
+        cols               |= [:id, :model_type, :created_at, :updated_at]
+        self.column_names ||= []
+        self.column_names  |= cols.map(&:to_sym)
+
+        cols.each do |col|
+          # Normal attributes setters and getters
+          class_eval <<-EV, __FILE__, __LINE__ + 1
+            def #{col}
+              read_attribute(:#{col})
+            end
+
+            def #{col}=(val)
+              write_attribute(:#{col}, val)
+            end
+          EV
+
+          # Relationship type columns: `user_id`, `app_id` to `user`, `app`
+          if col =~ /(.*)_id$/
+            class_eval <<-EV, __FILE__, __LINE__ + 1
+              def #{$1}
+                klass = "#{$1}".classify.constantize
+                klass.find(#{col})
+              end
+
+              def #{$1}=(val)
+                write_attribute(:#{$1}_id, val.id)
+              end
+            EV
+          end
+
+        end
+      end
+
+    end # /class << self
+
+    attr_reader :errors
+
+    def initialize(attributes = {})
+      @attributes = (attributes || {}).symbolize_keys
+
+      @attributes.each do |key, val|
+        if self.class.stringify_columns.include?(key)
+          self.send("#{key}=", val)
+        elsif !self.class.column_names.include?(key)
+          raise ArgumentError.new("#{key} is not a valid column name!")
+        end  
+      end
+
+      self.class.default_values.each do |k, v|
+        self.send("#{k}=", v) if self.send(k).nil?
+      end
+
+      @errors = ::Waistband::QuickError.new
+    end
+
+    def save
+      return false unless valid?
+
+      before_save
+
+      prev_id         = self.id
+      prev_created_at = self.created_at
+      prev_updated_at = self.updated_at
+
+      self.id         ||= generated_id
+      self.created_at ||= (Time.zone || Time).now.to_i
+      self.updated_at   = (Time.zone || Time).now.to_i
+      self.model_type   = self.class.name.downcase
+
+      stored_json = JSON.parse store!
+      stored      = !!stored_json.try(:[], 'ok')
+
+      if stored
+        after_save
+      else stored
+        self.id         = prev_id
+        self.created_at = prev_created_at
+        self.updated_at = prev_updated_at
+      end
+
+      stored
+    end
+
+    def before_save
+    end
+
+    def after_save
+    end
+
+    def valid?
+      self.class.validate_columns.each do |col|
+        @errors << "#{col} cannot be nil" if self.send(col).nil?
+      end
+
+      @errors.empty?
+    end
+
+    def attributes
+      Hash[self.class.column_names.map{|col| [col, self.send(col)] }]
+    end
+
+    def read_attribute(attribute)
+      @attributes[attribute]
+    end
+
+    def write_attribute(attribute, val)
+      val = val.to_s if self.class.stringify_columns.include?(attribute.to_sym)
+      @attributes[attribute] = val
+    end
+
+    private
+
+      def store!
+        self.class.index.store!(id, attributes)
+      end
+
+      def generated_id
+        @generated_id ||= Digest::SHA1.hexdigest "#{attributes}:#{rand(999999)}"
+      end
+
+    # /private
+
+  end
+end