rawscsi 1.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 335008cad4298d74c3a008b6dc3d9e6ab0f52588
+  data.tar.gz: b1dae029204a1a1db586a9018f8e2d7c314d96f2
+SHA512:
+  metadata.gz: 34f19cb0137d62bf6d5ded203ae279f3d885aa42b1e3536865aff772e84f70a925f23d2add0a4ecf3125747aa25b870a91eab97c2aa9c8a7b16cc8d300023d7e
+  data.tar.gz: e4d1196c9b3473289cd69c42ee586739f0fd317873f2492e7e8aaec59721a10cf08dfac551b6f2045d180d5b31b892856c46a071cab1c3aa2d22bab63e79290c

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+.DS_Store

data/.travis.yml

@@ -0,0 +1,13 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - 2.0.0
+  - 2.1.0
+  - 2.1.2
+branches:
+  only:
+    - master
+env:
+  global:
+  - secure: "J2/UOjDgJNI9qZ0sEQUKDb3iLctyvsAPb3s+pSxDF0FVG5nPx3kDUU0p5yYzr7xcvxlpWCVyjKOYJdA7eM9mupml1muwOH5adcU0nIxBtd8zshRw8SIvyUpjL1q7CeABHyi2QpbvwbYLE8bRngWMk3SqRHrqLEIFUb6mpyfZsy8=" 
+

data/Gemfile

@@ -0,0 +1,5 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in rawscsi.gemspec
+gem "codeclimate-test-reporter", :group => :test, :require => nil
+gemspec

data/LICENSE.txt

@@ -0,0 +1,23 @@
+Copyright (c) 2014 Steven Li
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/README.md

@@ -0,0 +1,288 @@
+# Rawscsi
+
+[![Build Status](https://travis-ci.org/StevenJL/rawscsi.svg)](https://travis-ci.org/StevenJL/rawscsi)
+[![Code Climate](https://codeclimate.com/github/StevenJL/rawscsi/badges/gpa.svg)](https://codeclimate.com/github/StevenJL/rawscsi)
+[![Test Coverage](https://codeclimate.com/github/StevenJL/rawscsi/badges/coverage.svg)](https://codeclimate.com/github/StevenJL/rawscsi)
+
+Ruby Amazon Web Services Cloud Search Interface (Rawscsi) is a flexible gem for searching and indexing AWS Cloud Search with the following characteristics:
+
+1) Maximal Flexibility: Rawscsi can construct very complex queries.  For example, using Cloud Search's default movies search domain, Rawscsi can find the top three James Bond movies excluding 'Casino', staring either 'Connery', 'Craig', or 'Moore', but not 'Brosnan', released after 1970. This is how `rawscsi` would construct this query:
+
+```ruby
+search_object.search(q: {
+                          and: [
+                                 { plot: "James Bond" }, 
+                                 { not: { 
+                                          title: "Casino"
+                                        },
+                                 },
+                                 { or: [
+                                         { actor: "Connery" },
+                                         { actor: "Craig" },
+                                         { actor: "Moore"}
+                                       ]
+                                 },
+                                 { not: {
+                                          actor: "Brosnan"
+                                        }
+                                 }]
+                         },
+                      date: {release_date: "[1970-01-01,}"},
+                      sort: "rating desc",
+                      limit: 3
+                    )
+```
+In theory, if the query is allowed on Cloudsearch, this gem can construct it.
+
+2) Supports Cloud Search Version api 2013-01-01.
+
+3) Smart indexing of data to a search domain. Namely, it calculates the size of your batch and breaks it up into chunks, each of which is less than 5Mb (the upload limit).
+
+4) Has some nice Active Record integration features.
+
+5) Supports Ruby 1.8.7, 1.9.3, 2.0.0, 2.1.0, and 2.1.2
+
+Here's the official aws cloud search documentation: http://docs.aws.amazon.com/cloudsearch/latest/developerguide/what-is-cloudsearch.html
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'rawscsi'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install rawscsi
+
+### Registering Search Domains 
+
+Suppose we have two search domains: Songs and Books.  Let's say Song is an `active_record` model in our project. We first register them to `Rawscsi`.
+
+```ruby
+Rawscsi.register 'Song' do |config|
+  config.domain_name = 'good-songs'
+  config.domain_id = 'akldfjakljf3894fjeaf9df'
+  config.region = 'us-east-1'
+  config.api_version = '2013-01-01' # rawscsi only supports api version 2013-01-01
+  config.attributes = [:title, :album, :artist, :release_date] # since Song is an active model, we can specify the attributes we want indexed
+end
+
+Rawscsi.register 'Book' do |config|
+  config.name = 'good-books'
+  config.id = 'dj43g6i77dof86lk34fsf2s'
+  config.region = 'us-east-1'
+  config.api_version = '2013-01-01' # rawscsi only supports api version 2013-01-01
+end
+```
+
+### Uploading indices
+```ruby
+song_indexer = Rawscsi::Index.new('Song', :active_record => true)
+# Since Song is an active record object and we specified the attributes in the config, we can use the active record option
+# Then we can just upload active record objects directly. Note this will use the active record id as the cloud search record id.
+song_indexer.upload([
+  # active record objects
+  <id: 4567, title: "Saturdays Reprise", artist: "Cut Copy", album: "Bright Like Neon Love">
+  <id: 5456, title: "Common Burn", artist: "Mazzy Star", album: "Seasons of Your Day">
+  <id: 5346, title: "Honey Power", artist: "My Bloody Valentine">
+  <id: 5762, title: "Introduction", artist: "Nick Drake">
+])
+
+book_indexer = Rawscsi::Index.new('Book')
+# since Book is not an active record object, we upload hashes instead
+# Note the id key in the hash  will be used as the cloud search record id.
+book_indexer.upload([
+  {id: 1234, title: "Surely you're Joking, Mr. Feynman", author: "Richard Feynman"},
+  {id: 5546, title: "Zen and the Art of Motorcycle Maintanence", author: "Robert Pirsig"}
+])
+```
+
+### Deleting indices
+```ruby
+song_indexer.delete([3244, 53452, 5435, 64545, 34342, 4545])
+# To delete records in the search domain, you just pass an array of ids (the cloud search record id)
+```
+
+##### Automatically Batches on cloud search's 5Mb Upload Limit 
+Rawscsi is smart enough to break up large batch uploads into chunks of 5 Mb (cloud search's upload size limit per post request).
+
+### Searching
+Instantiate a search object
+
+```ruby
+book_search_helper = Rawscsi::Search.new('Book')
+
+song_search_helper = Rawscsi::Search.new('Song', :active_record => true)
+# use the 'active record' option if 'Song' is a Active Record model in your project.
+# Calling search will return an array of active record objects ordered by cloud search's rank score.
+```
+
+#### Simple Search
+Simple search queries take one string as an argument and searches all text and text-array fields on the search domain.
+
+```ruby
+search_books_helper.search('Richard Feynman')
+  => [{id: 546, author: "Richard Feynman"  title: "Surely, You're Joking Mr. Feynman" },
+      {id: 657, author: "Richard Feynman", title: "What Do You Care What Other People Think"}]
+
+search_songs_helper.search('Air')
+# since we initiated this search helper with the active record option
+# it returns an array of active record objects
+  => [<Song id:156, artist: "White Stripes", title: "The Air Near My Fingers">,
+      <Song id:342, artist: "Air", title: "Mer du Japon">]
+```
+#### Compound Boolean Searches
+Compound search queries are queries consisting of multiple constraints, linked together by either `and` or `or` or some combination of the two. Rawscsi's philosophy in this aspect is maximal flexibility: Any query that is possible on cloud search, should be possible to construct in Rawscsi.
+
+Lets look at some sample compound boolean queries.
+
+```ruby
+search_songs_helper.search(q: {
+                                and: [{artist: "Daft Punk"}]
+                              }
+                           )
+# Even though we're searching only one term, it's not a simple query because we're looking
+# at just the artist field, not all text fields. Also note, even though its only one constrainst, we still use 'and'.
+```
+
+```ruby
+search_songs_helper.search(q: {
+                                and: [{ title: 'Angel' }, { artist: 'Smith' }]
+                              })
+# Conjunction of two constraints is done using the `and` key
+=> [{song_id: 12345, title: "Angel in the Snow", artist: "Elliot Smith"},
+    {song_id: 43534, title: "Angel, Angel Down We Go Together", artist: "The Smiths"}]
+```
+By default, the search returns all the fields in the domain.  But you can specify which fields to return. 
+
+```ruby
+search_songs.search(q: {and: [ {artist: "Cold Play"} ], 
+                    fields: [:title])
+
+=> [{ title: "Warning Sign"},
+    { title: "God Put a Smile Upon Your Face"},
+    { title: "Sparks"}]
+
+```
+The default sort order is Amazon Cloudsearch's rank-score but you can use your own sort as well as a limiting the number of results.
+```ruby
+search_songs.search(q: {and: [{genres: "Rock"}]}, sort: "rating desc", limit: 100)
+# Top 100 Rock songs.
+```
+
+Here is an example of using a date constraint in conjunction ("and")  with other constraints:
+```ruby
+search_songs.search(q: {
+                         and: [
+                                { genres: "Hip Hop" }, 
+                                { title: "Street"}
+                               ]
+                         },
+                    date: { release_date: "[1995-01-01,}"}
+                    limit: 5,
+                    sort: "rating desc"
+                   ) 
+# Conjunction of two constraints and date constraint (Top 5 Hip Hop songs with Street in title released after 1995)
+# Note date syntax is working in conjunction (and) with the frist two constraints. This is always the case.
+# Also note syntax for searching date ranges: "[1995-01-01,}" is all dates after Jan 1 1995 while "{,1995-01-01]" is all dates before.
+# Note we're limiting only 5 results and sorting by rating in descending order.
+=> [{song_id: 54642, title: "Street Dreams", artist: "Nas"},
+    {song_id: 98786, title: "Street Struck", artist: "Big L"},
+    {song_id: 54645, title: "Street Disputes", artist: "Wu-Tang Clan"},
+    {song_id: 54542, title: "Streets is Watching", artist: "Jay-Z"},
+    {song_id: 54644, title: "Street Corners", artist: "Wu-Tang Clan"}]
+```
+
+So far, we've only seen 'and' examples.  Now let's look at some 'or' examples. 
+
+```ruby
+search_songs.search(q: {
+                         or: [
+                               { artist: "Digitalism"},
+                               { artist: "Daft Punk"}, 
+                               { artist: "Justice"}
+                             ]
+                        }
+                    )
+# Find the union of all the songs by these awesome electro-house bands
+```
+```ruby
+# you can also combine `and` and `or` constraints
+search_songs.search(q: {
+                          and: [
+                                  { range: "rating:['9',}"}, 
+                                  { or: [
+                                          { artist: "Bob Dylan" },
+                                          { artist: "Lorde"}
+                                        ]
+                                  }
+                                ]
+                         }
+                     )
+# You only want highly rated songs by either Bob Dylan or Lorde (cuz you know, Lorde is the new Bob Dylan)
+```
+
+And of course, negation:
+
+```ruby
+# You love the song "All Along the Watchtower" but you didn't like the Dave Matthews Band cover 
+search_songs.search(q: {
+                         and: [ 
+                                { title: "All Along the Watchtower"},
+                                  not: {
+                                          artist: "Dave Matthews Band"
+                                       }
+                              ]
+                        }
+                    )
+```
+
+By default, the search returns all the fields in the domain.  But you can specify which fields to return. 
+
+```ruby
+search_songs.search(q: {and: [ {artist: "Cold Play"} ], 
+                    fields: [:title])
+
+=> [{ title: "Warning Sign"},
+    { title: "God Put a Smile Upon Your Face"},
+    { title: "Sparks"}]
+
+```
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+## License
+
+(The MIT License)
+
+Copyright © 2014 Steven Li
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+‘Software’), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED ‘AS IS’, WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/Rakefile

@@ -0,0 +1,11 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+task :default => :spec
+
+RSpec::Core::RakeTask.new
+
+namespace :spec do
+  Rake::Task["spec"].invoke
+end
+

data/lib/rawscsi/base.rb

@@ -0,0 +1,13 @@
+module Rawscsi
+  class Base
+    attr_accessor :model 
+    attr_reader :config, :is_active_record
+
+    def initialize(model_name, options={})
+      @is_active_record = options[:active_record]
+      @model = model_name
+      @config = Rawscsi.registered_models[model_name]
+    end
+  end
+end
+

data/lib/rawscsi/index.rb

@@ -0,0 +1,64 @@
+require "json"
+
+module Rawscsi
+  class Index < Rawscsi::Base
+    def upload(obj_array)
+      batch_size = 0
+      current_batch = []
+
+      obj_array.each do |obj|
+        sdf = format_to_sdf(obj)
+        size = sdf.to_json.bytesize
+        if (batch_size + size) < max_size
+          current_batch << sdf
+          batch_size = batch_size + size
+        else
+          post_to_amazon(current_batch)
+          current_batch = []
+          batch_size = 0
+        end
+      end
+      post_to_amazon(current_batch)
+    end
+
+    def delete(id_array)
+      if id_array.length < 20000
+        delete_from_amazon(id_array)
+      else
+        id_array.slice(20000) do |sub_array|
+         delete_from_amazon(sub_array)
+        end
+      end
+    end
+
+    private 
+    def max_size
+      5240000 # bytes
+    end
+
+    def format_to_sdf(obj)
+      Rawscsi::IndexHelpers::SdfAdd.new(obj, config.attributes).build
+    end
+
+    def delete_from_amazon(id_array)
+      sdf_del_array = id_array.map do |id|
+        Rawscsi::IndexHelpers::SdfDelete.new(id).build
+      end
+      post_to_amazon(sdf_del_array)
+    end
+
+    def post_to_amazon(payload)
+      resp = connection.post do |req|
+        req.url "#{config.api_version}/documents/batch"
+        req.headers["Content-type"] = "application/json"
+        req.body = payload.to_json
+      end
+      resp.body
+    end
+
+    def connection
+      @connection ||= Rawscsi::IndexHelpers::Connection.new(config).build
+    end
+  end
+end
+

data/lib/rawscsi/index_helpers/connection.rb

@@ -0,0 +1,23 @@
+require "faraday"
+require "faraday_middleware"
+
+module Rawscsi
+  module IndexHelpers
+    class Connection
+      attr_reader :url
+      
+      def initialize(config)
+        @url= "http://doc-#{config.domain_name}-#{config.domain_id}.#{config.region}.cloudsearch.amazonaws.com"
+      end
+
+      def build
+        connection = Faraday.new url do |builder|
+          builder.use FaradayMiddleware::EncodeJson
+          builder.adapter Faraday.default_adapter
+        end
+        connection
+      end
+    end
+  end
+end
+

data/lib/rawscsi/index_helpers/sdf_add.rb

@@ -0,0 +1,47 @@
+module Rawscsi
+  module IndexHelpers
+    class SdfAdd
+      attr_reader :doc, :attributes
+
+      def initialize(doc, attributes=nil) 
+        @doc = doc
+        @attributes = attributes || doc.keys
+      end
+      
+      def build
+        {
+          :id => doc_id,
+          :type => "add",
+          :fields => fields
+        }
+      end
+
+      private
+      def doc_id
+        if doc.is_a?(Hash)
+          doc[:id]
+        else
+          "#{doc.class}_#{doc.id}"
+        end
+      end
+
+      def fields
+        output = {}
+        attributes.each do |attr|
+          next if attr == :id
+          output[attr] = get_attr(doc, attr)
+        end
+        output
+      end
+
+      def get_attr(doc, attr)
+        if doc.is_a?(Hash)
+          doc[attr]
+        else
+          doc.send(attr)
+        end
+      end
+    end
+  end
+end
+

data/lib/rawscsi/index_helpers/sdf_delete.rb

@@ -0,0 +1,19 @@
+module Rawscsi
+  module IndexHelpers
+    class SdfDelete
+      attr_reader :id
+
+      def initialize(id)       
+        @id = id
+      end
+  
+      def build
+        {
+          :type => "delete",
+          :id => id
+        }
+      end
+    end
+  end
+end
+

data/lib/rawscsi/query/compound.rb

@@ -0,0 +1,114 @@
+module Rawscsi
+  module Query
+    class Compound
+      attr_reader :query_hash
+      def initialize(query_hash)
+        @query_hash = query_hash
+      end
+
+      def build
+        [
+          query,
+          date,
+          sort,
+          limit,
+          fields,
+          "q.parser=structured"
+        ].compact.join("&")
+      end
+
+      private
+      def query
+      "q=" + compound_bool(query_hash[:q])
+      end
+
+      def date
+        return nil unless date_hash = query_hash[:date]
+        output_str = "fq="
+        date_hash.each do |k,v|
+          output_str << "#{k}:#{FRC3339(v)}"
+        end
+        encode(output_str)
+      end
+
+      def FRC3339(date_str)
+        return date_str if /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}/ =~ date_str
+        date_str.gsub(/\d{4}-\d{2}-\d{2}/) do |dt|
+          "#{dt}T00:00:00Z"
+        end
+      end
+
+      def sort
+        return nil unless query_hash[:sort]
+        encode("sort=#{query_hash[:sort]}")
+      end
+      
+      def limit
+        return nil unless query_hash[:limit]
+        "size=#{query_hash[:limit]}"
+      end
+
+      def fields
+        return nil unless fields_array = query_hash[:fields]
+        output = []
+        fields_array.each do |field_sym|
+          output << field_sym.to_s
+        end
+        "return=" + output.join(",")
+      end
+
+      def compound_bool(hash)
+        if compound?(hash)
+          stringify_compound(hash)
+        else
+          stringify_noncompound(hash)
+        end
+      end
+
+      def compound?(hash) 
+        ar = hash.keys
+        ar.include?(:and) || ar.include?(:or)
+      end
+
+      def stringify_compound(hash)
+        bool_op = hash.keys.first
+        ar = hash[bool_op]
+        "(#{bool_op}" + encode(" #{bool_map(ar)}") + ")"
+      end
+
+      def stringify_noncompound(hash)
+        if not_hash = hash[:not]
+          "(not" + encode(" #{stringify(not_hash)}") + ")"
+        elsif range = hash[:range]
+          range 
+        else
+          encode(stringify(hash))
+        end
+      end
+
+     def bool_map(array)
+        output = ""
+        array.each do |pred|
+          output << compound_bool(pred)
+        end
+        output
+      end
+
+      def stringify(hash)
+        output_str = ""
+        hash.each do |k,v|
+          output_str << "#{k}:'#{v}'"
+        end
+        output_str
+      end
+
+      def encode(str)
+        # URI and CGI.escape don't quite work here
+        # For example, I need blank space as %20, but they encode it as +
+        # So I have to write my own
+        str.gsub(' ', '%20').gsub("'", '%27').gsub("[", '%5B').gsub("]",'%5D').gsub("{", '%7B').gsub("}", '%7D')
+      end
+    end
+  end
+end
+

data/lib/rawscsi/query/simple.rb

@@ -0,0 +1,16 @@
+require "cgi"
+
+module Rawscsi
+  module Query
+    class Simple
+      def initialize(query_string)
+        @query_string = query_string
+      end
+
+      def build
+        "q=#{CGI.escape(@query_string.to_s)}"
+      end
+    end
+  end
+end
+