s3search 0.0.1 → 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f2f409bf1d5943a4bba14de2d68380fd42de29e5
+  data.tar.gz: fa9aae7844c80c61be5fca34bfbf1d18be16b3f3
+SHA512:
+  metadata.gz: 17497e746c81e2e54ab2690b64e4a27f8b98cca4f24a79772bbe81044cb70acec9d0c1ad95dc2c5b6e80e9e28917f46d61c0a236bd81588120dd544e5ce75672
+  data.tar.gz: 0d79ac7927201e5a74a407d865c83622b0e072f2ca3f966c0c79a4c24a04f163b6afaba16ab211d636401666e575c97166ec1f9ee940b8d0de3ee0606625f764

data/.gitignore

@@ -15,3 +15,5 @@ spec/reports
 test/tmp
 test/version_tmp
 tmp
+.powenv
+.rbenv-vars

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format progress

data/Gemfile

@@ -2,3 +2,6 @@ source 'https://rubygems.org'
 
 # Specify your gem's dependencies in s3search.gemspec
 gemspec
+
+gem 'rspec'
+gem 'webmock'

data/README.md

@@ -1,6 +1,12 @@
-# S3search
+# S3Search
 
-TODO: Write a gem description
+[S3Search](http://addons.heroku.com/s3search) is an [add-on](http://addons.heroku.com) for providing powerful document-based full-text indexing and search to your application.
+
+Adding S3Search to your application will allow you search your documents based upon the actual text within the documents, as well as any metadata fields you assign to them. Yes, that's right! S3Search will index the text inside your documents.
+
+Do you already have documents stored and want to index them and make them searchable? No worries. S3Search works by you sending it a URL to fetch the document content to index, along with a hash of metadata attributes to record against it.  You can them perform powerful queries against that indexed data, based on the rich features of [elasticsearch](http://www.elasticsearch.org).
+
+S3Search is a Heroku add-on.
 
 ## Installation
 
@@ -12,13 +18,155 @@ And then execute:
 
     $ bundle
 
-Or install it yourself as:
+## Provisioning the add-on
+
+S3Search can be attached to a Heroku application via the  CLI:
+
+<div class="callout" markdown="1">
+A list of all plans available can be found [here](http://addons.heroku.com/s3search).
+</div>
+
+```term
+    $ heroku addons:add s3search
+    -----> Adding s3search to sharp-mountain-4005... done, v18 (free)
+```
+
+Once S3Search has been added a `S3SEARCH_URL` setting will be available in the app configuration and will contain your custom URL to access the newly provisioned S3Search service instance. This can be confirmed using the `heroku config:get` command.
+
+```term
+    $ heroku config:get S3SEARCH_URL
+    https://user:pass@api.s3searchapp.com
+```
+
+After installing S3Search the application should be configured to fully integrate with the add-on.
+
+## Using with Rails 3.x
+
+Ruby on Rails applications will need to add the following entry into their `Gemfile` specifying the S3Search client library.
+
+```ruby
+    gem 's3search'
+```
+
+Update application dependencies with bundler.
+
+```term
+    $ bundle install
+```
+
+Write some application code to index some documents. Use the special field, `_content_url`, if you want to specify a location to download content and make it searchable. S3Search will download the content and make it searchable via the special field `_document_content`.
+
+```ruby
+    S3Search.create title: 'MyDocument', _content_url: 'https://s3-us-east-1.amazonaws.com/my_bucket/my_document.pdf'
+    S3Search.create name: 'Bob Lob Law', resume_id: 25, _content_url: 'https://s3-us-east-1.amazonaws.com/resumes.mycompany.com/bob.pdf'
+```
+
+The documents don't even really need to be in S3.
+
+```ruby
+    S3Search.create name: 'Bitcoin Pirate', resume_id: 42, _content_url: 'https://user:pass@authenticatedlocation.com/docs/jenny.pdf'
+    S3Search.create title: 'Bitcoin', author: 'santoshin@gmx.com', tags: ['bitcoin', 'manifesto'], _content_url: 'http://bitcoin.org/bitcoin.pdf'
+```
+
+The documents don't even really need to be documents! You can use S3Search to use its powerful search capability over just your custom metadata.
+
+```ruby
+    S3Search.create customer_id: 32, first_name: 'His Holiness', last_name: 'The Dalia Lama', religion: 'Buddhist', twitter_handle: '@DalaiLama'
+    S3Search.create customer_id: 99, first_name: 'George', middle_name: 'R. R.', last_name: 'Martin', job_title: 'Author'
+```
+
+Now retrieve some documents via the powerful query API.
+
+Search by a single metadata field
+
+```ruby
+    results = S3Search.search('title:MyDocument')
+```
+
+Search all metadata fields AND the content of the documents.
+
+```ruby
+    results = S3Search.search('bitcoin')
+```
+
+Search only the content of the documents.
+
+```ruby
+    results = S3Search.search('_document_content:bitcoin')
+```
+
+ Boost the search ranking of a certain field.
+
+```ruby
+    results = S3Search.search('bitcoin', boost: { title: 2.5 })
+```
+ Find a single document based on its unique id.
+
+```ruby
+    document = S3Search.get '833FCA4EEEF2943AC2D8E0'
+```
+## Monitoring & Logging
+
+Stats and the current state of S3Search can be displayed via the CLI.
+
+```term
+    $ heroku s3search:status
+    documents_indexed: 32842
+    index_size: 640MB
+```
+
+S3Search activity can be observed within the Heroku log-stream.
+
+```term
+    $ heroku logs -t | grep 's3search'
+```
+
+## Dashboard
+
+<div class="callout" markdown="1">
+For more information on the features available within the S3Search dashboard please see the docs at [heroku.s3searchapp.com/docs](heroku.s3searchapp.com/docs).
+</div>
+
+The S3Search dashboard allows you to view the current status of your S3Search cluster.
+
+The dashboard can be accessed via the CLI:
+
+```term
+    $ heroku addons:open s3search
+    Opening s3search for sharp-mountain-4005…
+```
+
+or by visiting the [Heroku apps web interface](http://heroku.com/myapps) and selecting the application in question. Select S3Search from the Add-ons menu.
+
+## Migrating between plans
+
+<div class="note" markdown="1">Application owners should carefully manage the migration timing to ensure proper application function during the migration process.</div>
+
+Use the `heroku addons:upgrade` command to migrate to a new plan.
+
+```term
+    $ heroku addons:upgrade s3search:newplan
+    -----> Upgrading s3search:newplan to sharp-mountain-4005... done, v18 ($49/mo)
+           Your plan has been updated to: s3search:newplan
+```
+
+## Removing the add-on
+
+S3Search can be removed via the CLI.
+
+<div class="warning" markdown="1">This will destroy all metadata and indexes stored in S3Search and cannot be undone! Of course, documents indexed in S3Search but stored elsewhere will remain untouched.</div>
+
+```term
+    $ heroku addons:remove s3search
+    -----> Removing s3search from sharp-mountain-4005... done, v20 (free)
+```
+
+Before removing S3Search a data export can be performed by contacting support@s3searchapp.com directly.
 
-    $ gem install s3search
+## Support
 
-## Usage
+All S3Search support and runtime issues should be submitted via on of the [Heroku Support channels](support-channels). Any non-support related issues or product feedback is welcome at feedback@s3searchapp.com.
 
-TODO: Write usage instructions here
 
 ## Contributing
 

data/integration/document_spec.rb

@@ -0,0 +1,13 @@
+require 'integration_helper'
+
+describe '/document' do
+
+  it 'is implemented correctly' do
+    document = S3Search.create title: 'blah'
+    expect(document.title).to eq('blah')
+
+    id = document.id
+
+    expect(S3Search.get(id).title).to eq('blah')
+  end
+end

data/integration/search_spec.rb

@@ -0,0 +1,70 @@
+require 'integration_helper'
+
+describe '/searches' do
+
+  it 'is implemented correctly' do
+
+    S3Search.get_all(per_page: 9999).map do |doc|
+      S3Search.delete doc.id
+    end
+
+    title_a = SecureRandom.hex
+    title_b = SecureRandom.hex
+    
+    a_docs = 10.times.map do
+      {title: title_a}
+    end
+
+    b_docs = 10.times.map do
+      {title: title_b}
+    end
+
+    a_doc_ids = S3Search.create_many(a_docs).map(&:id)
+    b_doc_ids = S3Search.create_many(b_docs).map(&:id)
+    all_doc_ids = a_doc_ids + b_doc_ids
+
+    S3Search.get_all(per_page: 20).each do |doc|
+      expect(all_doc_ids).to include(doc.id)
+    end
+
+    results = S3Search.simple_search "title:#{title_a}", per_page: 4
+    expect(results.size).to eq(4)
+    results.each do |doc|
+      expect(a_doc_ids).to include(doc.id)
+    end
+
+    results.next
+    expect(results.size).to eq(4)
+    results.each do |doc|
+      expect(a_doc_ids).to include(doc.id)
+    end
+    
+    results.next
+    expect(results.size).to eq(2)
+    results.each do |doc|
+      expect(a_doc_ids).to include(doc.id)
+    end
+    expect(results.next?).to be_false
+
+    results = S3Search.simple_search "title:#{title_b}", per_page: 999
+    expect(results.size).to eq(10)
+    results.each do |doc|
+      expect(b_doc_ids).to include(doc.id)
+    end
+
+    # begin
+    #   results.each do |result|
+    #     S3Search.delete result.id
+    #   end
+    # end while results.next
+
+    # document = S3Search.create title: 'blah'
+    # expect(document.title).to eq('blah')
+
+    # id = document.id
+
+    # expect(S3Search.get(id).title).to eq('blah')
+
+    # puts S3Search.search 'blah'
+  end
+end

data/lib/s3search/api/documents.rb

@@ -0,0 +1,37 @@
+require 's3search/result_page'
+
+module S3Search
+  module API
+    module Documents
+
+      def create data
+        raise 'use create_many for multiple documents' if data.is_a?(Array)
+        _post('/v1/documents.json', {data: data})
+      end
+
+      def create_many data
+        response = _post('/v1/documents.json', {data: [data].flatten})
+        response.results
+      end
+
+      def delete id
+        _delete "/v1/documents/#{id}.json"
+      end
+
+      def get id
+        _get "/v1/documents/#{id}.json"
+      end
+
+      def update id, attributes
+        _put("/v1/documents/#{id}.json", {document: attributes})
+      end
+
+      def get_all options={}
+        options[:page] ||= 1
+        options[:per_page] ||= 25
+        response = _get("/v1/documents.json", options)
+        S3Search::ResultPage.new(response, self)
+      end
+    end
+  end
+end

data/lib/s3search/api/searches.rb

@@ -0,0 +1,49 @@
+require 's3search/result_page'
+
+module S3Search
+  module API
+    module Searches
+
+      def simple_search search_term, options={}
+        if search_term.include?(':')
+          field, value = search_term.split(':')[0..1]
+          options.merge!({where: { field => value }})
+        else
+          options.merge!({where: {_all: search_term }})
+        end
+        search options
+      end
+
+      def search options
+        raise 'where: {field: "value"} is a required option' unless options[:where]
+        query_terms = options.delete(:where).map do |field, value|
+          {
+            query_string: {
+              default_field: field,
+              query: value,
+              default_operator: 'AND'
+            }
+          }
+        end
+        query = {
+          query: {
+            bool: {
+              must: query_terms
+            }
+          }
+        }
+
+        page = options.delete(:page) || 1
+        per_page = options.delete(:per_page) || 25
+
+        query.merge!(options)
+        response = _post '/v1/searches.json', {
+          search: query,
+          page: page,
+          per_page: per_page
+        }
+        S3Search::ResultPage.new(response, self)
+      end
+    end
+  end
+end

data/lib/s3search/client.rb

@@ -0,0 +1,77 @@
+require 'active_support/core_ext/object/try'
+require 'faraday'
+require 'faraday_middleware'
+require 'hashie'
+require 'logger'
+require 's3search/api/documents'
+require 's3search/api/searches'
+require 's3search/request_error'
+require 's3search/version'
+
+module S3Search
+  class Client
+    include S3Search::API::Documents
+    include S3Search::API::Searches
+
+    attr_reader :url, :http, :logger
+
+    def initialize
+      @url = ENV['S3SEARCH_URL'] || 'https://api.s3searchapp.com'
+      @http = Faraday.new(:url => @url) do |builder|
+        builder.response :mashify
+        builder.response :json, :content_type => /\bjson$/
+        builder.request :json
+        builder.request :basic_auth, ENV['S3SEARCH_API_KEY'], ENV['S3SEARCH_API_SECRET']
+        builder.options[:read_timeout] = 4
+        builder.options[:open_timeout] = 2
+        builder.adapter :excon
+      end
+
+      @http.headers = {
+        accept:  'application/json',
+        user_agent: "S3Search Ruby Gem #{S3Search::VERSION}",
+        "Content-Type" => "application/json"
+      }
+
+      @logger = Logger.new(STDOUT)
+
+      # 0=DEBUG, 1=INFO, 2=WARN, 3=ERROR, 4=FATAL
+      @logger.level = ENV['S3SEARCH_LOG_LEVEL'].try(:to_i) || 2
+
+      @logger.formatter = proc do |severity, datetime, progname, msg|
+        "[S3Search][#{severity}]: #{msg}\n"
+      end
+    end
+
+    def _delete path, params={}
+     request :delete, path, params
+    end
+
+    def _get path, params={}
+     request :get, path, params
+    end
+
+    def _post path, params={}
+     request :post, path, params
+    end
+
+    def _put path, params={}
+     request :put, path, params
+    end
+
+    def request method, path, params
+      response = http.send(method, path, params)
+
+      case response.status
+      when 200..299
+        response.body
+      when 404, 410
+        raise RequestError::NotFound.new(response)
+      when 401
+        raise RequestError::Unauthorized.new(response)
+      else
+        raise RequestError.new(response)
+      end
+    end
+  end
+end

data/lib/s3search/request_error.rb

@@ -0,0 +1,19 @@
+module S3Search
+  class RequestError < StandardError
+    attr_reader :response
+
+    def initialize(response)
+      @response = response
+      msg = "Error processing request: (#{response.status})! #{response.env[:method]} URL: #{response.env[:url]}"
+      msg << "\n Resp Body: #{response.body}"
+      super msg
+    end
+
+    class NotFound < RequestError
+    end
+
+    class Unauthorized < RequestError
+    end
+
+  end
+end

data/lib/s3search/result_page.rb

@@ -0,0 +1,53 @@
+module S3Search
+  class ResultPage
+
+    attr_reader :response, :results
+
+    def initialize response, client
+      @response = response
+      @results = response.results
+      @client = client
+    end
+
+    def next
+      return unless next?
+      change_page(links.next)
+    end
+
+    def next?
+      links.next
+    end
+
+    def prev
+      return unless prev?
+      change_page(links.prev)
+    end
+
+    def prev?
+      links.prev
+    end
+
+    def respond_to_missing? method_name, include_private=false
+      results.respond_to?(method_name, include_private) || response.respond_to?(method_name, include_private)
+    end
+
+    private
+
+    def change_page link
+      @response = @client._get(link)
+      @results = @response.results
+      S3Search::ResultPage.new(@response, @client)
+    end
+
+    def method_missing method_name, *args, &block
+      if results.respond_to?(method_name) 
+        results.send method_name, *args, &block
+      elsif response.respond_to?(method_name)
+        response.send method_name, *args, &block
+      else
+        super
+      end
+    end
+
+  end
+end

data/lib/s3search/version.rb

@@ -1,3 +1,3 @@
-module S3search
-  VERSION = "0.0.1"
+module S3Search
+  VERSION = "0.0.2"
 end

data/lib/s3search.rb

@@ -1,5 +1,21 @@
-require "s3search/version"
+require 'json'
+require 's3search/client'
 
-module S3search
-  # Your code goes here...
+module S3Search
+  class << self
+    def client
+      @client ||= S3Search::Client.new
+    end
+
+    def respond_to_missing? method_name, include_private=false
+      client.respond_to?(method_name, include_private)
+    end
+
+    private
+
+    def method_missing method_name, *args, &block
+      return super unless client.respond_to?(method_name)
+      client.send(method_name, *args, &block)
+    end
+  end
 end

data/s3search.gemspec

@@ -3,21 +3,27 @@ lib = File.expand_path('../lib', __FILE__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require 's3search/version'
 
-Gem::Specification.new do |spec|
-  spec.name          = "s3search"
-  spec.version       = S3search::VERSION
-  spec.authors       = ["Chris Aitchison"]
-  spec.email         = ["chris.aitchison@sodalis.com.au"]
-  spec.description   = %q{Ruby implementation of the S3Search API}
-  spec.summary       = %q{S3Search Ruby API}
-  spec.homepage      = "https://github.com/sodalis/s3search-ruby"
-  spec.license       = "MIT"
+Gem::Specification.new do |gem|
+  gem.name          = "s3search"
+  gem.version       = S3Search::VERSION
+  gem.authors       = ["Chris Aitchison"]
+  gem.email         = ["chris.aitchison@sodalis.com.au"]
+  gem.description   = %q{Ruby implementation of the S3Search API}
+  gem.summary       = %q{S3Search Ruby API}
+  gem.homepage      = "https://github.com/sodalis/s3search-ruby"
+  gem.license       = "MIT"
 
-  spec.files         = `git ls-files`.split($/)
-  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
-  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
-  spec.require_paths = ["lib"]
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(spec|integration)/})
+  gem.require_paths = ["lib"]
 
-  spec.add_development_dependency "bundler", "~> 1.3"
-  spec.add_development_dependency "rake"
+  gem.add_dependency 'activesupport', ">= 3.0.0"
+  gem.add_dependency('faraday', '~> 0.8')
+  gem.add_dependency('faraday_middleware', '~> 0.9.0')
+  gem.add_dependency('excon', '>= 0.16')
+  gem.add_dependency('hashie', '~> 1.2.0')
+
+  gem.add_development_dependency "bundler", "~> 1.3"
+  gem.add_development_dependency "rake"
 end

data/spec/acceptance/document_create_many_spec.rb

@@ -0,0 +1,20 @@
+require 'spec_helper'
+
+describe 'creating many documents at once' do
+
+  let(:docs){ (0..9).map {|i| {title: "Doc#{i}"} } }
+  
+  before do
+    stub_api_http :post, '/v1/documents.json', {data: docs} do
+      {
+        status: 201,
+        body: {results: docs.map{|doc| Hashie::Mash.new(doc.merge(id: SecureRandom.hex))} }
+      }
+    end
+    @result = S3Search.create_many docs
+  end
+
+  it 'returns the created documents' do
+    expect(@result.size).to eq(10)
+  end
+end

data/spec/acceptance/document_create_spec.rb

@@ -0,0 +1,21 @@
+require 'spec_helper'
+
+describe 'creating a simple document' do
+
+  let(:create_params){ {title: 'Simple Document'} }
+  let(:created_id) { 'zMitmokAR5O4utuyz7qXRw' }
+  before do
+    stub_api_http :post, '/v1/documents.json', {data: create_params} do
+      {
+        status: 201,
+        body: Hashie::Mash.new(create_params.merge(id: created_id))
+      }
+    end
+    @document = S3Search.create create_params
+  end
+
+  it 'returns the created document' do
+    expect(@document.title).to eq('Simple Document')
+    expect(@document.id).to eq(created_id)
+  end
+end

data/spec/acceptance/document_delete_spec.rb

@@ -0,0 +1,17 @@
+require 'spec_helper'
+
+describe 'deleting a document' do
+
+  let(:document_id) { '3cbv4PUcRcOGsHnvIX9lJQ' }
+  before do
+    stub_api_http :delete, "/v1/documents/#{document_id}.json" do
+      {
+        status: 204
+      }
+    end
+  end
+
+  it 'deletes the document' do
+    S3Search.delete document_id
+  end
+end

data/spec/acceptance/document_get_all_spec.rb

@@ -0,0 +1,102 @@
+require 'spec_helper'
+
+describe 'getting all documents' do
+  let(:document_jsons) { (1..27).map{ |i| Hashie::Mash.new(id: "id#{i}", _score: 1.0, name: "doc#{i}") } }
+    
+  def link page, per_page
+    "https://api.s3searchapp.test/v1/documents.json?page=#{page}&per_page=#{per_page}"
+  end  
+
+  it 'takes per_page and page params' do
+    stub_get_all 2, 3
+    results = S3Search.get_all page: 2, per_page: 3
+    expect_page(results, 2, 3)
+
+    expect(results.map(&:name)).to eq(['doc4', 'doc5', 'doc6'])
+  end
+
+  it 'allows paging with next and prev' do
+    stub_get_all 4, 2
+    page = S3Search.get_all page: 4, per_page: 2
+    expect_page(page, 4, 2)
+
+    stub_get_all 5, 2
+    page.next
+    expect_page(page, 5, 2)
+
+    page.prev
+    expect_page(page, 4, 2)
+
+    stub_get_all 3, 2
+    page.prev
+    expect_page(page, 3, 2)
+
+    stub_get_all 2, 2
+    page.prev
+    expect_page(page, 2, 2)
+
+    stub_get_all 1, 2
+    page.prev
+    expect_page(page, 1, 2)
+    
+    page.prev
+    expect_page(page, 1, 2)
+  end
+
+  it 'defaults to page: 1, per_page: 25' do
+    stub_get_all 1, 25
+    page = S3Search.get_all
+    expect_page(page, 1, 25)
+
+    stub_get_all 2, 25
+    page.next
+    expect_page(page, 2, 25)
+
+    page.next
+    expect_page(page, 2, 25)
+  end
+
+  def expect_page result, page, per_page
+    from = from(page, per_page)
+    to = to(page, per_page)
+
+    expect(result.size).to eq((to - from) + 1)
+    expect(result.from).to eq(from)
+    expect(result.to).to eq(to)
+    expect(result.first).to eq(document_jsons[from - 1])
+    expect(result.last).to eq(document_jsons[to - 1])
+  end
+
+
+  def from page, per_page
+    [(page - 1) * per_page + 1, 1].max
+  end
+
+  def to page, per_page, total=27
+    [page * per_page, total].min
+  end
+  
+  def stub_get_all page, per_page
+    total = 27
+    from = from(page, per_page)
+    to = to(page, per_page)
+    links = {
+      self: link(page, per_page)
+    }
+    links[:next] = link(page + 1, per_page) if to < total
+    links[:prev] = link(page - 1, per_page) if page > 1
+
+    stub_api_http :get, "/v1/documents.json?page=#{page}&per_page=#{per_page}" do
+      {
+        body: Hashie::Mash.new(
+          total: total,
+          from: from,
+          to: to,
+          results: document_jsons[(from - 1)..(to - 1)],
+          links: links
+        ),
+        status: 200
+      }
+    end
+  end
+end

data/spec/acceptance/document_get_spec.rb

@@ -0,0 +1,22 @@
+require 'spec_helper'
+
+describe 'deleting a document' do
+
+  let(:document_id) { 'ZenfyafgRRGWAR6pqF0DeA' }
+  before do
+    stub_api_http :get, "/v1/documents/#{document_id}.json" do
+      {
+        body: Hashie::Mash.new({id: document_id, name: 'Some Document', some_property: 42}),
+        status: 200
+      }
+    end
+    @document = S3Search.get document_id
+  end
+
+
+  it 'gets the document' do
+    expect(@document.id).to eq(document_id)
+    expect(@document.name).to eq('Some Document')
+    expect(@document.some_property).to eq(42)
+  end
+end

data/spec/acceptance/document_update_spec.rb

@@ -0,0 +1,6 @@
+require 'spec_helper'
+
+describe 'updating a document' do
+
+  it 'updates the document and returns the updated version'
+end

data/spec/acceptance/search_simple_spec.rb

@@ -0,0 +1,154 @@
+require 'active_support/core_ext/hash/deep_merge'
+
+describe 'search' do
+  
+  def link id, page, per_page
+    "https://api.s3searchapp.test/v1/searches/abcde.json?page=#{page}&per_page=#{per_page}"
+  end
+
+  let(:found_documents) { (1..70).map{|i| {id: SecureRandom.hex, name: "doc#{i}", _score: 1.0 / i}}}
+
+  let(:simple_request_body) {
+    { 
+      search: {
+        query: {
+          bool: {
+            must: [
+              {
+                query_string:{
+                  default_field: '_all',
+                  query: "searchterm",
+                  default_operator:"AND"
+                }
+              }
+            ]
+          }
+        }
+      },
+      page: 1,
+      per_page: 25
+    }
+  }
+  it 'finds by simple query' do      
+    stub_api_http :post, "/v1/searches.json", simple_request_body do
+      {
+        body: Hashie::Mash.new(
+          total: 70,
+          from: 1,
+          to: 25,
+          results: found_documents[0..24],
+          links: {
+            self: link('abcde', 1, 25),
+            next: link('abcde', 2, 25)
+          }
+        ),
+        status: 200
+      }
+    end
+
+    results = S3Search.simple_search 'searchterm'
+    expect(results.first).to eq(
+      {
+        "_score" => 1.0,
+        "id" => results[0].id,
+        "name" => "doc1"
+      }
+    )
+    expect(results.last).to eq(
+      {
+        "_score" => 0.04,
+        "id" => results[24].id,
+        "name" => "doc25"
+      }
+    )
+
+    stub_api_http :get, "/v1/searches/abcde.json?page=2&per_page=25" do
+      {
+        body: Hashie::Mash.new(
+          total: 70,
+          from: 26,
+          to: 50,
+          results: found_documents[25..49],
+          links: {
+            self: link('abcde', 2, 25),
+            next: link('abcde', 3, 25),
+            prev: link('abcde', 1, 25)
+          }
+        ),
+        status: 200
+      }
+    end
+
+    results.next
+    expect(results.first).to eq(
+      {
+        "_score" => results[0]._score,
+        "id" => results[0].id,
+        "name" => "doc26"
+      }
+    )
+    expect(results.last).to eq(
+      {
+        "_score" => results[24]._score,
+        "id" => results[24].id,
+        "name" => "doc50"
+      }
+    )
+
+  end
+
+  let(:highlight_options) {
+    {
+      highlight: {
+        fields: [:name],
+        tag: '<strong class="highlight">',
+        number_of_fragments: 3,
+        fragment_size: 150          
+      }
+    }
+  }
+  let(:highlight_request_body) {
+    simple_request_body.deep_merge(search: highlight_options)
+  }
+
+  let(:highlight_documents) { 
+    (1..70).map do |i| 
+      {
+        id: SecureRandom.hex, 
+        name: "doc#{i}", 
+        description: 'a searchterm here',
+        _score: 1.0 / i,
+        _highlight: {
+          _description: ['a <strong class="highlight">searchterm</strong> here']
+        }
+      }
+    end
+  }
+
+  it 'allows highlighting of fields' do
+    stub_api_http :post, "/v1/searches.json", highlight_request_body do
+      {
+        body: Hashie::Mash.new(
+          total: 70,
+          from: 1,
+          to: 25,
+          results: highlight_documents[0..24],
+          links: {
+            self: link('abcde', 1, 25),
+            next: link('abcde', 2, 25)
+          }
+        ),
+        status: 200
+      }
+    end
+    results = S3Search.simple_search 'searchterm', highlight_options
+    expect(results.first).to eq({
+      "_highlight" => {"_description"=>['a <strong class="highlight">searchterm</strong> here']},
+      "_score" => 1.0,
+      "description" => "a searchterm here",
+      "id" => results[0].id,
+      "name" => "doc1"
+      })
+  end
+  
+end

data/spec/integration_helper.rb

@@ -0,0 +1,10 @@
+require 's3search'
+
+ENV['S3SEARCH_URL'] = 'http://s3search.dev'
+
+RSpec.configure do |config|
+  config.treat_symbols_as_metadata_keys_with_true_values = true
+  config.run_all_when_everything_filtered = true
+  config.filter_run :focus
+  config.order = 'random'
+end

data/spec/spec_helper.rb

@@ -0,0 +1,30 @@
+require 'webmock/rspec'
+require 's3search'
+require 'active_support/core_ext/object/try'
+
+ENV['S3SEARCH_API_KEY'] = 'api_key'
+ENV['S3SEARCH_API_SECRET'] = 'api_secret'
+ENV['S3SEARCH_URL'] = 'https://@api.s3searchapp.test'
+
+RSpec.configure do |config|
+  config.treat_symbols_as_metadata_keys_with_true_values = true
+  config.run_all_when_everything_filtered = true
+  config.filter_run :focus
+  config.order = 'random'
+end
+
+def stub_api_http method, resource, body=nil, &block
+  stub_request(
+    method,
+    "https://api.s3searchapp.test#{resource}"
+  ).with(
+    body: body.try(:to_json),
+    headers: {
+      'Accept' => 'application/json',
+      'Content-Type' => 'application/json',
+      'Authorization' => 'Basic YXBpX2tleTphcGlfc2VjcmV0'
+    }
+  ).to_return(
+    yield
+  )
+end

metadata

@@ -1,20 +1,88 @@
 --- !ruby/object:Gem::Specification
 name: s3search
 version: !ruby/object:Gem::Version
-  version: 0.0.1
-  prerelease: 
+  version: 0.0.2
 platform: ruby
 authors:
 - Chris Aitchison
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-06-05 00:00:00.000000000 Z
+date: 2013-06-17 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  name: faraday
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.8'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.8'
+- !ruby/object:Gem::Dependency
+  name: faraday_middleware
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.9.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.9.0
+- !ruby/object:Gem::Dependency
+  name: excon
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0.16'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0.16'
+- !ruby/object:Gem::Dependency
+  name: hashie
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.2.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.2.0
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -22,7 +90,6 @@ dependencies:
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -30,17 +97,15 @@ dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 description: Ruby implementation of the S3Search API
@@ -51,42 +116,63 @@ extensions: []
 extra_rdoc_files: []
 files:
 - .gitignore
+- .rspec
 - Gemfile
 - LICENSE.txt
 - README.md
 - Rakefile
+- integration/document_spec.rb
+- integration/search_spec.rb
 - lib/s3search.rb
+- lib/s3search/api/documents.rb
+- lib/s3search/api/searches.rb
+- lib/s3search/client.rb
+- lib/s3search/request_error.rb
+- lib/s3search/result_page.rb
 - lib/s3search/version.rb
 - s3search.gemspec
+- spec/acceptance/document_create_many_spec.rb
+- spec/acceptance/document_create_spec.rb
+- spec/acceptance/document_delete_spec.rb
+- spec/acceptance/document_get_all_spec.rb
+- spec/acceptance/document_get_spec.rb
+- spec/acceptance/document_update_spec.rb
+- spec/acceptance/search_simple_spec.rb
+- spec/integration_helper.rb
+- spec/spec_helper.rb
 homepage: https://github.com/sodalis/s3search-ruby
 licenses:
 - MIT
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: -1043739705409496661
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: -1043739705409496661
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.23
+rubygems_version: 2.0.2
 signing_key: 
-specification_version: 3
+specification_version: 4
 summary: S3Search Ruby API
-test_files: []
+test_files:
+- integration/document_spec.rb
+- integration/search_spec.rb
+- spec/acceptance/document_create_many_spec.rb
+- spec/acceptance/document_create_spec.rb
+- spec/acceptance/document_delete_spec.rb
+- spec/acceptance/document_get_all_spec.rb
+- spec/acceptance/document_get_spec.rb
+- spec/acceptance/document_update_spec.rb
+- spec/acceptance/search_simple_spec.rb
+- spec/integration_helper.rb
+- spec/spec_helper.rb