elastomer-client 0.4.1 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 81dcb0ea764810dd5a824281170423608a0ab5f2
-  data.tar.gz: 089264a40b4574bccf618b35cc258266bb06ec94
+  metadata.gz: 1ef2b4c790e85cb9d9141c34ae1945b230cadeb8
+  data.tar.gz: 463229368a9644620aab366e4ad67efa393e47fa
 SHA512:
-  metadata.gz: fd7298fd7537c9bcdc07f7be3bae04bcd77c83c20cbd3b0c7e410d339cc55f2ecaebb4f9dd3bc53305705cd19a44311cbd301f4f813c172035e2f36a1d1dcf12
-  data.tar.gz: 8eb7574b104215ee1a66a346a29c10d8ee00ea1273a4a213c8756b8ebdb1f07a25c8fe118bccfdca073c6a607e4a77e1bb32e14f999d855e5e9a8b7795c05238
+  metadata.gz: 904e906d03a23949bc0725932be4d7aaf6cd23185ab5a382160c7611e6c500adc03283813068a3fdfca5d067253736b309c7a74767d0c1a1a0d28b6a5e689689
+  data.tar.gz: 0f0d9fcb4527e8a0b1e43df7fba7c2d5fb822c78e8270e4ec4e239d0439647bbe6afbe2f569093ec0cffed21623187b1420ee3d7c4b4ae5148f6dd8accbe7ef9

data/.gitignore

@@ -2,6 +2,7 @@
 /vendor/gems
 /.bundle
 /.rbenv-version
+/.ruby-version
 /vendor/cache/*.gem
 /coverage
 Gemfile.lock

data/.travis.yml

@@ -0,0 +1,12 @@
+language: ruby
+rvm:
+  - 2.1.1
+  - 1.9.3
+
+services:
+  - elasticsearch
+
+notifications:
+  email: false
+
+sudo: false

data/CHANGELOG.md

@@ -1,3 +1,18 @@
+## 0.5.0 (2015-01-21)
+- BREAKING: rename action.available notification to action.ping
+- Index Component
+  - client.index no longer requires a name
+- Documents Component
+  - client.docs no longer requires an index name
+  - added an `exists?` method
+  - added `termvector` and `multi_termvectors` methods
+  - added a `search_shards` method
+  - added an `mget` alias for `multi_get`
+- Adding more documentation
+- Rename client.available? to client.ping (aliased as available?)
+- Updating tests to pass with ES 1.4.X
+- Enabling regular scroll queries vi `Client#scroll`
+
 ## 0.4.1 (2014-10-14)
 - Support for index `_recovery` endpoint
 - Fix Faraday 0.8 support

data/README.md

@@ -1,16 +1,16 @@
-# Elastomer
+# Elastomer Client [![Build Status](https://travis-ci.org/github/elastomer-client.svg)](https://travis-ci.org/github/elastomer-client)
 
 Making a stupid simple ElasticSearch client so your project can be smarter!
 
 ## Getting Started
 
-Use [Boxen](https://setup.githubapp.com) to get your machine set up. Then:
+Get started by cloning and running a few scripts:
 
 ```
 $ git clone https://github.com/github/elastomer-client.git
 $ cd elastomer-client
 $ script/bootstrap
-$ script/testsuite
+$ script/test
 ```
 
 ## Client
@@ -75,10 +75,9 @@ index.delete
 
 #### Docs
 
-This decomposition is the most questionable, but it's a starting point. The
-[Docs](lib/elastomer/client/docs.rb) class handles the indexing and searching
-of documents. Each instance is scoped to an index and optionally a document
-type.
+The [Docs](lib/elastomer/client/docs.rb) class handles the indexing and
+searching of documents. Each instance is scoped to an index and optionally a
+document type.
 
 ```ruby
 require 'elastomer/client'

data/Rakefile

@@ -7,3 +7,24 @@ Rake::TestTask.new do |t|
 end
 
 task :default => :test
+
+namespace :actions do
+  desc "list valid actions"
+  task :list do
+    # there are two distinct :action declarations we need to find
+    # the regular expressions below capture both
+    #
+    #   [:action] = 'some.value'
+    #   :action => 'some.value'
+    #
+    list = %x(grep '\\[\\?:action\\]\\?\\s\\+=' `find lib -name '*.rb'`).split("\n")
+    list.map! do |line|
+      m = /\A.*?\[?:action\]?\s+=>?\s+'(.*?)'.*\Z/.match line
+      m.nil? ? nil : m[1]
+    end
+
+    list.compact.sort.uniq.each do |action|
+      STDOUT.puts "- #{action}"
+    end
+  end
+end

data/docs/README.md

@@ -0,0 +1,44 @@
+# Elastomer Client in Depth
+
+We first started building the Elastomer Client gem when an
+[official client](https://github.com/elasticsearch/elasticsearch-ruby)
+was not yet available from ElasticSearch. We were looking for a client that
+provided a one-to-one mapping of the ElasticSearch APIs and avoided higher level
+complexity such as connection pooling, round-robin connections, thrift support,
+and the like. We think these things these things are bettered handled at
+different layers and by other software libraries.
+
+Our goal is to keep our ElasticSearch client simple and then compose
+higher level functionality from smaller components. This is the UNIX philosophy
+in action.
+
+To that end we have tried to be as faithful as possible to the ElasticSearch API
+with our implementation. There are a few places where it made sense to wrap the
+ElasticSearch API inside Ruby idioms. One notable location is the
+[scan-scroll](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html)
+search type; the Elastomer Client provides a Ruby iterator to work with these
+types of queries.
+
+Below are links to documents describing the various components of the Elastomer
+Client library. Start with the core components - specifically the **Client**
+document. All the other components are built atop the client.
+
+**Core Components**
+
+* [Client](client.md)
+* [Index](index.md)
+* [Documents](docs.md)
+* [Cluster](cluster.md)
+* [Templates](templates.md)
+* [Warmers](warmers.md)
+
+**Bulk Components**
+
+* [Bulk Indexing](bulk_indexing.md)
+* [Multi-Search](multi_search.md)
+* [Scan/Scroll](scan_scroll.md)
+
+**Operational Components**
+
+* [Snapshots](snapshots.md)
+* [Notifications](notifications.md)

data/docs/bulk_indexing.md

@@ -0,0 +1,3 @@
+# Elastomer Bulk Indexing Component
+
+![constructocat](https://octodex.github.com/images/constructocat2.jpg)

data/docs/client.md

@@ -0,0 +1,240 @@
+# Elastomer Client Component
+
+All methods in the Elastomer Client gem eventually make an HTTP request to
+ElasticSearch. The [`Elastomer::Client`](https://github.com/github/elastomer-client/blob/master/lib/elastomer/client.rb)
+class is responsible for connecting to an ElasticSearch instance, making HTTP
+requests, processing the response, and handling errors. Let's look at the
+details of how `Elastomer::Client` handles HTTP communication.
+
+### Connecting
+
+We use the [Faraday](https://github.com/lostisland/faraday) gem for all HTTP
+communication. Faraday provides a uniform wrapper around several different HTTP
+clients allowing any of these clients to be used at runtime. Faraday also has
+the concept of *middlewares* that operate on the HTTP request and response. We
+use Faraday middleware to encode and decode JSON messages exchanged with
+ElasticSearch.
+
+Without any options the `Elastomer::Client` will connect to the default
+ElasticSearch URL `http://localhost:9200`. The `Net:HTTP` client from the Ruby
+standard library will be used.
+
+```ruby
+client = Elastomer::Client.new
+client.host  #=> 'localhost'
+client.port  #=> 9200
+client.url   #=> 'http://localhost:9200'
+```
+
+[Boxen](https://boxen.github.com) configures ElasticSearch to listen on port
+`19200` instead of the standard port. We can provide either the full URL or just
+a different port number if ElasticSearch is running on `localhost`.
+
+```ruby
+client = Elastomer::Client.new :port => 19200
+client.host  #=> 'localhost'
+client.port  #=> 19200
+client.url   #=> 'http://localhost:19200'
+
+client = Elastomer::Client.new :url => "http://localhost:19200"
+```
+
+ElasticSearch works best with persistent connections. We can use the
+`Net::HTTP::Persistent` adapter with Faraday.
+
+```ruby
+client = Elastomer::Client.new \
+  :port    => 19200,
+  :adapter => :net_http_persistent
+```
+
+We also want to configure the `:open_timeout` (for making the initial connection
+to ElasticSearch) and the `:read_timeout` (used to limit each request). The open
+timeout should be short - it defaults to 2 seconds. The read timeout should be
+longer, but it can vary depending upon the type of request you are making. Large
+bulk requests will take longer than a quick count query.
+
+The open timeout is configured once when the client is first created. The read
+timeout can be set for each request.
+
+```ruby
+client = Elastomer::Client.new \
+  :url          => "http:/localhost:19200",
+  :adapter      => :net_http_persistent,
+  :open_timeout => 1,
+  :read_timeout => 5
+
+client.get("/", :read_timeout => 1)
+```
+
+Because each library handles read timeouts differently, some caution must be
+used. Persistent connections might or might not be closed and reopened when the
+read timeout is reached. If the connection is left open and reused, then the
+returned data might actually be from a previous request. This can lead to all
+kinds of horrible data leaks.
+
+ElasticSearch provides an `X-Opaque-Id` request header. Any value set in this
+request header will be returned in the corresponding response header. This
+allows the client to correlate the response from ElasticSearch with the request
+that was submitted. We have written an
+[OpaqueId](https://github.com/github/elastomer-client/blob/master/lib/elastomer/middleware/opaque_id.rb)
+middleware that will abort any request if the `X-Opaque-Id` headers disagree
+between the request and the response. You can use this feature by setting
+the `:opaque_id` flag.
+
+```ruby
+client = Elastomer::Client.new \
+  :url       => "http://localhost:19200",
+  :adapter   => :net_http_persistent,
+  :opaque_id => true
+```
+
+If you are not using persistent connections, then you do not need to worry about
+`X-Opaque-Id` headers.
+
+### HTTP Methods
+
+The standard HTTP verbs - `head`, `get`, `put`, `post`, `delete` - are exposed
+as methods on the `Elastomer::Client` class. Each method accepts a path and a
+Hash of parameters. Some parameters are applied as path expansions, some are
+reserved, and the remainder are used as URL parameters. We'll look at the
+reserved parameters first.
+
+#### Reserved Parameters
+
+**:body**
+
+The value passed in as the `:body` parameter will be used as the body of the
+HTTP request. The HTTP `head` method does not support a request body and ignores
+this parameter. The other HTTP methods all support request bodies.
+
+The `:body` value will be converted into JSON format before being sent to
+ElasticSearch unless the body is a String or an Array. If the body is a String
+it is assumed to already be JSON formatted, and it is sent to ElasticSearch as
+is without any modifications. When the body is an Array then all the items are
+joined with a newline character `\n` and a trailing newline is appended; this
+supports [bulk](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html)
+indexing and [multi-search](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-multi-search.html)
+requests.
+
+**:read_timeout**
+
+The read timeout for the HTTP network calls can be changed on a per-call basis.
+If a `:read_timeout` is supplied then it will be used instead of the global read
+timeout configured during initialization. This is useful for large bulk
+operations that might take longer than normal.
+
+```ruby
+client.cluster.health \
+  :index           => "test-index-1",
+  :wait_for_status => "green",
+  :timeout         => "5s",
+  :read_timeout    => 7
+```
+
+In the example above we are waiting for the named index to reach a green state.
+The `:timeout` of 5 seconds is passed to ElasticSearch. This call will return
+after 5 seconds even if the index has not yet reached green status. So we set
+our network call timeout to 7 seconds to ensure we don't kill the request before
+ElasticSearch has responded.
+
+**:action** and **:context**
+
+Each method in the Elastomer client gem has its own `:action` value that is
+used in conjunction with the [notifications](notifications.md) layer. The
+`:action` parameter cannot be changed by the user. Instead you can provide a
+`:context` value to each method call. This will be passed unchanged to the
+notifications layer, and it is useful for tracking where an Elastomer client
+method is called from within your application.
+
+#### URL Handling
+
+URLs are generated by combining a URL template with values extracted from the
+parameters. The [Addressable](https://github.com/sporkmonger/addressable) gem is
+used for this URL template expansion.
+
+With the [`Addressable::Template`](https://github.com/sporkmonger/addressable#uri-templates)
+a typical search URL takes the form `{/index}{/type}/_search`. The `:index` and
+`:type` values are taken from the parameters Hash and combined with the template
+to generate the URL. The internal
+[`expand_path`](https://github.com/github/elastomer-client/blob/master/lib/elastomer/client.rb#L245)
+method handles the URL generation.
+
+Here are a few examples to better illustrate the concept.
+
+```ruby
+client.expand_path("{/index}{/type}/_search", {
+  :q => "*:*"
+})
+#=> "/_search?q=*:*"
+
+client.expand_path("{/index}{/type}/_search", {
+  :index => "twitter",
+  :q     => "*:*"
+})
+#=> "/twitter/_search?q=*:*"
+
+client.expand_path("{/index}{/type}/_search", {
+  :index => "twitter",
+  :type  => "tweet",
+  :q     => "*:*"
+})
+#=> "/twitter/tweet/_search?q=*:*"
+
+client.expand_path("{/index}{/type}/_search", {
+  :index       => "twitter",
+  :type        => ["tweet", "user"],
+  :q           => "*:*"
+  :search_type => "count"
+})
+#=> "/twitter/tweet,user/_search?q=*:*&search_type=count"
+```
+
+In the examples above the path elements are optional. We can force them to be
+required and non-empty using a slightly different template syntax. In the
+example below we are requiring the `:index` and `:type` parameters to be
+provided. If either of them are missing (or nil or an empty string) then an
+`ArgumentError` is raised.
+
+```ruby
+client.expand_path("/{index}/{type}/_search", {
+  :index => "twitter",
+  :q     => "*:*"
+})
+#=> raises an ArgumentError - :type is missing
+
+client.expand_path("/{index}/{type}/_search", {
+  :index => "twitter",
+  :type  => " ",
+  :q     => "*:*"
+})
+#=> raises an ArgumentError - :type is an empty string
+```
+
+And that is the basic concept of the `expand_path` method. The URL template
+pattern is used extensively in the Elastomer client code, so it is definitely
+worth knowing about.
+
+### Errors
+
+Invariably things will go wrong where computers and networks are involved. The
+Elastomer client code makes no attempt to retry an operation in the face of an
+error. However, it does classify errors into *fatal* and *retryable* exceptions.
+
+Each class that inherits from
+[`Elastomer::Client::Error`](https://github.com/github/elastomer-client/blob/master/lib/elastomer/client/errors.rb)
+has a `fatal?` method (and the inverse `retry?` method). If an exception is
+fatal, then the request is fundamentally flawed and should not be retried.
+Passing a malformed search query or trying to search an index that does not
+exist are both examples of fatal errors - the request will never succeed.
+
+If an error is not fatal then it can be retried. If the ElasticSearch cluster
+has a full search queue then any query will fail. It not the fault of the user
+or the query itself - ElasticSearch just needs more capacity. The query can be
+safely retried.
+
+Therein lies the rub, though. Retrying a search or any operation will continue
+to add load to a search cluster that might already be experiencing problems.
+Blindly retrying an operation might do more harm than good. It is left to the
+user to implement their own exponential back-off scheme or to implement some
+status / back-pressure system.

data/docs/cluster.md

@@ -0,0 +1,148 @@
+# Elastomer Cluster Component
+
+The cluster component deals with commands for managing cluster state and
+monitoring cluster health. All the commands found under the
+[cluster API](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster.html)
+section of the ElasticSearch documentation are implemented by the
+[`cluster.rb`](https://github.com/github/elastomer-client/blob/master/lib/elastomer/client/cluster.rb)
+module and the [`nodes.rb`](https://github.com/github/elastomer-client/blob/master/lib/elastomer/client/nodes.rb)
+module.
+
+## Cluster
+
+API endpoints dealing with cluster level information and settings are found in
+the [`Cluster`](lib/elastomer/client/cluster.rb) class. Each of these methods
+corresponds to an API endpoint described in the ElasticSearch documentation
+(linked to above). The params listed in the documentation can be passed to these
+methods, so we do not take too much trouble to enumerate them all.
+
+#### health
+
+The cluster [health API](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html)
+returns a very simple cluster health status report.
+
+```ruby
+require 'elastomer/client'
+client = Elastomer::Client.new :port => 19200
+
+# the current health summary
+client.cluster.health
+```
+
+You can wait for a *yellow* status.
+
+```ruby
+client.cluster.health \
+  :wait_for_status => "yellow",
+  :timeout         => "10s",
+  :read_timeout    => 12
+```
+
+And you can request index level health details. The default timeout for the
+health endpoint is 30 seconds; hence, we set our read timeout to 32 seconds.
+
+```ruby
+client.cluster.health \
+  :level        => "indices",
+  :read_timeout => 32
+```
+
+#### state & stats
+
+If you need something more than basic health information, then the
+[`state`](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-state.html)
+and [`stats`](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-stats.html)
+endpoints are the next methods to call. Please look through the API
+documentation linked to above for all the details. And you can play with these
+endpoints via an IRB session.
+
+```ruby
+# detailed cluster state information
+client.cluster.state
+
+# cluster wide statistics
+client.cluster.stats
+```
+
+#### settings
+
+Cluster behavior is controlled via the
+[settings API](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-update-settings.html).
+The settings can be retrieved, and some settings can be modified at runtime to
+control shard allocations, routing, index replicas, and so forth. For example,
+when performing a [rolling restart](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_rolling_restarts.html)
+of a cluster, disabling shard allocation between restarts can reduce the
+cluster recovery time.
+
+```ruby
+# disable all shard allocation
+client.cluster.update_settings :transient => {
+  "cluster.routing.allocation.enable" => "none"
+}
+
+# shutdown the local node
+client.nodes('_local').shutdown
+
+# restart the local node and wait for it to rejoin the cluster
+
+# re-enable shard allocation
+client.cluster.update_settings :transient => {
+  "cluster.routing.allocation.enable" => "all"
+}
+```
+
+#### extras
+
+We've added a few extras to the `cluster` module purely for convenience. These
+are not API mappings; they are requests we frequently make from our
+applications.
+
+```ruby
+# the list of all index templates
+client.cluster.templates
+
+# list all the indices in the cluster
+client.cluster.indices
+
+# list all nodes that are currently part of the cluster
+client.cluster.nodes
+```
+
+Using these methods we can quickly get the names of all the indices in the
+cluster. The `indices` method returns a hash of the index settings keyed by the
+index name.
+
+```ruby
+client.cluster.indices.keys
+```
+
+The same method can be used for getting all the template names, as well.
+
+## Nodes
+
+There are also node level API methods that provide stats and information for
+individual (or multiple) nodes in the cluster. We expose these via the `nodes`
+module in elastomer-client.
+
+```ruby
+require 'elastomer/client'
+client = Elastomer::Client.new :port => 19200
+
+# gather OS, JVM, and process information from the local node
+client.nodes("_local").info(:info => %w[os jvm process])
+```
+
+More than one node can be queried at the same time.
+
+```ruby
+client.nodes(%w[node-1.local node-2.local]).stats(:stats => %w[os process])
+```
+
+Or you can query all nodes.
+
+```ruby
+client.nodes("_all").stats(:stats => "fs")
+```
+
+Take a look at the source code documentation for all the API calls provided by
+elastomer-client.