flex 0.4.2 → 1.0.1

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2012 by Domizio Demichelis
+Copyright (c) 2012-2013 by Domizio Demichelis
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,12 +1,51 @@
 # Flex
 
-Flex is a complete ruby client for [ElasticSearch](http://www.elasticsearch.org). It introduces a new way to send any kind of request (not just search queries) to the ES server, and to manage any kind of response. You can transparently integrate it with **ActiveRecord** and **Mongoid** models with or without automatic synchronization, with or without **Rails**. It is fast and efficient, easy to use and customize.
+Flex is the ultimate ruby client for [elasticsearch](http://elasticsearch.org). It is powerful, fast and efficient, easy to use and customize.
 
-## Useful Links
+It covers ALL the elasticsearch API, and transparently integrates it with your app and its components, like `Rails`, `ActiveRecord`, `Mongoid`, `ActiveModel`, `will_paginate`, `kaminari`, `elasticsearch-mapper-attachments`, ...
 
-* [Documentation](./flex/wiki)
-* [Issues](./flex/issues)
-* [Pull Requests](./flex/pulls)
+It also implements and integrates very advanced features like chainable scopes, live-reindex, cross-model syncing, query fragment reuse, parent/child relationships, templating, self-documenting tools, detailed debugging, ...
+
+## Quick Start
+
+The Flex documentation is very complete and detailed, so starting from the right topic for you will save you time. Please, pick the starting point that better describes you below:
+
+### For Tire Users
+
+1. You may be interested to start from [Why you should use Flex rather than Tire](http://ddnexus.github.io/flex/doc/7-Tutorials/1-Flex-vs-Tire.html) that is a direct comparison between the two projects.
+
+2. Depending on your elasticsearch knowledge you can read below the "Elasticsearch Beginner" or the "Elasticsearch Expert" starting point sections.
+
+### For Flex 0.x Users
+
+1. If you used an old flex version, please start with [How to migrate from flex 0.x](http://ddnexus.github.io/flex/doc/7-Tutorials/2-Migrate-from-0.x.html).
+
+2. Depending on your elasticsearch knowledge you can read below the "Elasticsearch Beginner" or the "Elasticsearch Expert" sections.
+
+### For Elasticsearch Beginners
+
+1. You may want to start with the [Index and Search External Data](http://ddnexus.github.io/flex/doc/7-Tutorials/4-Index-and-Search-External-Data.md) tutorial, since it practically doesn't require any elasticsearch knowledge. It will show you how to build your own search application with just a few lines of code. You will crawl a site, extract its content and build a simple user interface to search it with elasticsearch.
+
+2. Then you may want to read the [Usage Overview](http://ddnexus.github.io/flex/doc/1-Flex-Project/2-Usage-Overview.html) page. Follow the links from there in order to dig into the topics that interest you the most.
+
+3. You will probably like the [flex-scopes](http://ddnexus.github.io/flex/doc/3-flex-scopes) that allows you to easy search, chain toghether and reuse searching scopes in pure ruby.
+
+### For Elasticsearch Experts
+
+1. Flex provides the full elasticsearch APIs as ready to methods. Just take a look at the [API Metods](http://ddnexus.github.io/flex/doc/2-flex/2-API-Methods.html) page to appreciate its completeness.
+
+2. Then you may want to read the [Usage Overview](http://ddnexus.github.io/flex/doc/1-Flex-Project/2-Usage-Overview.html) page. Follow the links from there in order to dig into the topics that interest you.
+
+3. If you are used to create complex searching logic, you will certainly appreciate the [Templating System](http://ddnexus.github.io/flex/doc/2-flex/3-Templating) that gives you real power with great simplicity.
+
+4. As an elasticsearch expert, you will certainly appreciate the [Live-Reindex](http://ddnexus.github.io/flex/doc/6-flex-admin/2-Live-Reindex.html) feature: it encapsulates the solution to a quite complex problem in just one method call.
+
+## Links
+
+* [Flex Project (Global Documentation)](http://ddnexus.github.io/flex/doc/)
+* [flex Gem (Specific Documentation)](http://ddnexus.github.io/flex/doc/2-flex)
+* [Issues](https://github.com/ddnexus/flex/issues)
+* [Pull Requests](https://github.com/ddnexus/flex/pulls)
 
 ## Branches
 
@@ -18,5 +57,5 @@ Special thanks for their sponsorship to [Escalate Media](http://www.escalatemedi
 
 ## Copyright
 
-Copyright (c) 2012 by [Domizio Demichelis](mailto://dd.nexus@gmail.com)<br>
-See [LICENSE](./flex/blob/master/LICENSE) for details.
+Copyright (c) 2012-2013 by [Domizio Demichelis](mailto://dd.nexus@gmail.com)<br>
+See [LICENSE](https://github.com/ddnexus/flex/blob/master/LICENSE) for details.

data/VERSION

@@ -1 +1 @@
-0.4.2
+1.0.1

data/flex.gemspec

@@ -2,20 +2,23 @@ require 'date'
 
 Gem::Specification.new do |s|
   s.name                      = 'flex'
-  s.summary                   = 'Ruby Client for ElasticSearch'
+  s.summary                   = 'The ultimate ruby client for elasticsearch'
   s.description               = <<-desc
-Flex is an innovative ruby client for ElasticSearch. With Flex your code will be clean, easy to write and read, and very short: "poetic-short". Flex is fast and efficient, easy to use and customize, and offers ActiveRecord, Mongoid and Rails integration.
+Flex is the ultimate ruby client for elasticsearch. It is powerful, fast and efficient, easy to use and customize.
+
+It covers ALL the elasticsearch API, and transparently integrates it with your app and its components, like Rails, ActiveRecord, Mongoid, ActiveModel, will_paginate, kaminari, elasticsearch-mapper-attachments, ...
+
+It also implements and integrates very advanced features like chainable scopes, live-reindex, cross-model syncing, query fragment reuse, parent/child relationships, templating, self-documenting tools, detailed debugging, ...
   desc
   s.homepage                  = 'http://github.com/ddnexus/flex'
-  s.authors                   = ["Domizio Demichelis"]
+  s.authors                   = ['Domizio Demichelis']
   s.email                     = 'dd.nexus@gmail.com'
-  s.executables               = ['flexes']
   s.extra_rdoc_files          = %w[README.md]
   s.require_paths             = %w[lib]
   s.files                     = `git ls-files -z`.split("\0")
   s.version                   = File.read(File.expand_path('../VERSION', __FILE__)).strip
   s.date                      = Date.today.to_s
-  s.required_rubygems_version = ">= 1.3.6"
+  s.required_rubygems_version = '>= 1.3.6'
   s.rdoc_options              = %w[--charset=UTF-8]
   s.post_install_message      = <<EOM
 ________________________________________________________________________________
@@ -23,22 +26,13 @@ ________________________________________________________________________________
                              FLEX INSTALLATION NOTES
 ________________________________________________________________________________
 
-In order to use Flex, a supported http-client must be installed on this system.
-
-The suported http-client gems are "patron" and "rest-client".
-
-You should install "patron" (a libcurl based gem developed in C) for best
-performances, or install "rest-client" if you cannot use libcurl on your system.
+New Documentation:  http://ddnexus.github.io/flex
 
-As an alternative you could eventually develop your own http-client interface
-and set the Flex::Configuration.http_client option.
+Upgrading Tutorial: http://ddnexus.github.io/flex/doc/7-Tutorials/2-Migrate-from-0.x.html
 
 ________________________________________________________________________________
 EOM
-  s.add_runtime_dependency 'multi_json', '~> 1.3.4'
-  s.add_runtime_dependency 'progressbar', '~> 0.11.0'
-  s.add_runtime_dependency 'prompter', '~> 0.1.5'
-  s.add_development_dependency 'irt', '~> 1.2.10'
-  s.add_development_dependency 'patron', '~> 0.4.18'
-  s.add_development_dependency 'rest-client', '~> 1.6.7'
+  s.add_runtime_dependency     'multi_json',  '>= 1.3.4'
+  s.add_runtime_dependency     'progressbar', '>= 0.11.0', '~> 0.12.0'
+  s.add_runtime_dependency     'dye',         '~> 0.1.4'
 end

data/lib/flex.rb

@@ -1,3 +1,4 @@
+require 'dye'
 require 'yaml'
 require 'ostruct'
 require 'erb'
@@ -6,429 +7,102 @@ require 'flex/logger'
 require 'flex/errors'
 require 'flex/utils'
 
-require 'flex/structure/indifferent_access'
-require 'flex/structure/mergeable'
+require 'flex/struct/prunable'
+require 'flex/struct/symbolize'
+require 'flex/struct/hash'
+require 'flex/struct/array'
+
+require 'flex/variables'
 
 require 'flex/result'
-require 'flex/result/collection'
+require 'flex/struct/paginable'
 require 'flex/result/document'
-require 'flex/result/source_document'
 require 'flex/result/search'
-require 'flex/result/source_search'
+require 'flex/result/multi_get'
 require 'flex/result/bulk'
 
-require 'flex/variables'
-
-require 'flex/template/base'
+require 'flex/template/common'
 require 'flex/template/partial'
+require 'flex/template/logger'
 require 'flex/template'
 require 'flex/template/search'
 require 'flex/template/slim_search'
 require 'flex/template/tags'
-require 'flex/template/info'
-
-require 'flex/manager'
 
 require 'flex/class_proxy/base'
-require 'flex/class_proxy/loader'
-require 'flex/class_proxy/model_sync'
-require 'flex/class_proxy/model'
-require 'flex/class_proxy/related_model'
+require 'flex/class_proxy/templates/search'
+require 'flex/class_proxy/templates/doc'
 
-require 'flex/instance_proxy/base'
-require 'flex/instance_proxy/model'
-require 'flex/instance_proxy/related_model'
+require 'flex/class_proxy/templates'
 
-require 'flex/loader'
-require 'flex/related_model'
-require 'flex/model'
+require 'flex/templates'
 
-require 'flex/http_clients/patron'
-require 'flex/http_clients/rest_client'
+require 'flex/http_clients/base'
+require 'flex/http_clients/loader'
 require 'flex/configuration'
 require 'flex/utility_methods'
 
 require 'progressbar'
 require 'flex/prog_bar'
+require 'flex/deprecation'
 
-module Flex
-
-  VERSION  = File.read(File.expand_path('../../VERSION', __FILE__)).strip
-  LIB_PATH = __FILE__.sub(/flex.rb$/, '')
-
-
-  # The following lines are autogenerated by Flex.info
-
-  #  ########## Flex.indices_exists ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  indices_exists:
-  #  - HEAD
-  #  - /<<index>>
-  #
-  #  Usage:
-  #  Flex.indices_exists :index => "flex_test_index"
-  #
-  def Flex.indices_exists(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.exist? ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  exist?:
-  #  - HEAD
-  #  - /<<index>>
-  #
-  #  Usage:
-  #  Flex.exist? :index => "flex_test_index"
-  #
-  def Flex.exist?(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.create_index ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  create_index:
-  #  - PUT
-  #  - /<<index>>
-  #  - settings:
-  #      number_of_shards: <<number_of_shards= 5 >>
-  #      number_of_replicas: <<number_of_replicas= 1 >>
-  #
-  #  Usage:
-  #  Flex.create_index :index              => "flex_test_index",
-  #                    :number_of_shards   => 5,
-  #                    :number_of_replicas => 1
-  #
-  def Flex.create_index(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.put_index ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  put_index:
-  #  - PUT
-  #  - /<<index>>
-  #  - settings:
-  #      number_of_shards: <<number_of_shards= 5 >>
-  #      number_of_replicas: <<number_of_replicas= 1 >>
-  #
-  #  Usage:
-  #  Flex.put_index :index              => "flex_test_index",
-  #                 :number_of_shards   => 5,
-  #                 :number_of_replicas => 1
-  #
-  def Flex.put_index(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.post_index ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  post_index:
-  #  - POST
-  #  - /<<index>>
-  #  - settings:
-  #      number_of_shards: <<number_of_shards= 5 >>
-  #      number_of_replicas: <<number_of_replicas= 1 >>
-  #
-  #  Usage:
-  #  Flex.post_index :index              => "flex_test_index",
-  #                  :number_of_shards   => 5,
-  #                  :number_of_replicas => 1
-  #
-  def Flex.post_index(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.get_settings ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  get_settings:
-  #  - GET
-  #  - /<<index>>/_settings
-  #
-  #  Usage:
-  #  Flex.get_settings :index => "flex_test_index"
-  #
-  def Flex.get_settings(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.put_mapping ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  put_mapping:
-  #  - PUT
-  #  - /<<index>>/<<type>>/_mapping
-  #  - <<type>>:
-  #      properties: <<properties>>
-  #
-  #  Usage:
-  #  Flex.put_mapping :properties => properties,        # required
-  #                   :type       => nil,
-  #                   :index      => "flex_test_index"
-  #
-  def Flex.put_mapping(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.get_mapping ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  get_mapping:
-  #  - GET
-  #  - /<<index>>/<<type>>/_mapping
-  #
-  #  Usage:
-  #  Flex.get_mapping :index => "flex_test_index",
-  #                   :type  => nil
-  #
-  def Flex.get_mapping(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.delete_mapping ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  delete_mapping:
-  #  - DELETE
-  #  - /<<index>>/<<type>>
-  #
-  #  Usage:
-  #  Flex.delete_mapping :index => "flex_test_index",
-  #                      :type  => nil
-  #
-  def Flex.delete_mapping(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.delete_index ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  delete_index:
-  #  - DELETE
-  #  - /<<index>>
-  #
-  #  Usage:
-  #  Flex.delete_index :index => "flex_test_index"
-  #
-  def Flex.delete_index(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.delete_by_query ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  delete_by_query:
-  #  - DELETE
-  #  - /<<index>>/<<type>>/_query
-  #
-  #  Usage:
-  #  Flex.delete_by_query :index => "flex_test_index",
-  #                       :type  => nil
-  #
-  def Flex.delete_by_query(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.bulk ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  bulk:
-  #  - POST
-  #  - /_bulk
-  #  - << lines >>
-  #
-  #  Usage:
-  #  Flex.bulk :lines => lines  # required
-  #
-  def Flex.bulk(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.count ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  count:
-  #  - GET
-  #  - /<<index>>/<<type>>/_count
-  #
-  #  Usage:
-  #  Flex.count :index => "flex_test_index",
-  #             :type  => nil
-  #
-  def Flex.count(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.stats ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  stats:
-  #  - GET
-  #  - /<<index>>/_stats/<<end_point= ~ >>
-  #
-  #  Usage:
-  #  Flex.stats :index     => "flex_test_index",
-  #             :end_point => nil
-  #
-  def Flex.stats(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.store ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  store:
-  #  - PUT
-  #  - /<<index>>/<<type>>/<<id>>
-  #
-  #  Usage:
-  #  Flex.store :id    => id,                # required
-  #             :type  => nil,
-  #             :index => "flex_test_index"
-  #
-  def Flex.store(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.put_store ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  put_store:
-  #  - PUT
-  #  - /<<index>>/<<type>>/<<id>>
-  #
-  #  Usage:
-  #  Flex.put_store :id    => id,                # required
-  #                 :type  => nil,
-  #                 :index => "flex_test_index"
-  #
-  def Flex.put_store(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.post_store ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  post_store:
-  #  - POST
-  #  - /<<index>>/<<type>>
-  #
-  #  Usage:
-  #  Flex.post_store :index => "flex_test_index",
-  #                  :type  => nil
-  #
-  def Flex.post_store(vars={})
-    # this is a stub, used for reference
-  end
-
-
-  #  ########## Flex.remove ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  remove:
-  #  - DELETE
-  #  - /<<index>>/<<type>>/<<id>>
-  #
-  #  Usage:
-  #  Flex.remove :id    => id,                # required
-  #              :type  => nil,
-  #              :index => "flex_test_index"
-  #
-  def Flex.remove(vars={})
-    # this is a stub, used for reference
-  end
+require 'flex/api_stubs'
+require 'flex/tasks'
 
+module Flex
 
-  #  ########## Flex.get ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  get:
-  #  - GET
-  #  - /<<index>>/<<type>>/<<id>>
-  #
-  #  Usage:
-  #  Flex.get :id    => id,                # required
-  #           :type  => nil,
-  #           :index => "flex_test_index"
-  #
-  def Flex.get(vars={})
-    # this is a stub, used for reference
-  end
+  VERSION   = File.read(File.expand_path('../../VERSION', __FILE__)).strip
+  LIB_PATHS = [ File.dirname(__FILE__) ]
 
 
-  #  ########## Flex.multi_get ##########
-  #
-  #  Flex::Template:
-  #  ---
-  #  multi_get:
-  #  - GET
-  #  - /<<index>>/<<type>>/_mget
-  #  - ids: << ids >>
-  #
-  #  Usage:
-  #  Flex.multi_get :ids   => ids,               # required
-  #                 :type  => nil,
-  #                 :index => "flex_test_index"
-  #
-  def Flex.multi_get(vars={})
-    # this is a stub, used for reference
-  end
+  include ApiStubs
 
-  include Loader
-  flex.load_source File.expand_path('../flex/api_methods.yml', __FILE__)
+  include Templates
+  flex.load_source File.expand_path('../flex/api_templates/core_api.yml'   , __FILE__)
+  flex.load_source File.expand_path('../flex/api_templates/indices_api.yml', __FILE__)
+  flex.load_source File.expand_path('../flex/api_templates/cluster_api.yml', __FILE__)
 
   extend self
   extend UtilityMethods
 
-  def reload!
-    flex.variables = Configuration.variables.deep_dup
-    Loader.host_classes.each {|c| c.flex.reload!}
-    true
-  end
-
-  def info(*args)
-    flex.info *args
+  flex.wrap :post_bulk_string, :bulk do |*vars|
+    vars = Vars.new(*vars)
+    return if vars[:bulk_string].nil? || vars[:bulk_string].empty?
+    super vars
+  end
+
+  # get a document without using the get API (which doesn't support fields '*')
+  flex.wrap :search_by_id do |*vars|
+    vars   = Vars.new(*vars)
+    result = super(vars)
+    doc    = result['hits']['hits'].first
+    class << doc; self end.class_eval do
+      define_method(:raw_result){ result }
+    end
+    doc
+  end
+
+  # support for live-reindex
+  flex.wrap :store, :put_store, :post_store do |*vars|
+    vars         = Vars.new(*vars)
+    vars[:index] = LiveReindex.prefix_index(vars[:index]) if LiveReindex.should_prefix_index?
+    result = super(vars)
+    if LiveReindex.should_track_change?
+      vars.delete(:data)
+      LiveReindex.track_change(:index, dump_one(vars))
+    end
+    result
+  end
+
+  # support for live-reindex
+  flex.wrap :delete, :remove do |*vars|
+    vars         = Vars.new(*vars)
+    vars[:index] = LiveReindex.prefix_index(vars[:index]) if LiveReindex.should_prefix_index?
+    if LiveReindex.should_track_change?
+      vars.delete(:data)
+      LiveReindex.track_change(:delete, dump_one(vars))
+    end
+    super(vars)
   end
 
 end