agnostic_backend 0.9.0

data/doc/indexable.md

@@ -0,0 +1,292 @@
+# Indexable module Guide
+
+Broadly speaking, the `Indexable` module provides classes with
+functionality related to the following:
+
+- define what should be indexed (attributes names/values, nested
+  fields, field types etc.)
+- define who should be notified when an object (or its owner) needs to
+  be indexed (when the object changes in some way)
+- when should the above notifications occur
+
+In our examples below, we are working with `ActiveRecord` models, but
+`AgnosticBackend` can be used with any object. Also, whenever we
+mention the word "document" below, we take this to be a Ruby Hash.
+
+## Document contents
+
+Say we need to represent a workflow comprising a sequence of tasks
+within a case, using a `Workflow` AR model and a `Task` AR model
+connected by a one-to-many relationship, as follows:
+
+```ruby
+class Task < ActiveRecord::Base
+  belongs_to :workflow, class_name: 'Workflow'
+end
+
+class Workflow < ActiveRecord::Base
+  has_many :tasks, class_name: 'Task'
+end
+```
+
+We can specify what to index in the `Task` model by including
+`AgnosticBackend::Indexable` and using the `define_index_fields`
+method as follows:
+
+```ruby
+class Task < ActiveRecord::Base
+  include AgnosticBackend::Indexable
+  belongs_to :workflow, class_name: 'Workflow'
+
+  define_index_fields do
+    integer :id
+    date :last_assigned_at, value: :assigned_at
+    string :type, value: proc { task_category.name }
+  end
+end
+```
+
+In this case, we specify that the document to be generated when the
+time comes (more about that below) will include 3 fields: `id`,
+`last_assigned_at` and `type`. Let's look at each one of them in more
+detail. The document will contain a field with the key `id` and the
+value that will be generated when the object receives the message
+`:id` at document generation time. This means that in the simplest
+possible case, the field's key is a method to which we expect the
+object to respond. The document will also contain the
+`last_assigned_at` key whose value will be determined by sending the
+message `assigned_at` to the object. Finally, the document will also
+contain the field `type` which in this case is a computed value that
+will be determined at runtime at executing the specified `proc` in the
+context of `self` (i.e. in the context of the class's instance).
+
+## Nested documents
+
+`Indexable` supports the specification of nested documents by using
+the `struct` type as follows:
+
+```ruby
+class Task < ActiveRecord::Base
+  include AgnosticBackend::Indexable
+  belongs_to :workflow, class_name: 'Workflow'
+
+  define_index_fields do
+    integer :id
+    date :last_assigned_at, value: :assigned_at
+    string :type, value: proc { task_category.name }
+    struct :workflow, from: Workflow
+  end
+end
+```
+
+As a result, the document will also contain a `workflow` field whose
+value will be derived by requesting a document from the object's
+`workflow` reference (which is a `Workflow` instance). In order to get
+this to work, we need a corresponding definition in the `Workflow`
+class as follows:
+
+```
+class Workflow < ActiveRecord::Base
+  include AgnosticBackend::Indexable
+  has_many :tasks, class_name: 'Task'
+
+  define_index_fields(owner: Task) do
+    integer :id
+    date :created_at
+    text_array :notes, value: proc { notes.map(&:body) }
+  end
+end
+```
+
+Notice the use of the `owner: Task` argument in
+`define_index_fields`. This means that the document specified within
+the block is to be used only when requested by the `Task`'s document
+generation process. It also implies that we can specify multiple
+document definitions in the same class for different owners. When the
+owner is not specified, it is taken to be the class in which the
+definition is written.
+
+## Document Generation
+
+Use the `Indexable#generate_document` method in order to obtain a hash
+with the document's contents. For example, given a `Task` instance:
+
+```ruby
+> task.generate_document
+{:id=>5, :last_assigned_at=>2016-02-09 19:45:00 UTC, ...,
+ :workflow=>{:id=>6, ...}}
+```
+
+The document includes all fields specified in `Task` including the
+nested hash retrieved from `Workflow`.
+
+## When should a document be indexed
+
+`Indexable` does not specify when and in what way a document should be
+indexed. Instead, this decision is up to the client. The objective is
+to achieve the maximum flexibility with regard to different
+requirements, some of which are summarized below:
+
+- when the class is an AR model, the client would like to use AR
+  callbacks (such as `after_save` or `after_commit`) to index the
+  model.
+- the client may wish to implement document indexing in an
+  asynchronous manner for performance reasons.
+- the client may wish to decide whether to index the document only if
+  certain conditions are met.
+
+The entry point to indexing a document is
+`Indexable::InstanceMethods#trigger_index_notification`, which is
+responsible for notifying whoever has been registered in one or many
+`define_index_notifiers` blocks within the class. The message
+`:index_object` is sent to all objects that need to be notified. By
+default, `Indexable::InstanceMethods#index_object` will delegate to
+`Indexable::InstanceMethods#put_in_index` which will index the
+document synchronously.
+
+`Indexable::InstanceMethods#index_object` can be overriden in order to
+implement custom behaviour (such as asynchronous indexing through
+queueing). Any call to `put_in_index` will index
+
+- `Indexable::InstanceMethods#trigger_index_notification` must be used
+  in order to notify all registered objects
+- `Indexable::InstanceMethods#index_object`, by default, will index
+  the document synchronously (by calling
+  `Indexable::InstanceMethods#put_in_index`). `index_object` needs to
+  be overriden in order to implement custom behaviour (such as
+  asynchronous indexing)
+- `Indexable::InstanceMethods#put_in_index` is the method that
+  implements the actual indexing. Any custom implementation of
+  `index_object` must ultimately call `put_in_index` for indexing to
+  occur.
+
+
+## Field Types
+
+`Indexable` supports the following generic types:
+
+- `:integer`
+- `:double`
+- `:string`: this is a literal string (i.e. should be matched exactly)
+- `:string_array`: an array of literal strings
+- `:text`: text that can be interpreted as free text by a specific backend
+- `:text_array`: an array of text fields
+- `:date`: datetime field
+- `:boolean`
+- `:struct`: used to specify a nested structure
+
+## Document Schemas
+
+The specification of types in the definition of index fields implies
+that we can derive the document schema using the `Indexable#schema`
+method. E.g. given the `Task` class:
+
+```ruby
+> Task.schema
+{
+  "id" => :integer,
+  "last_assigned_at" => :date,
+  "type" => :string,
+  "workflow" => {
+    "id" => :integer,
+    "created_at" => :date,
+    "notes" => :text_array
+  }
+}
+```
+
+## Custom Field Attributes
+
+The definition of index fields within a class allows for the
+specification of custom attributes, for example:
+
+```ruby
+class Task < ActiveRecord::Base
+  include AgnosticBackend::Indexable
+  belongs_to :workflow, class_name: 'Workflow'
+
+  define_index_fields do
+    integer :id
+    date :last_assigned_at, value: :assigned_at,
+         is_column: true, label: 'Last Assigned At'
+    string :type, value: proc { task_category.name }
+           is_column: true, label: 'Task Type'
+    struct :workflow, from: Workflow
+  end
+end
+```
+
+In this example, we have specified two custom attributes for fields
+`label` and `is_column`, for use in UI elements.
+
+We can get these options back (say `:is_column`) by passing a block to
+`Indexable#schema` (that yields a `FieldType` instance) as follows:
+
+```ruby
+> task.schema {|field_type| field_type.get_option('is_column') }
+{:id=>nil, :last_assigned_at=>true, :type=>true, ...}
+```
+
+Custom attributes can be very useful in a variety of situations; for
+example, they can be used in the context of web views in order to
+control the visual/behavioural aspects of the document's fields.
+
+## Polymorphic relationships (for ActiveRecord classes)
+
+`Indexable` also supports AR polymorphic relationships as nested
+fields. Suppose we have the following model:
+
+```ruby
+class Task < ActiveRecord::Base
+  include AgnosticBackend::Indexable
+  has_one :concrete_task, polymorphic: true
+
+  define_index_fields do
+    struct :concrete_task
+  end
+end
+```
+
+that has a polymorphic relationship with a concrete task, which can be
+one of various classes, say `ConcreteTaskA` and `ConcreteTaskB`. When
+requesting the `Task`'s schema using `Task.schema` the algorithm can
+not figure out which class needs to be queried about its schema when
+it encounters the `struct` field. As a result, the schema is
+incomplete.
+
+This can be overcome by specifying the possible classes that can
+constitute a concrete task using the `from` attribute as:
+
+```ruby
+class Task < ActiveRecord::Base
+  include AgnosticBackend::Indexable
+  has_one :concrete_task, polymorphic: true
+
+  define_index_fields do
+    struct :concrete_task, from: [ConcreteTaskA, ConcreteTaskB]
+  end
+end
+```
+
+As a result, the schema will include a `concrete_task` field whose
+value will be the result of a merge between the schemas of all the
+classes specified in the `from` attribute.
+
+## RSpec Matchers
+
+`AgnosticBackends` also supplies `RSpec` matchers for verifying that a
+given class is `Indexable` and that it indexes the expected fields.
+
+In your `spec_helper.rb` use the following:
+
+```ruby
+require 'agnostic_backend/rspec/matchers'
+
+RSpec.configure do |config|
+  config.include AgnosticBackend::RSpec::Matchers
+end
+```
+
+This gives access to the matchers `be_indexable` and
+`define_index_field`. For usage examples, check the
+[corresponding test file](matchers_spec.rb).

data/lib/agnostic_backend/cloudsearch/index.rb

@@ -0,0 +1,99 @@
+module AgnosticBackend
+  module Cloudsearch
+    class Index < AgnosticBackend::Index
+
+      attr_reader :region,
+                  :domain_name,
+                  :document_endpoint,
+                  :search_endpoint,
+                  :access_key_id,
+                  :secret_access_key
+
+      def initialize(indexable_klass, **options)
+        super(indexable_klass)
+        @region = parse_option(options, :region)
+        @domain_name = parse_option(options, :domain_name)
+        @document_endpoint = parse_option(options, :document_endpoint)
+        @search_endpoint = parse_option(options, :search_endpoint)
+        @access_key_id = parse_option(options, :access_key_id)
+        @secret_access_key = parse_option(options, :secret_access_key)
+      end
+
+      def indexer
+        AgnosticBackend::Cloudsearch::Indexer.new(self)
+      end
+
+      def query_builder
+        AgnosticBackend::Queryable::Cloudsearch::QueryBuilder.new(self)
+      end
+
+      def schema
+        @schema ||= @indexable_klass.schema{|ftype| ftype}
+      end
+
+      def configure
+        define_fields_in_domain(indexer.flatten(schema))
+      end
+
+      def cloudsearch_client
+        @cloudsearch_client ||= Aws::CloudSearch::Client.new(region: region, access_key_id: access_key_id, secret_access_key: secret_access_key)
+      end
+
+      def cloudsearch_domain_client
+        @cloudsearch_domain_client ||= Aws::CloudSearchDomain::Client.new(endpoint: search_endpoint, access_key_id: access_key_id, secret_access_key: secret_access_key)
+      end
+
+      private
+
+      def remove_fields_from_domain(remote_fields, verbose: true)
+        remote_fields.map(&:index_field_name).each do |field_name|
+          puts "#{domain_name} > Removing obsolete field: #{field_name}" if verbose
+          cloudsearch_client.delete_index_field(domain_name: domain_name,
+                                                index_field_name: field_name)
+        end
+      end
+
+      def define_fields_in_domain(flat_schema, verbose: true)
+        remote_fields = cloudsearch_client.describe_index_fields(domain_name: domain_name).
+                         index_fields.map{|field| RemoteIndexField.new(field)}
+        puts "Found #{remote_fields.size} remote fields in #{domain_name}" if verbose
+        local_fields = index_fields(flat_schema)
+        puts "Found #{local_fields.size} local fields for that domain" if verbose
+
+        valid_remote_fields, obsolete_remote_fields =
+          RemoteIndexField.partition(local_fields, remote_fields)
+        puts "Found #{valid_remote_fields.size} valid remote fields" if verbose
+        puts "Found #{obsolete_remote_fields.size} obsolete remote fields" if verbose
+
+        remove_fields_from_domain(obsolete_remote_fields) unless obsolete_remote_fields.empty?
+
+        local_fields.each do |index_field|
+          # find the corresponding remote field
+          remote_field = valid_remote_fields.find do |remote_field|
+            remote_field.index_field_name == index_field.name
+          end
+          if remote_field.nil? ||
+             (remote_field.present? && !index_field.equal_to_remote_field?(remote_field))
+            puts "#{domain_name} > Defining new field: #{index_field.name}" if verbose
+            index_field.define_in_domain(index: self)
+          end
+        end
+        nil
+      end
+
+      def index_fields(flat_schema)
+        flat_schema.map do |field_name, field_type|
+          AgnosticBackend::Cloudsearch::IndexField.new(field_name, field_type)
+        end
+      end
+
+      def parse_option(options, option_name)
+        if options.has_key?(option_name)
+          options[option_name]
+        else
+          raise "#{option_name} must be specified"
+        end
+      end
+    end
+  end
+end

data/lib/agnostic_backend/cloudsearch/index_field.rb

@@ -0,0 +1,103 @@
+require "aws-sdk"
+
+module AgnosticBackend
+  module Cloudsearch
+    class IndexField
+      include AgnosticBackend::Utilities
+
+      TYPE_MAPPINGS = {
+        AgnosticBackend::Indexable::FieldType::STRING => "literal",
+        AgnosticBackend::Indexable::FieldType::STRING_ARRAY => "literal-array",
+        AgnosticBackend::Indexable::FieldType::DATE => "date",
+        AgnosticBackend::Indexable::FieldType::INTEGER => "int",
+        AgnosticBackend::Indexable::FieldType::DOUBLE => "double",
+        AgnosticBackend::Indexable::FieldType::BOOLEAN => "literal",
+        AgnosticBackend::Indexable::FieldType::TEXT => "text",
+        AgnosticBackend::Indexable::FieldType::TEXT_ARRAY => "text-array",
+      }.freeze
+
+      attr_reader :name, :type
+
+      def initialize(name, type)
+        @name = name
+        @type = type
+      end
+
+      def define_in_domain(index: )
+        with_exponential_backoff Aws::CloudSearch::Errors::Throttling do
+          index.cloudsearch_client.define_index_field(
+            :domain_name => index.domain_name,
+            :index_field => definition
+          )
+        end
+      end
+
+      def equal_to_remote_field?(remote_field)
+        remote_options = remote_field.send(options_name.to_sym)
+        local_options = options
+
+        remote_field.index_field_name == name.to_s &&
+          remote_field.index_field_type == cloudsearch_type &&
+          local_options.all?{|k, v| v == remote_options.send(k) }
+      end
+
+      def sortable?
+        type.has_option(:sortable) ? !!type.get_option(:sortable) : true
+      end
+
+      def searchable?
+        type.has_option(:searchable) ? !!type.get_option(:searchable) : true
+      end
+
+      def returnable?
+        type.has_option(:returnable) ? !!type.get_option(:returnable) : true
+      end
+
+      def facetable?
+        type.has_option(:facetable) ? !!type.get_option(:facetable) : false
+      end
+
+      private
+
+      def cloudsearch_type
+        @cloudsearch_type ||= TYPE_MAPPINGS[type.type]
+      end
+
+      def definition
+        {
+          :index_field_name => name.to_s,
+          :index_field_type => cloudsearch_type,
+          options_name.to_sym => options
+        }
+      end
+
+      def options_name
+        "#{cloudsearch_type.gsub('-', '_')}_options"
+      end
+
+      def options
+        opts = {
+            :sort_enabled => sortable?,
+            :search_enabled => searchable?,
+            :return_enabled => returnable?,
+            :facet_enabled => facetable?
+          }
+        # certain parameters are not included acc. to cloudsearch type
+        # we filter them out here
+        case cloudsearch_type
+        when 'text-array'
+          opts.delete(:sort_enabled)
+          opts.delete(:search_enabled)
+          opts.delete(:facet_enabled)
+        when 'text'
+          opts.delete(:search_enabled)
+          opts.delete(:facet_enabled)
+        when 'literal-array'
+          opts.delete(:sort_enabled)
+        end
+        opts
+      end
+
+    end
+  end
+end

data/lib/agnostic_backend/cloudsearch/indexer.rb

@@ -0,0 +1,84 @@
+require 'aws-sdk'
+
+module AgnosticBackend
+  module Cloudsearch
+    class Indexer < AgnosticBackend::Indexer
+      include AgnosticBackend::Utilities
+
+      def initialize(index)
+        @index = index
+      end
+
+      def publish(document)
+        with_exponential_backoff Aws::CloudSearch::Errors::Throttling do
+          client.upload_documents(
+            documents: document,
+            content_type:'application/json'
+          )
+        end
+      end
+
+      def delete(*document_ids)
+        documents = document_ids.map do |document_id|
+          {"type" => 'delete',
+           "id" => document_id}
+        end
+
+        with_exponential_backoff Aws::CloudSearch::Errors::Throttling do
+          client.upload_documents(
+            documents: convert_to_json(documents),
+            content_type:'application/json'
+
+          )
+        end
+      end
+
+      private
+
+      def client
+        index.cloudsearch_domain_client
+      end
+
+      def prepare(document)
+        document
+      end
+
+      def transform(document)
+        return {} if document.empty?
+
+        document = flatten document
+        document = reject_blank_values_from document
+        document = convert_bool_values_to_string_in document
+        document = date_format document
+        document = add_metadata_to document
+        document = convert_document_into_array(document)
+        convert_to_json document
+
+      end
+
+      def date_format(document)
+        document.each do |k, v|
+          if v.is_a?(Time)
+            document[k] = v.utc.strftime("%Y-%m-%dT%H:%M:%SZ")
+          end
+        end
+      end
+
+      def add_metadata_to(document)
+        {
+            "type" => "add",
+            "id" => document["id"].to_s,
+            "fields" => document,
+        }
+      end
+
+      def convert_to_json(transformed_document)
+        ActiveSupport::JSON.encode(transformed_document)
+      end
+
+      def convert_document_into_array(document)
+        [document]
+      end
+    end
+  end
+end

data/lib/agnostic_backend/cloudsearch/remote_index_field.rb

@@ -0,0 +1,32 @@
+module AgnosticBackend
+  module Cloudsearch
+    class RemoteIndexField
+
+      attr_reader :field, :status
+
+      # returns an array with two elements:
+      # the first is an array with the remote fields that correspond to local fields
+      # the second is an array with the remote that do not have corresponding local fields
+      def self.partition(local_fields, remote_fields)
+        local_field_names = local_fields.map(&:name)
+        remote_fields.partition do |remote_field|
+          local_field_names.include? remote_field.index_field_name
+        end
+      end
+
+      def initialize(remote_field_struct)
+        @field = remote_field_struct.options
+        @status = remote_field_struct.status
+      end
+
+      def method_missing(method_name)
+        if field.respond_to?(method_name)
+          field.send(method_name)
+        else
+          super
+        end
+      end
+
+    end
+  end
+end

data/lib/agnostic_backend/index.rb

@@ -0,0 +1,24 @@
+module AgnosticBackend
+  class Index
+
+    def initialize(indexable_klass)
+      @indexable_klass = indexable_klass
+    end
+
+    def name
+      @indexable_klass.index_name
+    end
+
+    def schema
+      @indexable_klass.schema
+    end
+
+    def indexer
+      raise NotImplementedError
+    end
+
+    def configure(new_schema = nil)
+      raise NotImplementedError
+    end
+  end
+end

data/lib/agnostic_backend/indexable/config.rb

@@ -0,0 +1,24 @@
+module AgnosticBackend
+  module Indexable
+
+    class Config
+
+      class ConfigEntry < Struct.new(:index_class, :options);
+      end
+
+      def self.indices
+        @indices ||= {}
+      end
+
+      def self.configure_index(indexable_class, index_class, **options)
+        indices[indexable_class.name] = ConfigEntry.new index_class, options
+      end
+
+      def self.create_index_for(indexable_class)
+        entry = indices[indexable_class.name]
+        entry.index_class.try(:new, indexable_class, entry.options)
+      end
+
+    end
+  end
+end