droonga-engine 1.0.1

data/doc/text/news.md

@@ -0,0 +1,106 @@
+# News
+
+## 1.0.1: 2014-03-29
+
+### Improvements
+
+  * More documents around plugin APIs are now available.
+    See the plugin [development tutorial](http://droonga.org/tutorial/plugin-development/) and the [plugin API reference](http://droonga.org/reference/plugin/).
+  * Some documented features of the `catalog.json` are actually implemented.
+    For example:
+    * A new `vectorOptions` option for a schema.
+    * New options `fact`, `dimension`, `nWorkers` and so on.
+    For more details, see [the reference of the `catalog.json`](http://droonga.org/reference/catalog/).
+  * Connections to other Droonga Engine nodes are automatically re-established correctly.
+  * Some improvements about the [`search` command](http://droonga.org/reference/commands/search/)
+    * The column name `_nsubrecs` is available as a source with `groupBy` and `sortBy`.
+    * The element `elapsedTime` is now available for a value of `elements`.
+    * A new parameter `adjusters` is introduced. (Not documented yet, so see also [Groonga's document](http://groonga.org/docs/reference/commands/select.html#select-adjuster))
+    * `groupBy` becomes faster.
+  * And some small bugfixes.
+
+## 1.0.0: 2014-02-28
+
+### Improvements
+
+  * Updated catalog.json specification to
+    [version2](http://droonga.org/reference/catalog/version2/).
+    [version1](http://droonga.org/reference/catalog/version1/) is
+    still usable. But It is deprecated.
+  * Supported log API in plugin.
+  * Supported auto catalog.json reload.
+  * Changed adapter API:
+
+    Old:
+
+        message.input_pattern  = []
+        message.output_pattern = []
+
+    New:
+
+        input_message.pattern  = []
+        output_message.pattern = []
+
+  * Supported developing a plugin for handling phase.
+    See [tutorial](http://droonga.org/tutorial/plugin-development/handler/)
+    for details.
+
+## 0.9.9: 2014-02-09
+
+### Improvements
+
+  * Supported gathering errors.
+  * Added more error handled cases.
+  * experimental: Added a MeCab filter that filters results from
+    N-gram tokenizer based search by MeCab based tokenized search. It
+    is disabled by default. You need to define
+    `DROONGA_ENABLE_SEARCH_MECAB_FILTER=yes` environment variable when
+    you run fluentd.
+  * Supported developing a plugin. You can custom adaption phase for now.
+    See [tutorial](http://droonga.org/tutorial/plugin-development/) for details.
+
+## 0.9.0: 2014-01-29
+
+### Improvements
+
+  * `search`: Supported `"attributes"` for `elements` of `output`.
+  * `table_remove`: Implemented Groonga compatible `table_remove`
+    command.
+  * `column_create`: Implemented error handling.
+  * `catalog`: Supported auto reloading.
+  * Supported reducing responses from two or more nodes for Groonga
+    compatible commands.
+  * Supported three or more partitions.
+
+## 0.8.0: 2013-12-29
+
+### Improvements
+
+  * `search`: Supported `groupBy` with multiple partitions.
+  * Changed job queue implementation to UNIX domain socket based
+    implementation from Groonga's queue based implementation. It
+    reduces shutdown time and fixes job queue break problem on
+    crash.
+  * Supported error response.
+  * `watch`: Fixed a bug that duplicated notification is pushed when
+    multiple column values are matched.
+  * `watch`: Supported subscriber garbage collection.
+  * Improved plugin API. Plugin API documentation will be published
+    in the next release.
+  * Added micro seconds information to time value. It uses W3C-DTF format
+    such as `2013-12-29T00:00:00.000000Z`.
+  * Changed the name of adapter plugin that provides Groonga's `select`
+    compatible API to `groonga` from `select`. The `groonga` adapter will
+    provide more Groonga compatible commands.
+  * Added `version` to `catalog.json`.
+  * Changed plugin path format to
+    `droonga/plugin/#{PLUGIN_TYPE}/#{PLUGIN_NAME}.rb` from
+    `droonga/plugin/#{PLUGIN_TYPE}_#{PLUGIN_NAME}.rb`. We use a
+    directory per plugin type instead of putting all plugins to
+    `droonga/plugin/` directory. Because it is more maintainable.
+  * Split adapter plugin into input adapter plugin and output adapter
+    plugin. You can in the next release.
+
+## 0.7.0: 2013-11-29
+
+The first release!!!

data/droonga-engine.gemspec

@@ -0,0 +1,52 @@
+# -*- mode: ruby; coding: utf-8 -*-
+#
+# Copyright (C) 2013-2014 Droonga Project
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+base_dir = File.dirname(__FILE__)
+$LOAD_PATH.unshift(File.join(base_dir, "lib"))
+
+require "droonga/engine/version"
+
+Gem::Specification.new do |gem|
+  gem.name          = "droonga-engine"
+  gem.version       = Droonga::Engine::VERSION
+  gem.authors       = ["Droonga Project"]
+  gem.email         = ["droonga@groonga.org"]
+  gem.summary       = "Droonga engine"
+  gem.description   =
+    "Droonga engine is a core component in Droonga system. " +
+    "Droonga is a scalable data processing engine based on Groonga. " +
+    "Droonga means Distributed Groonga."
+  gem.homepage      = "https://github.com/droonga/droonga-engine"
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+  gem.add_dependency "rroonga", ">= 3.1.0"
+  gem.add_dependency "groonga-command-parser"
+  gem.add_dependency "json"
+  gem.add_dependency "cool.io"
+  gem.add_dependency "serverengine"
+  gem.add_dependency "droonga-message-pack-packer"
+  gem.add_development_dependency "rake"
+  gem.add_development_dependency "bundler"
+  gem.add_development_dependency "droonga-client"
+  gem.add_development_dependency "test-unit"
+  gem.add_development_dependency "test-unit-notify"
+  gem.add_development_dependency "test-unit-rr"
+  gem.add_development_dependency "packnga"
+  gem.add_development_dependency "kramdown"
+end

data/lib/droonga/adapter.rb

@@ -0,0 +1,48 @@
+# Copyright (C) 2014 Droonga Project
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+require "droonga/pluggable"
+require "droonga/loggable"
+require "droonga/plugin/metadata/adapter_input_message"
+require "droonga/plugin/metadata/adapter_output_message"
+require "droonga/error_messages"
+
+module Droonga
+  class Adapter
+    extend Pluggable
+    include Loggable
+    include ErrorMessages
+
+    class << self
+      def input_message
+        Plugin::Metadata::AdapterInputMessage.new(self)
+      end
+
+      def output_message
+        Plugin::Metadata::AdapterOutputMessage.new(self)
+      end
+
+      def id
+        options[:id] || name || object_id.to_s
+      end
+    end
+
+    def adapt_input(input_message)
+    end
+
+    def adapt_output(output_message)
+    end
+  end
+end

data/lib/droonga/adapter_runner.rb

@@ -0,0 +1,104 @@
+# Copyright (C) 2013-2014 Droonga Project
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+require "droonga/loggable"
+require "droonga/message_matcher"
+require "droonga/input_message"
+require "droonga/output_message"
+require "droonga/adapter"
+
+module Droonga
+  class AdapterRunner
+    include Loggable
+
+    def initialize(dispatcher, plugins)
+      @dispatcher = dispatcher
+      default_plugins = ["error"]
+      plugins += (default_plugins - plugins)
+      logger.debug("activating plugins: #{plugins.join(", ")}")
+      @adapter_classes = Adapter.find_sub_classes(plugins)
+      logger.debug("activated:\n#{@adapter_classes.join("\n")}")
+    end
+
+    def shutdown
+    end
+
+    def adapt_input(message)
+      logger.trace("adapt_input: start",
+                   :dataset => message["dataset"],
+                   :type => message["type"])
+      adapted_message = message
+      adapted_message["appliedAdapters"] = []
+      @adapter_classes.each do |adapter_class|
+        adapter_class_id = adapter_class.id
+        pattern = adapter_class.input_message.pattern
+        if pattern
+          matcher = MessageMatcher.new(pattern)
+          logger.trace("adapt_input: skip: #{adapter_class_id}",
+                       :pattern => pattern)
+          next unless matcher.match?(adapted_message)
+        end
+        logger.trace("adapt_input: use: #{adapter_class_id}")
+        input_message = InputMessage.new(adapted_message)
+        adapter = adapter_class.new
+        adapter.adapt_input(input_message)
+        adapted_message = input_message.adapted_message
+        adapted_message["appliedAdapters"] << adapter_class_id
+      end
+      logger.trace("adapt_input: done",
+                   :dataset => adapted_message["dataset"],
+                   :type => adapted_message["type"])
+      adapted_message
+    end
+
+    def adapt_output(message)
+      logger.trace("adapt_output: start",
+                   :dataset => message["dataset"],
+                   :type => message["type"])
+      adapted_message = message
+      applied_adapters = adapted_message["appliedAdapters"]
+      @adapter_classes.reverse_each do |adapter_class|
+        adapter_class_id = adapter_class.id
+        if applied_adapters
+          logger.trace("adapt_output: skip: #{adapter_class_id}: " +
+                         "input adapter wasn't applied",
+                       :applied_adapters => applied_adapters)
+          next unless applied_adapters.include?(adapter_class.id)
+        end
+        pattern = adapter_class.output_message.pattern
+        if pattern
+          matcher = MessageMatcher.new(pattern)
+          logger.trace("adapt_output: skip: #{adapter_class_id}",
+                       :pattern => pattern)
+          next unless matcher.match?(adapted_message)
+        end
+        logger.trace("adapt_output: use: #{adapter_class_id}")
+        output_message = OutputMessage.new(adapted_message)
+        adapter = adapter_class.new
+        adapter.adapt_output(output_message)
+        adapted_message = output_message.adapted_message
+      end
+      logger.trace("adapt_output: done",
+                   :dataset => adapted_message["dataset"],
+                   :type => adapted_message["type"])
+      adapted_message
+    end
+
+    private
+    def log_tag
+      "adapter-runner"
+    end
+  end
+end

data/lib/droonga/catalog/base.rb

@@ -0,0 +1,41 @@
+# Copyright (C) 2013-2014 Droonga Project
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+require "digest/sha1"
+require "zlib"
+require "time"
+require "droonga/error_messages"
+require "droonga/catalog/errors"
+
+module Droonga
+  module Catalog
+    class Base
+      attr_reader :path, :base_path
+      def initialize(data, path)
+        @data = data
+        @path = path
+        @base_path = File.dirname(path)
+      end
+
+      def have_dataset?(name)
+        datasets.key?(name)
+      end
+
+      def dataset(name)
+        datasets[name]
+      end
+    end
+  end
+end

data/lib/droonga/catalog/collection_volume.rb

@@ -0,0 +1,106 @@
+# Copyright (C) 2014 Droonga Project
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+require "digest/sha1"
+require "zlib"
+
+require "droonga/catalog/slice"
+
+module Droonga
+  module Catalog
+    class CollectionVolume
+      def initialize(dataset, data)
+        @dataset = dataset
+        @data = data
+        compute_continuum if ratio_scaled_slicer?
+      end
+
+      def dimension
+        @data["dimension"] || "_key"
+      end
+
+      def slicer
+        @data["slicer"] || "hash"
+      end
+
+      def slices
+        @slices ||= @data["slices"].collect do |raw_slice|
+          Slice.new(@dataset, raw_slice)
+        end
+      end
+
+      def select_slices(range=0..-1)
+        slices.sort_by(&:label)[range]
+      end
+
+      def choose_slice(record)
+        return slices.first unless ratio_scaled_slicer?
+
+        key = stringify_key(record[dimension])
+        hash = Zlib.crc32(key)
+        min = 0
+        max = @continuum.size - 1
+        while (min < max)
+          index = (min + max) / 2
+          value, key = @continuum[index]
+          return key if value == hash
+          if value > hash
+            max = index
+          else
+            min = index + 1
+          end
+        end
+        @continuum[max][1]
+      end
+
+      def ratio_scaled_slicer?
+        slicer == "hash"
+      end
+
+      private
+      def compute_continuum
+        total_weight = compute_total_weight
+        continuum = []
+        n_slices = slices.size
+        slices.each do |slice|
+          weight = slice.weight
+          points = n_slices * 160 * weight / total_weight
+          points.times do |point|
+            hash = Digest::SHA1.hexdigest("#{@dataset.name}:#{point}")
+            continuum << [hash[0..7].to_i(16), slice]
+          end
+        end
+        @continuum = continuum.sort do |a, b|
+          a[0] - b[0]
+        end
+      end
+
+      def compute_total_weight
+        slices.reduce(0) do |result, slice|
+          result + slice.weight
+        end
+      end
+
+      def stringify_key(key)
+        case key
+        when Integer
+          key.to_s
+        else
+          key
+        end
+      end
+    end
+  end
+end

data/lib/droonga/catalog/dataset.rb

@@ -0,0 +1,69 @@
+# Copyright (C) 2014 Droonga Project
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+require "droonga/catalog/schema"
+require "droonga/catalog/volume"
+require "droonga/catalog/volume_collection"
+
+module Droonga
+  module Catalog
+    class Dataset
+      attr_reader :name
+
+      def initialize(name, data)
+        @name = name
+        @data = data
+        @schema = nil
+      end
+
+      # provided for compatibility
+      def [](key)
+        @data[key]
+      end
+
+      # provided for compatibility
+      def []=(key, value)
+        @data[key] = value
+      end
+
+      def schema
+        @schema ||= Schema.new(@name, @data["schema"])
+      end
+
+      def plugins
+        @data["plugins"] || []
+      end
+
+      def fact
+        @data["fact"]
+      end
+
+      def n_workers
+        @data["nWorkers"] || 0
+      end
+
+      def replicas
+        @replicas ||= VolumeCollection.new(create_volumes(@data["replicas"]))
+      end
+
+      private
+      def create_volumes(raw_volumes)
+        raw_volumes.collect do |raw_volume|
+          Volume.create(self, raw_volume)
+        end
+      end
+    end
+  end
+end

data/lib/droonga/catalog/errors.rb

@@ -0,0 +1,113 @@
+# Copyright (C) 2014 Droonga Project
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+require "droonga/error"
+
+module Droonga
+  module Catalog
+    class ValidationError < Error
+      class Detail
+        attr_reader :value_path, :message
+        def initialize(value_path, message)
+          @value_path = value_path
+          @message = message
+        end
+      end
+
+      attr_reader :path, :details
+      def initialize(path, details)
+        message = "validation error: <#{path}>"
+        details.each do |detail|
+          message << "\n  * #{detail.value_path}: #{detail.message}"
+        end
+        super(message)
+      end
+    end
+
+    class LegacyValidationError < Error
+      def initialize(message, path)
+        if path
+          super("[Validation Error <#{path}>]#{message}")
+        else
+          super(message)
+        end
+      end
+    end
+
+    class MissingRequiredParameter < LegacyValidationError
+      def initialize(name, path)
+        super("[#{name}] A required parameter is missing.", path)
+      end
+    end
+
+    class MismatchedParameterType < LegacyValidationError
+      def initialize(name, expected_types, actual, path)
+        expected_types = [expected_types] unless expected_types.is_a?(Array)
+        message = nil
+        if expected_types.size == 1
+          message = "[#{name}] Mismatched parameter type: " +
+                      "expected=<#{expected_types.first}>, actual=<#{actual}>"
+        else
+          message = "[#{name}] Mismatched parameter type: " +
+                      "expected=<#{expected_types.join(" or ")}>, " +
+                      "actual=<#{actual}>"
+        end
+        super(message, path)
+      end
+    end
+
+    class InvalidDate < LegacyValidationError
+      def initialize(name, value, path)
+        super("[#{name}] Invalid date string: <#{value}>", path)
+      end
+    end
+
+    class NegativeNumber < LegacyValidationError
+      def initialize(name, actual, path)
+        super("[#{name}] A positive number is expected, but <#{actual}>", path)
+      end
+    end
+
+    class SmallerThanOne < LegacyValidationError
+      def initialize(name, actual, path)
+        super("[#{name}] A number 1 or larger is expected, but <#{actual}>", path)
+      end
+    end
+
+    class FarmNotZoned < LegacyValidationError
+      def initialize(name, zones, path)
+        super("The farm does not appear in zones: <#{name}>, zones=<#{zones}>", path)
+      end
+    end
+
+    class UnknownFarmInZones < LegacyValidationError
+      def initialize(name, zones, path)
+        super("The farm is unknown: <#{name}>, zones=<#{zones}>", path)
+      end
+    end
+
+    class UnknownFarmForPartition < LegacyValidationError
+      def initialize(name, slice, path)
+        super("The farm is unknown: <{#name}>, slice=<#{slice}>", path)
+      end
+    end
+
+    class UnsupportedValue < LegacyValidationError
+      def initialize(name, value, path)
+        super("[#{name}] Not supported value: <#{value}>", path)
+      end
+    end
+  end
+end