fluent-plugin-groonga 1.0.3 → 1.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 34f2b071eacb59e8af9d91a384a8227e5ea64f58
-  data.tar.gz: aacc29322894f6a03cd5681e3bd7870b64df6bb7
+  metadata.gz: 91510d22d9e94a522c9ee54b3a4d0caf4c6f4cc9
+  data.tar.gz: 0e77c6fc21121575977a288c131272c780ae0ecb
 SHA512:
-  metadata.gz: 9e82b5797718f9fde41edf6c8f74cfb8ee484fe98f932aaacbfe58fd0c179b141dd4389a29d5f9a3fb69c4bb56cd4166d417580828e34d71dad2ef07ebb3aa51
-  data.tar.gz: 6d4309efaaf8bbbfb46e5f46f99322b4ad8f810425a2ea80b2eaa8f1c5b393884f2293b4eefd4d3b40072c7f755c1b423bc7cf957a144c53772db6872b876600
+  metadata.gz: 6d00c8ff58928fc8c94b1a6ab98f803cac53517a022bb968ecf1b7e4862442e89f6583c89cedcd2b3b6022e3ff6acb82bb9ad270ec17a98b14b998c00727425a
+  data.tar.gz: 1fb6b8d3faeae3bf096b1df0845f590620ecf08cc41c2e4234ade39297e42efecd2a908c61f64ad316096ddba98c9833c2a8b6439e83ed178133374617b06fc6

data/README.md

@@ -8,19 +8,59 @@ fluent-plugin-groonga
 
 ## Description
 
-Fluent-plugin-groonga is fluentd plugin collection for
-[groonga](http://groonga.org/) users. Groonga users can replicate
-their data by fluent-plugin-groonga.
+Fluent-plugin-groonga is a Fluentd plugin collection to use
+[Groonga](http://groonga.org/) with Fluentd. Fluent-plugin-groonga
+supports the following two usages:
+
+  * Store logs collected by Fluentd to Groonga.
+  * Implement replication system for Groonga.
+
+The first usage is normal usage. You can store logs to Groonga and
+find logs by full-text search.
+
+The second usage is for Groonga users. Groonga itself doesn't support
+replication. But Groonga users can replicate their data by
+fluent-plugin-groonga.
 
 Fluent-plugin-groonga includes an input plugin and an output
 plugin. Both of them are named `groonga`.
 
-The input plugin provides groonga compatible interface. It means that
-HTTP and GQTP interface. You can use the input plugin as groonga
-server. The input plugin receives groonga commands and sends them to
-the output plugin through zero or more fluentds.
+If you want to use fluent-plugin-groonga to store logs to Groonga, you
+need to use only `groonga` output plugin.
+
+The following configuration stores all data in `/var/log/messages`
+into Groonga:
+
+    <source>
+      type tail
+      format syslog
+      path /var/log/syslog.1
+      pos_file /tmp/messages.pos
+      tag log.messages
+      read_from_head true
+    </source>
+
+    <match log.**>
+      type groonga
+      table logs
+
+      protocol http
+      host 127.0.0.1
+
+      buffer_type file
+      buffer_path /tmp/buffer
+      flush_interval 1
+    </match>
 
-The output plugin sends received groonga commands to groonga. The
+If you want to use fluent-plugin-groonga to implement Groonga
+replication system, you need to use both plugins.
+
+The input plugin provides Groonga compatible interface. It means that
+HTTP and GQTP interface. You can use the input plugin as Groonga
+server. The input plugin receives Groonga commands and sends them to
+the output plugin through zero or more Fluentds.
+
+The output plugin sends received Groonga commands to Groonga. The
 output plugin supports all interfaces, HTTP, GQTP and command
 interface.
 
@@ -32,6 +72,78 @@ You can replicate your data by using `copy` output plugin.
 
 ## Usage
 
+There are two usages:
+
+  * Store logs collected by Fluentd to Groonga.
+  * Implement replication system for Groonga.
+
+They are described in other sections.
+
+### Store logs into Groonga
+
+You need to use `groonga` output plugin to store logs into Groonga.
+
+The output plugin has auto schema define feature. So you don't need to
+define schema in Groonga before running Fluentd. You just run Groonga.
+
+There is one required parameter:
+
+  * `table`: It specifies table name for storing logs.
+
+Here is a minimum configuration:
+
+    <match log.**>
+      type groonga
+      table logs
+    </match>
+
+The configuration stores logs into `logs` table in Groonga that runs
+on `localhost`.
+
+There are optional parameters:
+
+  * `protocol`: It specifies protocol to communicate Groonga server.
+    * Available values: `http`, `gqtp`, `command`
+    * Default: `http`
+  * `host`: It specifies host name or IP address of Groonga server.
+    * Default: `127.0.0.1`
+  * `port`: It specifies port number of Groonga server.
+    * Default for `http` protocol: `10041`
+    * Default for `gqtp` protocol: `10043`
+
+Here is a configuration that specifies optional parameters explicitly:
+
+    <match log.**>
+      type groonga
+      table logs
+
+      protocol http
+      host 127.0.0.1
+      port 10041
+    </match>
+
+`groonga` output plugin supports buffer. So you can use buffer related
+parameters. See
+[Buffer Plugin Overview | Fluentd](http://docs.fluentd.org/articles/buffer-plugin-overview)
+for details.
+
+Note that there is special tag name. You can't use
+`groonga.command.XXX` tag name for this usage. It means that you can't
+use the following configuration:
+
+    <match groonga.command.*>
+      type groonga
+      # ...
+    </match>
+
+`groonga.command.XXX` tag name is reserved for implementing
+replication system for Groonga.
+
+### Implement replication system for Groonga
+
+See the following documents how to implement replication system for
+Groonga:
+
 * [Configuration](doc/text/configuration.md)
   ([on the Web](http://groonga.org/fluent-plugin-groonga/en/file.configuration.html))
 * [Constitution](doc/text/constitution.md)

data/doc/text/news.md

@@ -2,6 +2,21 @@
 
 # News
 
+## 1.0.4: 2014-10-20
+
+### Improvements
+
+* Supported the latest http_parser gem.
+* Removed no buffer mode. Use `flush_interval 0` for no buffer like
+  behavior.
+* Changed the default port number to `10043` for `gqtp` protocol usage.
+  Because Groonga changed the default port number for `gqtp` protocol.
+* Reduced the number of `load` calls. It improves `load` performance.
+* Supported auto schema define. You don't need to define schema in Groonga
+  before running Fluentd.
+* Added document to use fluent-plugin-groonga to store logs into Groonga.
+  It fits normal Fluentd usage.
+
 ## 1.0.3: 2013-09-29
 
 ### Improvements

data/fluent-plugin-groonga.gemspec

@@ -1,6 +1,6 @@
 # -*- mode: ruby; coding: utf-8 -*-
 #
-# Copyright (C) 2012-2013  Kouhei Sutou <kou@clear-code.com>
+# Copyright (C) 2012-2014  Kouhei Sutou <kou@clear-code.com>
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -17,12 +17,12 @@
 
 Gem::Specification.new do |spec|
   spec.name = "fluent-plugin-groonga"
-  spec.version = "1.0.3"
+  spec.version = "1.0.4"
   spec.authors = ["Kouhei Sutou"]
   spec.email = ["kou@clear-code.com"]
-  spec.summary = "Fluentd plugin collection for groonga users"
+  spec.summary = "Fluentd plugin to store data into Groonga and implement Groonga replication system."
   spec.description =
-    "Groonga users can replicate their data by fluent-plugin-groonga"
+    "There are two usages. 1) Store data into Groonga. 2) Implement Groonga replication system. See documentation for details."
   spec.homepage = "https://github.com/groonga/fluent-plugin-groonga"
   spec.license = "LGPL-2.1"
 
@@ -35,7 +35,7 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_runtime_dependency("fluentd")
-  spec.add_runtime_dependency("gqtp", ">= 1.0.3")
+  spec.add_runtime_dependency("groonga-client")
   spec.add_runtime_dependency("groonga-command-parser")
 
   spec.add_development_dependency("rake")

data/lib/fluent/plugin/in_groonga.rb

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 #
-# Copyright (C) 2012-2013  Kouhei Sutou <kou@clear-code.com>
+# Copyright (C) 2012-2014  Kouhei Sutou <kou@clear-code.com>
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -16,6 +16,7 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
 require "English"
+require "uri"
 require "webrick/httputils"
 
 require "http_parser"
@@ -31,19 +32,22 @@ module Fluent
       super
     end
 
-    config_param :protocol, :string, :default => "http"
+    config_param :protocol, :defalut => :http do |value|
+      case value
+      when "http", "gqtp"
+        value.to_sym
+      else
+        raise ConfigError, "must be http or gqtp: <#{value}>"
+      end
+    end
 
     def configure(conf)
       super
       case @protocol
-      when "http"
+      when :http
         @input = HTTPInput.new
-      when "gqtp"
+      when :gqtp
         @input = GQTPInput.new
-      else
-        message = "unknown protocol: <#{@protocol.inspect}>"
-        $log.error message
-        raise ConfigError, message
       end
       @input.configure(conf)
     end
@@ -76,9 +80,9 @@ module Fluent
       include DetachMultiProcessMixin
 
       config_param :bind, :string, :default => "0.0.0.0"
-      config_param :port, :integer, :default => 10041
+      config_param :port, :integer, :default => nil
       config_param :real_host, :string
-      config_param :real_port, :integer, :default => 10041
+      config_param :real_port, :integer, :default => nil
       DEFAULT_EMIT_COMMANDS = [
         /\Atable_/,
         /\Acolumn_/,
@@ -102,6 +106,13 @@ module Fluent
         end
       end
 
+      def configure(conf)
+        super
+
+        @port ||= default_port
+        @real_port ||= default_port
+      end
+
       def start
         listen_socket = TCPServer.new(@bind, @port)
         detach_multi_process do
@@ -156,6 +167,10 @@ module Fluent
 
     class HTTPInput < BaseInput
       private
+      def default_port
+        10041
+      end
+
       def handler_class
         Handler
       end
@@ -188,8 +203,9 @@ module Fluent
         end
 
         def on_message_complete
-          params = WEBrick::HTTPUtils.parse_query(@parser.query_string)
-          path_info = @parser.request_path
+          uri = URI.parse(@parser.request_url)
+          params = WEBrick::HTTPUtils.parse_query(uri.query)
+          path_info = uri.path
           case path_info
           when /\A\/d\//
             command = $POSTMATCH
@@ -204,6 +220,10 @@ module Fluent
 
     class GQTPInput < BaseInput
       private
+      def default_port
+        10043
+      end
+
       def handler_class
         Handler
       end

data/lib/fluent/plugin/out_groonga.rb

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 #
-# Copyright (C) 2012  Kouhei Sutou <kou@clear-code.com>
+# Copyright (C) 2012-2014  Kouhei Sutou <kou@clear-code.com>
 #
 # This library is free software; you can redistribute it and/or
 # modify it under the terms of the GNU Lesser General Public
@@ -17,23 +17,26 @@
 
 require "fileutils"
 
+require "yajl"
+
+require "groonga/client"
+
 module Fluent
-  class GroongaOutput < Output
+  class GroongaOutput < BufferedOutput
     Plugin.register_output("groonga", self)
 
     def initialize
       super
     end
 
-    BufferedOutput.config_params.each do |name, (block, options)|
-      if options[:type]
-        config_param(name, options[:type], options)
+    config_param :protocol, :default => :http do |value|
+      case value
+      when "http", "gqtp", "command"
+        value.to_sym
       else
-        config_param(name, options, &block)
+        raise ConfigError, "must be http, gqtp or command: <#{value}>"
       end
     end
-
-    config_param :protocol, :string, :default => "http"
     config_param :table, :string, :default => nil
 
     def configure(conf)
@@ -42,168 +45,318 @@ module Fluent
       @client.configure(conf)
 
       @emitter = Emitter.new(@client, @table)
-      @output = create_output(@buffer_type, @emitter)
-      @output.configure(conf)
     end
 
     def start
       super
       @client.start
-      @output.start
+      @emitter.start
     end
 
     def shutdown
       super
-      @output.shutdown
+      @emitter.shutdown
       @client.shutdown
     end
 
-    def emit(tag, event_stream, chain)
-      @output.emit(tag, event_stream, chain)
+    def format(tag, time, record)
+      [tag, time, record].to_msgpack
     end
 
+    def write(chunk)
+      @emitter.emit(chunk)
+    end
+
+    private
     def create_client(protocol)
       case protocol
-      when "http"
-        HTTPClient.new
-      when "gqtp"
-        GQTPClient.new
-      when "command"
+      when :http, :gqtp
+        NetworkClient.new(protocol)
+      when :command
         CommandClient.new
       end
     end
 
-    def create_output(buffer_type, emitter)
-      if buffer_type == "none"
-        RawGroongaOutput.new(emitter)
-      else
-        BufferedGroongaOutput.new(emitter)
+    class Schema
+      def initialize(client, table_name)
+        @client = client
+        @table_name = table_name
+        @table = nil
+        @columns = nil
       end
-    end
 
-    class Emitter
-      def initialize(client, table)
-        @client = client
-        @table = table
+      def populate
+        # TODO
       end
 
-      def emit(tag, record)
-        if /\Agroonga\.command\./ =~ tag
-          name = $POSTMATCH
-          send_command(name, record)
-        else
-          store_chunk(data)
+      def update(records)
+        ensure_table
+        ensure_columns
+
+        nonexistent_columns = {}
+        records.each do |record|
+          record.each do |key, value|
+            column = @columns[key]
+            if column.nil?
+              nonexistent_columns[key] ||= []
+              nonexistent_columns[key] << value
+            end
+          end
+        end
+
+        nonexistent_columns.each do |name, values|
+          @columns[name] = create_column(name, values)
         end
       end
 
       private
-      def send_command(name, arguments)
-        command_class = Groonga::Command.find(name)
-        command = command_class.new(name, arguments)
-        @client.send(command)
+      def ensure_table
+        return if @table
+
+        table_list = @client.execute("table_list")
+        target_table = table_list.find do |table|
+          table.name == @table_name
+        end
+        if target_table
+          @table = Table.new(@table_name, target_table.domain)
+        else
+          # TODO: Check response
+          @client.execute("table_create",
+                          "name"  => @table_name,
+                          "flags" => "TABLE_NO_KEY")
+          @table = Table.new(@table_name, nil)
+        end
       end
 
-      def store_chunk(value)
-        return if @table.nil?
+      def ensure_columns
+        return if @columns
 
-        values = [value]
-        arguments = {
-          "table" => @table,
-          "values" => Yajl::Enocder.encode(values),
-        }
-        send_command("load", arguments)
+        column_list = @client.execute("column_list", "table" => @table_name)
+        @columns = {}
+        column_list.each do |column|
+          vector_p = column.flags.split("|").include?("COLUMN_VECTOR")
+          @columns[column.name] = Column.new(column.name,
+                                             column.range,
+                                             vector_p)
+        end
       end
-    end
 
-    class RawGroongaOutput < Output
-      def initialize(emitter)
-        @emitter = emitter
-        super()
+      def create_column(name, sample_values)
+        guesser = TypeGuesser.new(sample_values)
+        value_type = guesser.guess
+        vector_p = guesser.vector?
+        if vector_p
+          flags = "COLUMN_VECTOR"
+        else
+          flags = "COLUMN_SCALAR"
+        end
+        # TODO: Check response
+        @client.execute("column_create",
+                        "table" => @table_name,
+                        "name" => name,
+                        "flags" => flags,
+                        "type" => value_type)
+        Column.new(name, value_type, vector_p)
       end
 
-      def emit(tag, event_stream, chain)
-        event_stream.each do |time, record|
-          @emitter.emit(tag, record)
+      class TypeGuesser
+        def initialize(sample_values)
+          @sample_values = sample_values
         end
-        chain.next
-      end
-    end
 
-    class BufferedGroongaOutput < BufferedOutput
-      def initialize(emitter)
-        @emitter = emitter
-        super()
+        def guess
+          return "Time"          if time_values?
+          return "Int32"         if int32_values?
+          return "Int64"         if int64_values?
+          return "Float"         if float_values?
+          return "WGS84GeoPoint" if geo_point_values?
+
+          "Text"
+        end
+
+        def vector?
+          @sample_values.any? do |sample_value|
+            sample_value.is_a?(Array)
+          end
+        end
+
+        private
+        def time_values?
+          now = Time.now.to_i
+          year_in_seconds = 365 * 24 * 60 * 60
+          window = 10 * year_in_seconds
+          new = now + window
+          old = now - window
+          recent_range = old..new
+          @sample_values.all? do |sample_value|
+            sample_value.is_a?(Integer) and
+              recent_range.cover?(sample_value)
+          end
+        end
+
+        def integer_value?(value)
+          case value
+          when String
+            begin
+              Integer(value)
+              true
+            rescue ArgumentError
+              false
+            end
+          when Integer
+            true
+          else
+            false
+          end
+        end
+
+        def int32_values?
+          int32_min = -(2 ** 31)
+          int32_max = 2 ** 31 - 1
+          range = int32_min..int32_max
+          @sample_values.all? do |sample_value|
+            integer_value?(sample_value) and
+              range.cover?(Integer(sample_value))
+          end
+        end
+
+        def int64_values?
+          @sample_values.all? do |sample_value|
+            integer_value?(sample_value)
+          end
+        end
+
+        def float_value?(value)
+          case value
+          when String
+            begin
+              Float(value)
+              true
+            rescue ArgumentError
+              false
+            end
+          when Float
+            true
+          else
+            false
+          end
+        end
+
+        def float_values?
+          @sample_values.all? do |sample_value|
+            float_value?(sample_value)
+          end
+        end
+
+        def geo_point_values?
+          @sample_values.all? do |sample_value|
+            sample_value.is_a?(String) and
+              /\A-?\d+(?:\.\d+)[,x]-?\d+(?:\.\d+)\z/ =~ sample_value
+          end
+        end
       end
 
-      def format(tag, time, record)
-        [tag, time, record].to_msgpack
+      class Table
+        def initialize(name, key_type)
+          @name = name
+          @key_type = key_type
+        end
       end
 
-      def write(chunk)
-        chunk.msgpack_each do |tag, time, record|
-          @emitter.emit(tag, record)
+      class Column
+        def initialize(name, value_type, vector_p)
+          @name = name
+          @value_type = value_type
+          @vector_p = vector_p
         end
       end
     end
 
-    class HTTPClient
-      include Configurable
-
-      config_param :host, :string, :default => "localhost"
-      config_param :port, :integer, :default => 10041
+    class Emitter
+      def initialize(client, table)
+        @client = client
+        @table = table
+        @schema = nil
+      end
 
       def start
-        @loop = Coolio::Loop.new
+        @schema = Schema.new(@client, @table)
       end
 
       def shutdown
       end
 
-      def send(command)
-        client = GroongaHTTPClient.connect(@host, @port)
-        client.request("GET", command.to_uri_format)
-        @loop.attach(client)
-        @loop.run
+      def emit(chunk)
+        records = []
+        chunk.msgpack_each do |message|
+          tag, _, record = message
+          if /\Agroonga\.command\./ =~ tag
+            name = $POSTMATCH
+            unless records.empty?
+              store_records(records)
+              records.clear
+            end
+            @client.execute(name, record)
+          else
+            records << record
+          end
+        end
+        store_records(records) unless records.empty?
       end
 
-      class GroongaHTTPClient < Coolio::HttpClient
-        def on_body_data(data)
-        end
+      private
+      def store_records(records)
+        return if @table.nil?
+
+        @schema.update(records)
+
+        arguments = {
+          "table" => @table,
+          "values" => Yajl::Encoder.encode(records),
+        }
+        @client.execute("load", arguments)
+      end
+    end
+
+    class BaseClient
+      private
+      def build_command(name, arguments={})
+        command_class = Groonga::Command.find(name)
+        command_class.new(name, arguments)
       end
     end
 
-    class GQTPClient
+    class NetworkClient < BaseClient
       include Configurable
 
-      config_param :host, :string, :default => "localhost"
-      config_param :port, :integer, :default => 10041
+      config_param :host, :string, :default => nil
+      config_param :port, :integer, :default => nil
+
+      def initialize(protocol)
+        super()
+        @protocol = protocol
+      end
 
       def start
-        @loop = Coolio::Loop.new
         @client = nil
       end
 
       def shutdown
         return if @client.nil?
-        @client.close do
-          @loop.stop
-        end
-        @loop.run
+        @client.close
       end
 
-      def send(command)
-        @client ||= GQTP::Client.new(:address => @host,
-                                     :port => @port,
-                                     :connection => :coolio,
-                                     :loop => @loop)
-        @client.send(command.to_command_format) do |header, body|
-          @loop.stop
-        end
-        @loop.run
+      def execute(name, arguments={})
+        command = build_command(name, arguments)
+        @client ||= Groonga::Client.new(:protocol => @protocol,
+                                        :host     => @host,
+                                        :port     => @port,
+                                        :backend  => :synchronous)
+        @client.execute(command)
       end
     end
 
-    class CommandClient
+    class CommandClient < BaseClient
       include Configurable
 
       config_param :groonga, :string, :default => "groonga"
@@ -222,42 +375,45 @@ module Fluent
 
       def start
         run_groonga
-        wrap_io
       end
 
       def shutdown
-        @groonga_input.close
-        @groonga_output.close
-        @groonga_error.close
+        @input.close
+        read_output("shutdown")
+        @output.close
+        @error.close
         Process.waitpid(@pid)
       end
 
-      def send(command)
+      def execute(name, arguments={})
+        command = build_command(name, arguments)
         body = nil
         if command.name == "load"
           body = command.arguments.delete(:values)
         end
-        @groonga_input.write("#{command.to_uri_format}\n")
+        uri = command.to_uri_format
+        @input.write("#{uri}\n")
         if body
           body.each_line do |line|
-            @groonga_input.write("#{line}\n")
+            @input.write("#{line}\n")
           end
         end
-        @loop.run
+        @input.flush
+        read_output(uri)
       end
 
       private
       def run_groonga
         env = {}
-        @input = IO.pipe("ASCII-8BIT")
-        @output = IO.pipe("ASCII-8BIT")
-        @error = IO.pipe("ASCII-8BIT")
-        input_fd = @input[0].to_i
-        output_fd = @output[1].to_i
+        input = IO.pipe("ASCII-8BIT")
+        output = IO.pipe("ASCII-8BIT")
+        error = IO.pipe("ASCII-8BIT")
+        input_fd = input[0].to_i
+        output_fd = output[1].to_i
         options = {
           input_fd => input_fd,
           output_fd => output_fd,
-          :err => @error[1],
+          :err => error[1],
         }
         arguments = @arguments
         arguments += [
@@ -270,27 +426,42 @@ module Fluent
         end
         arguments << @database
         @pid = spawn(env, @groonga, *arguments, options)
-        @input[0].close
-        @output[1].close
-        @error[1].close
+        input[0].close
+        @input = input[1]
+        output[1].close
+        @output = output[0]
+        error[1].close
+        @error = error[0]
       end
 
-      def wrap_io
-        @loop = Coolio::Loop.new
+      def read_output(context)
+        output_message = ""
+        error_message = ""
+
+        loop do
+          readables = IO.select([@output, @error], nil, nil, 0)
+          break if readables.nil?
+
+          readables.each do |readable|
+            case readable
+            when @output
+              output_message << @output.gets
+            when @error
+              error_message << @error.gets
+            end
+          end
+        end
 
-        @groonga_input = Coolio::IO.new(@input[1])
-        on_write_complete = lambda do
-          @loop.stop
+        unless output_message.empty?
+          Engine.log.debug("[output][groonga][output]",
+                           :context => context,
+                           :message => output_message)
         end
-        @groonga_input.on_write_complete do
-          on_write_complete.call
+        unless error_message.empty?
+          Engine.log.error("[output][groonga][error]",
+                           :context => context,
+                           :message => error_message)
         end
-        @groonga_output = Coolio::IO.new(@output[0])
-        @groonga_error = Coolio::IO.new(@error[0])
-
-        @loop.attach(@groonga_input)
-        @loop.attach(@groonga_output)
-        @loop.attach(@groonga_error)
       end
     end
   end