fluent-plugin-groonga 1.0.0 → 1.0.1

data/.yardopts

@@ -0,0 +1,5 @@
+--output-dir doc/reference/en
+--use-cache .yardoc
+lib/**/*.rb
+-
+doc/text/**/*

data/README.md

@@ -1,3 +1,5 @@
+# @title README
+
 # README
 
 ## Name
@@ -16,7 +18,7 @@ plugin. Both of them are named `groonga`.
 The input plugin provides groonga compatible interface. It means that
 HTTP and GQTP interface. You can use the input plugin as groonga
 server. The input plugin receives groonga commands and sends them to
-the output plugin through some fluentds.
+the output plugin through zero or more fluentds.
 
 The output plugin sends received groonga commands to groonga. The
 output plugin supports all interfaces, HTTP, GQTP and command
@@ -30,57 +32,10 @@ You can replicate your data by using `copy` output plugin.
 
 ## Usage
 
-### The `groonga` input plugin
-
-Here are available parameters:
-
-* `protocol`: It specifies protocol for receiving groonga commands.
-  * available values: `http` and `gqtp`
-  * default: `http`
-* `bind`: It specifies bind address.
-   * default: `0.0.0.0`
-* `port`: It specifies port number.
-   * default: `10041`
-* `real_host`: It specifies real groonga server's address. It is required.
-   * default: no default.
-* `real_port`: It specifies real groonga server's port number.
-   * default: `10041`
-* `emit_commands`: TODO
-
-Here is an example:
-
-    <source>
-      type groonga
-      protocol http
-      bind 127.0.0.1
-      port 10041
-      real_host 192.168.0.1
-      real_port 10041
-    </source>
-
-### The `groonga` output plugin
-
-* `protocol`: It specifies protocol for sending groonga commands to groonga.
-  * available values: `http`, `gqtp` and `command`
-  * default: `http`
-* For `http` and `gqtp` use:
-  * `host`: It specifies groonga server's address.
-     * default: `localhost`
-  * `port`: It specifies groonga server's port number.
-     * default: `10041`
-* For `command` use:
-  * `groonga`: It specifies path of groonga command.
-     * default: `groonga`
-  * `database`: It specifies path of groonga database. It is required.
-     * default: no default.
-  * `arguments`: It specifies additional arguments for groonga command.
-     * default: no additional arguments.
-
-    <match groonga.command.*>
-      type groonga
-      protocol command
-      database /tmp/groonga/db
-    </match>
+* [Configuration](doc/text/configuration.md)
+  ([on the Web](http://groonga.org/fluent-plugin-groonga/en/file.configuration.html))
+* [Constitution](doc/text/constitution.md)
+  ([on the Web](http://groonga.org/fluent-plugin-groonga/en/file.constitution.html))
 
 ## Authors
 

data/doc/text/configuration.md

@@ -0,0 +1,61 @@
+# @title Configuration
+
+# Configuration
+
+Fluent-plugin-groonga includes two fluentd plugins. They are the
+`groonga` input plugin and the `groonga` output plugin. This documents
+describes configuration parameters of them.
+
+## The `groonga` input plugin
+
+Here are available parameters:
+
+* `protocol`: It specifies protocol for receiving groonga commands.
+  * available values: `http` and `gqtp`
+  * default: `http`
+* `bind`: It specifies bind address.
+   * default: `0.0.0.0`
+* `port`: It specifies port number.
+   * default: `10041`
+* `real_host`: It specifies real groonga server's address. It is required.
+   * default: no default.
+* `real_port`: It specifies real groonga server's port number.
+   * default: `10041`
+* `emit_commands`: TODO
+
+Here is an example:
+
+    <source>
+      type groonga
+      protocol http
+      bind 127.0.0.1
+      port 10041
+      real_host 192.168.0.1
+      real_port 10041
+    </source>
+
+## The `groonga` output plugin
+
+* `protocol`: It specifies protocol for sending groonga commands to groonga.
+  * available values: `http`, `gqtp` and `command`
+  * default: `http`
+* For `http` and `gqtp` use:
+  * `host`: It specifies groonga server's address.
+     * default: `localhost`
+  * `port`: It specifies groonga server's port number.
+     * default: `10041`
+* For `command` use:
+  * `groonga`: It specifies path of groonga command.
+     * default: `groonga`
+  * `database`: It specifies path of groonga database. It is required.
+     * default: no default.
+  * `arguments`: It specifies additional arguments for groonga command.
+     * default: no additional arguments.
+
+Here is an example:
+
+    <match groonga.command.*>
+      type groonga
+      protocol command
+      database /tmp/groonga/db
+    </match>

data/doc/text/constitution.md

@@ -0,0 +1,315 @@
+# @title Constitution
+
+# Constitution
+
+You can chose some system constitutions to implemented replication
+ready groonga system. This document describes some patterns.
+
+Here are available patterns:
+
+  * Master slave replication
+  * Resending data to recovered slave
+
+Here are unavailable patterns:
+
+  * Multi master replication
+  * Auto slave recovery
+  * Dynamic slave adding
+  * Failover
+  * No SPOF (Single Point of Failure) without downing service level
+
+## Master slave replication
+
+Master slave replication is available. This section describes how to
+configure your system constitution.
+
+### Small system
+
+In small system, you just has two servers. One is the master groonga
+server and the other is the slave groonga server. You send all update
+commands (e.g. `table_create`, `column_create`, `load` and so on.) to
+fluentd. In fluentd, the `groonga` input plugin receives commands from
+client, passes through them to master groonga server and passes
+through responses from master groonga server to client. The `groonga`
+input plugin converts update commands to fluent messages when the
+`groonga` input plugin passes through comamands and responses. The
+fluent messages are sent to slave groonga server by the `groonga`
+output plugin.
+
+Here is a diagram of this constitution.
+
+                update               update
+                 and                  and
+                search  +---------+  search  +---------+
+    +--------+ <------> | fluentd | <------> | master  |
+    |        |          +---------+          | groonga |
+    | client |        update |               +---------+
+    |        |              \_/
+    |        |  search  +---------+
+    +--------+ <------> |  slave  |
+                        | groonga |
+                        +---------+
+
+Fluentd should be placed at client or master groonga server. If you
+have only one client that updates data, client side is reasonable. If
+you have multiple clients that update data, master groonga server side
+is reasonable.
+
+You can use replication for high performance by providing search
+service with multi servers. You can't use replication for high
+availability. If master groonga server or fluentd is down, this system
+can't update data. (Searching is still available because slabe groonga
+server is alive.)
+
+Here is an example configuration file:
+
+    # For master groonga server
+    <source>
+      type groonga
+      protocol gqtp          # Or use the below line
+      # protocol http
+      bind 127.0.0.1         # For client side fluentd
+      # bind 192.168.0.1     # For master groonga server side fluentd
+      port 10041
+      real_host 192.168.29.1 # IP address of master groonga server
+      real_port 10041        # Port number of master groonga server
+      # real_port 20041      # Use different port number
+                             # for master groonga server side fluentd
+    </source>
+
+    # For slave groonga server
+    <match groonga.command.*>
+      type groonga
+      protocol gqtp            # Or use the below line
+      # protocol http          # You can use different protocol for
+                               # master groonga server and slave groonga server
+      host 192.168.29.29       # IP address of slave groonga server
+      port 10041               # Port number of slave groonga server
+
+      # Buffer
+      flush_interval 1s        # Use small value for less delay replication
+
+      ## Use the following configurations to support resending data to
+      ## recovered slave groonga server. If you don't care about slave
+      ## groonga server is down case, you don't need the following
+      ## configuration.
+
+      ## For supporting resending data after fluentd is restarted
+      # buffer_type file
+      # buffer_path /var/log/fluent/groonga.*.buffer
+      ## Use large value if a record has many data in load command.
+      ## A value in load command is a chunk.
+      # buffer_chunk_limit 256m
+      ## Use large value if you want to support resending data after
+      ## slave groonga server is down long time.
+      ## 17: about 1.5day =
+      ##       ((2 ** 0) + (2 ** 1) + ... + (2 ** 17)) / 60.0 / 60.0 / 24.0
+      ##     (default)
+      ## 18: about 3.0day = ((2 ** 0) + ... + (2 ** 18)) / ...
+      ## 19: about 6.0day = ((2 ** 0) + ... + (2 ** 19)) / ...
+      # retry_limit 19
+      ## Use large value if you load many records.
+      ## A value in load command is a chunk.
+      # buffer_queue_limit 10000
+    </match>
+
+#### How to recover from fluentd down
+
+If fluentd is down, you just restart fluentd. Note that you may resend
+the last command if fluentd is down while you are sending update
+commands.
+
+You cannot update data until fluentd is up.
+
+#### How to recover from master groonga server down
+
+Here are recover steps when master groonga server is down:
+
+  1. Stop fluentd.
+  2. Run `grndump /PATH/TO/SLAVE/GROONGA/SERVER/DB >
+     SLAVE_GROONGA_DUMP.grn` on slave groonga server host.
+  3. Run `groonga -n /PATH/TO/MASTER/GROONGA/SERVER/DB <
+     SLAVE_GROONGA_DUMP.grn` on master groonga server.
+  4. Run master groonga server.
+  5. Start fluentd.
+
+You cannot update data until you finish to recover.
+
+#### How to recover from slave groonga server down
+
+Here are recover steps when slave groonga server is down:
+
+  1. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
+     MASTER_GROONGA_DUMP.grn` on master groonga server host.
+  2. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
+     MASTER_GROONGA_DUMP.grn` on slave groonga server.
+  3. Run slave groonga server.
+
+You can update data while you recover. If your system can't process
+all search requests by only master groonga server, your system will be
+down.
+
+You need to recover slave groonga server before fluentd's buffer queue
+is full (see `buffer_queue_limit`) or fluentd gives up retrying (see
+`retry_limit`). Here are recover steps when you cannot recover slave
+groonga server before those situations:
+
+  1. Stop fluentd.
+  2. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
+     MASTER_GROONGA_DUMP.grn` on master groonga server host.
+  3. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
+     MASTER_GROONGA_DUMP.grn` on slave groonga server host.
+  4. Run slave groonga server.
+  5. Start fluentd.
+
+You cannot update data until you finish to recover.
+
+### Medium system
+
+In medium system, you have three or more slave groonga servers. Fluentd
+updates two or more slave groonga servers with the `copy` output
+plugin and the `groonga` output plugin.
+
+Here is a diagram of this constitution.
+
+                update               update
+                 and                  and
+                search  +---------+  search  +---------+
+    +--------+ <------> | fluentd | <------> | master  |
+    |        |          +---------+          | groonga |
+    | client |               +--------+      +---------+
+    |        |                        |
+    +--------+  search  +---------+   |
+    |        | <------> |  slave  | <-+ update
+    | client |          | groonga |   |
+    |        |          +---------+   |
+    +--------+  search  +---------+   |
+    |        | <------> |  slave  | <-+ update
+    | client |          | groonga |   |
+    |        |          +---------+   |
+    +- ...  -+   ...        ...      ...
+
+Here is an example configuration file:
+
+    # For master groonga server
+    <source>
+      type groonga
+      protocol gqtp          # Or use the below line
+      # protocol http
+      bind 127.0.0.1         # For client side fluentd
+      # bind 192.168.0.1     # For master groonga server side fluentd
+      port 10041
+      real_host 192.168.29.1 # IP address of master groonga server
+      real_port 10041        # Port number of master groonga server
+      # real_port 20041      # Use different port number
+                             # for master groonga server side fluentd
+    </source>
+
+    # For slave groonga servers
+    <match groonga.command.*>
+      type copy
+
+      # The first slave groonga server
+      <store>
+        type groonga
+        protocol gqtp            # Or use the below line
+        # protocol http          # You can use different protocol for
+                                 # master groonga server and slave groonga server
+        host 192.168.29.2        # IP address of slave groonga server
+        port 10041               # Port number of slave groonga server
+
+        # Buffer
+        flush_interval 1s        # Use small value for less delay replication
+
+        ## Use the following configurations to support resending data to
+        ## recovered slave groonga server. If you don't care about slave
+        ## groonga server is down case, you don't need the following
+        ## configuration.
+
+        ## For supporting resending data after fluentd is restarted
+        # buffer_type file
+        # buffer_path /var/log/fluent/groonga.*.buffer
+        ## Use large value if a record has many data in load command.
+        ## A value in load command is a chunk.
+        # buffer_chunk_limit 256m
+        ## Use large value if you want to support resending data after
+        ## slave groonga server is down long time.
+        ## 17: about 1.5day =
+        ##       ((2 ** 0) + (2 ** 1) + ... + (2 ** 17)) / 60.0 / 60.0 / 24.0
+        ##     (default)
+        ## 18: about 3.0day = ((2 ** 0) + ... + (2 ** 18)) / ...
+        ## 19: about 6.0day = ((2 ** 0) + ... + (2 ** 19)) / ...
+        # retry_limit 19
+        ## Use large value if you load many records.
+        ## A value in load command is a chunk.
+        # buffer_queue_limit 10000
+      </store>
+
+      # The second slave groonga server
+      <store>
+        type groonga
+        protocol gqtp            # Or use the below line
+        # protocol http          # You can use different protocol for
+                                 # master groonga server and slave groonga server
+        host 192.168.29.3        # IP address of slave groonga server
+        port 10041               # Port number of slave groonga server
+
+        # Buffer
+        # ...
+      </store>
+
+      # More slave groonga servers
+      # <store>
+      #   type groonga
+      #   ...
+      # </store>
+    </match>
+
+TODO: ...
+
+### Large system
+
+In large system, you have two or more slave groonga server clusters.
+Fluentd that connects with master groonga server updates two or more
+fluentds that are in slave groonga server clusters with the `copy`
+output plugin and the `forward` output plugin. A slave cluster has a
+fluentd. Fluentd in slave groonga server clusters updates slave
+groonga server in the same slave groonga server cluster by the `copy`
+output plugin and `groonga` output plugin.
+
+Here is a diagram of this constitution.
+
+                update                 update
+                 and                    and
+                search    +---------+  search  +---------+
+    +--------+ <--------> | fluentd | <------> | master  |
+    |        |            +---------+          | groonga |
+    | client |                |                +---------+
+    |        |                +------------------------------+
+    +--------+          +----------------------------------+ |
+    |        |          |        slave cluster             | |
+    | client |  search  | +---------+  update  +---------+ | |
+    |        | <------> | |  slave  | <------- | fluentd | <-+ update
+    +--------|          | | groonga |          +---------+ | |
+    |        |          | +---------+   +-----------+      | |
+    | client |  search  | +---------+   |                  | |
+    |        | <------> | |  slave  | <-+ update           | |
+    +--------|          | | groonga |   |                  | |
+    |        |          | +---------+   |                  | |
+    |  ...   |   ...    |     ...      ...                 | |
+                        +----------------------------------+ |
+    +--------+          +----------------------------------+ |
+    |        |          |        slave cluster             | |
+    | client |  search  | +---------+  update  +---------+ | |
+    |        | <------> | |  slave  | <------- | fluentd | <-+ update
+    +--------|          | | groonga |          +---------+ | |
+    |        |          | +---------+   +-----------+      | |
+    | client |  search  | +---------+   |                  | |
+    |        | <------> | |  slave  | <-+ update           | |
+    +--------|          | | groonga |   |                  | |
+    |        |          | +---------+   |                  | |
+    |  ...   |   ...    |     ...      ...                 | |
+                        +----------------------------------+ |
+                                      ...                   ...
+
+TODO: ...

data/doc/text/news.md

@@ -1,5 +1,20 @@
+# @title News
+
 # News
 
+## 1.0.1: 2012-12-29
+
+### Improvements
+
+* Added more destructive emit commands ("delete", "register", "truncate").
+* [out] Used close instead of sending "shutdown".
+* Placed documents to http://groonga.org/fluent-plugin-groonga/en/.
+* [doc] Updated documents:
+  * Added the documents of configuration and constitution.
+  * Added recover steps.
+  * Added documentation about master slave replication in
+    [small/medium/large] system.
+
 ## 1.0.0: 2012-11-29
 
 The first release!!!

data/fluent-plugin-groonga.gemspec

@@ -17,7 +17,7 @@
 
 Gem::Specification.new do |spec|
   spec.name = "fluent-plugin-groonga"
-  spec.version = "1.0.0"
+  spec.version = "1.0.1"
   spec.authors = ["Kouhei Sutou"]
   spec.email = ["kou@clear-code.com"]
   spec.summary = "Fluentd plugin collection for groonga users"
@@ -26,9 +26,11 @@ Gem::Specification.new do |spec|
   spec.homepage = "https://github.com/groonga/fluent-plugin-groonga"
 
   spec.files = ["README.md", "Gemfile", "#{spec.name}.gemspec"]
+  spec.files += [".yardopts"]
   spec.files += Dir.glob("lib/**/*.rb")
   spec.files += Dir.glob("sample/**/*")
   spec.files += Dir.glob("doc/text/**/*")
+  spec.test_files += Dir.glob("test/**/*")
   spec.require_paths = ["lib"]
 
   spec.add_runtime_dependency("fluentd")
@@ -37,6 +39,8 @@ Gem::Specification.new do |spec|
 
   spec.add_development_dependency("rake")
   spec.add_development_dependency("bundler")
+  spec.add_development_dependency("packnga", ">= 0.9.6")
   spec.add_development_dependency("test-unit")
   spec.add_development_dependency("test-unit-notify")
+  spec.add_development_dependency("redcarpet")
 end

data/lib/fluent/plugin/in_groonga.rb

@@ -82,6 +82,9 @@ module Fluent
       DEFAULT_EMIT_COMMANDS = [
         /\Atable_/,
         /\Acolumn_/,
+        "delete",
+        "register",
+        "truncate",
         "load",
       ]
       config_param :emit_commands, :default => DEFAULT_EMIT_COMMANDS do |value|

data/lib/fluent/plugin/out_groonga.rb

@@ -16,7 +16,6 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 
 require "fileutils"
-require "cgi/util"
 
 module Fluent
   class GroongaOutput < BufferedOutput
@@ -127,7 +126,7 @@ module Fluent
 
       def shutdown
         return if @client.nil?
-        @client.send("shutdown") do
+        @client.close do
           @loop.stop
         end
         @loop.run

data/test/run-test.rb

@@ -0,0 +1,42 @@
+#!/usr/bin/env ruby
+#
+# Copyright (C) 2012  Kouhei Sutou <kou@clear-code.com>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+# $VERBOSE = true
+
+Thread.abort_on_exception = true
+
+base_dir = File.expand_path(File.join(File.dirname(__FILE__), ".."))
+lib_dir = File.join(base_dir, "lib")
+test_dir = File.join(base_dir, "test")
+
+require "test-unit"
+require "test/unit/notify"
+
+Test::Unit::Priority.enable
+
+$LOAD_PATH.unshift(lib_dir)
+
+require "fluent/log"
+$log = Fluent::Log.new($stdout, Fluent::Log::LEVEL_WARN)
+
+Dir.glob("#{base_dir}/test/**/test{_,-}*.rb") do |file|
+  require file.sub(/\.rb$/, '')
+end
+
+ENV["TEST_UNIT_MAX_DIFF_TARGET_STRING_SIZE"] ||= "5000"
+
+exit Test::Unit::AutoRunner.run

data/test/test_input.rb

@@ -0,0 +1,176 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2012  Kouhei Sutou <kou@clear-code.com>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+require "time"
+require "cgi/util"
+require "net/http"
+require "webrick/config"
+require "webrick/httpresponse"
+
+require "fluent/test"
+require "fluent/plugin/in_groonga"
+
+require "http_parser"
+
+class GroongaInputTest < Test::Unit::TestCase
+  setup
+  def setup_fluent
+    Fluent::Test.setup
+    @now = Time.parse("2012-10-26T08:45:42Z").to_i
+    Fluent::Engine.now = @now
+  end
+
+  private
+  def create_driver
+    driver = Fluent::Test::InputTestDriver.new(Fluent::GroongaInput)
+    driver.configure(configuration)
+    driver
+  end
+
+  def configuration
+    <<-EOC
+EOC
+  end
+
+  class HTTPTest < self
+    setup :before => :append
+    def setup_real_server
+      @real_host = "127.0.0.1"
+      @real_port = 29292
+      @real_server = TCPServer.new(@real_host, @real_port)
+      @repeater = nil
+      response_config = WEBrick::Config::HTTP.dup.update(:Logger => $log)
+      @real_response = WEBrick::HTTPResponse.new(response_config)
+      Thread.new do
+        @repeater = @real_server.accept
+        @real_server.close
+        parser = HTTP::Parser.new
+        parser.on_message_complete = lambda do
+          @real_response.send_response(@repeater)
+          @repeater.close
+        end
+
+        loop do
+          break if @repeater.closed?
+          data = @repeater.readpartial(4096)
+          break if data.nil?
+          parser << data
+        end
+      end
+    end
+
+    teardown
+    def teardown_real_server
+      @real_server.close unless @real_server.closed?
+
+      if @repeater and not @repeater.closed?
+        @repeater.close
+      end
+    end
+
+    def setup
+      @host = "127.0.0.1"
+      @port = 2929
+
+      @driver = create_driver
+      @last_response = nil
+    end
+
+    def configuration
+      <<-EOC
+      protocol http
+      bind #{@host}
+      port #{@port}
+      real_host #{@real_host}
+      real_port #{@real_port}
+EOC
+    end
+
+    def test_target_command
+      @driver.expect_emit("groonga.command.table_create",
+                          @now,
+                          {"name" => "Users"})
+      @driver.run do
+        get("/d/table_create", "name" => "Users")
+        assert_equal("200", @last_response.code)
+      end
+    end
+
+    def test_not_target_command
+      @driver.run do
+        get("/d/status")
+        assert_equal("200", @last_response.code)
+      end
+      assert_empty(@driver.emits)
+    end
+
+    def test_load
+      json = <<-EOJ
+[
+{"name": "Alice"},
+{"name": "Bob"}
+]
+EOJ
+      @driver.expect_emit("groonga.command.load",
+                          @now,
+                          {
+                            "table" => "Users",
+                            "values" => json,
+                          })
+
+      @driver.run do
+        post("/d/load", json, "table" => "Users")
+        assert_equal("200", @last_response.code)
+      end
+    end
+
+    def test_not_command
+      @driver.run do
+        @real_response.status = 404
+        get("/index.html")
+        assert_equal("404", @last_response.code)
+      end
+    end
+
+    private
+    def get(path, parameters={})
+      http = Net::HTTP.new(@host, @port)
+      response = http.get(build_path(path, parameters))
+      @last_response = response
+      response
+    end
+
+    def post(path, body, parameters={})
+      http = Net::HTTP.new(@host, @port)
+      response = http.post(build_path(path, parameters),
+                           body,
+                           {"Content-Type" => "application/json"})
+      @last_response = response
+      response
+    end
+
+    def build_path(path, parameters)
+      unless parameters.empty?
+        url_encoded_parameters = parameters.collect do |key, value|
+          "#{CGI.escape(key)}=#{CGI.escape(value)}"
+        end
+        path += "?" + url_encoded_parameters.join("&")
+      end
+      path
+    end
+  end
+end

data/test/test_output.rb

@@ -0,0 +1,213 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2012  Kouhei Sutou <kou@clear-code.com>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1 as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+require "time"
+require "cgi/util"
+require "net/http"
+require "webrick/config"
+require "webrick/httpresponse"
+
+require "fluent/test"
+require "fluent/plugin/out_groonga"
+
+require "http_parser"
+
+class GroongaOutputTest < Test::Unit::TestCase
+  setup
+  def setup_fluent
+    Fluent::Test.setup
+  end
+
+  private
+  def create_driver(tag)
+    driver = Fluent::Test::BufferedOutputTestDriver.new(Fluent::GroongaOutput,
+                                                        tag)
+    driver.configure(configuration)
+    driver
+  end
+
+  def configuration
+    <<-EOC
+EOC
+  end
+
+  class HTTPTest < self
+    setup :before => :append
+    def setup_real_server
+      @real_host = "127.0.0.1"
+      @real_port = 29292
+      @real_server_pid = fork do
+        exit
+        real_server = TCPServer.new(@real_host, @real_port)
+        response_config = WEBrick::Config::HTTP.dup.update(:Logger => $log)
+        real_response = WEBrick::HTTPResponse.new(response_config)
+        request_headers = nil
+        request_body = ""
+        client = real_server.accept
+        real_server.close
+        parser = HTTP::Parser.new
+        parser.on_headers_complete = lambda do |headers|
+          request_headers = headers
+        end
+        parser.on_body = lambda do |chunk|
+          request_body << chunk
+        end
+        parser.on_message_complete = lambda do
+          real_response.send_response(client)
+          client.close
+        end
+
+        loop do
+          break if client.closed?
+          data = client.readpartial(4096)
+          break if data.nil?
+          parser << data
+        end
+      end
+    end
+
+    teardown
+    def teardown_real_server
+      Process.kill(:INT, @real_server_pid)
+      Process.kill(:KILL, @real_server_pid)
+      Process.waitpid(@real_server_pid)
+    end
+
+    def configuration
+      <<-EOC
+      protocol http
+      host #{@real_host}
+      port #{@real_port}
+EOC
+    end
+
+    class CommandTest < self
+      def test_basic_command
+        driver = create_driver("groonga.command.table_create")
+        time = Time.parse("2012-10-26T08:45:42Z").to_i
+        driver.emit({"name" => "Users"}, time)
+        driver.run
+        # p @request_headers
+        # p @request_body
+      end
+    end
+  end
+
+  class CommandLineTest < self
+    setup :before => :append
+    def setup_command
+      @temporary_directory = File.expand_path("tmp", File.dirname(__FILE__))
+      FileUtils.rm_rf(@temporary_directory)
+      FileUtils.mkdir_p(@temporary_directory)
+
+      @groonga_stub_path = File.join(@temporary_directory, "groonga")
+      @command_line_path = File.join(@temporary_directory, "command-line")
+      @input_path = File.join(@temporary_directory, "input")
+      @input_fd_path = File.join(@temporary_directory, "input-fd")
+      @output_fd_path = File.join(@temporary_directory, "output-fd")
+      @database_path = File.join(@temporary_directory, "database")
+
+      File.open(@groonga_stub_path, "w") do |groonga_stub|
+        groonga_stub.puts(<<-EOR)
+#!#{Gem.ruby}
+
+File.open(#{@command_line_path.inspect}, "a") do |file|
+  file.puts(ARGV)
+end
+
+input_fd = ARGV[ARGV.index("--input-fd") + 1]
+input = IO.new(input_fd.to_i)
+
+File.open(#{@input_fd_path.inspect}, "a") do |file|
+  file.print(input_fd)
+end
+
+File.open(#{@input_path.inspect}, "a") do |file|
+  input.each_line do |line|
+    file.print(line)
+  end
+end
+
+output_fd = ARGV[ARGV.index("--output-fd") + 1]
+output = IO.new(output_fd.to_i)
+
+File.open(#{@output_fd_path.inspect}, "a") do |file|
+  file.print(output_fd)
+end
+
+output.puts("done")
+output.flush
+EOR
+      end
+      FileUtils.chmod(0755, @groonga_stub_path)
+
+      FileUtils.touch(@command_line_path)
+      FileUtils.touch(@input_path)
+    end
+
+    teardown
+    def teardown_command
+      FileUtils.rm_rf(@temporary_directory)
+    end
+
+    def configuration
+      <<-EOC
+      protocol command
+      groonga #{@groonga_stub_path}
+      database #{@database_path}
+EOC
+    end
+
+    private
+    def actual_command_line
+      File.read(@command_line_path).split(/\n/)
+    end
+
+    def actual_input
+      File.read(@input_path)
+    end
+
+    def actual_input_fd
+      File.read(@input_fd_path)
+    end
+
+    def actual_output_fd
+      File.read(@output_fd_path)
+    end
+
+    class CommandTest < self
+      def test_basic_command
+        driver = create_driver("groonga.command.table_create")
+        time = Time.parse("2012-10-26T08:45:42Z")
+        driver.emit({"name" => "Users"}, time)
+        driver.run
+        assert_equal([
+                       [
+                         "--input-fd", actual_input_fd,
+                         "--output-fd", actual_output_fd,
+                         "-n", @database_path,
+                       ],
+                       "/d/table_create?name=Users\n",
+                     ],
+                     [
+                       actual_command_line,
+                       actual_input,
+                     ])
+      end
+    end
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-groonga
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.0.1
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-11-29 00:00:00.000000000 Z
+date: 2012-12-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -91,6 +91,22 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: packnga
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 0.9.6
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 0.9.6
 - !ruby/object:Gem::Dependency
   name: test-unit
   requirement: !ruby/object:Gem::Requirement
@@ -123,6 +139,22 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: redcarpet
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Groonga users can replicate their data by fluent-plugin-groonga
 email:
 - kou@clear-code.com
@@ -133,13 +165,19 @@ files:
 - README.md
 - Gemfile
 - fluent-plugin-groonga.gemspec
+- .yardopts
 - lib/fluent/plugin/out_groonga.rb
 - lib/fluent/plugin/in_groonga.rb
-- sample/http.conf
-- sample/command.conf
 - sample/gqtp.conf
+- sample/command.conf
+- sample/http.conf
 - doc/text/news.md
+- doc/text/constitution.md
 - doc/text/lgpl-2.1.txt
+- doc/text/configuration.md
+- test/run-test.rb
+- test/test_input.rb
+- test/test_output.rb
 homepage: https://github.com/groonga/fluent-plugin-groonga
 licenses: []
 post_install_message: 
@@ -152,17 +190,26 @@ required_ruby_version: !ruby/object:Gem::Requirement
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: -1000993838467277328
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: -1000993838467277328
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
 summary: Fluentd plugin collection for groonga users
-test_files: []
+test_files:
+- test/run-test.rb
+- test/test_input.rb
+- test/test_output.rb
 has_rdoc: