fluent-plugin-mqtt-io 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: dbaced6d7ef1c77842544fb27125007e2b9635c5
+  data.tar.gz: ab568a296508ad1a8f65fb5349193b2cd999c78f
+SHA512:
+  metadata.gz: ca0f40329239b618e5b5d588f9579c3b57d05db0786ff6ba5091fabc64f3cd27f5aa5d0be50d46900a60100ccfb791a7ee8acbcfd6acbbc7ce22d36fdb5a255b
+  data.tar.gz: ce8103acef9c93ae89ef2f747f7fcc69142ed49a960fe8dee684fa78a6c6f2817d0d3ba23a6e5fa450cdbef37114b1bca4338b131b77ca1298d10eded731cf0a

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.travis.yml

@@ -0,0 +1,4 @@
+language: ruby
+rvm:
+  - 2.2.2
+before_install: gem install bundler -v 1.10.2

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,13 @@
+# Contributor Code of Conduct
+
+As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, or religion.
+
+Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team.
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers.
+
+This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.0.0, available at [http://contributor-covenant.org/version/1/0/0/](http://contributor-covenant.org/version/1/0/0/)

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fluent-plugin-mqtt-io.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Toyokazu Akiyama
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

data/README.md

@@ -0,0 +1,122 @@
+# Fluent::Plugin::Mqtt::IO
+
+Fluent plugin for MQTT Input/Output
+Mqtt::IO Plugin is deeply inspired by Fluent::Plugin::Mqtt.
+
+https://github.com/yuuna/fluent-plugin-mqtt
+
+Mqtt::IO plugin focus on federating components, e.g. sensors, messaging platform and databases.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'fluent-plugin-mqtt-io'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install fluent-plugin-mqtt-io
+
+
+## Usage
+
+fluent-plugin-mqtt provides Input and Output Plugins for MQTT.
+
+Input Plugin can be used via source directive in the configuration.
+
+```
+
+<source>
+  type mqtt
+  bind 127.0.0.1
+  port 1883
+</source>
+
+```
+
+The default MQTT topic is "#". Configurable options are the following:
+
+- bind: IP address of MQTT broker
+- port: Port number of MQTT broker
+- topic: Topic name to be subscribed
+- username: User name for authentication
+- password: Password for authentication
+- ssl: set true if you want to use SSL/TLS. If set to true, the following parameter must be provided
+  - ca_file: CA certificate file path
+  - key_file: private key file path
+  - cert_file: certificate file path
+- format: Input parser can be chosen, e.g. json, xml
+
+Input Plugin supports @label directive.
+
+Output Plugin can be used via match directive.
+
+```
+
+<match topic.**>
+  type mqtt
+  bind 127.0.0.1
+  port 1883
+</match>
+
+```
+
+The options are basically the same as Input Plugin. The difference is the following.
+
+- time_key: An attribute name used for timestamp field. Default is 'timestamp'.
+- time_format: Output format of timestamp field. Default is ISO8601. You can specify your own format by using TimeParser.
+- topic_rewrite_pattern: Regexp pattern to extract replacement words from received topic or tag name
+- topic_rewrite_replacement: Topic name used for the publish using extracted pattern
+
+If you use different source, e.g. the other MQTT broker, log file and so on, there is no need to specifie topic rewriting. Skip the following descriptions.
+
+The topic name or tag name, e.g. "topic", received from an event can not be published without modification because if MQTT input plugin connecting to the identical MQTT broker is used as a source, the same message will become an input repeatedly. In order to support data conversion in single MQTT domain, simple topic rewriting should be supported. Since topic is rewritten using #gsub method, 'pattern' and 'replacement' are the same as #gsub arguments.
+
+
+```
+
+<match topic.**>
+  type mqtt
+  bind 127.0.0.1
+  port 1883
+  topic_rewrite_pattern '^([\w\/]+)$'
+  topic_rewrite_replacement '\1/rewritten'
+</match>
+
+```
+
+You can also use mqtt_buf type which is implemented as BufferedOutput.
+
+```
+
+<match topic.**>
+  type mqtt_buf
+  bind 127.0.0.1
+  port 1883
+  topic_rewrite_pattern '^([\w\/]+)$'
+  topic_rewrite_replacement '\1/rewritten'
+  # You can specify Buffer Plugin options
+  buffer_type memory
+  flush_interval 1s
+</match>
+
+```
+
+
+## Contributing
+
+1. Fork it ( http://github.com/toyokazu/fluent-plugin-mqtt-io/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList['test/**/*_test.rb']
+end
+
+task :default => :test

data/fluent-plugin-mqtt-io.gemspec

@@ -0,0 +1,25 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |spec|
+  spec.name          = "fluent-plugin-mqtt-io"
+  spec.version       = "0.0.1"
+  spec.authors       = ["Toyokazu Akiyama"]
+  spec.email         = ["toyokazu@gmail.com"]
+
+  spec.summary       = %q{fluentd input/output plugin for mqtt broker}
+  spec.description   = %q{fluentd input/output plugin for mqtt broker}
+  spec.homepage      = "https://github.com/toyokazu/fluent-plugin-mqtt-io"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.bindir        = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.10"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "mqtt"
+end

data/lib/fluent/plugin/in_mqtt.rb

@@ -0,0 +1,78 @@
+module Fluent
+  class MqttInput < Input
+    Plugin.register_input('mqtt', self)
+
+    config_param :port, :integer, :default => 1883
+    config_param :bind, :string, :default => '127.0.0.1'
+    config_param :topic, :string, :default => '#'
+    config_param :format, :string, :default => 'none'
+    config_param :username, :string, :default => nil
+    config_param :password, :string, :default => nil
+    config_param :ssl, :bool, :default => nil
+    config_param :ca_file, :string, :default => nil
+    config_param :key_file, :string, :default => nil
+    config_param :cert_file, :string, :default => nil
+
+    require 'mqtt'
+
+    # Define `router` method of v0.12 to support v0.10 or earlier
+    unless method_defined?(:router)
+      define_method("router") { Fluent::Engine }
+    end
+
+    def configure(conf)
+      super
+      @bind ||= conf['bind']
+      @topic ||= conf['topic']
+      @port ||= conf['port']
+      @username ||= conf['username']
+      @password ||= conf['password']
+      configure_parser(conf)
+    end
+
+    def configure_parser(conf)
+      @parser = Plugin.new_parser(conf['format'])
+      @parser.configure(conf)
+    end
+
+    def start
+      $log.debug "start mqtt #{@bind}"
+      opts = {
+        host: @bind,
+        port: @port,
+        username: @username,
+        password: @password
+      }
+      opts[:ssl] = @ssl if @ssl
+      opts[:ca_file] = @ca if @ca
+      opts[:cert_file] = @crt if @crt
+      opts[:key_file] = @key if @key
+      @connect = MQTT::Client.connect(opts)
+      @connect.subscribe(@topic)
+
+      @thread = Thread.new do
+        @connect.get do |topic,message|
+          emit topic, message
+        end
+      end
+    end
+
+    def emit topic, message
+      begin
+        topic.gsub!("/","\.")
+        @parser.parse(message) {|time, record|
+          $log.debug "#{topic}, #{time}, #{record}"
+          router.emit(topic, time, record)
+        }
+      rescue Exception => e
+        $log.error :error => e.to_s
+        $log.debug_backtrace(e.backtrace)
+      end
+    end
+
+    def shutdown
+      @thread.kill
+      @connect.disconnect
+    end
+  end
+end

data/lib/fluent/plugin/mqtt_output_mixin.rb

@@ -0,0 +1,94 @@
+module Fluent
+  module MqttOutputMixin
+    # config_param defines a parameter. You can refer a parameter via @path instance variable
+    # Without :default, a parameter is required.
+    
+    def self.included(base)
+      base.config_param :port, :integer, :default => 1883
+      base.config_param :bind, :string, :default => '127.0.0.1'
+      base.config_param :username, :string, :default => nil
+      base.config_param :password, :string, :default => nil
+      base.config_param :ssl, :bool, :default => nil
+      base.config_param :ca_file, :string, :default => nil
+      base.config_param :key_file, :string, :default => nil
+      base.config_param :cert_file, :string, :default => nil
+      base.config_param :time_key, :string, :default => 'timestamp'
+      base.config_param :time_format, :string, :default => nil
+      base.config_param :topic_rewrite_pattern, :string, :default => nil
+      base.config_param :topic_rewrite_replacement, :string, :default => nil
+    end
+
+    require 'mqtt'
+
+    # This method is called before starting.
+    # 'conf' is a Hash that includes configuration parameters.
+    # If the configuration is invalid, raise Fluent::ConfigError.
+    def configure(conf)
+      super
+
+      # You can also refer raw parameter via conf[name].
+      @bind ||= conf['bind']
+      @port ||= conf['port']
+      @username ||= conf['username']
+      @password ||= conf['password']
+      @time_key ||= conf['time_key']
+      @time_format ||= conf['time_format']
+      @topic_rewrite_pattern ||= conf['topic_rewrite_pattern']
+      @topic_rewrite_replacement ||= conf['topic_rewrite_replacement']
+    end
+
+    # This method is called when starting.
+    # Open sockets or files here.
+    def start
+      super
+
+      $log.debug "start mqtt #{@bind}"
+      opts = {
+        host: @bind,
+        port: @port,
+        username: @username,
+        password: @password
+      }
+      opts[:ssl] = @ssl if @ssl
+      opts[:ca_file] = @ca if @ca
+      opts[:cert_file] = @crt if @crt
+      opts[:key_file] = @key if @key
+      @connect = MQTT::Client.connect(opts)
+    end
+
+    # This method is called when shutting down.
+    # Shutdown the thread and close sockets or files here.
+    def shutdown
+      super
+
+      @connect.disconnect
+    end
+
+    def format_time(time)
+      if @time_format.nil?
+        Time.at(time).iso8601
+      else
+        time_parser = TimeParser.new(@time_format)
+        time_parser.parse(time)
+      end
+    end
+
+    def rewrite_tag(tag)
+      if @topic_rewrite_pattern.nil?
+        tag.gsub("\.", "/")
+      else
+        tag.gsub("\.", "/").gsub(Regexp.new(@topic_rewrite_pattern), @topic_rewrite_replacement)
+      end
+    end
+
+    def json_parse message
+      begin
+        y = Yajl::Parser.new
+        y.parse(message)
+      rescue
+        $log.error "JSON parse error", :error => $!.to_s, :error_class => $!.class.to_s
+        $log.warn_backtrace $!.backtrace         
+      end
+    end
+  end
+end

data/lib/fluent/plugin/out_mqtt.rb

@@ -0,0 +1,20 @@
+module Fluent
+  class MqttOutput < Output
+    require 'fluent/plugin/mqtt_output_mixin'
+    include Fluent::MqttOutputMixin
+
+    # First, register the plugin. NAME is the name of this plugin
+    # and identifies the plugin in the configuration file.
+    Fluent::Plugin.register_output('mqtt', self)
+
+    def emit(tag, es, chain)
+      es.each {|time,record|
+        $log.debug "#{tag}, #{format_time(time)}, #{record}"
+        @connect.publish(rewrite_tag(tag), record.merge(@time_key => format_time(time)).to_json)
+      }
+      $log.flush
+
+      chain.next
+    end
+  end
+end

data/lib/fluent/plugin/out_mqtt_buf.rb

@@ -0,0 +1,30 @@
+module Fluent
+  class MqttOutput < BufferedOutput
+    require 'fluent/plugin/mqtt_output_mixin'
+    include Fluent::MqttOutputMixin
+
+    # First, register the plugin. NAME is the name of this plugin
+    # and identifies the plugin in the configuration file.
+    Fluent::Plugin.register_output('mqtt_buf', self)
+
+    # This method is called when an event reaches to Fluentd.
+    # Convert the event to a raw string.
+    def format(tag, time, record)
+      [tag, time, record].to_json + "\n"
+    end
+
+    # This method is called every flush interval. Write the buffer chunk
+    # to files or databases here.
+    # 'chunk' is a buffer chunk that includes multiple formatted
+    # events. You can use 'data = chunk.read' to get all events and
+    # 'chunk.open {|io| ... }' to get IO objects.
+    #
+    # NOTE! This method is called by internal thread, not Fluentd's main thread. So IO wait doesn't affect other plugins.
+    def write(chunk)
+      json = json_parse(chunk.read)
+      $log.debug "#{json[0]}, #{format_time(json[1])}, #{json[2]}"
+      @connect.publish(rewrite_tag(json[0]), (json[2].merge(@time_key => format_time(json[1]))).to_json)
+      $log.flush
+    end
+  end
+end

data/test/helper.rb

@@ -0,0 +1,26 @@
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'test/unit'
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'fluent/test'
+unless ENV.has_key?('VERBOSE')
+  nulllogger = Object.new
+  nulllogger.instance_eval {|obj|
+    def method_missing(method, *args)
+      # pass
+    end
+  }
+  $log = nulllogger
+end
+
+class Test::Unit::TestCase
+end

metadata

@@ -0,0 +1,100 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-mqtt-io
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Toyokazu Akiyama
+autorequire: 
+bindir: []
+cert_chain: []
+date: 2015-11-05 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: mqtt
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: fluentd input/output plugin for mqtt broker
+email:
+- toyokazu@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".gitignore"
+- ".travis.yml"
+- CODE_OF_CONDUCT.md
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- fluent-plugin-mqtt-io.gemspec
+- lib/fluent/plugin/in_mqtt.rb
+- lib/fluent/plugin/mqtt_output_mixin.rb
+- lib/fluent/plugin/out_mqtt.rb
+- lib/fluent/plugin/out_mqtt_buf.rb
+- test/helper.rb
+homepage: https://github.com/toyokazu/fluent-plugin-mqtt-io
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5
+signing_key: 
+specification_version: 4
+summary: fluentd input/output plugin for mqtt broker
+test_files:
+- test/helper.rb