fluent-plugin-mqtt-io 0.0.4 → 0.0.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1c4996b6fab94e81580910023e41ba75014ded6f
-  data.tar.gz: 943ca60f56e475cd3d984430545e59369456c8e2
+  metadata.gz: b35a0ac007a40c1766646e950d678848fc4eb802
+  data.tar.gz: a4c1b6588ca8f74bfd311ea648d65a0aa2883beb
 SHA512:
-  metadata.gz: e7533937fe7d59eca39f8e41da1a57afc5595ab0b496cd210e8723664d3678f29ce1847f1af0a52038e1c73a7f3396dbe1849756fc751d6df24385779f605378
-  data.tar.gz: f668409785345ef55928f2dc1a23cae094d5e18425cd53bfd2265e2ef124951a4d31c49e7d890042f4e4d5b622160d1c522f3e9f08c1e4ec7c46e68ff31b8fdb
+  metadata.gz: 89c514b8981c952bccdd8de2aa8a7b23e9d39268dd34352301ebb6e6ae668277783abcb20f568f3138e06fbfb0c3cac685f80f35e2fe7b9201ff12a8ac2a0a83
+  data.tar.gz: c03a0b3312616ecea0431e1c5fdaa1c5e444ac896650fd6b7241839513baf91ce338feca03966c8e3a7e47f548157308ec4311dd98c44184aba688e53c60f554

data/README.md

@@ -24,7 +24,7 @@ Or install it yourself as:
 
 ## Usage
 
-fluent-plugin-mqtt provides Input and Output Plugins for MQTT.
+fluent-plugin-mqtt-io provides Input and Output Plugins for MQTT.
 
 Input Plugin can be used via source directive in the configuration.
 
@@ -50,7 +50,8 @@ The default MQTT topic is "#". Configurable options are the following:
   - key_file: private key file path
   - cert_file: certificate file path
 - format: Input parser can be chosen, e.g. json, xml
-  - in order to use xml format, you need to install [fluent-plugin-xml-parser](https://github.com/toyokazu/fluent-plugin-xml-parser]).
+  - in order to use xml format, you need to install [fluent-plugin-xml-parser](https://github.com/toyokazu/fluent-plugin-xml-parser).
+  - default format is json and default time_key for json format is 'time'
 
 Input Plugin supports @label directive.
 
@@ -109,8 +110,106 @@ You can also use mqtt_buf type which is implemented as BufferedOutput.
 ```
 
 
-## Example use case
+## Use case examples
 
+### Libelium sensor data collection
+
+There are many kinds of commercial sensor products on the market, e.g. [Libelium](http://www.libelium.com/). Major sensor products support MQTT to upload sensor data. The following example shows how to store uploaded Libelium sensor data into ElasticSearch.
+
+![Libelium sensor data collection](https://github.com/toyokazu/fluent-plugin-mqtt-io/blob/master/images/libelium_sensor_data_collection.png "Libelium sensor data")
+
+As described in the figure, fluent-plugin-mqtt-io, fluent-plugin-xml-parser and fluent-plugin-elasticsearch are used. The following is an example configuration.
+
+```
+<source>
+  type mqtt
+  bind 192.168.1.100
+  port 1883
+  topic 'Libelium/+/+'
+  format xml
+  time_xpath '["cap:alert/cap:info/cap:onset", "text"]'
+  time_key '@timestamp'
+  attr_xpaths '[["cap:alert/cap:info/cap:parameter/cap:valueName", "text"]]'
+  value_xpaths '[["cap:alert/cap:info/cap:parameter/cap:value", "text"]]'
+  @label @MQTT_OUT
+</source>
+
+<label @MQTT_OUT>
+  <match **>
+    <store>
+      type elasticsearch
+      host localhost
+      port 9200
+      index_name libelium
+      type_name smartcity
+      include_tag_key true
+      tag_key sensor_id
+      logstash_format false
+    </store>
+  </match>
+</label>
+
+```
+
+The following mapping is assumed to be created at ElasticSearch.
+
+```
+
+curl -XPUT 'http://localhost:9200/libelium/_mapping/smartcity' -d '
+  {"smartcity":
+    {"properties":
+      {
+        "sensor_id":{"type": "string"},
+        "DUST":{"type": "float"},
+        "MCP":{"type": "float"},
+        "HUMA":{"type": "float"},
+        "TCA":{"type": "float"},
+        "BAT":{"type": "float"},
+        "@timestamp":{"type":"date","format":"dateOptionalTime"}
+      }
+    }
+  }'
+
+```
+
+### MQTT message conversion
+
+Sometimes, MQTT message conversion must be done in the network because the processing entities does not have the conversion function. In that case, the configuration similar to the above example can be used. The difference resides output configuration. In this example, since the same MQTT broker is used to upload converted data, topic rewriting function is used for separating messages before and after conversion.
+
+![MQTT message conversion](https://github.com/toyokazu/fluent-plugin-mqtt-io/blob/master/images/mqtt_message_conversion.png "MQTT message conversion")
+
+```
+<source>
+  type mqtt
+  bind 192.168.1.100
+  port 1883
+  topic 'Libelium/+/+'
+  format xml
+  time_xpath '["cap:alert/cap:info/cap:onset", "text"]'
+  time_key '@timestamp'
+  attr_xpaths '[["cap:alert/cap:info/cap:parameter/cap:valueName", "text"]]'
+  value_xpaths '[["cap:alert/cap:info/cap:parameter/cap:value", "text"]]'
+  @label @MQTT_OUT
+</source>
+
+<label @MQTT_OUT>
+  <match **>
+    type mqtt
+    bind 192.168.1.100
+    port 1883
+    topic_rewrite_pattern '^([\w\/]+)$'
+    topic_rewrite_replacement '\1/rewritten'
+  </match>
+</label>
+
+```
+
+
+### Sensor data uploads from tiny computers, e.g. Raspberry Pi, Edison, etc
+
+MQTT output plugin can be used as the following. If you have tiny computers like Raspberry Pi equipped with sensors and their data are outputted as files, you can use fluent-plugin-mqtt-io for uploading those data.
+
+![Sensor data uploads from tiny computers](https://github.com/toyokazu/fluent-plugin-mqtt-io/blob/master/images/sensor_data_uploads_from_tiny_computers.png "Sensor data uploads from tiny computers")
 
 
 ## Contributing

data/fluent-plugin-mqtt-io.gemspec

@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
   spec.name          = "fluent-plugin-mqtt-io"
-  spec.version       = "0.0.4"
+  spec.version       = "0.0.5"
   spec.authors       = ["Toyokazu Akiyama"]
   spec.email         = ["toyokazu@gmail.com"]
 
@@ -13,13 +13,15 @@ Gem::Specification.new do |spec|
   spec.homepage      = "https://github.com/toyokazu/fluent-plugin-mqtt-io"
   spec.license       = "MIT"
 
-  spec.files         = `git ls-files`.split($/)
+  spec.files         = `git ls-files`.gsub(/images\/[\w\.]+\n/, "").split($/)
   spec.bindir        = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
 
+  spec.add_dependency 'fluentd', '>= 0.10.0'
+  spec.add_dependency "mqtt", "~> 0.3"
+
   spec.add_development_dependency "bundler", "~> 1.10"
   spec.add_development_dependency "rake", "~> 10.0"
-  spec.add_development_dependency "mqtt", "~> 0.3"
 end

data/lib/fluent/plugin/in_mqtt.rb

@@ -5,7 +5,7 @@ module Fluent
     config_param :port, :integer, :default => 1883
     config_param :bind, :string, :default => '127.0.0.1'
     config_param :topic, :string, :default => '#'
-    config_param :format, :string, :default => 'none'
+    config_param :format, :string, :default => 'json'
     config_param :username, :string, :default => nil
     config_param :password, :string, :default => nil
     config_param :ssl, :bool, :default => nil
@@ -61,6 +61,10 @@ module Fluent
       begin
         topic.gsub!("/","\.")
         @parser.parse(message) {|time, record|
+          if time.nil?
+            $log.debug "Since time_key field is nil, Fluent::Engine.now is used."
+            time = Fluent::Engine.now
+          end
           $log.debug "#{topic}, #{time}, #{record}"
           router.emit(topic, time, record)
         }

metadata

@@ -1,57 +1,71 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-mqtt-io
 version: !ruby/object:Gem::Version
-  version: 0.0.4
+  version: 0.0.5
 platform: ruby
 authors:
 - Toyokazu Akiyama
 autorequire: 
 bindir: []
 cert_chain: []
-date: 2015-11-06 00:00:00.000000000 Z
+date: 2015-12-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: bundler
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.10.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.10.0
+- !ruby/object:Gem::Dependency
+  name: mqtt
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.10'
-  type: :development
+        version: '0.3'
+  type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.10'
+        version: '0.3'
 - !ruby/object:Gem::Dependency
-  name: rake
+  name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '1.10'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '1.10'
 - !ruby/object:Gem::Dependency
-  name: mqtt
+  name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.3'
+        version: '10.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.3'
+        version: '10.0'
 description: fluentd input/output plugin for mqtt broker
 email:
 - toyokazu@gmail.com
@@ -98,3 +112,4 @@ specification_version: 4
 summary: fluentd input/output plugin for mqtt broker
 test_files:
 - test/helper.rb
+has_rdoc: