fluent-plugin-azurestorage 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 173125873d3f72fb76b104b180df63b23472c02d
+  data.tar.gz: 4895ded3228acfb7c5e3e6458f9cc081c0ce69da
+SHA512:
+  metadata.gz: 87d07b8903865caccf0e90833cbbc5edf9fb4ac81bcf453e526d9b3555a0c92af0aff6abee645abdab4789ecdd98ede4cfcaddc0c53e1b0eadb6c905f71585e8
+  data.tar.gz: 12b97072aa1633a683b7061212f6ac7a7c3c30c5374f9068edac973ab4972cbb086f4397c58e35a95d2dbf26ff20febd883f8ec0d91d8e181505c1a296fb17c7

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fluent-plugin-azurestorage.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Hidemasa Togashi
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,208 @@
+# Azure Storage output plugin for Fluentd
+
+## Overview
+
+Azure Storate output plugin buffers logs in local file and upload them to Azure Storage periodically.
+This plugin is porting from [fluent-plugin-s3](https://github.com/fluent/fluent-plugin-s3) for AzureStorage.
+
+## Installation
+
+Install from RubyGems:
+```
+$ gem install fluent-plugin-azurestorage
+```
+
+## Configuration
+
+```
+<match pattern>
+  type azurestorage
+
+  azure_storage_account    <your azure storage account>
+  azure_storage_access_key <your azure storage access key>
+  azure_container          <your azure storage container>
+  azure_storage_type       blob
+  store_as                 gzip 
+  auto_create_container    true
+  path                     logs/
+  azure_object_key_format  %{path}%{time_slice}_%{index}.%{file_extension}
+  buffer_path              /var/log/fluent/azurestorage
+
+  time_slice_format        %Y%m%d-%H
+  time_slice_wait          10m
+  utc
+</match>
+```
+
+### azure_storage_account
+
+Your Azure Storage Account Name. This can be got from Azure Management potal.
+This parameter is required when environment variable 'AZURE_STORAGE_ACCOUNT' is not set.
+
+### azure_storage_access_key
+
+Your Azure Storage Access Key(Primary or Secondary). This also can be got from Azure Management potal.
+This parameter is required when environment variable 'AZURE_STORAGE_ACCESS_KEY' is not set.
+
+### azure_container (Required)
+
+Azure Storage Container name
+
+### auto_create_container
+
+This plugin create container if not exist when you set 'auto_create_container' to true.
+
+### azure_storage_type
+
+Azure Storage type. Now supports only 'blob'(default). 'tables' and 'queues' are not implemented.
+
+### azure_object_key_format
+
+The format of Azure Storage object keys. You can use several built-in variables:
+
+- %{path}
+- %{time_slice}
+- %{index}
+- %{file_extension}
+
+to decide keys dynamically.
+
+%{path} is exactly the value of *path* configured in the configuration file. E.g., "logs/" in the example configuration above.
+%{time_slice} is the time-slice in text that are formatted with *time_slice_format*.
+%{index} is the sequential number starts from 0, increments when multiple files are uploaded to Azure Storage in the same time slice.
+%{file_extention} is always "gz" for now.
+
+The default format is "%{path}%{time_slice}_%{index}.%{file_extension}".
+
+For instance, using the example configuration above, actual object keys on Azure Storage will be something like:
+
+```
+"logs/20130111-22_0.gz"
+"logs/20130111-23_0.gz"
+"logs/20130111-23_1.gz"
+"logs/20130112-00_0.gz"
+```
+
+With the configuration:
+
+```
+azure_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}.%{file_extension}
+path log
+time_slice_format %Y%m%d-%H
+```
+
+You get:
+
+```
+"log/events/ts=20130111-22/events_0.gz"
+"log/events/ts=20130111-23/events_0.gz"
+"log/events/ts=20130111-23/events_1.gz"
+"log/events/ts=20130112-00/events_0.gz"
+```
+
+The [fluent-mixin-config-placeholders](https://github.com/tagomoris/fluent-mixin-config-placeholders) mixin is also incorporated, so additional variables such as %{hostname}, %{uuid}, etc. can be used in the azure_object_key_format. This could prove useful in preventing filename conflicts when writing from multiple servers.
+
+```
+azure_object_key_format %{path}/events/ts=%{time_slice}/events_%{index}-%{hostname}.%{file_extension}
+```
+
+### store_as
+
+Archive format on Azure Storage. You can use following types:
+
+- gzip (default)
+- json
+- text
+- lzo (Need lzop command)
+- lzma2 (Need xz command)
+
+### format
+
+Change one line format in the Azure Storage object. Supported formats are 'out_file', 'json', 'ltsv' and 'single_value'.
+
+- out_file (default)
+
+```
+time\ttag\t{..json1..}
+time\ttag\t{..json2..}
+...
+```
+
+- json
+
+```
+{..json1..}
+{..json2..}
+...
+```
+
+At this format, "time" and "tag" are omitted.
+But you can set these information to the record by setting "include_tag_key" / "tag_key" and "include_time_key" / "time_key" option.
+If you set following configuration in AzureStorage output:
+
+```
+format json
+include_time_key true
+time_key log_time # default is time
+```
+
+then the record has log_time field.
+
+```
+{"log_time":"time string",...}
+```
+
+- ltsv
+
+```
+key1:value1\tkey2:value2
+key1:value1\tkey2:value2
+...
+```
+
+"ltsv" format also accepts "include_xxx" related options. See "json" section.
+
+- single_value
+
+Use specified value instead of entire recode. If you get '{"message":"my log"}', then contents are
+
+```
+my log1
+my log2
+...
+```
+
+You can change key name by "message_key" option.
+
+### path
+
+path prefix of the files on Azure Storage. Default is "" (no prefix).
+
+### buffer_path (required)
+
+path prefix of the files to buffer logs.
+
+### time_slice_format
+
+Format of the time used as the file name. Default is '%Y%m%d'. Use '%Y%m%d%H' to split files hourly.
+
+### time_slice_wait
+
+The time to wait old logs. Default is 10 minutes. Specify larger value if old logs may reache.
+
+### utc
+
+Use UTC instead of local time.
+
+## License
+Azure Storage output plugin is licensed according to the terms of the Apache License, Version 2.0.
+
+The full version of this lisence can be found at http://www.apache.org/licenses/LICENSE-2.0 [TXT](http://www.apache.org/licenses/LICENSE-2.0.txt) or [HTML](http://www.apache.org/licenses/LICENSE-2.0.html)
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/fluent-plugin-azurestorage.gemspec

@@ -0,0 +1,25 @@
+# coding: utf-8
+#lib = File.expand_path('../lib', __FILE__)
+#$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |spec|
+  spec.name          = "fluent-plugin-azurestorage"
+  spec.version       = "0.0.1"
+  spec.authors       = ["Hidemasa Togashi"]
+  spec.email         = ["togachiro@gmail.com"]
+  spec.description   = %q{Fluent plugin for store to Azure Storage}
+  spec.summary       = %q{Fluent plugin for store to Azure Storage}
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+
+  spec.add_runtime_dependency "fluentd"
+  spec.add_runtime_dependency "azure"
+end

data/lib/fluent/plugin/out_azurestorage.rb

@@ -0,0 +1,162 @@
+module Fluent
+  require 'fluent/mixin/config_placeholders'
+
+  class AzureStorageOutput < Fluent::TimeSlicedOutput
+    Fluent::Plugin.register_output('azurestorage', self)
+
+    def initialize
+      super
+      require 'azure'
+      require 'zlib'
+      require 'time'
+      require 'tempfile'
+      require 'open3'
+    end
+
+    config_param :path, :string, :default => ''
+    config_param :azure_storage_account, :string, :default => nil
+    config_param :azure_storage_access_key, :string, :default => nil
+    config_param :azure_container, :string, :default => nil
+    config_param :azure_storage_type, :string, :default => 'blob'
+    config_param :azure_object_key_format, :string, :default => "%{path}%{time_slice}_%{index}.%{file_extension}"
+    config_param :store_as, :string, :default => 'gzip'
+    config_param :auto_create_container, :bool, :default => true
+    config_param :format, :string, :default => 'out_file'
+    config_param :command_parameter, :string, :default => nil
+
+    include Fluent::Mixin::ConfigPlaceholders
+
+    def configure(conf)
+      super
+
+      @ext = case @store_as
+             when 'gzip'
+               'gz'
+             when 'lzo'
+               check_command('lzop', 'LZO')
+               @command_parameter = '-qf1' if @command_parameter.nil?
+               'lzo'
+             when 'lzma2'
+               check_command('xz', 'LZMA2')
+               @command_paramter = '-qf0' if @command_parameter.nil?
+               'xz'
+             when 'json'
+               'json'
+             else
+               'txt'
+             end
+
+      @storage_type = case @azure_storage_type
+                      when 'tables'
+                        raise NotImplementedError
+                      when 'queues'
+                        raise NotImplementedError
+                      else
+                        'blob'
+                      end
+
+      conf['format'] = @format
+      @formatter = TextFormatter.create(conf)
+
+      if @localtime
+        @path_slicer = Proc.new {|path|
+          Time.now.strftime(path)
+        }
+      else
+        @path_slicer = Proc.new {|path|
+          Time.now.utc.strftime(path)
+        }
+      end
+
+      if @azure_container.nil?
+        raise ConfigError, 'azure_container is needed'
+      end
+    end
+
+    def start
+      super
+
+      if (!@azure_storage_account.nil? && !@azure_storage_access_key)
+        Azure.configure do |config|
+          config.storage_account_name = @azure_storage_account
+          config.storage_access_key   = @azure_storage_access_key
+        end
+      end
+      @bs = Azure::BlobService.new
+
+      ensure_container
+    end
+
+    def format(tag, time, record)
+      @formatter.format(tag, time, record)
+    end
+
+    def write(chunk)
+      blob_list = @bs.list_blobs(@azure_container)
+      i = 0
+      begin
+        path = @path_slicer.call(@path)
+        values_for_object_key = {
+          'path' => path,
+          'time_slice' => chunk.key,
+          'file_extension' => @ext,
+          'index' => i
+        }
+        storage_path = @azure_object_key_format.gsub(%r(%{[^}]+})) { |expr|
+          values_for_object_key[expr[2...expr.size-1]]
+        }
+        i += 1
+      end while blob_list.find { |elem| elem.name == storage_path }
+ 
+      tmp = Tempfile.new("azure-")
+      begin
+        case @store_as
+        when 'gzip'
+          w = Zlib::GzipWriter.new(tmp)
+          chunk.write_to(w)
+          w.close
+        when 'lzo'
+          w = Tempfile.new('chunk-tmp')
+          chunk.write_to(w)
+          w.close
+          tmp.close
+          system "lzop #{@command_parameter} -o #{tmp.path} #{w.path}"
+        when 'lzma2'
+          w = Tempfile.new('chunk-xz-tmp')
+          chunk.write_to(w)
+          w.close
+          tmp.close
+          system "xz #{@command_parameter} -c #{w.path} > #{tmp.path}"
+        else
+          chunk.write_to(tmp)
+          tmp.close
+        end
+        content = File.open(tmp.path, 'rb') { |file| file.read }
+        @bs.create_block_blob(@azure_container, storage_path, content)
+      ensure
+        tmp.close(true) rescue nil
+        w.close rescue nil
+        w.unlink rescue nil
+      end
+    end
+
+    private
+    def ensure_container
+      if ! @bs.list_containers.find { |c| c.name == @azure_container }
+        if @auto_create_container
+          @bs.create_container(@azure_container)
+        else
+          raise "The specified container does not exist: container = #{@azure_container}"
+        end
+      end
+    end
+
+    def check_command(command, algo)
+      begin
+        Open3.capture3("#{command} -V")
+      rescue Errno::ENOENT
+        raise ConfigError, "'#{command}' utility must be in PATH for #{algo} compression"
+      end
+    end
+  end
+end

metadata

@@ -0,0 +1,107 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-azurestorage
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Hidemasa Togashi
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-08-03 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: azure
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Fluent plugin for store to Azure Storage
+email:
+- togachiro@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- fluent-plugin-azurestorage.gemspec
+- lib/fluent/plugin/out_azurestorage.rb
+homepage: ''
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.14
+signing_key: 
+specification_version: 4
+summary: Fluent plugin for store to Azure Storage
+test_files: []