logstash-input-qingstor 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4a30583b85c2e846709cecde14a016f28f1907b3
+  data.tar.gz: 7834fd79bf3ad2c7de3e971faf42e347fe38f769
+SHA512:
+  metadata.gz: e63c11974c60a1fa33525c5b758ebbe7d80202d0e534605060bdbc7833df608d87db40e92c4cc4775be37ae78373df264ed6d96b226feab083bee8a658ec35f5
+  data.tar.gz: 0723a8e49fd65c445d5546030f8fc4130794cc96148ebeebd8e517b9280f00f2db732416a90eff7d8dbe3b7d499f2a9e166978bace5b2a624075cb51f4fbad0a

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.0
+  - Basic functions: Fetch file(.log/.gz), delete atfer processing, backup to remote bucket/local dir.

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,44 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at pengsrc@yunify.com. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][http://contributor-covenant.org], version 1.4, available at [http://contributor-covenant.org/version/1/4][http://contributor-covenant.org/version/1/4].

data/CONTRIBUTING.md

@@ -0,0 +1,43 @@
+# Contributing Guidelines
+
+We love contributions from everyone. By participating in this project, you agree to abide by the [_`Contributor Covenant Code of Conduct`_](./CODE_OF_CONDUCT.md) and the conventions described in this documentation.
+
+## Getting Started
+
+### Reporting Issues
+
+A great way to contribute to the project is to send a detailed report when you encounter an issue. We always appreciate a well-written, thorough bug report, and will thank you for it!
+
+Check that [our issues list](https://github.com/Tacinight/logstash-output-qingstor/issues) doesn't already include that problem or suggestion before submitting an issue. If you find a match, you can use the "subscribe" button to get notified on updates. Do not leave random "+1" or "I have this too" comments, as they only clutter the discussion, and don't help resolving it. However, if you have ways to reproduce the issue or have additional information that may help resolving the issue, please leave a comment.
+
+When reporting issues, please describe what happened clearly, and include the steps required to reproduce the problem if possible and applicable. This information will help us review and fix your issue faster. When sending lengthy log-files, consider posting them as a gist (https://gist.github.com). Don't forget to remove sensitive data from your log files before posting (you can replace those parts with "CONFIDENTIAL").
+
+### Sending Pull Request
+
+Pull requests are always welcome, we will appreciate it.
+
+Any significant improvement should be documented as a GitHub [issue](https://github.com/Tacinight/logstash-output-qingstor/issues) before anybody starts working on it.
+
+1. Fork it ( https://github.com/Tacinight/logstash-output-qingstor/fork )
+2. Create your feature branch (`git checkout -b new-feature`)
+3. Commit your changes (`git commit -asm 'Add some feature'`)
+4. Push to the branch (`git push origin new-feature`)
+5. Create a new Pull Request
+
+## Conventions
+
+### Patch Content
+
+* Always remove the trailing whitespace
+* Keep a newline at the end of text files
+* Exclude irrelevant files, such as `*.a`, `.bundle`, and `.DS_Store`
+* Always pass all the tests before submitting
+* Let the propose of each patch simple and clear
+
+### Commit Message
+
+* Capitalize the subject line
+* Limit the subject line to 50 characters
+* Do not end the subject line with a period
+* Wrap the body at 72 characters
+* Use the body to explain what happened

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Evan Zhao - tacingiht@gmail.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-input-qingstor
+Example input plugin. This should help bootstrap your effort to write your own input plugin!

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+gemspec
+

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,168 @@
+# Logstash Input Plugin for Qingstor 
+
+这是一个适配了[Qingstor](https://www.qingcloud.com/products/storage#qingstor), 工作在logstash中的input插件.  Qingstor是[Qingcloud](https://www.qingcloud.com/)推出的对象存储服务.  
+作为一个input插件, 它能下载存储在Qingstor上的日志文件, 读入到logstash中进行进一步处理.  
+详细功能参考下面配置说明.  
+
+目前代码尚未提交至官方插件库, 需要手动安装, 安装方法参考下文.
+
+## 1. 配置说明
+
+#### 1.1 最小运行配置
+- 使用'-f' 接受一个*.conf文件或者使用'-e'参数最小运行配置时, 至少需要以下三项
+```sh
+input {
+    qingstor {
+        access_key_id => 'your_access_key_id'           #required 
+        secret_access_key => 'your_secret_access_key'   #required  
+        bucket => 'bucket_name'                         #required 
+        # region => "pek3a"                             #optional, default value "pek3a"                                
+    }
+}
+
+```
+
+#### 1.2 其他可选参数说明
+```sh
+input {
+    qingstor {
+        ......
+        # 指定下载文件的前缀. 
+        # 默认nil, 
+        prefix => 'aprefix'
+
+        # 本地保存临时文件的目录. 
+        # 默认: 系统临时文件目录下的qingstor2logstash文件夹, 例如linux下 "/tmp/qingstor2logstash".
+        tmpdir => '/local/temporary/directory' 
+
+        # 是否在处理之后, 删除远程bucket中的文件.
+        # 默认: false
+        delete_later => true
+
+        # 如果指定一个本地目录, 那么在处理完之后将文件备份至该位置.
+        # 默认:　nil 
+        local_dir => 'your/local/directory'
+
+        # 如果指定了该值, 那么在处理完之后将文件上传到Qingstor指定的bucket中.
+        # 默认: nil
+        backup_bucket => 'backupbucket'
+
+        # 配合上一项使用, 指定备份bucket所在的region.
+        # 默认: "pek3a", 可选枚举值: ["pek3a", "sh1a"]
+        backup_region => "sh1a"
+
+        # 备份文件的前缀
+        # 默认: nil 
+        backup_prefix => "logstash/backup"
+
+        # 指定一个sincedb的保存位置, sincedb用于记录上一次抓取文件的时间
+        # 没有指定时默认在用户HOME目录下创建
+        # 默认: nil
+        sincedb_path => "~/qingstor/.sincedb" 
+
+        # 每次抓取的时间间隔, 单位秒
+        # 默认: 10(s)
+        interval => 30
+                                       
+    }
+}
+
+```
+
+## 2. 安装插件
+
+#### 2.1 直接运行本地的插件
+
+- 编辑Logstash目录下的Genfile, 添加插件的路径, 例如
+```ruby
+gem "logstash-input-qingstor", :path => "/your/local/logstash-input-qingstor"
+```
+- 安装插件
+```sh
+bin/logstash-plugin install --no-verify
+```
+- 使用插件运行
+```sh
+bin/logstash -e "input {qingstor {  access_key_id => 'your_access_key_id'            
+        secret_access_key => 'your_secret_access_key'     
+        bucket => 'bucket_name'                          }}'
+```
+此时你对插件所做的任意的代码上的修改都会直接生效.
+
+#### 2.2 安装一个本地插件然后运行
+
+这一步你需要生成一个插件的gem包, 然后通过logstash来安装到logstash的插件目录下
+- 在logstash-input-qingstor项目目录下生成gem
+```sh
+gem build logstash-input-qingstor.gemspec
+```
+- 在Logstash的目录下使用logstash-plugin安装
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-input-qingstor.gem
+```
+- 安装完毕之后, 就可以使用Logstash运行开始测试了.
+
+
+# Logstash Output Plugin for Qingstor 
+
+As an input plugin, it can download the log files from [Qingstor](https://www.qingcloud.com/products/storage#qingstor), and read them into logstash.  
+Qingstor is a remarkable object storage service provided by [Qingcloud](https://www.qingcloud.com/).
+
+For now, We haven't submitted this to official plugin repository. If you want to have a try, please refer to the following guide to install it manually.
+
+## 1. Configuration Guide
+
+#### 1.1 Run in minimal Configuration Items
+```sh
+input {
+    qingstor {
+        access_key_id => 'your_access_key_id'           #required 
+        secret_access_key => 'your_secret_access_key'   #required  
+        bucket => 'bucket_name'                         #required 
+        # region => "pek3a"                             #optional, default value "pek3a"                                
+    }
+}
+
+```
+
+More configuration details please refer to [code part](lib/logstash/inputs/qingstor.rb).
+
+## 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-input-qingstor", :path => "/your/local/logstash-input-qingstor"
+```
+- Install plugin
+```sh
+bin/logstash-plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e "input {qingstor { access_key_id => 'your_access_key_id'            
+        secret_access_key => 'your_secret_access_key'     
+        bucket => 'bucket_name'                          }}"
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-input-qingstor.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-input-qingstor.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+Please see [Contributing Guidelines](./CONTRIBUTING.md) of this project before submitting patches.
+
+## LICENSE
+The Apache License (Version 2.0, January 2004).

data/lib/logstash/inputs/qingstor.rb

@@ -0,0 +1,262 @@
+# encoding: utf-8
+require "logstash/inputs/base"
+require "logstash/namespace"
+require "stud/interval"
+require "qingstor/sdk"
+require "fileutils"
+require "tmpdir"
+
+class LogStash::Inputs::Qingstor < LogStash::Inputs::Base
+  require "logstash/inputs/qingstor/qingstor_validator"
+  config_name "qingstor"
+
+  default :codec, "plain"
+
+  # The key id to access your QingStor
+  config :access_key_id, :validate => :string, :required => true
+
+  # The key to access your QingStor
+  config :secret_access_key, :validate => :string, :required => true
+
+  # The name of the qingstor bucket
+  config :bucket, :validate => :string, :required => true
+
+  # The region of the QingStor bucket
+  config :region, :validate => ["pek3a", "sh1a"], :default => "pek3a"
+
+  # If specified, it would only download the files with the specified prefix.
+  config :prefix, :validate => :string, :default => nil
+
+  # Set the directory where logstash store the tmp files before 
+  # sending it to logstash, default directory in linux /tmp/logstash2qingstor
+  config :tmpdir, :validate => :string, :default => File.join(Dir.tmpdir, "qingstor2logstash")
+
+  # If this set to true, the remote file will be deleted after processed
+  config :delete_later, :validate => :boolean, :default => false
+  
+  # If this set to true, the file will backup to a local dir,
+  # please make sure you can access to this dir.
+  config :local_dir, :validate => :string, :default => File.expand_path("~/")
+
+  # If specified, the file will be upload to this bucket of the given region
+  config :backup_bucket, :validate => :string, :default => nil
+
+  # Specified the backup region in Qingstor.
+  config :backup_region, :validate => ["pek3a", "sh1a"], :default => "pek3a"
+
+  # This prefix will add before backup filename.
+  config :backup_prefix, :validate => :string, :default => nil
+  
+  # Use sincedb to record the last download time 
+  config :sincedb_path, :validate => :string, :default => nil 
+  
+  # Set how frequently messages should be sent.
+  # The default, `10`, means send a message every 10 seconds.
+  config :interval, :validate => :number, :default => 10
+
+  public
+  def register
+
+    if !@tmpdir.nil? && !directory_valid?(@tmpdir)
+      raise LogStash::ConfigurationError, "Logstash must have the permissions to write to the temporary directory: #{@tmpdir}"
+    end
+
+    if !@local_dir.nil? && !directory_valid?(@local_dir)
+      raise LogStash::ConfigurationError, "Logstash must have the permissions to write to the temporary directory: #{@local_dir}"
+    end
+
+
+    @logger.info "Registering QingStor plugin", :bucket => @bucket, :region => @region
+    
+    @qs_config = QingStor::SDK::Config.init @access_key_id, @secret_access_key
+    @qs_service = QingStor::SDK::Service.new @qs_config
+    @qs_bucket = @qs_service.bucket @bucket, @region
+
+    QingstorValidator.bucket_valid?(@qs_bucket)
+    QingstorValidator.prefix_valid?(@backup_prefix)
+
+  end # def register
+
+  def run(queue)
+    @logger.info "starting processing"
+    @current_thread = Thread.current
+    Stud.interval(@interval) do
+      process_files(queue)
+    end
+  end # def run
+
+   
+  def process_files(queue)
+    objects = list_new_files
+
+    # For each record in objects, it will be download and read 
+    # into logstash. Then it will do deletion or backup job 
+    # according to preset flags.
+    objects.each do |key, time|
+      process_log(queue, key)
+      backup_to_bucket key unless @backup_bucket.nil?
+      backup_to_local_dir unless @local_dir.nil?
+      @qs_bucket.delete_object key if @delete_later
+    end
+
+    # if fetched nothing, just return
+    return if objects.empty?
+    _time = objects.values.max 
+    sincedb.write(Time.at(_time))
+  end
+
+  private 
+  def list_new_files
+    objects = {}
+    @logger.info "starting fetching objects"
+    res = @qs_bucket.list_objects({'prefix' => @prefix})
+    res[:keys].each do |log|
+      next unless is_desired?(log[:key])
+      next unless sincedb.newer?(log[:modified])
+      objects[log[:key]] = log[:modified]
+      @logger.info("QingStor input: Adding to object:", :key => log[:key])
+    end
+    return objects
+  end
+
+  def process_log(queue, key)
+    # a global var, for the next possible upload and copy job 
+    @tmp_file_path = File.join(@tmpdir, File.basename(key))
+
+    File.open(@tmp_file_path, 'wb') do |logfile|
+      logfile.write @qs_bucket.get_object(key)[:body]
+    end
+    process_local_log(queue, @tmp_file_path)
+  end
+ 
+  def backup_to_bucket(key)
+    properties = {'bucket-name' => @backup_bucket, 'zone' => @backup_region}
+    bucket = QingStor::SDK::Bucket.new @qs_config, properties
+    
+    if bucket.head[:status_code] != 200
+      res = bucket.put
+      if res[:status_code] != 201
+        @logger.error("ERROR : cannot create the bucket ", res[:message])
+        raise LogStash::ConfigurationError, "cannot create the bucket"
+      end 
+    end 
+    
+    md5_string = Digest::MD5.file(@tmp_file_path).to_s
+
+    new_key = if backup_prefix.end_with?('/')
+                backup_prefix + key 
+              else 
+                backup_prefix + '/' + key 
+              end 
+
+    bucket.put_object new_key, { 
+      'content_md5' => md5_string,
+      'body' => File.open(@tmp_file_path)
+    }
+  end 
+ 
+  def backup_to_local_dir
+    FileUtils.mkdir_p @local_dir unless File.exist? @local_dir
+    FileUtils.cp @tmp_file_path, @local_dir
+  end
+ 
+  def process_local_log(queue, filename)
+    read_file(filename) do |line|
+      @codec.decode(line) do |event|
+        decorate(event)
+        queue << event
+      end
+    end
+  end
+   
+  def read_file(filename, &block)
+    if gzip?(filename)
+      read_gzip_file(filename, block)
+    else
+      read_plain_file(filename, block)
+    end
+  end
+
+  def read_gzip_file(filename, block)
+    begin
+      Zlib::GzipReader.open(filename) do |decoder|
+        decoder.each_line { |line| block.call(line) }
+      end
+    rescue Zlib::Error, Zlib::GzipFile::Error => e 
+      @logger.error("Gzip codec: Cannot uncompress the file", :filename => filename)
+      raise e 
+    end 
+  end 
+   
+  def read_plain_file(filename, block)
+    File.open(filename, 'rb') do |file|
+      file.each(&block)
+    end
+  end
+
+  def is_desired?(filename)
+    return logger?(filename) || gzip?(filename)
+  end
+ 
+  def logger?(filename)
+    return filename.end_with?('.log')
+  end
+ 
+  def gzip?(filename)
+    return filename.end_with?('.gz')
+  end
+ 
+  def sincedb
+    @sincedb ||= if @sincedb_path.nil?
+                    @logger.info("Using default path for the sincedb", :filename => sincedb_file)
+                    SinceDB::File.new(sincedb_file)
+                 else
+                    @logger.info("Using the provided sincedb_path")
+                    SinceDB::File.new(@sincedb_path)
+                 end
+  end 
+ 
+  def sincedb_file
+    File.join(ENV["HOME"], ".sincedb_" + Digest::MD5.hexdigest("#{@bucket}+#{@prefix}"))
+  end
+
+  module SinceDB
+    class File
+      def initialize(file)
+        @sincedb_path = file
+      end
+
+      def newer?(date)
+        Time.at(date) > read 
+      end 
+
+      def read 
+        if ::File.exist?(@sincedb_path)
+          content = ::File.read(@sincedb_path).chomp.strip
+          return content.empty? ? Time.new(0) : Time.parse(content)
+        else 
+          return Time.new(0)
+        end
+      end 
+
+      def write(since = nil)
+        since = Time.now() if since.nil?
+        ::File.open(@sincedb_path, 'w') { |file| file.write(since.to_s) }
+      end 
+    end 
+  end  
+
+  public 
+  def stop
+   Stud.stop!(@current_thread)
+  end
+
+  def directory_valid?(path)
+    begin 
+      FileUtils.mkdir_p(path) unless Dir.exist?(path)
+      ::File.writable?(path)
+    rescue 
+      false 
+    end 
+  end 
+end # class LogStash::Inputs::Qingstor

data/lib/logstash/inputs/qingstor/qingstor_validator.rb

@@ -0,0 +1,30 @@
+# encoding: utf-8
+require "qingstor/sdk"
+require "fileutils"
+
+module LogStash
+  module Inputs
+    class Qingstor
+      class QingstorValidator
+
+        def bucket_valid?(bucket)
+            res = bucket.head
+            case res[:status_code] 
+            when 401
+              raise LogStash::ConfigurationError, "Incorrect key id or access key."
+            when 404
+              raise LogStash::ConfigurationError, "Incorrect bucket/region name."
+            end 
+            true
+        end 
+
+        def prefix_valid?(prefix)
+          if prefix.start_with?("/") || prefix.length >= 1024
+            raise LogStash::ConfigurationError, "Prefix must not start with '/' with length less than 1024 "
+          end 
+          true
+        end 
+      end 
+    end 
+  end 
+end 

data/logstash-input-qingstor.gemspec

@@ -0,0 +1,27 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-input-qingstor'
+  s.version       = '0.1.0'
+  s.licenses      = ['Apache License (2.0)']
+  s.summary       = 'logstash input plugin for QingStor'
+  s.description   = 'Use this plugin to fetch file as a input of logstash from QingStor'
+  s.homepage      = 'https://github.com/Tacinight/logstash-input-qingstor'
+  s.authors       = ['Evan Zhao']
+  s.email         = 'tacingiht@gmail.com'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", ">=1.6", "<=2.99"
+  s.add_runtime_dependency 'logstash-codec-plain'
+  s.add_runtime_dependency 'stud', '>= 0.0.22'
+  s.add_runtime_dependency "qingstor-sdk", "=1.9.2"
+
+  s.add_development_dependency 'logstash-devutils'
+end

data/spec/fixtures/logstash.log

@@ -0,0 +1,100 @@
+83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] "GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js HTTP/1.1" 200 2892 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/sad-medic.png HTTP/1.1" 200 430406 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] "GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Bold.ttf HTTP/1.1" 200 38720 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] "GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Regular.ttf HTTP/1.1" 200 41820 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] "GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1" 200 52878 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:43 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard.png HTTP/1.1" 200 321631 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/Dreamhost_logo.svg HTTP/1.1" 200 2126 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:43 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard2.png HTTP/1.1" 200 394967 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/apache-icon.gif HTTP/1.1" 200 8095 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/nagios-sms5.png HTTP/1.1" 200 78075 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/redis.png HTTP/1.1" 200 25230 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:47 +0000] "GET /presentations/logstash-monitorama-2013/images/elasticsearch.png HTTP/1.1" 200 8026 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:47 +0000] "GET /presentations/logstash-monitorama-2013/images/logstashbook.png HTTP/1.1" 200 54662 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:47 +0000] "GET /presentations/logstash-monitorama-2013/images/github-contributions.png HTTP/1.1" 200 34245 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:47 +0000] "GET /presentations/logstash-monitorama-2013/css/print/paper.css HTTP/1.1" 200 4254 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:47 +0000] "GET /presentations/logstash-monitorama-2013/images/1983_delorean_dmc-12-pic-38289.jpeg HTTP/1.1" 200 220562 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/simple-inputs-filters-outputs.jpg HTTP/1.1" 200 1168622 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:46 +0000] "GET /presentations/logstash-monitorama-2013/images/tiered-outputs-to-inputs.jpg HTTP/1.1" 200 1079983 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+83.149.9.216 - - [04/Jan/2015:05:13:53 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
+24.236.252.67 - - [04/Jan/2015:05:14:10 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0"
+93.114.45.13 - - [04/Jan/2015:05:14:32 +0000] "GET /articles/dynamic-dns-with-dhcp/ HTTP/1.1" 200 18848 "http://www.google.ro/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CCwQFjAB&url=http%3A%2F%2Fwww.semicomplete.com%2Farticles%2Fdynamic-dns-with-dhcp%2F&ei=W88AU4n9HOq60QXbv4GwBg&usg=AFQjCNEF1X4Rs52UYQyLiySTQxa97ozM4g&bvm=bv.61535280,d.d2k" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [04/Jan/2015:05:14:32 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/articles/dynamic-dns-with-dhcp/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [04/Jan/2015:05:14:33 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/articles/dynamic-dns-with-dhcp/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [04/Jan/2015:05:14:33 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [04/Jan/2015:05:14:33 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/dynamic-dns-with-dhcp/" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+93.114.45.13 - - [04/Jan/2015:05:14:33 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0"
+66.249.73.135 - - [04/Jan/2015:05:15:03 +0000] "GET /blog/tags/ipv6 HTTP/1.1" 200 12251 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+50.16.19.13 - - [04/Jan/2015:05:15:15 +0000] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "http://www.semicomplete.com/blog/tags/puppet?flav=rss20" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)"
+66.249.73.185 - - [04/Jan/2015:05:15:23 +0000] "GET / HTTP/1.1" 200 37932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+110.136.166.128 - - [04/Jan/2015:05:16:11 +0000] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&cad=rja&sqi=2&ved=0CFYQFjAE&url=http%3A%2F%2Fwww.semicomplete.com%2Fprojects%2Fxdotool%2F&ei=6cwAU_bRHo6urAeI0YD4Ag&usg=AFQjCNE3V_aCf3-gfNcbS924S6jZ6FqffA&bvm=bv.61535280,d.bmk" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"
+46.105.14.53 - - [04/Jan/2015:05:16:17 +0000] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/"
+110.136.166.128 - - [04/Jan/2015:05:16:22 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"
+110.136.166.128 - - [04/Jan/2015:05:16:22 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"
+110.136.166.128 - - [04/Jan/2015:05:16:22 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"
+110.136.166.128 - - [04/Jan/2015:05:16:22 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"
+123.125.71.35 - - [04/Jan/2015:05:16:31 +0000] "GET /blog/tags/release HTTP/1.1" 200 40693 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
+110.136.166.128 - - [04/Jan/2015:05:16:22 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0"
+50.150.204.184 - - [04/Jan/2015:05:17:06 +0000] "GET /images/googledotcom.png HTTP/1.1" 200 65748 "http://www.google.com/search?q=https//:google.com&source=lnms&tbm=isch&sa=X&ei=4-r8UvDrKZOgkQe7x4CICw&ved=0CAkQ_AUoAA&biw=320&bih=441" "Mozilla/5.0 (Linux; U; Android 4.0.4; en-us; LG-MS770 Build/IMM76I) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30"
+207.241.237.225 - - [04/Jan/2015:05:17:35 +0000] "GET /blog/tags/examples HTTP/1.0" 200 9208 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
+200.49.190.101 - - [04/Jan/2015:05:17:39 +0000] "GET /reset.css HTTP/1.1" 200 1015 "-" "-"
+200.49.190.100 - - [04/Jan/2015:05:17:37 +0000] "GET /blog/tags/web HTTP/1.1" 200 44019 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1"
+200.49.190.101 - - [04/Jan/2015:05:17:41 +0000] "GET /style2.css HTTP/1.1" 200 4877 "-" "-"
+200.49.190.101 - - [04/Jan/2015:05:17:48 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "-" "QS304 Profile/MIDP-2.0 Configuration/CLDC-1.1"
+66.249.73.185 - - [04/Jan/2015:05:18:48 +0000] "GET /reset.css HTTP/1.1" 200 1015 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+66.249.73.135 - - [04/Jan/2015:05:18:55 +0000] "GET /blog/tags/munin HTTP/1.1" 200 9746 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+66.249.73.135 - - [04/Jan/2015:05:19:16 +0000] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+66.249.73.135 - - [04/Jan/2015:05:20:03 +0000] "GET /blog/geekery/eventdb-ideas.html HTTP/1.1" 200 11418 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+67.214.178.190 - - [04/Jan/2015:05:20:48 +0000] "GET / HTTP/1.0" 200 37932 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0"
+67.214.178.190 - - [04/Jan/2015:05:20:59 +0000] "GET /blog/geekery/installing-windows-8-consumer-preview.html HTTP/1.0" 200 8948 "http://www.semicomplete.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:21.0) Gecko/20100101 Firefox/21.0"
+207.241.237.220 - - [04/Jan/2015:05:21:16 +0000] "GET /blog/tags/projects HTTP/1.0" 200 28370 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
+46.105.14.53 - - [04/Jan/2015:05:21:17 +0000] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/"
+207.241.237.227 - - [04/Jan/2015:05:21:19 +0000] "GET /blog/geekery/soekris-gpio.html HTTP/1.0" 200 9587 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
+91.177.205.119 - - [04/Jan/2015:05:22:03 +0000] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "http://en.wikipedia.org/wiki/Xvfb" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)"
+91.177.205.119 - - [04/Jan/2015:05:22:03 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)"
+91.177.205.119 - - [04/Jan/2015:05:22:03 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)"
+91.177.205.119 - - [04/Jan/2015:05:22:03 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)"
+91.177.205.119 - - [04/Jan/2015:05:22:03 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://semicomplete.com/blog/geekery/xvfb-firefox.html" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)"
+91.177.205.119 - - [04/Jan/2015:05:22:04 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Win64; x64; Trident/6.0)"
+66.249.73.185 - - [04/Jan/2015:05:22:13 +0000] "GET /doc/index.html?org/elasticsearch/action/search/SearchResponse.html HTTP/1.1" 404 294 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+207.241.237.228 - - [04/Jan/2015:05:22:23 +0000] "GET /blog/tags/defcon HTTP/1.0" 200 24142 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
+207.241.237.101 - - [04/Jan/2015:05:22:27 +0000] "GET /blog/tags/regex HTTP/1.0" 200 14888 "http://www.semicomplete.com/blog/tags/C" "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)"
+87.169.99.232 - - [04/Jan/2015:05:23:41 +0000] "GET /presentations/puppet-at-loggly/puppet-at-loggly.pdf.html HTTP/1.1" 200 24747 "https://www.google.de/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
+209.85.238.199 - - [04/Jan/2015:05:23:47 +0000] "GET /blog/tags/firefox?flav=rss20 HTTP/1.1" 200 16021 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 3 subscribers; feed-id=14171215010336145331)"
+209.85.238.199 - - [04/Jan/2015:05:24:00 +0000] "GET /test.xml HTTP/1.1" 200 1370 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=11390274670024826467)"
+81.220.24.207 - - [04/Jan/2015:05:24:57 +0000] "GET /blog/geekery/ssl-latency.html HTTP/1.1" 200 17147 "http://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CE4QFjAE&url=http%3A%2F%2Fwww.semicomplete.com%2Fblog%2Fgeekery%2Fssl-latency.html&ei=ZdEAU9mGGuWX1AW09IDoBw&usg=AFQjCNHw6zioJpizqX8Q0YpKKaF4zdCSEg&bvm=bv.61535280,d.d2k" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11"
+81.220.24.207 - - [04/Jan/2015:05:24:57 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11"
+81.220.24.207 - - [04/Jan/2015:05:24:57 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11"
+81.220.24.207 - - [04/Jan/2015:05:24:57 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11"
+81.220.24.207 - - [04/Jan/2015:05:24:57 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11"
+81.220.24.207 - - [04/Jan/2015:05:24:58 +0000] "GET /favicon.ico HTTP/1.1" 200 3638 "http://www.semicomplete.com/blog/geekery/ssl-latency.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.73.11 (KHTML, like Gecko) Version/7.0.1 Safari/537.73.11"
+66.249.73.135 - - [04/Jan/2015:05:25:05 +0000] "GET /blog/geekery/vmware-cpu-performance.html HTTP/1.1" 200 12908 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+46.105.14.53 - - [04/Jan/2015:05:26:17 +0000] "GET /blog/tags/puppet?flav=rss20 HTTP/1.1" 200 14872 "-" "UniversalFeedParser/4.2-pre-314-svn +http://feedparser.org/"
+218.30.103.62 - - [04/Jan/2015:05:27:05 +0000] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+218.30.103.62 - - [04/Jan/2015:05:27:10 +0000] "GET /robots.txt HTTP/1.1" 200 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+218.30.103.62 - - [04/Jan/2015:05:27:15 +0000] "GET /projects/fex/ HTTP/1.1" 200 14352 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+74.125.40.20 - - [04/Jan/2015:05:27:22 +0000] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "FeedBurner/1.0 (http://www.FeedBurner.com)"
+71.212.224.97 - - [04/Jan/2015:05:27:34 +0000] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://suckless.org/rocks" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
+71.212.224.97 - - [04/Jan/2015:05:27:34 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
+71.212.224.97 - - [04/Jan/2015:05:27:35 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
+71.212.224.97 - - [04/Jan/2015:05:27:35 +0000] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
+71.212.224.97 - - [04/Jan/2015:05:27:35 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
+218.30.103.62 - - [04/Jan/2015:05:27:36 +0000] "GET /projects/xdotool/xdotool.xhtml HTTP/1.1" 304 - "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+108.174.55.234 - - [04/Jan/2015:05:27:45 +0000] "GET /?flav=rss20 HTTP/1.1" 200 29941 "-" "-"
+218.30.103.62 - - [04/Jan/2015:05:27:57 +0000] "GET /blog/geekery/c-vs-python-bdb.html HTTP/1.1" 200 11388 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+121.107.188.202 - - [04/Jan/2015:05:27:57 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36"
+218.30.103.62 - - [04/Jan/2015:05:28:21 +0000] "GET /blog/productivity/better-zsh-xterm-title-fix.html HTTP/1.1" 200 10185 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+218.30.103.62 - - [04/Jan/2015:05:28:43 +0000] "GET /blog/geekery/xvfb-firefox.html HTTP/1.1" 200 10975 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+218.30.103.62 - - [04/Jan/2015:05:29:06 +0000] "GET /blog/geekery/puppet-facts-into-mcollective.html HTTP/1.1" 200 9872 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+198.46.149.143 - - [04/Jan/2015:05:29:13 +0000] "GET /blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 9316 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)"
+198.46.149.143 - - [04/Jan/2015:05:29:13 +0000] "GET /blog/geekery/solving-good-or-bad-problems.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 10756 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)"
+218.30.103.62 - - [04/Jan/2015:05:29:26 +0000] "GET /blog/geekery/jquery-interface-puffer.html%20target= HTTP/1.1" 200 202 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+218.30.103.62 - - [04/Jan/2015:05:29:48 +0000] "GET /blog/geekery/ec2-reserved-vs-ondemand.html HTTP/1.1" 200 11834 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
+66.249.73.135 - - [04/Jan/2015:05:30:06 +0000] "GET /blog/web/firefox-scrolling-fix.html HTTP/1.1" 200 8956 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
+86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /projects/xdotool/ HTTP/1.1" 200 12292 "http://www.haskell.org/haskellwiki/Xmonad/Frequently_asked_questions" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"
+86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"
+86.1.76.62 - - [04/Jan/2015:05:30:37 +0000] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/projects/xdotool/" "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205 Firefox/24.0 Iceweasel/24.3.0"

data/spec/inputs/qingstor_spec.rb

@@ -0,0 +1,58 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/inputs/qingstor"
+require_relative "./qs_access_helper"
+require "tmpdir"
+
+describe LogStash::Inputs::Qingstor do
+
+  before do 
+    Thread.abort_on_exception = true
+
+    upload_file("../fixtures/logstash.log", "log3.log")
+    upload_file("../fixtures/logstash.log.gz", "log3.log.gz")
+  end
+
+  after do 
+    delete_remote_file "log3.log"    
+    delete_remote_file "log3.log.gz"
+  end
+
+  let(:config) { { 
+      "access_key_id" => ENV['access_key_id'],
+      "secret_access_key" => ENV['secret_access_key'],
+      "bucket" => ENV['bucket'],
+      "region" => ENV['region']
+  } }
+  let(:key1) { "log3.log" }
+  let(:key2) { "log3.log.gz" }
+  let(:backup) { "logstash-backup" }
+  let(:local_backup_dir) { File.join(Dir.tmpdir, backup) }
+
+  context "local backup" do 
+    it "backup to local dir" do 
+      fetch_events(config.merge({"local_dir" => local_backup_dir }))
+      expect(File.exists?(File.join(local_backup_dir, key1))).to be_truthy
+      expect(File.exists?(File.join(local_backup_dir, key2))).to be_truthy
+    end 
+    
+    after do 
+      FileUtils.rm_r(File.join(local_backup_dir, key1))
+      FileUtils.rm_r(File.join(local_backup_dir, key2))
+    end 
+  end 
+
+  context "remote backup" do 
+    it "backup to another bucket" do 
+      fetch_events(config.merge({"backup_bucket" => backup}))
+      expect(list_remote_file(backup).size).to eq(2)
+    end 
+
+    after do 
+      delete_bucket(backup)
+    end 
+  end 
+
+  context ""
+  
+end

data/spec/inputs/qingstor_spec_validator_spec.rb

@@ -0,0 +1,37 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/inputs/qingstor"
+
+describe LogStash::Inputs::Qingstor do
+  let(:config) { { 
+      "access_key_id" => ENV['access_key_id'],
+      "secret_access_key" => ENV['secret_access_key'],
+      "bucket" => ENV['bucket'],
+      "region" => ENV['region']
+  } }
+  
+  it "raise error if it has incorrect key id or access key" do
+    config["access_key_id"] = "wrongid"
+    expect{ described_class.new(config).register }.to raise_error(LogStash::ConfigurationError)
+  end
+
+  it "raise error if it has incorrect key id or access key" do
+    config["secret_access_key"] = "wrongaccesskey"
+    expect{ described_class.new(config).register }.to raise_error(LogStash::ConfigurationError)
+  end
+
+  it "raise error if it has incorrect bucket/region name" do
+    config["bucket"] = "wrongbucket"
+    expect{ described_class.new(config).register }.to raise_error(LogStash::ConfigurationError)
+  end
+
+  it "raise error if it has incorrect bucket/region name" do
+    config["region"] = "wrongregion"
+    expect{ described_class.new(config).register }.to raise_error(LogStash::ConfigurationError)
+  end
+
+  it "use default region if it is not set" do 
+    config.delete("region")
+    expect(described_class.new(config).register ).to be_truthy
+  end
+end 

data/spec/inputs/qs_access_helper.rb

@@ -0,0 +1,43 @@
+def fetch_events(settings)
+  queue = []
+  qs = LogStash::Inputs::Qingstor.new(settings)
+  qs.register
+  qs.process_files(queue)
+  queue
+end  
+
+def qs_init_config(access_key_id = ENV['access_key_id'],
+                   secret_access_key = ENV['secret_access_key'])
+  return QingStor::SDK::Config.init access_key_id, secret_access_key
+end 
+
+def qs_init_bucket(bucket = ENV['bucket'], region = ENV['region'])
+  config = qs_init_config
+  properties = {'bucket-name' => bucket, 'zone' => region }
+  return QingStor::SDK::Bucket.new(config, properties)
+end 
+
+def upload_file(local_file, remote_file)
+  bucket = qs_init_bucket
+  file = File.expand_path(File.join(File.dirname(__FILE__), local_file))
+  md5_string = Digest::MD5.file(file).to_s
+  bucket.put_object remote_file, { 
+      'content_md5' => md5_string,
+      'body' => File.open(file)
+  }
+end 
+
+def delete_remote_file(key)
+  bucket = qs_init_bucket
+  bucket.delete_object key
+end
+
+def list_remote_file(bucket = ENV['bucket'], region = ENV['region'])
+  bucket = qs_init_bucket(bucket, region)
+  return bucket.list_objects[:keys]
+end
+
+def delete_bucket(bucket)
+  bucket = qs_init_bucket(bucket)
+  bucket.delete
+end 

metadata

@@ -0,0 +1,142 @@
+--- !ruby/object:Gem::Specification
+name: logstash-input-qingstor
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Evan Zhao
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2017-03-07 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.6'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.6'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+  name: stud
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.9.2
+  name: qingstor-sdk
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 1.9.2
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Use this plugin to fetch file as a input of logstash from QingStor
+email: tacingiht@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CODE_OF_CONDUCT.md
+- CONTRIBUTING.md
+- CONTRIBUTORS
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/inputs/qingstor.rb
+- lib/logstash/inputs/qingstor/qingstor_validator.rb
+- logstash-input-qingstor.gemspec
+- spec/fixtures/logstash.log
+- spec/fixtures/logstash.log.gz
+- spec/inputs/qingstor_spec.rb
+- spec/inputs/qingstor_spec_validator_spec.rb
+- spec/inputs/qs_access_helper.rb
+homepage: https://github.com/Tacinight/logstash-input-qingstor
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
+specification_version: 4
+summary: logstash input plugin for QingStor
+test_files:
+- spec/fixtures/logstash.log
+- spec/fixtures/logstash.log.gz
+- spec/inputs/qingstor_spec.rb
+- spec/inputs/qingstor_spec_validator_spec.rb
+- spec/inputs/qs_access_helper.rb