Nordes-logstash-input-azureblob 0.9.5.pre.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: c95d8dd858297e06d69f2366d28e2846b2c35fb4
+  data.tar.gz: 1a1921f5e3db235698d6c0387d4b615008ccc5c0
+SHA512:
+  metadata.gz: 1684e18df37bcf098189d61265a008d21d20bc3d768956347563529e2633b71744578142a8b2a27df072ac33ce13c5b9a0fd749e51021a0f79d54ba7caf263ad
+  data.tar.gz: 05f3b519d14dd26fb8614f1c00af111f492bf05d5ad4bdffdcb0f6a3627ad4942d0ca5150a57cb1e5f49608f213f485bf1c40a519458253673d0b5cb991a0b99

data/CHANGELOG.md

@@ -0,0 +1,16 @@
+## 2016.07.01
+* Updated the *README.md*
+* Implemented *sleep_time*
+* Added *sincedb* parameter (Use Azure table)
+* Added *ignore_older* parameter (work as file plugin)
+* Added *start_position* parameter (work as file plugin)
+* Added *path_prefix* parameter (no wildcard accepted, using Azure blob API)
+* Added some logs for debugging
+
+### Not yet completed
+* Removed the milestone (deprecated)
+
+## 2016.05.05
+* Made the plugin to respect Logstash shutdown signal.
+* Updated the *logstash-core* runtime dependency requirement to '~> 2.0'.
+* Updated the *logstash-devutils* development dependency requirement to '>= 0.0.16'

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,17 @@
+
+Copyright (c) Microsoft.  All rights reserved.
+Microsoft would like to thank its contributors, a list
+of whom are at http://aka.ms/entlib-contributors
+
+Licensed under the Apache License, Version 2.0 (the "License"); you
+may not use this file except in compliance with the License. You may
+obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+implied. See the License for the specific language governing permissions
+and limitations under the License.
+

data/README.md

@@ -0,0 +1,146 @@
+# Logstash input plugin for Azure Storage Blobs
+
+## Summary
+This plugin reads and parses data from Azure Storage Blobs.
+
+## Installation
+You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:
+```sh
+logstash-plugin install logstash-input-azureblob
+```
+
+For more information, see Logstash reference [Working with plugins](https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html).
+
+## Configuration
+### Required Parameters
+__*storage_account_name*__
+
+The Azure storage account name.
+
+__*storage_access_key*__
+
+The access key to the storage account.
+
+__*container*__
+
+The blob container name.
+
+### Optional Parameters
+__*codec*__
+
+The codec used to decode the blob. By default *json_lines* is selected. For normal log file, use *line* or other existing codec.
+
+* **Default value:** *json_lines*
+
+__*sleep_time*__
+
+The sleep time before scanning for new data. 
+
+* **Default value:** *10* seconds.
+* **Note:** Does not seems to be implemented
+
+__*sincedb*__
+
+The Azure Table name to keep track of what have been done like when we 
+use the file plugin. This define the table name that will be used during 
+the process. **IMPORTANT** This will __not__ take into account any *.lock* 
+files. It will also __not__ create any *.lock* files. 
+
+* **Default value:** No default value, if a value is defined, than it will 
+create the *sincedb* table in the blob account.
+
+__*ignore_older*__
+
+When the file input discovers a file that was last modified before the 
+specified timespan in seconds, the file is ignored. After it's discovery, 
+if an ignored file is modified it is no longer ignored and any new data 
+is read. The default is 24 hours.
+
+* **Default value:** *24*60*60* (24h)
+
+__*start_position*__
+
+Choose where Logstash starts initially reading blob: at the beginning or
+at the end. The default behavior treats files like live streams and thus
+starts at the end. If you have old data you want to import, set this
+to 'beginning'.
+
+This option only modifies *"first contact"* situations where a file
+is new and not seen before, **i.e.** files that don't have a current
+position recorded in a sincedb read by Logstash. If a file
+has already been seen before, this option has no effect and the
+position recorded in the sincedb file will be used.
+
+* **Possible values:** [beginning | end]
+* **Dependency:** *sincedb* needs to be activated. 
+* **Default value:** *beginning*
+
+__*path_prefix*__
+
+Array of blob "path" prefixes. It defines the path prefix to watch in the 
+blob container. Path are defined by the blob name (i.e.: ["path/to/blob.log"]). 
+Regex cannot really be used to optimize perfs.
+
+I recommend to use the paths in order to speed up the processing. *By example, 
+WebApp on azure with IIS logging enabled will create one folder per hour. If 
+you keep the logs for a long retention it will select all before keeping only 
+the last modified ones.*
+
+* **Default value:** *[""]*
+
+***
+
+### Example 1 Basic (out of the box)
+Read from a blob (any type) and send it to ElasticSearch.
+```
+input
+{
+    azureblob
+    {
+        storage_account_name => "mystorageaccount"
+        storage_access_key => "VGhpcyBpcyBhIGZha2Uga2V5Lg=="
+        container => "mycontainer"
+    }
+}
+output
+{
+  elasticsearch {
+    hosts => "localhost"
+    index => "logstash-azureblob-%{+YYYY-MM-dd}"
+  }
+} 
+```
+
+#### What will it do
+It will get the blob, create an empty lock file (512 bytes) and read the entire blob **only once**. Each iteration of the plugin will take a new file and create a new lock file and push the original file to ElasticSearch. If any modification are made on the blob file, it won't be taken into account to push the new data to ElasticSearch. (*No use of sincedb in this situation*)
+
+### Example 2 Advanced (Using sincedb and some other features)
+Read from a blob and send it to ElasticSearch and keep track of where we are in the file.
+
+```
+input
+{
+    azureblob
+    {
+        storage_account_name => "mystorageaccount"
+        storage_access_key   => "VGhpcyBpcyBhIGZha2Uga2V5Lg=="
+        container            => "mycontainer"
+        codec                => "line"         # Optional override => use line instead of json
+        sincedb              => "sincedb"      # Optional          => Activate the sincedb in Azure table
+        sleep_time           => 60             # Optional override => Azure IIS blob are updated each minutes
+        ignore_older         => 2*60*60        # Optional override => Set to 2hours instead of 24 hours
+        path_prefix          => ["ServerA/iis/2016/06/30/", "ServerA/log4net/"]
+        start_position       => "end"          # Optional override => First contact set the sincedb to the end
+    }
+}
+output
+{
+  elasticsearch {
+    hosts => "localhost"
+    index => "logstash-azureblob-%{+YYYY-MM-dd}"
+  }
+}
+```
+
+## More information
+The source code of this plugin is hosted in GitHub repo [Microsoft Azure Diagnostics with ELK](https://github.com/Azure/azure-diagnostics-tools). We welcome you to provide feedback and/or contribute to the project.

data/lib/logstash/inputs/azureblob.rb

@@ -0,0 +1,269 @@
+# encoding: utf-8
+require "logstash/inputs/base"
+require "logstash/namespace"
+
+require "azure"
+require "base64"
+require "securerandom"
+
+# Reads events from Azure Blobs
+class LogStash::Inputs::Azureblob < LogStash::Inputs::Base
+  # Define the plugin name
+  config_name "azureblob"
+
+  # Codec
+  # *Possible values available at https://www.elastic.co/guide/en/logstash/current/codec-plugins.html
+  # *Most used: json_lines, line, etc.
+  default :codec, "json_lines"
+  
+  # storage_account_name
+  # *Define the Azure Storage Account Name
+  config :storage_account_name, :validate => :string, :required => true
+
+  # storage_access_key
+  # *Define the Azure Storage Access Key (available through the portal)
+  config :storage_access_key, :validate => :string, :required => true
+
+  # container
+  # *Define the container to watch
+  config :container, :validate => :string, :required => true
+
+  # sleep_time
+  # *Define the sleep_time between scanning for new data
+  config :sleep_time, :validate => :number, :default => 10, :required => false
+    
+  # [New]
+  # path_prefix
+  # *Define the path prefix in the container in order to not take everything
+  config :path_prefix, :validate => :array, :default => [""], :required => false
+
+  # sincedb
+  # *Define the Azure Storage Table where we can drop information about the the blob we're collecting. 
+  # *Important! The sincedb will be on the container we're watching.
+  # *By default, we don't use the sincedb but I recommend it if files gets updated.
+  config :sincedb, :validate => :string, :required => false
+
+  # ignore_older
+  # When the file input discovers a file that was last modified
+  # before the specified timespan in seconds, the file is ignored.
+  # After it's discovery, if an ignored file is modified it is no
+  # longer ignored and any new data is read. The default is 24 hours.
+  config :ignore_older, :validate => :number, :default => 24 * 60 * 60, :required => false
+
+  # Choose where Logstash starts initially reading blob: at the beginning or
+  # at the end. The default behavior treats files like live streams and thus
+  # starts at the end. If you have old data you want to import, set this
+  # to 'beginning'.
+  #
+  # This option only modifies "first contact" situations where a file
+  # is new and not seen before, i.e. files that don't have a current
+  # position recorded in a sincedb file read by Logstash. If a file
+  # has already been seen before, this option has no effect and the
+  # position recorded in the sincedb file will be used.
+  config :start_position, :validate => [ "beginning", "end"], :default => "end", :required => false
+
+  # Initialize the plugin
+  def initialize(*args)
+    super(*args)
+  end # def initialize
+  
+  public
+  def register
+    Azure.configure do |config|
+      config.storage_account_name = @storage_account_name
+      config.storage_access_key = @storage_access_key
+      config.storage_table_host = "https://#{@storage_account_name}.table.core.windows.net"
+    end
+    @azure_blob = Azure::Blob::BlobService.new
+    
+    if (@sincedb)
+      @azure_table = Azure::Table::TableService.new
+      init_wad_table
+    end
+  end # def register
+  
+  # Initialize the WAD Table if we have a sincedb defined.
+  def init_wad_table
+    if (@sincedb)
+      begin
+        @azure_table.create_table(@sincedb) # Be sure that the table name is properly named.
+      rescue
+        @logger.info("#{DateTime.now} Table #{@sincedb} already exists.")
+      end
+    end
+  end # def init_wad_table
+  
+  # List the blob names in the container. If we have any path pattern defined, it will filter 
+  # the blob names from the list. The status of the blobs will be persisted in the WAD table.
+  #
+  # Returns the list of blob_names to read from.
+  def list_blobs
+    blobs = Hash.new
+    now_time = DateTime.now.new_offset(0)
+
+    @logger.info("#{DateTime.now} Looking for blobs in #{path_prefix.length} paths (#{path_prefix.to_s})...")
+
+    path_prefix.each do |prefix|
+      continuation_token = NIL
+      loop do
+        entries = @azure_blob.list_blobs(@container, { :timeout => 10, :marker => continuation_token, :prefix => prefix})
+        entries.each do |entry|
+          entry_last_modified = DateTime.parse(entry.properties[:last_modified]) # Normally in GMT 0
+          elapsed_seconds = ((now_time - entry_last_modified) * 24 * 60 * 60).to_i
+          if (elapsed_seconds <= @ignore_older)
+            blobs[entry.name] = entry
+          end
+        end
+        continuation_token = entries.continuation_token
+        break if continuation_token.empty?
+      end
+    end 
+
+    @logger.info("#{DateTime.now} Finished looking for blobs. #{blobs.length} are queued for possible candidate with new data")
+
+    return blobs
+  end # def list_blobs
+
+  # Acquire the lock on the blob. Default duration is 60 seconds with a timeout of 10 seconds.
+  # *blob_name: Blob name to threat
+  # Returns true if the aquiring works
+  def acquire_lock(blob_name)
+    @azure_blob.create_page_blob(@container, blob_name, 512)
+    @azure_blob.acquire_lease(@container, blob_name,{:duration=>60, :timeout=>10, :proposed_lease_id=>SecureRandom.uuid})
+
+    return true
+    
+    # Shutdown signal for graceful shutdown in LogStash
+    rescue LogStash::ShutdownSignal => e
+      raise e
+    rescue => e
+      @logger.error("#{DateTime.now} Caught exception while locking", :exception => e)
+    return false
+  end # def acquire_lock
+  
+  # Do the official lock on the blob
+  # *blob_names: Array of blob names to threat
+  def lock_blob(blobs)
+    # Take all the blobs without a lock file.
+    real_blobs = blobs.select { |name, v| !name.end_with?(".lock") }
+  
+    # Return the first one not marked as locked + lock it.
+    real_blobs.each do |blob_name, blob|
+      if !blobs.keys.include?(blob_name + ".lock")      
+        if acquire_lock(blob_name + ".lock")
+          return blob
+        end
+      end
+    end
+
+    return NIL
+  end # def lock_blob
+
+  def list_sinceDbContainerEntities
+    entities = Set.new
+
+    #loop do
+      continuation_token = NIL
+
+      entries = @azure_table.query_entities(@sincedb, { :filter => "PartitionKey eq '#{container}'", :continuation_token => continuation_token}) 
+      entries.each do |entry|
+          entities << entry
+      end
+      #continuation_token = entries.continuation_token
+      #break if continuation_token.empty?
+    #end
+
+    return entities
+  end # def list_sinceDbContainerEntities
+
+  # Process the plugin ans start watching.
+  def process(output_queue)
+    blobs = list_blobs
+
+    # use the azure table in order to set the :start_range and :end_range
+    # When we do that, we shouldn't use the lock strategy, since we know where we are at. It would still be interesting in a multi-thread
+    # environment.
+    if (@sincedb)
+      existing_entities = list_sinceDbContainerEntities # @azure_table.query_entities(@sincedb, { :filter => "PartitionKey eq '#{container}'"}) # continuation_token...
+
+      blobs.each do |blob_name, blob_info|
+        blob_name_encoded = Base64.strict_encode64(blob_info.name)
+        entityIndex = existing_entities.find_index {|entity| entity.properties["RowKey"] == blob_name_encoded }
+
+        entity = { 
+          "PartitionKey" => @container, 
+          "RowKey" => blob_name_encoded, 
+          "ByteOffset" => 0, # First contact, start_position is beginning by default
+          "ETag" => NIL,
+          "BlobName" => blob_info.name
+        }
+
+        if (entityIndex)
+          # exists in table
+          foundEntity = existing_entities.to_a[entityIndex];
+          entity["ByteOffset"] = foundEntity.properties["ByteOffset"]
+          entity["ETag"] = foundEntity.properties["ETag"] 
+        elsif (@start_position === "end")
+          # first contact
+          entity["ByteOffset"] = blob_info.properties[:content_length]
+        end
+
+        if (entity["ETag"] === blob_info.properties[:etag])
+          # nothing to do...
+          # @logger.info("#{DateTime.now} Blob already up to date #{blob_info.name}")
+        else
+          @logger.info("#{DateTime.now} Processing #{blob_info.name}")
+          blob, content = @azure_blob.get_blob(@container,  blob_info.name, { :start_range => entity["ByteOffset"], :end_range =>  blob_info.properties[:content_length] })
+
+          @codec.decode(content) do |event|
+            decorate(event) # we could also add the host name that read the blob in the event from here.
+            # event["host"] = hostname...
+            output_queue << event
+          end 
+
+          # Update the entity with the latest informations we used while processing the blob. If we have a crash, 
+          # we will re-process the last batch.
+          entity["ByteOffset"] = blob_info.properties[:content_length]
+          entity["ETag"] = blob_info.properties[:etag]
+          @azure_table.insert_or_merge_entity(@sincedb, entity)
+        end
+      end
+    else
+      # Process the ones not yet processed. (The newly locked blob)
+      blob_info = lock_blob(blobs)
+    
+      # Do what we were doing before
+      return if !blob_info
+        @logger.info("#{DateTime.now} Processing #{blob_info.name}")
+        
+        blob, content = @azure_blob.get_blob(@container, blob_info.name)
+        @codec.decode(content) do |event|
+          decorate(event) # we could also add the host name that read the blob in the event from here.
+          # event["host"] = hostname...
+          output_queue << event
+      end
+    end
+    
+    # Shutdown signal for graceful shutdown in LogStash
+    rescue LogStash::ShutdownSignal => e
+      raise e
+    rescue => e
+      @logger.error("#{DateTime.now} Oh My, An error occurred.", :exception => e)
+  end # def process
+  
+  # Run the plugin (Called directly by LogStash)
+  public
+  def run(output_queue)
+    # Infinite processing loop.
+    while !stop?
+      process(output_queue)      
+      sleep sleep_time
+    end # loop
+  end # def run
+ 
+  public
+  def teardown
+    # Nothing to do.
+    @logger.info("Teardown")
+  end # def teardown
+end # class LogStash::Inputs::Azureblob

data/logstash-input-azureblob.gemspec

@@ -0,0 +1,24 @@
+Gem::Specification.new do |s|
+  s.name          = 'Nordes-logstash-input-azureblob'
+  s.version       = '0.9.5-1'
+  s.licenses      = ['Apache License (2.0)']
+  s.summary       = "This plugin collects Microsoft Azure Diagnostics data from Azure Storage Blobs."
+  s.description   = "This gem is a Logstash plugin. It reads and parses data from Azure Storage Blobs."
+  s.authors       = ["Microsoft Corporation"]
+  s.email         = 'azdiag@microsoft.com'
+  s.homepage      = "https://github.com/Azure/azure-diagnostics-tools"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','Gemfile','LICENSE']
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+  s.add_runtime_dependency 'azure', '~> 0.7.1'
+  s.add_development_dependency 'logstash-devutils'
+end

data/spec/inputs/azureblob_spec.rb

@@ -0,0 +1 @@
+require "logstash/devutils/rspec/spec_helper"

metadata

@@ -0,0 +1,101 @@
+--- !ruby/object:Gem::Specification
+name: Nordes-logstash-input-azureblob
+version: !ruby/object:Gem::Version
+  version: 0.9.5.pre.1
+platform: ruby
+authors:
+- Microsoft Corporation
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2017-05-15 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.7.1
+  name: azure
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.7.1
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a Logstash plugin. It reads and parses data from Azure Storage Blobs.
+email: azdiag@microsoft.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/inputs/azureblob.rb
+- logstash-input-azureblob.gemspec
+- spec/inputs/azureblob_spec.rb
+homepage: https://github.com/Azure/azure-diagnostics-tools
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">"
+    - !ruby/object:Gem::Version
+      version: 1.3.1
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
+specification_version: 4
+summary: This plugin collects Microsoft Azure Diagnostics data from Azure Storage Blobs.
+test_files:
+- spec/inputs/azureblob_spec.rb