logstash-input-azure_blob_storage 0.10.0 → 0.10.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 47775d226b17cd57d8ce290569e35cf893713f42757bd953fd46b93055733527
-  data.tar.gz: de73405430b71405ecc0a4873a72feefabdb7dd7f410734dd550928623a39c53
+  metadata.gz: db216440cf4319f70a5fbdb001a53da72826afdcbce43c04ada28b63c5d9e1f8
+  data.tar.gz: e2e519090c0d67b6b65f4570c34be4f0b02592864459f261c7be613486f2941b
 SHA512:
-  metadata.gz: a028a0df1310312d9a1826de016407693ca14002c316d08267de3be51abc94aaa9db997e3fb19677cd23db551d029d7e1acffc65068eb0b2ca4a2a6d408dbbe7
-  data.tar.gz: b54aa9e59046793f26bfaa5fcd2795c809f2fdd535449f00a8ab6b7392107f8c9a2d3f937a3d24d1c0bcb2890d60493baeb360b29c1188666781ecc9e4e7b014
+  metadata.gz: 341abd35cf3b732c1a0bada111cbac79cafa92fb48fd14ad24e0121245c67f1819c1d6e5e4647dda645773231da6aa2a61b7e1aaee4027fa97fe9e857a8a334f
+  data.tar.gz: 93542f740dda404889f623c9a9d460a1ac6f7181ddd6974de8d775e4e29585b8fccd43b9bd05a880250a4da2709121bf5f018f15a8ce49a9d758b1c645482cbf

data/CHANGELOG.md

@@ -1,2 +1,3 @@
-## 0.1.0
+## 0.10.0
   - Plugin created with the logstash plugin generator
+  - Reimplemented logstash-input-azureblob with incompatible config and data/registry

data/README.md

@@ -10,10 +10,13 @@ All plugin documentation are placed under one [central location](http://www.elas
 
 ## Need Help?
 
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum. For real problems or feature requests, raise a github issue. Pull requests will ionly be merged after discussion through an issue.
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum. For real problems or feature requests, raise a github issue [GITHUB/janmg/logstash-input-azure_blob_storage/](https://github.com/janmg/logstash-input-azure_blob_storage). Pull requests will ionly be merged after discussion through an issue.
 
 ## Purpose
-This plugin can read from Azure Storage Blobs, after every interval it will write a registry to the storageaccount to save the information of how many bytes per blob (file) are read and processed. After all files are processed and at least one interval has passed a new file list is generated and a worklist is constructed that will be processed. When a file has already been processed before, partial files are read from the offset to the filesize at the time of the file listing. If the codec is JSON partial files will be have the header and tail will be added. They can be configured. If logtype is nsgflowlog, the plugin will process the splitting into individual tuple events. The logtype wadiis may in the future be used to process the grok formats to split into log lines. Any other format is fed into the queue as one event per file or partial file. It's then up to the filter to split and mutate the file format. use source => message in the filter {} block.
+This plugin can read from Azure Storage Blobs, for instance diagnostics logs for NSG flow logs or accesslogs from App Services. 
+[Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/)
+
+After every interval it will write a registry to the storageaccount to save the information of how many bytes per blob (file) are read and processed. After all files are processed and at least one interval has passed a new file list is generated and a worklist is constructed that will be processed. When a file has already been processed before, partial files are read from the offset to the filesize at the time of the file listing. If the codec is JSON partial files will be have the header and tail will be added. They can be configured. If logtype is nsgflowlog, the plugin will process the splitting into individual tuple events. The logtype wadiis may in the future be used to process the grok formats to split into log lines. Any other format is fed into the queue as one event per file or partial file. It's then up to the filter to split and mutate the file format. use source => message in the filter {} block.
 
 ## Installation 
 This plugin can be installed through logstash-plugin
@@ -49,6 +52,9 @@ curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/j
 
 
 ## Configuration Examples
+The minimum configuration required as input is storageaccount, access_key and container.
+
+For nsgflowlogs, a simple configuration looks like this
 ```
 input {
     azure_blob_storage {
@@ -79,7 +85,7 @@ output {
 }
 ```
 
-You can include additional options to tweak the operations
+It's possible to specify the optional parameters to overwrite the defaults. The iplookup, use_redis and iplist parameters are used for additional information about the source and destination ip address. Redis can be used for caching the results and iplist is to configure an array of ip addresses.
 ```
 input {
     azure_blob_storage {
@@ -90,7 +96,7 @@ input {
         logtype => "nsgflowlog"
         prefix => "resourceId=/"
         registry_create_policy => "resume"
-        interval => 60
+        interval => 300
         iplookup => "http://10.0.0.5:6081/ripe.php?ip="
         use_redis => true
         iplist => [
@@ -100,3 +106,47 @@ input {
     }
 }
 ```
+
+For WAD IIS and App Services the HTTP AccessLogs can be retrieved from a storage account as line based events and parsed through GROK. The date stamp can also be parsed with %{TIMESTAMP_ISO8601:log_timestamp}. For WAD IIS logfiles the container is wad-iis-logfiles. In the future grokking may happen already by the plugin.
+```
+input {
+    azure_blob_storage {
+        storageaccount => "yourstorageaccountname"
+        access_key => "Ba5e64c0d3=="
+        container => "access-logs"
+        interval => 300
+        codec => line
+    }
+}
+
+filter {
+  if [message] =~ "^#" {
+    drop {}
+  }
+
+  mutate {
+    strip => "message"
+  }
+
+  grok {
+    match => ['message', '(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}\d+) %{NOTSPACE:instanceId} %{WORD:httpMethod} %{URIPATH:requestUri} %{NOTSPACE:requestQuery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientIP} %{NOTSPACE:userAgent} %{NOTSPACE:cookie} %{NOTSPACE:referer} %{NOTSPACE:host} %{NUMBER:httpStatus} %{NUMBER:subresponse} %{NUMBER:win32response} %{NUMBER:sentBytes:int} %{NUMBER:receivedBytes:int} %{NUMBER:timeTaken:int}']
+  }
+
+  date {
+    match => [ "timestamp", "YYYY-MM-dd HH:mm:ss" ]
+    target => "@timestamp"
+  }
+
+  mutate {
+    remove_field => ["log_timestamp"]
+    remove_field => ["message"]
+    remove_field => ["win32response"]
+    remove_field => ["subresponse"]
+    remove_field => ["username"]
+    remove_field => ["clientPort"]
+    remove_field => ["port"]
+    remove_field => ["timestamp"]
+  }
+}
+```
+

data/lib/logstash/inputs/azure_blob_storage.rb

@@ -7,10 +7,10 @@ require 'azure/storage/blob'
 #require 'date'
 #require 'json'
 #require 'thread'
-#require "redis"
+#require 'redis'
 #require 'net/http'
 
-# This is a logstash input plugin for files in Azure Blob Storage. There is a storage explorer in the portal and an application with the same name https://storageexplorer.com. A storage account has by default a globally unique name, {storageaccount}.blob.core.windows.net which is a CNAME to  Azures blob servers blob.*.store.core.windows.net. A storageaccount has an container and those have a directory and blobs (like files) and blobs are constructed of or more blocks. Some Azure diagnostics can send events to an EventHub that can be parse through the plugin logstash-input-azure_event_hubs, but for the events that are only stored in an storage account, use this plugin. The original logstash-input-azureblob from azure-diagnostics-tools is great for low volumes, but it suffers from outdated client, slow reads, lease locking issues and json parse errors.
+# This is a logstash input plugin for files in Azure Blob Storage. There is a storage explorer in the portal and an application with the same name https://storageexplorer.com. A storage account has by default a globally unique name, {storageaccount}.blob.core.windows.net which is a CNAME to  Azures blob servers blob.*.store.core.windows.net. A storageaccount has an container and those have a directory and blobs (like files). Blobs have one or more blocks. After writing the blocks, they can be committed. Some Azure diagnostics can send events to an EventHub that can be parse through the plugin logstash-input-azure_event_hubs, but for the events that are only stored in an storage account, use this plugin. The original logstash-input-azureblob from azure-diagnostics-tools is great for low volumes, but it suffers from outdated client, slow reads, lease locking issues and json parse errors.
 # https://azure.microsoft.com/en-us/services/storage/blobs/
 class LogStash::Inputs::AzureBlobStorage < LogStash::Inputs::Base
   config_name "azure_blob_storage"
@@ -23,16 +23,19 @@ class LogStash::Inputs::AzureBlobStorage < LogStash::Inputs::Base
 
   # The storage account is accessed through Azure::Storage::Blob::BlobService, it needs either a sas_token, connection string or a storageaccount/access_key pair.
   # https://github.com/Azure/azure-storage-ruby/blob/master/blob/lib/azure/storage/blob/blob_service.rb#L42
-  config :connection_string, :validate => :password
+  config :connection_string, :validate => :password, :required => false
 
   # The storage account name for the azure storage account.
-  config :storageaccount, :validate => :string
+  config :storageaccount, :validate => :string, :required => false
+
+  # DNS Suffix other then blob.core.windows.net
+  config :dns_suffix, :validate => :string, :required => false, :default => 'core.windows.net'
 
   # The (primary or secondary) Access Key for the the storage account. The key can be found in the portal.azure.com or through the azure api StorageAccounts/ListKeys. For example the PowerShell command Get-AzStorageAccountKey.
-  config :access_key, :validate => :password
+  config :access_key, :validate => :password, :required => false
 
   # SAS is the Shared Access Signature, that provides restricted access rights. If the sas_token is absent, the access_key is used instead.
-  config :sas_token, :validate => :password
+  config :sas_token, :validate => :password, :required => false
 
   # The container of the blobs.
   config :container, :validate => :string, :default => 'insights-logs-networksecuritygroupflowevent'
@@ -91,21 +94,22 @@ class LogStash::Inputs::AzureBlobStorage < LogStash::Inputs::Base
 
   # Optional to enrich NSGFLOWLOGS with netname and subnet the iplookup value points to a webservice that provides the information in JSON format like this.
   # {"ip":"8.8.8.8","netname":"Google","subnet":"8.8.8.0\/24","hostname":"google-public-dns-a.google.com"}
-  config :iplookup, :validate => :string, :required => false, :default => 'http://127.0.0.1/ripe.php?ip='
-
-  # Optional Redis IP cache
-  config :use_redis, :validate => :boolean, :required => false, :default => false
-
+  # In the query parameter has the <ip> tag will be replaced by the IP address to lookup, other parameters are optional and according to your lookup service. 
+  config :iplookup, :validate => :string, :required => false, :default => 'http://127.0.0.1/ripe.php?ip=<ip>&TOKEN=token'
 
   # Optional array of JSON objects that don't require a lookup
   config :iplist, :validate => :array, :required => false, :default => ['{"ip":"10.0.0.4","netname":"Application Gateway","subnet":"10.0.0.0\/24","hostname":"appgw"}']
 
+  # Optional Redis IP cache
+  config :use_redis, :validate => :boolean, :required => false, :default => false
+
 
 
 public
 def register
     @pipe_id = Thread.current[:name].split("[").last.split("]").first
     @logger.info("=== "+config_name+"/"+@pipe_id+"/"+@id[0,6]+" ===")
+    @logger.info("Contact me at jan@janmg.com, if something in this plugin doesn't work")
     # TODO: consider multiple readers, so add pipeline @id or use logstash-to-logstash communication?
     # TODO: Implement retry ... Error: Connection refused - Failed to open TCP connection to
 
@@ -118,11 +122,12 @@ def register
     # 2. connection_string
     # 3. storageaccount / access_key
 
-    conn = connection_string
+    unless connection_string.nil?
+	conn = connection_string.value
+    end
     unless sas_token.nil?
-        # TODO: Fix SAS Tokens
         unless sas_token.value.start_with?('?')
-		conn = "BlobEndpoint=https://#{storageaccount}.blob.core.windows.net;SharedAccessSignature=#{sas_token.value}"
+		conn = "BlobEndpoint=https://#{storageaccount}.#{dns_suffix};SharedAccessSignature=#{sas_token.value}"
         else
 		conn = sas_token.value
     	end
@@ -132,6 +137,7 @@ def register
     else
         @blob_client = Azure::Storage::Blob::BlobService.create(
             storage_account_name: storageaccount,
+	    storage_dns_suffix: dns_suffix,
             storage_access_key: access_key.value,
         )
     end
@@ -200,7 +206,7 @@ def run(queue)
         
         # Worklist is the subset of files where the already read offset is smaller than the file size
         worklist = filelist.select {|name,file| file[:offset] < file[:length]}
-        @logger.info(@pipe_id+" worklist contains #{worklist.size} blobs to process")
+        @logger.debug(@pipe_id+" worklist contains #{worklist.size} blobs to process")
         # This would be ideal for threading since it's IO intensive, would be nice with a ruby native ThreadPool
         worklist.each do |name, file|
             res = resource(name)
@@ -255,6 +261,8 @@ def stop
 end
 
 
+
+private
 def full_read(filename)
     return @blob_client.get_blob(container, filename)[1]
 end
@@ -371,6 +379,7 @@ end
 def learn_encapsulation
     # From one file, read first block and last block to learn head and tail
     blob = @blob_client.list_blobs(container, { maxresults: 1, prefix: @prefix }).first
+    return if blob.nil?
     blocks = @blob_client.list_blob_blocks(container, blob.name)[:committed]
     @logger.info(@pipe_id+" using #{blob.name} to learn the json header and tail")
     @head = @blob_client.get_blob(container, blob.name, start_range: 0, end_range: blocks.first.size-1)[1]
@@ -397,6 +406,7 @@ end
 
 
 # Optional lookup for netname and hostname for the srcip and dstip returned in a Hash
+# TODO: split out to own class
 def addip(srcip, dstip)
     #TODO: return anonymous merge
     srcjson = JSON.parse(lookup(srcip))
@@ -409,8 +419,10 @@ def lookup(ip)
     unless @red.nil?
         res = @red.get(ip)
     end
+    uri = URI.parse(iplookup.sub('<ip>',ip))
+    res = Net::HTTP.get(uri)
     if res.nil?
-        res = Net::HTTP.get(URI(iplookup + ip))
+        res = Net::HTTP.get(uri)
 	unless @red.nil?
             @red.set(ip, res)
             @red.expire(ip,604800)

data/logstash-input-azure_blob_storage.gemspec

@@ -1,9 +1,20 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-input-azure_blob_storage'
-  s.version       = '0.10.0'
+  s.version       = '0.10.1'
   s.licenses      = ['Apache-2.0']
   s.summary       = 'This logstash plugin reads and parses data from Azure Storage Blobs.'
-  s.description   = 'This gem is a Logstash plugin. It reads and parses data from Azure Storage Blobs. The azure_blob_storage is a rewrite to replace azureblob from azure-diagnostics-tools/Logstash. It can deal with larger volumes and partial file reads and eliminating a delay when rebuilding the registry'
+  s.description   = <<-EOF
+ This gem is a Logstash plugin. It reads and parses data from Azure Storage Blobs. The azure_blob_storage is a reimplementation to replace azureblob from azure-diagnostics-tools/Logstash. It can deal with larger volumes and partial file reads and eliminating a delay when rebuilding the registry.
+
+ The logstash pipeline configuration would look like this
+  input {
+    azure_blob_storage {
+        storageaccount => "yourstorageaccountname"
+        access_key => "Ba5e64c0d3=="
+        container => "insights-logs-networksecuritygroupflowevent"
+    }
+  }
+EOF
   s.homepage      = 'https://github.com/janmg/logstash-input-azure_blob_storage'
   s.authors       = ['Jan Geertsma']
   s.email         = 'jan@janmg.com'

data/spec/inputs/azure_blob_storage_spec.rb

@@ -8,4 +8,11 @@ describe LogStash::Inputs::AzureBlobStorage do
     let(:config) { { "interval" => 100 } }
   end
 
+  def test_helper_methodes 
+      assert_equal('b', AzureBlobStorage.val('a=b')
+      assert_equal('whatever', AzureBlobStorage.strip_comma(',whatever')
+      assert_equal('whatever', AzureBlobStorage.strip_comma('whatever,')
+      assert_equal('whatever', AzureBlobStorage.strip_comma(',whatever,')
+      assert_equal('whatever', AzureBlobStorage.strip_comma('whatever')
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-input-azure_blob_storage
 version: !ruby/object:Gem::Version
-  version: 0.10.0
+  version: 0.10.1
 platform: ruby
 authors:
 - Jan Geertsma
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2019-02-27 00:00:00.000000000 Z
+date: 2019-03-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -86,10 +86,17 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 1.0.0
-description: This gem is a Logstash plugin. It reads and parses data from Azure Storage
-  Blobs. The azure_blob_storage is a rewrite to replace azureblob from azure-diagnostics-tools/Logstash.
-  It can deal with larger volumes and partial file reads and eliminating a delay when
-  rebuilding the registry
+description: |2
+   This gem is a Logstash plugin. It reads and parses data from Azure Storage Blobs. The azure_blob_storage is a reimplementation to replace azureblob from azure-diagnostics-tools/Logstash. It can deal with larger volumes and partial file reads and eliminating a delay when rebuilding the registry.
+
+   The logstash pipeline configuration would look like this
+    input {
+      azure_blob_storage {
+          storageaccount => "yourstorageaccountname"
+          access_key => "Ba5e64c0d3=="
+          container => "insights-logs-networksecuritygroupflowevent"
+      }
+    }
 email: jan@janmg.com
 executables: []
 extensions: []