logstash-input-azure_blob_storage 0.11.5 → 0.12.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3d446aed971a95e6e17a27ed1e9ec8b141f939b53697fb9c332cfb130404745a
-  data.tar.gz: 4a1321f6c6a30f6787d2133642ca23840371d6f4e18102cb775d345b09eb176a
+  metadata.gz: 202f43b2d085a872fef61594e05dbc3eb19188863a20b5f4ffece2ef66608d15
+  data.tar.gz: 3b13f7a610c7b541ee7265340103011998d5c1799dfc7a1057add7307f1550dc
 SHA512:
-  metadata.gz: b4f48a0bebcd6e3594584a4473b223838359d44e9ef591f958aa4c80c4c22953f6b0f708b19faeaf0517c66f47185bda4de75ab4e3618b23e2e7f23f71cb4bee
-  data.tar.gz: 508cd39ea159a4655e590f46ad0108c3b6e6de95ed575c4456da0230bae73fb384ecb7697ed710e7afb1542fe01cbd8a62130acedcbf0ba9c3040ace1f9d76d0
+  metadata.gz: 0af0f2e0f0f955849840190ebfd60185ae5b0465e4d509f40b99bf075da44ed50e0120c8d781d2455b0b16969cc4c75f8a3ff9e71431e73e0bfaa03dc284b708
+  data.tar.gz: 3e7907fe4700bfe3640b5783d63b17d0f153061b0a57bb61964ca08f83332e616e158f754904807a1b804be0ba7a64be4c6d92aa7ee9393e42ca33a8ac9e54a4

data/CHANGELOG.md

@@ -1,6 +1,22 @@
+## 0.12.0
+  - version 2 of azure-storage
+  - saving current files registry, not keeping historical files
+
+## 0.11.7
+  - implemented skip_learning
+  - start ignoring failed files and not retry
+
+## 0.11.6
+  - fix in json head and tail learning the max_results
+  - broke out connection setup in order to call it again if connection exceptions come
+  - deal better with skipping of empty files.
+
 ## 0.11.5
-  - Added optional filename into the message
-  - plumbing for emulator, start_over not learning from registry
+  - added optional addfilename to add filename in message
+  - NSGFLOWLOG version 2 uses 0 as value instead of NULL in src and dst values
+  - added connection exception handling when full_read files
+  - rewritten json header footer learning to ignore learning from registry  
+  - plumbing for emulator
 
 ## 0.11.4
   - fixed listing 3 times, rather than retrying to list max 3 times

data/README.md

@@ -1,30 +1,34 @@
-# Logstash Plugin
+# Logstash 
 
-This is a plugin for [Logstash](https://github.com/elastic/logstash).
+This is a plugin for [Logstash](https://github.com/elastic/logstash). It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way. All logstash plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/). Need generic logstash help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
 
-It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+For problems or feature requests with this specific plugin, raise a github issue [GITHUB/janmg/logstash-input-azure_blob_storage/](https://github.com/janmg/logstash-input-azure_blob_storage). Pull requests will also be welcomed after discussion through an issue.
 
-## Documentation
-
-All logstash plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+## Purpose
+This plugin can read from Azure Storage Blobs, for instance JSON diagnostics logs for NSG flow logs or LINE based accesslogs from App Services. 
+[Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/)
 
-## Need Help?
+The plugin depends on the [Ruby library azure-storage-blon](https://rubygems.org/gems/azure-storage-blob/versions/1.1.0) from Microsoft, that depends on Faraday for the HTTPS connection to Azure.
 
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum. For real problems or feature requests, raise a github issue [GITHUB/janmg/logstash-input-azure_blob_storage/](https://github.com/janmg/logstash-input-azure_blob_storage). Pull requests will ionly be merged after discussion through an issue.
+The plugin executes the following steps
+1. Lists all the files in the azure storage account. where the path of the files are matching pathprefix
+2. Filters on path_filters to only include files that match the directory and file glob (e.g. **/*.json)
+3. Save the listed files in a registry of known files and filesizes. (data/registry.dat on azure, or in a file on the logstash instance)
+4. List all the files again and compare the registry with the new filelist and put the delta in a worklist
+5. Process the worklist and put all events in the logstash queue.
+6. if there is time left, sleep to complete the interval. If processing takes more than an inteval, save the registry and continue processing.
+7. If logstash is stopped, a stop signal will try to finish the current file, save the registry and than quit
 
-## Purpose
-This plugin can read from Azure Storage Blobs, for instance diagnostics logs for NSG flow logs or accesslogs from App Services. 
-[Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/)
-This 
 ## Installation 
 This plugin can be installed through logstash-plugin
 ```
-logstash-plugin install logstash-input-azure_blob_storage
+/usr/share/logstash/bin/logstash-plugin install logstash-input-azure_blob_storage
 ```
 
 ## Minimal Configuration
 The minimum configuration required as input is storageaccount, access_key and container.
 
+/etc/logstash/conf.d/test.conf
 ```
 input {
     azure_blob_storage {
@@ -36,27 +40,29 @@ input {
 ```
 
 ## Additional Configuration
-The registry_create_policy is used when the pipeline is started to either resume from the last known unprocessed file, or to start_fresh ignoring old files or start_over to process all the files from the beginning.
+The registry keeps track of files in the storage account, their size and how many bytes have been processed. Files can grow and the added part will be processed as a partial file. The registry is saved todisk every interval.
 
-interval defines the minimum time the registry should be saved to the registry file (by default 'data/registry.dat'), this is only needed in case the pipeline dies unexpectedly. During a normal shutdown the registry is also saved.
+The registry_create_policy determines at the start of the pipeline if processing should resume from the last known unprocessed file, or to start_fresh ignoring old files and start only processing new events that came after the start of the pipeline. Or start_over to process all the files ignoring the registry.
 
-When registry_local_path is set to a directory, the registry is save on the logstash server in that directory. The filename is the pipe.id
+interval defines the minimum time the registry should be saved to the registry file (by default to 'data/registry.dat'), this is only needed in case the pipeline dies unexpectedly. During a normal shutdown the registry is also saved.
 
-with registry_create_policy set to resume and the registry_local_path set to a directory where the registry isn't yet created, should load from the storage account and save the registry on the local server
+When registry_local_path is set to a directory, the registry is saved on the logstash server in that directory. The filename is the pipe.id
 
-During the pipeline start for JSON codec, the plugin uses one file to learn how the JSON header and tail look like, they can also be configured manually.
+with registry_create_policy set to resume and the registry_local_path set to a directory where the registry isn't yet created, should load the registry from the storage account and save the registry on the local server. This allows for a migration to localstorage
+
+For pipelines that use the JSON codec or the JSON_LINE codec, the plugin uses one file to learn how the JSON header and tail look like, they can also be configured manually. Using skip_learning the learning can be disabled.
 
 ## Running the pipeline
 The pipeline can be started in several ways.
  - On the commandline
    ```
-   /usr/share/logstash/bin/logtash -f /etc/logstash/pipeline.d/test.yml
+   /usr/share/logstash/bin/logtash -f /etc/logstash/conf.d/test.conf
    ```
  - In the pipeline.yml
    ```
    /etc/logstash/pipeline.yml
    pipe.id = test
-   pipe.path = /etc/logstash/pipeline.d/test.yml
+   pipe.path = /etc/logstash/conf.d/test.conf
    ```
  - As managed pipeline from Kibana
 
@@ -95,7 +101,9 @@ The log level of the plugin can be put into DEBUG through
 curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d'{"logger.logstash.inputs.azureblobstorage" : "DEBUG"}'
 ```
 
-because debug also makes logstash chatty, there are also debug_timer and debug_until that can be used to print additional informantion on what the pipeline is doing and how long it takes. debug_until is for the number of events until debug is disabled.
+Because logstash debug makes logstash very chatty, the option debug_until will for a number of processed events and stops debuging. One file can easily contain thousands of events. The debug_until is useful to monitor the start of the plugin and the processing of the first files.
+
+debug_timer will show detailed information on how much time listing of files took and how long the plugin will sleep to fill the interval and the listing and processing starts again.
 
 ## Other Configuration Examples
 For nsgflowlogs, a simple configuration looks like this
@@ -121,6 +129,10 @@ filter {
     }
 }
 
+output {
+  stdout { }
+}
+
 output {
     elasticsearch {
         hosts => "elasticsearch"
@@ -128,21 +140,35 @@ output {
     }
 }
 ```
-
+A more elaborate input configuration example
 ```
 input {
     azure_blob_storage {
+        codec => "json"
         storageaccount => "yourstorageaccountname"
         access_key => "Ba5e64c0d3=="
         container => "insights-logs-networksecuritygroupflowevent"
-        codec => "json"
         logtype => "nsgflowlog"
         prefix => "resourceId=/"
+        path_filters => ['**/*.json']
+        addfilename => true
         registry_create_policy => "resume"
+        registry_local_path => "/usr/share/logstash/plugin"
         interval => 300
+        debug_timer => true
+        debug_until => 100
+    }
+}
+
+output {
+    elasticsearch {
+        hosts => "elasticsearch"
+        index => "nsg-flow-logs-%{+xxxx.ww}"
     }
 }
 ```
+The configuration documentation is in the first 100 lines of the code
+[GITHUB/janmg/logstash-input-azure_blob_storage/blob/master/lib/logstash/inputs/azure_blob_storage.rb](https://github.com/janmg/logstash-input-azure_blob_storage/blob/master/lib/logstash/inputs/azure_blob_storage.rb)
 
 For WAD IIS and App Services the HTTP AccessLogs can be retrieved from a storage account as line based events and parsed through GROK. The date stamp can also be parsed with %{TIMESTAMP_ISO8601:log_timestamp}. For WAD IIS logfiles the container is wad-iis-logfiles. In the future grokking may happen already by the plugin.
 ```

data/lib/logstash/inputs/azure_blob_storage.rb

@@ -61,7 +61,9 @@ config :registry_create_policy, :validate => ['resume','start_over','start_fresh
 # Z00000000000000000000000000000000 2     ]}
 config :interval, :validate => :number, :default => 60
 
+# add the filename into the events
 config :addfilename, :validate => :boolean, :default => false, :required => false
+
 # debug_until will for a maximum amount of processed messages shows 3 types of log printouts including processed filenames. This is a lightweight alternative to switching the loglevel from info to debug or even trace 
 config :debug_until, :validate => :number, :default => 0, :required => false
 
@@ -71,6 +73,9 @@ config :debug_timer, :validate => :boolean, :default => false, :required => fals
 # WAD IIS Grok Pattern
 #config :grokpattern, :validate => :string, :required => false, :default => '%{TIMESTAMP_ISO8601:log_timestamp} %{NOTSPACE:instanceId} %{NOTSPACE:instanceId2} %{IPORHOST:ServerIP} %{WORD:httpMethod} %{URIPATH:requestUri} %{NOTSPACE:requestQuery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientIP} %{NOTSPACE:httpVersion} %{NOTSPACE:userAgent} %{NOTSPACE:cookie} %{NOTSPACE:referer} %{NOTSPACE:host} %{NUMBER:httpStatus} %{NUMBER:subresponse} %{NUMBER:win32response} %{NUMBER:sentBytes:int} %{NUMBER:receivedBytes:int} %{NUMBER:timeTaken:int}'
 
+# skip learning if you use json and don't want to learn the head and tail, but use either the defaults or configure them.
+config :skip_learning, :validate => :boolean, :default => false, :required => false
+
 # The string that starts the JSON. Only needed when the codec is JSON. When partial file are read, the result will not be valid JSON unless the start and end are put back. the file_head and file_tail are learned at startup, by reading the first file in the blob_list and taking the first and last block, this would work for blobs that are appended like nsgflowlogs. The configuration can be set to override the learning. In case learning fails and the option is not set, the default is to use the 'records' as set by nsgflowlogs.
 config :file_head, :validate => :string, :required => false, :default => '{"records":['
 # The string that ends the JSON
@@ -113,34 +118,7 @@ def run(queue)
     @processed = 0
     @regsaved = @processed
 
-    # Try in this order to access the storageaccount
-    # 1. storageaccount / sas_token
-    # 2. connection_string
-    # 3. storageaccount / access_key
-
-    unless connection_string.nil?
-	conn = connection_string.value
-    end
-    unless sas_token.nil?
-        unless sas_token.value.start_with?('?')
-	    conn = "BlobEndpoint=https://#{storageaccount}.#{dns_suffix};SharedAccessSignature=#{sas_token.value}"
-        else
-	    conn = sas_token.value
-    	end
-    end
-    unless conn.nil?
-        @blob_client = Azure::Storage::Blob::BlobService.create_from_connection_string(conn)
-    else
-#        unless use_development_storage?
-        @blob_client = Azure::Storage::Blob::BlobService.create(
-            storage_account_name: storageaccount,
-	    storage_dns_suffix: dns_suffix,
-            storage_access_key: access_key.value,
-        )
-#        else
-#            @logger.info("not yet implemented")
-#        end
-    end
+    connect
 
     @registry = Hash.new
     if registry_create_policy == "resume"
@@ -175,7 +153,7 @@ def run(queue)
     if registry_create_policy == "start_fresh"
         @registry = list_blobs(true)
 	save_registry(@registry)
-	@logger.info("starting fresh, writing a clean the registry to contain #{@registry.size} blobs/files")
+	@logger.info("starting fresh, writing a clean registry to contain #{@registry.size} blobs/files")
     end
 
     @is_json = false
@@ -188,12 +166,14 @@ def run(queue)
     @tail = ''
     # if codec=json sniff one files blocks A and Z to learn file_head and file_tail
     if @is_json
-        learn_encapsulation
         if file_head
-           @head = file_head
+            @head = file_head
         end
         if file_tail
-           @tail = file_tail
+            @tail = file_tail
+        end
+        if file_head and file_tail and !skip_learning
+            learn_encapsulation
         end
         @logger.info("head will be: #{@head} and tail is set to #{@tail}")
     end
@@ -233,7 +213,10 @@ def run(queue)
 	end
         # size nilClass when the list doesn't grow?!
         # Worklist is the subset of files where the already read offset is smaller than the file size
-	worklist.clear
+        @registry = newreg
+        worklist.clear
+        chunk = nil
+
 	worklist = newreg.select {|name,file| file[:offset] < file[:length]}
 	if (worklist.size > 4) then @logger.info("worklist contains #{worklist.size} blobs") end
 
@@ -246,17 +229,28 @@ def run(queue)
             size = 0
             if file[:offset] == 0
                 # This is where Sera4000 issue starts
-                begin
-                    chunk = full_read(name)
-                    size=chunk.size
-                rescue Exception => e
-                    @logger.error("Failed to read #{name} because of: #{e.message} .. will continue and pretend this never happened")
+                # For an append blob, reading full and crashing, retry, last_modified? ... lenght? ... committed? ...
+                # length and skip reg value
+                if (file[:length] > 0)
+                    begin
+                        chunk = full_read(name)
+                        size=chunk.size
+                    rescue Exception => e
+                        @logger.error("Failed to read #{name} because of: #{e.message} .. will continue, set file as read and pretend this never happened")
+                        @logger.error("#{size} size and #{file[:length]} file length")
+                        size = file[:length]
+                    end 
+                else
+                    @logger.info("found a zero size file #{name}")
+                    chunk = nil 
                 end
             else
                 chunk = partial_read_json(name, file[:offset], file[:length])
                 @logger.debug("partial file #{name} from #{file[:offset]} to #{file[:length]}")
             end
             if logtype == "nsgflowlog" && @is_json
+              # skip empty chunks
+              unless chunk.nil?
                 res = resource(name)
                 begin
 		    fingjson = JSON.parse(chunk)
@@ -265,6 +259,7 @@ def run(queue)
                 rescue JSON::ParserError
                     @logger.error("parse error on #{res[:nsg]} [#{res[:date]}] offset: #{file[:offset]} length: #{file[:length]}")
                 end
+              end
             # TODO: Convert this to line based grokking.
             # TODO: ECS Compliance?
             elsif logtype == "wadiis" && !@is_json
@@ -272,7 +267,7 @@ def run(queue)
             else
                 counter = 0
                 begin
-                  @codec.decode(chunk) do |event|
+                    @codec.decode(chunk) do |event|
                     counter += 1
                     if @addfilename
                       event.set('filename', name)
@@ -282,6 +277,7 @@ def run(queue)
                   end
                 rescue Exception => e
                     @logger.error("codec exception: #{e.message} .. will continue and pretend this never happened")
+                    @registry.store(name, { :offset => file[:length], :length => file[:length] })
                     @logger.debug("#{chunk}")
                 end
                 @processed += counter
@@ -321,8 +317,54 @@ end
 
 
 private
+def connect
+    # Try in this order to access the storageaccount
+    # 1. storageaccount / sas_token
+    # 2. connection_string
+    # 3. storageaccount / access_key
+
+    unless connection_string.nil?
+        conn = connection_string.value
+    end
+    unless sas_token.nil?
+        unless sas_token.value.start_with?('?')
+            conn = "BlobEndpoint=https://#{storageaccount}.#{dns_suffix};SharedAccessSignature=#{sas_token.value}"
+        else
+            conn = sas_token.value
+        end
+    end
+    unless conn.nil?
+        @blob_client = Azure::Storage::Blob::BlobService.create_from_connection_string(conn)
+    else
+#        unless use_development_storage?
+        @blob_client = Azure::Storage::Blob::BlobService.create(
+            storage_account_name: storageaccount,
+            storage_dns_suffix: dns_suffix,
+            storage_access_key: access_key.value,
+        )
+#        else
+#            @logger.info("not yet implemented")
+#        end
+    end
+end
+
 def full_read(filename)
-    return @blob_client.get_blob(container, filename)[1]
+    tries ||= 2
+    begin
+        return @blob_client.get_blob(container, filename)[1]
+    rescue Exception => e
+        @logger.error("caught: #{e.message} for full_read")
+        if (tries -= 1) > 0
+           if e.message = "Connection reset by peer"
+               connect
+           end
+           retry
+        end
+    end
+    begin
+        chuck = @blob_client.get_blob(container, filename)[1]
+    end
+    return chuck
 end
 
 def partial_read_json(filename, offset, length)
@@ -347,6 +389,7 @@ end
 
 def nsgflowlog(queue, json, name)
     count=0
+    begin
     json["records"].each do |record|
       res = resource(record["resourceId"])
       resource = { :subscription => res[:subscription], :resourcegroup => res[:resourcegroup], :nsg => res[:nsg] }
@@ -376,6 +419,9 @@ def nsgflowlog(queue, json, name)
           end
       end
     end
+    rescue Exception => e
+      @logger.error("NSG Flowlog problem for #{name}, with #{json["records"].size} records and error message #{e.message}")
+    end
     return count 
 end
 
@@ -473,9 +519,10 @@ end
 
 
 def learn_encapsulation
+  @logger.info("learn_encapsulation, this can be skipped by setting skip_learning => true. Or set both head_file and tail_file")
     # From one file, read first block and last block to learn head and tail
     begin
-        blobs = @blob_client.list_blobs(container, { maxresults: 3, prefix: @prefix})
+        blobs = @blob_client.list_blobs(container, { max_results: 3, prefix: @prefix})
         blobs.each do |blob|
             unless blob.name == registry_path
               begin

data/logstash-input-azure_blob_storage.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-input-azure_blob_storage'
-  s.version       = '0.11.5'
+  s.version       = '0.12.1'
   s.licenses      = ['Apache-2.0']
   s.summary       = 'This logstash plugin reads and parses data from Azure Storage Blobs.'
   s.description   = <<-EOF
@@ -22,6 +22,6 @@ EOF
   # Gem dependencies
   s.add_runtime_dependency 'logstash-core-plugin-api', '~> 2.1'
   s.add_runtime_dependency 'stud', '~> 0.0.23'
-  s.add_runtime_dependency 'azure-storage-blob', '~> 1.1'
+  s.add_runtime_dependency 'azure-storage-blob', '~> 2', '>= 2.0.3'
   #s.add_development_dependency 'logstash-devutils', '~> 2'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-input-azure_blob_storage
 version: !ruby/object:Gem::Version
-  version: 0.11.5
+  version: 0.12.1
 platform: ruby
 authors:
 - Jan Geertsma
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-12-19 00:00:00.000000000 Z
+date: 2021-12-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -17,8 +17,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '2.1'
   name: logstash-core-plugin-api
-  type: :runtime
   prerelease: false
+  type: :runtime
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
@@ -31,8 +31,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: 0.0.23
   name: stud
-  type: :runtime
   prerelease: false
+  type: :runtime
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
@@ -43,15 +43,21 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.1'
+        version: '2'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.3
   name: azure-storage-blob
-  type: :runtime
   prerelease: false
+  type: :runtime
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.1'
+        version: '2'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.3
 description: " This gem is a Logstash plugin. It reads and parses data from Azure\
   \ Storage Blobs. The azure_blob_storage is a reimplementation to replace azureblob\
   \ from azure-diagnostics-tools/Logstash. It can deal with larger volumes and partial\
@@ -92,7 +98,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.6
+rubygems_version: 3.1.6
 signing_key:
 specification_version: 4
 summary: This logstash plugin reads and parses data from Azure Storage Blobs.