logstash-output-azure_loganalytics 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 960c239563a176eb66e1b6c48b6dac8c5edd2df5
+  data.tar.gz: a89509bfd7dcf3f3211882de0009f37778b0d997
+SHA512:
+  metadata.gz: 7fe84eace2a157cb4c00730f666d0679bd923ccccea515967f0b440dc413d59a5ace185fef46865b34741ef619d498b17003349d5414318b1e3e07445da4757d
+  data.tar.gz: b53d6da49d70b80877c12d87d6af8adfe98a20ba92d3c1e692e87e898a9699381594b8c5c0c76f06a90b7c9fd023f26d31ed8398a05a846d9346bf98eaeb0415

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+## 0.1.0
+
+* Initial Release

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Yoichi Kawasaki (yokawasa)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,119 @@
+# Azure Log Analytics output plugin for Logstash 
+logstash-output-azure_loganalytics is a logstash plugin to output to Azure Log Analytics. [Logstash](https://www.elastic.co/products/logstash) is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite [destinations](https://www.elastic.co/products/logstash). [Log Analytics](https://azure.microsoft.com/en-us/services/log-analytics/) is a service in Operations Management Suite (OMS) that helps you collect and analyze data generated by resources in your cloud and on-premises environments. It gives you real-time insights using integrated search and custom dashboards to readily analyze millions of records across all of your workloads and servers regardless of their physical location. The plugin stores in-coming events to Azure Log Analytics by leveraging [Log Analytics HTTP Data Collector API](https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api)
+
+## Installation
+
+You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:
+```
+bin/plugin install logstash-output-azure_loganalytics
+# or
+bin/logstash-plugin install logstash-output-azure_loganalytics  (Newer versions of Logstash)
+```
+Please see [Logstash reference](https://www.elastic.co/guide/en/logstash/current/offline-plugins.html) for more information.
+
+## Configuration
+
+```
+output {
+    azure_loganalytics {
+        customer_id => "<OMS WORKSPACE ID>"
+        shared_key => "<CLIENT AUTH KEY>"
+        log_type => "<LOG TYPE NAME>"
+        key_names  => ['key1','key2','key3'..] ## list of Key names (array)
+        flush_items => <FLUSH_ITEMS_NUM>
+        flush_interval_time => <FLUSH INTERVAL TIME(sec)>
+    }
+}
+```
+
+ * **customer\_id (required)** - Your Operations Management Suite workspace ID
+ * **shared\_key (required)** - The primary or the secondary Connected Sources client authentication key.
+ * **log\_type (required)** - The name of the event type that is being submitted to Log Analytics. This must be only alpha characters. 
+ * **key\_names (optional)** - Default:[] (empty array). list of Key names in in-coming record to deliver.
+ * **flush_items (optional)** - Default 50. Max number of items to buffer before flushing (1 - 1000).
+ * **flush_interval_time (optional)** - Default 5. Max number of seconds to wait between flushes.
+
+## Tests
+
+Here is an example configuration where Logstash's event source and destination are configured as Apache2 access log and Azure Log Analytics respectively.
+
+### Example Configuration
+```
+input {
+    file {
+        path => "/var/log/apache2/access.log"
+        start_position => "beginning"
+    }
+}
+
+filter {
+    if [path] =~ "access" {
+        mutate { replace => { "type" => "apache_access" } }
+        grok {
+            match => { "message" => "%{COMBINEDAPACHELOG}" }
+        }
+    }
+    date {
+        match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
+    }
+}
+
+output {
+    azure_loganalytics {
+        customer_id => "818f7bbc-8034-4cc3-b97d-f068dd4cd659"
+        shared_key => "ppC5500KzCcDsOKwM1yWUvZydCuC3m+ds/2xci0byeQr1G3E0Jkygn1N0Rxx/yVBUrDE2ok3vf4ksXxcBmQQHw==(dummy)"
+        log_type => "ApacheAccessLog"
+        key_names  => ['logid','date','processing_time','remote','user','method','status','agent']
+        flush_items => 10
+        flush_interval_time => 5
+    }
+    # for debug
+    stdout { codec => rubydebug }
+}
+```
+
+You can find example configuration files in logstash-output-azure_loganalytics/examples.
+
+### Run the plugin with the example configuration
+
+Now you run logstash with the the example configuration like this:
+```
+# Test your logstash configuration before actually running the logstash
+bin/logstash -f logstash-apache2-to-loganalytics.conf --configtest
+# run
+bin/logstash -f logstash-apache2-to-loganalytics.conf
+```
+
+Here is an expected output for sample input (Apache2 access log):
+
+<u>Apache2 access log</u>
+```
+106.143.121.169 - - [29/Dec/2016:01:38:16 +0000] "GET /test.html HTTP/1.1" 304 179 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
+```
+
+<u>Output (rubydebug)</u>
+```
+{
+        "message" => "106.143.121.169 - - [29/Dec/2016:01:38:16 +0000] \"GET /test.html HTTP/1.1\" 304 179 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\"",
+       "@version" => "1",
+     "@timestamp" => "2016-12-29T01:38:16.000Z",
+           "path" => "/var/log/apache2/access.log",
+           "host" => "yoichitest01",
+           "type" => "apache_access",
+       "clientip" => "106.143.121.169",
+          "ident" => "-",
+           "auth" => "-",
+      "timestamp" => "29/Dec/2016:01:38:16 +0000",
+           "verb" => "GET",
+        "request" => "/test.html",
+    "httpversion" => "1.1",
+       "response" => "304",
+          "bytes" => "179",
+       "referrer" => "\"-\"",
+          "agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\""
+}
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/logstash-output-azure_loganalytics.

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/lib/logstash/outputs/azure_loganalytics.rb

@@ -0,0 +1,94 @@
+# encoding: utf-8
+require "logstash/outputs/base"
+require "logstash/namespace"
+require "stud/buffer"
+
+class LogStash::Outputs::AzureLogAnalytics < LogStash::Outputs::Base
+  include Stud::Buffer
+
+  config_name "azure_loganalytics"
+
+  # Your Operations Management Suite workspace ID
+  config :customer_id, :validate => :string, :required => true
+
+  # The primary or the secondary Connected Sources client authentication key
+  config :shared_key, :validate => :string, :required => true
+
+  # The name of the event type that is being submitted to Log Analytics. This must be only alpha characters.
+  config :log_type, :validate => :string, :required => true
+
+  # list of Key names in in-coming record to deliver.
+  config :key_names, :validate => :array, :default => []
+  
+  # Max number of items to buffer before flushing. Default 50.
+  config :flush_items, :validate => :number, :default => 50
+  
+  # Max number of seconds to wait between flushes. Default 5
+  config :flush_interval_time, :validate => :number, :default => 5
+
+  public
+  def register
+    require 'azure/loganalytics/datacollectorapi/client'
+
+    ## Configure
+    if not @log_type.match(/^[[:alpha:]]+$/)
+      raise ArgumentError, 'log_type must be only alpha characters' 
+    end
+
+    ## Start 
+    @client=Azure::Loganalytics::Datacollectorapi::Client::new(@customer_id,@shared_key)
+
+    buffer_initialize(
+      :max_items => @flush_items,
+      :max_interval => @flush_interval_time,
+      :logger => @logger
+    )
+
+  end # def register
+
+  public
+  def receive(event)
+    # Simply save an event for later delivery
+    buffer_receive(event)
+  end # def receive
+
+  # called from Stud::Buffer#buffer_flush when there are events to flush
+  public
+  def flush (events, close=false)
+  
+    documents = []  #this is the array of hashes to add Azure Log Analytics
+    events.each do |event|
+      document = {}
+      event_hash = event.to_hash()
+      if @key_names.length > 0
+        @key_names.each do |key|
+          if event_hash.include?(key)
+            document[key] = event_hash[key]
+          end
+        end
+      else
+        document = event_hash
+      end
+      # Skip if document doesn't contain any items
+      next if (document.keys).length < 1
+
+      documents.push(document)
+    end
+
+    # Skip in case there are no candidate documents to deliver
+    if documents.length < 1
+      return
+    end
+
+    begin
+      res = @client.post_data(@log_type, documents)
+      if not Azure::Loganalytics::Datacollectorapi::Client.is_success(res)
+        $logger.error("DataCollector API request failure: error code: #{res.code}, data=>" + (documents.to_json).to_s)
+      end
+    rescue Exception => ex
+      $logger.error("Exception occured in posting to DataCollector API: '#{ex}', data=>" + (documents.to_json).to_s)
+    end
+
+  end # def flush
+
+end # class LogStash::Outputs::AzureLogAnalytics

data/logstash-output-azure_loganalytics.gemspec

@@ -0,0 +1,26 @@
+Gem::Specification.new do |s|
+  s.name = 'logstash-output-azure_loganalytics'
+  s.version    =  File.read("VERSION").strip
+  s.authors = ["Yoichi Kawasaki"]
+  s.email = "yoichi.kawasaki@outlook.com"
+  s.summary = %q{logstash output plugin to store events into Azure Log Analytics}
+  s.description = s.summary
+  s.homepage = "http://github.com/yokawasa/logstash-output-azure_loganalytics"
+  s.licenses = ["Apache License (2.0)"]
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT', 'VERSION']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "rest-client"
+  s.add_runtime_dependency "azure-loganalytics-datacollector-api", ">= 0.1.2"
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0", "< 3.0.0"
+  s.add_runtime_dependency "logstash-codec-plain"
+  s.add_development_dependency "logstash-devutils"
+end

data/spec/outputs/azure_loganalytics_spec.rb

@@ -0,0 +1,68 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/outputs/azure_loganalytics"
+require "logstash/codecs/plain"
+require "logstash/event"
+
+describe LogStash::Outputs::AzureLogAnalytics do
+
+  let(:customer_id) { '<Customer ID aka WorkspaceID String>' }
+  let(:shared_key) { '<Primary Key String>' }
+  let(:log_type) { 'ApacheAccessLog' }
+  let(:key_names) { ['logid','date','processing_time','remote','user','method','status','agent'] }
+
+  let(:azure_loganalytics_config) {
+    { 
+      "customer_id" => customer_id, 
+      "shared_key" => shared_key,
+      "log_type" => log_type,
+      "key_names" => key_names
+    }
+  }
+
+  let(:azure_loganalytics_output) { LogStash::Outputs::AzureLogAnalytics.new(azure_loganalytics_config) }
+
+  before do
+     azure_loganalytics_output.register
+  end 
+
+  describe "#flush" do
+    it "Should successfully send the event to Azure Log Analytics" do
+      events = []
+      log1 = {
+        :logid => "5cdad72f-c848-4df0-8aaa-ffe033e75d57",
+        :date => "2016-12-10 09:44:32 JST",
+        :processing_time => "372",
+        :remote => "101.202.74.59",
+        :user => "-",
+        :method => "GET / HTTP/1.1",
+        :status => "304",
+        :size => "-",
+        :referer => "-",
+        :agent => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:27.0) Gecko/20100101 Firefox/27.0"
+      }
+
+      log2 = {
+        :logid => "7260iswx-8034-4cc3-uirtx-f068dd4cd659",
+        :date => "2016-12-10 09:45:14 JST",
+        :processing_time => "105",
+        :remote => "201.78.74.59",
+        :user => "-",
+        :method => "GET /manager/html HTTP/1.1",
+        :status =>"200",
+        :size => "-",
+        :referer => "-",
+        :agent => "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
+      }
+
+      event1 =  LogStash::Event.new(log1) 
+      event2 =  LogStash::Event.new(log2) 
+      azure_loganalytics_output.receive(event1)
+      azure_loganalytics_output.receive(event2)
+      events.push(event1)
+      events.push(event2)
+      expect {azure_loganalytics_output.flush(events)}.to_not raise_error
+    end
+  end
+
+end

metadata

@@ -0,0 +1,131 @@
+--- !ruby/object:Gem::Specification
+name: logstash-output-azure_loganalytics
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Yoichi Kawasaki
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2016-12-29 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: rest-client
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.1.2
+  name: azure-loganalytics-datacollector-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.1.2
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  name: logstash-core
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: logstash output plugin to store events into Azure Log Analytics
+email: yoichi.kawasaki@outlook.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- README.md
+- VERSION
+- lib/logstash/outputs/azure_loganalytics.rb
+- logstash-output-azure_loganalytics.gemspec
+- spec/outputs/azure_loganalytics_spec.rb
+homepage: http://github.com/yokawasa/logstash-output-azure_loganalytics
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: output
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
+specification_version: 4
+summary: logstash output plugin to store events into Azure Log Analytics
+test_files:
+- spec/outputs/azure_loganalytics_spec.rb