logstash-output-redisearch 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 60a5ad2c1a084053f9275022336e2e32620c08ff5a44041743e7a3b4860bf063
+  data.tar.gz: 8769d6da6888847da02eab1ef5972d0015a0567ec4246b73d1380fb58039fbac
+SHA512:
+  metadata.gz: df4d0505f0e869b38fcb74de6749ff86e81a483a57d9eab620515bd1494277ede9b6115ca1a3adaafaaa07f46d8aae4191c0253e2a732974b29446da98b2ad21
+  data.tar.gz: d43c3d54cf4363fb1a4a25a102a4a21f55b1b6d83ec94b80d4f567bdfc676269b3ef351f717212f942e0749e0aee6f050c11a84bb1ac8c87bfe930c5ee944e46

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.0
+  - Plugin created with the logstash plugin generator

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+*  - HashedIn
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-output-redisearch
+Example output plugin. This should help bootstrap your effort to write your own output plugin!

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,181 @@
+# Logstash Output Plugin
+
+This logstash output plugin is for Redisearch.
+- Note: Plugin has not been tested with cluster mode.
+
+### 1. Plugin Development and Testing
+
+#### Requirements
+* JRuby (Use Ruby Version Manger(RVM))
+* JDK
+* Git
+* bundler
+* Redisearch
+* Logstash
+
+#### Install requirements
+* RVM
+```bash
+gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
+\curl -sSL https://get.rvm.io | bash
+source ~/.rvm/scripts/rvm
+```
+* JRuby
+```bash
+rvm install jruby
+```
+
+* JDK
+```bash
+sudo apt install default-jdk
+echo "export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which javac) )))" >> ~/.profile
+source ~/.profile
+```
+
+* bundler
+```bash
+gem install bundler
+```
+
+* Redisearch
+```bash
+git clone --recursive https://github.com/RediSearch/RediSearch.git
+make build
+make run
+```
+
+#### Code
+
+- Clone Project
+```bash
+git clone https://github.com/hashedin/logstash-output-redisearch.git
+``` 
+- Use JRuby
+```bash
+rvm use jruby
+```
+
+- Install dependencies
+```bash
+bundle install
+```
+
+#### Test
+
+- Run tests
+
+```bash
+bundle exec rspec
+```
+
+### 2. Running your Plugin in Logstash
+
+* Install Logstash
+```bash
+wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
+sudo apt-get install apt-transport-https
+echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
+sudo apt-get update && sudo apt-get install logstash
+sudo /usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd
+```
+
+#### Run in a local Logstash
+
+- Build Gemfile
+
+```bash
+gem build logstash-output-redisearch.gemspec
+```
+
+- Deploy Gemfile to Logstash
+
+```bash
+bin/logstash-plugin install /path/to/logstash-output-redisearch-0.1.0.gem
+```
+
+- Verify installed plugin
+```bash
+bin/logstash-plugin list
+There should be logstash-output-redisearch
+```
+#### Start logstash output plugin
+
+- Configuration options
+
+| Name | Description | Type | Default | 
+| --- | --- | --- | --- |
+| host | Redis-server IP address | string | "127.0.0.1" | 
+| port | Redis-server port number | number | 6379 |
+| index | Name an index in redisearch | string | "logstash-current-date" |
+| batch_events | Max number of events in a buffer before flush | number | 50 |
+| batch_timeout | Max interval to pass before flush | number | 5 |
+| ssl | SSL authentication | boolean | false |
+| password | Password for authentication | password | - |
+
+* Usage
+```bash
+output {
+    redisearch {
+   }
+}
+```
+OR
+
+```bash
+output {
+    redisearch {
+        host => '192.168.0.1'
+        port => 6379
+        index => logstash
+        batch_events => 20
+        batch_timeout => 2
+        ssl => true
+        password => "123"
+    }
+}
+```
+
+#### Example
+
+Let's create a logstash pipleline using filebeat as input plugin and redisearch as output plugin:
+1. Install filebeat and configure /etc/filebeat/filebeat.yml as following:
+- Install filebeat:
+```bash
+sudo apt-get install filebeat 
+```
+- Enable filebeat input to read from file:
+```bash
+filebeat.inputs:
+	enabled: true
+	paths:
+          -  /path/to/logfile
+```
+
+- Change filebeat output from elasticsearch to logstash:
+```bash
+output.logstash:
+	hosts: [“localhost:5044”]
+```
+
+2. Create a conf file in /etc/logstash/conf.d
+```bash
+input {
+	beats {
+		Port => 5044
+	}
+output {
+	redisearch {
+}
+}
+```
+
+3. After configuring, restart logstash and filebeat services and check the data stashing into redisearch.
+```bash 
+sudo service logstash restart
+sudo service filebeat restart
+```
+
+#### References
+
+* https://github.com/logstash-plugins/logstash-output-redis : Redis Output Plugin
+* https://github.com/logstash-plugins/logstash-output-elasticsearch : Elasticsearch Output Plugin

data/lib/logstash/outputs/redisearch.rb

@@ -0,0 +1,118 @@
+# encoding: utf-8
+# require "logstash/outputs/template"
+require "logstash/outputs/template"
+require "logstash/outputs/base"
+require 'redisearch-rb'
+require "stud/buffer"
+require 'json'
+require 'time'
+
+# An redisearch output will store data into Redisearch.
+class LogStash::Outputs::Redisearch < LogStash::Outputs::Base
+  include Stud::Buffer
+  config_name "redisearch"
+  default :codec, "json"
+
+  # Hostname of rediserver to connect to rediserver.
+  config :host, :validate => :string, :default => "127.0.0.1"
+  
+  # Port number to connect to rediserver
+  config :port, :validate => :number, :default => 6379
+
+  # Index name to create or to connect the existing old spec
+  config :index, :validate => :string, :default => nil
+
+  # Interval to reconnect if Failure in redis connection 
+  config :reconnect_interval, :validate => :number, :default => 1
+
+  # Max number of events to add in a list 
+  config :batch_events, :validate => :number, :default => 50
+
+  # Max interval to pass before flush 
+  config :batch_timeout, :validate => :number, :default => 5
+
+  # SSL flag for aunthentication
+  config :ssl, :validate => :boolean, :default => false
+
+  # Password for aunthentication
+  config :password, :validate => :password
+
+  # Method is a constructor to this class. 
+  # Used to intialize buffer, redisearch client and also to create a index if it is not present
+  public
+  def register
+    
+    buffer_initialize(
+      :max_items => @batch_events,
+      :max_interval => @batch_timeout,
+    )
+
+    params = {
+      "host"=>@host,
+      "port"=>@port,
+      "index"=>@index,
+      "ssl"=>@ssl
+    }
+
+    if @password
+      params["password"] = @password.value  
+    end
+
+    @idx = Index.new(params)
+    @redisearch_client = @idx.connect()
+    @codec.on_event(&method(:send_to_redisearch))
+  
+  end # def register
+
+  # Method is to receive event and encode it in json format. 
+  public
+  def receive(event)
+    begin
+      @codec.encode(event)
+    rescue StandardError => e
+      @logger.warn("Error encoding event", :exception => e,
+                   :event => event)
+      sleep @reconnect_interval
+      retry
+    end
+  end # def event
+ 
+  # Method is called from Stud::Buffer when max_items/max_interval is reached
+  def flush(events, close=false)
+    #buffer_flush should pass here the :final boolean value.
+    @redisearch_client.add_docs(events)
+    @logger.info("Buffer Inserted Successfully", :length => events.length)
+  end
+
+  # Method is called from Stud::Buffer when an error occurs
+  def on_flush_error(e)
+    @logger.warn("Failed to send backlog of events to Redisearch",
+      :exception => e,
+      :backtrace => e.backtrace
+    )
+  end
+
+  # Method is for final bookkeeping and cleanup when plugin thread exit
+  def close
+      # Force full flush call to ensure that all accumulated messages are flushed.
+      buffer_flush(:final => true)
+  end
+
+  # Method to assign uuid to each event (formatting event as per document required by redisearch)
+  # and to append each event to buffer 
+  def send_to_redisearch(event, payload)
+    begin
+      doc_data = JSON.parse(payload)
+      doc_id = @idx.get_id()
+      document = [doc_id,doc_data]
+      buffer_receive(document)
+
+    rescue => e
+      @logger.warn("Failed to send event to Redisearch", :event => event,
+                   :exception => e,
+                   :backtrace => e.backtrace)
+      sleep @reconnect_interval
+      retry
+    end
+  end
+end # class LogStash::Outputs::Redisearch

data/lib/logstash/outputs/template.json

@@ -0,0 +1,9 @@
+{
+    "index": "logstash-*",
+    "schema": { 
+        "@timestamp": ["TEXT", "SORTABLE"],
+        "host" : "TEXT", 
+        "@version": "TEXT",
+        "message": "TEXT"
+   }
+}

data/lib/logstash/outputs/template.rb

@@ -0,0 +1,65 @@
+require 'json'
+require 'redisearch-rb'
+require 'time'
+require 'redis'
+require 'securerandom'
+require 'base64'
+
+# Redisearch index management
+class Index
+    # initialize and create redis instance using params value
+    def initialize(params)
+        begin
+
+            @template_data = read_template()
+            @redis = Redis.new(
+                host: params["host"], 
+                port: params["port"],
+                ssl: params["ssl"], 
+                password:params["password"])
+            
+            if params["index"] == nil
+                time = Time.new
+                @idx_name = @template_data['index'].sub! '*', time.strftime("%Y-%m-%d")
+            else
+                @idx_name = params["index"]
+            end
+
+        rescue => e
+            @logger.debug("Exception in Index initialization",e)
+        end
+    end
+
+
+    # Reads json file and returns data
+    def read_template()
+        begin
+            filepath = ::File.expand_path('template.json', ::File.dirname(__FILE__))
+            file_data = ::IO.read(filepath)
+            data = JSON.load(file_data)
+        rescue => e
+            @logger.debug("Exception in reading template", e)
+        end
+        return data
+    end
+
+    # Creates redisearch instance using redis instance.
+    # Using redisearch instance, connects to index if it is present, else creates a new index
+    def connect()
+        begin
+            @rs = RediSearch.new(@idx_name,@redis)
+            @rs.info()
+        rescue
+            @schema = @template_data['schema']
+            @rs.create_index(@schema)
+        end
+        return @rs
+    end
+
+    # Id for each document in redisearch
+    def get_id()
+        uuid = SecureRandom.uuid
+        id = Base64.encode64([ uuid.tr('-', '') ].pack('H*')).gsub(/\=*\n/, '')
+        return id
+    end
+end #Class Index

data/logstash-output-redisearch.gemspec

@@ -0,0 +1,29 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-output-redisearch'
+  s.version       = '0.1.1'
+  s.licenses      = ['Apache-2.0']
+  s.summary       = 'Stash logs in Redisearch'
+  s.description   = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
+  s.homepage      = 'https://github.com/hashedin/logstash-output-redisearch'
+  s.authors       = ['HashedIn Technologies']
+  s.email         = 'redis-connectors@hashedin.com'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
+  s.add_runtime_dependency "logstash-codec-plain", '~> 3.0'
+  s.add_runtime_dependency 'logstash-codec-json', '~> 3.0'
+  s.add_runtime_dependency 'redis', '~> 4'
+  s.add_runtime_dependency 'redisearch-rb', '~> 1'
+  s.add_development_dependency 'logstash-devutils'
+  s.add_development_dependency 'flores'
+  
+end

data/spec/outputs/batch_spec.rb

@@ -0,0 +1,28 @@
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/outputs/redisearch"
+require "redisearch"
+
+describe LogStash::Outputs::Redisearch do
+
+  context "batch of events" do
+
+    let(:config) {
+      {
+        "batch_events" => 10,
+        "batch_timeout" => 1,
+       }
+    }
+    let(:redisearch) { described_class.new(config) }
+
+    it "should call buffer_receive" do
+      redisearch.register
+      expect(redisearch).to receive(:buffer_receive).exactly(100).times.and_call_original
+      expect(redisearch).to receive(:flush).exactly(10).times
+      expect(redisearch).not_to receive(:on_flush_error)
+
+      100.times do |i|
+        expect{redisearch.receive(LogStash::Event.new({"message" => "test-#{i}"}))}.to_not raise_error
+      end
+    end
+  end
+end

data/spec/outputs/redisearch_spec.rb

@@ -0,0 +1,47 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/outputs/redisearch"
+require "logstash/event"
+require "redisearch-rb"
+require "flores/random"
+require "redis"
+
+describe LogStash::Outputs::Redisearch do
+context "to check if events are inserted to redisearch" do
+  subject { described_class.new(config) }
+
+    options = {
+      "host" => "127.0.0.1",
+      "port" => 6379,
+      "index" => "test_idx",
+    }
+
+    let(:config) {
+      options
+    }
+    let(:event_count) { Flores::Random.integer(0..1000) }
+    let(:message) { Flores::Random.text(0..2) }
+    redis = Redis.new(host: options["host"], port: options["port"])
+    rs = RediSearch.new(options["index"],redis)
+
+    before do
+      subject.register
+      event_count.times do |i|
+        event = LogStash::Event.new("sequence" => i, "message" => message)
+        subject.receive(event)
+      end
+      subject.close
+    end
+
+    it "search for a text in redisearch" do
+        info = rs.search("message")
+        insist { info.count } == 0
+    end
+     
+    it "count number of docs in redisearch are same as number of events" do
+      info = rs.info()
+      insist { info['num_docs'].to_i } == event_count
+    end
+
+  end
+end

metadata

@@ -0,0 +1,159 @@
+--- !ruby/object:Gem::Specification
+name: logstash-output-redisearch
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- HashedIn Technologies
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2020-05-05 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  name: logstash-codec-json
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4'
+  name: redis
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1'
+  name: redisearch-rb
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: flores
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a Logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This
+  gem is not a stand-alone program
+email: redis-connectors@hashedin.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/outputs/redisearch.rb
+- lib/logstash/outputs/template.json
+- lib/logstash/outputs/template.rb
+- logstash-output-redisearch.gemspec
+- spec/outputs/batch_spec.rb
+- spec/outputs/redisearch_spec.rb
+homepage: https://github.com/hashedin/logstash-output-redisearch
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: output
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.7.10
+signing_key:
+specification_version: 4
+summary: Stash logs in Redisearch
+test_files:
+- spec/outputs/batch_spec.rb
+- spec/outputs/redisearch_spec.rb