logstash-input-vespa 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 65e09408c1ab044377bbcda59b2a201e1b05f7e599d69277b292922e2129449f
-  data.tar.gz: 5aa868b54f814c5425dcd5c009ddd99753d0e00a6c214558b877a05cfe643bc0
+  metadata.gz: 5685d4ff93454cd5bb00019d87ac57e65ec96f80a27aba4e2a2a417b631250f0
+  data.tar.gz: 3456a806bc19ee7b64c64521719562e944aaa2eb2ea2c7c1524848f9659eabc7
 SHA512:
-  metadata.gz: 9d91224c4efd9b85c5a3ea9fd1d0256072c77d83e4f7013ac643e0e140024e880e4006535c1063c083978494be5b904218681a89515b028ed1cb357c0b3d9d99
-  data.tar.gz: 3c0b794ddc12bb622a49a393e97c4209007e7f568c953f09185f072f99b2ef09fca22d49f30ce2ca0d577335e7265fc5d649d52e5dbca7ebccefd76e2b17d704
+  metadata.gz: 8643c2ea6899c96047806b23d710278eb50fe76dca5b6a18cd428a4e2474307b99fe17383a8718efe6b44c5ccd535e57f01f87e600508a3316feafdfca8d5d98
+  data.tar.gz: 91e734d688e73b662cab9eea66c685ade66837007f0245d4a498ca1727d13ce95d3f06464634f6cf2da1b6a8e0b86e9cac8ca7a4a3e02312fd7b2b510f05a486

data/CHANGELOG.md

@@ -1,14 +1,4 @@
-## 2.0.7
-  - Docs: Set the default_codec doc attribute.
+## 0.1.0
 
-## 2.0.6
- - Docs: Add documentation template
-## 2.0.4
- - Add encoding: utf-8 line to spec file.  This can prevent issues with tests.
-## 2.0.1
- - Simplify the shutdown implementation a bit for easier understanding
-## 2.0.0
- - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
-   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
- - Dependency on logstash-core update to 2.0
+Initial version. Can connect to an HTTP endpoint of Vespa and pull documents via Visit and use the continuation token
 

data/README.md

@@ -1,98 +1,26 @@
-# Logstash Plugin
+# Logstash Input Plugin for Vespa
 
-[![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-input-example.svg)](https://travis-ci.com/logstash-plugins/logstash-input-example)
+Plugin for [Logstash](https://github.com/elastic/logstash) to read from [Vespa](https://vespa.ai). Apache 2.0 license.
 
-This is a plugin for [Logstash](https://github.com/elastic/logstash).
-
-It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
-
-## Documentation
-
-Logstash provides infrastructure to automatically build documentation for this plugin. We provide a template file, index.asciidoc, where you can add documentation. The contents of this file will be converted into html and then placed with other plugin documentation in a [central location](http://www.elastic.co/guide/en/logstash/current/).
-
-- For formatting config examples, you can use the asciidoc `[source,json]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
-
-## Need Help?
-
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
-
-## Developing
-
-### 1. Plugin Developement and Testing
-
-#### Code
-- To get started, you'll need JRuby with the Bundler gem installed.
-
-- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
-
-- Install dependencies
-```sh
-bundle install
-```
-
-#### Test
-
-- Update your dependencies
-
-```sh
-bundle install
-```
-
-- Run tests
-
-```sh
-bundle exec rspec
-```
-
-### 2. Running your unpublished Plugin in Logstash
-
-#### 2.1 Run in a local Logstash clone
-
-- Edit Logstash `Gemfile` and add the local plugin path, for example:
-```ruby
-gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
-```
-- Install plugin
-```sh
-# Logstash 2.3 and higher
-bin/logstash-plugin install --no-verify
-
-# Prior to Logstash 2.3
-bin/plugin install --no-verify
+## Installation
 
+Download and unpack/install Logstash, then:
 ```
-- Run Logstash with your plugin
-```sh
-bin/logstash -e 'filter {awesome {}}'
+bin/logstash-plugin install logstash-input-vespa
 ```
-At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
 
-#### 2.2 Run in an installed Logstash
-
-You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
-
-- Build your plugin gem
-```sh
-gem build logstash-filter-awesome.gemspec
-```
-- Install the plugin from the Logstash home
-```sh
-# Logstash 2.3 and higher
-bin/logstash-plugin install --no-verify
-
-# Prior to Logstash 2.3
-bin/plugin install --no-verify
+## Usage
 
+Minimal Logstash config example:
 ```
-- Start Logstash and proceed to test the plugin
-
-## Contributing
-
-All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
-
-Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
-
-It is more important to the community that you are able to contribute.
+input {
+  vespa {
+    vespa_url => "http://localhost:8080"
+    cluster => "test_cluster"
+  }
+}
 
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
+output {
+  stdout {}
+}
+```

data/lib/logstash/inputs/vespa.rb

@@ -4,6 +4,7 @@ require "logstash/namespace"
 require "net/http"
 require "uri"
 require "json"
+require "openssl"
 
 # This is the logstash vespa input plugin. It is used to read from Vespa
 # via Visit : https://docs.vespa.ai/en/reference/document-v1-api-reference.html#visit
@@ -21,28 +22,71 @@ class LogStash::Inputs::Vespa < LogStash::Inputs::Base
   # The cluster parameter to use in the request.
   config :cluster, :validate => :string, :required => true
 
+  # Path to the client certificate file for mTLS.
+  config :client_cert, :validate => :path
+
+  # Path to the client key file for mTLS.
+  config :client_key, :validate => :path
+
+  # desired page size for the visit request, i.e. the wantedDocumentCount parameter
+  config :page_size, :validate => :number, :default => 100
+
+  # backend concurrency for the visit request, i.e. the concurrency parameter
+  config :backend_concurrency, :validate => :number, :default => 1
+
+  # selection. A query in Vespa selector language
+  config :selection, :validate => :string
+
+  # timeout for each HTTP request
+  config :timeout, :validate => :number, :default => 180
+
+  # lower timestamp limit for the visit request, i.e. the fromTimestamp parameter
+  # microseconds since epoch
+  config :from_timestamp, :validate => :number
+
+  # upper timestamp limit for the visit request, i.e. the toTimestamp parameter
+  config :to_timestamp, :validate => :number
+
   public
   def register
-    # nothing to do here
+    if @client_cert != nil
+      @cert = OpenSSL::X509::Certificate.new(File.read(@client_cert))
+    end
+    if @client_key != nil
+      @key = OpenSSL::PKey::RSA.new(File.read(@client_key))
+    end
+
+    if @client_cert.nil? ^ @client_key.nil?
+      raise LogStash::ConfigurationError, "Both client_cert and client_key must be set, you can't have just one"
+    end
+    
+    @uri_params = {
+      :cluster => @cluster,
+      :wantedDocumentCount => @page_size,
+      :concurrency => @backend_concurrency,
+      :timeout => @timeout
+    }
+
+    if @selection != nil
+      @uri_params[:selection] = @selection
+    end
+
+    if @from_timestamp != nil
+      @uri_params[:fromTimestamp] = @from_timestamp
+    end
+
+    if @to_timestamp != nil
+      @uri_params[:toTimestamp] = @to_timestamp
+    end
   end # def register
 
   def run(queue)
-    uri = URI.parse("#{@vespa_url}/document/v1/?cluster=#{@cluster}")
+    uri = URI.parse("#{@vespa_url}/document/v1/")
+    uri.query = URI.encode_www_form(@uri_params)
     continuation = nil
 
     loop do
-
-      if continuation != nil
-        uri.query = URI.encode_www_form({:cluster => @cluster, :continuation => continuation})
-      end
-      
-      # TODO we'll want to retry here
-      begin
-        response = Net::HTTP.get_response(uri)
-      rescue => e
-        @logger.error("Failed to make HTTP request to Vespa", :error => e.message)
-        break
-      end
+      response = fetch_documents_from_vespa(uri)
       # response should look like:
       # {
       #   "pathId":"/document/v1/","documents":[
@@ -52,29 +96,31 @@ class LogStash::Inputs::Vespa < LogStash::Inputs::Base
       # }
 
       if response.is_a?(Net::HTTPSuccess)
-        begin
-          response_parsed = JSON.parse(response.body)
-        rescue JSON::ParserError => e
-          @logger.error("Failed to parse JSON response", :error => e.message)
-          break
-        end
+        response_parsed = parse_response(response)
+        break unless response_parsed
 
         document_count = response_parsed["documentCount"]
         # record the continuation token for the next request (if it exists)
         continuation = response_parsed["continuation"]
         documents = response_parsed["documents"]
 
-        documents.each do |document|
-          event = LogStash::Event.new(document["fields"])
-          decorate(event)
-          queue << event
-        end # documents.each
+        process_documents(documents, queue)
 
         # Exit the loop if there are no more documents to process
-        break if document_count == 0
+        if continuation != nil
+          uri.query = URI.encode_www_form(@uri_params.merge({:continuation => continuation}))
+        else
+          @logger.info("No continuation ID => no more documents to fetch from Vespa")
+          break
+        end
+
+        if @stopping
+          @logger.info("Stopping Vespa input")
+          break
+        end
 
       else
-        @logger.error("Failed to fetch documents from Vespa",
+        @logger.error("Failed to fetch documents from Vespa", :request => uri.to_s,
                       :response_code => response.code, :response_message => response.message)
         break # TODO retry? Only on certain codes?
       end # if response.is_a?(Net::HTTPSuccess)
@@ -82,11 +128,38 @@ class LogStash::Inputs::Vespa < LogStash::Inputs::Base
     end # loop do
   end # def run
 
+  def fetch_documents_from_vespa(uri)
+    http = Net::HTTP.new(uri.host, uri.port)
+    if uri.scheme == "https"
+      http.use_ssl = true
+      http.cert = @cert
+      http.key = @key
+      http.verify_mode = OpenSSL::SSL::VERIFY_PEER
+    end
+
+    request = Net::HTTP::Get.new(uri.request_uri)
+    http.request(request)
+  rescue => e
+      @logger.error("Failed to make HTTP request to Vespa", :error => e.message)
+      nil
+  end # def fetch_documents_from_vespa
+
+  def parse_response(response)
+    JSON.parse(response.body)
+  rescue JSON::ParserError => e
+    @logger.error("Failed to parse JSON response", :error => e.message)
+    nil
+  end # def parse_response
+
+  def process_documents(documents, queue)
+    documents.each do |document|
+      event = LogStash::Event.new(document)
+      decorate(event)
+      queue << event
+    end
+  end # def process_documents
+
   def stop
-    # TODO
-    # examples of common "stop" tasks:
-    #  * close sockets (unblocking blocking reads/accepts)
-    #  * cleanup temporary files
-    #  * terminate spawned threads
+    @stopping = true
   end
 end # class LogStash::Inputs::Vespa

data/logstash-input-vespa.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name = 'logstash-input-vespa'
-  s.version         = '0.1.0'
+  s.version         = '0.2.0'
   s.licenses = ['Apache-2.0']
   s.summary = "Logstash input plugin reading from Vespa"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-input-vespa
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Radu Gheorghe
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-08-28 00:00:00.000000000 Z
+date: 2024-08-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -86,7 +86,6 @@ files:
 - README.md
 - lib/logstash/inputs/vespa.rb
 - logstash-input-vespa.gemspec
-- spec/inputs/vespa_spec.rb
 homepage: https://vespa.ai
 licenses:
 - Apache-2.0
@@ -112,5 +111,4 @@ rubygems_version: 3.3.26
 signing_key:
 specification_version: 4
 summary: Logstash input plugin reading from Vespa
-test_files:
-- spec/inputs/vespa_spec.rb
+test_files: []

data/spec/inputs/vespa_spec.rb

@@ -1,12 +0,0 @@
-# encoding: utf-8
-require "logstash/devutils/rspec/spec_helper"
-require "logstash/devutils/rspec/shared_examples"
-require "logstash/inputs/vespa"
-
-describe LogStash::Inputs::Vespa do
-
-  it_behaves_like "an interruptible input plugin" do
-    let(:config) { { "interval" => 100 } }
-  end
-
-end