logstash-output-vespa_feed 0.8.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: da41467fd259e14c680f84a9ab406885326deec8968af3fbaa12f6abcd3c35dc
-  data.tar.gz: 5fbe4d025c77730760f3103d19fd5d61c3249536b4947ea42fa5413fc727c845
+  metadata.gz: b3821846f444b171f16fdd64199d52bd4a6e9e16a0fcaeb70acc4e0e28eb72c1
+  data.tar.gz: 0e479e7f5e999ebd1d6099c85bbab3ea62259326ae31296e4e6f255d52007b52
 SHA512:
-  metadata.gz: 233f69ae4a1e2909c800b5c91e4181d9342046aea93c2c7ec423466cb861188b7ea01b1d569063143bc0c3e950f5542718f8586af380eaccb907c2d3c798d8d6
-  data.tar.gz: 885c8572e25dad3b1366cdd2747cbeef834e89bb1e9ced2f444b7bbce740de1db1ab5fa2b3c7e6d930686486570f289862aa487ccf132dfb410384834501dfa8
+  metadata.gz: 7bacdfd3c56daea32a1501ac69da32da04a98dd396819d7b257c3e046faad491b7ce4ae225f699e2afa8858c56b0f08b46e1b1c9a8c48d69c2b6b8b4d5c672d1
+  data.tar.gz: 9d9f5cbc881d448298c2f60e6667c0e4aa9b073744d57912ac7ddba4a91954de4ad2758a658ebc0bb87404e830096ec8e715346c41b0fa1c9524926b186e83bb

data/README.md

@@ -2,48 +2,99 @@
 
 Plugin for [Logstash](https://github.com/elastic/logstash) to write to [Vespa](https://vespa.ai). Apache 2.0 license.
 
-## Installation
-
-Download and unpack/install Logstash, then:
+## Table of Contents
+- [Quick start](#quick-start)
+- [Usage](#usage)
+  - [Mode 1: Generating an application package](#mode-1-generating-an-application-package)
+  - [Mode 2: Sending data to Vespa](#mode-2-sending-data-to-vespa)
+- [Development](#development)
+  - [Integration tests](#integration-tests)
+  - [Publishing the gem](#publishing-the-gem)
+
+## Quick start
+
+[Download and unpack/install Logstash](https://www.elastic.co/downloads/logstash), then install the plugin:
 ```
 bin/logstash-plugin install logstash-output-vespa_feed
 ```
 
-## Development
-If you're developing the plugin, you'll want to do something like:
+Write a Logstash config file (a couple of examples are below), then start Logstash, pointing it to the config file:
 ```
-# build the gem
-./gradlew gem
-# run tests
-./gradlew test
-# install it as a Logstash plugin
-/opt/logstash/bin/logstash-plugin install /path/to/logstash-output-vespa/logstash-output-vespa_feed-0.7.0.gem
-# profit
-/opt/logstash/bin/logstash
+bin/logstash -f logstash.conf
 ```
-Some more good info about Logstash Java plugins can be found [here](https://www.elastic.co/guide/en/logstash/current/java-output-plugin.html).
 
-It looks like the JVM options from [here](https://github.com/logstash-plugins/.ci/blob/main/dockerjdk17.env)
-are useful to make JRuby's `bundle install` work.
+## Usage
+There are two modes of operation:
+1. **Generating an application package**: this is useful if you're just getting started with Vespa and you don't have e.g. a schema yet.
+2. **Sending data to Vespa**: once you have an application package deployed, you can send data to Vespa.
 
-### Integration tests
-To run integration tests, you'll need to have a Vespa instance running with an app deployed that supports an "id" field. And Logstash installed.
+### Mode 1: generating an application package
 
-Check out the `integration-test` directory for more information.
+If you're just getting started with Vespa, you can use the `detect_schema` option to generate an application package that works with the data you're sending. That application package can be deployed to [Vespa Cloud](https://cloud.vespa.ai) or a local Vespa instance.
+
+To process the data, the `input` and `filter` sections are the same as when you send data to Vespa (see the [config example below](#mode-2-sending-data-to-vespa)). The output section is different. Here's an example for Vespa Cloud:
 
 ```
-cd integration-test
-./run_tests.sh
+output {
+  vespa_feed {
+    # enable detect schema mode
+    detect_schema => true
+    # to get copy-paste-able Vespa CLI commands
+    deploy_package => true
+
+    # Vespa Cloud application details
+    vespa_cloud_tenant => "my_tenant"
+    vespa_cloud_application => "my_application"
+
+    ### optional settings
+    # where to save the generated application package
+    application_package_dir => "/OS_TMPDIR/vespa_app"
+    # How long to wait (in empty batches) before showing the CLI commands for deploying the application
+    # This is useful if Logstash doesn't exit quickly and waits (e.g. when tailing a file)
+    # Otherwise, the plugin will show the CLI commands before Logstash exits
+    idle_batches => 10
+    # whether to generate mTLS certificates (defaults to true for Vespa Cloud, because you'll need them)
+    generate_mtls_certificates => true
+    # common name for the mTLS certificates
+    certificate_common_name => "cloud.vespa.logstash"
+    # validity days for the mTLS certificates
+    certificate_validity_days => 30
+    # where should the client certificate and key be saved
+    client_cert => "/OS_TMPDIR/vespa_app/security/clients.pem"
+    client_key => "/OS_TMPDIR/vespa_app/data-plane-private-key.pem"
+  }
+}
 ```
 
-### Publishing the gem
+For self-hosted Vespa, options are slightly different:
+```
+output {
+  vespa_feed {
+    # enable detect schema mode
+    detect_schema => true
+    # whether to actually deploy the application package
+    deploy_package => true
+
+    # config server endpoint (derived from vespa_url, which defaults to http://localhost:8080)
+    # used for deploying the application package, if deploy_package=true
+    config_server => "http://localhost:19071"
+
+    ### same optional settings as for Vespa Cloud
+    ### exception: generate_mtls_certificates defaults to false
+  }
+}
+```
 
-Note to self: for some reason, `bundle exec rake publish_gem` fails, but `gem push logstash-output-vespa_feed-$VERSION.gem`
-does the trick.
+In the end, you should have an application package in `application_package_dir`, which defaults to your OS's temp directory + `vespa_app`. We encourage you to check it out and change it as needed. If Logstash didn't already deploy it, you can do so with the [Vespa CLI](https://docs.vespa.ai/en/vespa-cli.html):
+```
+cd /path/to/application_package
+### show deployment logs for up to 15 minutes
+vespa deploy --wait 900
+```
 
-## Usage
+### Mode 2: sending data to Vespa
 
-Some more Logstash config examples can be found [in this blog post](https://blog.vespa.ai/logstash-vespa-tutorials/), but here's one with all the output plugin options:
+Some more Logstash config examples can be found [in this blog post](https://blog.vespa.ai/logstash-vespa-tutorials/), but here's one with all the relevant output options:
 
 ```
 # read stuff
@@ -94,16 +145,20 @@ output {
     vespa_url => "http://localhost:8080"
     
     # for HTTPS URLS (e.g. Vespa Cloud), you may want to provide a certificate and key for mTLS authentication
-    client_cert => "/home/radu/vespa_apps/myapp/security/clients.pem"
+    # the defaults are relative to application_package_dir (see the detect_schema section above)
+    client_cert => "/OS_TMPDIR/vespa_app/security/clients.pem"
     # make sure the key isn't password-protected
     # if it is, you can create a new key without a password like this:
     # openssl rsa -in myapp_key_with_pass.pem -out myapp_key.pem
-    client_key => "/home/radu/vespa_apps/myapp_key.pem"
+    client_key => "/OS_TMPDIR/vespa_app/data-plane-private-key.pem"
+
+    # for Vespa Cloud, you can use an auth token instead of mTLS certificates
+    auth_token => "vespa_cloud_TOKEN_GOES_HERE"
     
     # namespace could be static or in the %{field} format, picking from a field in the document
-    namespace => "no_default_provide_yours"
+    namespace => "defaults_to_the_document_type_value"
     # similarly, doc type could be static or in the %{field} format
-    document_type => "no_default_provide_yours_from_schema"
+    document_type => "doctype"
     
     # operation can be "put", "update", "remove" or dynamic (in the %{field} format)
     operation => "put"
@@ -155,7 +210,34 @@ output {
 }
 ```
 
-Then you can start Logstash while pointing to the config file like:
+## Development
+If you're developing the plugin, you'll want to do something like:
 ```
-bin/logstash -f logstash.conf
+# build the gem
+./gradlew gem
+# run tests
+./gradlew test
+# install it as a Logstash plugin
+/opt/logstash/bin/logstash-plugin install /path/to/logstash-output-vespa/logstash-output-vespa_feed-1.0.0.gem
+# profit
+/opt/logstash/bin/logstash
+```
+Some more good info about Logstash Java plugins can be found [here](https://www.elastic.co/guide/en/logstash/current/java-output-plugin.html).
+
+It looks like the JVM options from [here](https://github.com/logstash-plugins/.ci/blob/main/dockerjdk17.env)
+are useful to make JRuby's `bundle install` work.
+
+### Integration tests
+To run integration tests, you'll need to have a Vespa instance running with an app deployed that supports an "id" field. And Logstash installed.
+
+Check out the `integration-test` directory for more information.
+
 ```
+cd integration-test
+./run_tests.sh
+```
+
+### Publishing the gem
+
+Note to self: for some reason, `bundle exec rake publish_gem` fails, but `gem push logstash-output-vespa_feed-$VERSION.gem`
+does the trick.

data/VERSION

@@ -1 +1 @@
-0.8.0
+1.0.0

data/docs/index.asciidoc

@@ -51,6 +51,26 @@ Writes documents to Vespa.
 | <<plugins-{type}s-{plugin}-max_streams>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-operation_timeout>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-grace_period>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-doom_period>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-auth_token>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-enable_dlq>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-dlq_path>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-max_queue_size>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-max_segment_size>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-flush_interval>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-detect_schema>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-application_package_dir>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-deploy_package>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-idle_batches>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-config_server>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-generate_mtls_certificates>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-certificate_common_name>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-certificate_validity_days>> |<<number,number>>|No
+| <<plugins-{type}s-{plugin}-type_mappings_file>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-type_conflict_resolution_file>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-vespa_cloud_tenant>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-vespa_cloud_application>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-vespa_cloud_instance>> |<<string,string>>|No
 |=======================================================================
 
 [id="plugins-{type}s-{plugin}-vespa_url"]
@@ -198,6 +218,15 @@ After this time (seconds), the circuit breaker will be half-open:
 it will ping the endpoint to see if it's back,
 then resume sending requests when it's back.
 
+[id="plugins-{type}s-{plugin}-doom_period"]
+===== `doom_period`
+
+* Value type is <<number,number>>
+* Default value is `60`
+
+After this time (seconds), if the connection is still broken, the connection will close
+and stop sending requests until the Vespa service comes back up.
+
 [id="plugins-{type}s-{plugin}-enable_dlq"]
 ===== `enable_dlq`
 
@@ -239,6 +268,124 @@ Maximum size of each Dead Letter Queue segment file in bytes.
 
 How often to commit the Dead Letter Queue to disk, in milliseconds.
 
+[id="plugins-{type}s-{plugin}-detect_schema"]
+===== `detect_schema`
+
+* Value type is <<boolean,boolean>>
+* Default value is `false`
+
+Enable detect schema mode. This will not send documents to Vespa, but will generate an application package
+based on the incoming data. This is useful for quickly getting started with Vespa when you don't have
+a schema yet.
+
+[id="plugins-{type}s-{plugin}-application_package_dir"]
+===== `application_package_dir`
+
+* Value type is <<string,string>>
+* Default value is `/tmp/vespa_app` (system temporary directory)
+
+Directory where the generated application package will be stored. Only relevant if `detect_schema` is `true`.
+
+[id="plugins-{type}s-{plugin}-deploy_package"]
+===== `deploy_package`
+
+* Value type is <<boolean,boolean>>
+* Default value is `true`
+
+Whether to deploy the generated application package. Only relevant if `detect_schema` is `true`.
+
+[id="plugins-{type}s-{plugin}-idle_batches"]
+===== `idle_batches`
+
+* Value type is <<number,number>>
+* Default value is `10`
+
+Number of empty batches to wait before deploying the application package. Only relevant if `detect_schema` is `true`
+and `deploy_package` is `true`.
+
+[id="plugins-{type}s-{plugin}-config_server"]
+===== `config_server`
+
+* Value type is <<string,string>>
+* Default value is `nil` (uses the host from `vespa_url`, if defined, with port 19071)
+
+URL to the Vespa config server for deployment. Only relevant if `detect_schema` is `true` and `deploy_package` is `true`.
+
+[id="plugins-{type}s-{plugin}-generate_mtls_certificates"]
+===== `generate_mtls_certificates`
+
+* Value type is <<boolean,boolean>>
+* Default value is `false`
+
+Whether to generate self-signed mTLS certificates for the application package. Only relevant if `detect_schema` is `true`.
+Defaults to `true` if using Vespa Cloud, otherwise `false`.
+
+[id="plugins-{type}s-{plugin}-certificate_common_name"]
+===== `certificate_common_name`
+
+* Value type is <<string,string>>
+* Default value is `cloud.vespa.logstash`
+
+Common name for the generated mTLS certificates. Only relevant if `detect_schema` is `true` and `generate_mtls_certificates` is `true`.
+
+[id="plugins-{type}s-{plugin}-certificate_validity_days"]
+===== `certificate_validity_days`
+
+* Value type is <<number,number>>
+* Default value is `30`
+
+Validity period in days for the generated mTLS certificates. Only relevant if `detect_schema` is `true` and `generate_mtls_certificates` is `true`.
+
+[id="plugins-{type}s-{plugin}-type_mappings_file"]
+===== `type_mappings_file`
+
+* Value type is <<string,string>>
+* Default value is `nil`
+
+Path to a YAML file containing custom type mappings for schema detection. Only relevant if `detect_schema` is `true`.
+
+[id="plugins-{type}s-{plugin}-type_conflict_resolution_file"]
+===== `type_conflict_resolution_file`
+
+* Value type is <<string,string>>
+* Default value is `nil`
+
+Path to a YAML file containing custom type conflict resolution rules. Only relevant if `detect_schema` is `true`.
+
+[id="plugins-{type}s-{plugin}-vespa_cloud_tenant"]
+===== `vespa_cloud_tenant`
+
+* Value type is <<string,string>>
+* Default value is `nil`
+
+Tenant name for Vespa Cloud deployment. Only relevant if `detect_schema` is `true` and deploying to Vespa Cloud.
+
+[id="plugins-{type}s-{plugin}-vespa_cloud_application"]
+===== `vespa_cloud_application`
+
+* Value type is <<string,string>>
+* Default value is `nil`
+
+Application name for Vespa Cloud deployment. Only relevant if `detect_schema` is `true` and deploying to Vespa Cloud.
+
+[id="plugins-{type}s-{plugin}-vespa_cloud_instance"]
+===== `vespa_cloud_instance`
+
+* Value type is <<string,string>>
+* Default value is `default`
+
+Instance name for Vespa Cloud deployment. Only relevant if `detect_schema` is `true` and deploying to Vespa Cloud.
+
+[id="plugins-{type}s-{plugin}-auth_token"]
+===== `auth_token`
+
+* Value type is <<string,string>>
+* Default value is `nil`
+
+Authentication token for Vespa Cloud. If provided, it will be sent as a Bearer token in the Authorization header.
+
+Note: This is mutually exclusive with client certificate authentication (`client_cert` and `client_key`). If both are provided, the client certificate will be used.
+
 // The full list of Value Types is here: 
 // https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
 

data/lib/logstash-output-vespa_feed_jars.rb

@@ -2,4 +2,4 @@
 # encoding: utf-8
 
 require 'jar_dependencies'
-require_jar('org.logstashplugins', 'logstash-output-vespa_feed', '0.8.0')
+require_jar('org.logstashplugins', 'logstash-output-vespa_feed', '1.0.0')

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-vespa_feed
 version: !ruby/object:Gem::Version
-  version: 0.8.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Radu Gheorghe
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2025-02-03 00:00:00.000000000 Z
+date: 2025-04-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -76,7 +76,7 @@ files:
 - lib/logstash-output-vespa_feed_jars.rb
 - lib/logstash/outputs/vespa_feed.rb
 - logstash-output-vespa_feed.gemspec
-- vendor/jar-dependencies/org/logstashplugins/logstash-output-vespa_feed/0.8.0/logstash-output-vespa_feed-0.8.0.jar
+- vendor/jar-dependencies/org/logstashplugins/logstash-output-vespa_feed/1.0.0/logstash-output-vespa_feed-1.0.0.jar
 homepage: https://vespa.ai
 licenses:
 - Apache-2.0