logstash-output-otlp 0.4.7-java → 0.5.0-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 99edd5380c8a604ff1635507ffe85f213a05092e4b9508f7273df289da3abe31
-  data.tar.gz: 7ed7840fb269a6108d3679c77250c9e68b3953661a48903c4bb1d7e30ef025fa
+  metadata.gz: 9b52eb6dd3c022d4044a25d15b65b0b4f0c49fed621dbad3719fbd1bc1c64550
+  data.tar.gz: a553cb6bb1fe87ce81d44d56b11c32f324041a351daad834ad0a6b2aa979bf50
 SHA512:
-  metadata.gz: b45a36f07a5c1818247675b4a3806fc52d5f911e79cf2563858ad38a0c4576b7662de4ec816c66a7b8084eee97c6b1eda1814addea4e486f30c900116e2ff148
-  data.tar.gz: 5bd376d2998654a9c6ee5f39f79fa421b95880111600c84c77948076ba3596f4e22214d86aea040b8d47ce84707bd79c4ae6fb7fb2606e692991b5f0b9753882
+  metadata.gz: e817e73a885e5c451468b073ed4baab8a700b8ba907f13dbbaead07da6ffce65e15058b4006d8062b8a2301511887bc07f0cca436d55466c0a5c09e30a0b4d18
+  data.tar.gz: 00b32d6a761e488dd1622fc081b93b768427af912871d983ed2e180ca25ebcfb0cdaef7dcf3e2c89bbc9b835c139edc82f79b4ce927809ea8493d40082c7e52f

data/README.md

@@ -104,6 +104,10 @@ output {
 | compression                  | [string](https://www.elastic.co/guide/en/logstash/8.12/configuration-file-structure.html#string), one of ["gzip", "none"] | No |
 | connect_timeout              | [long](https://www.elastic.co/guide/en/logstash/8.12/configuration-file-structure.html#number)                            | No |
 | timeout                      | [long](https://www.elastic.co/guide/en/logstash/8.12/configuration-file-structure.html#number)   | No |
+| max_queue_size               | long | No |
+| max_batch_size               | long | No |
+| schedule_delay_millis        | long | No |
+| export_timeout_millis        | long | No |
 | ssl_disable_tls_verification | [boolean](https://www.elastic.co/guide/en/logstash/8.12/configuration-file-structure.html#string)                         | No |
 | ssl_certificate_authorities  | [string](https://www.elastic.co/guide/en/logstash/8.12/configuration-file-structure.html#string)                          | No |
 | resource                     | [Hash](https://www.elastic.co/guide/en/logstash/latest/configuration-file-structure.html#hash)                            | No |
@@ -136,6 +140,30 @@ An endpoint that supports otlp to which logs are sent.
 
 - Value type is [long](https://www.elastic.co/guide/en/logstash/8.12/configuration-file-structure.html#number)
 - Default is: `10` (seconds)
+- Must be positive and no greater than `export_timeout_millis / 1000`.
+
+Batch settings:
+
+| Setting | Default | Purpose |
+| --- | --- | --- |
+| `max_queue_size` | `2048` | Maximum records waiting in memory. |
+| `max_batch_size` | `512` | Maximum records in one export. |
+| `schedule_delay_millis` | `1000` | Interval for exporting an incomplete batch; a full batch can send earlier. |
+| `export_timeout_millis` | `30000` | Maximum wait for an export result in the batch processor. |
+
+All four values must be positive integers, and `max_batch_size` must not exceed
+`max_queue_size`. Invalid or overflowing values fail at startup.
+
+The OTLP SDK retries transient failures with exponential backoff and jitter,
+up to five attempts including the original request. The initial delay is about
+one second, multiplied by 1.5, with a nominal five-second backoff cap. The request
+`timeout` can end retries earlier; five attempts are not guaranteed. See the
+[SDK retry policy](https://github.com/open-telemetry/opentelemetry-java/blob/v1.62.0/sdk/common/src/main/java/io/opentelemetry/sdk/common/export/RetryPolicy.java).
+
+Keep `export_timeout_millis` above the request timeout with some margin; the
+processor's wait does not cancel the network request. The queue is bounded and
+in memory: excess events are dropped when it fills, and batches that exhaust
+their retries are not requeued. This does not guarantee delivery during an outage.
 
 `protocol`
 
@@ -224,17 +252,40 @@ For unit tests, build the Logstash core jar first:
 
 ```bash
 make logstashcorejar
-JAVA_HOME=/opt/homebrew/Cellar/openjdk@17/17.0.18/libexec/openjdk.jdk/Contents/Home ./gradlew test -PLOGSTASH_CORE_PATH=/Users/willianmarchan/Projects/BCP/O11Y/logstash-output-otlp/assets/logstash-9.0.0/logstash-core
+./gradlew test -PLOGSTASH_CORE_PATH="$PWD/assets/logstash-9.0.0/logstash-core"
 ```
 
 `make gem` also builds the Logstash core jar before packaging the local plugin gem.
 
+## Monitoring
+
+The output exposes the Java SDK's `LATEST` log processor and exporter metrics
+under `sdk_metrics` in Logstash's Node Stats API, preserving component,
+endpoint and error attributes. The [metrics demo](tests/integration/queue/README.md)
+shows successful exports, failures and queue saturation in Prometheus.
+`kuskoman/logstash-exporter` v1.9.2 needs additional collector support for these
+custom fields; the demo uses JSON Exporter mappings.
+
 ## Running locally
 
-`docker-compose up`
+With Java 17 and Docker running:
 
-The local Dockerfile installs a gem built from this repository with `logstash-plugin install --no-verify --local`.
-That pattern is intended for local smoke tests of the local gem only.
+```sh
+make gem
+docker compose up --build
+```
+
+Logstash sends ten records over OTLP/HTTP. The Collector (`otel`) prints them
+using its `debug` exporter with the body `OTLP local example`.
+
+From another terminal:
+
+```sh
+docker compose logs otel
+docker compose down
+```
+
+The Dockerfile installs the local gem built from this repository.
 
 For a production-like image that installs the published gem from RubyGems, use:
 
@@ -243,8 +294,10 @@ FROM docker.elastic.co/logstash/logstash:9.4.4@sha256:c1aeca2bbf56148c1c868957e1
 RUN logstash-plugin install logstash-output-otlp
 ```
 
-The certificates under `config/tls` are local test certificates used by the Docker Compose example.
-Do not reuse those private keys or certificates in shared, staging, or production environments.
+### Integration tests
+
+See [tests/integration](tests/integration/README.md) for manual tests of export
+error logging, queue metrics in Prometheus and the timestamp fix for Loki.
 
 ## Notes
 

data/VERSION

@@ -1 +1 @@
-0.4.7
+0.5.0

data/docs/blog.md

@@ -0,0 +1,179 @@
+# De Logback y Elastic a OpenTelemetry: Logstash como puente OTLP
+
+## Cómo conectar logs, trazas y métricas sin reescribir las aplicaciones Java
+
+Muchas plataformas Java centralizan sus logs con una arquitectura conocida:
+
+![Classic centralized logging architecture](./images/classic-elastic-logging-architecture.png)
+
+Logback escribe en `stdout` o en un archivo. Filebeat o Elastic Agent recolecta el evento, Logstash lo transforma, Elasticsearch lo indexa y Kibana permite buscarlo.
+
+El modelo funciona, pero aparece una separación cuando se incorpora el OpenTelemetry Java Agent:
+
+- trazas y métricas salen por OTLP;
+- los logs siguen viajando por Logback y Logstash;
+- el contexto de una misma petición termina repartido entre plataformas.
+
+`logstash-output-otlp` cierra esa brecha: toma el evento procesado por Logstash, lo convierte en un `LogRecord` OTLP y lo envía a un OpenTelemetry Collector.
+
+## Una identidad compartida
+
+![Logs, traces, and metrics share one identity](./images/otel-three-signals.png)
+
+El plugin exporta **logs**. El Java Agent continúa exportando **trazas y métricas**. La correlación funciona porque las tres señales comparten:
+
+- `service.name`;
+- `trace.id`;
+- `span.id`;
+- ambiente, versión e identidad de Kubernetes.
+
+El Java Agent puede inyectar `trace.id` y `span.id` en el MDC de Logback. El formato del log debe conservarlos:
+
+```json
+{
+  "@timestamp": "2026-07-27T14:50:15.922Z",
+  "log.level": "WARN",
+  "message": "El pago excedió el tiempo esperado",
+  "service.name": "payments-api",
+  "trace.id": "5b8aa5a2d2c872e8321cf37308d69df2",
+  "span.id": "051581bf3cb55c13"
+}
+```
+
+Si Filebeat o los filtros de Logstash eliminan estos campos, la correlación se pierde.
+
+## Instalar y configurar el plugin
+
+```bash
+bin/logstash-plugin install logstash-output-otlp
+```
+
+Durante la transición, Logstash puede conservar Elasticsearch y añadir OTLP como segunda salida:
+
+```ruby
+input {
+  beats {
+    port => 5044
+  }
+}
+
+output {
+  elasticsearch {
+    hosts => ["https://elasticsearch:9200"]
+    index => "application-logs-%{+YYYY.MM.dd}"
+    user => "${ELASTIC_USER}"
+    password => "${ELASTIC_PASSWORD}"
+    ssl_enabled => true
+    ssl_certificate_authorities => ["/etc/logstash/certs/elastic-ca.crt"]
+  }
+
+  otlp {
+    endpoint => "https://otel-collector:4317"
+    protocol => "grpc"
+    compression => "gzip"
+    ssl_certificate_authorities => "/etc/logstash/certs/otel-ca.crt"
+
+    body => "message"
+    severity_text => "[log][level]"
+    trace_id => "trace.id"
+    span_id => "span.id"
+
+    resource => {
+      "service.name" => "payments-api"
+      "deployment.environment.name" => "production"
+    }
+  }
+}
+```
+
+Antes de activar OTLP conviene inspeccionar un evento real con `stdout { codec => rubydebug }` y confirmar los nombres de los campos.
+
+## El Collector como frontera neutral
+
+![Signal routing through the OpenTelemetry Collector](./images/otel-collector-routing.png)
+
+El Collector desacopla productores y destinos:
+
+- logs de Logstash → Loki;
+- trazas del Java Agent → Tempo;
+- métricas del Java Agent → Prometheus.
+
+Una configuración mínima puede separar las señales así:
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+
+processors:
+  memory_limiter:
+    check_interval: 1s
+    limit_mib: 512
+  batch: {}
+
+exporters:
+  otlphttp/loki:
+    endpoint: http://loki:3100/otlp
+  otlp/tempo:
+    endpoint: tempo:4317
+    tls:
+      insecure: true
+  prometheusremotewrite:
+    endpoint: http://prometheus:9090/api/v1/write
+
+service:
+  pipelines:
+    logs:
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [otlphttp/loki]
+    traces:
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [otlp/tempo]
+    metrics:
+      receivers: [otlp]
+      processors: [memory_limiter, batch]
+      exporters: [prometheusremotewrite]
+```
+
+Los endpoints, TLS y mecanismos de autenticación deben adaptarse al entorno.
+
+## De un log a una traza
+
+Con el contexto preservado, el recorrido en Grafana es directo:
+
+```text
+alerta en Prometheus
+→ servicio afectado
+→ log en Loki
+→ trace.id
+→ traza completa en Tempo
+```
+
+`trace.id` no debería convertirse en una etiqueta indexada de Loki: su cardinalidad es muy alta. Es mejor conservarlo como metadata estructurada o campo consultable.
+
+## Migrar sin un “big bang”
+
+![Gradual migration roadmap](./images/otel-migration-roadmap.png)
+
+1. **Conservar:** mantener Elastic mientras se valida el flujo actual.
+2. **Correlacionar:** inyectar `trace.id` y `span.id` en el MDC.
+3. **Duplicar:** añadir la salida OTLP junto a Elasticsearch.
+4. **Conectar:** consultar logs, trazas y métricas desde Grafana.
+
+Durante la convivencia se debe comparar volumen, timestamps, severidad, atributos y comportamiento ante fallos.
+
+## Consideraciones de producción
+
+- Usar TLS y verificar la CA del Collector.
+- No habilitar `ssl_disable_tls_verification`.
+- Definir una lista explícita de atributos para evitar datos sensibles.
+- Aplicar redacción y filtros también en el Collector.
+- Habilitar persistent queues si se requiere tolerar interrupciones.
+- Supervisar reintentos, descartes y presión de memoria.
+- Fijar versiones y digests de las imágenes.
+
+La meta no es reemplazar Elastic de inmediato. Es aprovechar el pipeline existente y convertir Logstash en una vía de adopción gradual hacia OTLP, Loki, Tempo, Prometheus u otros backends compatibles.

data/lib/logstash-output-otlp_jars.rb

@@ -2,4 +2,4 @@
 # encoding: utf-8
 
 require 'jar_dependencies'
-require_jar('org.otlp', 'logstash-output-otlp', '0.4.7')
+require_jar('org.otlp', 'logstash-output-otlp', '0.5.0')

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-otlp
 version: !ruby/object:Gem::Version
-  version: 0.4.7
+  version: 0.5.0
 platform: java
 authors:
 - Willian Marchan
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-07-27 00:00:00.000000000 Z
+date: 2026-11-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   requirement: !ruby/object:Gem::Requirement
@@ -70,11 +70,17 @@ files:
 - LICENSE
 - README.md
 - VERSION
+- docs/blog.md
+- docs/images/classic-elastic-logging-architecture.png
+- docs/images/logstash-otlp-architecture.png
+- docs/images/otel-collector-routing.png
+- docs/images/otel-migration-roadmap.png
+- docs/images/otel-three-signals.png
 - docs/index.asciidoc
 - lib/logstash-output-otlp_jars.rb
 - lib/logstash/outputs/otlp.rb
 - logstash-output-otlp.gemspec
-- vendor/jar-dependencies/org/otlp/logstash-output-otlp/0.4.7/logstash-output-otlp-0.4.7.jar
+- vendor/jar-dependencies/org/otlp/logstash-output-otlp/0.5.0/logstash-output-otlp-0.5.0.jar
 homepage: https://github.com/wjma90/logstash-output-otlp
 licenses:
 - Apache-2.0