julewire-gcp 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 7770e24c27f9957004b7e38a1fc5b49161a07f65173fe687bea6a8102f3796d5
+  data.tar.gz: 37376ea4aa3f56d0488a15a36d5d2a9e82a4244c5accd18fca224c50a9e070f3
+SHA512:
+  metadata.gz: 853abdb1eee58dfb2d61178f50ec572df0504131d2af476abd7fd9a4b8b141249dbe82b585b0880a8c17d882b110fdd0be0e3fecce74cfe2b2441afa02f6c79c
+  data.tar.gz: 97b6621b589e538254db6455522270860d1b643a6f7f730f10f9b448d0d534b0abf4d04129bb028707aef90c739978180a052e01b996d84d203eccdd4dd86d33

data/CHANGELOG.md

@@ -0,0 +1,6 @@
+## Unreleased
+
+## 1.0.0 - 2026-06-21
+
+- Initial release: Google Cloud Logging formatter, trace fields, source
+  locations, Error Reporting shape, and CLI transcode support.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Alexander Grebennik
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,40 @@
+# Julewire GCP
+
+Google Cloud Logging structured JSON formatter and direct-output destination
+for Julewire records.
+
+It does not own queues, batching, retries, or network transport.
+
+## Quickstart
+
+```ruby
+gem "julewire-gcp"
+```
+
+```ruby
+Julewire.configure do |config|
+  config.destinations.use(
+    :gcp,
+    formatter: Julewire::GCP::Formatter.new(project_id: "my-project"),
+    output: $stdout
+  )
+end
+```
+
+Default shape:
+
+- Cloud Logging special fields: `severity`, `message`, `time`, `httpRequest`,
+  labels, operation, source location, trace, span, and trace sampled
+- remaining Julewire fields in JSON payload
+- records with neutral HTTP attributes mapped to `httpRequest`
+- stack traces promoted for Error Reporting when core-shaped errors include
+  backtrace lines
+
+## Docs
+
+- [Configuration](docs/configuration.md)
+- [Advanced Configuration](docs/advanced-configuration.md)
+- [Shape](docs/shape.md)
+- [Trace](docs/trace.md)
+- [Error Reporting](docs/error-reporting.md)
+- [Development](docs/development.md)

data/docs/advanced-configuration.md

@@ -0,0 +1,25 @@
+# Advanced Configuration
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `trace_headers_paths` | `carry.http.request_headers`, then secondary paths | Request trace-header paths. |
+| `trace_id_path` | `nil` | Explicit path to trace id. |
+| `span_id_path` | `nil` | Explicit path to span id. |
+| `trace_sampled_path` | `nil` | Explicit path to sampling flag. |
+| `label_formatter` | `nil` | Custom label formatter. |
+| `label_options` | `{}` | Options passed to the default label formatter. |
+
+Use explicit trace paths when trace facts are already normalized into record
+fields and no header parsing is needed:
+
+```ruby
+Julewire::GCP::Formatter.new(
+  project_id: "my-project",
+  trace_id_path: %i[attributes trace_id],
+  span_id_path: %i[attributes span_id],
+  trace_sampled_path: %i[attributes trace_sampled]
+)
+```
+
+Use `label_formatter` only when the default label limits or key formatting are
+not enough.

data/docs/configuration.md

@@ -0,0 +1,33 @@
+# Configuration
+
+## Default Path
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `project_id` | `nil` | Expand bare trace IDs to Cloud Trace resource names. |
+| `service_context` | `nil` | Add Error Reporting service/version context. |
+| `max_record_bytes` | `256 KiB` | Destination record-size cap from `Julewire::GCP::Destination`. |
+
+## Common Knobs
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `operation_producer` | `nil` | Override `logging.googleapis.com/operation.producer`. |
+| `max_labels` | `64` | Maximum Cloud Logging labels emitted. |
+| `max_label_key_bytes` | `512` | Maximum label key byte size. |
+| `max_label_value_bytes` | `65_536` | Maximum label value byte size. |
+
+Oversized label values are truncated. Oversized label keys are dropped because
+Cloud Logging rejects invalid label keys.
+
+Example:
+
+```ruby
+Julewire::GCP::Formatter.new(
+  project_id: "my-project",
+  service_context: { service: "web", version: "2026-05-31" }
+)
+```
+
+`Julewire::GCP::Destination` uses `Julewire::GCP::RECOMMENDED_MAX_RECORD_BYTES`
+as a conservative `max_record_bytes` default.

data/docs/development.md

@@ -0,0 +1,7 @@
+# Development
+
+Run the normal gem checks:
+
+```sh
+bundle exec rake
+```

data/docs/error-reporting.md

@@ -0,0 +1,31 @@
+# Error Reporting
+
+When a record contains an `error` hash with backtrace lines, the formatter
+promotes those lines to top-level JSON payload field `stack_trace` for Cloud
+Logging/Error Reporting ingestion.
+
+The stack trace starts with the exception summary, includes nested causes, and
+respects core's `error_backtrace_lines` setting. If
+`error_backtrace_lines = 0`, core-shaped errors have no backtrace and the
+formatter emits no `stack_trace`.
+
+When `stack_trace` is promoted, nested `julewire.error.backtrace` fields are
+removed to avoid carrying the same stack twice in one log entry. Exception
+class, message, and cause metadata remain in `julewire.error`.
+
+If the record has no message and no HTTP-derived message can be derived, the
+top-level `message` uses the exception summary, such as
+`RuntimeError: boom`.
+
+When no explicit `payload.gcp.source_location` is present, the formatter first
+uses neutral `code.*` fields from the record's `neutral` section, then infers
+`logging.googleapis.com/sourceLocation` from the first error backtrace frame.
+
+References:
+
+- Google Cloud Logging structured logging:
+  https://docs.cloud.google.com/logging/docs/structured-logging
+- Google Cloud Logging `LogEntry`:
+  https://docs.cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+- Google Cloud Error Reporting log formatting:
+  https://docs.cloud.google.com/error-reporting/docs/formatting-error-messages

data/docs/shape.md

@@ -0,0 +1,104 @@
+# Shape
+
+The formatter maps Julewire records to the special JSON fields recognized by
+Cloud Logging agents:
+
+- `severity`
+- `message`
+- `time`
+- `httpRequest`
+- `logging.googleapis.com/labels`
+- `logging.googleapis.com/operation`
+- `logging.googleapis.com/sourceLocation`
+- `logging.googleapis.com/trace`
+- `logging.googleapis.com/spanId`
+- `logging.googleapis.com/trace_sampled`
+
+The remaining non-empty Julewire data stays queryable in the JSON payload:
+
+- `payload`
+- `attributes`
+- `serviceContext` when configured
+- `julewire.kind`
+- `julewire.event`
+- `julewire.logger`
+- `julewire.source`
+- `julewire.execution`
+- `julewire.context`
+- `julewire.error`
+- `julewire.metrics`
+
+Records are mapped into `httpRequest` when they include provider-neutral HTTP
+fields in the record's `neutral` section:
+
+- `http.request.method`
+- `url.full`
+- `url.path`
+- `http.response.status_code`
+- `user_agent.original`
+- `client.address`
+- `http.response.body.size`
+
+`httpRequest.latency` comes from `metrics.duration_ms` when present.
+
+The formatter reads these core neutral HTTP attributes to build `httpRequest`.
+Integration-specific attribute namespaces stay queryable.
+
+Records are mapped into `logging.googleapis.com/sourceLocation` when they carry
+neutral `code.file.path`, `code.line.number`, or `code.function.name` attributes
+in the record's `neutral` section. Explicit `payload.gcp.source_location` wins.
+
+When a record has HTTP method, URL or path, and status but no explicit message,
+the formatter emits a concise HTTP-derived message such as
+`GET /orders -> 200 in 24.1ms`.
+
+Request summaries are marked as `logging.googleapis.com/operation.last = true`.
+The formatter does not infer `operation.first`; mark that specific record
+explicitly when it matters:
+
+```ruby
+Julewire.with_execution(type: :job, id: "job-1") do
+  Julewire.emit(
+    event: "job.started",
+    source: "worker",
+    payload: Julewire::GCP.operation(first: true)
+  )
+end
+```
+
+GCP output keeps public execution fields under `julewire.execution`.
+Lineage internals are omitted. Execution fields promoted to GCP-native fields,
+such as `logging.googleapis.com/operation.id` or configured trace paths, are
+not duplicated there.
+
+For records without an execution, provide an operation id:
+
+```ruby
+Julewire.emit(
+  event: "script.started",
+  source: "script",
+  payload: Julewire::GCP.operation(
+    id: "nightly-import-20260531",
+    producer: "script",
+    first: true
+  )
+)
+```
+
+The formatter consumes `payload.gcp.operation` and
+`payload.gcp.source_location` as control metadata and removes those control keys
+from emitted application payload.
+
+Add source-location metadata when the application has a meaningful code location
+to attach to a record:
+
+```ruby
+Julewire.emit(
+  event: "import.started",
+  payload: Julewire::GCP.source_location(
+    file: "app/jobs/import_job.rb",
+    line: 42,
+    function: "ImportJob#perform"
+  )
+)
+```

data/docs/trace.md

@@ -0,0 +1,49 @@
+# Trace
+
+The formatter exposes the trace header names it needs as:
+
+```ruby
+Julewire::GCP::CARRY_REQUEST_HEADERS
+```
+
+Put selected headers into Julewire carry before emitting records:
+
+```ruby
+headers = {
+  "traceparent" => "00-...",
+  "tracestate" => "...",
+  "x-cloud-trace-context" => "..."
+}
+
+Julewire.with_execution(type: :request, id: "request-1") do
+  Julewire.carry.add(http: { request_headers: headers })
+  Julewire.emit(message: "handled")
+end
+```
+
+By default the formatter looks for request headers at
+`carry.http.request_headers`, then checks `payload.request_headers` and
+`context.request_headers`. It understands W3C `traceparent` first, then Google
+`x-cloud-trace-context`.
+
+`tracestate` is carried with `traceparent`, but Cloud Logging special fields do
+not use it directly. If `project_id` is configured, a bare trace ID is expanded
+to:
+
+```text
+projects/[PROJECT-ID]/traces/[TRACE-ID]
+```
+
+Already-expanded trace resource names are left unchanged.
+
+You can point trace extraction at different record paths when another
+integration or processor stores trace facts elsewhere:
+
+```ruby
+Julewire::GCP::Formatter.new(
+  project_id: "my-project",
+  trace_id_path: %i[context trace_id],
+  span_id_path: %i[context span_id],
+  trace_sampled_path: %i[context trace_sampled]
+)
+```

data/julewire-gcp.gemspec

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+require_relative "lib/julewire/gcp/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "julewire-gcp"
+  spec.version = Julewire::GCP::VERSION
+  spec.authors = ["Alexander Grebennik"]
+  spec.email = ["slbug@users.noreply.github.com", "sl.bug.sl@gmail.com"]
+
+  spec.summary = "Google Cloud Logging formatter for Julewire."
+  spec.description = "Google Cloud Logging structured JSON formatter for Julewire records."
+  spec.homepage = "https://github.com/slbug/julewire"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 3.4"
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/slbug/julewire/tree/main/gems/gcp"
+  spec.metadata["changelog_uri"] = "https://github.com/slbug/julewire/blob/main/gems/gcp/CHANGELOG.md"
+
+  spec.metadata["rubygems_mfa_required"] = "true"
+
+  spec.files = Dir.chdir(__dir__) do
+    Dir[
+      "CHANGELOG.md",
+      "LICENSE.txt",
+      "README.md",
+      "docs/**/*.md",
+      "julewire-gcp.gemspec",
+      "lib/**/*.rb"
+    ]
+  end
+  spec.executables = []
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "julewire-core", ">= 1.0"
+  spec.add_dependency "zeitwerk", ">= 2.8.1"
+end

data/lib/julewire/gcp/destination.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module Julewire
+  module GCP
+    class Destination < Julewire::Core::Destinations::Destination
+      def initialize(output:, name: :gcp, formatter: nil, encoder: Julewire::JsonEncoder.new,
+                     max_record_bytes: DEFAULT_MAX_RECORD_BYTES, close_output: false, on_drop: nil,
+                     on_failure: nil)
+        super(
+          name: name,
+          close_output: close_output,
+          encoder: encoder,
+          formatter: formatter || Formatter.new,
+          max_record_bytes: max_record_bytes,
+          on_drop: on_drop,
+          on_failure: on_failure,
+          output: output
+        )
+      end
+    end
+  end
+end

data/lib/julewire/gcp/execution_payload.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module Julewire
+  module GCP
+    class ExecutionPayload
+      def initialize(trace_id_path:, span_id_path:, trace_sampled_path:)
+        @trace_keys = trace_execution_keys(trace_id_path, span_id_path, trace_sampled_path)
+      end
+
+      def call(record, operation_options:)
+        execution = record.fetch(:execution)
+        output = nil
+        execution.each do |key, value|
+          next if promoted_key?(key, value, operation_options)
+
+          (output ||= {})[key] = value
+        end
+        output
+      end
+
+      private
+
+      def promoted_key?(key, value, operation_options)
+        return true if key == :id && !operation_options[:id]
+
+        @trace_keys.key?(key) && !Core::Integration::Values::Read.blank?(value)
+      end
+
+      def trace_execution_keys(*paths)
+        paths.map { trace_execution_key(it) }.to_h { [it, true] }
+      end
+
+      def trace_execution_key(path)
+        return unless path&.length == 2
+        return unless path.fetch(0) == :execution
+
+        path.fetch(1)
+      end
+    end
+  end
+end

data/lib/julewire/gcp/formatter.rb

@@ -0,0 +1,238 @@
+# frozen_string_literal: true
+
+module Julewire
+  module GCP
+    class Formatter
+      SEVERITIES = {
+        debug: "DEBUG",
+        info: "INFO",
+        warn: "WARNING",
+        error: "ERROR",
+        fatal: "CRITICAL",
+        unknown: "DEFAULT"
+      }.freeze
+      EMPTY_HASH = {}.freeze
+      TRUE_VALUES = %w[true 1 yes].freeze
+      private_constant :EMPTY_HASH, :TRUE_VALUES
+
+      def initialize(project_id: nil,
+                     operation_producer: nil,
+                     service_context: nil,
+                     trace_headers_paths: [
+                       %i[carry http request_headers],
+                       %i[payload request_headers],
+                       %i[context request_headers]
+                     ],
+                     **options)
+        FormatterOptions.validate!(options)
+        @project_id = project_id
+        @operation_producer = operation_producer
+        @service_context = frozen_service_context(service_context)
+        @trace_headers_paths = FormatterOptions.trace_headers_paths(trace_headers_paths)
+        @label_formatter = FormatterOptions.label_formatter(options)
+        @trace_id_path = FormatterOptions.trace_value_path(options[:trace_id_path])
+        @span_id_path = FormatterOptions.trace_value_path(options[:span_id_path])
+        @trace_sampled_path = FormatterOptions.trace_value_path(options[:trace_sampled_path])
+        @execution_payload = ExecutionPayload.new(
+          trace_id_path: @trace_id_path,
+          span_id_path: @span_id_path,
+          trace_sampled_path: @trace_sampled_path
+        )
+      end
+
+      def call(record)
+        Julewire::Record.validate_normalized!(record)
+
+        error = record.fetch(:error)
+        operation_options = operation_options(record)
+        neutral_attributes = Core::Fields::AttributeKeys.from(record.fetch(:neutral))
+        source_location_options = SourceLocationOptions.call(record, neutral_attributes)
+        entry = {
+          "severity" => severity(record.fetch(:severity)),
+          "time" => record.fetch(:timestamp)
+        }
+        append_log_field(entry, "message", Core::Records::DisplayMessage.call(record))
+        entry.tap do |log_entry|
+          append_special_fields(
+            log_entry,
+            record,
+            error: error,
+            operation_options: operation_options,
+            source_location_options: source_location_options,
+            neutral_attributes: neutral_attributes
+          )
+          append_payload_fields(log_entry, record, error: error, operation_options: operation_options)
+        end
+      end
+
+      private
+
+      def frozen_service_context(value) = value ? Core::Fields::FieldSet.frozen_copy(value) : nil
+
+      def append_special_fields(entry, record, error:, operation_options:, source_location_options:,
+                                neutral_attributes:)
+        trace_context = trace_context(record)
+        append_log_field(entry, "httpRequest", HttpRequestFields.http_request(record, neutral_attributes))
+        append_log_field(entry, "logging.googleapis.com/labels", labels(record))
+        append_log_field(entry, "logging.googleapis.com/operation", operation(record, operation_options))
+        append_log_field(
+          entry,
+          "logging.googleapis.com/sourceLocation",
+          source_location(error, source_location_options)
+        )
+        append_log_field(entry, "logging.googleapis.com/trace", trace(record, trace_context))
+        append_log_field(
+          entry,
+          "logging.googleapis.com/spanId",
+          trace_value(record, trace_context, @span_id_path, :span_id)
+        )
+        append_log_field(
+          entry,
+          "logging.googleapis.com/trace_sampled",
+          trace_value(record, trace_context, @trace_sampled_path, :trace_sampled)
+        )
+      end
+
+      def append_payload_fields(entry, record, error:, operation_options:)
+        stack_trace = stack_trace(error)
+        append_log_field(entry, JULEWIRE_PAYLOAD_FIELD, julewire_payload(record,
+                                                                         error: error,
+                                                                         operation_options: operation_options,
+                                                                         stack_trace: stack_trace))
+        append_log_field(entry, "attributes", attributes_payload(record))
+        append_log_field(entry, "payload", application_payload(record))
+        append_log_field(entry, "stack_trace", stack_trace)
+        append_log_field(entry, "serviceContext", @service_context)
+      end
+
+      def append_log_field(entry, key, value)
+        return if value.nil?
+        return if (value.is_a?(Hash) || value.is_a?(Array)) && value.empty?
+
+        entry[key] = value
+        nil
+      end
+
+      def severity(value) = SEVERITIES.fetch(value)
+
+      def labels(record)
+        @label_formatter.call(record.fetch(:labels))
+      end
+
+      def operation(record, options)
+        execution = record.fetch(:execution)
+        id = options[:id] || execution[:id] || record.lineage.root_reference&.fetch(:id, nil)
+        return unless id
+
+        operation = {
+          "id" => id.to_s
+        }
+        append_log_field(operation, "producer", operation_producer(record, options))
+        operation["first"] = true if true_value?(options[:first])
+        operation["last"] = true if record.fetch(:kind) == :summary || true_value?(options[:last])
+        operation
+      end
+
+      def operation_options(record)
+        options = record.dig(:payload, :gcp, :operation)
+        options.is_a?(Hash) ? options : EMPTY_HASH
+      end
+
+      def operation_producer(record, options)
+        options[:producer] || @operation_producer || record[:source] || record[:logger]
+      end
+
+      def source_location(error, options)
+        return SourceLocation.call(options) unless options.empty?
+        return unless error.is_a?(Hash) && !error.empty?
+
+        SourceLocation.from_error(error)
+      end
+
+      def trace(record, trace_context)
+        value = trace_value(record, trace_context, @trace_id_path, :trace_id)
+        return unless value
+
+        trace = value.to_s
+        return trace if trace.start_with?("projects/") || @project_id.nil?
+
+        "projects/#{@project_id}/traces/#{trace}"
+      end
+
+      def trace_context(record)
+        @trace_headers_paths.each do |path|
+          context = TraceContext.extract(Core::Integration::Values::Read.path_value(record, path))
+          return context unless context.empty?
+        end
+        EMPTY_HASH
+      end
+
+      def trace_value(record, trace_context, path, key)
+        value = Core::Integration::Values::Read.path_value(record, path) if path
+        Core::Integration::Values::Read.blank?(value) ? trace_context[key] : value
+      end
+
+      def application_payload(record)
+        payload = record.fetch(:payload)
+        control = payload[:gcp]
+        remove_gcp_control_payload(payload, control)
+      end
+
+      def attributes_payload(record)
+        record.fetch(:attributes)
+      end
+
+      def remove_gcp_control_payload(payload, control)
+        return payload unless control.is_a?(Hash) && gcp_control_payload?(control)
+
+        cleaned_payload = payload.dup
+        cleaned_control = control.dup
+        cleaned_control.delete(:operation)
+        cleaned_control.delete(:source_location)
+        if cleaned_control.empty?
+          cleaned_payload.delete(:gcp)
+        else
+          cleaned_payload[:gcp] = cleaned_control
+        end
+        cleaned_payload
+      end
+
+      def gcp_control_payload?(control)
+        control.key?(:operation) || control.key?(:source_location)
+      end
+
+      def julewire_payload(record, error:, operation_options:, stack_trace: nil)
+        {}.tap do |payload|
+          append_log_field(payload, :kind, record.fetch(:kind))
+          append_log_field(payload, :event, record.fetch(:event))
+          append_log_field(payload, :logger, record.fetch(:logger))
+          append_log_field(payload, :source, record.fetch(:source))
+          append_log_field(
+            payload,
+            :execution,
+            @execution_payload.call(record, operation_options: operation_options)
+          )
+          append_log_field(payload, :context, record.fetch(:context))
+          append_log_field(payload, :error, julewire_error(error, stack_trace: stack_trace))
+          append_log_field(payload, :metrics, record[:metrics])
+        end
+      end
+
+      def julewire_error(error, stack_trace:)
+        return error unless error.is_a?(Hash) && stack_trace
+
+        StackTrace.remove_backtraces(error)
+      end
+
+      def stack_trace(error)
+        return unless error.is_a?(Hash) && !error.empty?
+
+        StackTrace.call(error)
+      end
+
+      def true_value?(value)
+        TRUE_VALUES.include?(value.to_s.downcase)
+      end
+    end
+  end
+end