julewire-core 1.0.0

data/docs/records-and-data-policy.md

@@ -0,0 +1,230 @@
+# Records and Data Policy
+
+Core records are symbol-key hashes before formatting. Public ingress accepts
+string or symbol keys defensively. Processors receive a mutable
+`Julewire::RecordDraft`; destinations and formatters receive an immutable
+`Julewire::Record`. The canonical symbol-key shape is:
+
+```ruby
+{
+  timestamp: Time.now.utc,
+  severity: :info,
+  kind: :point,
+  event: "log",
+  message: "message",
+  logger: nil,
+  source: nil,
+  execution: {},
+  context: {},
+  carry: {},
+  neutral: {},
+  attributes: {},
+  labels: {},
+  payload: {},
+  metrics: {},
+  error: nil
+}
+```
+
+Raw input is normalized through `Julewire::RecordDraft.build`, then processors
+mutate the draft owned by the current emit. After processors finish, core
+validates the draft and freezes it through `Julewire::RecordDraft#to_record`
+before destinations and formatters see it. Immutable `Record` objects are not
+the public raw-input construction API.
+
+`Julewire::RecordFormatter` turns that normalized record into a public log
+projection. It omits internal keys such as `:carry` and execution lineage
+internals such as `:depth` and `:root`. Direct core destinations pass formatter
+payloads to an encoder. The default encoder serializes, compacts, and writes one
+JSON object plus a newline.
+
+Records built from log-safe field values are `Ractor.shareable?` after
+finalization. Use `Serializer.call` before crossing ractor, process, or network
+boundaries with arbitrary application objects.
+
+## Record Kinds
+
+Core supports:
+
+- `:point` for immediate records from `emit`
+- `:summary` for final execution summaries
+
+Unknown kinds are treated as extension bugs and contained as Julewire
+normalization failures. They are not silently coerced.
+
+Invalid explicit record severity is treated as caller data quality trouble, not
+a logging outage. Core normalizes the record severity to `:info`, writes one
+process warning, and counts the normalization in health with value class plus
+source/event metadata when available. Level filtering then applies normally, so
+an invalid explicit severity can still be dropped when `config.level` is above
+`:info`. Configuration severities remain strict.
+
+## Structured Sections
+
+Structured sections prefer hashes:
+
+```ruby
+execution context carry neutral attributes labels payload metrics
+```
+
+Labels are operator metadata, not payload storage. Treat them as non-sensitive,
+low-cardinality dimensions that are safe to copy into diagnostic records and
+indexes. PII and secrets belong in payload/context fields that a processor can
+handle before formatting.
+
+Carry is propagated correlation data. It is present on normalized records and
+carried through propagation envelopes, but it is not emitted by the default
+formatter and is not execution identity. Use it for small facts that
+integrations or formatters need on every record. Put large diagnostic snapshots
+in summary payloads instead.
+
+If application code supplies a non-hash value for a structured section, core
+preserves it under `:value` instead of dropping it:
+
+```ruby
+Julewire.emit(payload: "raw")
+# payload: { value: "raw" }
+```
+
+Runtime mutation helpers are forgiving for the same reason: logging should not
+crash the app.
+
+String keys inside structured sections are normalized to symbols before
+processors and destinations run. Encoders and custom destinations may convert
+the record to string-key payloads, but the core Ruby record contract stays
+symbol-keyed.
+
+## Optional Metadata
+
+`logger` is the logical logger entrypoint, for example `"app"` or
+`"billing.audit"`.
+
+`source` is the producer source. Raw application emits leave it `nil`;
+integrations should set it.
+
+`timestamp` defaults to `Time.now.utc` when omitted. If caller code supplies an
+explicit timestamp, core preserves that value as caller data. Output code that
+requires a time-like timestamp should validate or coerce it before export.
+
+## Execution Lineage
+
+Normalized nested executions include cheap relationship metadata:
+
+```ruby
+execution: {
+  type: "job",
+  id: "job-1",
+  depth: 2,
+  root: { type: "request", id: "req-1" },
+  parent: { type: "request", id: "req-1" }
+}
+```
+
+The full ancestor chain is available through the explicit lineage accessor:
+
+```ruby
+record.lineage.ancestors
+record.lineage.truncated?
+```
+
+Core keeps at most 42 serialized ancestors when that accessor is used. Live
+context inheritance still works across all active levels. The default formatter
+omits lineage internals and keeps only public execution identity plus caller
+fields.
+
+Lineage is still available to processors. Processors may deliberately promote
+selected relationship fields into output-facing
+sections before formatting:
+
+```ruby
+Julewire.configure do |config|
+  config.processors.use do |draft|
+    root_id = draft.lineage.root_reference&.fetch(:id, nil)
+    depth = draft.lineage.depth
+    ancestor_count = draft.lineage.ancestors.length
+
+    draft[:labels][:root_execution_id] = root_id if root_id
+    draft[:payload][:execution_depth] = depth if depth
+    draft[:payload][:ancestor_count] = ancestor_count
+  end
+end
+```
+
+That promotion is explicit policy. Core does not emit full lineage by default.
+
+## Raw by Policy
+
+Core does not redact. Normalized records and propagation envelopes keep
+application data raw, including `error.message`, `error.to_s`, payload fields,
+context fields, and carry fields. Default formatting may omit carry, but it does
+not sanitize values it does emit.
+
+The built-in serializer is an error-pruning layer, not a privacy layer. It makes
+values safe to format, caps worst cases, and avoids recursive crashes. It does
+not decide which values are secrets.
+
+Put redaction in processors or a separate policy gem before formatting.
+
+## Serializer Bounds
+
+`Julewire::Serializer` applies:
+
+- max nesting depth
+- circular reference detection
+- max string bytes
+- max array items
+- max hash keys
+- invalid UTF-8 repair
+- non-finite float sentinels
+- non-primitive numeric strings for `BigDecimal`, `Rational`, `Complex`, etc.
+- bounded exception backtraces and cause chains
+
+Truncated containers, containers pruned by `max_depth`, and circular container
+references get `_julewire_truncation` metadata. Serializer keys beginning with
+`_julewire_` are reserved by convention for core metadata; user payloads must
+not use that namespace. Public Julewire record contracts use symbol keys
+internally. Payloads should also use one JSON field name per value; mixed key
+types that stringify to the same JSON field are outside the serializer contract.
+Long strings use a `...[Truncated]` suffix. User data that already contains that
+suffix is preserved as user data.
+
+Unknown objects collapse to safe class markers instead of dumping arbitrary
+`inspect` output. If object serialization itself raises, the value becomes a
+bounded marker such as `"[Unserializable: RuntimeError]"`.
+
+Exceptions are shaped through `Julewire::Core::Serialization::ExceptionShape`
+before encoding:
+
+```ruby
+{
+  class: "RuntimeError",
+  message: "wrapper",
+  backtrace: ["app.rb:1:in ..."],
+  cause: {
+    class: "ArgumentError",
+    message: "root"
+  }
+}
+```
+
+Cause chains are bounded and cycle-safe.
+Set `config.error_backtrace_lines = 0` to omit `backtrace` fields from
+core-shaped exceptions in records and the default formatter output.
+The same limit is applied when core receives a core-shaped error hash with
+`backtrace` fields at the top level or inside nested `cause` hashes. This keeps
+the contract consistent for integrations that pre-shape errors before handing
+them to core.
+
+## Internal Error Records
+
+When core can still format and write a replacement record, internal failures may
+produce minimal records such as:
+
+- `julewire.emit_error`
+- `julewire.processor_error`
+
+Those records intentionally omit original payloads. They include bounded,
+serializer-scrubbed exception class details and safe record metadata such as
+source, event, severity, logger, and labels when available. They do not include
+raw exception messages by default. Use `on_failure` for local diagnostics when
+you need the original exception object.

data/docs/security-and-wire.md

@@ -0,0 +1,45 @@
+# Security and Wire Keys
+
+Julewire separates safety, privacy, and propagation. Core bounds output shape
+and contains logger-path failures. It does not redact application data by
+itself; redaction is processor policy.
+
+## Trust Boundaries
+
+Treat inbound carriers as trust-boundary data:
+
+- Request headers are not restored automatically. Integrations choose explicit
+  outbound carry headers.
+- Job carriers live inside serialized job data. That is usually an internal
+  queue boundary.
+- Message headers can come from other producers. Use an integration-level
+  carrier filter when topics accept external producers.
+
+Carry is baggage-shaped: small propagated facts, not log content. Keep large
+payloads in record `payload` or `attributes`, where processors and formatters
+can apply policy before output.
+
+## Capture and Redaction
+
+Body and broad header capture are opt-in because they can contain secrets.
+Framework parameter filtering may cover framework event payloads, but full
+Julewire record filtering is a processor decision. Install a record-filtering
+processor before enabling broad capture.
+
+Health snapshots, internal error records, and invalid-severity diagnostics omit
+raw exception messages and raw payloads. They carry safe coordinates such as
+exception class, event, source, severity, phase, component, and timestamp.
+
+## Wire Keys
+
+| Key | Location | Purpose |
+| --- | --- | --- |
+| `julewire` | Generic carrier key and message header | Serialized propagation carrier. |
+| `julewire.carrier` | Serialized job data | Propagation carrier inside job payloads. |
+| `neutral` | record section | Neutral formatter-coordination fields. Default JSON strips it. |
+| `_julewire_truncation` | Serialized hashes | Truncation metadata inserted by bounded serializers/transforms. |
+
+These names are similar by design, but they live at different boundaries:
+wire carriers move context across processes, `neutral` coordinates formatters
+inside one record, and `_julewire_truncation` reports output
+bounding.

data/docs/tail.md

@@ -0,0 +1,91 @@
+# Developer Tail
+
+`Julewire::Tail` is a bounded in-memory destination for local inspection. It is
+not delivery, buffering, or audit storage.
+
+```ruby
+tail = Julewire::Tail.attach!(capacity: 200)
+
+Julewire.info("booted", service: "api")
+
+puts tail.render(limit: 20)
+```
+
+`Julewire.tail` attaches a tail destination to the default runtime:
+
+```ruby
+tail = Julewire.tail(capacity: 100)
+```
+
+The tail stores public formatter output after Julewire's normal processors have
+run. It omits hidden carry data and execution-lineage internals, and it applies
+core serializer bounds before records enter the ring buffer. `tail.records`
+returns frozen snapshots; `tail.render` returns one-line text; `tail.write(io)`
+writes that text to an IO.
+
+Use `Julewire::ConsoleFormatter` with `Julewire::TextEncoder` for normal
+stdout text destinations. Tail uses the same text encoder for its rendered
+view.
+
+## CLI Tail
+
+`julewire tail` reads JSON logs from a file or stdin and renders compact text:
+
+```sh
+julewire tail --format core log/development.jsonl
+```
+
+Provider gems can register their own file decoders, so mixed platform stdout can
+be tailed without teaching core about provider log envelopes:
+
+```sh
+platform logs -f service/my-app | julewire tail --format provider_json --raw-invalid -
+```
+
+Core decoders own Julewire's core JSON shape; provider gems own provider wire
+shapes.
+
+`julewire --version` prints the core CLI version.
+
+Parsing is strict by default. Use `--skip-invalid` to ignore non-JSON lines or
+`--raw-invalid` to print boot noise and platform lines unchanged while decoded
+Julewire records are rendered normally.
+
+`julewire transcode` uses the same provider-owned decoders, then writes another
+registered output format. Core ships `core` JSON and `console` text; provider
+gems can register their own output formats:
+
+```sh
+platform logs service/my-app | julewire transcode --from provider_json --to core --raw-invalid -
+```
+
+`Julewire::TextEncoder.new(theme: :punk)` is a high-contrast local console
+style. `Julewire.punk!` configures a named runtime with that formatter/encoder
+pair.
+
+`julewire doctor --punk` renders the doctor report as loud terminal text. Plain
+`julewire doctor` stays JSON for scripts.
+
+`Julewire.dev!` configures the same console and attaches a tail in one core-only
+call:
+
+```ruby
+tail = Julewire.dev!(tail: { capacity: 500 })
+```
+
+Use a custom formatter when the local view should differ from default output:
+
+```ruby
+tail = Julewire::Tail.attach!(
+  formatter: ->(record) { { severity: record.fetch(:severity), message: record.fetch(:message) } }
+)
+```
+
+The tail is a normal custom destination. `Julewire.health` reports its size,
+capacity, capture count, and formatter failures under the tail destination name.
+
+`Julewire::TailSampling` is separate from the developer tail. It wraps another
+destination and buffers records until an execution summary decides whether to
+keep the execution. Errors and slow executions are kept; the rest are sampled
+with Julewire's deterministic sampling hash. Pass `decider:` for a custom callable
+policy; it receives the summary record and `key:`.

data/exe/julewire

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "julewire/core"
+
+exit Julewire::Core::CLI.call

data/julewire-core.gemspec

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require_relative "lib/julewire/core/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "julewire-core"
+  spec.version = Julewire::Core::VERSION
+  spec.authors = ["Alexander Grebennik"]
+  spec.email = ["slbug@users.noreply.github.com", "sl.bug.sl@gmail.com"]
+
+  spec.summary = "Execution-scoped structured logging core for Ruby applications."
+  spec.description =
+    "Provider-neutral records, execution context, processors, destinations, " \
+    "and formatters for Julewire."
+  spec.homepage = "https://github.com/slbug/julewire"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 3.4"
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/slbug/julewire/tree/main/gems/core"
+  spec.metadata["changelog_uri"] = "https://github.com/slbug/julewire/blob/main/gems/core/CHANGELOG.md"
+
+  spec.metadata["rubygems_mfa_required"] = "true"
+
+  spec.files = Dir.chdir(__dir__) do
+    Dir[
+      "CHANGELOG.md",
+      "LICENSE.txt",
+      "README.md",
+      "docs/**/*.md",
+      "exe/*",
+      "julewire-core.gemspec",
+      "lib/**/*.rb"
+    ]
+  end
+  spec.bindir = "exe"
+  spec.executables = ["julewire"]
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "concurrent-ruby", ">= 1.3"
+  spec.add_dependency "zeitwerk", ">= 2.8.1"
+end

data/lib/julewire/core/cli/doctor.rb

@@ -0,0 +1,143 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Julewire
+  module Core
+    class CLI
+      class Doctor
+        FLAGS = {
+          "--color" => [:color, true],
+          "--json" => %i[format json],
+          "--no-color" => [:color, false],
+          "--plain" => %i[theme plain],
+          "--punk" => %i[theme punk],
+          "--text" => %i[format text]
+        }.freeze
+
+        def initialize(argv:, stdout:)
+          @argv = argv
+          @stdout = stdout
+        end
+
+        def call
+          options = doctor_options
+          report = Julewire.doctor
+          options.fetch(:format) == :json ? write_json(report) : write_text(report, options)
+          0
+        end
+
+        private
+
+        def doctor_options
+          { color: color_output?, format: :json, theme: :plain }.tap do |options|
+            until @argv.empty?
+              value = @argv.shift
+              if (assignment = FLAGS[value])
+                apply_option(options, assignment)
+              else
+                raise ArgumentError, "unknown option #{value}"
+              end
+            end
+          end
+        end
+
+        def apply_option(options, assignment)
+          key = assignment.fetch(0)
+          value = assignment.fetch(1)
+          options[key] = value
+          options[:format] = :text if key == :theme && value == :punk
+        end
+
+        def color_output?
+          @stdout.respond_to?(:tty?) && @stdout.tty?
+        end
+
+        def write_json(report)
+          @stdout.write(JSON.pretty_generate(report))
+          @stdout.write("\n")
+        end
+
+        def write_text(report, options)
+          theme = options.fetch(:theme)
+          color = options.fetch(:color)
+          lines = [
+            title(theme),
+            status_line(report, theme: theme, color: color),
+            runtime_line(report.fetch(:runtime)),
+            pipeline_line(report.fetch(:pipeline)),
+            component_line("destinations", report.dig(:pipeline, :destinations) || {}),
+            component_line("runtime_integrations", report.fetch(:integrations)),
+            component_line("process_integrations", report.fetch(:process_integrations)),
+            warning_lines(report.fetch(:warnings), theme: theme)
+          ].flatten.compact
+          @stdout.write("#{lines.join("\n")}\n")
+        end
+
+        def title(theme)
+          theme == :punk ? "!! JULEWIRE DOCTOR !!" : "Julewire Doctor"
+        end
+
+        def status_line(report, theme:, color:)
+          status = report.fetch(:status).to_s
+          label = theme == :punk ? punk_label(status) : "status=#{status}"
+          colorize(label, severity_for_status(status), color: color, theme: theme)
+        end
+
+        def runtime_line(runtime)
+          parts = [
+            "level=#{runtime.fetch(:level)}",
+            "generation=#{runtime.fetch(:generation)}",
+            "closed=#{runtime.fetch(:closed)}"
+          ]
+          "runtime #{parts.join(" ")}"
+        end
+
+        def pipeline_line(pipeline)
+          "pipeline configured=#{pipeline.fetch(:configured)} status=#{pipeline.fetch(:status)}"
+        end
+
+        def component_line(name, components)
+          return "#{name}=none" if components.empty?
+
+          values = components.map { |key, value| "#{key}:#{value.fetch(:status)}" }
+          "#{name}=#{values.join(",")}"
+        end
+
+        def warning_lines(warnings, theme:)
+          return "warnings=none" if warnings.empty?
+
+          header = theme == :punk ? "!! warnings=#{warnings.length}" : "warnings=#{warnings.length}"
+          [header, *warnings.map { warning_line(it, theme: theme) }]
+        end
+
+        def warning_line(warning, theme:)
+          prefix = theme == :punk ? "!!" : "-"
+          "#{prefix} #{warning.fetch(:code)}: #{warning.fetch(:message)}"
+        end
+
+        def punk_label(status)
+          glyph = Serialization::TextEncoder.punk_glyph(severity_for_status(status))
+          "#{glyph} status=#{status.upcase} #{glyph}"
+        end
+
+        def colorize(value, severity, color:, theme:)
+          return value unless color
+
+          styles = severity_styles(theme)
+          "\e[#{styles.fetch(severity.to_s, styles.fetch("unknown"))}m#{value}\e[0m"
+        end
+
+        def severity_for_status(status)
+          status == "ok" ? "info" : "error"
+        end
+
+        def severity_styles(theme)
+          return Serialization::TextEncoder::PUNK_SEVERITY_STYLES if theme == :punk
+
+          Serialization::TextEncoder::SEVERITY_STYLES
+        end
+      end
+    end
+  end
+end

data/lib/julewire/core/cli/line_helpers.rb

@@ -0,0 +1,77 @@
+# frozen_string_literal: true
+
+module Julewire
+  module Core
+    class CLI
+      module LineHelpers
+        private
+
+        def parse_command_options(options, command:)
+          yield(options, @argv.shift) until @argv.empty?
+          raise ArgumentError, "#{command} log path is required" unless options[:path]
+
+          options
+        end
+
+        def next_symbol_option(name)
+          value = @argv.shift
+          raise ArgumentError, "#{name} value is required" unless value
+
+          value.to_sym
+        end
+
+        def positive_integer_option(name)
+          value = @argv.shift
+          raise ArgumentError, "#{name} value is required" unless value
+
+          Validation.validate_integer_limit!(Integer(value, 10), name: name.delete_prefix("--"), positive: true)
+        rescue ArgumentError
+          raise ArgumentError, "#{name} must be a positive integer"
+        end
+
+        def apply_path_option(options, value, command:)
+          raise ArgumentError, "unknown option #{value}" if value.start_with?("-") && value != "-"
+          raise ArgumentError, "#{command} accepts one log path" if options[:path]
+
+          options[:path] = value
+        end
+
+        def handle_invalid_line(line, error, mode)
+          case mode
+          when :skip
+            nil
+          when :raw
+            @stdout.write(raw_line(line))
+          else
+            raise error
+          end
+        end
+
+        def raw_line(line)
+          line.end_with?("\n") ? line : "#{line}\n"
+        end
+
+        def indexed_lines(lines)
+          lines.each_with_index.filter_map do |line, index|
+            [index + 1, line] unless line.strip.empty?
+          end
+        end
+
+        def console_text_encoder(options)
+          LogFormats::ConsoleText.new(
+            color: options.fetch(:color),
+            max_value_bytes: options.fetch(:max_value_bytes),
+            theme: options.fetch(:theme)
+          )
+        end
+
+        def write_encoded_record_line(line, line_number, input_format:, invalid:, encoder:)
+          record = LogFormats.record_from_json_line(line, line_number: line_number, format: input_format)
+          @stdout.write(encoder.call(record))
+        rescue ArgumentError => e
+          handle_invalid_line(line, e, invalid)
+        end
+      end
+    end
+  end
+end

data/lib/julewire/core/cli/log_formats/console_text.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module Julewire
+  module Core
+    class CLI
+      module LogFormats
+        class ConsoleText
+          def initialize(color: false, max_value_bytes: Serialization::TextEncoder::DEFAULT_MAX_VALUE_BYTES,
+                         theme: :plain)
+            @formatter = Records::ConsoleFormatter.new
+            @encoder = Serialization::TextEncoder.new(
+              color: color,
+              max_value_bytes: max_value_bytes,
+              theme: theme
+            )
+          end
+
+          def call(record)
+            @encoder.call(@formatter.call(record))
+          end
+        end
+      end
+    end
+  end
+end