logquill 3.0.0.rc5-x64-mingw-ucrt
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/LICENSE +21 -0
- data/README.md +99 -0
- data/lib/logquill/chain.rb +49 -0
- data/lib/logquill/helpers.rb +59 -0
- data/lib/logquill/json_safe.rb +74 -0
- data/lib/logquill/logger.rb +140 -0
- data/lib/logquill/native/logquill_ffi.dll +0 -0
- data/lib/logquill/native.rb +97 -0
- data/lib/logquill/plugins.rb +43 -0
- data/lib/logquill/version.rb +7 -0
- data/lib/logquill.rb +13 -0
- metadata +67 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 2037333253900ab0ab7edf16fcedde2915301bac9ac8886a77c549628a4e326c
|
|
4
|
+
data.tar.gz: ca5906d910ef525736b3d9a0bad468a79c5a1bdbe33304dcace5e2215cc1bf03
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 1852323eabc56c23f001fd717b89555b76e173935fd6612d7d92e1cc4d2f34f080c219a863844c2e821eaf0da0bbcb55ee31146bfec6eadbd9a435f872a0b097
|
|
7
|
+
data.tar.gz: c8fe6071da34820bdc293300101b589c410dcfe6f2439fd568ee6d88ccf5135b22b7793353059c27bc2f27d7c48ebcd9567510df83e4df8b2df55361a8aea3a1
|
data/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Nikhil Arora
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# logquill (Ruby)
|
|
2
|
+
|
|
3
|
+
Structured logging for Ruby on LogQuill's Rust core, through its C ABI (`crates/logquill-ffi`). The same record shape,
|
|
4
|
+
the same redaction and the same tamper-evident hash chain as the Python, Node, browser and other bindings: redaction and
|
|
5
|
+
the level filter run in Rust before anything is written.
|
|
6
|
+
|
|
7
|
+
> **Not published yet.** The gem is unreleased: the 3.0 line is MIT licensed and a release candidate (3.0.0-rc.5) is prepared but not published
|
|
8
|
+
> (`docs/stages/stage-9.md`). See [Build and test](#build-and-test) to use it from this repository.
|
|
9
|
+
|
|
10
|
+
## Use it
|
|
11
|
+
|
|
12
|
+
```ruby
|
|
13
|
+
require "logquill"
|
|
14
|
+
|
|
15
|
+
log = Logquill::Logger.new(
|
|
16
|
+
name: "web",
|
|
17
|
+
plugins: [{ type: "redact" }, { type: "pii" }],
|
|
18
|
+
transports: [{ type: "file", path: "web.log" }]
|
|
19
|
+
)
|
|
20
|
+
|
|
21
|
+
log.info("signed in", user: "alice@example.com", password: "hunter2")
|
|
22
|
+
log.child("db", component: "pg").warn("slow query", ms: 840)
|
|
23
|
+
log.close
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
`web.log` now has two JSON lines. In the first, `password` is `"***"` and the address in `user` is `***`; the second has
|
|
27
|
+
`"logger": "web.db"` and `component` and `ms` in `meta`. Call `close` (or `flush`) before the process ends: records are
|
|
28
|
+
written on a background thread.
|
|
29
|
+
|
|
30
|
+
### The logger
|
|
31
|
+
|
|
32
|
+
- `Logquill::Logger.new(name:, level:, plugins:, transports:, content_policy:, tamper_evident:, queue:, return_record:)`.
|
|
33
|
+
`level` is `:trace :debug :info :warn :error :fatal` (default `:info`). A wrong option raises `Logquill::ConfigError`
|
|
34
|
+
saying which one and what is allowed; an unknown option is an error, not ignored, so a typo cannot quietly leave a
|
|
35
|
+
secret in the log.
|
|
36
|
+
- `log.info(message, key: value, ..., llm: { model:, tokens_in:, ... })`, and the same for `trace debug warn error fatal`.
|
|
37
|
+
Returns `true` when the record was kept and `false` when it was filtered (or could not be logged). With
|
|
38
|
+
`return_record: true` it returns the finished record as a Hash instead. **A log call never raises**: a value JSON cannot
|
|
39
|
+
carry (NaN, a cycle, an object whose `to_s` raises, bytes that are not UTF-8) is turned into text, and a failure is
|
|
40
|
+
reported with `Kernel#warn`.
|
|
41
|
+
- `log.child("db", component: "pg")` is a logger named `web.db` with that meta on every record, sharing the level and
|
|
42
|
+
transports. `log.set_level(:error)` changes the level for the logger, its parent and its children.
|
|
43
|
+
- `plugins:` are `{ type: "redact", keys:, classes:, replacement: }`, `{ type: "pii", replacement: }`, `{ type: "sampling", rate: }`,
|
|
44
|
+
`{ type: "rate_limit", max_records:, per_seconds:, key: }` and `{ type: "adaptive_sampling", ... }` (or build them with
|
|
45
|
+
`Logquill::Plugins.redact`, `.pii`, `.sampling(rate:)`, `.rate_limit(...)`, `.adaptive_sampling(...)`), run in order in
|
|
46
|
+
Rust. `content_policy:` is `off` (the default: prompt, completion and tool content is withheld), `hash`, `truncate` or
|
|
47
|
+
`full`. `tamper_evident: true` chains each record as it is written; check a log file with `Logquill.verify_chain`.
|
|
48
|
+
- `transports:` takes Hashes for Rust transports (`{ type: "console", format: "json", colour: false }` or
|
|
49
|
+
`{ type: "file", path:, format:, max_bytes:, backup_count:, compression: "gz" }`) and also any object that responds to
|
|
50
|
+
`write(line, record)`: a Ruby transport, called on your thread with the finished, already-redacted record. It may also
|
|
51
|
+
define `flush` and `close`.
|
|
52
|
+
|
|
53
|
+
### Context
|
|
54
|
+
|
|
55
|
+
```ruby
|
|
56
|
+
Logquill.with_context(request_id: "r-1") { log.info("handling") } # request_id is in meta
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
The context is fiber-local: other threads do not see it, and a thread started inside the block does not inherit it. The
|
|
60
|
+
call's own fields win over the context.
|
|
61
|
+
|
|
62
|
+
### Helpers and the hash chain
|
|
63
|
+
|
|
64
|
+
Every stateless helper of the core is a method: `Logquill.canonical_json`, `pii_redact_text`, `redact_meta`,
|
|
65
|
+
`verify_chain`, `verify_signed_chain`, `format_text`, `parse_record`, `build_trace`, ... (`Logquill::HELPERS` lists them by
|
|
66
|
+
their Node-addon names; `Logquill.call("canonicalJson", ...)` runs one by that name). A record, `meta` or list of records
|
|
67
|
+
may be a Hash or a String of JSON text; **pass text when a number's spelling matters**, since `1` and `1.0` hash
|
|
68
|
+
differently in a chain. `Logquill::Chain` stamps records and signs its head:
|
|
69
|
+
|
|
70
|
+
```ruby
|
|
71
|
+
chain = Logquill::Chain.new
|
|
72
|
+
records = [chain.stamp(record_one), chain.stamp(record_two)]
|
|
73
|
+
Logquill.verify_chain(records) # => { ok: true, records_checked: 2, head_hash: "..." }
|
|
74
|
+
Logquill.verify_signed_chain(records, "key", chain.sign_head("key")) # also catches a deleted last line
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## Not in this gem (yet)
|
|
78
|
+
|
|
79
|
+
The medium-tier bindings ship the facade, context, console / file transports and Ruby transports. Not here: HTTP, cloud and
|
|
80
|
+
database transports (write a Ruby transport), framework adapters and SDK patching, the standard-library `Logger` bridge,
|
|
81
|
+
and tail sampling, the flight recorder and the run summary (the C ABI does not carry them yet). Host-language *plugins* that change a record are not possible either: the Rust
|
|
82
|
+
transports have written it by the time Ruby sees it; use a Ruby transport to act on a record.
|
|
83
|
+
|
|
84
|
+
## Build and test
|
|
85
|
+
|
|
86
|
+
You need Ruby 3.1 or newer and a Rust toolchain. From the repository root:
|
|
87
|
+
|
|
88
|
+
```sh
|
|
89
|
+
cargo build -p logquill-ffi --release # the shared library
|
|
90
|
+
cd bindings/ruby
|
|
91
|
+
gem install ffi minitest
|
|
92
|
+
ruby -Ilib test/test_logquill.rb # 21 tests
|
|
93
|
+
ruby -Ilib test/test_readme.rb # the example above is a file that is run, and this README must contain it
|
|
94
|
+
ruby scripts/pack_smoke.rb # build a platform gem, install it into an empty gem home, use it by name
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
The gem looks for the library in `LOGQUILL_LIB` (a path to the file), then inside the gem (`lib/logquill/native/`, how a
|
|
98
|
+
platform gem ships it), then in `target/release` and `target/debug` of this repository. The C ABI version is checked at
|
|
99
|
+
load: a library with a different one is refused.
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
|
|
5
|
+
module Logquill
|
|
6
|
+
# A hash chain: each record stamped through it carries `meta.prev_hash` and `meta.hash`, so a log can be checked later
|
|
7
|
+
# for an edited or removed line (Logquill.verify_chain). Safe to share between threads.
|
|
8
|
+
class Chain
|
|
9
|
+
# `genesis` is the hash the chain starts from; nil is the standard one.
|
|
10
|
+
def initialize(genesis = nil)
|
|
11
|
+
error = Native.error_slot
|
|
12
|
+
handle = Native.lq_chain_new(Native.c_text(genesis), error)
|
|
13
|
+
raise Error, Native.error_from(error) if handle.null?
|
|
14
|
+
|
|
15
|
+
@handle = FFI::AutoPointer.new(handle, Native.method(:lq_chain_free))
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
# Chains `record` (a Hash, or a String of JSON text) and returns it as a Hash with the hashes set.
|
|
19
|
+
def stamp(record)
|
|
20
|
+
text = record.is_a?(String) ? record : JsonSafe.dump(record)
|
|
21
|
+
error = Native.error_slot
|
|
22
|
+
out = Native.lq_chain_stamp(@handle, Native.c_text(text), error)
|
|
23
|
+
raise Error, Native.error_from(error) if out.null?
|
|
24
|
+
|
|
25
|
+
JSON.parse(Native.take(out))
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# The hash of the last record chained.
|
|
29
|
+
def head
|
|
30
|
+
error = Native.error_slot
|
|
31
|
+
out = Native.lq_chain_head(@handle, error)
|
|
32
|
+
raise Error, Native.error_from(error) if out.null?
|
|
33
|
+
|
|
34
|
+
Native.take(out)
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
# HMAC-SHA256 of the head under `key` (a String of bytes), hex-encoded: a signature that a deleted last line breaks.
|
|
38
|
+
def sign_head(key)
|
|
39
|
+
bytes = key.to_s.b
|
|
40
|
+
buffer = FFI::MemoryPointer.new(:uint8, [bytes.bytesize, 1].max)
|
|
41
|
+
buffer.put_bytes(0, bytes)
|
|
42
|
+
error = Native.error_slot
|
|
43
|
+
out = Native.lq_chain_sign_head(@handle, buffer, bytes.bytesize, error)
|
|
44
|
+
raise Error, Native.error_from(error) if out.null?
|
|
45
|
+
|
|
46
|
+
Native.take(out)
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
end
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
|
|
5
|
+
module Logquill
|
|
6
|
+
class Error < StandardError; end
|
|
7
|
+
|
|
8
|
+
# Raised by Logger.new for a config the core rejects; the message says which option and what is allowed.
|
|
9
|
+
class ConfigError < Error; end
|
|
10
|
+
|
|
11
|
+
# The stateless helpers of the C ABI, named as the Node addon names them. Each is also a snake_case method on Logquill
|
|
12
|
+
# (`Logquill.canonical_json`, `Logquill.verify_chain`, ...). `Logquill.call` runs one by its original name.
|
|
13
|
+
HELPERS = %w[
|
|
14
|
+
canonicalJson formatJson formatText formatLogfmt parseLogfmt parseRecord genesisHash verifyChain signHead
|
|
15
|
+
verifyHeadSignature verifySignedChain redactMeta redactLocal piiRedactMeta piiRedactText applyPolicyValue
|
|
16
|
+
applyContentPolicy contentFields fieldClasses keysInClass toEcsDocument sqlRow createTableSql graphRow cypher
|
|
17
|
+
leafPlaceholder levelFromJson utcTimestamp formatTimestamp buildLlmBlock parseTraceHeader spanIdFor traceIdFor
|
|
18
|
+
traceIdFromParent captureEnv captureContentEnabled semconvResolve epochNs classifyRecord logAttributes exportSpec
|
|
19
|
+
buildTrace
|
|
20
|
+
].freeze
|
|
21
|
+
|
|
22
|
+
# Arguments that are binary keys cross as base64, for these helpers, at these positions.
|
|
23
|
+
KEY_ARGUMENTS = { "signHead" => 1, "verifyHeadSignature" => 1, "verifySignedChain" => 1 }.freeze
|
|
24
|
+
|
|
25
|
+
class << self
|
|
26
|
+
# Runs the helper `name` (e.g. "canonicalJson") with `args` and returns the parsed result. A record, `meta` or list of
|
|
27
|
+
# records may be a Hash / Array, or a String holding its JSON text (pass text when a number's spelling matters: `1`
|
|
28
|
+
# and `1.0` hash differently in a tamper-evident chain). Raises Logquill::Error with the core's message.
|
|
29
|
+
def call(name, *args)
|
|
30
|
+
key_at = KEY_ARGUMENTS[name.to_s]
|
|
31
|
+
wire = args.each_with_index.map do |arg, i|
|
|
32
|
+
i == key_at ? [arg.to_s.b].pack("m0") : JsonSafe.call(arg)
|
|
33
|
+
end
|
|
34
|
+
error = Native.error_slot
|
|
35
|
+
out = Native.lq_call(Native.c_text(name), JSON.generate(wire), error)
|
|
36
|
+
raise Error, Native.error_from(error) if out.null?
|
|
37
|
+
|
|
38
|
+
JSON.parse(Native.take(out))
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
HELPERS.each do |helper|
|
|
42
|
+
snake = helper.gsub(/([A-Z])/) { "_#{Regexp.last_match(1).downcase}" }
|
|
43
|
+
define_method(snake) { |*args| call(helper, *args) }
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
# The verification results use Ruby's naming.
|
|
47
|
+
%i[verify_chain verify_signed_chain].each do |method|
|
|
48
|
+
original = instance_method(method)
|
|
49
|
+
define_method(method) do |*args|
|
|
50
|
+
original.bind_call(self, *args).transform_keys { |k| k.gsub(/([A-Z])/) { "_#{Regexp.last_match(1).downcase}" }.to_sym }
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
# The library's version, e.g. "3.0.0-dev.0".
|
|
55
|
+
def core_version
|
|
56
|
+
Native.lq_version
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
end
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
require "time"
|
|
5
|
+
|
|
6
|
+
module Logquill
|
|
7
|
+
# Turns any Ruby value into one the JSON encoder accepts, without raising: logging must never crash the app over what it
|
|
8
|
+
# was asked to log. The rules are Python's `default=str` and the other bindings':
|
|
9
|
+
#
|
|
10
|
+
# - Hash keys become strings, Symbols become strings, Times become ISO 8601 (UTC);
|
|
11
|
+
# - NaN and the infinities (which JSON cannot carry) become the strings "NaN", "Infinity", "-Infinity";
|
|
12
|
+
# - an object that is not plain JSON becomes its `to_s`, or "<unprintable Klass>" if that raises;
|
|
13
|
+
# - a String that is not valid UTF-8 (or is binary) keeps its valid characters and shows U+FFFD for the bad bytes;
|
|
14
|
+
# - a cycle, or nesting deeper than DEPTH, becomes "<circular>" / "<too deep>" instead of recursing forever.
|
|
15
|
+
module JsonSafe
|
|
16
|
+
DEPTH = 50
|
|
17
|
+
|
|
18
|
+
def self.call(value, depth = 0, seen = {})
|
|
19
|
+
return "<too deep>" if depth > DEPTH
|
|
20
|
+
|
|
21
|
+
case value
|
|
22
|
+
when nil, true, false, Integer then value
|
|
23
|
+
when String then utf8(value)
|
|
24
|
+
when Float then finite(value)
|
|
25
|
+
when Symbol then value.to_s
|
|
26
|
+
when Time then value.getutc.iso8601(3)
|
|
27
|
+
when Hash then container(value, seen) { value.each_with_object({}) { |(k, v), out| out[text(k)] = call(v, depth + 1, seen) } }
|
|
28
|
+
when Array then container(value, seen) { value.map { |item| call(item, depth + 1, seen) } }
|
|
29
|
+
else printable(value)
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def self.container(value, seen)
|
|
34
|
+
return "<circular>" if seen[value.object_id]
|
|
35
|
+
|
|
36
|
+
seen[value.object_id] = true
|
|
37
|
+
begin
|
|
38
|
+
yield
|
|
39
|
+
ensure
|
|
40
|
+
seen.delete(value.object_id)
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def self.utf8(string)
|
|
45
|
+
return string if string.encoding == Encoding::UTF_8 && string.valid_encoding?
|
|
46
|
+
|
|
47
|
+
raw = string.encoding == Encoding::BINARY ? string.dup.force_encoding(Encoding::UTF_8) : string
|
|
48
|
+
raw.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: "\u{FFFD}")
|
|
49
|
+
rescue EncodingError
|
|
50
|
+
string.b.force_encoding(Encoding::UTF_8).scrub("\u{FFFD}")
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
def self.finite(float)
|
|
54
|
+
return float if float.finite?
|
|
55
|
+
|
|
56
|
+
float.nan? ? "NaN" : (float.positive? ? "Infinity" : "-Infinity")
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def self.text(key)
|
|
60
|
+
key.is_a?(String) ? key : printable(key).to_s
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def self.printable(value)
|
|
64
|
+
value.to_s
|
|
65
|
+
rescue StandardError, NotImplementedError
|
|
66
|
+
"<unprintable #{value.class}>"
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
# JSON text for any value.
|
|
70
|
+
def self.dump(value)
|
|
71
|
+
JSON.generate(call(value))
|
|
72
|
+
end
|
|
73
|
+
end
|
|
74
|
+
end
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "json"
|
|
4
|
+
|
|
5
|
+
module Logquill
|
|
6
|
+
LEVELS = %i[trace debug info warn error fatal].freeze
|
|
7
|
+
|
|
8
|
+
# The context bound by Logquill.with_context, merged into every record logged inside the block. Fiber-local: another
|
|
9
|
+
# thread or fiber does not see it, and a thread started inside the block does not inherit it.
|
|
10
|
+
def self.current_context
|
|
11
|
+
(Thread.current[:logquill_context] || []).reduce({}) { |merged, fields| merged.merge(fields) }
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
# Binds `fields` to every record logged inside the block, by any logger. The call's own fields win over the context.
|
|
15
|
+
def self.with_context(**fields)
|
|
16
|
+
stack = (Thread.current[:logquill_context] ||= [])
|
|
17
|
+
stack.push(fields.transform_keys(&:to_s))
|
|
18
|
+
yield
|
|
19
|
+
ensure
|
|
20
|
+
stack.pop
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
# A logger on the Rust core: level filter, redaction and the hash chain run in Rust before anything is written.
|
|
24
|
+
#
|
|
25
|
+
# log = Logquill::Logger.new(name: "web", level: :info,
|
|
26
|
+
# plugins: [{ type: "redact" }, { type: "pii" }],
|
|
27
|
+
# transports: [{ type: "file", path: "web.log" }])
|
|
28
|
+
# log.info("signed in", user: "alice", password: "hunter2") # => true; the file has password "***"
|
|
29
|
+
#
|
|
30
|
+
# `transports` takes two kinds of entry. A Hash is a Rust transport (`{type: "console"}` or `{type: "file", path:}`,
|
|
31
|
+
# written on a background thread). An object that responds to `write(line, record)` is a Ruby transport: it is called
|
|
32
|
+
# on the caller's thread with the finished, already-redacted record. The constructor raises ConfigError for a config the
|
|
33
|
+
# core rejects; a log call never raises.
|
|
34
|
+
class Logger
|
|
35
|
+
attr_reader :name
|
|
36
|
+
|
|
37
|
+
def initialize(name: "app", level: :info, plugins: [], transports: [], content_policy: nil, tamper_evident: false,
|
|
38
|
+
queue: nil, return_record: false)
|
|
39
|
+
rust, host = transports.partition { |transport| transport.is_a?(Hash) }
|
|
40
|
+
config = {
|
|
41
|
+
name: name, level: level.to_s, plugins: plugins, transports: rust, tamper_evident: tamper_evident,
|
|
42
|
+
queue: queue, content_policy: content_policy&.to_s
|
|
43
|
+
}.compact
|
|
44
|
+
error = Native.error_slot
|
|
45
|
+
handle = Native.lq_logger_new(JsonSafe.dump(config), error)
|
|
46
|
+
raise ConfigError, Native.error_from(error, "the logger could not be created") if handle.null?
|
|
47
|
+
|
|
48
|
+
adopt(handle, name, host, return_record)
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
# A logger named "<name>.<suffix>" with `fixed_meta` added to every record. It shares this logger's level and transports.
|
|
52
|
+
def child(suffix, **fixed_meta)
|
|
53
|
+
error = Native.error_slot
|
|
54
|
+
meta = fixed_meta.empty? ? nil : JsonSafe.dump(fixed_meta)
|
|
55
|
+
handle = Native.lq_logger_child(@handle, Native.c_text(suffix), meta, error)
|
|
56
|
+
raise Error, Native.error_from(error) if handle.null?
|
|
57
|
+
|
|
58
|
+
other = Logger.allocate
|
|
59
|
+
other.send(:adopt, handle, "#{@name}.#{suffix}", @host, @return_record)
|
|
60
|
+
other
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
LEVELS.each do |level|
|
|
64
|
+
# log.info("message", key: value, llm: { model: "..." }) => true when kept, false when filtered; the record when
|
|
65
|
+
# the logger was built with return_record: true.
|
|
66
|
+
define_method(level) { |message = "", llm: nil, **meta| log(level, message, meta, llm) }
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
# The method behind the level methods, for a level chosen at run time.
|
|
70
|
+
def log(level, message, meta = {}, llm = nil)
|
|
71
|
+
context = Logquill.current_context
|
|
72
|
+
merged = context.empty? ? meta : context.merge(meta.transform_keys(&:to_s))
|
|
73
|
+
want_record = @return_record || !@host.empty?
|
|
74
|
+
slot = want_record ? FFI::MemoryPointer.new(:pointer) : nil
|
|
75
|
+
error = Native.error_slot
|
|
76
|
+
code = Native.lq_log(
|
|
77
|
+
@handle, level.to_s, nil, Native.c_text(message), merged.empty? ? nil : Native.c_text(JsonSafe.dump(merged)),
|
|
78
|
+
llm.nil? ? nil : Native.c_text(JsonSafe.dump(llm)), slot, error
|
|
79
|
+
)
|
|
80
|
+
return false if code.zero?
|
|
81
|
+
return failed(Native.error_from(error)) if code.negative?
|
|
82
|
+
|
|
83
|
+
return true unless want_record
|
|
84
|
+
|
|
85
|
+
record = JSON.parse(Native.take(slot.read_pointer))
|
|
86
|
+
deliver(record)
|
|
87
|
+
@return_record ? record : true
|
|
88
|
+
rescue StandardError => e
|
|
89
|
+
failed(e.message)
|
|
90
|
+
end
|
|
91
|
+
|
|
92
|
+
# Changes the level for this logger, its parent and its children. Raises ArgumentError for a level that does not exist.
|
|
93
|
+
def set_level(level)
|
|
94
|
+
error = Native.error_slot
|
|
95
|
+
return self unless Native.lq_logger_set_level(@handle, level.to_s, error).negative?
|
|
96
|
+
|
|
97
|
+
raise ArgumentError, Native.error_from(error)
|
|
98
|
+
end
|
|
99
|
+
alias level= set_level
|
|
100
|
+
|
|
101
|
+
# Waits up to `timeout` seconds for queued records to be written (Rust transports), and flushes Ruby transports that
|
|
102
|
+
# can. true when everything was written in time.
|
|
103
|
+
def flush(timeout: 5.0)
|
|
104
|
+
@host.each { |transport| transport.flush if transport.respond_to?(:flush) }
|
|
105
|
+
Native.lq_logger_flush(@handle, (timeout * 1000).to_i) == 1
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
# Flushes, stops the background thread, and closes Ruby transports that respond to `close`. Call once, on shutdown.
|
|
109
|
+
def close(timeout: 5.0)
|
|
110
|
+
done = Native.lq_logger_close(@handle, (timeout * 1000).to_i) == 1
|
|
111
|
+
@host.each { |transport| transport.close if transport.respond_to?(:close) }
|
|
112
|
+
done
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
private
|
|
116
|
+
|
|
117
|
+
def adopt(handle, name, host, return_record)
|
|
118
|
+
@handle = FFI::AutoPointer.new(handle, Native.method(:lq_logger_free))
|
|
119
|
+
@name = name
|
|
120
|
+
@host = host
|
|
121
|
+
@return_record = return_record
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
def deliver(record)
|
|
125
|
+
return if @host.empty?
|
|
126
|
+
|
|
127
|
+
line = JSON.generate(record)
|
|
128
|
+
@host.each do |transport|
|
|
129
|
+
transport.write(line, record)
|
|
130
|
+
rescue StandardError => e
|
|
131
|
+
Kernel.warn("logquill: #{transport.class} failed to write a record: #{e.message}")
|
|
132
|
+
end
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
def failed(message)
|
|
136
|
+
Kernel.warn("logquill: #{message}")
|
|
137
|
+
false
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
end
|
|
Binary file
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "ffi"
|
|
4
|
+
|
|
5
|
+
module Logquill
|
|
6
|
+
# The C ABI of the Rust core (crates/logquill-ffi, include/logquill.h), loaded with the `ffi` gem. Nothing outside this
|
|
7
|
+
# file touches a pointer: the rest of the gem sees Ruby strings and raises Logquill::Error.
|
|
8
|
+
module Native
|
|
9
|
+
extend FFI::Library
|
|
10
|
+
|
|
11
|
+
LIBRARY = case RbConfig::CONFIG["host_os"]
|
|
12
|
+
when /darwin/ then "liblogquill_ffi.dylib"
|
|
13
|
+
when /mswin|mingw|cygwin/ then "logquill_ffi.dll"
|
|
14
|
+
else "liblogquill_ffi.so"
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
# Where the shared library is looked for, first match wins: the LOGQUILL_LIB environment variable (a path to the
|
|
18
|
+
# file), a copy shipped inside the gem (precompiled platform gems), then a build in this repository's target/ directory
|
|
19
|
+
# (development). Returns the path, or raises with every place it looked.
|
|
20
|
+
def self.library_path
|
|
21
|
+
tried = []
|
|
22
|
+
from_env = ENV.fetch("LOGQUILL_LIB", nil)
|
|
23
|
+
candidates = []
|
|
24
|
+
candidates << from_env if from_env && !from_env.empty?
|
|
25
|
+
candidates << File.expand_path("native/#{LIBRARY}", __dir__)
|
|
26
|
+
root = File.expand_path("../../../..", __dir__)
|
|
27
|
+
%w[release debug].each { |profile| candidates << File.join(root, "target", profile, LIBRARY) }
|
|
28
|
+
candidates.each do |path|
|
|
29
|
+
return path if File.file?(path)
|
|
30
|
+
|
|
31
|
+
tried << path
|
|
32
|
+
end
|
|
33
|
+
raise LoadError,
|
|
34
|
+
"logquill: could not find #{LIBRARY}. Looked in:\n #{tried.join("\n ")}\n" \
|
|
35
|
+
"Set LOGQUILL_LIB to the library's path, install a platform build of the gem, or run " \
|
|
36
|
+
"`cargo build -p logquill-ffi --release` in the repository."
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
ffi_lib library_path
|
|
40
|
+
|
|
41
|
+
attach_function :lq_version, [], :string
|
|
42
|
+
attach_function :lq_abi_version, [], :uint32
|
|
43
|
+
attach_function :lq_string_free, [:pointer], :void
|
|
44
|
+
attach_function :lq_call, %i[string string pointer], :pointer
|
|
45
|
+
|
|
46
|
+
attach_function :lq_chain_new, %i[string pointer], :pointer
|
|
47
|
+
attach_function :lq_chain_stamp, %i[pointer string pointer], :pointer
|
|
48
|
+
attach_function :lq_chain_head, %i[pointer pointer], :pointer
|
|
49
|
+
attach_function :lq_chain_sign_head, %i[pointer pointer size_t pointer], :pointer
|
|
50
|
+
attach_function :lq_chain_free, [:pointer], :void
|
|
51
|
+
|
|
52
|
+
attach_function :lq_logger_new, %i[string pointer], :pointer
|
|
53
|
+
attach_function :lq_logger_child, %i[pointer string string pointer], :pointer
|
|
54
|
+
attach_function :lq_log, %i[pointer string string string string string pointer pointer], :int32
|
|
55
|
+
attach_function :lq_logger_name, %i[pointer pointer], :pointer
|
|
56
|
+
attach_function :lq_logger_set_level, %i[pointer string pointer], :int32
|
|
57
|
+
attach_function :lq_logger_flush, %i[pointer uint32], :int32
|
|
58
|
+
attach_function :lq_logger_close, %i[pointer uint32], :int32
|
|
59
|
+
attach_function :lq_logger_free, [:pointer], :void
|
|
60
|
+
|
|
61
|
+
# The ABI version this gem was written against. A library with a different one has changed a function's meaning.
|
|
62
|
+
ABI_VERSION = 1
|
|
63
|
+
|
|
64
|
+
if lq_abi_version != ABI_VERSION
|
|
65
|
+
raise LoadError, "logquill: #{library_path} speaks C ABI #{lq_abi_version}, this gem needs #{ABI_VERSION}"
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# A fresh out-parameter for `char **error`.
|
|
69
|
+
def self.error_slot
|
|
70
|
+
FFI::MemoryPointer.new(:pointer)
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
# Reads and frees an owned C string.
|
|
74
|
+
def self.take(pointer)
|
|
75
|
+
text = pointer.read_string.force_encoding(Encoding::UTF_8)
|
|
76
|
+
lq_string_free(pointer)
|
|
77
|
+
text
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
# The message a call left in `slot`, freed; a generic one if it left none.
|
|
81
|
+
def self.error_from(slot, fallback = "the call failed")
|
|
82
|
+
pointer = slot.read_pointer
|
|
83
|
+
pointer.null? ? fallback : take(pointer)
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
# A Ruby string as the UTF-8, NUL-free text a C string needs. Invalid bytes become U+FFFD and NUL becomes U+2400 rather
|
|
87
|
+
# than raising: a log call must not crash the caller over what it was asked to log.
|
|
88
|
+
def self.c_text(value)
|
|
89
|
+
return nil if value.nil?
|
|
90
|
+
|
|
91
|
+
text = value.to_s
|
|
92
|
+
text = text.dup.force_encoding(Encoding::UTF_8) if text.encoding == Encoding::BINARY
|
|
93
|
+
text = text.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: "\u{FFFD}")
|
|
94
|
+
text.include?("\0") ? text.tr("\0", "\u{2400}") : text
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
end
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Logquill
|
|
4
|
+
# Builders for the +plugins:+ option of Logquill::Logger: plugins that run in Rust, in order. A plugin is a plain hash with a
|
|
5
|
+
# +type+, so a hash written by hand works too; these only spell the options out and leave unset ones to the core's defaults.
|
|
6
|
+
module Plugins
|
|
7
|
+
module_function
|
|
8
|
+
|
|
9
|
+
# Masks the values of keys (case-insensitively, at every depth); +nil+ keys means password, token, secret, api_key and authorization.
|
|
10
|
+
def redact(keys: nil, classes: nil, replacement: nil)
|
|
11
|
+
build("redact", keys: keys, classes: classes, replacement: replacement)
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
# Masks emails, card numbers, phone numbers, SSNs and API keys inside any string of the record's meta.
|
|
15
|
+
def pii(replacement: nil)
|
|
16
|
+
build("pii", replacement: replacement)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# Keeps about +rate+ (0 to 1) of the records, chosen at random.
|
|
20
|
+
def sampling(rate:)
|
|
21
|
+
build("sampling", rate: rate)
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
# Drops records once more than +max_records+ share a key inside +per_seconds+; +key+ is "logger_level" (the default),
|
|
25
|
+
# "logger" or "message".
|
|
26
|
+
def rate_limit(max_records:, per_seconds:, key: nil)
|
|
27
|
+
build("rate_limit", max_records: max_records, per_seconds: per_seconds, key: key)
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
# Keeps errors and slow spans, samples the rest at a rate that follows demand, and caps the bytes written per window.
|
|
31
|
+
# An option left +nil+ keeps the core's default.
|
|
32
|
+
def adaptive_sampling(base_rate: nil, min_rate: nil, max_rate: nil, keep_at: nil, slow_ms: nil,
|
|
33
|
+
max_bytes_per_second: nil, window_seconds: nil)
|
|
34
|
+
build("adaptive_sampling", base_rate: base_rate, min_rate: min_rate, max_rate: max_rate, keep_at: keep_at,
|
|
35
|
+
slow_ms: slow_ms, max_bytes_per_second: max_bytes_per_second, window_seconds: window_seconds)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def build(type, **options)
|
|
39
|
+
{ type: type }.merge(options.compact)
|
|
40
|
+
end
|
|
41
|
+
private_class_method :build
|
|
42
|
+
end
|
|
43
|
+
end
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Logquill
|
|
4
|
+
# The gem's version. Lockstep with the crates and the other bindings (docs/rust-core-plan.md, section 7); a RubyGems
|
|
5
|
+
# pre-release is spelled "3.0.0.dev0" where Cargo and npm spell it "3.0.0-dev.0".
|
|
6
|
+
VERSION = "3.0.0.rc5"
|
|
7
|
+
end
|
data/lib/logquill.rb
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "logquill/version"
|
|
4
|
+
require_relative "logquill/native"
|
|
5
|
+
require_relative "logquill/json_safe"
|
|
6
|
+
require_relative "logquill/helpers"
|
|
7
|
+
require_relative "logquill/chain"
|
|
8
|
+
require_relative "logquill/plugins"
|
|
9
|
+
require_relative "logquill/logger"
|
|
10
|
+
|
|
11
|
+
# LogQuill for Ruby: structured logging on the Rust core, through its C ABI. See README.md.
|
|
12
|
+
module Logquill
|
|
13
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: logquill
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 3.0.0.rc5
|
|
5
|
+
platform: x64-mingw-ucrt
|
|
6
|
+
authors:
|
|
7
|
+
- Nikhil Arora
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies:
|
|
12
|
+
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: ffi
|
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
|
15
|
+
requirements:
|
|
16
|
+
- - "~>"
|
|
17
|
+
- !ruby/object:Gem::Version
|
|
18
|
+
version: '1.15'
|
|
19
|
+
type: :runtime
|
|
20
|
+
prerelease: false
|
|
21
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
22
|
+
requirements:
|
|
23
|
+
- - "~>"
|
|
24
|
+
- !ruby/object:Gem::Version
|
|
25
|
+
version: '1.15'
|
|
26
|
+
description: 'LogQuill for Ruby: the same record shape, redaction, sampling and tamper-evident
|
|
27
|
+
hash chain as the Python, Node, browser and other bindings, through a C ABI. Redaction
|
|
28
|
+
runs before anything is written.'
|
|
29
|
+
executables: []
|
|
30
|
+
extensions: []
|
|
31
|
+
extra_rdoc_files: []
|
|
32
|
+
files:
|
|
33
|
+
- LICENSE
|
|
34
|
+
- README.md
|
|
35
|
+
- lib/logquill.rb
|
|
36
|
+
- lib/logquill/chain.rb
|
|
37
|
+
- lib/logquill/helpers.rb
|
|
38
|
+
- lib/logquill/json_safe.rb
|
|
39
|
+
- lib/logquill/logger.rb
|
|
40
|
+
- lib/logquill/native.rb
|
|
41
|
+
- lib/logquill/native/logquill_ffi.dll
|
|
42
|
+
- lib/logquill/plugins.rb
|
|
43
|
+
- lib/logquill/version.rb
|
|
44
|
+
homepage: https://github.com/nikhilvdev/logquill
|
|
45
|
+
licenses:
|
|
46
|
+
- MIT
|
|
47
|
+
metadata:
|
|
48
|
+
source_code_uri: https://github.com/nikhilvdev/logquill
|
|
49
|
+
rubygems_mfa_required: 'true'
|
|
50
|
+
rdoc_options: []
|
|
51
|
+
require_paths:
|
|
52
|
+
- lib
|
|
53
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
54
|
+
requirements:
|
|
55
|
+
- - ">="
|
|
56
|
+
- !ruby/object:Gem::Version
|
|
57
|
+
version: '3.1'
|
|
58
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
59
|
+
requirements:
|
|
60
|
+
- - ">="
|
|
61
|
+
- !ruby/object:Gem::Version
|
|
62
|
+
version: '0'
|
|
63
|
+
requirements: []
|
|
64
|
+
rubygems_version: 4.0.23
|
|
65
|
+
specification_version: 4
|
|
66
|
+
summary: Structured, agent-aware logging for Ruby, on a Rust core
|
|
67
|
+
test_files: []
|