rjq 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5556b3edad7d8c167e8cc364b1f40f071be01d783125d1a9308bfde9a2eb2884
+  data.tar.gz: d40945b36cd710c8d07acf7885c82db3299f6e903830ef16884ec5cc275e8f09
+SHA512:
+  metadata.gz: d4046883d388b8e1d7bb169ac8e55c374cb79a010ebb6060707bc40cf91df8dd89085c0597b9a95ced0fdad50bbd279050f3d3c6742eaa18cc9a1a0e0308ccee
+  data.tar.gz: 53c472f7c37ead61b345b8168e7495e92a928b9cefe6dba3f9619bca41d41e51ffcc860a9ad4f985dcef84d2ae6fc064ec1e2ba9ca3fef698420bc6ff7a97815

data/ARCHITECTURE.md

@@ -0,0 +1,38 @@
+# Architecture
+
+The execution path is deliberately split into explicit phases:
+
+1. The lexer records token offsets and source names.
+2. The parser builds filter and module-directive AST nodes.
+3. `ModuleLoader` resolves dependencies through an injected resolver, evaluates constant metadata, and constructs namespaced definitions.
+4. `BytecodeCompiler` lowers the executable AST to instructions and constants.
+5. `SemanticAnalyzer` validates function and builtin names and arities before input is consumed.
+6. `VM` executes enumerator-backed value streams.
+7. `Runtime` coordinates incremental input records, input builtins, filenames, line numbers, and output budgets.
+8. `JSON::Dumper` writes values directly to an IO without recursively building container output strings.
+
+## Streaming model
+
+`JSON::InputBuffer` reads fixed-size chunks. The normal parser yields complete top-level values; the stream parser yields path events within containers. `Runtime::InputQueue` adds one-record lookahead for `input` and `inputs`. Owned file handles are closed on completion, failure, and downstream early termination.
+
+Array constructors, JSON slurp, raw slurp, sorting, grouping, uniqueness, and other jq aggregation boundaries collect by definition. Generators such as `range`, `recurse`, `repeat`, `while`, `until`, and `inputs` remain lazy so `first` and `limit` apply backpressure.
+
+## Number model
+
+`Rjq::Number` retains the original decimal literal and its binary64 value. Exact untouched literals use decimal components for equality and ordering without expanding large exponents. Arithmetic returns computed Ruby numeric values, which the dumper renders separately from untouched lexemes.
+
+## Modules
+
+Production code has no fixture-name fallback. `ModuleResolver` accepts explicit paths, canonicalizes real paths, and limits file size. `ModuleLoader` limits dependency depth, detects canonical cycles, caches parsed modules, handles data imports, and applies namespace rewriting to AST calls rather than source strings. Tests inject their own fixture resolver.
+
+## Resource boundaries
+
+JSON parsing uses jq's 256-container depth limit. Module bytes and dependency depth are limited. Embedded callers can
+set `max_outputs` and `input_chunk_size`; `input_max_depth` can lower the parser depth limit. `max_number_digits` and
+`max_string_bytes` are optional, unlimited by default, and stop oversized tokens incrementally before numeric
+conversion or decoded-string growth. Public runtime and compiler APIs reject unknown options and invalid limits at
+construction. Filter parsing is capped at 256 nested expressions by default. Continuation-free user calls in tail position use an
+explicit VM trampoline; non-tail calls can be bounded with `max_call_depth`. An optional bytecode-instruction budget is
+charged lazily, and execution limits bypass jq-level error handlers so filters cannot disable host safety policy.
+`max_replay_cache` optionally bounds the memoized values needed to replay effectful index filters.
+Cyclic Ruby values and invalid JSON value types are rejected before copying or output.

data/CHANGELOG.md

@@ -0,0 +1,7 @@
+# Changelog
+
+## Unreleased
+
+## 0.1.0 - 2026-08-13
+
+- Initial release

data/COMPATIBILITY.md

@@ -0,0 +1,48 @@
+# Compatibility
+
+`rjq` targets a practical subset of jq 1.7.1. Compatibility is tested at two levels:
+
+- The bundled, adapted `jq.test` and `onig.test` fixtures exercise 447 and 40 cases. Expected JSON is decoded by Ruby's independent stdlib parser and compared without rjq's parser or value comparator. `%%FAIL` cases verify error categories instead of accepting any error.
+- The differential suite invokes an independently installed jq 1.7.1 executable and compares stdout bytes, normalized stderr, exit status, output count, and order. Run it with `JQ_BIN=/path/to/jq-1.7.1 bundle exec rake differential`.
+
+The fixture source tag, upstream commit, adaptation marker, and file checksums are recorded in `spec/fixtures/jq/manifest.json`.
+
+## Compatibility areas
+
+| Area | Current contract |
+| --- | --- |
+| JSON numbers | Untouched decimal literals retain their lexeme; computed values use binary64-style rendering. |
+| Filter streams | Pipes, commas, branching, interpolation, filter arguments, slices, reduce, foreach, range, recurse, repeat, while, and until preserve multiple outputs. |
+| Errors | Compile, runtime, input parse, `halt`, `halt_error`, and `-e` statuses follow the jq status classes documented below. |
+| Input | Top-level JSON and `--stream` parse incrementally. Multiple files form one logical stream; only slurp and aggregating filters intentionally collect. |
+| Modules | Directives are parsed syntax, paths are canonicalized, fixture resolution is test-only, and traversal, symlink, size, depth, and cycle checks are enforced. |
+| Filter locations | `$__loc__` and disassembly use filter-source positions, including module and interpolation sources. `-f` filenames are reported as canonical absolute paths. Input filenames and line numbers remain separate. |
+| Regex | Ruby's regular-expression engine is used. jq's `m`, `s`, and `p` mode mapping is preserved; advanced Oniguruma behavior can differ. Ruby API callers may set `regexp_timeout` on runtimes that support per-expression timeouts. |
+| Date/time | UTC conversion is host-timezone independent; platform date ranges can still differ. |
+| Math | jq 1.7.1 builtin names and arities are declared. Bessel, fused multiply-add, and IEEE remainder use the platform C math library through `fiddle` when available; scaling uses a portable implementation. |
+
+## Known differences
+
+- This is not libjq and does not claim complete source, diagnostic-text, regex-engine, module-layout, or performance compatibility.
+- Regex flag `l`, Unicode offsets, and advanced engine constructs can differ because Ruby Regexp is not jq's bundled Oniguruma build. In particular, zero-width matches on multibyte UTF-8 strings advance by codepoint in rjq and by byte in jq, so `split/2` can contain a different number of empty fields.
+- Ruby Regexp rejects some variable-length lookbehind expressions that jq's Oniguruma accepts. These remain controlled regular-expression errors rather than loading a second native regex engine.
+- `fma` and IEEE remainder require their exact native symbols and raise controlled runtime errors when unavailable. `scalb` and `scalbln` use a portable `ldexp`-based implementation with jq-compatible truncation, saturation, signed zero, and non-finite handling.
+- rjq extensions include `@base32`, `@base32d`, `dateadd`, `datesub`, `ascii`, `to_number`, and compatibility aliases such as `leaf_paths`. They remain callable but are excluded from jq's `builtins` result; embedders can inspect `Rjq::Builtins::EXTENSION_ARITIES`.
+- `get_jq_origin` reports the rjq installation root. Default module search paths are rjq's actual expanded paths, not jq's symbolic `$ORIGIN` entries.
+- JSON object keys supplied through the Ruby API must be strings. Cyclic Ruby arrays and hashes are rejected.
+- `env`, `now`, local time, module loading, file arguments, and diagnostic builtins access process capabilities unless the caller avoids those features or injects controlled options.
+
+## Exit statuses
+
+| Condition | Status |
+| --- | ---: |
+| Option or system usage error | 2 |
+| Filter compile error | 3 |
+| `-e` with no output | 4 |
+| Runtime or input JSON error | 5 |
+| `halt` | 0 |
+| `halt_error(n)` | `n` |
+
+## Ruby API values
+
+Inputs and outputs are composed of `nil`, booleans, `Numeric`, UTF-8 `String`, `Array`, and `Hash` with string keys. Assignment operations copy affected values. Mutable strings are copied at constant and deep-copy boundaries. A compiled program may be reused sequentially; concurrent reuse has not yet been declared a stable guarantee.

data/CONTRIBUTING.md

@@ -0,0 +1,30 @@
+# Contributing
+
+Install dependencies and run the complete local gate:
+
+```sh
+bundle install
+bundle exec rake
+```
+
+When jq 1.7.1 is available, also run the independent byte-level comparison:
+
+```sh
+JQ_BIN=/path/to/jq-1.7.1 bundle exec rake differential
+```
+
+Build and smoke-test the gem before changing packaging:
+
+```sh
+gem build rjq.gemspec
+gem install ./rjq-*.gem
+rjq -nc '{installed: true}'
+```
+
+Changes to semantics should include a focused regression test and, where jq behavior is the contract, a differential case. Avoid deriving both expected and actual values through the same parser or comparator. Preserve partial output and verify the terminal error/status separately.
+
+The bundled jq fixtures are adapted regression inputs. If they change, update `spec/fixtures/jq/manifest.json` deliberately and record the upstream tag or commit, checksum, and reason in the change.
+
+## Release
+
+Update `Rjq::VERSION` and `CHANGELOG.md` in one reviewed change. After CI passes, create and push a matching `vX.Y.Z` tag. `.github/workflows/release.yml` verifies the tag, reruns the gate, and publishes with RubyGems trusted publishing. The repository and `release.yml` workflow must first be registered as a RubyGems trusted publisher using the `release` environment.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yudai Takada
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,192 @@
+# rjq
+
+`rjq` is a Ruby JSON processor implementing a practical jq 1.7.1-compatible subset with a command line and Ruby API.
+
+The runtime uses its own incremental JSON parser and writer instead of Ruby's `json` library. The gem has no project-specific native extension; Bessel math functions call the platform C math library through `fiddle`.
+
+## Installation
+
+Install the gem and add it to your application's Gemfile by executing:
+
+```sh
+bundle add rjq
+```
+
+If bundler is not being used to manage dependencies, install the gem by executing:
+
+```sh
+gem install rjq
+```
+
+From a checkout, install dependencies and run the executable directly:
+
+```sh
+bundle install
+bin/rjq --version
+```
+
+## Usage
+
+Use `rjq` from the command line:
+
+```sh
+rjq '.foo | map(select(.active)) | .[].name' <<'JSON'
+{"foo":[{"name":"Ada","active":true},{"name":"Linus","active":false}]}
+JSON
+```
+
+Useful CLI examples:
+
+```sh
+rjq -c '[range(3)]' <<< 'null'
+rjq -r '.message' <<< '{"message":"hello"}'
+rjq --arg name Ada '{"hello": $name}' <<< 'null'
+rjq -n '[inputs]' <<< $'1\n2\n3'
+```
+
+Use `rjq` as a Ruby library:
+
+```ruby
+require "rjq"
+
+Rjq.run(".foo | .[]", {"foo" => [1, 2, 3]}).to_a
+# => [1, 2, 3]
+
+# Optional resource limits for untrusted JSON input and filters:
+Rjq.run_stream(".", io: input_io, opts: {
+  input_chunk_size: 16_384,
+  input_max_depth: 128,
+  max_number_digits: 1_000,
+  max_string_bytes: 1_048_576,
+  max_filter_depth: 128,
+  max_call_depth: 64,
+  max_instructions: 1_000_000,
+  max_replay_cache: 10_000,
+  max_outputs: 10_000,
+  regexp_timeout: 0.1
+}).to_a
+```
+
+`max_number_digits` counts all digits in a JSON number literal, including fractional and exponent digits.
+`max_string_bytes` limits each decoded JSON string (including object keys) by UTF-8 byte size. Both default to
+`nil` (unlimited) for jq compatibility. Invalid or unknown Ruby API options are rejected before compilation or input
+processing. Passing `nil` for `stderr` or `module_resolver` selects the normal process stderr or default module
+resolver, respectively. Option hashes and their `variables` and `library_path` containers are copied before lazy
+execution begins.
+
+Filter nesting defaults to 256. User-function calls in continuation-free tail positions are trampolined, so bounded
+countdown-style recursion does not consume the Ruby stack. `max_call_depth` and `max_instructions` are unlimited by default; the latter counts
+bytecode instructions actually demanded by the consumer. Call and instruction limits are host safety boundaries and
+cannot be caught by jq `try` or `?`. The same three controls are available to the CLI as
+`--max-filter-depth`, `--max-call-depth`, and `--max-instructions`.
+`max_replay_cache` (also available as `--max-replay-cache`) bounds values retained when a generated index filter must
+be replayed for multiple upstream values; it is unlimited by default for jq compatibility.
+
+Inspect compiled bytecode:
+
+```ruby
+puts Rjq.compile(".foo | .[]").disasm
+```
+
+## Compatibility
+
+The current implementation passes the bundled, adapted jq 1.7.1 regression fixtures:
+
+```sh
+ruby script/official_compat.rb
+# checked=447 failures=0
+
+ruby script/official_compat.rb spec/fixtures/jq/onig.test
+# checked=40 failures=0
+```
+
+The same fixtures are integrated into RSpec under `spec/compat`, so CI runs them as normal tests. Expected JSON uses Ruby's independent stdlib parser and comparator, and `%%FAIL` cases check diagnostic categories. The adapted fixtures remain a broad regression corpus rather than proof of complete jq compatibility.
+
+An independent differential suite invokes a checksum-pinned jq 1.7.1 executable and compares stdout bytes, normalized stderr, exit status, output count, and ordering:
+
+```sh
+JQ_BIN=/path/to/jq-1.7.1 bundle exec rake differential
+```
+
+Supported areas include:
+
+- jq values, ordering, truthiness, and numeric edge cases
+- incremental JSON parser and direct writer, including jq-style `NaN`, `Infinity`, and `-0`
+- jq-compatible `--stream` input, including close markers, and `--stream-errors` parse-error arrays
+- field/index/slice access, iteration, pipes, commas, conditionals, `try/catch`, labels and breaks
+- bindings, structured bindings, functions, local `def`, filter arguments, and recursive functions
+- path expressions, assignment/update operators, `del`, `getpath`, `setpath`, `delpaths`
+- core, array, string, math, date/time, format, SQL-style, stream, and regex builtins
+- `jq.test` success cases, `%%FAIL` rejection cases, and `onig.test` success cases
+
+See [COMPATIBILITY.md](COMPATIBILITY.md) for the compatibility contract, known differences, exit statuses, extensions, and accepted Ruby value types.
+
+## Bytecode VM
+
+`Rjq.compile` parses filters into an AST, compiles the executable filter body into a bytecode `Rjq::Program`, then runs that program through `Rjq::VM`.
+
+The VM is a stack machine with explicit instructions such as:
+
+- `load_input`, `load_const`, `field`, `index_const`, `index_filter`, `slice_const`, `each`
+- `path`, `pipe`, `append`, `array`, `object`, `call`, `unary`, `binary`, `branch`
+- `try`, `reduce`, `foreach`, `label`, `break`, `assign`, `scoped_def`, `recurse`
+
+All parsed AST nodes used by the runtime are lowered to bytecode. Path expressions, update assignment,
+`try/catch`, local `def`, recursion, and module metadata are handled by VM instructions and runtime context
+rather than an `eval_node` fallback.
+
+The VM stores instruction results as enumerator-backed streams on the stack. `each`, `pipe`, comma-style append,
+branching, binary cross-products, object construction, `reduce`, `foreach`, local function calls, filter arguments,
+recursion, `range`, `repeat`, `while`, `until`, and input builtins preserve continuations lazily. Array constructors,
+slurp, and aggregating builtins collect only at the jq semantic boundaries that require a complete value. Input files
+and stream events are read incrementally and owned file handles close when downstream evaluation stops early.
+
+See [ARCHITECTURE.md](ARCHITECTURE.md) for execution phases, number semantics, module resolution, streaming, and resource boundaries.
+
+## Development
+
+After checking out the repo, run:
+
+```sh
+bundle install
+bundle exec rake
+```
+
+This runs:
+
+- RSpec, including `spec/compat`
+- `script/compat_probe.rb`
+
+Individual checks:
+
+```sh
+bundle exec rake spec
+bundle exec rake compat
+bundle exec rake differential
+ruby script/official_compat.rb
+ruby script/official_compat.rb spec/fixtures/jq/onig.test
+```
+
+Run the benchmark suite:
+
+```sh
+bundle exec ruby benchmark/jq_compare.rb
+```
+
+If `jq` is available on `PATH`, the benchmark prints a Markdown comparison table and checks stdout equality. Increase the sample size with:
+
+```sh
+ITERATIONS=50 bundle exec ruby benchmark/jq_compare.rb
+```
+
+GitHub Actions runs Ruby 3.1 through 4.0 plus experimental Ruby head, macOS, Windows, musl, minimum dependencies,
+coverage thresholds, the checksum-pinned jq 1.7.1 differential suite, and a built-gem install smoke test. A scheduled
+workflow records median and p95 process benchmarks and rejects output mismatches.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/ydah/rjq. See [CONTRIBUTING.md](CONTRIBUTING.md) and [SECURITY.md](SECURITY.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](LICENSE.txt).

data/SECURITY.md

@@ -0,0 +1,10 @@
+# Security policy
+
+Please report suspected vulnerabilities privately through GitHub's security advisory interface for this repository. Do not open a public issue with exploit details before a fix is available.
+
+Supported releases are the current `main` branch and the latest published gem version.
+
+Security-sensitive boundaries include module path resolution, symlink and traversal checks, incremental JSON parsing, regex resource consumption, cyclic host values, and process capabilities exposed by file, environment, time, and diagnostic builtins.
+
+Current safeguards include canonical module roots, module byte/depth/cycle limits, a 256-level JSON parse limit, cyclic-value rejection, incremental input, optional output budgets, and early file closure. Ruby Regexp can still be exposed to expensive patterns; callers processing untrusted patterns should use process-level time and memory limits.
+

data/bin/rjq

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+lib = File.expand_path('../lib', __dir__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+require 'rjq/cli'
+
+exit Rjq::CLI.new(ARGV, stdin: $stdin, stdout: $stdout, stderr: $stderr).run