janeway-jsonpath 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 10f6ca7766f713b6272f971fd9f6b119895a59385dcebf0cfdb6e0b7d7b27de8
-  data.tar.gz: b33f60624ea2fc41740c888ce00c818c8a873ca56c3d8f85f49b00abee7ea466
+  metadata.gz: 23479bd7bd49f2ac64ce5e8f27399df191efd5c7efe76aaca4fd5615cf429319
+  data.tar.gz: b1dea6ae594f5ab51f0105fb32f81ad03a5028223214f38e01c05cae0fad011b
 SHA512:
-  metadata.gz: 2cf900d4b424ba23d436a6a8d3714e274ad5b6198c3062ec73760d489634105effed9350c665c29673032f899a855ea7d946b85b583b83e3b0d0730ec1e2244f
-  data.tar.gz: 5b28155e3e56346e95f512688cde87310329626d8deb50af76f7ff2d59529f3bb8816b1c62d2bc402266feda338b29bd7197835543c96ea2930d572199cdb293
+  metadata.gz: 46d9ea3bfe2b012d538f53abcad7f29f604ce5c54de51f7f0354d78aa9be4871955cf4666898fdf0ba0e6262147b4d9f29570ca394046b1473597b2c8118f023
+  data.tar.gz: f323676b0cfeb66a2cbbad9331ff782dbd2691d42258ca607d9e6aa4bf511f6cc6e439962d40720b75a0aceee746f5ed224f4c3f4fbd68c29a2d093395807830

data/README.md

@@ -1,64 +1,135 @@
 # Janeway JSONPath parser
 
-### Purpose
-
 This is a [JsonPath](https://goessner.net/articles/JsonPath/) parser.
+It strictly follows [RFC 9535](https://www.rfc-editor.org/rfc/rfc9535.html) and passes the [JSONPath Compliance Test Suite](https://github.com/jsonpath-standard/jsonpath-compliance-test-suite).
 
-It reads a JSON input file and a query.
-It uses the query to find and return a set of matching values from the input.
+It reads a JSON input file and a query, and uses the query to find and return a set of matching values from the input.
 This does for JSON the same job that XPath does for XML.
 
 This project includes:
+
     * command-line tool to run jsonpath queries on a JSON input
     * ruby library to run jsonpath queries on a JSON input
 
-### Goals
+**Contents**
 
-* parse Goessner JSONPath, similar to https://github.com/joshbuddy/jsonpath
-* implement all of [IETF RFC 9535](https://github.com/ietf-wg-jsonpath)
-* don't use regular expressions for parsing, for performance
-* don't use `eval`, which is known to be an attack vector
-* be simple and fast with minimal dependencies
-* use helpful query parse errors which help understand and improve queries, rather than describing issues in the code
-* modern, linted ruby 3 code with frozen string literals
+- [Install](#install)
+- [Usage](#usage)
+- [Related projects](#related-projects)
+- [Goals](#goals)
+- [Non-goals](#non-goals)
 
-### Non-goals
+### Install
 
-* Changing behavior to follow [other implementations](https://cburgmer.github.io/json-path-comparison/)
+Install the gem from the command-line:
+```
+    gem install janeway-jsonpath`
+```
 
-The JSONPath RFC was in draft status for a long time and has seen many changes.
-There are many implementations based on older drafts, or which add features that were never in the RFC at all.
+or add it to your Gemfile:
+
+```
+    gem 'janeway-jsonpath', '~> 0.2.0'
+```
+
+### Usage
+
+#### Janeway command-line tool
+
+Give it a query and some input JSON, and it prints a JSON result.
+Use single quotes around the JSON query to avoid shell interaction.
+Example:
+
+```
+    $ janeway '$..book[?(@.price<10)]' example.json
+    [
+      {
+        "category": "reference",
+        "author": "Nigel Rees",
+        "title": "Sayings of the Century",
+        "price": 8.95
+      },
+      {
+        "category": "fiction",
+        "author": "Herman Melville",
+        "title": "Moby Dick",
+        "isbn": "0-553-21311-3",
+        "price": 8.99
+      }
+    ]
+```
+
+You can also pipe JSON into it:
+```
+    $ cat example.json | janeway '$..book[?(@.price<10)]'
+```
+
+See the help message for more capabilities: `janeway --help`
 
-The goal here is perfect adherence to the finalized [RFC 9535](https://github.com/ietf-wg-jsonpath) rather than adding features that are in other implementations.
+#### Janeway ruby libarary
 
-The RFC was finalized in 2024, and it has a rigorous [suite of compliance tests.](https://github.com/jsonpath-standard/jsonpath-compliance-test-suite)
+Here's an example of using Janeway to execute a JSONPath query in ruby code:
+```ruby
+require 'janeway'
+require 'json'
 
-With these tools it is possible to have JSONPath implementations in many languages with identical behavior.
+data = JSON.parse(File.read(ARGV.first))
+results = Janeway.find_all('$..book[?(@.price<10)]', data)
+```
 
-### Differences from joshbuddy/jsonpath
+Alternatively, compile the query once, and share it between threads or ractors.
 
-The only other serious ruby implementation of jsonpath is joshbuddy/jsonpath.
-This implementation has been around for a long time.
-I personally have used it in a software project for several years.
-Here are differences I've found in porting this application from joshbuddy/jsonpath to janeway.
+The Janeway::AST::Query object is not modified after parsing, so it is easy to freeze and share concurrently.
 
-* joshbuddy/jsonpath allows unquoted strings in filter comparisons.
-Examples:
-$ jsonpath '$.store.book[?(@.category==reference)]' example.json
-$ janeway '$.store.book[?(@.category=="reference")]' example.json
+```ruby
+    # Create ractors with their own data sources
+    ractors =
+      Array.new(4) do |index|
+        Ractor.new(index) do |i|
+          query = receive
+          data = JSON.parse File.read("input-file-#{i}.json")
+          puts query.find_all(data)
+        end
+      end
 
-* joshbuddy/jsonpath allows root selector to be omitted
-grid.first('gid').to_i
+    # Construct JSONPath query object and send it to each ractor
+    query = Janeway.compile('$..book[?(@.price<10)]')
+    ractors.each { |ractor| ractor.send(query).take }
+```
 
-$ jsonpath 'store' example.json
-$ janeway '$.store' example.json
+### Related Projects
 
-*
-$ jsonpath '$.nodes..services..id' spec/resources/grid-full.json
+- [joshbuddy/jsonpath](https://github.com/joshbuddy/jsonpath)
 
-### Implementation
+This is the classic 'jsonpath' ruby gem. It has been around since 2008.
+It is not compliant with RFC 9535, because it was written long before the standard was finalized, but it's a capable and useful parser and has long been the best jsonpath library available for ruby.
+
+See [Porting](PORTING.md) for tips on converting a ruby project from [joshbuddy/jsonpath](https://github.com/joshbuddy/jsonpath) to janeway.
+
+- [JPT - reference implementation based on parsing the ABNF grammar of RFC 9535](https://github.com/cabo/jpt)
+
+Also there are many non-ruby implementations of RFC 9535, here are just a few:
+- [jesse (dart)](https://github.com/f3ath/jessie)
+- [python-jsonpath-rfc9535 (python)](https://github.com/jg-rp/python-jsonpath-rfc9535)
+- [theory/jsonpath (go)](https://github.com/theory/jsonpath)
+
+### Goals
+
+* maintain perfect compliance with [IETF RFC 9535](https://www.rfc-editor.org/rfc/rfc9535.html)
+* raise helpful query parse errors designed to help users understand and improve queries, rather than describing issues in the code
+* don't use regular expressions for parsing, for performance
+* don't use `eval`, which is known to be an attack vector
+* be simple and fast with minimal dependencies
+* provide ruby-like accessors (eg. #each, #delete_if) for processing results
+* modern, linted ruby 3 code with frozen string literals
+
+### Non-goals
+
+* Changing behavior to follow [other implementations](https://cburgmer.github.io/json-path-comparison/)
+
+The JSONPath RFC was in draft status for a long time and has seen many changes.
+There are many implementations based on older drafts, and others which add features that were never in the RFC at all.
 
-Functionality is based on [IETF RFC 9535, "JSONPath: Query Expressions for JSON"](https://www.rfc-editor.org/rfc/rfc9535.html#filter-selector)
-The examples in the RFC have been implemented as unit tests.
+The goal is adherence to the [RFC 9535](https://www.rfc-editor.org/rfc/rfc9535.html) rather than adding features that are in other implementations. This implementation's results are supposed to be identical to other RFC-compliant implementations in [dart](https://github.com/f3ath/jessie), [python](https://github.com/jg-rp/python-jsonpath-rfc9535) and other languages.
 
-For details not covered in the RFC, it does the most reasonable thing based on [what other JSONPath parsers do](https://cburgmer.github.io/json-path-comparison/). However, this is always secondary to following the RFC. Many of the popular behaviors there contradict the RFC, so it has to be one or the other.
+The RFC was finalized in 2024. With the finalized RFC and the rigorous [suite of compliance tests](https://github.com/jsonpath-standard/jsonpath-compliance-test-suite), it is now possible to have JSONPath implementations in many languages with identical behavior.

data/lib/janeway/ast/array_slice_selector.rb

@@ -136,14 +136,16 @@ module Janeway
         [lower, upper]
       end
 
-      # ignores the brackets: argument, this always needs surrounding brackets
+      # @param brackets [Boolean] add surrounding brackets if true
       # @return [String]
-      def to_s(*)
-        if @step
-          "[#{@start}:#{@end}:#{@step}]"
-        else
-          "[#{@start}:#{@end}]"
-        end
+      def to_s(brackets: true, **_ignored)
+        index_str =
+          if @step
+            "#{@start}:#{@end}:#{@step}"
+          else
+            "#{@start}:#{@end}"
+          end
+        brackets ? "[#{index_str}]" : index_str
       end
 
       # @param level [Integer]

data/lib/janeway/ast/binary_operator.rb

@@ -2,19 +2,21 @@
 
 module Janeway
   module AST
+    # AST node for binary operators:
+    #   == != <= >= < > || &&
     class BinaryOperator < Janeway::AST::Expression
-      attr_reader :operator, :left, :right
+      attr_reader :name, :left, :right
 
       def initialize(operator, left = nil, right = nil)
         super(nil)
         raise ArgumentError, "expect symbol, got #{operator.inspect}" unless operator.is_a?(Symbol)
 
-        @operator = operator
+        @name = operator # eg. :equal
         self.left = left if left
         self.right = right if right
       end
 
-      # Set the left-hand-side value
+      # Set the left-hand-side expression
       # @param expr [AST::Expression]
       def left=(expr)
         if comparison_operator? && !(expr.literal? || expr.singular_query?)
@@ -32,7 +34,7 @@ module Janeway
         @left = expr
       end
 
-      # Set the left-hand-side value
+      # Set the left-hand-side expression
       # @param expr [AST::Expression]
       def right=(expr)
         if comparison_operator? && !(expr.literal? || expr.singular_query?)
@@ -52,23 +54,6 @@ module Janeway
         "(#{@left} #{operator_to_s} #{@right})"
       end
 
-      private
-
-      def operator_to_s
-        case operator
-        when :and then '&&'
-        when :equal then '=='
-        when :greater_than then '>'
-        when :greater_than_or_equal then '>='
-        when :less_than then '<'
-        when :less_than_or_equal then '<='
-        when :not_equal then '!='
-        when :or then '||'
-        else
-          raise "unknown binary operator #{operator}"
-        end
-      end
-
       # @param level [Integer]
       # @return [Array]
       def tree(level)
@@ -91,12 +76,29 @@ module Janeway
         operator_type == :logical
       end
 
+      private
+
+      def operator_to_s
+        case name
+        when :and then '&&'
+        when :equal then '=='
+        when :greater_than then '>'
+        when :greater_than_or_equal then '>='
+        when :less_than then '<'
+        when :less_than_or_equal then '<='
+        when :not_equal then '!='
+        when :or then '||'
+        else
+          raise "unknown binary operator #{name}"
+        end
+      end
+
       def operator_type
-        case operator
+        case name
         when :and, :or then :logical
         when :equal, :not_equal, :greater_than, :greater_than_or_equal, :less_than, :less_than_or_equal then :comparison
         else
-          raise "unknown binary operator #{operator}"
+          raise "unknown binary operator #{name}"
         end
       end
     end

data/lib/janeway/ast/current_node.rb

@@ -29,8 +29,11 @@ module Janeway
     #
     # Construct accepts an optional Selector which will be applied to the "current" node
     class CurrentNode < Janeway::AST::Expression
+      # Subsequent expression that modifies the output of this expression
+      attr_accessor :next
+
       def to_s
-        "@#{@value}"
+        "@#{@next}"
       end
 
       # True if this is the root of a singular-query.
@@ -38,10 +41,10 @@ module Janeway
       #
       # @return [Boolean]
       def singular_query?
-        return true unless @value # there are no following selectors
+        return true unless @next # there are no following selectors
 
         selector_types = []
-        selector = @value
+        selector = @next
         loop do
           selector_types << selector.class
           selector = selector.next
@@ -54,13 +57,13 @@ module Janeway
       # True if this is a bare current node operator, without a following expression
       # @return [Boolean]
       def empty?
-        @value.nil?
+        @next.nil?
       end
 
       # @param level [Integer]
       # @return [Array]
       def tree(level)
-        [indented(level, '@'), @value.tree(indent + 1)]
+        [indented(level, '@'), @next.tree(indent + 1)]
       end
     end
   end

data/lib/janeway/ast/descendant_segment.rb

@@ -32,7 +32,7 @@ module Janeway
       # @param level [Integer]
       # @return [Array]
       def tree(level)
-        [indented(level, "..#{@value}"), child&.tree(level + 1)]
+        [indented(level, "..#{@value}"), @next&.tree(level + 1)]
       end
     end
   end

data/lib/janeway/ast/expression.rb

@@ -7,13 +7,23 @@ module Janeway
   module AST
     INDENT = '  '
 
+    # Base class for jsonpath expressions.
+    #
+    # Every AST node is a subclass of this.
+    # This includes selectors, root and child identifiers, descendant segments,
+    # and the nodes that occur within a filter selector such as the current
+    # node identifier, operators and literals.
     class Expression
+      # Value provided by subclass constructor.
       attr_accessor :value
 
+      # Next expression in the AST, if any
+      attr_reader :next
+
       def initialize(val = nil)
         # don't set the instance variable if unused, because it makes the
         # "#inspect" output cleaner in rspec test failures
-        @value = val unless val.nil?  # false must be stored though!
+        @value = val unless val.nil? # false must be stored though!
       end
 
       # @return [String]

data/lib/janeway/ast/filter_selector.rb

@@ -57,7 +57,7 @@ module Janeway
       # @return [Array]
       def tree(level)
         if @next
-          [indented(level, to_s), indented(level + 1, @next&.tree)]
+          [indented(level, to_s), @next.tree(level + 1)]
         else
           [indented(level, to_s)]
         end

data/lib/janeway/ast/helpers.rb

@@ -2,6 +2,7 @@
 
 module Janeway
   module AST
+    # Helper methods for AST Expressions.
     module Helpers
       # @param str [String] ascii string, CamelCase
       # @return [String] lowercase, with underscores

data/lib/janeway/ast/index_selector.rb

@@ -17,9 +17,8 @@ module Janeway
       end
 
       # @param brackets [Boolean] include brackets around selector
-      # ** ignores keyword arguments that don't apply to this selector
-      def to_s(brackets: true, **)
-        brackets ? "[#{@value}]" : @value.to_s
+      def to_s(brackets: true, **_ignored)
+        brackets ? "[#{@value}]#{@next}" : "#{@value}#{@next}"
       end
     end
   end

data/lib/janeway/ast/name_selector.rb

@@ -17,8 +17,8 @@ module Janeway
         raise "Invalid name: #{value.inspect}:#{value.class}" unless value.is_a?(String)
       end
 
-      # @param bracket [Boolean] request for bracket syntax
-      # @param bracket [Boolean] include . prefix, if shorthand notation is used
+      # @param brackets [Boolean] request for bracket syntax
+      # @param dot_prefix [Boolean] include . prefix, if shorthand notation is used
       def to_s(brackets: false, dot_prefix: true)
         # Add quotes and surrounding brackets if the name includes chars that require quoting.
         # These chars are not allowed in dotted notation, only bracket notation.

data/lib/janeway/ast/null.rb

@@ -2,6 +2,7 @@
 
 module Janeway
   module AST
+    # Represents JSON null literal within a filter expression
     class Null < Janeway::AST::Expression
       def to_s
         'null'

data/lib/janeway/ast/query.rb

@@ -32,13 +32,54 @@ module Janeway
         Janeway::Interpreter.new(self).interpret(input)
       end
 
+      # Iterate through each value matched by the JSONPath query.
+      #
+      # @param input [Hash, Array] ruby object to be searched
+      # @yieldparam [Object] matched value
+      # @return [void]
+      def each(input, &block)
+        return enum_for(:each, input) unless block_given?
+
+        interpreter = Janeway::Interpreter.new(self)
+        interpreter.push Janeway::Interpreters::Yielder.new(&block)
+        interpreter.interpret(input)
+      end
+
       def to_s
         @root.to_s
       end
 
+      # Return a list of all the nodes in the AST.
+      # The AST of a jsonpath query is a straight line, so this is expressible as an array.
+      # The only part of the AST with branches is inside a filter selector, but that doesn't show up here.
+      # @return [Array<Expression>]
+      def node_list
+        nodes = []
+        node = @root
+        loop do
+          nodes << node
+          break unless node.next
+
+          node = node.next
+        end
+        nodes
+      end
+
+      # Remove the last selector in the query from the node list, and return the removed selector.
+      # @return [AST::Selector, nil] last selector in the query, if any
+      def pop
+        nodes = node_list
+        return nil if node_list.size == 1 # only 1 node, don't pop
+
+        # Remove the last selector and return it
+        last_node = nodes.pop
+        nodes.last.next = nil # delete the second-last node's link to the last node
+        last_node
+      end
+
       # Queries are considered equal if their ASTs evaluate to the same JSONPath string.
       #
-      # The string output is generated by the AST and should be considered a "normalized"
+      # The string output is generated from the AST and should be considered a "normalized"
       # form of the query. It may have different whitespace and parentheses than the original
       # input but will be semantically equivalent.
       def ==(other)

data/lib/janeway/ast/root_node.rb

@@ -14,8 +14,11 @@ module Janeway
     #   $(? $.key1 == $.key2 )
     #
     class RootNode < Janeway::AST::Expression
+      # Subsequent expression that modifies the output of this expression
+      attr_accessor :next
+
       def to_s
-        "$#{@value}"
+        "$#{@next}"
       end
 
       # True if this is the root of a singular-query.
@@ -23,13 +26,13 @@ module Janeway
       #
       # @return [Boolean]
       def singular_query?
-        return true unless @value # there are no following selectors
+        return true unless @next # there are no following selectors
 
         selector_types = []
-        selector = @value
+        selector = @next
         loop do
           selector_types << selector.class
-          selector = selector.next
+          selector = selector&.next
           break unless selector
         end
         allowed = [AST::IndexSelector, AST::NameSelector]
@@ -39,7 +42,7 @@ module Janeway
       # @param level [Integer]
       # @return [Array]
       def tree(level = 0)
-        [indented(level, '$'), @value.tree(level + 1)]
+        [indented(level, '$'), @next.tree(level + 1)]
       end
     end
   end

data/lib/janeway/ast/string_type.rb

@@ -2,6 +2,8 @@
 
 module Janeway
   module AST
+    # Represents a string literal within a filter expression.
+    # May be used within a comparison or a jsonpath function call.
     class StringType < Janeway::AST::Expression
       def to_s
         if @value.include?("'")

data/lib/janeway/ast/unary_operator.rb

@@ -4,11 +4,11 @@ module Janeway
   module AST
     # Represent unary operators "!", "-"
     class UnaryOperator < Janeway::AST::Expression
-      attr_accessor :operator, :operand
+      attr_accessor :name, :operand
 
       def initialize(operator, operand = nil)
         super()
-        @operator = operator
+        @name = operator # eg. :not
         @operand = operand
       end
 

data/lib/janeway/ast/wildcard_selector.rb

@@ -22,6 +22,7 @@ module Janeway
         @next = nil
       end
 
+      # @return [String]
       def to_s(brackets: false, dot_prefix: true)
         if brackets
           "[*]#{@next&.to_s(brackets: true)}"

data/lib/janeway/error.rb

@@ -22,7 +22,7 @@ module Janeway
 
     def detailed_message
       msg = "Error: #{message}\nQuery: #{query}\n"
-      msg += (' ' * location.col) + '^' if @location
+      msg += "#{' ' * location.col}^" if @location
       msg
     end
   end

data/lib/janeway/functions/count.rb

@@ -27,16 +27,14 @@ module Janeway
       arg = parse_function_parameter
       parameters = [arg]
       raise Error, "Invalid parameter - count() expects node list, got #{arg.value.inspect}" if arg.literal?
-      unless current.type == :group_end
-        raise Error, 'Too many parameters for count() function call'
-      end
+      raise Error, 'Too many parameters for count() function call' unless current.type == :group_end
 
       # Define function body
       AST::Function.new('count', parameters) do |node_list|
         if node_list.is_a?(Array)
           node_list.size
         else
-          1 # The count of a non-empty singulare noselist such as count(@) is always 1
+          1 # the count of a non-empty singular nodelist such as count(@) is always 1.
         end
       end
     end

data/lib/janeway/functions/length.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 module Janeway
+  # Mixin to provide JSONPath function handlers for Parser
   module Functions
     # The length() function extension provides a way to compute the length of a value
     # and make that available for further processing in the filter expression:
@@ -18,9 +19,7 @@ module Janeway
       unless arg.singular_query? || arg.literal?
         raise Error, "Invalid parameter - length() expects literal value or singular query, got #{arg.value.inspect}"
       end
-      unless current.type == :group_end
-        raise Error, 'Too many parameters for length() function call'
-      end
+      raise Error, 'Too many parameters for length() function call' unless current.type == :group_end
 
       # Meaning of return value depends on the JSON type:
       #   * string - number of Unicode scalar values in the string.

data/lib/janeway/functions/match.rb

@@ -41,22 +41,16 @@ module Janeway
       # Read first parameter
       parameters = []
       parameters << parse_function_parameter
+      raise Error, 'Not enough parameters for match() function call' unless current.type == :union
 
-      # Consume comma
-      if current.type == :union
-        consume # ,
-      else
-        raise Error, 'Not enough parameters for match() function call'
-      end
+      consume # ,
 
       # Read second parameter (the regexp)
       # This could be a string, in which case it is available now.
       # Otherwise it is an expression that takes the regexp from the input document,
       # and the iregexp will not be available until interpretation.
       parameters << parse_function_parameter
-      unless current.type == :group_end
-        raise Error, 'Too many parameters for match() function call'
-      end
+      raise Error, 'Too many parameters for match() function call' unless current.type == :group_end
 
       AST::Function.new('match', parameters) do |str, str_iregexp|
         if str.is_a?(String) && str_iregexp.is_a?(String)