sparkql 1.3.6 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -0
- data/README.md +17 -0
- data/VERSION +1 -1
- data/lib/sparkql/evaluator.rb +34 -0
- data/lib/sparkql/parser_tools.rb +171 -27
- data/test/unit/evaluator_test.rb +23 -0
- data/test/unit/parse_tree_test.rb +130 -0
- data/test/unit/parser_test.rb +22 -0
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: aa4fb1fb72c6db01168e3b411de7fb2e1e409d6446ccb9cf34c8d5e486f7df47
|
|
4
|
+
data.tar.gz: f49564f84f78cb5deb97cdbd34c2bd177c47173f288ec841b15580b380a3be9e
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 23aac5651ae1e314d79063c47f790c5fa4cbd63945d1ed390ca61fc89b747850ce8db1fe73a3a594b543a4fc225084481f5da75cdd0e817ff5810bbc88f6ca4c
|
|
7
|
+
data.tar.gz: f13283590cf799878cc1d1f064d93afeaab5c86f2f10e17abcf5bc9d094a5e3fd2f0e0830f8fba1095ebc902fc55ced385571fa451084be39ffc3348b777f7ca
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,14 @@
|
|
|
1
|
+
v1.4.0 2026-10-13
|
|
2
|
+
-------------------
|
|
3
|
+
* [IMPROVEMENT] `Parser#parse_tree` returns the filter as a syntax tree, with And/Not
|
|
4
|
+
binding tighter than Or. `Parser#parse` is now built by flattening that tree; its
|
|
5
|
+
output is unchanged.
|
|
6
|
+
* [CHANGE] The grammar actions `tokenize_expression`, `tokenize_conjunction`,
|
|
7
|
+
`tokenize_unary_conjunction`, and `tokenize_group` now return tree nodes instead of
|
|
8
|
+
expression arrays. Only relevant to subclasses overriding them (none known).
|
|
9
|
+
* [DOCS] Documented two Evaluator bugs with nested groups (see Sparkql::Evaluator), with
|
|
10
|
+
pending tests.
|
|
11
|
+
|
|
1
12
|
v1.3.6, 2026-06-23
|
|
2
13
|
* [BUGFIX] Parser crash when days()/weekdays() exceed max offset
|
|
3
14
|
|
data/README.md
CHANGED
|
@@ -39,6 +39,23 @@ The return value will be an array with one expression element containing the que
|
|
|
39
39
|
# ...
|
|
40
40
|
}
|
|
41
41
|
|
|
42
|
+
That flat list describes nesting through each expression's `:level`, `:block_group`,
|
|
43
|
+
`:conjunction`, and `:unary` keys, which can't express every filter's structure (e.g. the
|
|
44
|
+
middle group in `(A) And ((B) Or (C))`). When a consumer needs the real structure, use
|
|
45
|
+
`parse_tree`, which returns a syntax tree instead:
|
|
46
|
+
|
|
47
|
+
tree = Parser.new.parse_tree("A Eq 1 Or B Eq 2 And C Eq 3")
|
|
48
|
+
# { node: :conjunction, conjunction: "Or",
|
|
49
|
+
# left: { node: :expression, expression: { field: "A", ... } },
|
|
50
|
+
# right: { node: :conjunction, conjunction: "And",
|
|
51
|
+
# left: { node: :expression, expression: { field: "B", ... } },
|
|
52
|
+
# right: { node: :expression, expression: { field: "C", ... } } } }
|
|
53
|
+
|
|
54
|
+
Nodes are `:expression` (a comparison -- the same Hash `parse` returns), `:conjunction`
|
|
55
|
+
(`And`/`Or`/`Not`, where `Not` means "And Not"), `:unary` (`Not`), and `:group`
|
|
56
|
+
(parentheses). And/Not bind tighter than Or, as in SQL. `parse` is itself built by
|
|
57
|
+
flattening this tree, so the two always agree. See test/unit/parse_tree_test.rb.
|
|
58
|
+
|
|
42
59
|
|
|
43
60
|
Development
|
|
44
61
|
-------------
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
1.
|
|
1
|
+
1.4.0
|
data/lib/sparkql/evaluator.rb
CHANGED
|
@@ -4,6 +4,40 @@
|
|
|
4
4
|
# Namely, this class will handle all the nesting, boolean algebra, and dropped
|
|
5
5
|
# fields. Plus, it has some optimizations built in to skip the processing for
|
|
6
6
|
# any expressions that don't contribute to the net result of the filter.
|
|
7
|
+
#
|
|
8
|
+
# == KNOWN ISSUES -- read before relying on this for nested filters
|
|
9
|
+
#
|
|
10
|
+
# Used by prospector's SparkqlReductor to decide whether a listing matches a
|
|
11
|
+
# saved search (newsfeeds, listing matching), so a wrong answer here means a
|
|
12
|
+
# listing matches in prospector but not in search, or vice versa.
|
|
13
|
+
#
|
|
14
|
+
# This class reconstructs the filter's structure from Parser#parse's *flat*
|
|
15
|
+
# expression list (:level, :block_group, :conjunction, :unary, ...), and that
|
|
16
|
+
# list can't always describe the structure. Two confirmed bugs follow (see the
|
|
17
|
+
# pending tests in test/unit/evaluator_test.rb):
|
|
18
|
+
#
|
|
19
|
+
# 1. Sibling groups nested in a group with no expression of its own are
|
|
20
|
+
# attached to the wrong parent:
|
|
21
|
+
# "(A) And ((B) Or (C))" with A=false, B=true, C=true => true (should be false)
|
|
22
|
+
# The middle group never appears in the flat list, and :block_group is the
|
|
23
|
+
# most recently *opened* group, not the containing one, so #build_structures
|
|
24
|
+
# files (B)'s block under (A)'s.
|
|
25
|
+
#
|
|
26
|
+
# 2. A leading unary Not on a group of two or more expressions negates only
|
|
27
|
+
# the group's first expression:
|
|
28
|
+
# "Not (A Or B) And C" is evaluated as "((Not A) Or B) And C"
|
|
29
|
+
# The flat list records the Not on A with :unary_level 0 (outside the
|
|
30
|
+
# group), but #process_structures applies :unary per expression and
|
|
31
|
+
# ignores :unary_level. "Not (A)" -- a single expression -- is unaffected.
|
|
32
|
+
#
|
|
33
|
+
# And/Or precedence is *not* an issue: evaluating left to right, but stopping
|
|
34
|
+
# at an Or once the result is true (and otherwise taking the Or's right side
|
|
35
|
+
# afresh), gives the same result as SQL's And-before-Or.
|
|
36
|
+
#
|
|
37
|
+
# The fix is to evaluate Parser#parse_tree's syntax tree instead, which
|
|
38
|
+
# represents both cases exactly -- a recursive walk over its
|
|
39
|
+
# expression/conjunction/unary/group nodes, keeping this class's :drop and
|
|
40
|
+
# short-circuit semantics.
|
|
7
41
|
class Sparkql::Evaluator
|
|
8
42
|
include Sparkql::Token
|
|
9
43
|
|
data/lib/sparkql/parser_tools.rb
CHANGED
|
@@ -9,16 +9,66 @@ module Sparkql::ParserTools
|
|
|
9
9
|
GROUP = 'Group'.freeze
|
|
10
10
|
NEGATION = 'Negation'.freeze
|
|
11
11
|
|
|
12
|
+
##
|
|
13
|
+
# Parses a filter into the flat expression list every parser consumer
|
|
14
|
+
# (Parser#tokenize, Evaluator, the API's SQL parsers) has always used: one
|
|
15
|
+
# Hash per comparison, in source order, with nesting described by each
|
|
16
|
+
# expression's :level, :block_group, :conjunction, :conjunction_level,
|
|
17
|
+
# :unary, and :unary_level keys.
|
|
18
|
+
#
|
|
19
|
+
# Built by flattening the same syntax tree #parse_tree returns, so the two
|
|
20
|
+
# always agree; see #flatten_tree for how the flat metadata is derived.
|
|
21
|
+
#
|
|
22
|
+
# @return [Array<Hash>, nil, 0] the expressions; nil on a syntax error; 0
|
|
23
|
+
# for an empty filter
|
|
12
24
|
def parse(str)
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
results = do_parse
|
|
16
|
-
return if results.nil?
|
|
25
|
+
tree = build_tree(str)
|
|
26
|
+
return tree unless tree.is_a?(Hash)
|
|
17
27
|
|
|
28
|
+
results = flatten_tree(tree)
|
|
18
29
|
validate_expressions results
|
|
19
30
|
results
|
|
20
31
|
end
|
|
21
32
|
|
|
33
|
+
##
|
|
34
|
+
# Parses a filter into a syntax tree, for consumers that need the filter's
|
|
35
|
+
# real structure -- the flat #parse list can't always be turned back into
|
|
36
|
+
# it (e.g. "(A) And ((B) Or (C))": the middle group has no expression of
|
|
37
|
+
# its own, so nothing in the list records it).
|
|
38
|
+
#
|
|
39
|
+
# Nodes are Hashes keyed by :node --
|
|
40
|
+
# { node: :expression, expression: <Hash> } -- a comparison; the same
|
|
41
|
+
# Hash #parse would return
|
|
42
|
+
# { node: :conjunction, conjunction: "And" | "Or" | "Not",
|
|
43
|
+
# conjunction_level:, left: <node>, right: <node> }
|
|
44
|
+
# { node: :unary, unary: "Not", unary_level:, operand: <node> }
|
|
45
|
+
# { node: :group, operand: <node> } -- parentheses
|
|
46
|
+
#
|
|
47
|
+
# A "Not" conjunction means "And Not": "A Not B" is A AND NOT B. And/Not
|
|
48
|
+
# bind tighter than Or, as in SQL -- "A Or B And C" is A OR (B AND C) --
|
|
49
|
+
# and chained conjunctions of the same strength group left to right. A
|
|
50
|
+
# unary Not applies to the single comparison or group that follows it.
|
|
51
|
+
#
|
|
52
|
+
# Errors are reported exactly as #parse reports them (see #errors).
|
|
53
|
+
#
|
|
54
|
+
# @return [Hash, nil] the root node; nil for an empty filter or a syntax error
|
|
55
|
+
def parse_tree(str)
|
|
56
|
+
tree = build_tree(str)
|
|
57
|
+
return unless tree.is_a?(Hash)
|
|
58
|
+
|
|
59
|
+
# Annotates each leaf with #parse's flat metadata, so a tree's leaves
|
|
60
|
+
# match #parse's output. Must run on the raw tree, before
|
|
61
|
+
# #apply_precedence, and only once (see #flatten_tree).
|
|
62
|
+
expressions = flatten_tree(tree)
|
|
63
|
+
|
|
64
|
+
# Records the same fatal "maximum expression count" error #parse does.
|
|
65
|
+
# validate_expressions also truncates the list it's given, which only
|
|
66
|
+
# matters for #parse's return value -- here the list is discarded, so the
|
|
67
|
+
# tree keeps every expression; callers must check #errors either way.
|
|
68
|
+
validate_expressions(expressions)
|
|
69
|
+
apply_precedence(tree)
|
|
70
|
+
end
|
|
71
|
+
|
|
22
72
|
def next_token
|
|
23
73
|
t = @lexer.shift
|
|
24
74
|
t = @lexer.shift while (t[0] == :SPACE) || (t[0] == :NEWLINE)
|
|
@@ -91,36 +141,23 @@ module Sparkql::ParserTools
|
|
|
91
141
|
message: "Operator not supported for this type and value string", status: :fatal)
|
|
92
142
|
end
|
|
93
143
|
@expression_count += 1
|
|
94
|
-
|
|
144
|
+
{ node: :expression, expression: expression }
|
|
95
145
|
end
|
|
96
146
|
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
147
|
+
# Grammar action for "expressions CONJUNCTION expression". The grammar
|
|
148
|
+
# chains conjunctions left to right; #apply_precedence regroups them.
|
|
149
|
+
# @lexer.level here is the level the conjunction itself sits at.
|
|
150
|
+
def tokenize_conjunction(left, conj, right)
|
|
151
|
+
{ node: :conjunction, conjunction: conj, conjunction_level: @lexer.level, left: left, right: right }
|
|
101
152
|
end
|
|
102
153
|
|
|
103
|
-
def tokenize_unary_conjunction(conj,
|
|
104
|
-
|
|
105
|
-
# begins with a unary operator, and is nested, such as:
|
|
106
|
-
# Not (Not Field Eq 1)
|
|
107
|
-
# In this instance we treat the outer unary as a conjunction. With any other
|
|
108
|
-
# expression this would be the case, so that should make processing
|
|
109
|
-
# consistent.
|
|
110
|
-
if exp.first[:unary] && @lexer.level.zero?
|
|
111
|
-
exp.first[:conjunction] = conj
|
|
112
|
-
exp.first[:conjunction_level] = @lexer.level
|
|
113
|
-
else
|
|
114
|
-
exp.first[:unary] = conj
|
|
115
|
-
exp.first[:unary_level] = @lexer.level
|
|
116
|
-
end
|
|
117
|
-
|
|
118
|
-
exp
|
|
154
|
+
def tokenize_unary_conjunction(conj, operand)
|
|
155
|
+
{ node: :unary, unary: conj, unary_level: @lexer.level, operand: operand }
|
|
119
156
|
end
|
|
120
157
|
|
|
121
|
-
def tokenize_group(
|
|
158
|
+
def tokenize_group(operand)
|
|
122
159
|
@lexer.leveldown
|
|
123
|
-
|
|
160
|
+
{ node: :group, operand: operand }
|
|
124
161
|
end
|
|
125
162
|
|
|
126
163
|
def tokenize_arithmetic_group(lhs)
|
|
@@ -374,6 +411,16 @@ module Sparkql::ParserTools
|
|
|
374
411
|
end
|
|
375
412
|
end
|
|
376
413
|
|
|
414
|
+
# Records a fatal error past max_expressions and truncates results in
|
|
415
|
+
# place. Unlike the value/argument limits below, which trim mid-parse and
|
|
416
|
+
# so cap later work, this runs after parsing: the truncation only shortens
|
|
417
|
+
# the list #parse returns. The error is fatal, so callers shouldn't use
|
|
418
|
+
# the results anyway; kept for backward compatibility.
|
|
419
|
+
#
|
|
420
|
+
# TODO(2.0): stop slicing results here -- record the error and leave the
|
|
421
|
+
# list whole. A behavior change for #parse's return value (on an already
|
|
422
|
+
# fatal parse), so it waits for a major release. The value/argument
|
|
423
|
+
# slicing in validate_multiple_values/validate_multiple_arguments stays.
|
|
377
424
|
def validate_expressions(results)
|
|
378
425
|
if results.size > max_expressions
|
|
379
426
|
compile_error(token: results[max_expressions][:field], expression: results[max_expressions],
|
|
@@ -383,6 +430,7 @@ module Sparkql::ParserTools
|
|
|
383
430
|
end
|
|
384
431
|
end
|
|
385
432
|
|
|
433
|
+
# TODO(2.0): stop slicing results here
|
|
386
434
|
def validate_multiple_values(values)
|
|
387
435
|
values = Array(values)
|
|
388
436
|
if values.size > max_values
|
|
@@ -393,6 +441,7 @@ module Sparkql::ParserTools
|
|
|
393
441
|
end
|
|
394
442
|
end
|
|
395
443
|
|
|
444
|
+
# TODO(2.0): stop slicing results here
|
|
396
445
|
def validate_multiple_arguments(args)
|
|
397
446
|
args = Array(args)
|
|
398
447
|
if args.size > max_values
|
|
@@ -463,6 +512,101 @@ module Sparkql::ParserTools
|
|
|
463
512
|
|
|
464
513
|
private
|
|
465
514
|
|
|
515
|
+
# Runs the grammar, whose actions (#tokenize_expression,
|
|
516
|
+
# #tokenize_conjunction, #tokenize_unary_conjunction, #tokenize_group)
|
|
517
|
+
# build the raw tree: conjunctions chained left to right, before
|
|
518
|
+
# #apply_precedence.
|
|
519
|
+
# @return [Hash, nil, 0] the root node; nil on a syntax error; 0 when empty
|
|
520
|
+
def build_tree(str)
|
|
521
|
+
@lexer = Sparkql::Lexer.new(str)
|
|
522
|
+
@expression_count = 0
|
|
523
|
+
do_parse
|
|
524
|
+
end
|
|
525
|
+
|
|
526
|
+
##
|
|
527
|
+
# Flattens a raw tree (straight from #build_tree, before #apply_precedence)
|
|
528
|
+
# into #parse's expression list, annotating each expression Hash in place.
|
|
529
|
+
#
|
|
530
|
+
# :level and :block_group were already set from the lexer as each
|
|
531
|
+
# expression was parsed. The conjunction and unary keys are applied here in
|
|
532
|
+
# the order the grammar reduces them -- bottom-up, left to right -- because
|
|
533
|
+
# the unary rule depends on what's already been applied: a Not in front of
|
|
534
|
+
# a group whose first expression already has a unary Not becomes that
|
|
535
|
+
# expression's :conjunction instead, e.g. "Not (Not Field Eq 1)".
|
|
536
|
+
#
|
|
537
|
+
# Annotates in place, so must run exactly once per parse.
|
|
538
|
+
def flatten_tree(node)
|
|
539
|
+
case node[:node]
|
|
540
|
+
when :expression
|
|
541
|
+
[node[:expression]]
|
|
542
|
+
when :group
|
|
543
|
+
flatten_tree(node[:operand])
|
|
544
|
+
when :unary
|
|
545
|
+
expressions = flatten_tree(node[:operand])
|
|
546
|
+
first = expressions.first
|
|
547
|
+
if first[:unary] && node[:unary_level].zero?
|
|
548
|
+
first[:conjunction] = node[:unary]
|
|
549
|
+
first[:conjunction_level] = node[:unary_level]
|
|
550
|
+
else
|
|
551
|
+
first[:unary] = node[:unary]
|
|
552
|
+
first[:unary_level] = node[:unary_level]
|
|
553
|
+
end
|
|
554
|
+
expressions
|
|
555
|
+
when :conjunction
|
|
556
|
+
left = flatten_tree(node[:left])
|
|
557
|
+
right = flatten_tree(node[:right])
|
|
558
|
+
right.first[:conjunction] = node[:conjunction]
|
|
559
|
+
right.first[:conjunction_level] = node[:conjunction_level]
|
|
560
|
+
left + right
|
|
561
|
+
end
|
|
562
|
+
end
|
|
563
|
+
|
|
564
|
+
##
|
|
565
|
+
# Regroups the grammar's left-to-right conjunction chains so And/Not bind
|
|
566
|
+
# tighter than Or: "A Or B And C" becomes Or(A, And(B, C)). Returns new
|
|
567
|
+
# conjunction/unary/group nodes; expression nodes (and their Hashes) are
|
|
568
|
+
# shared with the input.
|
|
569
|
+
#
|
|
570
|
+
# The grammar's right operand is never itself a conjunction (only a
|
|
571
|
+
# comparison or a group), so a chain is the run of conjunctions down the
|
|
572
|
+
# left side of the tree.
|
|
573
|
+
def apply_precedence(node)
|
|
574
|
+
case node[:node]
|
|
575
|
+
when :conjunction
|
|
576
|
+
operators = []
|
|
577
|
+
operands = []
|
|
578
|
+
current = node
|
|
579
|
+
while current[:node] == :conjunction
|
|
580
|
+
operators.unshift(current)
|
|
581
|
+
operands.unshift(apply_precedence(current[:right]))
|
|
582
|
+
current = current[:left]
|
|
583
|
+
end
|
|
584
|
+
operands.unshift(apply_precedence(current))
|
|
585
|
+
|
|
586
|
+
or_operators = []
|
|
587
|
+
or_terms = []
|
|
588
|
+
term = operands.first
|
|
589
|
+
operators.each_with_index do |operator, i|
|
|
590
|
+
if operator[:conjunction] == Sparkql::Token::OR
|
|
591
|
+
or_operators << operator
|
|
592
|
+
or_terms << term
|
|
593
|
+
term = operands[i + 1]
|
|
594
|
+
else
|
|
595
|
+
term = operator.merge(left: term, right: operands[i + 1])
|
|
596
|
+
end
|
|
597
|
+
end
|
|
598
|
+
or_terms << term
|
|
599
|
+
|
|
600
|
+
or_terms.drop(1).each_with_index.inject(or_terms.first) do |left, (right, i)|
|
|
601
|
+
or_operators[i].merge(left: left, right: right)
|
|
602
|
+
end
|
|
603
|
+
when :unary, :group
|
|
604
|
+
node.merge(operand: apply_precedence(node[:operand]))
|
|
605
|
+
else
|
|
606
|
+
node
|
|
607
|
+
end
|
|
608
|
+
end
|
|
609
|
+
|
|
466
610
|
def report_function_resolver_errors(resolver)
|
|
467
611
|
resolver.errors.each do |error|
|
|
468
612
|
tokenizer_error(token: error.token || @lexer.last_field,
|
data/test/unit/evaluator_test.rb
CHANGED
|
@@ -189,6 +189,29 @@ class EvaluatorTest < Test::Unit::TestCase
|
|
|
189
189
|
assert_equal 3, @evaluator.processed_count
|
|
190
190
|
end
|
|
191
191
|
|
|
192
|
+
# And/Or precedence matches SQL's (And before Or), not just left to right.
|
|
193
|
+
def test_and_before_or
|
|
194
|
+
assert sample("Test Eq true Or Test Eq true And Test Eq false")
|
|
195
|
+
assert sample("Test Eq false And Test Eq false Or Test Eq true")
|
|
196
|
+
assert !sample("Test Eq false Or Test Eq true And Test Eq false")
|
|
197
|
+
assert sample("Test Eq true And Test Eq false Or Test Eq true And Test Eq true")
|
|
198
|
+
end
|
|
199
|
+
|
|
200
|
+
# Known issue 1 in Sparkql::Evaluator's class comment. These pend while the
|
|
201
|
+
# bug exists, and fail once it's fixed -- move them to real tests then.
|
|
202
|
+
def test_known_issue_sibling_groups_in_an_expressionless_group
|
|
203
|
+
pend("Evaluator mis-nests sibling groups -- see Sparkql::Evaluator KNOWN ISSUES") do
|
|
204
|
+
assert !sample("(Test Eq false) And ((Test Eq true) Or (Test Eq true))")
|
|
205
|
+
end
|
|
206
|
+
end
|
|
207
|
+
|
|
208
|
+
# Known issue 2 in Sparkql::Evaluator's class comment.
|
|
209
|
+
def test_known_issue_unary_not_on_a_multi_expression_group
|
|
210
|
+
pend("Evaluator negates only the group's first expression -- see Sparkql::Evaluator KNOWN ISSUES") do
|
|
211
|
+
assert !sample("Not (Test Eq true Or Test Eq true) And Test Eq true")
|
|
212
|
+
end
|
|
213
|
+
end
|
|
214
|
+
|
|
192
215
|
# Here's some examples from prospector's tests that have been simplified a bit.
|
|
193
216
|
def test_advanced
|
|
194
217
|
assert !sample("MlsStatus Eq false And PropertyType Eq true And (City Eq true Or City Eq false)")
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
require 'test_helper'
|
|
2
|
+
|
|
3
|
+
class ParseTreeTest < Test::Unit::TestCase
|
|
4
|
+
include Sparkql
|
|
5
|
+
|
|
6
|
+
def setup
|
|
7
|
+
@parser = Parser.new
|
|
8
|
+
end
|
|
9
|
+
|
|
10
|
+
# Renders a tree compactly by field name, e.g. "Or[A, And[B, C]]".
|
|
11
|
+
def shape(node)
|
|
12
|
+
case node[:node]
|
|
13
|
+
when :expression then node[:expression][:field]
|
|
14
|
+
when :group then "(#{shape(node[:operand])})"
|
|
15
|
+
when :unary then "#{node[:unary]} #{shape(node[:operand])}"
|
|
16
|
+
when :conjunction then "#{node[:conjunction]}[#{shape(node[:left])}, #{shape(node[:right])}]"
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def assert_shape(expected, filter)
|
|
21
|
+
tree = @parser.parse_tree(filter)
|
|
22
|
+
assert !@parser.errors?, "#{filter}: #{@parser.errors.map(&:message)}"
|
|
23
|
+
assert_equal expected, shape(tree), filter
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def leaves(node)
|
|
27
|
+
case node[:node]
|
|
28
|
+
when :expression then [node[:expression]]
|
|
29
|
+
when :group, :unary then leaves(node[:operand])
|
|
30
|
+
when :conjunction then leaves(node[:left]) + leaves(node[:right])
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def test_single_expression
|
|
35
|
+
tree = @parser.parse_tree('Test Eq 10')
|
|
36
|
+
|
|
37
|
+
assert_equal :expression, tree[:node]
|
|
38
|
+
assert_equal 'Test', tree[:expression][:field]
|
|
39
|
+
assert_equal 'Eq', tree[:expression][:operator]
|
|
40
|
+
assert_equal '10', tree[:expression][:value]
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
def test_conjunctions_of_one_kind_group_left_to_right
|
|
44
|
+
assert_shape 'And[And[A, B], C]', 'A Eq 1 And B Eq 1 And C Eq 1'
|
|
45
|
+
assert_shape 'Or[Or[A, B], C]', 'A Eq 1 Or B Eq 1 Or C Eq 1'
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def test_and_binds_tighter_than_or
|
|
49
|
+
assert_shape 'Or[A, And[B, C]]', 'A Eq 1 Or B Eq 1 And C Eq 1'
|
|
50
|
+
assert_shape 'Or[And[A, B], C]', 'A Eq 1 And B Eq 1 Or C Eq 1'
|
|
51
|
+
assert_shape 'Or[And[A, B], And[C, D]]', 'A Eq 1 And B Eq 1 Or C Eq 1 And D Eq 1'
|
|
52
|
+
assert_shape 'Or[Or[A, And[B, C]], D]', 'A Eq 1 Or B Eq 1 And C Eq 1 Or D Eq 1'
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
# A binary Not means "And Not", so it binds like And.
|
|
56
|
+
def test_binary_not_binds_like_and
|
|
57
|
+
assert_shape 'Or[A, Not[B, C]]', 'A Eq 1 Or B Eq 1 Not C Eq 1'
|
|
58
|
+
assert_shape 'Or[Not[And[A, B], C], D]', 'A Eq 1 And B Eq 1 Not C Eq 1 Or D Eq 1'
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def test_groups_override_precedence
|
|
62
|
+
assert_shape 'And[(Or[A, B]), C]', '(A Eq 1 Or B Eq 1) And C Eq 1'
|
|
63
|
+
assert_shape 'Or[A, And[B, (Or[C, And[D, E]])]]', 'A Eq 1 Or B Eq 1 And (C Eq 1 Or D Eq 1 And E Eq 1)'
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
# The flat #parse list can't represent these: the middle group has no
|
|
67
|
+
# expression of its own, and block_group is the most recently opened group
|
|
68
|
+
# rather than the containing one.
|
|
69
|
+
def test_sibling_groups_keep_their_structure
|
|
70
|
+
assert_shape 'And[(A), (Or[(B), (C)])]', '(A Eq 1) And ((B Eq 1) Or (C Eq 1))'
|
|
71
|
+
assert_shape 'Or[(And[A, (B)]), (And[(C), D])]', '(A Eq 1 And (B Eq 1)) Or ((C Eq 1) And D Eq 1)'
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
def test_unary_not_applies_to_the_following_comparison_or_group
|
|
75
|
+
assert_shape 'And[Not A, B]', 'Not A Eq 1 And B Eq 1'
|
|
76
|
+
assert_shape 'And[Not (Or[A, B]), C]', 'Not (A Eq 1 Or B Eq 1) And C Eq 1'
|
|
77
|
+
assert_shape 'Not (Not A)', 'Not (Not A Eq 1)'
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
def test_conjunction_and_unary_nodes_carry_their_levels
|
|
81
|
+
tree = @parser.parse_tree('Not (A Eq 1 Or B Eq 1) And C Eq 1')
|
|
82
|
+
unary = tree[:left]
|
|
83
|
+
inner_or = unary[:operand][:operand]
|
|
84
|
+
|
|
85
|
+
assert_equal [:conjunction, 'And', 0], tree.values_at(:node, :conjunction, :conjunction_level)
|
|
86
|
+
assert_equal [:unary, 'Not', 0], unary.values_at(:node, :unary, :unary_level)
|
|
87
|
+
assert_equal [:conjunction, 'Or', 1], inner_or.values_at(:node, :conjunction, :conjunction_level)
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
# A tree's leaves are the same expression Hashes #parse returns, with the
|
|
91
|
+
# same flat metadata.
|
|
92
|
+
def test_leaves_match_parse_output
|
|
93
|
+
[
|
|
94
|
+
'A Eq 1 Or B Eq 1 And C Eq 1',
|
|
95
|
+
'(A Eq 1) And ((B Eq 1) Or (C Eq 1))',
|
|
96
|
+
'Not (A Eq 1 Or B Eq 1) And C Eq 1',
|
|
97
|
+
'Not (Not A Eq 1)',
|
|
98
|
+
"A Eq 1 Not (B Eq 'x' Or C Bt 1,2)"
|
|
99
|
+
].each do |filter|
|
|
100
|
+
assert_equal Parser.new.parse(filter), leaves(@parser.parse_tree(filter)), filter
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
def test_empty_filter_is_nil
|
|
105
|
+
assert_nil @parser.parse_tree('')
|
|
106
|
+
assert !@parser.errors?
|
|
107
|
+
end
|
|
108
|
+
|
|
109
|
+
def test_syntax_error_is_nil_with_errors
|
|
110
|
+
assert_nil @parser.parse_tree('A Eq 1 And')
|
|
111
|
+
assert @parser.fatal_errors?
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
def test_validation_errors_are_reported_like_parse
|
|
115
|
+
filter = (1..80).map { |i| "F#{i} Eq #{i}" }.join(' And ')
|
|
116
|
+
tree = @parser.parse_tree(filter)
|
|
117
|
+
|
|
118
|
+
assert_not_nil tree
|
|
119
|
+
assert(@parser.errors.any? { |e| e.message.include?('maximum expression count') })
|
|
120
|
+
|
|
121
|
+
nested = Parser.new
|
|
122
|
+
nested.parse_tree('A Eq 1 And (((B Eq 1)))')
|
|
123
|
+
assert(nested.errors.any? { |e| e.message.include?('maximum nesting level') })
|
|
124
|
+
end
|
|
125
|
+
|
|
126
|
+
def test_parser_can_be_reused
|
|
127
|
+
assert_shape 'And[A, B]', 'A Eq 1 And B Eq 1'
|
|
128
|
+
assert_shape 'Or[C, D]', 'C Eq 1 Or D Eq 1'
|
|
129
|
+
end
|
|
130
|
+
end
|
data/test/unit/parser_test.rb
CHANGED
|
@@ -11,6 +11,28 @@ class ParserTest < Test::Unit::TestCase
|
|
|
11
11
|
parse "Test Eq 'false'", "'false'"
|
|
12
12
|
end
|
|
13
13
|
|
|
14
|
+
# #parse is built by flattening #parse_tree's tree; these pin the flat
|
|
15
|
+
# nesting metadata to what the grammar has always produced, quirks
|
|
16
|
+
# included (block_group is the most recently opened group, not the
|
|
17
|
+
# containing one -- see D below), so existing consumers see no change.
|
|
18
|
+
def test_flat_nesting_metadata_is_unchanged
|
|
19
|
+
keys = %i[field conjunction conjunction_level level block_group unary unary_level]
|
|
20
|
+
flat = ->(filter) { Parser.new.parse(filter).map { |e| e.values_at(*keys).compact } }
|
|
21
|
+
|
|
22
|
+
assert_equal [['A', 'And', 0, 1, 1], ['B', 'And', 0, 2, 3], ['C', 'Or', 1, 2, 4]],
|
|
23
|
+
flat.call('(A Eq 1) And ((B Eq 1) Or (C Eq 1))')
|
|
24
|
+
assert_equal [['A', 'And', 0, 1, 1], ['B', 'And', 1, 2, 2], ['C', 'Or', 0, 2, 4], ['D', 'And', 1, 1, 4]],
|
|
25
|
+
flat.call('(A Eq 1 And (B Eq 1)) Or ((C Eq 1) And D Eq 1)')
|
|
26
|
+
# The outer Not becomes A's conjunction when A already has a unary Not.
|
|
27
|
+
assert_equal [['A', 'Not', 0, 1, 1, 'Not', 1]], flat.call('Not (Not A Eq 1)')
|
|
28
|
+
assert_equal [['A', 'And', 0, 1, 1, 'Not', 0], ['B', 'Or', 1, 1, 1], ['C', 'And', 0, 0, 0]],
|
|
29
|
+
flat.call('Not (A Eq 1 Or B Eq 1) And C Eq 1')
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def test_empty_filter_parses_to_zero
|
|
33
|
+
assert_equal 0, Parser.new.parse('')
|
|
34
|
+
end
|
|
35
|
+
|
|
14
36
|
def test_conjunction
|
|
15
37
|
@parser = Parser.new
|
|
16
38
|
expression = @parser.parse('Test Eq 10 And Test Ne 11')
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: sparkql
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Wade McEwen
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-
|
|
11
|
+
date: 2026-10-26 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: georuby
|
|
@@ -153,6 +153,7 @@ files:
|
|
|
153
153
|
- test/unit/function_resolver_test.rb
|
|
154
154
|
- test/unit/geo/record_circle_test.rb
|
|
155
155
|
- test/unit/lexer_test.rb
|
|
156
|
+
- test/unit/parse_tree_test.rb
|
|
156
157
|
- test/unit/parser_compatability_test.rb
|
|
157
158
|
- test/unit/parser_test.rb
|
|
158
159
|
- test/unit/security_test.rb
|