jgrep 1.4.1 → 1.5.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.markdown +5 -0
- data/README.markdown +13 -12
- data/Rakefile +6 -1
- data/bin/jgrep +115 -129
- data/lib/jgrep.rb +286 -377
- data/lib/parser/parser.rb +109 -125
- data/lib/parser/scanner.rb +148 -149
- data/spec/Rakefile +3 -3
- data/spec/spec_helper.rb +1 -2
- data/spec/unit/jgrep_spec.rb +233 -233
- data/spec/unit/parser_spec.rb +132 -127
- data/spec/unit/scanner_spec.rb +82 -86
- metadata +6 -18
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: bc2ae50ca974a9033f6bca65254c009705305587
|
4
|
+
data.tar.gz: 36c22db07d3980bd372071c537bde74c846ccfed
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 85342e07b0e672f19cebda3b426271736b02097df611b98171f54d297e5f2d394c9a2b83fe3a42ce4936f281bdd26806ff89659a2bd52fc3ee54d05b2772b3cd
|
7
|
+
data.tar.gz: 86ac2ba2a35df8aa5a7884f3b223a1e82a2d4b2980b69e86b84cdfde19227a3fd541f522d427b63891f1f482ce5d4b47a689798123bdfb5e142f28fc2a437798
|
data/CHANGELOG.markdown
CHANGED
@@ -1,5 +1,10 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## 1.5.0
|
4
|
+
* Dropped support for Ruby 1.8.3
|
5
|
+
* Added support for modern Ruby versions (Tested up to 2.4.0)
|
6
|
+
* Added utility method to validate expressions
|
7
|
+
|
3
8
|
## 1.4.1
|
4
9
|
* Fix binary exit code to be 1 when no matches are found (Mickaël Canévet)
|
5
10
|
|
data/README.markdown
CHANGED
@@ -1,12 +1,12 @@
|
|
1
1
|
JGrep is a command line tool and API for parsing JSON documents based on logical expressions.
|
2
2
|
|
3
|
-
###Installation
|
3
|
+
### Installation:
|
4
4
|
|
5
5
|
jgrep is available as a gem:
|
6
6
|
|
7
7
|
gem install jgrep
|
8
8
|
|
9
|
-
###JGrep binary usage
|
9
|
+
### JGrep binary usage:
|
10
10
|
|
11
11
|
jgrep [expression] -i foo.json
|
12
12
|
|
@@ -14,7 +14,7 @@ or
|
|
14
14
|
|
15
15
|
cat "foo.json" | jgrep [expression]
|
16
16
|
|
17
|
-
###Flags
|
17
|
+
### Flags:
|
18
18
|
|
19
19
|
-s, --simple [FIELDS] : Greps the JSON and only returns the value of the field(s) specified
|
20
20
|
-c, --compat : Returns the JSON in its non-pretty flat form
|
@@ -26,7 +26,7 @@ or
|
|
26
26
|
--start FIELD : Starts the grep at a specific key in the document
|
27
27
|
--slice [RANGE] : A range of the form 'n' or 'n..m', indicating which documents to extract from the final output
|
28
28
|
|
29
|
-
###Expressions
|
29
|
+
### Expressions:
|
30
30
|
|
31
31
|
JGrep uses the following logical symbols to define expressions.
|
32
32
|
|
@@ -62,7 +62,7 @@ JGrep uses the following logical symbols to define expressions.
|
|
62
62
|
|
63
63
|
Performs the operations inside the perentheses first.
|
64
64
|
|
65
|
-
###Statements
|
65
|
+
### Statements:
|
66
66
|
|
67
67
|
A statement is defined as some value in a json document compared to another value.
|
68
68
|
Available comparison operators are '=', '<', '>', '<=', '>='
|
@@ -73,7 +73,7 @@ Examples:
|
|
73
73
|
foo.bar>0
|
74
74
|
foo.bar<=1.3
|
75
75
|
|
76
|
-
###Complex expressions
|
76
|
+
### Complex expressions:
|
77
77
|
|
78
78
|
Given a json document, {"foo":1, "bar":null}, the following are examples of valid expressions
|
79
79
|
|
@@ -95,12 +95,12 @@ Examples:
|
|
95
95
|
|
96
96
|
... returns true
|
97
97
|
|
98
|
-
###CLI missing an expression
|
98
|
+
### CLI missing an expression:
|
99
99
|
|
100
100
|
If JGrep is executed without a set expression, it will return an unmodified JSON document. The
|
101
101
|
-s flag can still be applied to the result.
|
102
102
|
|
103
|
-
###In document comparison
|
103
|
+
### In document comparison:
|
104
104
|
|
105
105
|
If a document contains an array, the '[' and ']' operators can be used to define a comparison where
|
106
106
|
statements are checked for truth on a per element basis which will then be combined.
|
@@ -152,7 +152,7 @@ will return
|
|
152
152
|
|
153
153
|
**Note**: In document comparison cannot be nested.
|
154
154
|
|
155
|
-
###The -s flag
|
155
|
+
### The -s flag:
|
156
156
|
|
157
157
|
The s flag simplifies the output returned by JGrep. Given a JSON document
|
158
158
|
|
@@ -167,7 +167,7 @@ will output
|
|
167
167
|
1
|
168
168
|
|
169
169
|
The s flag can also be used with multiple field, which will return JSON as output which only contain the specified fields.
|
170
|
-
**Note**: Separate fields by a space and enclose all fields in quotes (see example below)
|
170
|
+
**Note**: Separate fields by a space and enclose all fields in quotes (see example below)
|
171
171
|
|
172
172
|
Given:
|
173
173
|
|
@@ -190,7 +190,7 @@ will output
|
|
190
190
|
}
|
191
191
|
]
|
192
192
|
|
193
|
-
###The --start flag
|
193
|
+
### The --start flag:
|
194
194
|
|
195
195
|
Some documents do not comply to our expected format, they might have an array embedded deep in a field. The --start
|
196
196
|
flag lets you pick a starting point for the grep.
|
@@ -230,7 +230,7 @@ With the --stream or -n flag, jgrep will process multiple JSON inputs (newline
|
|
230
230
|
separated) until standard input is closed. Each JSON input will be processed
|
231
231
|
as usual, but the output immediately printed.
|
232
232
|
|
233
|
-
###JGrep Gem usage
|
233
|
+
### JGrep Gem usage:
|
234
234
|
|
235
235
|
require 'jgrep'
|
236
236
|
|
@@ -242,3 +242,4 @@ as usual, but the output immediately printed.
|
|
242
242
|
sflags = "foo"
|
243
243
|
|
244
244
|
JGrep::jgrep(json, expression, sflags)
|
245
|
+
|
data/Rakefile
CHANGED
data/bin/jgrep
CHANGED
@@ -1,153 +1,139 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
2
|
|
3
|
-
require
|
4
|
-
require
|
3
|
+
require "jgrep"
|
4
|
+
require "optparse"
|
5
5
|
|
6
|
-
@options = {:
|
6
|
+
@options = {flat: false, start: nil, field: [], slice: nil}
|
7
7
|
|
8
8
|
def print_json(result)
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
9
|
+
if @options[:flat]
|
10
|
+
puts(result.first.to_json)
|
11
|
+
else
|
12
|
+
result = result.first if @options[:stream]
|
13
|
+
puts(JSON.pretty_generate(result))
|
14
|
+
end
|
15
15
|
end
|
16
16
|
|
17
17
|
def do_grep(json, expression)
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
18
|
+
if @options[:field].empty?
|
19
|
+
result = JGrep.jgrep(json, expression, nil, @options[:start])
|
20
|
+
result = result.slice(@options[:slice]) if @options[:slice]
|
21
|
+
|
22
|
+
exit 1 if result == []
|
23
|
+
|
24
|
+
print_json(result) unless @options[:quiet] == true
|
25
|
+
elsif @options[:field].size > 1
|
26
|
+
JGrep.validate_filters(@options[:field])
|
27
|
+
result = JGrep.jgrep(json, expression, @options[:field], @options[:start])
|
28
|
+
result = result.slice(@options[:slice]) if @options[:slice]
|
29
|
+
|
30
|
+
exit 1 if result == []
|
31
|
+
|
32
|
+
print_json(result) unless @options[:quiet] == true
|
33
|
+
|
34
|
+
else
|
35
|
+
JGrep.validate_filters(@options[:field][0])
|
36
|
+
result = JGrep.jgrep(json, expression, @options[:field][0], @options[:start])
|
37
|
+
result = result.slice(@options[:slice]) if @options[:slice]
|
38
|
+
exit 1 if result == []
|
39
|
+
if result.is_a?(Array) && !(result.first.is_a?(Hash) || result.flatten.first.is_a?(Hash))
|
40
|
+
unless @options[:quiet] == true
|
41
|
+
result.map {|x| puts x unless x.nil?}
|
42
|
+
end
|
25
43
|
else
|
26
|
-
|
27
|
-
JGrep::validate_filters(@options[:field])
|
28
|
-
result = JGrep::jgrep((json), expression, @options[:field], @options[:start])
|
29
|
-
result = result.slice(@options[:slice]) if @options[:slice]
|
30
|
-
exit 1 if result == []
|
31
|
-
unless @options[:quiet] == true
|
32
|
-
print_json(result)
|
33
|
-
end
|
34
|
-
|
35
|
-
else
|
36
|
-
JGrep::validate_filters(@options[:field][0])
|
37
|
-
result = JGrep::jgrep((json), expression, @options[:field][0], @options[:start])
|
38
|
-
result = result.slice(@options[:slice]) if @options[:slice]
|
39
|
-
exit 1 if result == []
|
40
|
-
if result.is_a?(Array) && !(result.first.is_a?(Hash) || result.flatten.first.is_a?(Hash))
|
41
|
-
unless @options[:quiet] == true
|
42
|
-
result.map{|x| puts x unless x.nil?}
|
43
|
-
end
|
44
|
-
else
|
45
|
-
unless @options[:quiet] == true
|
46
|
-
print_json(result)
|
47
|
-
end
|
48
|
-
end
|
49
|
-
end
|
44
|
+
print_json(result) unless @options[:quiet] == true
|
50
45
|
end
|
46
|
+
end
|
51
47
|
end
|
52
48
|
|
53
49
|
begin
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
@options[:flat] = true
|
66
|
-
end
|
67
|
-
|
68
|
-
opts.on("-n", "--stream", "Display continuous output from continuous input") do
|
69
|
-
@options[:stream] = true
|
70
|
-
end
|
71
|
-
|
72
|
-
opts.on("-f", "--flatten", "Makes output as flat as possible") do
|
73
|
-
JGrep::flatten_on
|
74
|
-
end
|
75
|
-
|
76
|
-
opts.on("-i", "--input [FILENAME]", "Specify input file to parse") do |filename|
|
77
|
-
@options[:file] = filename
|
78
|
-
end
|
79
|
-
|
80
|
-
opts.on("-q", "--quiet", "Quiet; don't write to stdout. Exit with zero status if match found.") do
|
81
|
-
@options[:quiet] = true
|
82
|
-
end
|
83
|
-
|
84
|
-
opts.on("-v", "--verbose", "Verbose output") do
|
85
|
-
JGrep::verbose_on
|
86
|
-
end
|
87
|
-
|
88
|
-
opts.on("--start [FIELD]", "Where in the data to start from") do |field|
|
89
|
-
@options[:start] = field
|
90
|
-
end
|
91
|
-
|
92
|
-
opts.on("--slice [RANGE]", "A range of the form 'n' or 'n..m', indicating which documents to extract from the final output") do |field|
|
93
|
-
range_nums = field.split('..').map{ |x| x.to_i }
|
94
|
-
@options[:slice] = range_nums.length == 1 ? range_nums[0] : Range.new(*range_nums)
|
95
|
-
end
|
96
|
-
end.parse!
|
97
|
-
rescue OptionParser::InvalidOption => e
|
98
|
-
puts e.to_s.capitalize
|
99
|
-
exit 1
|
50
|
+
OptionParser.new do |opts|
|
51
|
+
opts.banner = "Usage: jgrep [options] \"expression\""
|
52
|
+
opts.on("-s", "--simple [FIELDS]", "Display only one or more fields from each of the resulting json documents") do |field|
|
53
|
+
raise "-s flag requires a field value" if field.nil?
|
54
|
+
|
55
|
+
@options[:field].concat(field.split(" "))
|
56
|
+
end
|
57
|
+
|
58
|
+
opts.on("-c", "--compact", "Display non pretty json") do
|
59
|
+
@options[:flat] = true
|
60
|
+
end
|
100
61
|
|
101
|
-
|
102
|
-
|
103
|
-
|
62
|
+
opts.on("-n", "--stream", "Display continuous output from continuous input") do
|
63
|
+
@options[:stream] = true
|
64
|
+
end
|
65
|
+
|
66
|
+
opts.on("-f", "--flatten", "Makes output as flat as possible") do
|
67
|
+
JGrep.flatten_on
|
68
|
+
end
|
69
|
+
|
70
|
+
opts.on("-i", "--input [FILENAME]", "Specify input file to parse") do |filename|
|
71
|
+
@options[:file] = filename
|
72
|
+
end
|
73
|
+
|
74
|
+
opts.on("-q", "--quiet", "Quiet; don't write to stdout. Exit with zero status if match found.") do
|
75
|
+
@options[:quiet] = true
|
76
|
+
end
|
77
|
+
|
78
|
+
opts.on("-v", "--verbose", "Verbose output") do
|
79
|
+
JGrep.verbose_on
|
80
|
+
end
|
81
|
+
|
82
|
+
opts.on("--start [FIELD]", "Where in the data to start from") do |field|
|
83
|
+
@options[:start] = field
|
84
|
+
end
|
85
|
+
|
86
|
+
opts.on("--slice [RANGE]", "A range of the form 'n' or 'n..m', indicating which documents to extract from the final output") do |field|
|
87
|
+
range_nums = field.split("..").map(&:to_i)
|
88
|
+
@options[:slice] = range_nums.length == 1 ? range_nums[0] : Range.new(*range_nums)
|
89
|
+
end
|
90
|
+
end.parse!
|
91
|
+
rescue OptionParser::InvalidOption => e
|
92
|
+
puts e.to_s.capitalize
|
93
|
+
exit 1
|
94
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
95
|
+
puts e
|
96
|
+
exit 1
|
104
97
|
end
|
105
98
|
|
106
99
|
begin
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
end
|
100
|
+
expression = nil
|
101
|
+
|
102
|
+
# Identify the expression from command line arguments
|
103
|
+
ARGV.each do |argument|
|
104
|
+
if argument =~ /<|>|=|\+|-/
|
105
|
+
expression = argument
|
106
|
+
ARGV.delete(argument)
|
115
107
|
end
|
108
|
+
end
|
116
109
|
|
117
|
-
|
110
|
+
expression = "" if expression.nil?
|
118
111
|
|
119
|
-
|
120
|
-
|
121
|
-
|
112
|
+
# Continuously gets if inputstream in constant
|
113
|
+
# Load json from standard input if tty is false
|
114
|
+
# else find and load file from command line arugments
|
122
115
|
|
123
|
-
|
124
|
-
|
125
|
-
while json = gets
|
126
|
-
do_grep(json, expression)
|
127
|
-
end
|
128
|
-
else
|
129
|
-
raise "No json input specified"
|
130
|
-
end
|
131
|
-
else
|
132
|
-
if @options[:file]
|
133
|
-
json = File.read(@options[:file])
|
134
|
-
do_grep(json, expression)
|
135
|
-
elsif ! STDIN.tty?
|
136
|
-
json = STDIN.read
|
137
|
-
do_grep(json, expression)
|
138
|
-
else
|
139
|
-
raise "No json input specified"
|
140
|
-
end
|
141
|
-
end
|
116
|
+
if @options[:stream]
|
117
|
+
raise "No json input specified" if STDIN.tty?
|
142
118
|
|
143
|
-
|
144
|
-
|
145
|
-
exit 1
|
146
|
-
rescue Exception => e
|
147
|
-
if e.is_a?(SystemExit)
|
148
|
-
exit e.status
|
149
|
-
else
|
150
|
-
STDERR.puts "Error - #{e}"
|
151
|
-
exit 1
|
119
|
+
while json = gets
|
120
|
+
do_grep(json, expression)
|
152
121
|
end
|
122
|
+
elsif @options[:file]
|
123
|
+
json = File.read(@options[:file])
|
124
|
+
do_grep(json, expression)
|
125
|
+
elsif !STDIN.tty?
|
126
|
+
json = STDIN.read
|
127
|
+
do_grep(json, expression)
|
128
|
+
else
|
129
|
+
raise "No json input specified"
|
130
|
+
end
|
131
|
+
rescue Interrupt
|
132
|
+
STDERR.puts "Exiting..."
|
133
|
+
exit 1
|
134
|
+
rescue SystemExit
|
135
|
+
exit e.status
|
136
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
137
|
+
STDERR.puts "Error - #{e}"
|
138
|
+
exit 1
|
153
139
|
end
|
data/lib/jgrep.rb
CHANGED
@@ -1,446 +1,355 @@
|
|
1
|
-
|
2
|
-
|
3
|
-
require
|
4
|
-
require
|
5
|
-
require 'rubygems'
|
6
|
-
require 'json'
|
7
|
-
#require 'yajl/json_gem'
|
1
|
+
require "parser/parser.rb"
|
2
|
+
require "parser/scanner.rb"
|
3
|
+
require "rubygems"
|
4
|
+
require "json"
|
8
5
|
|
9
6
|
module JGrep
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
else
|
51
|
-
errors = "One or more the json documents could not be parsed. Run jgrep -v for to display documents"
|
52
|
-
end
|
53
|
-
end
|
54
|
-
end
|
7
|
+
@verbose = false
|
8
|
+
@flatten = false
|
9
|
+
|
10
|
+
def self.verbose_on
|
11
|
+
@verbose = true
|
12
|
+
end
|
13
|
+
|
14
|
+
def self.flatten_on
|
15
|
+
@flatten = true
|
16
|
+
end
|
17
|
+
|
18
|
+
# Parse json and return documents that match the logical expression
|
19
|
+
# Filters define output by limiting it to only returning a the listed keys.
|
20
|
+
# Start allows you to move the pointer indicating where parsing starts.
|
21
|
+
# Default is the first key in the document heirarchy
|
22
|
+
def self.jgrep(json, expression, filters = nil, start = nil)
|
23
|
+
errors = ""
|
24
|
+
|
25
|
+
begin
|
26
|
+
JSON.create_id = nil
|
27
|
+
json = JSON.parse(json)
|
28
|
+
json = [json] if json.is_a?(Hash)
|
29
|
+
|
30
|
+
json = filter_json(json, start).flatten if start
|
31
|
+
|
32
|
+
result = []
|
33
|
+
|
34
|
+
if expression == ""
|
35
|
+
result = json
|
36
|
+
else
|
37
|
+
call_stack = Parser.new(expression).execution_stack
|
38
|
+
|
39
|
+
json.each do |document|
|
40
|
+
begin
|
41
|
+
result << document if eval_statement(document, call_stack)
|
42
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
43
|
+
if @verbose
|
44
|
+
require "pp"
|
45
|
+
pp document
|
46
|
+
STDERR.puts "Error - #{e} \n\n"
|
55
47
|
else
|
56
|
-
|
48
|
+
errors = "One or more the json documents could not be parsed. Run jgrep -v for to display documents"
|
57
49
|
end
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
58
53
|
|
59
|
-
|
60
|
-
puts errors
|
61
|
-
end
|
54
|
+
puts errors unless errors == ""
|
62
55
|
|
63
|
-
|
64
|
-
return result
|
65
|
-
else
|
66
|
-
filter_json(result, filters)
|
67
|
-
end
|
56
|
+
return result unless filters
|
68
57
|
|
69
|
-
|
70
|
-
|
71
|
-
|
58
|
+
filter_json(result, filters)
|
59
|
+
rescue JSON::ParserError
|
60
|
+
STDERR.puts "Error. Invalid JSON given"
|
72
61
|
end
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
62
|
+
end
|
63
|
+
|
64
|
+
# Validates an expression, true when no errors are found else a string representing the issues
|
65
|
+
def self.validate_expression(expression)
|
66
|
+
Parser.new(expression)
|
67
|
+
true
|
68
|
+
rescue
|
69
|
+
$!.message
|
70
|
+
end
|
71
|
+
|
72
|
+
# Strips filters from json documents and returns those values as a less bloated json document
|
73
|
+
def self.filter_json(documents, filters)
|
74
|
+
result = []
|
75
|
+
|
76
|
+
if filters.is_a? Array
|
77
|
+
documents.each do |doc|
|
78
|
+
tmp_json = {}
|
79
|
+
|
80
|
+
filters.each do |filter|
|
81
|
+
filtered_result = dig_path(doc, filter)
|
82
|
+
unless (filtered_result == doc) || filtered_result.nil?
|
83
|
+
tmp_json[filter] = filtered_result
|
84
|
+
end
|
84
85
|
end
|
86
|
+
result << tmp_json
|
87
|
+
end
|
88
|
+
else
|
89
|
+
documents.each do |r|
|
90
|
+
filtered_result = dig_path(r, filters)
|
91
|
+
|
92
|
+
unless (filtered_result == r) || filtered_result.nil?
|
93
|
+
result << filtered_result
|
94
|
+
end
|
95
|
+
end
|
96
|
+
end
|
85
97
|
|
86
|
-
|
87
|
-
|
88
|
-
begin
|
89
|
-
for i in 0..(documents.size - 1) do
|
90
|
-
tmp = documents[i]
|
91
|
-
unless mark == ""
|
92
|
-
mark.split(".").each_with_index do |m,i|
|
93
|
-
tmp = tmp[m] unless i == mark.split(".").size - 1
|
94
|
-
end
|
95
|
-
end
|
98
|
+
result.flatten if @flatten == true && result.size == 1
|
96
99
|
|
97
|
-
|
98
|
-
|
100
|
+
result
|
101
|
+
end
|
99
102
|
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
103
|
+
# Validates if filters do not match any of the parser's logical tokens
|
104
|
+
def self.validate_filters(filters)
|
105
|
+
if filters.is_a? Array
|
106
|
+
filters.each do |filter|
|
107
|
+
if filter =~ /=|<|>|^and$|^or$|^!$|^not$/
|
108
|
+
raise "Invalid field for -s filter : '#{filter}'"
|
104
109
|
end
|
105
|
-
|
106
|
-
|
107
|
-
|
110
|
+
end
|
111
|
+
elsif filters =~ /=|<|>|^and$|^or$|^!$|^not$/
|
112
|
+
raise "Invalid field for -s filter : '#{filters}'"
|
108
113
|
end
|
109
114
|
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
120
|
-
|
115
|
+
nil
|
116
|
+
end
|
117
|
+
|
118
|
+
# Correctly format values so we can do the correct type of comparison
|
119
|
+
def self.format(kvalue, value)
|
120
|
+
if kvalue.to_s =~ /^\d+$/ && value.to_s =~ /^\d+$/
|
121
|
+
[Integer(kvalue), Integer(value)]
|
122
|
+
elsif kvalue.to_s =~ /^\d+.\d+$/ && value.to_s =~ /^\d+.\d+$/
|
123
|
+
[Float(kvalue), Float(value)]
|
124
|
+
else
|
125
|
+
[kvalue, value]
|
126
|
+
end
|
127
|
+
end
|
128
|
+
|
129
|
+
# Check if the json key that is defined by statement is defined in the json document
|
130
|
+
def self.present?(document, statement)
|
131
|
+
statement.split(".").each do |key|
|
132
|
+
if document.is_a? Hash
|
133
|
+
if document.value?(nil)
|
134
|
+
document.each do |k, _|
|
135
|
+
document[k] = "null" if document[k].nil?
|
136
|
+
end
|
121
137
|
end
|
138
|
+
end
|
122
139
|
|
123
|
-
|
140
|
+
if document.is_a? Array
|
141
|
+
rval = false
|
142
|
+
document.each do |doc|
|
143
|
+
rval ||= present?(doc, key)
|
144
|
+
end
|
145
|
+
return rval
|
146
|
+
end
|
124
147
|
|
125
|
-
|
126
|
-
for i in 0..(documents.size - 1) do
|
127
|
-
tmp = documents[i]
|
128
|
-
unless mark == ""
|
129
|
-
mark.split(".").each_with_index do |m,i|
|
130
|
-
tmp = tmp[m] unless i == mark.split(".").size - 1
|
131
|
-
end
|
132
|
-
end
|
148
|
+
document = document[key]
|
133
149
|
|
134
|
-
|
135
|
-
|
150
|
+
return false if document.nil?
|
151
|
+
end
|
136
152
|
|
137
|
-
|
138
|
-
|
139
|
-
STDERR.puts "Error. Invalid position specified in JSON document"
|
140
|
-
exit!
|
141
|
-
end
|
153
|
+
true
|
154
|
+
end
|
142
155
|
|
143
|
-
|
156
|
+
# Check if key=value is present in document
|
157
|
+
def self.has_object?(document, statement)
|
158
|
+
key, value = statement.split(/<=|>=|=|<|>/)
|
144
159
|
|
160
|
+
if statement =~ /(<=|>=|<|>|=)/
|
161
|
+
op = $1
|
162
|
+
else
|
163
|
+
op = statement
|
145
164
|
end
|
146
165
|
|
147
|
-
|
148
|
-
def self.filter_json(documents, filters)
|
149
|
-
result = []
|
166
|
+
tmp = dig_path(document, key)
|
150
167
|
|
151
|
-
|
152
|
-
documents.each do |doc|
|
153
|
-
tmp_json = {}
|
154
|
-
filters.each do |filter|
|
155
|
-
filtered_result = dig_path(doc, filter)
|
156
|
-
unless (filtered_result == doc) || filtered_result.nil?
|
157
|
-
tmp_json[filter] = filtered_result
|
158
|
-
end
|
159
|
-
end
|
160
|
-
result << tmp_json
|
161
|
-
end
|
168
|
+
tmp = tmp.first if tmp.is_a?(Array) && tmp.size == 1
|
162
169
|
|
163
|
-
|
164
|
-
return result
|
170
|
+
tmp, value = format(tmp, (value.gsub(/"|'/, "") unless value.nil?)) # rubocop:disable Style/FormatString
|
165
171
|
|
166
|
-
|
167
|
-
|
168
|
-
filtered_result = dig_path(r, filters)
|
169
|
-
unless (filtered_result == r) || filtered_result.nil?
|
170
|
-
result << filtered_result
|
171
|
-
end
|
172
|
-
end
|
172
|
+
# Deal with null comparison
|
173
|
+
return true if tmp.nil? && value == "null"
|
173
174
|
|
174
|
-
|
175
|
-
|
176
|
-
|
177
|
-
end
|
175
|
+
# Deal with booleans
|
176
|
+
return true if tmp == true && value == "true"
|
177
|
+
return true if tmp == false && value == "false"
|
178
178
|
|
179
|
-
#
|
180
|
-
|
181
|
-
|
182
|
-
filters.each do |filter|
|
183
|
-
if filter =~ /=|<|>|^and$|^or$|^!$|^not$/
|
184
|
-
raise "Invalid field for -s filter : '#{filter}'"
|
185
|
-
end
|
186
|
-
end
|
187
|
-
else
|
188
|
-
if filters =~ /=|<|>|^and$|^or$|^!$|^not$/
|
189
|
-
raise "Invalid field for -s filter : '#{filters}'"
|
190
|
-
end
|
191
|
-
end
|
192
|
-
return
|
179
|
+
# Deal with regex matching
|
180
|
+
if !tmp.nil? && value =~ /^\/.*\/$/
|
181
|
+
tmp.match(Regexp.new(value.delete("/"))) ? (return true) : (return false)
|
193
182
|
end
|
194
183
|
|
195
|
-
#
|
196
|
-
|
197
|
-
|
198
|
-
|
199
|
-
|
200
|
-
|
201
|
-
|
202
|
-
|
203
|
-
|
184
|
+
# Deal with everything else
|
185
|
+
case op
|
186
|
+
when "="
|
187
|
+
return tmp == value
|
188
|
+
when "<="
|
189
|
+
return tmp <= value
|
190
|
+
when ">="
|
191
|
+
return tmp >= value
|
192
|
+
when ">"
|
193
|
+
return tmp > value
|
194
|
+
when "<"
|
195
|
+
return tmp < value
|
204
196
|
end
|
197
|
+
end
|
205
198
|
|
199
|
+
# Check if key=value is present in a sub array
|
200
|
+
def self.is_object_in_array?(document, statement)
|
201
|
+
document.each do |item|
|
202
|
+
return true if has_object?(item, statement)
|
203
|
+
end
|
206
204
|
|
207
|
-
|
208
|
-
|
209
|
-
|
210
|
-
|
211
|
-
|
212
|
-
|
213
|
-
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
219
|
-
|
220
|
-
|
221
|
-
|
222
|
-
|
223
|
-
rval ||= present?(doc, key)
|
224
|
-
end
|
225
|
-
return rval
|
226
|
-
end
|
227
|
-
|
228
|
-
document = document[key]
|
229
|
-
if document.nil?
|
230
|
-
return false
|
231
|
-
end
|
232
|
-
end
|
233
|
-
return true
|
205
|
+
false
|
206
|
+
end
|
207
|
+
|
208
|
+
# Check if complex statement (defined as [key=value...]) is
|
209
|
+
# present over an array of key value pairs
|
210
|
+
def self.has_complex?(document, compound)
|
211
|
+
field = ""
|
212
|
+
tmp = document
|
213
|
+
result = []
|
214
|
+
fresult = []
|
215
|
+
|
216
|
+
compound.each do |token|
|
217
|
+
if token[0] == "statement"
|
218
|
+
field = token
|
219
|
+
break
|
220
|
+
end
|
234
221
|
end
|
235
222
|
|
236
|
-
|
237
|
-
def self.has_object?(document, statement)
|
223
|
+
field = field[1].split(/=|<|>/).first
|
238
224
|
|
239
|
-
|
225
|
+
field.split(".").each_with_index do |item, _|
|
226
|
+
tmp = tmp[item]
|
240
227
|
|
241
|
-
|
242
|
-
op = $1
|
243
|
-
else
|
244
|
-
op = statement
|
245
|
-
end
|
228
|
+
return false if tmp.nil?
|
246
229
|
|
247
|
-
|
230
|
+
next unless tmp.is_a?(Array)
|
248
231
|
|
249
|
-
|
250
|
-
|
251
|
-
end
|
252
|
-
|
253
|
-
tmp, value = format(tmp, (value.gsub(/"|'/, "") unless value.nil?))
|
232
|
+
tmp.each do |doc|
|
233
|
+
result = []
|
254
234
|
|
255
|
-
|
256
|
-
|
257
|
-
|
235
|
+
compound.each do |token|
|
236
|
+
case token[0]
|
237
|
+
when "and"
|
238
|
+
result << "&&"
|
239
|
+
when "or"
|
240
|
+
result << "||"
|
241
|
+
when /not|\!/
|
242
|
+
result << "!"
|
243
|
+
when "statement"
|
244
|
+
op = token[1].match(/.*<=|>=|=|<|>/)
|
245
|
+
left = token[1].split(op[0]).first.split(".").last
|
246
|
+
right = token[1].split(op[0]).last
|
247
|
+
new_statement = left + op[0] + right
|
248
|
+
result << has_object?(doc, new_statement)
|
249
|
+
end
|
258
250
|
end
|
259
251
|
|
260
|
-
#
|
261
|
-
|
262
|
-
|
263
|
-
elsif tmp == false and value == 'false'
|
264
|
-
return true
|
265
|
-
end
|
252
|
+
fresult << eval(result.join(" ")) # rubocop:disable Security/Eval
|
253
|
+
(fresult << "||") unless doc == tmp.last
|
254
|
+
end
|
266
255
|
|
267
|
-
|
268
|
-
if ((value =~ /^\/.*\/$/) && tmp != nil)
|
269
|
-
(tmp.match(Regexp.new(value.gsub("/", "")))) ? (return true) : (return false)
|
270
|
-
end
|
271
|
-
|
272
|
-
#Deal with everything else
|
273
|
-
case op
|
274
|
-
when "="
|
275
|
-
(tmp == value) ? (return true) : (return false)
|
276
|
-
when "<="
|
277
|
-
(tmp <= value) ? (return true) : (return false)
|
278
|
-
when ">="
|
279
|
-
(tmp >= value) ? (return true) : (return false)
|
280
|
-
when ">"
|
281
|
-
(tmp > value) ? (return true) : (return false)
|
282
|
-
when "<"
|
283
|
-
(tmp < value) ? (return true) : (return false)
|
284
|
-
end
|
256
|
+
return eval(fresult.join(" ")) # rubocop:disable Security/Eval
|
285
257
|
end
|
286
|
-
|
287
|
-
|
288
|
-
|
289
|
-
|
290
|
-
|
291
|
-
|
292
|
-
|
293
|
-
|
258
|
+
end
|
259
|
+
|
260
|
+
# Evaluates the call stack en returns true of selected document
|
261
|
+
# matches logical expression
|
262
|
+
def self.eval_statement(document, callstack)
|
263
|
+
result = []
|
264
|
+
|
265
|
+
callstack.each do |expression|
|
266
|
+
case expression.keys.first
|
267
|
+
when "statement"
|
268
|
+
if expression.values.first.is_a?(Array)
|
269
|
+
result << has_complex?(document, expression.values.first)
|
270
|
+
else
|
271
|
+
result << has_object?(document, expression.values.first)
|
294
272
|
end
|
295
|
-
|
296
|
-
|
273
|
+
when "+"
|
274
|
+
result << present?(document, expression.values.first)
|
275
|
+
when "-"
|
276
|
+
result << !present?(document, expression.values.first)
|
277
|
+
when "and"
|
278
|
+
result << "&&"
|
279
|
+
when "or"
|
280
|
+
result << "||"
|
281
|
+
when "("
|
282
|
+
result << "("
|
283
|
+
when ")"
|
284
|
+
result << ")"
|
285
|
+
when "not"
|
286
|
+
result << "!"
|
287
|
+
end
|
297
288
|
end
|
298
289
|
|
299
|
-
|
300
|
-
|
301
|
-
def self.has_complex?(document, compound)
|
302
|
-
field = ""
|
303
|
-
tmp = document
|
304
|
-
result = []
|
305
|
-
fresult = []
|
290
|
+
eval(result.join(" ")) # rubocop:disable Security/Eval
|
291
|
+
end
|
306
292
|
|
307
|
-
|
308
|
-
|
309
|
-
|
310
|
-
|
311
|
-
end
|
312
|
-
end
|
313
|
-
field = field[1].split(/=|<|>/).first
|
293
|
+
# Digs to a specific path in the json document and returns the value
|
294
|
+
def self.dig_path(json, path)
|
295
|
+
index = nil
|
296
|
+
path = path.gsub(/^\./, "")
|
314
297
|
|
315
|
-
|
316
|
-
|
317
|
-
|
318
|
-
return false
|
319
|
-
end
|
320
|
-
if tmp.is_a? Array
|
321
|
-
tmp.each do |doc|
|
322
|
-
result = []
|
323
|
-
compound.each do |token|
|
324
|
-
case token[0]
|
325
|
-
when "and"
|
326
|
-
result << "&&"
|
327
|
-
when "or"
|
328
|
-
result << "||"
|
329
|
-
when /not|\!/
|
330
|
-
result << "!"
|
331
|
-
when "statement"
|
332
|
-
op = token[1].match(/.*<=|>=|=|<|>/)
|
333
|
-
left = token[1].split(op[0]).first.split(".").last
|
334
|
-
right = token[1].split(op[0]).last
|
335
|
-
new_statement = left + op[0] + right
|
336
|
-
result << has_object?(doc, new_statement)
|
337
|
-
end
|
338
|
-
end
|
339
|
-
fresult << eval(result.join(" "))
|
340
|
-
(fresult << "||") unless doc == tmp.last
|
341
|
-
end
|
342
|
-
return eval(fresult.join(" "))
|
343
|
-
end
|
344
|
-
end
|
298
|
+
if path =~ /(.*)\[(.*)\]/
|
299
|
+
path = $1
|
300
|
+
index = $2
|
345
301
|
end
|
346
302
|
|
347
|
-
|
348
|
-
#matches logical expression
|
349
|
-
def self.eval_statement(document, callstack)
|
350
|
-
result = []
|
351
|
-
callstack.each do |expression|
|
352
|
-
case expression.keys.first
|
353
|
-
when "statement"
|
354
|
-
if expression.values.first.is_a? Array
|
355
|
-
result << has_complex?(document, expression.values.first)
|
356
|
-
else
|
357
|
-
result << has_object?(document, expression.values.first)
|
358
|
-
end
|
359
|
-
when "+"
|
360
|
-
result << present?(document, expression.values.first)
|
361
|
-
when "-"
|
362
|
-
result << !(present?(document, expression.values.first))
|
363
|
-
when "and"
|
364
|
-
result << "&&"
|
365
|
-
when "or"
|
366
|
-
result << "||"
|
367
|
-
when "("
|
368
|
-
result << "("
|
369
|
-
when ")"
|
370
|
-
result << ")"
|
371
|
-
when "not"
|
372
|
-
result << "!"
|
373
|
-
end
|
374
|
-
end
|
303
|
+
return json if path == ""
|
375
304
|
|
376
|
-
|
305
|
+
if json.is_a? Hash
|
306
|
+
json.keys.each do |k|
|
307
|
+
if path.start_with?(k) && k.include?(".")
|
308
|
+
return dig_path(json[k], path.gsub(k, ""))
|
309
|
+
end
|
310
|
+
end
|
377
311
|
end
|
378
312
|
|
379
|
-
|
380
|
-
def self.dig_path(json, path)
|
381
|
-
index = nil
|
382
|
-
path = path.gsub(/^\./, "")
|
313
|
+
path_array = path.split(".")
|
383
314
|
|
384
|
-
|
385
|
-
|
386
|
-
index = $2
|
387
|
-
end
|
315
|
+
if path_array.first == "*"
|
316
|
+
tmp = []
|
388
317
|
|
389
|
-
|
390
|
-
|
391
|
-
|
318
|
+
json.each do |j|
|
319
|
+
tmp << dig_path(j[1], path_array.drop(1).join("."))
|
320
|
+
end
|
392
321
|
|
393
|
-
|
394
|
-
|
395
|
-
if path.start_with?(k) && k.include?('.')
|
396
|
-
return dig_path(json[k], path.gsub(k, ""))
|
397
|
-
end
|
398
|
-
end
|
399
|
-
end
|
322
|
+
return tmp
|
323
|
+
end
|
400
324
|
|
401
|
-
|
325
|
+
json = json[path_array.first] if json.is_a? Hash
|
402
326
|
|
403
|
-
|
404
|
-
|
405
|
-
|
406
|
-
tmp << dig_path(j[1], path_array.drop(1).join("."))
|
407
|
-
end
|
408
|
-
return tmp
|
327
|
+
if json.is_a? Hash
|
328
|
+
return json if path == path_array.first
|
329
|
+
return dig_path(json, path.include?(".") ? path_array.drop(1).join(".") : path)
|
409
330
|
|
410
|
-
|
331
|
+
elsif json.is_a? Array
|
332
|
+
if path == path_array.first && (json.first.is_a?(Hash) && !json.first.keys.include?(path))
|
333
|
+
return json
|
334
|
+
end
|
411
335
|
|
412
|
-
|
413
|
-
|
414
|
-
if json.is_a? Hash
|
415
|
-
if path == path_array.first
|
416
|
-
return json
|
417
|
-
else
|
418
|
-
return dig_path(json, (path.include?('.') ? path_array.drop(1).join(".") : path))
|
419
|
-
end
|
336
|
+
tmp = []
|
420
337
|
|
421
|
-
|
422
|
-
|
423
|
-
|
424
|
-
|
425
|
-
tmp = []
|
426
|
-
json.each do |j|
|
427
|
-
tmp_path = dig_path(j, (path.include?('.') ? path_array.drop(1).join(".") : path))
|
428
|
-
unless tmp_path.nil?
|
429
|
-
tmp << tmp_path
|
430
|
-
end
|
431
|
-
end
|
432
|
-
unless tmp.empty?
|
433
|
-
(index) ? (return tmp.flatten[index.to_i]) : (return tmp)
|
434
|
-
end
|
435
|
-
end
|
338
|
+
json.each do |j|
|
339
|
+
tmp_path = dig_path(j, (path.include?(".") ? path_array.drop(1).join(".") : path))
|
340
|
+
tmp << tmp_path unless tmp_path.nil?
|
341
|
+
end
|
436
342
|
|
437
|
-
|
438
|
-
|
343
|
+
unless tmp.empty?
|
344
|
+
return index ? tmp.flatten[index.to_i] : tmp
|
345
|
+
end
|
439
346
|
|
440
|
-
|
441
|
-
|
347
|
+
elsif json.nil?
|
348
|
+
return nil
|
442
349
|
|
443
|
-
|
350
|
+
else
|
351
|
+
return json
|
444
352
|
|
445
353
|
end
|
354
|
+
end
|
446
355
|
end
|