enumpath 0.1.2 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +180 -31
- data/lib/enumpath/operator/base.rb +26 -11
- data/lib/enumpath/operator/subscript_expression.rb +50 -1
- data/lib/enumpath/path.rb +10 -3
- data/lib/enumpath/resolver/object_inspector.rb +70 -0
- data/lib/enumpath/resolver/property.rb +227 -8
- data/lib/enumpath/resolver/simple.rb +35 -1
- data/lib/enumpath/version.rb +1 -1
- data/lib/enumpath.rb +13 -2
- metadata +43 -46
- data/.github/ISSUE_TEMPLATE/bug_report.md +0 -38
- data/.github/ISSUE_TEMPLATE/feature_request.md +0 -20
- data/.github/workflows/greetings.yml +0 -13
- data/.github/workflows/rubocop-analysis.yml +0 -47
- data/.github/workflows/ruby.yml +0 -35
- data/.gitignore +0 -15
- data/.rspec +0 -3
- data/.rubocop.yml +0 -12
- data/.yardopts +0 -1
- data/CONTRIBUTING.md +0 -96
- data/Gemfile +0 -8
- data/Gemfile.lock +0 -61
- data/Rakefile +0 -13
- data/bin/console +0 -15
- data/bin/setup +0 -8
- data/enumpath.gemspec +0 -45
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4a759aab90deef7e5fe16fb2373738f61d26247efc4d20553740b86c025e1e29
|
|
4
|
+
data.tar.gz: f6dddd9c820e26fdde6eedd8999dae7fd94ab5bf578f33c81f60d56f08b0c559
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3984dc1111b7435e76868a400685144b7eb7467e1a61b66fe65b1447aac3c82c93c722fa6a8ff1ff84c8c0cef1dfabf15c6f1b7b291a0b678e733ae2fb435be4
|
|
7
|
+
data.tar.gz: 6be4ed326e7dcd4fb9c00d2cefda0ded4efc8ac57d1e0535bc307ca289bb30ad0a705d5d4f4654c955f5d6443c549c1b395bed331a3ddcc4ea68edf1cc1af03c
|
data/README.md
CHANGED
|
@@ -1,14 +1,18 @@
|
|
|
1
|
-
# Enumpath
|
|
1
|
+
# Enumpath
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Navigate nested Ruby objects with JSONPath-style path expressions, safely by default.
|
|
4
4
|
|
|
5
5
|
## Introduction
|
|
6
6
|
|
|
7
|
-
Enumpath
|
|
7
|
+
Enumpath lets you describe a path through nested Ruby objects as a string, using [JSONPath][jsonpath]-style syntax, and pull out whatever it matches. It works on the data you actually have in memory: hashes with symbol, string or integer keys, arrays, `Struct`s, plain Ruby objects and `OpenStruct`s, in any mix. It is like Ruby's native `dig`, but a path is a plain string that can use wildcards, filters, slices and recursive descent, so it can live in configuration or a database and change without a deploy. (`dig(*keys)` also takes keys at runtime, but only as a fixed chain: it can't say "every item" or "items where the price is over 100".) That makes it a good fit for flexible ETL (Extract, Transform, Load) processes and anywhere else a path should be data instead of code.
|
|
8
|
+
|
|
9
|
+
Paths are safe by default: a path can read your data but cannot call mutating or destructive methods such as `clear` or `destroy`. See [`:property_resolver`](#property_resolver) to allow more. See [Object traversal](#object-traversal) for examples, and [Alternatives](#alternatives) for when another gem is the better choice.
|
|
10
|
+
|
|
11
|
+
Only accept paths from sources you trust. The default resolver stops a path from calling methods that change your data, but a hostile path can still be expensive to evaluate: a regular expression in a filter can take exponential time to match (see [Known limitations](#known-limitations)).
|
|
8
12
|
|
|
9
13
|
Enumpath path expressions look like this:
|
|
10
14
|
|
|
11
|
-
```
|
|
15
|
+
```plain
|
|
12
16
|
$.pets.cats.0.name
|
|
13
17
|
|
|
14
18
|
$.pets[cats,dogs].*.name
|
|
@@ -22,13 +26,13 @@ pets..name
|
|
|
22
26
|
pets.cats.-1
|
|
23
27
|
```
|
|
24
28
|
|
|
25
|
-
Enumpath has the following benefits over vanilla `
|
|
29
|
+
Enumpath has the following benefits over vanilla `dig`:
|
|
26
30
|
|
|
27
31
|
- Paths can be described as simple strings
|
|
28
|
-
- It's smart enough to figure out which path segments are integer indexes versus symbolic keys versus string keys
|
|
32
|
+
- It's smart enough to figure out which path segments are integer indexes versus symbolic keys versus string keys versus struct members versus object readers
|
|
29
33
|
- It enables the use of wildcard, recursive descent, filter, subscript, union, and slice operators to describe complex paths through the data
|
|
30
34
|
|
|
31
|
-
Like `
|
|
35
|
+
Like `dig`, Enumpath protects against missing path segments and returns safely if the full path cannot be resolved.
|
|
32
36
|
|
|
33
37
|
## Installation
|
|
34
38
|
|
|
@@ -40,11 +44,15 @@ gem 'enumpath'
|
|
|
40
44
|
|
|
41
45
|
And then execute:
|
|
42
46
|
|
|
43
|
-
|
|
47
|
+
```shell
|
|
48
|
+
bundle
|
|
49
|
+
```
|
|
44
50
|
|
|
45
51
|
Or install it yourself as:
|
|
46
52
|
|
|
47
|
-
|
|
53
|
+
```shell
|
|
54
|
+
gem install enumpath
|
|
55
|
+
```
|
|
48
56
|
|
|
49
57
|
## Usage
|
|
50
58
|
|
|
@@ -72,6 +80,64 @@ Enumpath.apply("drinks.*", party) # => []
|
|
|
72
80
|
|
|
73
81
|
> This is a thoughtful deviation from the original JSONPath spec which would return `false` on no matches.
|
|
74
82
|
|
|
83
|
+
## Object traversal
|
|
84
|
+
|
|
85
|
+
The same path syntax works on whatever shape your data has. These examples are mirrored by `spec/integration_tests/object_traversal_spec.rb`, so they stay accurate.
|
|
86
|
+
|
|
87
|
+
A hash with symbol keys, using a wildcard and a filter:
|
|
88
|
+
|
|
89
|
+
```ruby
|
|
90
|
+
order = { customer: { name: 'Ada' }, items: [{ sku: 'A1', qty: 2 }, { sku: 'B2', qty: 1 }] }
|
|
91
|
+
Enumpath.apply('items.*.sku', order) # => ["A1", "B2"]
|
|
92
|
+
Enumpath.apply('items[?(@.qty > 1)].sku', order) # => ["A1"]
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
A `Struct` inside a hash, with a relative index:
|
|
96
|
+
|
|
97
|
+
```ruby
|
|
98
|
+
Point = Struct.new(:x, :y)
|
|
99
|
+
route = { stops: [Point.new(1, 2), Point.new(3, 4)] }
|
|
100
|
+
Enumpath.apply('stops.*.x', route) # => [1, 3]
|
|
101
|
+
Enumpath.apply('stops.-1.y', route) # => [4]
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
A plain Ruby object. Under the default `:safe` policy its `attr_reader`s resolve, and so do read-only methods such as `length` on the values they return:
|
|
105
|
+
|
|
106
|
+
```ruby
|
|
107
|
+
class Author
|
|
108
|
+
attr_reader :name, :books
|
|
109
|
+
|
|
110
|
+
def initialize(name, books)
|
|
111
|
+
@name = name
|
|
112
|
+
@books = books
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
def label
|
|
116
|
+
"#{name} (#{books.length} books)"
|
|
117
|
+
end
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
library = { authors: [Author.new('Le Guin', %w[Earthsea Dispossessed]), Author.new('Butler', %w[Kindred])] }
|
|
121
|
+
Enumpath.apply('authors.*.name', library) # => ["Le Guin", "Butler"]
|
|
122
|
+
Enumpath.apply('authors.0.books.length', library) # => [2]
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
A computed reader such as `label` is a method you wrote, not an `attr_reader`, so `:safe` does not call it. Ask for it with `property_resolver: :readers`, which also runs methods from installed gems and is not read-only, so use it on data you trust:
|
|
126
|
+
|
|
127
|
+
```ruby
|
|
128
|
+
Enumpath.apply('authors.*.label', library) # => []
|
|
129
|
+
Enumpath.apply('authors.*.label', library, property_resolver: :readers)
|
|
130
|
+
# => ["Le Guin (2 books)", "Butler (1 books)"]
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
An `OpenStruct`:
|
|
134
|
+
|
|
135
|
+
```ruby
|
|
136
|
+
require 'ostruct'
|
|
137
|
+
config = OpenStruct.new(db: OpenStruct.new(host: 'localhost', port: 5432))
|
|
138
|
+
Enumpath.apply('db.host', config) # => ["localhost"]
|
|
139
|
+
```
|
|
140
|
+
|
|
75
141
|
## Operator Reference
|
|
76
142
|
|
|
77
143
|
Enumpath currently implements the following path operators:
|
|
@@ -96,11 +162,11 @@ Child operators match on an index, key, member, or property of the enumerable. I
|
|
|
96
162
|
1. as an integer key or index (if the segment is integer-like),
|
|
97
163
|
2. then as a string key,
|
|
98
164
|
3. then as a symbol key,
|
|
99
|
-
4. and finally as a public property (i.e. a public method of the target that expects no arguments)
|
|
165
|
+
4. and finally as a public property (i.e. a public method of the target that expects no arguments), if the [`:property_resolver`](#property_resolver) option allows it. By default (`:safe`) only read-only access is allowed: allowlisted methods such as `length`, `size`, `count`, `first`, `keys` or `upcase` on core data (arrays, hashes, strings, symbols, numbers, structs, `Data` objects, `nil`, `true`, `false`), and `attr_reader`s on plain Ruby objects. A path segment like `clear`, `pop`, `sort!` or `destroy` matches nothing. Use `property_resolver: :readers` to also allow computed readers and ActiveRecord attributes (this also runs methods from installed gems and is not read-only; see [`:property_resolver`](#property_resolver)).
|
|
100
166
|
|
|
101
|
-
#### Examples
|
|
167
|
+
#### Operator Examples
|
|
102
168
|
|
|
103
|
-
```
|
|
169
|
+
```ruby
|
|
104
170
|
Car = Struct.new(:color, :transmition, :owners)
|
|
105
171
|
hyundai = Car.new('blue', :automatic, [{ name: 'Bill' }, { name: 'Ted' }])
|
|
106
172
|
subaru = Car.new('gold', :standard, [{ name: 'Kate' }])
|
|
@@ -141,7 +207,7 @@ The operator accepts a mixed bag of argument combinations. For instance, these a
|
|
|
141
207
|
|
|
142
208
|
Syntax: `[child1,child2,...]`
|
|
143
209
|
|
|
144
|
-
The union operator combines the results of two or more child operators. There is no limit to the number of child operators you can specify in a single union. Each child operator is separated by a comma (`,`). White space is stripped from around each child operator. Child operators can optionally be wrapped in single quotes. Bracket notation is not
|
|
210
|
+
The union operator combines the results of two or more child operators. There is no limit to the number of child operators you can specify in a single union. Each child operator is separated by a comma (`,`). White space is stripped from around each child operator. Child operators can optionally be wrapped in single quotes. Bracket notation is not supported in this context.
|
|
145
211
|
|
|
146
212
|
The following are all valid union operators:
|
|
147
213
|
|
|
@@ -161,7 +227,7 @@ The following are all valid filter expressions:
|
|
|
161
227
|
- `[?(isbn)]`: equivalent to the previous example
|
|
162
228
|
- `[?(@.price == 8)]`: any member whose `price` value is equal to 8
|
|
163
229
|
- `[?(@.price == 8 || @.price == 10)]`: members with a `price` of 8 _or_ 10
|
|
164
|
-
- `[?(@.price > 2 && @.price < 10)]`: members with a `price` greater than
|
|
230
|
+
- `[?(@.price > 2 && @.price < 10)]`: members with a `price` greater than 2 _and_ less than 10
|
|
165
231
|
- `[?(@.name =~ /bob/i || @.name == 'Mark')]`: any member whose `name` matches the regex `/bob/i` or equals `'Mark'`
|
|
166
232
|
|
|
167
233
|
> Regular expression operands are safely parsed using the `to_regexp` gem
|
|
@@ -170,15 +236,24 @@ The following are all valid filter expressions:
|
|
|
170
236
|
|
|
171
237
|
Syntax: `[(expression)]`
|
|
172
238
|
|
|
173
|
-
A subscript expression is made up of a
|
|
239
|
+
A subscript expression is made up of a single expression that consists of a child operator (`@.child` or `child`; the leading `@.` is optional), plus an optional pair of arithmetic operator and operand. The arithmetic operator can be any one of `+`, `-`, `**`, `*`, `/`, or `%`. The operand can be a string (`'some string'`), symbol (`:some_symbol`), or numeric value (`10` or `1.0`). The expression is evaluated and the result becomes the subscript. If an operator and operand are not included in an expression then the value located at the child operator is used as the subscript. If the subscript represents a valid child path for the enumerable, the value of that member will be passed along for further processing of the path.
|
|
174
240
|
|
|
175
|
-
The following are all valid
|
|
241
|
+
The following are all valid subscript expressions:
|
|
176
242
|
|
|
177
243
|
- `[(@.length - 1)]`: the subscript becomes the length of the current enumerable, minus 1
|
|
178
244
|
- `[(length / 2)]`: the subscript becomes the index at half the length of the enumerable
|
|
179
245
|
- `[(@.type)]`: the subscript becomes the value at the `type` key, member, or property of the enumerable
|
|
180
246
|
|
|
181
|
-
|
|
247
|
+
An expression that cannot be evaluated against the data, such as a division by zero or a string plus a number, matches nothing instead of raising an error. So does an expression that could use an unbounded amount of CPU or memory, because the operand comes from the path. The limits are:
|
|
248
|
+
|
|
249
|
+
- an integer operand must fit in 64 bits,
|
|
250
|
+
- repeating a string or array with `*` may produce at most 10,000 characters or elements,
|
|
251
|
+
- an integer raised to a power with `**` must produce a result of at most 1,024 bits, and
|
|
252
|
+
- `%` only applies to numbers (on a string it would be `format`, which can build a huge string).
|
|
253
|
+
|
|
254
|
+
These are far above anything useful for an index or a key.
|
|
255
|
+
|
|
256
|
+
## Usage Examples
|
|
182
257
|
|
|
183
258
|
Given the same store example from the JSONPath project:
|
|
184
259
|
|
|
@@ -379,6 +454,45 @@ Enumpath.apply("food.*", party, result_type: :path) # => ["$['food'][0]", "$['fo
|
|
|
379
454
|
|
|
380
455
|
Each returned path is a valid path expression that can be used in calls to `Enumpath.apply`. If you want to be explicit about returning values instead of paths you can specify that with the option `result_type: :value`.
|
|
381
456
|
|
|
457
|
+
### :property_resolver
|
|
458
|
+
|
|
459
|
+
When a path segment is not an index, key, or struct or `Data` member, Enumpath can resolve it by calling a method on the object (step 4 of the [child operator](#child-operator)). Wildcards, filters and recursive descent also use this option when they need the keys of an object that is not an array, hash, struct or `Data` object (they call its `to_h` only if the option allows it). The `:property_resolver` option controls what a path is allowed to call:
|
|
460
|
+
|
|
461
|
+
value | what resolves | read-only?
|
|
462
|
+
--- | --- | ---
|
|
463
|
+
`:safe` (default) | allowlisted read-only methods on core data, `attr_reader`s on plain objects | yes
|
|
464
|
+
`:readers` | `:safe`, plus public no-argument methods written in Ruby, including installed gems (Ruby's built-in and standard library methods excluded), and ActiveModel-style `attributes` | no
|
|
465
|
+
`:none` | nothing; only indexes, keys and members | yes
|
|
466
|
+
`:all` | any public method that expects no arguments (the behavior before this option existed) | no
|
|
467
|
+
a callable | whatever the callable allows | up to you
|
|
468
|
+
|
|
469
|
+
- `:safe` (the default): read-only access. On core data (arrays, hashes, strings, symbols, numbers, structs, `Data` objects, `nil`, `true`, `false`) Enumpath calls a fixed allowlist of non-mutating methods such as `length`, `size`, `count`, `first`, `last`, `min`, `max`, `sum`, `keys`, `values`, `to_a`, `to_h`, `to_s`, `empty?`, `zero?`, `upcase` or `strip`, and only when the receiver's class does not override them. On plain Ruby objects it reads instance variables that have a public reader of the same name (`attr_reader`s); the instance variable is read directly, so the method itself never runs. Names ending in `!` or `=` are never called, and neither is any method that requires arguments.
|
|
470
|
+
- `:readers`: everything `:safe` allows, plus, on objects that are not core data, any public method that expects no arguments and is written in Ruby, except Ruby's built-in methods, methods in the Ruby standard library directory, and a short denylist of persistence verbs (`destroy`, `delete`, `save`, `update`, `reload`, `touch`, `delete_all`, `destroy_all`, `toggle`). It also reads the keys of an ActiveModel-style `attributes` Hash. So computed readers like `full_name`, `attr_reader`s and ActiveRecord columns like `email` work. Methods implemented in C (core classes, `StringIO`, C-extension gems) are never called; for objects of C-extension classes use a callable or `:all`. Core data is treated exactly like `:safe`. **This is not read-only.** It includes methods from installed gems, including default and bundled gems such as Logger and CSV, and ActiveModel or ActiveRecord: for example `Logger#close` would run, and so would a model method like `archive` or `cancel`. The denylist is a safety net, not a guarantee. Use `:readers` for data you trust or objects you built; for untrusted paths use `:safe` or a callable.
|
|
471
|
+
- `:none`: never call methods.
|
|
472
|
+
- `:all`: call any public method that expects no arguments. This is how Enumpath behaved before this option existed, and it lets a path run methods such as `clear`, `pop`, `sort!` or `destroy`. Only use it with paths you trust.
|
|
473
|
+
- a callable, such as a lambda: called with the object and the method name (a String). The method is called only if it returns true. The callable has full control: none of the checks above are applied, so call `Enumpath::Resolver::Property.safe?(object, name)` or `.reader?(object, name)` from it if you want to keep them and allow more.
|
|
474
|
+
|
|
475
|
+
```ruby
|
|
476
|
+
party = { food: %w[pizza tacos] }
|
|
477
|
+
Enumpath.apply('food.pop', party) # => [] and party is unchanged
|
|
478
|
+
Enumpath.apply('food.length', party) # => [2]
|
|
479
|
+
Enumpath.apply('food.length', party, property_resolver: :none) # => []
|
|
480
|
+
|
|
481
|
+
# ActiveRecord models you trust: computed readers and columns
|
|
482
|
+
Enumpath.apply('[?(@.published)].title', Post.all.to_a, property_resolver: :readers)
|
|
483
|
+
|
|
484
|
+
# Recommended for objects you do not control: keep the safe defaults and allow specific methods per class
|
|
485
|
+
ALLOWED = { 'Invoice' => %w[total due_on], 'Customer' => %w[name email] }.freeze
|
|
486
|
+
policy = lambda do |object, name|
|
|
487
|
+
Enumpath::Resolver::Property.safe?(object, name) || ALLOWED.fetch(object.class.name, []).include?(name)
|
|
488
|
+
end
|
|
489
|
+
Enumpath.apply('invoices.*.total', account, property_resolver: policy)
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
The option applies to a single call. A chained `results.apply(...)` uses the default unless you pass the option again. Any other value raises an `ArgumentError`.
|
|
493
|
+
|
|
494
|
+
To decide whether an object has a method, Enumpath asks Ruby (`respond_to?` and `method`), which calls the object's `respond_to_missing?` if it defines one. So even under `:safe`, a path can run a little of your application's code that way.
|
|
495
|
+
|
|
382
496
|
### :verbose
|
|
383
497
|
|
|
384
498
|
Seeing how your path expression is being applied to an enumerable can be helpful in understanding the path expression syntax. Enumpath has a built-in logger to assist with this. It can be enabled by simply passing `verbose: true` as an option on `Enumpath.apply`. By default this will log debugging information to STDOUT, however you can provide your own logger.
|
|
@@ -391,7 +505,7 @@ Enumpath.logger.logger = ::Logger.new('log/enumpath.log')
|
|
|
391
505
|
|
|
392
506
|
Once enabled, it will log debugging information like so:
|
|
393
507
|
|
|
394
|
-
```
|
|
508
|
+
```plain
|
|
395
509
|
Enumpath.apply('$.store.book', store_info, verbose: true)
|
|
396
510
|
|
|
397
511
|
--------------------------------------
|
|
@@ -428,7 +542,7 @@ You can also control verbose mode via `Enumpath.verbose = true` and `Enumpath.ve
|
|
|
428
542
|
|
|
429
543
|
## Path normalization
|
|
430
544
|
|
|
431
|
-
When you give a string path to Enumpath it will automatically normalize it to an array of path segments. You can also pass it an array of path segments to avoid the normalization, for instance if the normalization process is having trouble parsing your path, or you happen to have a pre-normalized path already. For example the path `['pets']..[?(@.age > 10)].name` is represented in normalized form as `['pets', '..', '?(@.age > 10)', 'name']`. For the most part you should stick with string paths and let Enumpath normalize on its own.
|
|
545
|
+
When you give a string path to Enumpath it will automatically normalize it to an array of path segments. You can also pass it an array of path segments to avoid the normalization, for instance if the normalization process is having trouble parsing your path, or you happen to have a pre-normalized path already. For example the path `['pets']..[?(@.age > 10)].name` is represented in normalized form as `['pets', '..', '?(@.age > 10)', 'name']`. The segments of an array path are converted to strings and the array is used as given, without normalizing or caching it. A path that is neither a String nor an Array raises an `ArgumentError`. For the most part you should stick with string paths and let Enumpath normalize on its own.
|
|
432
546
|
|
|
433
547
|
### Normalized path caching
|
|
434
548
|
|
|
@@ -437,20 +551,52 @@ To save a little bit of time on consecutive calls Enumpath caches the normalized
|
|
|
437
551
|
## Deviations from the Original JSONPath Spec
|
|
438
552
|
|
|
439
553
|
1. The JSONPath spec required that `false` be returned when no matches were found, but Enumpath will return an empty result set (`[]`) instead. This is a thoughtful divergence based on the principle of least astonishment and the robustness principle.
|
|
440
|
-
|
|
441
554
|
2. Enumpath supports relative child indexes, which the original implementation did not support. For instance:
|
|
442
555
|
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
556
|
+
```ruby
|
|
557
|
+
# Get the last element. Both are equivalent to `$..book[-1:]`
|
|
558
|
+
Enumpath.apply('$..book.-1', store_info)
|
|
559
|
+
Enumpath.apply('$..book[-1]', store_info)
|
|
560
|
+
```
|
|
448
561
|
|
|
562
|
+
3. The original implementations of JSONPath allowed unchecked evaluation of filter and subscript expressions. Enumpath limits those expressions to a reasonable subset of operations as detailed in the [Operator Reference](#operator-reference) section. It never uses `eval`: the comparison and arithmetic operators are called by name from a fixed list, and a path can only call methods on your data as the [`:property_resolver`](#property_resolver) option allows (read-only by default).
|
|
449
563
|
4. The original JSONPath spec did not include support for using logical operators to chain expressions in filter expression operators. This addition was inspired by [Gergely Brautigam's][replace-eval-with-object-send-and-a-parser] work on [joshbuddy/jsonpath][jsonpath-gh]
|
|
450
564
|
|
|
565
|
+
## Known limitations
|
|
566
|
+
|
|
567
|
+
These are tracked as issues, and contributions are welcome.
|
|
568
|
+
|
|
569
|
+
- A quoted bracket child cannot contain a dot: `['a.b']` matches nothing ([issue 41](https://github.com/chrisbloom7/enumpath/issues/41)).
|
|
570
|
+
- `&&` and `||` inside a quoted string in a filter expression split the expression ([issue 42](https://github.com/chrisbloom7/enumpath/issues/42)).
|
|
571
|
+
- A key whose value is `nil` returns the same empty result as a missing key ([issue 43](https://github.com/chrisbloom7/enumpath/issues/43)).
|
|
572
|
+
- A negative slice step such as `[::-1]` matches nothing ([issue 44](https://github.com/chrisbloom7/enumpath/issues/44)).
|
|
573
|
+
- Malformed paths are not rejected: `a[` is read as `a` ([issue 45](https://github.com/chrisbloom7/enumpath/issues/45)).
|
|
574
|
+
- Recursive descent raises `SystemStackError` on data nested a few thousand levels deep ([issue 46](https://github.com/chrisbloom7/enumpath/issues/46)).
|
|
575
|
+
- A regular expression in a filter can take exponential time to match, so only accept paths you trust ([issue 60](https://github.com/chrisbloom7/enumpath/issues/60)).
|
|
576
|
+
|
|
577
|
+
## Alternatives
|
|
578
|
+
|
|
579
|
+
Enumpath is not the only JSONPath library for Ruby, and for some data it is not the best one:
|
|
580
|
+
|
|
581
|
+
- [`json_p3`][json-p3] follows [RFC 9535][rfc-9535] (the JSONPath standard) and works on parsed JSON only, by design. It is the standards-compliant choice and faster than Enumpath (about 2 to 4 times in a benchmark on a 10,000-element store example).
|
|
582
|
+
- [`jsonpath`][jsonpath-gh] is the long-standing de facto Ruby gem. It was about as fast as Enumpath in the same benchmark.
|
|
583
|
+
- Enumpath is **not** RFC 9535 compliant. It implements the original JSONPath spec with some deviations (see [above](#deviations-from-the-original-jsonpath-spec)).
|
|
584
|
+
|
|
585
|
+
If your data is parsed JSON with string keys, either alternative works and `json_p3` is the standards-compliant, faster choice. Enumpath is for data that is not plain JSON, in particular symbol keys (`jsonpath` loses track of objects nested inside a symbol-keyed Hash) and paths that should only be allowed to read, not call arbitrary methods (see [`:property_resolver`](#property_resolver)). The same queries behave like this across the three gems (checked with `jsonpath` 1.1.5 and `json_p3` 1.0.1 on Ruby 3.4):
|
|
586
|
+
|
|
587
|
+
data | Enumpath | `jsonpath` | `json_p3`
|
|
588
|
+
--- | --- | --- | ---
|
|
589
|
+
Hash and Array with string keys | works | works | works
|
|
590
|
+
Hash with symbol keys | works | no match | no match
|
|
591
|
+
`Struct` | works | works for top-level members; no match once a `Struct` is inside a symbol-keyed Hash | no match
|
|
592
|
+
Plain Ruby object with readers | works (`attr_reader`s under `:safe`, computed readers under `:readers`) | works (it calls any method), except inside a symbol-keyed Hash | no match
|
|
593
|
+
`OpenStruct` | works | works, except inside a symbol-keyed Hash | no match
|
|
594
|
+
|
|
595
|
+
If you only need a fixed path into data you already know, plain `dig` is simpler and much faster (about 50 times faster than Enumpath on a simple path).
|
|
596
|
+
|
|
451
597
|
## Requirements
|
|
452
598
|
|
|
453
|
-
Enumpath requires Ruby 2.
|
|
599
|
+
Enumpath requires Ruby 2.6.0 or higher.
|
|
454
600
|
|
|
455
601
|
## Development
|
|
456
602
|
|
|
@@ -458,7 +604,7 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
|
|
|
458
604
|
|
|
459
605
|
## Contributing
|
|
460
606
|
|
|
461
|
-
Bug reports and pull requests are welcome on GitHub at [chrisbloom7/enumpath][enumpath-gh]. Please read [CONTRIBUTING.md]
|
|
607
|
+
Bug reports and pull requests are welcome on GitHub at [chrisbloom7/enumpath][enumpath-gh]. Please read [CONTRIBUTING.md][contributing] for details on our code of conduct, and the process for submitting pull requests to us.
|
|
462
608
|
|
|
463
609
|
## Versioning
|
|
464
610
|
|
|
@@ -489,18 +635,21 @@ Enumpath is based on [Stefan Goessner's JSONPath spec][jsonpath], and was inspir
|
|
|
489
635
|
|
|
490
636
|
[apache-license-2-0]: http://www.apache.org/licenses/LICENSE-2.0
|
|
491
637
|
[chris-bloom-gh]: https://github.com/chrisbloom7
|
|
492
|
-
[ci]: https://circleci.com/gh/chrisbloom7/enumpath
|
|
493
638
|
[ci-image]: https://circleci.com/gh/chrisbloom7/enumpath.svg?style=svg
|
|
639
|
+
[ci]: https://circleci.com/gh/chrisbloom7/enumpath
|
|
640
|
+
[contributing]: https://github.com/chrisbloom7/enumpath/blob/main/CONTRIBUTING.md
|
|
494
641
|
[contributors]: https://github.com/chrisbloom7/enumpath/graphs/contributors
|
|
495
642
|
[enumpath-gh]: https://github.com/chrisbloom7/enumpath
|
|
496
|
-
[hound]: https://houndci.com
|
|
497
643
|
[hound-badge-image]: https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg
|
|
498
|
-
[
|
|
644
|
+
[hound]: https://houndci.com
|
|
499
645
|
[jsonpath-gh]: https://github.com/joshbuddy/jsonpath
|
|
646
|
+
[json-p3]: https://github.com/jg-rp/ruby-json-p3
|
|
647
|
+
[jsonpath]: http://goessner.net/articles/JsonPath/
|
|
500
648
|
[keypath-ruby]: https://github.com/nickcharlton/keypath-ruby
|
|
501
649
|
[replace-eval-with-object-send-and-a-parser]: https://skarlso.github.io/2017/05/28/replace-eval-with-object-send-and-a-parser/
|
|
650
|
+
[rfc-9535]: https://www.rfc-editor.org/rfc/rfc9535
|
|
502
651
|
[semver]: http://semver.org/
|
|
503
652
|
[tags]: https://github.com/chrisbloom7/enumpath/tags
|
|
504
|
-
[version]: https://badge.fury.io/rb/enumpath
|
|
505
653
|
[version-image]: https://badge.fury.io/rb/enumpath.svg
|
|
654
|
+
[version]: https://badge.fury.io/rb/enumpath
|
|
506
655
|
[youearnedit]: http://youearnedit.com
|
|
@@ -16,7 +16,7 @@ module Enumpath
|
|
|
16
16
|
# @param enum [Enumerable] an enum that can be used to assist in detection. Not all subclasses require an enum
|
|
17
17
|
# for detection.
|
|
18
18
|
# @return [true, false] whether the operator param appears to represent the operator class
|
|
19
|
-
def detect?(
|
|
19
|
+
def detect?(operator, enum = nil)
|
|
20
20
|
raise NotImplementedError
|
|
21
21
|
end
|
|
22
22
|
end
|
|
@@ -47,7 +47,7 @@ module Enumpath
|
|
|
47
47
|
# @yieldparam enum [Enumerable] the new enum after applying the operator
|
|
48
48
|
# @yieldparam resolved_path [Array] the new resolved_path after applying the operator
|
|
49
49
|
# @yieldreturn [void]
|
|
50
|
-
def apply(
|
|
50
|
+
def apply(remaining_path, enum, resolved_path)
|
|
51
51
|
raise NotImplementedError
|
|
52
52
|
end
|
|
53
53
|
|
|
@@ -62,22 +62,37 @@ module Enumpath
|
|
|
62
62
|
private
|
|
63
63
|
|
|
64
64
|
# Returns a set of keys, member names, or indices for a given enumerable. Useful for operators that need to
|
|
65
|
-
# iterate over each member of an enumerable.
|
|
66
|
-
#
|
|
67
|
-
#
|
|
65
|
+
# iterate over each member of an enumerable.
|
|
66
|
+
#
|
|
67
|
+
# - Arrays return their indexes, Hashes their keys, and Structs and Data objects their member names. These are
|
|
68
|
+
# read through Ruby's own implementations, so a subclass that overrides them is not called.
|
|
69
|
+
# - Any other object returns the keys of its `to_h`, but only if the current property resolver policy (see
|
|
70
|
+
# {Enumpath::Resolver::Property}) allows calling `to_h` and it returns a Hash. Under the default `:safe`
|
|
71
|
+
# policy a plain object's own `to_h` is never called, so iterating over it cannot run its code or consume an
|
|
72
|
+
# Enumerator.
|
|
73
|
+
# - Everything else returns an empty array.
|
|
68
74
|
#
|
|
69
75
|
# @param enum [Enumerable] the object to detect keys, member names, or indices on.
|
|
70
76
|
# @return [Array] a set of keys, member names, or indices for the object, or an empty set if they could not be
|
|
71
77
|
# determined.
|
|
72
78
|
def keys(enum)
|
|
73
|
-
|
|
74
|
-
return
|
|
79
|
+
core_keys = core_keys(enum)
|
|
80
|
+
return core_keys unless core_keys.nil?
|
|
75
81
|
|
|
76
|
-
|
|
77
|
-
|
|
82
|
+
hash = Enumpath::Resolver::Property.resolve('to_h', enum)
|
|
83
|
+
hash.is_a?(Hash) ? hash.keys : []
|
|
84
|
+
end
|
|
78
85
|
|
|
79
|
-
|
|
80
|
-
|
|
86
|
+
def core_keys(enum)
|
|
87
|
+
if enum.is_a?(Array)
|
|
88
|
+
(0...Array.instance_method(:length).bind(enum).call).to_a
|
|
89
|
+
elsif enum.is_a?(Hash)
|
|
90
|
+
Hash.instance_method(:keys).bind(enum).call
|
|
91
|
+
elsif enum.is_a?(Struct)
|
|
92
|
+
Struct.instance_method(:members).bind(enum).call
|
|
93
|
+
elsif Enumpath::Resolver::Simple.data?(enum)
|
|
94
|
+
Enumpath::Resolver::Simple.data_members(enum)
|
|
95
|
+
end
|
|
81
96
|
end
|
|
82
97
|
end
|
|
83
98
|
end
|
|
@@ -8,6 +8,21 @@ module Enumpath
|
|
|
8
8
|
ARITHMETIC_OPERATOR_REGEX = %r{(\+|-|\*\*|\*|\/|%)}
|
|
9
9
|
OPERATOR_REGEX = /^\((.*)\)$/
|
|
10
10
|
|
|
11
|
+
# The operand of an arithmetic expression comes from the path, so an expression that could use an unbounded
|
|
12
|
+
# amount of CPU or memory is not evaluated and matches nothing. A subscript is an index or a key, so none of
|
|
13
|
+
# these limits get in the way of a useful expression.
|
|
14
|
+
#
|
|
15
|
+
# The widest integer operand allowed, in bits
|
|
16
|
+
MAX_OPERAND_BITS = 64
|
|
17
|
+
# The most elements or characters that repeating a string or array (`*`) may produce
|
|
18
|
+
MAX_REPEAT_SIZE = 10_000
|
|
19
|
+
# The widest result allowed for an integer raised to a power (`**`), in bits
|
|
20
|
+
MAX_POWER_BITS = 1_024
|
|
21
|
+
|
|
22
|
+
# Errors from evaluating an expression against data that does not fit it, such as a string plus a number or a
|
|
23
|
+
# division by zero. The expression matches nothing in these cases.
|
|
24
|
+
EVALUATION_ERRORS = [NoMethodError, TypeError, ArgumentError, ZeroDivisionError, RangeError].freeze
|
|
25
|
+
|
|
11
26
|
class << self
|
|
12
27
|
# Whether the operator matches {Enumpath::Operator::SubscriptExpression::OPERATOR_REGEX}
|
|
13
28
|
#
|
|
@@ -58,13 +73,18 @@ module Enumpath
|
|
|
58
73
|
else
|
|
59
74
|
evaluate_with_operator(operator, operand, value)
|
|
60
75
|
end
|
|
61
|
-
rescue
|
|
76
|
+
rescue *EVALUATION_ERRORS
|
|
62
77
|
Enumpath.log('Subscript could not be evaluated') { { subscript: nil } }
|
|
63
78
|
nil
|
|
64
79
|
end
|
|
65
80
|
|
|
66
81
|
def evaluate_with_operator(operator, operand, value)
|
|
67
82
|
typecast_operand = variable_typecaster(operand)
|
|
83
|
+
unless within_limits?(operator, typecast_operand, value)
|
|
84
|
+
Enumpath.log('Subscript exceeds the arithmetic limits') { { operator: operator, operand: typecast_operand } }
|
|
85
|
+
return nil
|
|
86
|
+
end
|
|
87
|
+
|
|
68
88
|
result = value.public_send(operator.to_sym, typecast_operand)
|
|
69
89
|
Enumpath.log('Evaluated subscript') do
|
|
70
90
|
{ value: value, operator: operator, operand: typecast_operand, result: result }
|
|
@@ -72,6 +92,35 @@ module Enumpath
|
|
|
72
92
|
result
|
|
73
93
|
end
|
|
74
94
|
|
|
95
|
+
# Whether evaluating `value <operator> operand` stays inside the limits above, before any work is done
|
|
96
|
+
def within_limits?(operator, operand, value)
|
|
97
|
+
return false if operand.is_a?(Integer) && operand.abs.bit_length > MAX_OPERAND_BITS
|
|
98
|
+
|
|
99
|
+
case operator
|
|
100
|
+
when '*' then repetition_within_limit?(value, operand)
|
|
101
|
+
when '**' then power_within_limit?(value, operand)
|
|
102
|
+
when '%' then value.is_a?(Numeric) # `String#%` is `format`, which can build a huge string
|
|
103
|
+
else true
|
|
104
|
+
end
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
# Repeating a string or array by an integer multiplies its size. (`Array#*` with a string joins instead.)
|
|
108
|
+
def repetition_within_limit?(value, operand)
|
|
109
|
+
return true unless operand.is_a?(Integer) && (value.is_a?(String) || value.is_a?(Array))
|
|
110
|
+
|
|
111
|
+
operand >= 0 && value.size * operand <= MAX_REPEAT_SIZE
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
# A float overflows to Infinity at once. An integer power is only allowed if its result fits in MAX_POWER_BITS,
|
|
115
|
+
# estimated as the bit length of the base times the exponent. Any other kind of number is not allowed.
|
|
116
|
+
def power_within_limit?(value, operand)
|
|
117
|
+
case value
|
|
118
|
+
when Integer then operand.is_a?(Integer) && value.abs.bit_length * operand.abs <= MAX_POWER_BITS
|
|
119
|
+
when Float then true
|
|
120
|
+
else false
|
|
121
|
+
end
|
|
122
|
+
end
|
|
123
|
+
|
|
75
124
|
def variable_typecaster(variable)
|
|
76
125
|
if variable =~ /\A('|").+\1\z/ || variable =~ /^:.+/
|
|
77
126
|
# It quacks like a string or symbol
|
data/lib/enumpath/path.rb
CHANGED
|
@@ -11,7 +11,8 @@ module Enumpath
|
|
|
11
11
|
# @return [Enumpath::Results] the current results array
|
|
12
12
|
attr_reader :results
|
|
13
13
|
|
|
14
|
-
# @param path [String, Array<String>] the path expression to apply to the enumerable
|
|
14
|
+
# @param path [String, Array<String>] the path expression to apply to the enumerable, or an array of path segments
|
|
15
|
+
# @raise [ArgumentError] if the path is neither a String nor an Array
|
|
15
16
|
# @param result_type (see Enumpath::Results#initialize)
|
|
16
17
|
def initialize(path, result_type: nil)
|
|
17
18
|
@path = path
|
|
@@ -57,7 +58,7 @@ module Enumpath
|
|
|
57
58
|
|
|
58
59
|
def apply_segments(path_segments, enum, resolved_path, nesting_level)
|
|
59
60
|
segment = path_segments.first
|
|
60
|
-
remaining_path = path_segments[1
|
|
61
|
+
remaining_path = path_segments[1..]
|
|
61
62
|
operator = Enumpath::Operator.detect(segment, enum)
|
|
62
63
|
operator&.apply(remaining_path, enum, resolved_path) do |s, e, c|
|
|
63
64
|
trace(s, e, c, nesting_level + 1)
|
|
@@ -68,8 +69,14 @@ module Enumpath
|
|
|
68
69
|
@cache ||= Enumpath.path_cache
|
|
69
70
|
end
|
|
70
71
|
|
|
72
|
+
# String paths are normalized once and cached. An array is already a list of path segments, so it only has its
|
|
73
|
+
# segments converted to strings and never touches the cache.
|
|
71
74
|
def normalize!
|
|
72
|
-
@path =
|
|
75
|
+
@path = case @path
|
|
76
|
+
when String then cache.get_or_set(@path) { Enumpath::Path::NormalizedPath.new(@path) }
|
|
77
|
+
when Array then Enumpath::Path::NormalizedPath.new(@path.map(&:to_s))
|
|
78
|
+
else raise ArgumentError, 'path must be a String or an Array of path segments'
|
|
79
|
+
end
|
|
73
80
|
end
|
|
74
81
|
end
|
|
75
82
|
end
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'rbconfig'
|
|
4
|
+
|
|
5
|
+
module Enumpath
|
|
6
|
+
module Resolver
|
|
7
|
+
# Helpers for inspecting an object without trusting it. Kernel methods are called through
|
|
8
|
+
# `Kernel.instance_method(...).bind`, so an object that overrides `is_a?`, `respond_to?`, `method` or the instance
|
|
9
|
+
# variable methods cannot change the answer.
|
|
10
|
+
#
|
|
11
|
+
# @private
|
|
12
|
+
module ObjectInspector
|
|
13
|
+
# The Kernel methods {.kernel} can call
|
|
14
|
+
KERNEL = %i[is_a? respond_to? method instance_variable_defined? instance_variable_get].map do |name|
|
|
15
|
+
[name, Kernel.instance_method(name)]
|
|
16
|
+
end.to_h.freeze
|
|
17
|
+
|
|
18
|
+
# Ruby's standard library directory (`RbConfig::CONFIG['rubylibdir']`). Installed gems, including default and
|
|
19
|
+
# bundled gems such as Logger or CSV, live elsewhere and are not covered by it.
|
|
20
|
+
RUBY_LIBRARY_DIR = File.join(RbConfig::CONFIG['rubylibdir'], '')
|
|
21
|
+
|
|
22
|
+
module_function
|
|
23
|
+
|
|
24
|
+
# Calls one of the {KERNEL} methods on an object
|
|
25
|
+
#
|
|
26
|
+
# @raise [TypeError] if the object does not include Kernel (a BasicObject)
|
|
27
|
+
def kernel(object, method_name, *args)
|
|
28
|
+
KERNEL[method_name].bind(object).call(*args)
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# The public method of that name, if the object has one that expects no arguments
|
|
32
|
+
#
|
|
33
|
+
# @return [Method, nil]
|
|
34
|
+
def public_method_without_required_args(object, name)
|
|
35
|
+
return unless kernel(object, :respond_to?, name)
|
|
36
|
+
|
|
37
|
+
method = kernel(object, :method, name)
|
|
38
|
+
method unless method.parameters.any? { |type, _| %i[req keyreq].include?(type) }
|
|
39
|
+
rescue NameError, TypeError
|
|
40
|
+
nil
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
# Whether `:readers` may call a method: it is written in Ruby (it has a source location, so it is not
|
|
44
|
+
# implemented in C), it is not one of Ruby's built-in methods (`<internal:...>`), and it is not in the Ruby
|
|
45
|
+
# standard library directory ({RUBY_LIBRARY_DIR}). Methods from installed gems pass this check, including
|
|
46
|
+
# default and bundled gems such as Logger and CSV, and so do attr_readers and methods made with
|
|
47
|
+
# `define_method` or `class_eval`.
|
|
48
|
+
#
|
|
49
|
+
# @param method [Method, nil]
|
|
50
|
+
# @return [true, false]
|
|
51
|
+
def application_method?(method)
|
|
52
|
+
location = method&.source_location
|
|
53
|
+
return false if location.nil? || location.empty?
|
|
54
|
+
|
|
55
|
+
file = location.first.to_s
|
|
56
|
+
!file.start_with?('<internal:') && !file.start_with?(RUBY_LIBRARY_DIR)
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# Whether the object is an instance of any of the classes. Objects that do not include Kernel count as a match,
|
|
60
|
+
# so callers treat them as off limits.
|
|
61
|
+
#
|
|
62
|
+
# @return [true, false]
|
|
63
|
+
def instance_of_any?(object, classes)
|
|
64
|
+
classes.any? { |klass| kernel(object, :is_a?, klass) }
|
|
65
|
+
rescue TypeError
|
|
66
|
+
true
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|