evalidate 2.1.4__tar.gz → 2.1.6__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {evalidate-2.1.4 → evalidate-2.1.6}/.gitignore +1 -0
- evalidate-2.1.6/PKG-INFO +387 -0
- evalidate-2.1.6/README.md +370 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/evalidate/__init__.py +7 -1
- {evalidate-2.1.4 → evalidate-2.1.6}/evalidate/cli/jg.py +35 -4
- {evalidate-2.1.4 → evalidate-2.1.6}/evalidate/credits.txt +1 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/evalidate/security.py +14 -0
- evalidate-2.1.4/PKG-INFO +0 -371
- evalidate-2.1.4/README.md +0 -354
- evalidate-2.1.4/products.json +0 -10002
- {evalidate-2.1.4 → evalidate-2.1.6}/LICENSE +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/benchmark/benchmark.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/evalidate/cli/genfakeproducts.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/examples/basic.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/pyproject.toml +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/setup.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/tests/__init__.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/tests/test_exc.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/tests/test_expr.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/tests/test_feature.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/tests/test_jailbreak.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/tests/test_locals_globals.py +0 -0
- {evalidate-2.1.4 → evalidate-2.1.6}/tests/test_no_mutation.py +0 -0
evalidate-2.1.6/PKG-INFO
ADDED
|
@@ -0,0 +1,387 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: evalidate
|
|
3
|
+
Version: 2.1.6
|
|
4
|
+
Summary: Validation and secure evaluation of untrusted python expressions
|
|
5
|
+
Project-URL: Homepage, https://github.com/yaroslaff/evalidate
|
|
6
|
+
Project-URL: Issues, https://github.com/yaroslaff/evalidate/issues
|
|
7
|
+
Author-email: Yaroslav Polyakov <yaroslaff@gmail.com>
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Keywords: eval,evaluate,evaluation,expression,filter,grep,jg,jq,json,python,safe,sandbox,secure,security,validate
|
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
+
Classifier: Operating System :: OS Independent
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Requires-Python: >=3.8
|
|
14
|
+
Provides-Extra: generate
|
|
15
|
+
Requires-Dist: faker; extra == 'generate'
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
|
|
18
|
+
# Evalidate
|
|
19
|
+
|
|
20
|
+
Evalidate is a simple Python library for safely evaluating untrusted, user-supplied Python expressions. It validates expressions before execution and rejects potentially malicious code.
|
|
21
|
+
|
|
22
|
+
Evalidate also comes with a convenient, high-performance CLI tool, `jg`, for filtering JSON lists using Python expressions:
|
|
23
|
+
|
|
24
|
+
```shell
|
|
25
|
+
jg 'attributes["storage"] == "512GB" and price<300 and rating>4' products.json
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Purpose
|
|
29
|
+
|
|
30
|
+
Evalidate was originally developed to filter complex data structures. For example, to find inexpensive smartphones that are available for sale:
|
|
31
|
+
|
|
32
|
+
```python
|
|
33
|
+
category=="smartphones" and price<300 and stock>0
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
However, it can also evaluate other types of expressions, including arithmetic expressions:
|
|
37
|
+
|
|
38
|
+
```python
|
|
39
|
+
a+b-100
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Evalidate is the fastest secure Python `eval()` alternative I know of.
|
|
43
|
+
|
|
44
|
+
## Installation
|
|
45
|
+
|
|
46
|
+
```shell
|
|
47
|
+
pip3 install evalidate
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## TL;DR: Just give me safe `eval()`!
|
|
51
|
+
|
|
52
|
+
```python
|
|
53
|
+
from evalidate import base_eval_model, EvalException
|
|
54
|
+
|
|
55
|
+
src = 'a + 10 > b'
|
|
56
|
+
# src = "__import__('os').system('clear')"
|
|
57
|
+
|
|
58
|
+
model = base_eval_model.clone()
|
|
59
|
+
expr = model.expr(src)
|
|
60
|
+
|
|
61
|
+
try:
|
|
62
|
+
print(expr.eval({'a':10, 'b':142}))
|
|
63
|
+
except EvalException as e:
|
|
64
|
+
print(e)
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Output:
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
True
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
If you try to execute the potentially dangerous expression (uncomment the second `src` line), the output will be:
|
|
74
|
+
|
|
75
|
+
```
|
|
76
|
+
ERR: Operation type Call is not allowed
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Here's a more practical example: finding inexpensive smartphones that meet specific requirements.
|
|
80
|
+
|
|
81
|
+
```python
|
|
82
|
+
from evalidate import base_eval_model, EvalException
|
|
83
|
+
|
|
84
|
+
mobiles = [
|
|
85
|
+
{'brand': 'Apple', 'model': 'iPhone 15', 'price': 799, 'RAM': 6, 'storage': 128},
|
|
86
|
+
{'brand': 'Apple', 'model': 'iPhone 14', 'price': 799, 'RAM': 6, 'storage': 128},
|
|
87
|
+
{'brand': 'Samsung', 'model': 'Galaxy S23 Ultra', 'price': 1199, 'RAM': 8, 'storage': 256},
|
|
88
|
+
{'brand': 'Samsung', 'model': 'Galaxy A54', 'price': 449, 'RAM': 8, 'storage': 128},
|
|
89
|
+
{'brand': 'Samsung', 'model': 'Galaxy Z Flip 5', 'price': 999, 'RAM': 8, 'storage': 256},
|
|
90
|
+
{'brand': 'Google', 'model': 'Pixel 8', 'price': 699, 'RAM': 8, 'storage': 128},
|
|
91
|
+
{'brand': 'Google', 'model': 'Pixel 7a', 'price': 499, 'RAM': 8, 'storage': 128},
|
|
92
|
+
{'brand': 'OnePlus', 'model': 'OnePlus 11', 'price': 699, 'RAM': 16, 'storage': 256},
|
|
93
|
+
# intentionally incomplete:
|
|
94
|
+
{'brand': 'Xiaomi', 'model': 'Xiaomi 13 Pro', 'price': 999, 'RAM': 12, },
|
|
95
|
+
{'brand': 'Xiaomi', 'model': 'Redmi Note 12 Pro', 'price': 299, 'RAM': 8, 'storage': 256},
|
|
96
|
+
]
|
|
97
|
+
|
|
98
|
+
src = 'storage >= 256 and RAM >= 8 and price <= 1000'
|
|
99
|
+
|
|
100
|
+
model = base_eval_model.clone()
|
|
101
|
+
try:
|
|
102
|
+
expr = model.expr(src)
|
|
103
|
+
except EvalException as e:
|
|
104
|
+
print(f"Error in expression ({src}): {e}")
|
|
105
|
+
exit(1)
|
|
106
|
+
|
|
107
|
+
for mobile in mobiles:
|
|
108
|
+
try:
|
|
109
|
+
if expr.eval(mobile):
|
|
110
|
+
print(f"{mobile['brand']} {mobile['model']} {mobile['storage']}/{mobile['RAM']} ({mobile['price']})")
|
|
111
|
+
|
|
112
|
+
except EvalException as e:
|
|
113
|
+
print(f"ERROR: evaluating expression for {mobile['brand']} {mobile['model']}: {e}")
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
## Security
|
|
117
|
+
|
|
118
|
+
Python's built-in functions, such as `compile()` and `eval()`, are powerful enough to execute arbitrary user-supplied code. Using them with untrusted input can therefore be dangerous. For example, an expression might attempt to execute `os.system("rm -rf /")`.
|
|
119
|
+
|
|
120
|
+
Evalidate uses a whitelist-based approach: an expression is accepted only if it consists of operations explicitly permitted by the configured security model. What counts as safe depends on the model's configuration and the author's security assumptions. Your requirements may differ, so you can define your own list of allowed operations.
|
|
121
|
+
|
|
122
|
+
Although the built-in models are designed to be secure, Evalidate allows you to extend them. Do so carefully: even an attribute that appears harmless, such as `.format`, can introduce vulnerabilities.
|
|
123
|
+
|
|
124
|
+
## Exceptions
|
|
125
|
+
|
|
126
|
+
Evalidate defines three exception classes: `CompilationException`, `ValidationException`, and `ExecutionException`. All three inherit from the base exception class `EvalException`.
|
|
127
|
+
|
|
128
|
+
## Models: Configuring Validation
|
|
129
|
+
|
|
130
|
+
Evalidate is highly configurable. Whether an expression passes validation or raises an exception depends on the security model used.
|
|
131
|
+
|
|
132
|
+
`EvalModel` defines the security policy for expression evaluation. It specifies the allowed AST nodes, function calls, and attributes, as well as a dictionary of imported functions. Evalidate provides a built-in model, `base_eval_model`, which permits a basic set of operations considered safe by the author.
|
|
133
|
+
|
|
134
|
+
You can create an empty model and extend it as needed:
|
|
135
|
+
|
|
136
|
+
```python
|
|
137
|
+
my_model = evalidate.EvalModel()
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
By default, an empty model allows no operations. Even `1+2` is rejected.
|
|
141
|
+
|
|
142
|
+
Alternatively, you can clone `base_eval_model` and extend it:
|
|
143
|
+
|
|
144
|
+
```python
|
|
145
|
+
from evalidate import base_eval_model
|
|
146
|
+
|
|
147
|
+
my_model = base_eval_model.clone()
|
|
148
|
+
my_model.nodes.append('Mult')
|
|
149
|
+
|
|
150
|
+
expr = my_model.expr('2*2')
|
|
151
|
+
expr.eval()
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
To enable calls to `int()`, allow the `Call` node and add `int` to the list of permitted functions:
|
|
155
|
+
|
|
156
|
+
```python
|
|
157
|
+
my_model.nodes.append('Call')
|
|
158
|
+
my_model.allowed_functions.append('int')
|
|
159
|
+
|
|
160
|
+
my_model.expr('int(36.6)').eval()
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
To enable attribute access and method calls, configure the model as follows:
|
|
164
|
+
|
|
165
|
+
```python
|
|
166
|
+
m = base_eval_model.clone()
|
|
167
|
+
m.nodes.extend(['Call', 'Attribute'])
|
|
168
|
+
m.attributes.append('startswith')
|
|
169
|
+
|
|
170
|
+
src = '"abcdef".startswith("abc")'
|
|
171
|
+
|
|
172
|
+
r = m.expr(src).eval()
|
|
173
|
+
print(r)
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
Even with this configuration, an expression such as `__builtins__["eval"](1)` is rejected.
|
|
177
|
+
|
|
178
|
+
### Exporting Functions to Expressions
|
|
179
|
+
|
|
180
|
+
You can explicitly expose your own functions to evaluated expressions:
|
|
181
|
+
|
|
182
|
+
```python
|
|
183
|
+
from evalidate import base_eval_model
|
|
184
|
+
|
|
185
|
+
def one():
|
|
186
|
+
return 1
|
|
187
|
+
|
|
188
|
+
m = base_eval_model.clone()
|
|
189
|
+
m.nodes.append('Call')
|
|
190
|
+
m.imported_functions["one"] = one
|
|
191
|
+
m.expr('one()').eval()
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
### Testing a Model's Security
|
|
195
|
+
|
|
196
|
+
Evalidate includes a built-in collection of potential attacks and a `test_security()` method that tests a model against them.
|
|
197
|
+
|
|
198
|
+
Passing `test_security()` is necessary, but not sufficient, to establish that a model is secure. Even if `test_security()` returns `True`, the model may still be vulnerable to attacks that the test suite does not cover.
|
|
199
|
+
|
|
200
|
+
```python
|
|
201
|
+
from evalidate import base_eval_model
|
|
202
|
+
|
|
203
|
+
m = base_eval_model.clone()
|
|
204
|
+
m.nodes.append('Pow')
|
|
205
|
+
|
|
206
|
+
assert(m.test_security())
|
|
207
|
+
|
|
208
|
+
src = '10**10**10'
|
|
209
|
+
m.expr(src).eval()
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
## CLI Tools
|
|
213
|
+
|
|
214
|
+
### `genfakeproducts`
|
|
215
|
+
|
|
216
|
+
`genfakeproducts` generates a JSON array containing a specified number of fake product records for testing.
|
|
217
|
+
|
|
218
|
+
For example, to generate one million records (this takes about one minute on my computer):
|
|
219
|
+
|
|
220
|
+
```shell
|
|
221
|
+
genfakeproducts -n 1000000 -o products.json
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
By default, it generates 10,000 records and writes them to `products.json`. This takes approximately one second.
|
|
225
|
+
|
|
226
|
+
Example record:
|
|
227
|
+
|
|
228
|
+
```json
|
|
229
|
+
{
|
|
230
|
+
"id": "d5af9f93-c164-4872-a535-4fb56cf62e6e",
|
|
231
|
+
"sku": "SKU-D5AF9F93",
|
|
232
|
+
"title": "Google ET-3843 Pro",
|
|
233
|
+
"brand": "Google",
|
|
234
|
+
"category": "phones",
|
|
235
|
+
"description": "Foot trial city avoid he wish real college investment tend include draw window either over somebody history detail risk support glass. Google phones designed for modern users.",
|
|
236
|
+
"price": 340.94,
|
|
237
|
+
"currency": "USD",
|
|
238
|
+
"stock": 24,
|
|
239
|
+
"rating": 4.23,
|
|
240
|
+
"reviews_count": 222,
|
|
241
|
+
"attributes": {
|
|
242
|
+
"storage": "512GB",
|
|
243
|
+
"color": "red"
|
|
244
|
+
},
|
|
245
|
+
"tags": [
|
|
246
|
+
"limited",
|
|
247
|
+
"new"
|
|
248
|
+
]
|
|
249
|
+
}
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
To use `genfakeproducts`, install Evalidate with the `generate` extra:
|
|
253
|
+
|
|
254
|
+
```shell
|
|
255
|
+
pip3 install evalidate[generate]
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
### `jg` (JSON grep)
|
|
259
|
+
|
|
260
|
+
`jg` is a high-performance CLI tool for filtering JSON data using Python expressions. It accepts an expression and a filename, or reads input from standard input.
|
|
261
|
+
|
|
262
|
+
```
|
|
263
|
+
jg 'category=="phones"' products.json
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
`jg` implements only a small subset of `jq`'s functionality, but it is twice as fast and much simpler to use.
|
|
267
|
+
|
|
268
|
+
**Options**
|
|
269
|
+
|
|
270
|
+
- `-b` — Benchmark mode. Suppresses JSON output and measures filtering time.
|
|
271
|
+
- `-v` — Verbose mode.
|
|
272
|
+
- `-l` — Read input as JSON Lines (JSONL), with one JSON object per line, instead of a JSON array.
|
|
273
|
+
- `-k` — Specify the key path to the list, such as `"shop::products::onstock"`, when the input is a nested dictionary.
|
|
274
|
+
- `-f` — Specify a custom output format for non-JSON output, e.g. `'{sku} {price} ({stock}) {title!r}'`.
|
|
275
|
+
|
|
276
|
+
## Advanced Topics
|
|
277
|
+
|
|
278
|
+
### Improve Performance by Using Native `eval()` on Validated Code
|
|
279
|
+
|
|
280
|
+
Evalidate is fast, but its evaluation machinery still introduces some overhead. If you need maximum performance, you can use Python's native [`eval()`](https://docs.python.org/3/library/functions.html#eval) directly on the validated code:
|
|
281
|
+
|
|
282
|
+
```python
|
|
283
|
+
from evalidate import base_eval_model
|
|
284
|
+
|
|
285
|
+
m = base_eval_model.clone()
|
|
286
|
+
d = dict(a=1, b=2)
|
|
287
|
+
expr = m.expr('a+b')
|
|
288
|
+
r = eval(expr.code, None, d) # <-- native python eval, will run at eval() speed
|
|
289
|
+
print(r)
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
This approach provides the same expression-validation guarantees as `Model.Expr.eval()`, because `expr.code` has already passed validation.
|
|
293
|
+
|
|
294
|
+
The differences are minor but important: executing `expr.code` directly can raise arbitrary exceptions, whereas `expr.eval()` raises only `ExecutionException`. Also, if you want to expose functions to the expression, you must do so manually.
|
|
295
|
+
|
|
296
|
+
### Custom Mapping Classes
|
|
297
|
+
|
|
298
|
+
You can use custom mapping classes instead of `dict`. As with native Python `eval()`, a custom mapping class can be used only as the local evaluation context.
|
|
299
|
+
|
|
300
|
+
Example:
|
|
301
|
+
|
|
302
|
+
```python
|
|
303
|
+
from collections import UserDict
|
|
304
|
+
from evalidate import base_eval_model
|
|
305
|
+
|
|
306
|
+
class LazyDict(UserDict):
|
|
307
|
+
def __missing__(self, key):
|
|
308
|
+
return 42
|
|
309
|
+
|
|
310
|
+
ctx = LazyDict(a=100)
|
|
311
|
+
|
|
312
|
+
model = base_eval_model.clone()
|
|
313
|
+
|
|
314
|
+
expr = model.expr("a+b")
|
|
315
|
+
|
|
316
|
+
res = expr.eval(ctx_locals=ctx)
|
|
317
|
+
print(res) # 142
|
|
318
|
+
```
|
|
319
|
+
|
|
320
|
+
### Limitations
|
|
321
|
+
|
|
322
|
+
Evalidate uses [`ast.parse()`](https://docs.python.org/3/library/ast.html#ast.parse) to generate an [abstract syntax tree (AST)](https://docs.python.org/3/library/ast.html#node-classes), which it then validates.
|
|
323
|
+
|
|
324
|
+
> **Warning**
|
|
325
|
+
>
|
|
326
|
+
> A sufficiently large or deeply nested input string can crash the Python interpreter because of stack-depth limitations in Python's AST compiler.
|
|
327
|
+
|
|
328
|
+
In my tests, expressions containing 200 nested `int()` calls work:
|
|
329
|
+
|
|
330
|
+
```python
|
|
331
|
+
int(int(.... int(1)...))
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
but expressions with 201 nested calls do not. The source code is over 1,000 characters long. Even if Evalidate receives such an expression, it will raise `CompilationException`.
|
|
335
|
+
|
|
336
|
+
## Similar Projects and Benchmarks
|
|
337
|
+
|
|
338
|
+
### [asteval](https://newville.github.io/asteval/)
|
|
339
|
+
|
|
340
|
+
`asteval` supports more complex code, including function definitions and access to Python's mathematical libraries. However, it has two drawbacks:
|
|
341
|
+
|
|
342
|
+
- It is significantly slower than Evalidate, which can evaluate validated code at native `eval()` speed.
|
|
343
|
+
- User-supplied expressions can run for a long time and consume excessive resources.
|
|
344
|
+
|
|
345
|
+
### [simpleeval](https://github.com/danthedeckie/simpleeval)
|
|
346
|
+
|
|
347
|
+
`simpleeval` is a similar project that also uses an AST-based approach and is optimized for repeatedly evaluating pre-parsed expressions.
|
|
348
|
+
|
|
349
|
+
However, it evaluates expressions using a higher-level representation based on [`ast.Expr`](https://docs.python.org/3/library/ast.html#ast.Expr), which makes evaluation several times slower. Evalidate instead compiles validated expressions into Python's native `code` objects, allowing execution at native `eval()` speed.
|
|
350
|
+
|
|
351
|
+
Evalidate is particularly well suited to evaluating the same expression against many different data objects.
|
|
352
|
+
|
|
353
|
+
## Benchmarking
|
|
354
|
+
|
|
355
|
+
The benchmark script is `benchmark/benchmark.py` in this repository.
|
|
356
|
+
|
|
357
|
+
The benchmark creates a list of one million products. In practice, it takes a sample of 100 products and repeats it 10,000 times. It then filters the list using an untrusted user-supplied expression to find matching products. The expression in this test is `price < 20`.
|
|
358
|
+
|
|
359
|
+
```
|
|
360
|
+
Products: 1000000 items
|
|
361
|
+
evalidate_raw_eval(): 0.266s
|
|
362
|
+
evalidate_eval(): 0.326s
|
|
363
|
+
test_simpleeval(): 1.824s
|
|
364
|
+
test_asteval(): 26.106s
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
As the results show, Evalidate is several times faster than `simpleeval`, and both are significantly faster than `asteval`.
|
|
368
|
+
|
|
369
|
+
The benchmark may not be optimally implemented; I am not an expert in `simpleeval` or `asteval`. If you can suggest a faster filtering implementation, I will consider including it. Any proposed benchmark code must treat the expression as unknown in advance and potentially untrusted.
|
|
370
|
+
|
|
371
|
+
## Further Reading: The Risks of `eval()`
|
|
372
|
+
|
|
373
|
+
- [Eval really is dangerous](https://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html)
|
|
374
|
+
- [Breaking Python 3 eval protections](https://netsec.expert/posts/breaking-python3-eval-protections/)
|
|
375
|
+
- [Python eval() function](https://realpython.com/python-eval-function/)
|
|
376
|
+
|
|
377
|
+
**Note:** The Real Python article demonstrates a concise source-validation technique based on `code.co_names`. However, this technique is vulnerable: it accepts the "bomb" from Ned Batchelder's article, which has an empty `co_names` tuple and can crash the interpreter.
|
|
378
|
+
|
|
379
|
+
Evalidate can reject this code and similar bombs, unless you deliberately configure the model to allow the specific operations required by an attack. Evalidate makes it harder to shoot yourself in the foot, but it is still possible if you try hard enough.
|
|
380
|
+
|
|
381
|
+
## More Information
|
|
382
|
+
|
|
383
|
+
For more information, see the module's source code. It is short, straightforward, and easy to modify.
|
|
384
|
+
|
|
385
|
+
## Contact
|
|
386
|
+
|
|
387
|
+
Email: yaroslaff at gmail.com
|