pipescript 0.0.15__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. pipescript-0.0.15/MANIFEST.in +4 -0
  2. pipescript-0.0.15/PKG-INFO +657 -0
  3. pipescript-0.0.15/README.md +612 -0
  4. pipescript-0.0.15/docs/HISTORY.rst +37 -0
  5. pipescript-0.0.15/docs/LICENSE.txt +11 -0
  6. pipescript-0.0.15/pipescript/__init__.py +103 -0
  7. pipescript-0.0.15/pipescript/__main__.py +26 -0
  8. pipescript-0.0.15/pipescript/_version.py +658 -0
  9. pipescript-0.0.15/pipescript/analysis/__init__.py +0 -0
  10. pipescript-0.0.15/pipescript/analysis/dynamic_macros.py +162 -0
  11. pipescript-0.0.15/pipescript/analysis/extract_names.py +64 -0
  12. pipescript-0.0.15/pipescript/analysis/placeholders.py +231 -0
  13. pipescript-0.0.15/pipescript/api/__init__.py +7 -0
  14. pipescript-0.0.15/pipescript/api/static_macros.py +182 -0
  15. pipescript-0.0.15/pipescript/api/utils.py +95 -0
  16. pipescript-0.0.15/pipescript/constants.py +3 -0
  17. pipescript-0.0.15/pipescript/extension.py +166 -0
  18. pipescript-0.0.15/pipescript/patches/__init__.py +0 -0
  19. pipescript-0.0.15/pipescript/patches/completion_patch.py +31 -0
  20. pipescript-0.0.15/pipescript/patches/traceback_patch.py +39 -0
  21. pipescript-0.0.15/pipescript/tracers/__init__.py +7 -0
  22. pipescript-0.0.15/pipescript/tracers/macro_tracer.py +553 -0
  23. pipescript-0.0.15/pipescript/tracers/optional_chaining_tracer.py +330 -0
  24. pipescript-0.0.15/pipescript/tracers/pipeline_tracer.py +1062 -0
  25. pipescript-0.0.15/pipescript/utils.py +41 -0
  26. pipescript-0.0.15/pipescript/version.py +23 -0
  27. pipescript-0.0.15/pipescript.egg-info/PKG-INFO +657 -0
  28. pipescript-0.0.15/pipescript.egg-info/SOURCES.txt +36 -0
  29. pipescript-0.0.15/pipescript.egg-info/dependency_links.txt +1 -0
  30. pipescript-0.0.15/pipescript.egg-info/entry_points.txt +2 -0
  31. pipescript-0.0.15/pipescript.egg-info/not-zip-safe +1 -0
  32. pipescript-0.0.15/pipescript.egg-info/requires.txt +24 -0
  33. pipescript-0.0.15/pipescript.egg-info/top_level.txt +1 -0
  34. pipescript-0.0.15/pyproject.toml +43 -0
  35. pipescript-0.0.15/setup.cfg +76 -0
  36. pipescript-0.0.15/setup.py +7 -0
  37. pipescript-0.0.15/versioneer.py +2205 -0
@@ -0,0 +1,4 @@
1
+ include README.md docs/HISTORY.rst
2
+ recursive-exclude test *
3
+ include versioneer.py
4
+ include pipescript/_version.py
@@ -0,0 +1,657 @@
1
+ Metadata-Version: 2.4
2
+ Name: pipescript
3
+ Version: 0.0.15
4
+ Summary: Powerful pipeline syntax for IPython and Jupyter
5
+ Home-page: https://github.com/smacke/pipescript
6
+ Author: Stephen Macke
7
+ Author-email: stephen.macke@gmail.com
8
+ License: BSD-3-Clause
9
+ Classifier: Development Status :: 3 - Alpha
10
+ Classifier: Intended Audience :: Developers
11
+ Classifier: License :: OSI Approved :: BSD License
12
+ Classifier: Natural Language :: English
13
+ Classifier: Programming Language :: Python :: 3.9
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Classifier: Programming Language :: Python :: 3.14
19
+ Requires-Python: >=3.9
20
+ Description-Content-Type: text/markdown; charset=UTF-8
21
+ License-File: docs/LICENSE.txt
22
+ Requires-Dist: pyccolo>=0.0.85
23
+ Requires-Dist: typing-extensions
24
+ Provides-Extra: test
25
+ Requires-Dist: black; extra == "test"
26
+ Requires-Dist: hypothesis; extra == "test"
27
+ Requires-Dist: isort; extra == "test"
28
+ Requires-Dist: mypy; extra == "test"
29
+ Requires-Dist: pytest; extra == "test"
30
+ Requires-Dist: pytest-cov; extra == "test"
31
+ Requires-Dist: ruff; extra == "test"
32
+ Provides-Extra: dev
33
+ Requires-Dist: build; extra == "dev"
34
+ Requires-Dist: pycln; extra == "dev"
35
+ Requires-Dist: twine; extra == "dev"
36
+ Requires-Dist: versioneer; extra == "dev"
37
+ Requires-Dist: black; extra == "dev"
38
+ Requires-Dist: hypothesis; extra == "dev"
39
+ Requires-Dist: isort; extra == "dev"
40
+ Requires-Dist: mypy; extra == "dev"
41
+ Requires-Dist: pytest; extra == "dev"
42
+ Requires-Dist: pytest-cov; extra == "dev"
43
+ Requires-Dist: ruff; extra == "dev"
44
+ Dynamic: license-file
45
+
46
+ pipescript
47
+ ==========
48
+
49
+ [![CI Status](https://github.com/smacke/pipescript/workflows/pipescript/badge.svg)](https://github.com/smacke/pipescript/actions)
50
+ [![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)
51
+ [![License: BSD3](https://img.shields.io/badge/License-BSD3-maroon.svg)](https://opensource.org/licenses/BSD-3-Clause)
52
+ [![Python Versions](https://img.shields.io/pypi/pyversions/pipescript.svg)](https://pypi.org/project/pipescript)
53
+ [![PyPI Version](https://img.shields.io/pypi/v/pipescript.svg)](https://pypi.org/project/pipescript)
54
+
55
+ Pipescript is an IPython extension that brings a pipe operator `|>` and
56
+ powerful placeholder and macro expansion syntax extensions to IPython and Jupyter.
57
+
58
+ For a quick example, consider the following code snippet, which is not super easy
59
+ to read (which function call does the keyword parameter `initial=1.0` go with?):
60
+
61
+ ```python
62
+ result = max(
63
+ np.max(np.abs(array[np.isfinite(array)]), initial=1.0)
64
+ for array in arrays
65
+ )
66
+ ```
67
+
68
+ This mess of nested function calls can be written in pipescript as follows:
69
+
70
+ ```python
71
+ result = arrays |> map[$
72
+ |> $array[np.isfinite($array)]
73
+ |> np.abs
74
+ |> np.max($, initial=1.0)
75
+ ] |> max
76
+ ```
77
+
78
+ If you're familiar with the [magrittr](https://magrittr.tidyverse.org/) package
79
+ for R, then you'll be right at home with pipescript.
80
+
81
+
82
+ ## Getting Started
83
+
84
+ Run the following in IPython or Jupyter to install pipescript and load
85
+ the extension:
86
+
87
+ ```python
88
+ %pip install pipescript
89
+ %load_ext pipescript
90
+ ```
91
+
92
+ The `%load_ext pipescript` invocation is what enables the new pipe syntax
93
+ in your current session.
94
+
95
+ ## Features by Example
96
+
97
+ Let's look at a few examples to give a flavor of what you can do with pipescript:
98
+
99
+ ```python
100
+ # Render a sorted version of a tuple
101
+ >>> tup = (3, 4, 1, 5, 6)
102
+ >>> tup |> sorted |> tuple
103
+ (1, 3, 4, 5, 6)
104
+ ```
105
+ The above example showcases the `|>`, or "pipe", operator, which is a much-loved
106
+ feature of functional programming that has become increasingly mainstream. Its
107
+ primary benefit is that the flow of execution follows natural left-to-right
108
+ reading / writing order of the code. Whether or not such pipeline syntax is
109
+ available, it's not uncommon for programmers to execute pipelines like the above
110
+ multiple times during to verify the computation at each step, particularly in
111
+ interactive programming environments like Jupyter. With `|>`, this type of
112
+ incremental verification becomes a breeze: first execute `tup |> sorted`, then
113
+ append ` |> tuple` to execute the full chain `tup |> sorted |> tuple`, each time
114
+ using the last-expression rendering capabilities of the notebook or REPL to
115
+ inspect and verify the result.
116
+
117
+ ### Placeholders
118
+
119
+ The power of the `|>` operator is amplified via placeholder syntax for implicit
120
+ function construction: for pipescript, we use `$` to stand in for function arguments
121
+ and induce function creation:
122
+
123
+ ```python
124
+ # Sort a list in reverse order
125
+ >>> lst = [3, 4, 1, 5, 6]
126
+ >>> lst |> sorted($, reverse=True)
127
+ [6, 5, 4, 3, 1]
128
+ ```
129
+
130
+ `$` is analogous to magrittr's `.` placeholder. It can also be used outside
131
+ of pipeline contexts:
132
+
133
+ ```python
134
+ # Sort a list in reverse order and print the result
135
+ lst = [3, 4, 1, 5, 6]
136
+ reverse_sorter = sorted($, reverse=True)
137
+
138
+ # The following are equivalent:
139
+ print(reverse_sorter(lst))
140
+ lst |> reverse_sorter |> print
141
+ ```
142
+
143
+ Each time `$` appears, it represents a new argument, so `sorted($, reverse=$)`
144
+ represents a function with two arguments:
145
+
146
+ ```python
147
+ import random
148
+
149
+ # Sort a list in either ascending or descending order with probablility 0.5:
150
+ lst = [3, 4, 1, 5, 6]
151
+ sorter = sorted($, reverse=$)
152
+ reverse = random.random() < 0.5
153
+
154
+ # The following are equivalent:
155
+ print(sorter(lst, reverse))
156
+ lst |> sorter($, reverse) |> print
157
+ ```
158
+
159
+ Placeholders can appear anywhere -- not just as arguments to function calls:
160
+
161
+ ```python
162
+ # Sort a list and find the position of element 4:
163
+ >>> lst = [3, 4, 1, 5, 6]
164
+ >>> lst |> sorted |> $.index(3)
165
+ 1
166
+ ```
167
+
168
+ ### Named Placeholders
169
+
170
+ There are situations that would benefit from referencing the same placeholder multiple times, for which
171
+ pipescript permits *named placeholders* by prefixing `$` to an identifier:
172
+
173
+ ```python
174
+ # Pair even entries from a range with their adjacent odd entry
175
+ range(6) |> list |> zip($v[::2], $v[1::2]) |> list
176
+ >>> [(0, 1), (2, 3), (4, 5)]
177
+ ```
178
+
179
+ In the above example, we could have used any name for `$v`, the important
180
+ thing is that the same name was used -- otherwise pipescript would have
181
+ induced a function with two arguments instead of one.
182
+
183
+ ### Undetermined Pipelines
184
+
185
+ Similar to magrittr's behavior, if any number of placeholders appear in the first
186
+ step of an pipescript pipeline, this *undetermined pipeline* will represent a function:
187
+
188
+ ```python
189
+ >>> second_largest_value = $ |> sorted($, reverse=True) |> $[1]
190
+ >>> [3, 8, 6, 5, 1] |> second_largest_value
191
+ 6
192
+ ```
193
+
194
+ ### Macros and Partial Function Syntax
195
+
196
+ In some cases, it may be desirable to curry a function with parameters at its start,
197
+ akin to the typical usage of `functools.partial`. For example:
198
+
199
+ ```python
200
+ >>> add_reducer = reduce(lambda x, y: x + y, $, $)
201
+ >>> add_reducer([1, 2, 3], 0)
202
+ 6
203
+ >>> add_reducer([[1, 2, 3], [4, 5, 6]], [])
204
+ [1, 2, 3, 4, 5, 6]
205
+ ```
206
+
207
+ To avoid writing out a `$` placeholder for each and every tail argument, you can
208
+ prefix the call itself with a `$` and omit subsequent arguments, just like in
209
+ [coconut](https://coconut-lang.org/):
210
+
211
+ ```python
212
+ >>> add_reducer = reduce$(lambda x, y: x + y)
213
+ >>> add_reducer([1, 2, 3], 0)
214
+ 6
215
+ >>> add_reducer([[1, 2, 3], [4, 5, 6]], [])
216
+ [1, 2, 3, 4, 5, 6]
217
+ ```
218
+
219
+ Or even more simply, since the induced partial function retains all the same
220
+ argument defaults as the original `reduce`, we can omit the base case:
221
+
222
+ ```python
223
+ >>> add_reducer = reduce$(lambda x, y: x + y)
224
+ >>> add_reducer([1, 2, 3])
225
+ 6
226
+ >>> add_reducer([[1, 2, 3], [4, 5, 6]])
227
+ [1, 2, 3, 4, 5, 6]
228
+ ```
229
+
230
+ For common functional programming tools like `map`, `reduce`, and `filter`, the above
231
+ pattern is so common that pipescript provides corresponding macros, in which the function used
232
+ to curry each higher order function is specified between brackets:
233
+
234
+ ```python
235
+ >>> add_reducer = reduce[lambda x, y: x + y]
236
+ >>> [1, 2, 3] |> add_reducer
237
+ 6
238
+ >>> [[1, 2, 3], [4, 5, 6]] |> add_reducer
239
+ [1, 2, 3, 4, 5, 6]
240
+ ```
241
+
242
+ We're still writing out `lambda x, y: x + y`, which is kind of tedious -- for these
243
+ kinds of simple lambda constructions, pipescript provides a *quick lambda macro*, `f`:
244
+
245
+ ```python
246
+ >>> add_reducer = reduce[f[$ + $]]
247
+ >>> [1, 2, 3] |> add_reducer
248
+ 6
249
+ >>> [[1, 2, 3], [4, 5, 6]] |> add_reducer
250
+ [1, 2, 3, 4, 5, 6]
251
+ ```
252
+
253
+ `f` can also be used on its own:
254
+
255
+ ```python
256
+ >>> f[$ + $](2, 3)
257
+ 5
258
+
259
+ >>> f[$a*$b + $b*$c + $a*$c](2, 3, 4)
260
+ 26
261
+ ```
262
+
263
+ Furthermore, pipescript allows you to omit the `f` from higher order
264
+ functional macros, so that you can simply do `add_reducer = reduce[$ + $]` instead.
265
+ Here are a couple of nifty constructions utilizing this compact syntax:
266
+
267
+ ```python
268
+ # factorial
269
+ >>> range(1, 5) |> reduce[$ * $]
270
+ 24
271
+
272
+ # compute a number from decimal digits
273
+ >>> [2, 3, 4] |> reduce[10*$ + $]
274
+ 234
275
+ ```
276
+
277
+ ### Additional Pipe Operators
278
+
279
+ There are a few other variants of the `|>` operator offered by
280
+ pipescript, covered in this section.
281
+
282
+ #### Assignment Pipe
283
+
284
+ The *assignment pipe*, `|>>`, writes the left hand side value to the variable
285
+ whose name is specified on the right hand side. Furthermore, it evaluates to
286
+ the left hand side value. For example:
287
+
288
+ ```python
289
+ >>> 2 |> $ + 2 |>> two_plus_two |> $ + 3 |>> two_plus_two_plus_three
290
+ 7
291
+ >>> (two_plus_two, two_plus_two_plus_three)
292
+ (4, 7)
293
+ ```
294
+
295
+ #### Varargs Pipe
296
+
297
+ The *varargs pipe*, `*|>`, unpacks the iterable on the left hand side before
298
+ passing its values as inputs to the function on the right hand side. For
299
+ example:
300
+
301
+ ```python
302
+ # Add two numbers:
303
+ >>> (2, 3) *|> $ + $
304
+ 5
305
+ ```
306
+
307
+ A common pattern is using `*|>` to expand an undetermined pipeline
308
+ appearing inside of a `map[...]`:
309
+
310
+ ```python
311
+ # Take the product of consecutive pairs of even-odd integers
312
+ >>> consecutive_pairs = range(10) |> list |> ($v[::2], $v[1::2]) *|> zip
313
+ >>> consecutive_pairs |> map[$ *|> $ * $] |> list
314
+ [0, 6, 20, 42, 72]
315
+ ```
316
+
317
+ #### Function Pipe
318
+
319
+ The other commonly used pipe is the *function pipe*, `.>`, which is used to compose
320
+ the functions specified on the left hand side and right hand side together, with the
321
+ function on the left hand side being applied first in the composition (note that this
322
+ behavior is reversed from normal function composition, but follows the flow of data better).
323
+ For example:
324
+
325
+ ```python
326
+ >>> reverse = reversed .> list
327
+ >>> [1, 2, 3] |> reverse
328
+ [3, 2, 1]
329
+ ```
330
+
331
+ #### Other Pipes
332
+
333
+ Besides `|>>`, `*|>`, and `.>`, pipescript offers a few less commonly used operators as well. The below
334
+ table describes the complete set of forward pipe operators available:
335
+
336
+ | Operator | Pipescript Syntax | Python Syntax |
337
+ |--------------------|-----------------------------------------------------|-----------------------------------------|
338
+ | <code>\|></code> | <code>y = x \|> f</code> | `y = f(x)` |
339
+ | <code>\|>></code> | <code>x \|>> y</code> | `y = x; y` |
340
+ | <code>*\|></code> | <code>y = x *\|> f</code> where `x` is an iterable | `y = f(*x)` |
341
+ | <code>**\|></code> | <code>y = x **\|> f</code> where `x` is a dict | `y = f(**x)` |
342
+ | `.>` | `h = g .> f` | `h = lambda *a, **kw: g(f(*a, **kw))` |
343
+ | `*.>` | `h = g *.> f` | `h = lambda *a, **kw: g(*f(*a, **kw))` |
344
+ | `**.>` | `h = g **.> f` | `h = lambda *a, **kw: g(**f(*a, **kw))` |
345
+ | `?>` | `y = x ?> f` | `y = None if x is None else f(x)` |
346
+ | `*?>` | `y = x *?> f` where `x` is an iterable, or `None` | `y = None if x is None else f(*x)` |
347
+ | `**?>` | `y = x **?> f` where `x` is a dict, or `None` | `y = None if x is None else f(**x)` |
348
+ | `$>` | `g = x $> f` | `g = functools.partial(f, x)` |
349
+ | `*$>` | `g = x *$> f` where `x` is an iterable | `g = functools.partial(f, *x)` |
350
+ | `**$>` | `g = x **$> f` where `x` is a dict | `g = functools.partial(f, **x)` |
351
+
352
+ Except for `|>>`, each and every operator has a corresponding *backward* variant; e.g. `<|` is the backward variant
353
+ of `|>` and is a low-precedence apply. For example:
354
+
355
+ ```python
356
+ >>> reversed .> list <| [1, 2, 3]
357
+ [3, 2, 1]
358
+ ```
359
+
360
+ All pipe operators are applied in order from left to right (including backward pipes).
361
+ Furthermore, all pipe operators are left associative and operate at the same precedence
362
+ as `|` (bitwise or), meaning that any pipeline steps that include an `|` binary operation
363
+ must be wrapped in parentheses.
364
+
365
+ ### Additional Macros and Helper Utilities
366
+
367
+ #### `do` macro
368
+
369
+ Similar to [toolz](https://github.com/pytoolz/toolz), pipescript offers a `do` macro
370
+ implementing something similar to the following higher order function:
371
+
372
+ ```python
373
+ def do(func, obj):
374
+ func(obj)
375
+ return obj
376
+ ```
377
+
378
+ In the case of pipescript, the input function `func` is specified inside of brackets,
379
+ just as with other functional macros:
380
+
381
+ ```python
382
+ >>> 2 |> $ + 2 |> do[print] |> $ + 2 |>> result
383
+ 4
384
+ 6
385
+ ```
386
+
387
+ While any function expression, including undetermined pipelines, can appear inside `do[...]` brackets,
388
+ `do[print]` is so common that pipescript provides a `peek` utility that implements the very same:
389
+
390
+ ```python
391
+ >>> 2 |> $ + 2 |> peek |> $ + 2 |>> result
392
+ 4
393
+ 6
394
+ ```
395
+
396
+ To suppress the automatic expression rendering of a pipeline result, pipescript also offers a `null` utility function
397
+ (as in `/dev/null`), which essentially swallows its input:
398
+
399
+ ```python
400
+ >>> 2 |> $ + 2 |> peek |> $ + 2 |>> result |> null
401
+ 4
402
+ ```
403
+
404
+ #### `fork` and `parallel` macros
405
+
406
+ If you wish to move beyond linear chains and apply the same input to multiple pipelines,
407
+ pipescript provides `fork` and `parallel` macros, which return the results of each function
408
+ as a tuple:
409
+
410
+ ```python
411
+ >>> range(10) |> list |> fork[
412
+ map[2 * $] .> filter[$ % 3 == 0],
413
+ map[3 * $] .> filter[$ % 2 == 0],
414
+ ]
415
+ ([0, 6, 12, 18], [0, 6, 12, 18, 24])
416
+ ```
417
+
418
+ `parallel` does the same thing as `fork` but executes each function passed to it concurrently.
419
+
420
+ #### `when` `unless`, `otherwise`, `repeat`, `until` macros
421
+
422
+ The `when` macro takes as input a value and conditional expression that, upon passing,
423
+ forwards the value, and upon failing, terminates computation with `None`. It is particularly powerful
424
+ when combined with `fork` and `collapse` (the latter of which extracts the non-null value out of
425
+ the tuple that results from the `fork`):
426
+
427
+ ```python
428
+ >>> collatz = when[$ != 1] .> fork[
429
+ when[$ % 2 == 0] .> $ // 2,
430
+ when[$ % 2 == 1] .> $ * 3 + 1,
431
+ ] .> collapse .> peek
432
+ ```
433
+
434
+ You can also use `unless`, which is just the opposite of `when`:
435
+
436
+ ```python
437
+ >>> collatz = when[$ != 1] .> fork[
438
+ when[$ % 2 == 0] .> $ // 2,
439
+ unless[$ % 2 == 0] .> $ * 3 + 1,
440
+ ] .> collapse .> peek
441
+ ```
442
+
443
+ If you don't want to explicitly write out the negative conditional, `fork` lets you
444
+ use the `otherwise` macro as the last expression:
445
+
446
+ ```python
447
+ >>> collatz = when[$ != 1] .> fork[
448
+ when[$ % 2 == 0] .> $ // 2,
449
+ otherwise[$ * 3 + 1],
450
+ ] .> collapse .> peek
451
+ ```
452
+
453
+ Of course, this can be written more naturally and succinctly with
454
+ a ternary conditional expression:
455
+
456
+ ```python
457
+ >>> collatz = when[$ != 1] .> f[$v // 2 if $v % 2 == 0 else $v * 3 + 1] .> peek
458
+ ```
459
+
460
+ Regardless of how we write the conditional, pipescript allows you to
461
+ exponentiate single-argument functions with power the composition (`.**`)
462
+ operator, so that we don't need to write out
463
+ `42 |> collatz |> collatz |> ... |> collatz`:
464
+
465
+ ```python
466
+ >>> 42 |> collatz .** 20
467
+ 21
468
+ 64
469
+ 32
470
+ 16
471
+ 8
472
+ 4
473
+ 2
474
+ 1
475
+ ```
476
+
477
+ If you don't want to guess the upper bound of how many steps to run it, you can
478
+ use the `repeat` and `until` macros (`until` is just an alias of `unless`):
479
+
480
+ ```python
481
+ >>> collatz = f[$v // 2 if $v % 2 == 0 else $v * 3 + 1]
482
+ >>> 42 |> repeat[until[$ == 1] .> collatz .> peek] |> null
483
+ 21
484
+ 64
485
+ 32
486
+ 16
487
+ 8
488
+ 4
489
+ 2
490
+ 1
491
+ ```
492
+
493
+ #### `future` macro
494
+
495
+ Finally, to schedule a function to run in another thread and immediately
496
+ return a future to the eventual result, pipescript provides a `future` macro:
497
+
498
+ ```python
499
+ >>> 2 |> future[$ + 2] |> $.result()
500
+ 4
501
+ >>> [1, 2, 3] |> future[sum] |> $.result()
502
+ 6
503
+ ```
504
+
505
+ ## Placeholder Scope
506
+
507
+ A natural question is: how does pipescript know what part of the code should
508
+ be included in the body of the function induced by placeholder use? The
509
+ rules are as follows:
510
+
511
+ 1. If there is a macro or pipeline step enclosing the placeholder, the induced
512
+ function body includes the "smallest" such enclosing macro or pipeline step.
513
+ 2. Otherwise, the function body expands to include the nearest "chain"
514
+ of function calls, attribute accesses, and / or subscript accesses.
515
+
516
+ An example of a "chain" would be something like `np.array($).T.astype(int)`,
517
+ which induces a lambda that converts its argument to a numpy array,
518
+ transposes it, and then converts the result to use `int64` dtype. That is,
519
+ the lambda body expands to include not just `np.array($)`, but the entire
520
+ "chain" in the expression.
521
+
522
+ To see a concrete example of where this matters, consider the following
523
+ two placeholder expressions:
524
+
525
+ ```python
526
+ # The following sorters do different things!
527
+ sorter1 = sorted($, key=$[1])
528
+ sorter2 = sorted($, key=f[$[1]])
529
+ ```
530
+
531
+ `sorter1` is a function that takes two arguments: a sequence, and a list of
532
+ functions, the second of which will be used to compute the sort key, which it then
533
+ uses to sort the first argument.
534
+ `sorter2`, on the other hand, is a function that takes a single argument, which
535
+ is a sequence that it sorts using the second element of each value in said
536
+ sequence value as sort key. In most cases, `sorter2` probably gives the desired
537
+ behavior.
538
+
539
+ ## Optional Chaining, Permissive Attribute Chaining, and Nullish Coalescing
540
+
541
+ Pipescript also provides typescript-style optional chaining and nullish coalescing.
542
+ That is, `a?.b.c.d().e` resolves to `None` when `a` is `None`, as does `a?.()`.
543
+ Also, `a ?? obj` evaluates to `obj` only when `a` is `None`, but evaluates to `a`
544
+ whenever `a` is some other falsey value like `""`, `0`, `False`, or `[]`. Note that,
545
+ like normal boolean `or`, the nullish coalescing operator `??` is lazy and will not
546
+ evaluate expressions on its right hand side when its left hand side is not `None`.
547
+
548
+ Unlike Javascript, Python does not resolve unavailable attribute accesses to
549
+ `undefined`, but will rather throw `AttributeError`. In pipescript, if you would
550
+ like to perform some kind of permissive attribute access like in Javascript, you
551
+ can use the *permissive chaining operator* `.?` (where the `?` appears after the
552
+ `.`) and access `b` as `a.?b`, which is equivalent to `getattr(a, "b", None)`.
553
+ Note however that if the aforementioned expression resolves to `None`, something
554
+ like `a.?b.c` will still throw an `AttributeError` -- to avoid that, you need to
555
+ combine both permissive attribute chaining and optional chaining as `a.?b?.c`.
556
+
557
+ ## Performance Overhead
558
+
559
+ Because pipescript is implemented using instrumentation (see [How it works](#how-it-works)),
560
+ it does incur overhead. For top-level code written in a Jupyter cell (e.g.,
561
+ code that doesn't have any indentation), the additional overhead generally doesn't matter,
562
+ as it tends to be insignificant when compared to data-intensive dataframe operations
563
+ and SQL queries common in data science workloads. Furthermore, overhead is only incurred
564
+ when pipescript syntax is actually used -- there's no penalty for any code written in vanilla
565
+ Python, **even when pipescript has been enabled in your current REPL session**.
566
+
567
+ ## More Examples
568
+ I developed pipescript while working on
569
+ [Advent of Code 2025](https://adventofcode.com/2025) in parallel,
570
+ and used it for most of the input processesing portions of my solutions.
571
+ You can find these solutions at https://github.com/smacke/aoc2025. In particular,
572
+ the [solution for day 6](https://github.com/smacke/aoc2025/blob/main/aoc6.ipynb)
573
+ showcases the upper limits of what is possible with pipescript. Note however that it is
574
+ optimized for pipescript usage and not readability, which I generally wouldn't recommend.
575
+
576
+ ## What pipescript is and is not
577
+
578
+ For now, pipescript is not a general purpose functional programming language on top of
579
+ Python. It is very much not intended for production use cases, and instead
580
+ caters toward quick-and-dirty one-off / scratchpad type computations in IPython
581
+ and Jupyter specifically. In short, pipescript aims to provide simple but powerful
582
+ pipeline and placeholder syntax to interactive Python programming environments.
583
+
584
+ Particularly, pipescript is:
585
+ - Currently only for interactive Python environments built on top of IPython, such as
586
+ Jupyter, or IPython itself
587
+ - Just a library you can install from PyPI, compatible with a wide range of Python 3
588
+ versions -- no fancy installation instructions, no complicated language distribution
589
+ to install
590
+ - Fully compatible with all existing Python standard and third-party libraries that
591
+ you already know and love, since it's just Python function calls under the hood
592
+
593
+ All the different pipeline operators like `|>`, `<|`, `*|>`, etc. essentially
594
+ transpile down to an instrumented variant of the bitwise-or (`|`) operator, and
595
+ therefore every new operator left-associates at the same level of precedence,
596
+ meaning that pipeline steps run from left to right in the order that they
597
+ appear. Pipescript aims to optimize for simplicity, readability / writability, and
598
+ predictability over feature completeness (though I'd like to think it strikes a
599
+ fairly good balance in this regard). Pipescript may be expanded beyond IPython / Jupyter
600
+ depending on traction.
601
+
602
+ ## How it works
603
+
604
+ Pipescript works by transforming syntax in two stages. First, it rewrites token spans
605
+ like `|>` and `*|>` that are illegal in Python to legal ones -- for the previous
606
+ examples, both spans are rewritten to bitwise or, `|`. After these transformations,
607
+ the resulting code is valid (but likely not runnable) Python syntax. Pipescript uses
608
+ the [pyccolo](https://github.com/smacke/pyccolo) library to perform these rewrites,
609
+ which remembers the positions of the rewrites where they occurred, so that the eventual
610
+ `ast.BinOp` AST node can be associated with the `|>` operator.
611
+
612
+ Pyccolo is a library I developed during my PhD which provides an event-driven
613
+ architecture for declarative AST transformations. Its key selling point is that
614
+ it allows you to layer multiple AST transformations on top of each other in a
615
+ composable fashion. In short, you specify handlers for different AST nodes such
616
+ as `ast.BinOp`, and pyccolo instruments these nodes by emitting events for them,
617
+ so that when the code runs, all the handlers for a particular event are run.
618
+ Such event handlers are what allow us to change the behavior of `ast.BinOp`
619
+ nodes that have been associated with various custom operators like `|>`.
620
+
621
+ Because the same event emission transformation can be leveraged by multiple
622
+ associated handlers, you generally don't need to worry about said
623
+ transformations rewriting the AST in ways that conflict with each other. This
624
+ composability lies in stark contrast with the challenges you would face if you
625
+ were to just create a bunch of `ast.NodeTransformer` instances to perform
626
+ transformations. The strategy employed by pyccolo therefore allows for
627
+ incremental and iterative feature development without requiring large rewrites
628
+ as new features are introduced.
629
+
630
+ To summarize, pipescript rewrites its syntax to valid Python, and then runs this Python in
631
+ an instrumented fashion using pyccolo. Because everything is just running in
632
+ Python, pipescript is effectively a Python superset, and because the transformed
633
+ Python that is instrumented is fairly similar visually to pipescript syntax,
634
+ various Jupyter ergonomical features like readable stack traces and jedi-based
635
+ autocomplete can continue to function as normal (for the most part).
636
+
637
+ Implementation-wise, thanks to pyccolo's heavy lifting, I was able to write the
638
+ initial release of pipescript entirely over the course of time off during the
639
+ 2025 holiday season. At the time of this writing, pipescript occupies about 2000
640
+ lines of code (excluding tests), each of which was produced *without* the help
641
+ of any AI agents.
642
+
643
+ ## Inspiration
644
+
645
+ Pipescript draws inspiration largely from
646
+ [magrittr](https://magrittr.tidyverse.org/), but also from efforts like
647
+ [coconut](https://coconut-lang.org/) (a functional superset of Python),
648
+ as well as from libraries like [Pipe](https://github.com/JulienPalard/Pipe) and [toolz](https://github.com/pytoolz/toolz) which
649
+ fill some of Python's pipe and functional programming gaps with elegant APIs.
650
+
651
+ ## Disclaimer
652
+
653
+ **Warning: use pipescript at your own risk!** It is very much not guaranteed to
654
+ be bug-free -- I implemented it in a hurry before it was time to go back to work.
655
+
656
+ ## License
657
+ Code in this project licensed under the [BSD-3-Clause License](https://opensource.org/licenses/BSD-3-Clause).