runnable 0.12.1__py3-none-any.whl → 0.12.3__py3-none-any.whl

Sign up to get free protection for your applications and to get access to all the features.
@@ -1,453 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: runnable
3
- Version: 0.12.1
4
- Summary: A Compute agnostic pipelining software
5
- Home-page: https://github.com/vijayvammi/runnable
6
- License: Apache-2.0
7
- Author: Vijay Vammi
8
- Author-email: mesanthu@gmail.com
9
- Requires-Python: >=3.9,<3.13
10
- Classifier: License :: OSI Approved :: Apache Software License
11
- Classifier: Programming Language :: Python :: 3
12
- Classifier: Programming Language :: Python :: 3.9
13
- Classifier: Programming Language :: Python :: 3.10
14
- Classifier: Programming Language :: Python :: 3.11
15
- Classifier: Programming Language :: Python :: 3.12
16
- Provides-Extra: database
17
- Provides-Extra: docker
18
- Provides-Extra: notebook
19
- Requires-Dist: click
20
- Requires-Dist: click-plugins (>=1.1.1,<2.0.0)
21
- Requires-Dist: dill (>=0.3.8,<0.4.0)
22
- Requires-Dist: docker ; extra == "docker"
23
- Requires-Dist: mlflow-skinny
24
- Requires-Dist: ploomber-engine (>=0.0.31,<0.0.32) ; extra == "notebook"
25
- Requires-Dist: pydantic (>=2.5,<3.0)
26
- Requires-Dist: rich (>=13.5.2,<14.0.0)
27
- Requires-Dist: ruamel.yaml
28
- Requires-Dist: ruamel.yaml.clib
29
- Requires-Dist: sqlalchemy ; extra == "database"
30
- Requires-Dist: stevedore (>=3.5.0,<4.0.0)
31
- Requires-Dist: typing-extensions ; python_version < "3.8"
32
- Project-URL: Documentation, https://github.com/vijayvammi/runnable
33
- Project-URL: Repository, https://github.com/vijayvammi/runnable
34
- Description-Content-Type: text/markdown
35
-
36
-
37
-
38
-
39
-
40
- <p align="center">
41
-
42
- ,////,
43
- /// 6|
44
- // _|
45
- _/_,-'
46
- _.-/'/ \ ,/;,
47
- ,-' /' \_ \ / _/
48
- `\ / _/\ ` /
49
- | /, `\_/
50
- | \'
51
- /\_ /` /\
52
- /' /_``--.__/\ `,. / \
53
- |_/` `-._ `\/ `\ `.
54
- `-.__/' `\ |
55
- `\ \
56
- `\ \
57
- \_\__
58
- \___)
59
-
60
- </p>
61
- <hr style="border:2px dotted orange">
62
-
63
- <p align="center">
64
- <a href="https://pypi.org/project/runnable/"><img alt="python:" src="https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10-blue.svg"></a>
65
- <a href="https://pypi.org/project/runnable/"><img alt="Pypi" src="https://badge.fury.io/py/runnable.svg"></a>
66
- <a href="https://github.com/vijayvammi/runnable/blob/main/LICENSE"><img alt"License" src="https://img.shields.io/badge/license-Apache%202.0-blue.svg"></a>
67
- <a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
68
- <a href="https://github.com/python/mypy"><img alt="MyPy Checked" src="https://www.mypy-lang.org/static/mypy_badge.svg"></a>
69
- <a href="https://github.com/vijayvammi/runnable/actions/workflows/release.yaml"><img alt="Tests:" src="https://github.com/vijayvammi/runnable/actions/workflows/release.yaml/badge.svg">
70
- <a href="https://github.com/vijayvammi/runnable/actions/workflows/docs.yaml"><img alt="Docs:" src="https://github.com/vijayvammi/runnable/actions/workflows/docs.yaml/badge.svg">
71
- </p>
72
- <hr style="border:2px dotted orange">
73
-
74
- runnable is a simplified workflow definition language that helps in:
75
-
76
- - **Streamlined Design Process:** runnable enables users to efficiently plan their pipelines with
77
- [stubbed nodes](https://astrazeneca.github.io/runnable-core/concepts/stub), along with offering support for various structures such as
78
- [tasks](https://astrazeneca.github.io/runnable-core/concepts/task), [parallel branches](https://astrazeneca.github.io/runnable-core/concepts/parallel), and [loops or map branches](https://astrazeneca.github.io/runnable-core/concepts/map)
79
- in both [yaml](https://astrazeneca.github.io/runnable-core/concepts/pipeline) or a [python SDK](https://astrazeneca.github.io/runnable-core/sdk) for maximum flexibility.
80
-
81
- - **Incremental Development:** Build your pipeline piece by piece with runnable, which allows for the
82
- implementation of tasks as [python functions](https://astrazeneca.github.io/runnable-core/concepts/task/#python_functions),
83
- [notebooks](https://astrazeneca.github.io/runnable-core/concepts/task/#notebooks), or [shell scripts](https://astrazeneca.github.io/runnable-core/concepts/task/#shell),
84
- adapting to the developer's preferred tools and methods.
85
-
86
- - **Robust Testing:** Ensure your pipeline performs as expected with the ability to test using sampled data. runnable
87
- also provides the capability to [mock and patch tasks](https://astrazeneca.github.io/runnable-core/configurations/executors/mocked)
88
- for thorough evaluation before full-scale deployment.
89
-
90
- - **Seamless Deployment:** Transition from the development stage to production with ease.
91
- runnable simplifies the process by requiring [only configuration changes](https://astrazeneca.github.io/runnable-core/configurations/overview)
92
- to adapt to different environments, including support for [argo workflows](https://astrazeneca.github.io/runnable-core/configurations/executors/argo).
93
-
94
- - **Efficient Debugging:** Quickly identify and resolve issues in pipeline execution with runnable's local
95
- debugging features. Retrieve data from failed tasks and [retry failures](https://astrazeneca.github.io/runnable-core/concepts/run-log/#retrying_failures)
96
- using your chosen debugging tools to maintain a smooth development experience.
97
-
98
- Along with the developer friendly features, runnable also acts as an interface to production grade concepts
99
- such as [data catalog](https://astrazeneca.github.io/runnable-core/concepts/catalog), [reproducibility](https://astrazeneca.github.io/runnable-core/concepts/run-log),
100
- [experiment tracking](https://astrazeneca.github.io/runnable-core/concepts/experiment-tracking)
101
- and secure [access to secrets](https://astrazeneca.github.io/runnable-core/concepts/secrets).
102
-
103
- <hr style="border:2px dotted orange">
104
-
105
- ## What does it do?
106
-
107
-
108
- ![works](assets/work.png)
109
-
110
- <hr style="border:2px dotted orange">
111
-
112
- ## Documentation
113
-
114
- [More details about the project and how to use it available here](https://astrazeneca.github.io/runnable-core/).
115
-
116
- <hr style="border:2px dotted orange">
117
-
118
- ## Installation
119
-
120
- The minimum python version that runnable supports is 3.8
121
-
122
- ```shell
123
- pip install runnable
124
- ```
125
-
126
- Please look at the [installation guide](https://astrazeneca.github.io/runnable-core/usage)
127
- for more information.
128
-
129
- <hr style="border:2px dotted orange">
130
-
131
- ## Example
132
-
133
- Your application code. Use pydantic models as DTO.
134
-
135
- Assumed to be present at ```functions.py```
136
- ```python
137
- from pydantic import BaseModel
138
-
139
- class InnerModel(BaseModel):
140
- """
141
- A pydantic model representing a group of related parameters.
142
- """
143
-
144
- foo: int
145
- bar: str
146
-
147
-
148
- class Parameter(BaseModel):
149
- """
150
- A pydantic model representing the parameters of the whole pipeline.
151
- """
152
-
153
- x: int
154
- y: InnerModel
155
-
156
-
157
- def return_parameter() -> Parameter:
158
- """
159
- The annotation of the return type of the function is not mandatory
160
- but it is a good practice.
161
-
162
- Returns:
163
- Parameter: The parameters that should be used in downstream steps.
164
- """
165
- # Return type of a function should be a pydantic model
166
- return Parameter(x=1, y=InnerModel(foo=10, bar="hello world"))
167
-
168
-
169
- def display_parameter(x: int, y: InnerModel):
170
- """
171
- Annotating the arguments of the function is important for
172
- runnable to understand the type of parameters you want.
173
-
174
- Input args can be a pydantic model or the individual attributes.
175
- """
176
- print(x)
177
- # >>> prints 1
178
- print(y)
179
- # >>> prints InnerModel(foo=10, bar="hello world")
180
- ```
181
-
182
- ### Application code using driver functions.
183
-
184
- The code is runnable without any orchestration framework.
185
-
186
- ```python
187
- from functions import return_parameter, display_parameter
188
-
189
- my_param = return_parameter()
190
- display_parameter(my_param.x, my_param.y)
191
- ```
192
-
193
- ### Orchestration using runnable
194
-
195
- <table>
196
- <tr>
197
- <th>python SDK</th>
198
- <th>yaml</th>
199
- </tr>
200
- <tr>
201
- <td valign="top"><p>
202
-
203
- Example present at: ```examples/python-tasks.py```
204
-
205
- Run it as: ```python examples/python-tasks.py```
206
-
207
- ```python
208
- from runnable import Pipeline, Task
209
-
210
- def main():
211
- step1 = Task(
212
- name="step1",
213
- command="examples.functions.return_parameter",
214
- )
215
- step2 = Task(
216
- name="step2",
217
- command="examples.functions.display_parameter",
218
- terminate_with_success=True,
219
- )
220
-
221
- step1 >> step2
222
-
223
- pipeline = Pipeline(
224
- start_at=step1,
225
- steps=[step1, step2],
226
- add_terminal_nodes=True,
227
- )
228
-
229
- pipeline.execute()
230
-
231
-
232
- if __name__ == "__main__":
233
- main()
234
- ```
235
-
236
- </p></td>
237
-
238
- <td valign="top"><p>
239
-
240
- Example present at: ```examples/python-tasks.yaml```
241
-
242
-
243
- Execute via the cli: ```runnable execute -f examples/python-tasks.yaml```
244
-
245
- ```yaml
246
- dag:
247
- description: |
248
- This is a simple pipeline that does 3 steps in sequence.
249
- In this example:
250
- 1. First step: returns a "parameter" x as a Pydantic model
251
- 2. Second step: Consumes that parameter and prints it
252
-
253
- This pipeline demonstrates one way to pass small data from one step to another.
254
-
255
- start_at: step 1
256
- steps:
257
- step 1:
258
- type: task
259
- command_type: python # (2)
260
- command: examples.functions.return_parameter # (1)
261
- next: step 2
262
- step 2:
263
- type: task
264
- command_type: python
265
- command: examples.functions.display_parameter
266
- next: success
267
- success:
268
- type: success
269
- fail:
270
- type: fail
271
- ```
272
-
273
- </p></td>
274
-
275
- </tr>
276
- </table>
277
-
278
- ### Transpile to argo workflows
279
-
280
- No code change, just change the configuration.
281
-
282
- ```yaml
283
- executor:
284
- type: "argo"
285
- config:
286
- image: runnable:demo
287
- persistent_volumes:
288
- - name: runnable-volume
289
- mount_path: /mnt
290
-
291
- run_log_store:
292
- type: file-system
293
- config:
294
- log_folder: /mnt/run_log_store
295
- ```
296
-
297
- More details can be found in [argo configuration](https://astrazeneca.github.io/runnable-core/configurations/executors/argo).
298
-
299
- Execute the code as ```runnable execute -f examples/python-tasks.yaml -c examples/configs/argo-config.yam```
300
-
301
- <details>
302
- <summary>Expand</summary>
303
-
304
- ```yaml
305
- apiVersion: argoproj.io/v1alpha1
306
- kind: Workflow
307
- metadata:
308
- generateName: runnable-dag-
309
- annotations: {}
310
- labels: {}
311
- spec:
312
- activeDeadlineSeconds: 172800
313
- entrypoint: runnable-dag
314
- podGC:
315
- strategy: OnPodCompletion
316
- retryStrategy:
317
- limit: '0'
318
- retryPolicy: Always
319
- backoff:
320
- duration: '120'
321
- factor: 2
322
- maxDuration: '3600'
323
- serviceAccountName: default-editor
324
- templates:
325
- - name: runnable-dag
326
- failFast: true
327
- dag:
328
- tasks:
329
- - name: step-1-task-uvdp7h
330
- template: step-1-task-uvdp7h
331
- depends: ''
332
- - name: step-2-task-772vg3
333
- template: step-2-task-772vg3
334
- depends: step-1-task-uvdp7h.Succeeded
335
- - name: success-success-igzq2e
336
- template: success-success-igzq2e
337
- depends: step-2-task-772vg3.Succeeded
338
- - name: step-1-task-uvdp7h
339
- container:
340
- image: runnable:demo
341
- command:
342
- - runnable
343
- - execute_single_node
344
- - '{{workflow.parameters.run_id}}'
345
- - step%1
346
- - --log-level
347
- - WARNING
348
- - --file
349
- - examples/python-tasks.yaml
350
- - --config-file
351
- - examples/configs/argo-config.yaml
352
- volumeMounts:
353
- - name: executor-0
354
- mountPath: /mnt
355
- imagePullPolicy: ''
356
- resources:
357
- limits:
358
- memory: 1Gi
359
- cpu: 250m
360
- requests:
361
- memory: 1Gi
362
- cpu: 250m
363
- - name: step-2-task-772vg3
364
- container:
365
- image: runnable:demo
366
- command:
367
- - runnable
368
- - execute_single_node
369
- - '{{workflow.parameters.run_id}}'
370
- - step%2
371
- - --log-level
372
- - WARNING
373
- - --file
374
- - examples/python-tasks.yaml
375
- - --config-file
376
- - examples/configs/argo-config.yaml
377
- volumeMounts:
378
- - name: executor-0
379
- mountPath: /mnt
380
- imagePullPolicy: ''
381
- resources:
382
- limits:
383
- memory: 1Gi
384
- cpu: 250m
385
- requests:
386
- memory: 1Gi
387
- cpu: 250m
388
- - name: success-success-igzq2e
389
- container:
390
- image: runnable:demo
391
- command:
392
- - runnable
393
- - execute_single_node
394
- - '{{workflow.parameters.run_id}}'
395
- - success
396
- - --log-level
397
- - WARNING
398
- - --file
399
- - examples/python-tasks.yaml
400
- - --config-file
401
- - examples/configs/argo-config.yaml
402
- volumeMounts:
403
- - name: executor-0
404
- mountPath: /mnt
405
- imagePullPolicy: ''
406
- resources:
407
- limits:
408
- memory: 1Gi
409
- cpu: 250m
410
- requests:
411
- memory: 1Gi
412
- cpu: 250m
413
- templateDefaults:
414
- activeDeadlineSeconds: 7200
415
- timeout: 10800s
416
- arguments:
417
- parameters:
418
- - name: run_id
419
- value: '{{workflow.uid}}'
420
- volumes:
421
- - name: executor-0
422
- persistentVolumeClaim:
423
- claimName: runnable-volume
424
-
425
- ```
426
-
427
- </details>
428
-
429
- ## Pipelines can be:
430
-
431
- ### Linear
432
-
433
- A simple linear pipeline with tasks either
434
- [python functions](https://astrazeneca.github.io/runnable-core/concepts/task/#python_functions),
435
- [notebooks](https://astrazeneca.github.io/runnable-core/concepts/task/#notebooks), or [shell scripts](https://astrazeneca.github.io/runnable-core/concepts/task/#shell)
436
-
437
- [![](https://mermaid.ink/img/pako:eNpl0bFuwyAQBuBXQVdZTqTESpxMDJ0ytkszhgwnOCcoNo4OaFVZfvcSx20tGSQ4fn0wHB3o1hBIyLJOWGeDFJ3Iq7r90lfkkA9HHfmTUpnX1hFyLvrHzDLl_qB4-1BOOZGGD3TfSikvTDSNFqdj2sT2vBTr9euQlXNWjqycsN2c7UZWFMUE7udwP0L3y6JenNKiyfvz8t8_b-gavT9QJYY0PcDtjeTLptrAChriBq1JzeoeWkG4UkMKZCoN8k2Bcn1yGEN7_HYaZOBIK4h3g4EOFi-MDcgKa59SMja0_P7s_vAJ_Q_YOH6o?type=png)](https://mermaid.live/edit#pako:eNpl0bFuwyAQBuBXQVdZTqTESpxMDJ0ytkszhgwnOCcoNo4OaFVZfvcSx20tGSQ4fn0wHB3o1hBIyLJOWGeDFJ3Iq7r90lfkkA9HHfmTUpnX1hFyLvrHzDLl_qB4-1BOOZGGD3TfSikvTDSNFqdj2sT2vBTr9euQlXNWjqycsN2c7UZWFMUE7udwP0L3y6JenNKiyfvz8t8_b-gavT9QJYY0PcDtjeTLptrAChriBq1JzeoeWkG4UkMKZCoN8k2Bcn1yGEN7_HYaZOBIK4h3g4EOFi-MDcgKa59SMja0_P7s_vAJ_Q_YOH6o)
438
-
439
- ### [Parallel branches](https://astrazeneca.github.io/runnable-core/concepts/parallel)
440
-
441
- Execute branches in parallel
442
-
443
- [![](https://mermaid.ink/img/pako:eNp9k01rwzAMhv-K8S4ZtJCzDzuMLmWwwkh2KMQ7eImShiZ2sB1KKf3vs52PpsWNT7LySHqlyBeciRwwwUUtTtmBSY2-YsopR8MpQUfAdCdBBekWNBpvv6-EkFICzGAtWcUTDW3wYy20M7lr5QGBK2j-anBAkH4M1z6grnjpy17xAiTwDII07jj6HK8-VnVZBspITnpjztyoVkLLJOy3Qfrdm6gQEu2370Io7WLORo84PbRoA_oOl9BBg4UHbHR58UkMWq_fxjrOnhLRx1nH0SgkjlBjh7ekxNKGc0NelDLknhePI8qf7MVNr_31nm1wwNTeM2Ao6pmf-3y3Mp7WlqA7twOnXfKs17zt-6azmim1gQL1A0NKS3EE8hKZE4Yezm3chIVFiFe4AdmwKjdv7mIjKNYHaIBiYsycySPFlF8NxzotkjPPMNGygxXu2pxp2FSslKzBpGC1Ml7IKy3krn_E7i1f_wEayTcn?type=png)](https://mermaid.live/edit#pako:eNp9k01rwzAMhv-K8S4ZtJCzDzuMLmWwwkh2KMQ7eImShiZ2sB1KKf3vs52PpsWNT7LySHqlyBeciRwwwUUtTtmBSY2-YsopR8MpQUfAdCdBBekWNBpvv6-EkFICzGAtWcUTDW3wYy20M7lr5QGBK2j-anBAkH4M1z6grnjpy17xAiTwDII07jj6HK8-VnVZBspITnpjztyoVkLLJOy3Qfrdm6gQEu2370Io7WLORo84PbRoA_oOl9BBg4UHbHR58UkMWq_fxjrOnhLRx1nH0SgkjlBjh7ekxNKGc0NelDLknhePI8qf7MVNr_31nm1wwNTeM2Ao6pmf-3y3Mp7WlqA7twOnXfKs17zt-6azmim1gQL1A0NKS3EE8hKZE4Yezm3chIVFiFe4AdmwKjdv7mIjKNYHaIBiYsycySPFlF8NxzotkjPPMNGygxXu2pxp2FSslKzBpGC1Ml7IKy3krn_E7i1f_wEayTcn)
444
-
445
- ### [loops or map](https://astrazeneca.github.io/runnable-core/concepts/map)
446
-
447
- Execute a pipeline over an iterable parameter.
448
-
449
- [![](https://mermaid.ink/img/pako:eNqVlF1rwjAUhv9KyG4qKNR-3AS2m8nuBgN3Z0Sy5tQG20SSdE7E_76kVVEr2CY3Ied9Tx6Sk3PAmeKACc5LtcsKpi36nlGZFbXciHwfLN79CuWiBLMcEULWGkBSaeosA2OCxbxdXMd89Get2bZASsLiSyuvQE2mJZXIjW27t2rOmQZ3Gp9rD6UjatWnwy7q6zPPukd50WTydmemEiS_QbQ79RwxGoQY9UaMuojRA8TCXexzyHgQZNwbMu5Cxl3IXNX6OWMyiDHpzZh0GZMHjOK3xz2mgxjT3oxplzG9MPp5_nVOhwJjteDwOg3HyFj3L1dCcvh7DUc-iftX18n6Waet1xX8cG908vpKHO6OW7cvkeHm5GR2b3drdvaSGTODHLW37mxabYC8fLgRhlfxpjNdwmEets-Dx7gCXTHBXQc8-D2KbQEVUEzckjO9oZjKo9Ox2qr5XmaYWF3DGNdbzizMBHOVVWGSs9K4XeDCKv3ZttSmsx7_AYa341E?type=png)](https://mermaid.live/edit#pako:eNqVlF1rwjAUhv9KyG4qKNR-3AS2m8nuBgN3Z0Sy5tQG20SSdE7E_76kVVEr2CY3Ied9Tx6Sk3PAmeKACc5LtcsKpi36nlGZFbXciHwfLN79CuWiBLMcEULWGkBSaeosA2OCxbxdXMd89Get2bZASsLiSyuvQE2mJZXIjW27t2rOmQZ3Gp9rD6UjatWnwy7q6zPPukd50WTydmemEiS_QbQ79RwxGoQY9UaMuojRA8TCXexzyHgQZNwbMu5Cxl3IXNX6OWMyiDHpzZh0GZMHjOK3xz2mgxjT3oxplzG9MPp5_nVOhwJjteDwOg3HyFj3L1dCcvh7DUc-iftX18n6Waet1xX8cG908vpKHO6OW7cvkeHm5GR2b3drdvaSGTODHLW37mxabYC8fLgRhlfxpjNdwmEets-Dx7gCXTHBXQc8-D2KbQEVUEzckjO9oZjKo9Ox2qr5XmaYWF3DGNdbzizMBHOVVWGSs9K4XeDCKv3ZttSmsx7_AYa341E)
450
-
451
- ### [Arbitrary nesting](https://astrazeneca.github.io/runnable-core/concepts/nesting/)
452
- Any nesting of parallel within map and so on.
453
-