nextpipe 0.1.0.dev5__tar.gz → 0.1.0.dev6__tar.gz

nextpipe-0.1.0.dev5/.github/workflows/python-lint.yml → nextpipe-0.1.0.dev6/.github/workflows/lint.yml

@@ -1,11 +1,11 @@
-name: python lint
+name: lint
 on: [push]
 jobs:
-  python-lint:
+  lint:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
     steps:
       - name: git clone
         uses: actions/checkout@v4

nextpipe-0.1.0.dev5/.github/workflows/python-test.yml → nextpipe-0.1.0.dev6/.github/workflows/test.yml

@@ -1,11 +1,11 @@
-name: python test
+name: test
 on: [push]
 jobs:
-  python-test:
+  test:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
     steps:
       - name: git clone
         uses: actions/checkout@v4
@@ -21,4 +21,6 @@ jobs:
           pip install .[dev]
 
       - name: unit tests
+        env:
+          NEXTMV_API_KEY_NEXTPIPE: ${{ secrets.NEXTMV_API_KEY_NEXTPIPE }}
         run: python -m unittest

{nextpipe-0.1.0.dev5 → nextpipe-0.1.0.dev6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nextpipe
-Version: 0.1.0.dev5
+Version: 0.1.0.dev6
 Summary: Framework for Decision Pipeline modeling and execution
 Project-URL: Homepage, https://www.nextmv.io
 Project-URL: Documentation, https://www.nextmv.io/docs
@@ -98,17 +98,18 @@ License-File: LICENSE.md
 Keywords: decision automation,decision engineering,decision pipelines,decision science,decision workflows,decisions,nextmv,operations research,optimization,pipelines,workflows
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
-Requires-Python: >=3.8
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.9
 Requires-Dist: dataclasses-json>=0.6.7
 Requires-Dist: nextmv>=0.13.1
 Requires-Dist: pathos>=0.3.2
 Requires-Dist: requests>=2.31.0
 Provides-Extra: dev
+Requires-Dist: goldie>=0.1.6; extra == 'dev'
 Requires-Dist: ruff>=0.6.4; extra == 'dev'
 Description-Content-Type: text/markdown
 
@@ -116,6 +117,89 @@ Description-Content-Type: text/markdown
 
 Framework for Decision Pipeline modeling and execution.
 
+## Installation
+
+```bash
+pip install nextpipe
+```
+
+## Preview
+
+Example of a pipeline utilizing multiple routing solvers, and picking the best result.
+
+```mermaid
+graph LR
+  fetch_data(prepare_data)
+  fetch_data --> run_nextroute
+  fetch_data --> run_ortools
+  fetch_data --> run_pyvroom
+  run_nextroute{ }
+  run_nextroute_join{ }
+  run_nextroute_0(run_nextroute_0)
+  run_nextroute --> run_nextroute_0
+  run_nextroute_0 --> run_nextroute_join
+  run_nextroute_1(run_nextroute_1)
+  run_nextroute --> run_nextroute_1
+  run_nextroute_1 --> run_nextroute_join
+  run_nextroute_2(run_nextroute_2)
+  run_nextroute --> run_nextroute_2
+  run_nextroute_2 --> run_nextroute_join
+  run_nextroute_join --> pick_best
+  run_ortools(run_ortools)
+  run_ortools --> pick_best
+  run_pyvroom(run_pyvroom)
+  run_pyvroom --> pick_best
+  pick_best(pick_best)
+```
+
+## Modeling
+
+Pipeline modeling is done by defining a flow as steps and decorating them.
+The following example demonstrates a 3 step pipeline that makes use of dynamic fanout and joining of results.
+
+```python
+from nextpipe import AppOption, AppRunConfig, FlowSpec, app, foreach, join, needs, step
+
+
+class Flow(FlowSpec):
+    @foreach()  # Run the successor step for each item in the result list of this step
+    @step
+    def prepare(data: dict):
+        """
+        Creates 3 copies of the input and configures them for 3 different app parameters.
+        """
+        inputs = [copy.deepcopy(data) for _ in range(3)]
+        run_configs = [AppRunConfig(input, [AppOption("param", i)]) for i, input in enumerate(inputs)]
+        return run_configs
+
+    @app(app_id="echo")
+    @needs(predecessors=[prepare])
+    @step
+    def solve():
+        """
+        Imitates a solver app.
+        """
+        pass
+
+    @needs(predecessors=[solve])
+    @join()  # Collect the results from the previous 'foreach' step and combine them into a list passed as the arg
+    @step
+    def merge(results: list[dict]):
+        """Merges the results."""
+        return results
+```
+
+The pipeline can simply be executed from the main func of the app.
+
+```python
+# Run workflow
+flow = Flow("DecisionFlow", input.data)
+flow.run()
+
+# Write out the result
+print(json.dumps(flow.get_result(flow.merge)))
+```
+
 ## Examples
 
-You can find examples of how to use `nextpipe` in the [examples](./examples) directory.
+You can find further examples of how to use `nextpipe` in the [examples](./examples) directory.

nextpipe-0.1.0.dev6/README.md

@@ -0,0 +1,90 @@
+# nextpipe
+
+Framework for Decision Pipeline modeling and execution.
+
+## Installation
+
+```bash
+pip install nextpipe
+```
+
+## Preview
+
+Example of a pipeline utilizing multiple routing solvers, and picking the best result.
+
+```mermaid
+graph LR
+  fetch_data(prepare_data)
+  fetch_data --> run_nextroute
+  fetch_data --> run_ortools
+  fetch_data --> run_pyvroom
+  run_nextroute{ }
+  run_nextroute_join{ }
+  run_nextroute_0(run_nextroute_0)
+  run_nextroute --> run_nextroute_0
+  run_nextroute_0 --> run_nextroute_join
+  run_nextroute_1(run_nextroute_1)
+  run_nextroute --> run_nextroute_1
+  run_nextroute_1 --> run_nextroute_join
+  run_nextroute_2(run_nextroute_2)
+  run_nextroute --> run_nextroute_2
+  run_nextroute_2 --> run_nextroute_join
+  run_nextroute_join --> pick_best
+  run_ortools(run_ortools)
+  run_ortools --> pick_best
+  run_pyvroom(run_pyvroom)
+  run_pyvroom --> pick_best
+  pick_best(pick_best)
+```
+
+## Modeling
+
+Pipeline modeling is done by defining a flow as steps and decorating them.
+The following example demonstrates a 3 step pipeline that makes use of dynamic fanout and joining of results.
+
+```python
+from nextpipe import AppOption, AppRunConfig, FlowSpec, app, foreach, join, needs, step
+
+
+class Flow(FlowSpec):
+    @foreach()  # Run the successor step for each item in the result list of this step
+    @step
+    def prepare(data: dict):
+        """
+        Creates 3 copies of the input and configures them for 3 different app parameters.
+        """
+        inputs = [copy.deepcopy(data) for _ in range(3)]
+        run_configs = [AppRunConfig(input, [AppOption("param", i)]) for i, input in enumerate(inputs)]
+        return run_configs
+
+    @app(app_id="echo")
+    @needs(predecessors=[prepare])
+    @step
+    def solve():
+        """
+        Imitates a solver app.
+        """
+        pass
+
+    @needs(predecessors=[solve])
+    @join()  # Collect the results from the previous 'foreach' step and combine them into a list passed as the arg
+    @step
+    def merge(results: list[dict]):
+        """Merges the results."""
+        return results
+```
+
+The pipeline can simply be executed from the main func of the app.
+
+```python
+# Run workflow
+flow = Flow("DecisionFlow", input.data)
+flow.run()
+
+# Write out the result
+print(json.dumps(flow.get_result(flow.merge)))
+```
+
+## Examples
+
+You can find further examples of how to use `nextpipe` in the [examples](./examples) directory.

nextpipe-0.1.0.dev6/examples/README.md

@@ -0,0 +1,18 @@
+# Examples
+
+This directory contains examples of how to use `nextpipe`.
+
+## Prerequisites
+
+- Create and push all necessary apps in your account (check the README of the example and look for `@app` in the pipeline definition).
+  - Some simple apps are available in the `./apps` directory. Push them as described in their READMEs.
+  - Some examples make use of marketplace apps. Follow instructions in the example READMEs to subscribe to them.
+    - Alternative, you can use the `nextmv community clone -a <app-id>` command to clone the app from the community apps repository.
+
+## Table of contents
+
+- [pipeline-chain](./pipeline-chain): Example of a simple pipeline with a chain of steps.
+- [pipeline-foreach](./pipeline-foreach): Example of a pipeline with a fanout step (`@foreach`) that runs the same input through a solve step with different app options.
+- [pipeline-complex](./pipeline-complex): A more complex pipeline combining some concepts and multiple solvers.
+- [pipeline-ensemble](./pipeline-ensemble): Example of a pipeline ensembling the results of multiple solvers.
+- [pipeline-preprocess](./pipeline-preprocess): Example of a pipeline doing some preprocessing before running a solver.

{nextpipe-0.1.0.dev5 → nextpipe-0.1.0.dev6}/examples/apps/echo/main.py

@@ -1,3 +1,4 @@
+import sys
 import time
 
 import nextmv
@@ -5,7 +6,12 @@ import nextmv
 before = time.time()
 input = nextmv.load_local()
 output = nextmv.Output(
-    solution={"echo": input.data},
+    solution={
+        "echo": {
+            "data": input.data,
+            "args": sys.argv[1:],
+        },
+    },
     statistics={"run": {"duration": time.time() - before}},
 )
 nextmv.write_local(output)

{nextpipe-0.1.0.dev5 → nextpipe-0.1.0.dev6}/examples/pipeline-chain/README.md

@@ -5,7 +5,7 @@ A simple chain pipeline.
 ## Graph
 
 ```mermaid
-graph TD
+graph LR
   prepare(prepare)
   prepare --> solve
   solve(solve)
@@ -21,5 +21,5 @@ graph TD
 
 ```bash
 nextmv app push -a <app-id>
-echo '{"hello": "world!"}' | nextmv app run -a <app-id> -e "8c16gb12h"
+echo '{"hello": "world!"}' | nextmv app run -a <app-id>
 ```

{nextpipe-0.1.0.dev5 → nextpipe-0.1.0.dev6}/examples/pipeline-complex/README.md

@@ -5,7 +5,7 @@ A more complex pipeline combining some concepts.
 ## Graph
 
 ```mermaid
-graph TD
+graph LR
   fetch_data(fetch_data)
   fetch_data --> run_nextroute
   fetch_data --> run_ortools
@@ -40,5 +40,5 @@ graph TD
 
 ```bash
 nextmv app push -a <app-id>
-echo '{}' | nextmv app run -a <app-id> -e "8c16gb12h"
+echo '{}' | nextmv app run -a <app-id>
 ```

{nextpipe-0.1.0.dev5 → nextpipe-0.1.0.dev6}/examples/pipeline-ensemble/README.md

@@ -5,7 +5,7 @@ A basic ensemble pipeline.
 ## Graph
 
 ```mermaid
-graph TD
+graph LR
   run_nextroute{ }
   run_nextroute_join{ }
   run_nextroute_0(run_nextroute_0)
@@ -30,5 +30,5 @@ graph TD
 
 ```bash
 nextmv app push -a <app-id>
-cat /path/to/routing/input.json | nextmv app run -a <app-id> -e "8c16gb12h" -o 'instance=v171-5s'
+cat /path/to/routing/input.json | nextmv app run -a <app-id> -o 'instance=v171-5s'
 ```

nextpipe-0.1.0.dev6/examples/pipeline-foreach/README.md

@@ -0,0 +1,26 @@
+# Foreach fanout example
+
+Example of a pipeline with a fanout step (`@foreach`) that runs the same input through a solve step with different app options.
+
+## Graph
+
+```mermaid
+graph LR
+  prepare{ }
+  prepare(prepare)
+  prepare -- foreach --> solve
+  solve(solve)
+  solve -- join --> enhance
+  enhance(enhance)
+```
+
+## Pre-requisites
+
+- Push the echo app as described in the [echo app README](../apps/echo/README.md)`
+
+## Usage
+
+```bash
+nextmv app push -a <app-id>
+echo '{}' | nextmv app run -a <app-id>
+```

nextpipe-0.1.0.dev6/examples/pipeline-foreach/main.py

@@ -0,0 +1,56 @@
+import copy
+import json
+import os
+
+import nextmv
+import nextmv.cloud
+
+from nextpipe import AppOption, AppRunConfig, FlowSpec, app, foreach, join, needs, step
+
+
+class Flow(FlowSpec):
+    @foreach()  # Run the successor step for each item in the result list of this step
+    @step
+    def prepare(data: dict):
+        """
+        Creates 3 copies of the input and configures them for 3 different app parameters.
+        """
+        inputs = [copy.deepcopy(data) for _ in range(3)]
+        run_configs = [AppRunConfig(input, [AppOption("param", i)]) for i, input in enumerate(inputs)]
+        return run_configs
+
+    @app(app_id="echo")
+    @needs(predecessors=[prepare])
+    @step
+    def solve():
+        """
+        Runs the model.
+        """
+        pass
+
+    @needs(predecessors=[solve])
+    @join()  # Collect the results from the previous 'foreach' step and combine them into a list passed as the arg
+    @step
+    def merge(results: list[dict]):
+        """Merges the results."""
+        return results
+
+
+def main():
+    # Read API key from file (until secrets management support)
+    with open("key.json") as f:
+        os.environ["NEXTMV_API_KEY"] = json.load(f)["nextmv_api_key"]
+
+    # Load input data
+    input = nextmv.load_local()
+
+    # Run workflow
+    flow = Flow("DecisionFlow", input.data)
+    flow.run()
+
+    # Write out the result
+    print(json.dumps(flow.get_result(flow.merge)))
+
+
+if __name__ == "__main__":
+    main()

{nextpipe-0.1.0.dev5 → nextpipe-0.1.0.dev6}/examples/pipeline-preprocess/README.md

@@ -5,7 +5,7 @@ An example of a pipeline fetching CSV data, converting it to JSON, ensembling ac
 ## Graph
 
 ```mermaid
-graph TD
+graph LR
   fetch(fetch)
   fetch --> convert
   convert(convert)
@@ -42,5 +42,5 @@ graph TD
 
 ```bash
 nextmv app push -a <app-id>
-echo '{}' | nextmv app run -a <app-id> -e "8c16gb12h"
+echo '{}' | nextmv app run -a <app-id>
 ```

nextpipe-0.1.0.dev6/examples/pipeline-preprocess/app.yaml

@@ -0,0 +1,7 @@
+type: python
+runtime: ghcr.io/nextmv-io/runtime/python:3.11
+files:
+    - main.py
+    - key.json
+python:
+    pip-requirements: requirements.txt

nextpipe-0.1.0.dev6/examples/pipeline-preprocess/requirements.txt

@@ -0,0 +1 @@
+nextpipe==0.1.0.dev3

nextpipe-0.1.0.dev6/nextpipe/__about__.py

@@ -0,0 +1 @@
+__version__ = "v0.1.0.dev6"

{nextpipe-0.1.0.dev5 → nextpipe-0.1.0.dev6}/nextpipe/__init__.py

@@ -2,12 +2,16 @@
 
 from .__about__ import __version__
 from .decorators import app as app
+from .decorators import foreach as foreach
+from .decorators import join as join
 from .decorators import needs as needs
 from .decorators import optional as optional
 from .decorators import repeat as repeat
 from .decorators import step as step
 from .flow import FlowGraph as FlowGraph
 from .flow import FlowSpec as FlowSpec
+from .schema import AppOption as AppOption
+from .schema import AppRunConfig as AppRunConfig
 
 VERSION = __version__
 """The version of the nextpipe package."""

nextpipe-0.1.0.dev6/nextpipe/config.py

@@ -0,0 +1,22 @@
+from dataclasses import dataclass
+
+from dataclasses_json import dataclass_json
+
+
+@dataclass_json
+@dataclass
+class Configuration:
+    """Configuration for the pipeline."""
+
+    thread_count: int = 0
+    """
+    Number of threads to use for parallel processing.
+    If 0, the number of threads is set to the number of available CPUs.
+    """
+    max_step_inputs: int = 50
+    """
+    Maximum number of inputs to a step.
+    This is used to avoid accidental combinatorial explosions due to the Cartesian product
+    of inputs used when a step has multiple predecessors which are themselves repeated or
+    foreach steps.
+    """