async-graph-data-flow 1.4.0__tar.gz → 1.5.1__tar.gz

async_graph_data_flow-1.5.1/CHANGELOG.md

@@ -0,0 +1,68 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
+and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
+
+## [Unreleased] - YYYY-MM-DD
+
+### Added
+
+### Changed
+
+### Deprecated
+
+### Removed
+
+### Fixed
+
+### Security
+
+## [1.5.1] - 2024-11-07
+
+### Added
+- Added static type checking using mypy. (#12)
+
+### Fixed
+- Fixed documentation to actually show example code. (#12)
+
+## [1.5.0] - 2024-10-30
+
+### Added
+- Python 3.13 is officially supported and tested on CI. (#11)
+
+### Changed
+- Migrated the docs to the Read The Docs site. (#11)
+
+## [1.4.0] - 2023-11-30
+
+### Added
+- Python 3.12 is officially supported and tested on CI.
+- Added a technical notes page to the Sphinx documentation.
+
+## [1.3.0] - 2023-06-20
+
+### Added
+- `AsyncGraph.add_node` has the new keyword argument `check_async_gen` 
+  to optionally disable the async generator function check for advanced usage.
+
+## [1.2.0] - 2023-04-25
+
+### Added
+- An `AsyncExecutor` instance now has the `start_nodes` property.
+  It maps start nodes to their arguments passed in at the beginning of graph execution. 
+
+### Fixed
+- Before a graph executes, the `exceptions` property of an `AsyncExecutor` instance 
+  now returns `None`. Previously, accessing this property before graph execute
+  would raise an exception.
+
+## [1.1.0] - 2023-03-02
+
+### Added
+- Keep track of and expose exceptions. (#6)
+
+## [1.0.0] - 2023-01-05
+
+First release!

{async-graph-data-flow-1.4.0 → async_graph_data_flow-1.5.1}/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2022-2023, Civis Analytics
+Copyright (c) 2022-2024, Civis Analytics
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

async_graph_data_flow-1.5.1/MANIFEST.in

@@ -0,0 +1 @@
+include CHANGELOG.md

{async-graph-data-flow-1.4.0/src/async_graph_data_flow.egg-info → async_graph_data_flow-1.5.1}/PKG-INFO

@@ -1,9 +1,10 @@
 Metadata-Version: 2.1
 Name: async-graph-data-flow
-Version: 1.4.0
+Version: 1.5.1
 Summary: Asynchronous functions that pass data along a directed acyclic graph
 Author-email: Civis Analytics <opensource@civisanalytics.com>
 License: BSD 3-Clause
+Project-URL: Documentation, https://async-graph-data-flow.readthedocs.io
 Project-URL: Source, https://github.com/civisanalytics/async-graph-data-flow
 Keywords: async,asynchronous,asyncio,data flow,graph
 Classifier: Development Status :: 5 - Production/Stable
@@ -17,21 +18,24 @@ Classifier: Programming Language :: Python :: 3 :: Only
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: Framework :: AsyncIO
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
 Provides-Extra: dev
-Requires-Dist: black==23.11.0; extra == "dev"
-Requires-Dist: flake8==6.1.0; extra == "dev"
-Requires-Dist: pytest==7.4.3; extra == "dev"
-Requires-Dist: build==1.0.3; extra == "dev"
-Requires-Dist: twine==4.0.2; extra == "dev"
-Requires-Dist: Sphinx==7.2.6; extra == "dev"
-Requires-Dist: furo==2023.9.10; extra == "dev"
-Requires-Dist: numpydoc==1.6.0; extra == "dev"
-Requires-Dist: sphinx-copybutton==0.5.2; extra == "dev"
-Requires-Dist: sphinxcontrib-mermaid==0.9.2; extra == "dev"
+Requires-Dist: black==24.10.0; extra == "dev"
+Requires-Dist: flake8==7.1.1; extra == "dev"
+Requires-Dist: pytest==8.3.3; extra == "dev"
+Requires-Dist: mypy==1.13.0; extra == "dev"
+Requires-Dist: build==1.2.2; extra == "dev"
+Requires-Dist: twine==5.1.1; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: Sphinx==8.1.3; extra == "docs"
+Requires-Dist: furo==2024.8.6; extra == "docs"
+Requires-Dist: numpydoc==1.8.0; extra == "docs"
+Requires-Dist: sphinx-copybutton==0.5.2; extra == "docs"
+Requires-Dist: sphinxcontrib-mermaid==1.0.0; extra == "docs"
 
 # async-graph-data-flow
 
@@ -41,7 +45,7 @@ Requires-Dist: sphinxcontrib-mermaid==0.9.2; extra == "dev"
 
 ## Full Documentation
 
-Please visit https://civisanalytics.github.io/async-graph-data-flow
+Please visit https://async-graph-data-flow.readthedocs.io
 
 Please also check out this
 [blog post](https://www.civisanalytics.com/blog/open-source-software-release-introducing-async-graph-data-flow-a-python-library-for-efficient-data-pipelines/)
@@ -71,17 +75,18 @@ flake8 src tests examples
 black --check src tests examples
 ```
 
-## For Maintainers
+## Building Documentation
 
-To update the Sphinx documentation,
-the source files that need editing are under `docs/source/`;
-everything else under `docs/` is auto-generated.
-After the manual updates under `docs/source/` are ready,
-the HTML pages are updated as follows:
+We use the Sphinx framework. The documentation source files are in `docs/source/`.
+These files can be updated as necessary.
 
-```bash
-rm -r docs/_sources docs/_static && rm docs/*.html
+The public documentation is accessible at https://async-graph-data-flow.readthedocs.io.
+The doc build is configured by `.readthedocs.yaml`. 
+Normally, even when we need to update the documentation or make a new release of async-graph-data-flow,
+neither this configuration YAML file nor Civis's account on the Read the Docs site need any updates.
+The builds by the Read The Docs site generate the necessary files (the HTML pages and other things)
+for the public documentation. All these auto-generated files are explicitly not versioned (see `.gitignore`).
 
-# Run the following command *twice* -- certain HTML updates only show up after multiple `sphinx-build` runs.
-sphinx-build docs/source docs
-```
+To build the documentation locally (for testing and development),
+install the full doc-related dependencies: `pip install -r docs/requirements.txt`,
+then run `sphinx-build -b html docs/ docs/build/`.

{async-graph-data-flow-1.4.0 → async_graph_data_flow-1.5.1}/README.md

@@ -6,7 +6,7 @@
 
 ## Full Documentation
 
-Please visit https://civisanalytics.github.io/async-graph-data-flow
+Please visit https://async-graph-data-flow.readthedocs.io
 
 Please also check out this
 [blog post](https://www.civisanalytics.com/blog/open-source-software-release-introducing-async-graph-data-flow-a-python-library-for-efficient-data-pipelines/)
@@ -36,17 +36,18 @@ flake8 src tests examples
 black --check src tests examples
 ```
 
-## For Maintainers
+## Building Documentation
 
-To update the Sphinx documentation,
-the source files that need editing are under `docs/source/`;
-everything else under `docs/` is auto-generated.
-After the manual updates under `docs/source/` are ready,
-the HTML pages are updated as follows:
+We use the Sphinx framework. The documentation source files are in `docs/source/`.
+These files can be updated as necessary.
 
-```bash
-rm -r docs/_sources docs/_static && rm docs/*.html
+The public documentation is accessible at https://async-graph-data-flow.readthedocs.io.
+The doc build is configured by `.readthedocs.yaml`. 
+Normally, even when we need to update the documentation or make a new release of async-graph-data-flow,
+neither this configuration YAML file nor Civis's account on the Read the Docs site need any updates.
+The builds by the Read The Docs site generate the necessary files (the HTML pages and other things)
+for the public documentation. All these auto-generated files are explicitly not versioned (see `.gitignore`).
 
-# Run the following command *twice* -- certain HTML updates only show up after multiple `sphinx-build` runs.
-sphinx-build docs/source docs
-```
+To build the documentation locally (for testing and development),
+install the full doc-related dependencies: `pip install -r docs/requirements.txt`,
+then run `sphinx-build -b html docs/ docs/build/`.

{async-graph-data-flow-1.4.0 → async_graph_data_flow-1.5.1}/pyproject.toml

@@ -1,10 +1,10 @@
 [build-system]
-requires = ["setuptools >= 65.3.0", "wheel"]
+requires = ["setuptools >= 69.0.0", "wheel"]
 build-backend = "setuptools.build_meta"
 
 [project]
 name = "async-graph-data-flow"
-version = "1.4.0"
+version = "1.5.1"
 description = "Asynchronous functions that pass data along a directed acyclic graph"
 readme = "README.md"
 requires-python = ">= 3.10"
@@ -29,29 +29,32 @@ classifiers = [
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
     "Framework :: AsyncIO",
 ]
 
 [project.urls]
+Documentation = "https://async-graph-data-flow.readthedocs.io"
 Source = "https://github.com/civisanalytics/async-graph-data-flow"
 
 [project.optional-dependencies]
 dev = [
     # Running tests and linters
-    "black == 23.11.0",
-    "flake8 == 6.1.0",
-    "pytest == 7.4.3",
+    "black == 24.10.0",
+    "flake8 == 7.1.1",
+    "pytest == 8.3.3",
+    "mypy == 1.13.0",
 
     # Managing source distributions
-    "build == 1.0.3",
-    "twine == 4.0.2",
-
-    # Documentation
-    "Sphinx == 7.2.6",
-    "furo == 2023.9.10",
-    "numpydoc == 1.6.0",
+    "build == 1.2.2",
+    "twine == 5.1.1",
+]
+docs = [
+    "Sphinx == 8.1.3",
+    "furo == 2024.8.6",
+    "numpydoc == 1.8.0",
     "sphinx-copybutton == 0.5.2",
-    "sphinxcontrib-mermaid == 0.9.2",
+    "sphinxcontrib-mermaid == 1.0.0",
 ]
 
 [tool.setuptools.packages.find]
@@ -62,5 +65,5 @@ zip-safe = false
 package-dir = { "" = "src" }
 
 [tool.pytest.ini_options]
-addopts = "-vv --durations=0"
+addopts = "-vv --durations=0 --strict-markers"
 testpaths = ["tests"]

{async-graph-data-flow-1.4.0 → async_graph_data_flow-1.5.1}/src/async_graph_data_flow/executor.py

@@ -20,7 +20,7 @@ class AsyncExecutor:
         self,
         graph: AsyncGraph,
         *,
-        logger: logging.Logger = None,
+        logger: logging.Logger | None = None,
         max_exceptions: int = 1_000,
     ):
         """Initialize an executor.
@@ -41,23 +41,23 @@ class AsyncExecutor:
         if not isinstance(self._graph, AsyncGraph):
             raise TypeError(f"{self._graph} must be an AsyncGraph instance")
 
-        self._node_queues = {}
-        self._consumer_tasks = {}
+        self._node_queues: dict[str, asyncio.Queue] = {}
+        self._consumer_tasks: dict[str, asyncio.Task] = {}
         self._halt_pipeline_execution = False
         self._logger = logger if logger else _LOG
         self._max_exceptions = max_exceptions
 
-        self._data_flow_stats = None
+        self._data_flow_stats: dict[str, dict[str, int]] | None = None
         self._data_flow_logging_lock = asyncio.Lock()
         self._data_flow_logging = False
         self._data_flow_logging_node_format = _DEFAULT_DATA_FLOW_LOGGING_NODE_FORMAT
         self._data_flow_logging_time_interval = _DEFAULT_DATA_FLOW_LOGGING_TIME_INTERVAL
-        self._data_flow_logging_node_filter = self._graph._nodes.keys()
-        self._data_flow_logging_last_timestamp = 0
+        self._data_flow_logging_node_filter: Iterable[str] = self._graph._nodes.keys()
+        self._data_flow_logging_last_timestamp = 0.0
 
-        self._start_node_args = None
+        self._start_node_args: dict[str, tuple] | None = None
 
-        self._exceptions = None
+        self._exceptions: dict[str, deque[Exception]] | None = None
 
     @property
     def graph(self) -> AsyncGraph:
@@ -72,7 +72,7 @@ class AsyncExecutor:
         raised from the node.
         """
         if self._exceptions is None:
-            return
+            return None
         from_deque_to_list = {}
         for node_name, excs in self._exceptions.items():
             # `excs` is a deque. Turning it into a list for user-friendliness.
@@ -106,7 +106,7 @@ class AsyncExecutor:
         node_filter: Iterable[str] | None = None,
         time_interval: int | None = None,
         logger: logging.Logger | None = None,
-    ):
+    ) -> None:
         """Turn on and configure data flow logging.
 
         For a long-running graph execution,
@@ -148,11 +148,11 @@ class AsyncExecutor:
         if logger and isinstance(logger, logging.Logger):
             self._logger = logger
 
-    def turn_off_data_flow_logging(self):
+    def turn_off_data_flow_logging(self) -> None:
         """Turn off data flow logging."""
         self._data_flow_logging = False
 
-    async def _log_data_flow_nodes(self):
+    def _log_data_flow_nodes(self):
         for node, flow in self._data_flow_stats.items():
             if (
                 self._data_flow_logging_node_filter
@@ -186,7 +186,7 @@ class AsyncExecutor:
                             current_timestamp - self._data_flow_logging_last_timestamp
                             > self._data_flow_logging_time_interval
                         ):
-                            await self._log_data_flow_nodes()
+                            self._log_data_flow_nodes()
                             self._data_flow_logging_last_timestamp = current_timestamp
 
                 queue = self._node_queues[node_name]
@@ -202,17 +202,17 @@ class AsyncExecutor:
                 try:
                     if len(params) == 0:
                         coro = node.func()
-                    elif node.unpack_input and (
-                        (is_tuple := isinstance(data, tuple)) or isinstance(data, dict)
-                    ):
-                        coro = node.func(*data) if is_tuple else node.func(**data)
+                    elif node.unpack_input and isinstance(data, tuple):
+                        coro = node.func(*data)
+                    elif node.unpack_input and isinstance(data, dict):
+                        coro = node.func(**data)
                     else:
                         coro = node.func(data)
                 except asyncio.CancelledError:
                     continue
                 except Exception as exc:
-                    await self._update_data_flow_error_stats(node_name)
-                    await self._update_exceptions(node_name, exc)
+                    self._update_data_flow_error_stats(node_name)
+                    self._update_exceptions(node_name, exc)
                     self._logger.error(traceback.format_exc())
                     if self._graph.halt_on_exception or node.halt_on_exception:
                         self._logger.error(
@@ -238,8 +238,8 @@ class AsyncExecutor:
                     except asyncio.CancelledError:
                         break
                     except Exception as exc:
-                        await self._update_data_flow_error_stats(node_name)
-                        await self._update_exceptions(node_name, exc)
+                        self._update_data_flow_error_stats(node_name)
+                        self._update_exceptions(node_name, exc)
                         self._logger.error(traceback.format_exc())
                         if self._graph.halt_on_exception or node.halt_on_exception:
                             # close current agen
@@ -255,27 +255,33 @@ class AsyncExecutor:
                             continue
                     else:
                         node_edges = self._graph._nodes_to_edges[node_name]
-                        await self._update_data_flow_in_out_stats(node_name, node_edges)
+                        self._update_data_flow_in_out_stats(node_name, node_edges)
                         await self._add_to_node_queue(node_edges, next_data_item)
 
                 queue.task_done()
             except asyncio.CancelledError:
                 break
 
-    async def _update_data_flow_in_out_stats(self, in_node: str, out_nodes: set[str]):
+    def _update_data_flow_in_out_stats(self, in_node: str, out_nodes: set[str]):
+        if self._data_flow_stats is None:
+            return None
         self._data_flow_stats[in_node]["out"] += 1
         for node in out_nodes:
             self._data_flow_stats[node]["in"] += 1
 
-    async def _update_data_flow_error_stats(self, node: str):
+    def _update_data_flow_error_stats(self, node: str):
+        if self._data_flow_stats is None:
+            return None
         self._data_flow_stats[node]["err"] += 1
 
-    async def _update_exceptions(self, node: str, exc: Exception):
+    def _update_exceptions(self, node: str, exc: Exception):
+        if self._exceptions is None:
+            return None
         self._exceptions[node].append(exc)
 
     async def _pipeline_execution(self):
-        self._data_flow_stats: dict[str, dict[str, int]] = {}
-        self._exceptions: dict[str, deque[Exception]] = {}
+        self._data_flow_stats = {}
+        self._exceptions = {}
 
         for node_name, node in self._graph._nodes.items():
             queue = asyncio.Queue(maxsize=node.queue_size)
@@ -299,7 +305,7 @@ class AsyncExecutor:
         await asyncio.gather(*self._consumer_tasks.values())
 
         if self._data_flow_logging:
-            await self._log_data_flow_nodes()
+            self._log_data_flow_nodes()
 
     def _get_start_node_args(self, start_node_args) -> dict[str, tuple]:
         if start_node_args is None:
@@ -313,7 +319,7 @@ class AsyncExecutor:
                 raise TypeError(f"args for the node '{node}' isn't a tuple: {args}")
         return start_node_args
 
-    def execute(self, start_nodes: dict[str, tuple] = None) -> None:
+    def execute(self, start_nodes: dict[str, tuple] | None = None) -> None:
         """Start executing the functions along the graph.
 
         Parameters

{async-graph-data-flow-1.4.0 → async_graph_data_flow-1.5.1}/src/async_graph_data_flow/graph.py

@@ -3,9 +3,6 @@ from collections import OrderedDict
 from typing import Any, Callable, NamedTuple
 
 
-_DEFAULT_NUM_OF_WORKERS = 1
-
-
 class InvalidAsyncGraphError(Exception):
     pass
 

{async-graph-data-flow-1.4.0 → async_graph_data_flow-1.5.1/src/async_graph_data_flow.egg-info}/PKG-INFO

@@ -1,9 +1,10 @@
 Metadata-Version: 2.1
 Name: async-graph-data-flow
-Version: 1.4.0
+Version: 1.5.1
 Summary: Asynchronous functions that pass data along a directed acyclic graph
 Author-email: Civis Analytics <opensource@civisanalytics.com>
 License: BSD 3-Clause
+Project-URL: Documentation, https://async-graph-data-flow.readthedocs.io
 Project-URL: Source, https://github.com/civisanalytics/async-graph-data-flow
 Keywords: async,asynchronous,asyncio,data flow,graph
 Classifier: Development Status :: 5 - Production/Stable
@@ -17,21 +18,24 @@ Classifier: Programming Language :: Python :: 3 :: Only
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: Framework :: AsyncIO
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
 Provides-Extra: dev
-Requires-Dist: black==23.11.0; extra == "dev"
-Requires-Dist: flake8==6.1.0; extra == "dev"
-Requires-Dist: pytest==7.4.3; extra == "dev"
-Requires-Dist: build==1.0.3; extra == "dev"
-Requires-Dist: twine==4.0.2; extra == "dev"
-Requires-Dist: Sphinx==7.2.6; extra == "dev"
-Requires-Dist: furo==2023.9.10; extra == "dev"
-Requires-Dist: numpydoc==1.6.0; extra == "dev"
-Requires-Dist: sphinx-copybutton==0.5.2; extra == "dev"
-Requires-Dist: sphinxcontrib-mermaid==0.9.2; extra == "dev"
+Requires-Dist: black==24.10.0; extra == "dev"
+Requires-Dist: flake8==7.1.1; extra == "dev"
+Requires-Dist: pytest==8.3.3; extra == "dev"
+Requires-Dist: mypy==1.13.0; extra == "dev"
+Requires-Dist: build==1.2.2; extra == "dev"
+Requires-Dist: twine==5.1.1; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: Sphinx==8.1.3; extra == "docs"
+Requires-Dist: furo==2024.8.6; extra == "docs"
+Requires-Dist: numpydoc==1.8.0; extra == "docs"
+Requires-Dist: sphinx-copybutton==0.5.2; extra == "docs"
+Requires-Dist: sphinxcontrib-mermaid==1.0.0; extra == "docs"
 
 # async-graph-data-flow
 
@@ -41,7 +45,7 @@ Requires-Dist: sphinxcontrib-mermaid==0.9.2; extra == "dev"
 
 ## Full Documentation
 
-Please visit https://civisanalytics.github.io/async-graph-data-flow
+Please visit https://async-graph-data-flow.readthedocs.io
 
 Please also check out this
 [blog post](https://www.civisanalytics.com/blog/open-source-software-release-introducing-async-graph-data-flow-a-python-library-for-efficient-data-pipelines/)
@@ -71,17 +75,18 @@ flake8 src tests examples
 black --check src tests examples
 ```
 
-## For Maintainers
+## Building Documentation
 
-To update the Sphinx documentation,
-the source files that need editing are under `docs/source/`;
-everything else under `docs/` is auto-generated.
-After the manual updates under `docs/source/` are ready,
-the HTML pages are updated as follows:
+We use the Sphinx framework. The documentation source files are in `docs/source/`.
+These files can be updated as necessary.
 
-```bash
-rm -r docs/_sources docs/_static && rm docs/*.html
+The public documentation is accessible at https://async-graph-data-flow.readthedocs.io.
+The doc build is configured by `.readthedocs.yaml`. 
+Normally, even when we need to update the documentation or make a new release of async-graph-data-flow,
+neither this configuration YAML file nor Civis's account on the Read the Docs site need any updates.
+The builds by the Read The Docs site generate the necessary files (the HTML pages and other things)
+for the public documentation. All these auto-generated files are explicitly not versioned (see `.gitignore`).
 
-# Run the following command *twice* -- certain HTML updates only show up after multiple `sphinx-build` runs.
-sphinx-build docs/source docs
-```
+To build the documentation locally (for testing and development),
+install the full doc-related dependencies: `pip install -r docs/requirements.txt`,
+then run `sphinx-build -b html docs/ docs/build/`.

{async-graph-data-flow-1.4.0 → async_graph_data_flow-1.5.1}/src/async_graph_data_flow.egg-info/SOURCES.txt

@@ -1,9 +1,12 @@
+CHANGELOG.md
 LICENSE.txt
+MANIFEST.in
 README.md
 pyproject.toml
 src/async_graph_data_flow/__init__.py
 src/async_graph_data_flow/executor.py
 src/async_graph_data_flow/graph.py
+src/async_graph_data_flow/py.typed
 src/async_graph_data_flow.egg-info/PKG-INFO
 src/async_graph_data_flow.egg-info/SOURCES.txt
 src/async_graph_data_flow.egg-info/dependency_links.txt

async_graph_data_flow-1.5.1/src/async_graph_data_flow.egg-info/requires.txt

@@ -0,0 +1,15 @@
+
+[dev]
+black==24.10.0
+flake8==7.1.1
+pytest==8.3.3
+mypy==1.13.0
+build==1.2.2
+twine==5.1.1
+
+[docs]
+Sphinx==8.1.3
+furo==2024.8.6
+numpydoc==1.8.0
+sphinx-copybutton==0.5.2
+sphinxcontrib-mermaid==1.0.0

async-graph-data-flow-1.4.0/src/async_graph_data_flow.egg-info/requires.txt

@@ -1,12 +0,0 @@
-
-[dev]
-black==23.11.0
-flake8==6.1.0
-pytest==7.4.3
-build==1.0.3
-twine==4.0.2
-Sphinx==7.2.6
-furo==2023.9.10
-numpydoc==1.6.0
-sphinx-copybutton==0.5.2
-sphinxcontrib-mermaid==0.9.2