planai 0.1.0__tar.gz

planai-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2024 Niels Provos
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

planai-0.1.0/PKG-INFO

@@ -0,0 +1,141 @@
+Metadata-Version: 2.1
+Name: planai
+Version: 0.1.0
+Summary: A simple framework for coordinating classical compute and LLM-based tasks.
+Home-page: https://github.com/provos/planai
+License: Apache-2.0
+Keywords: ai,automation,workflow,llm
+Author: Niels Provos
+Author-email: planai@provos.org
+Requires-Python: >=3.10,<4.0
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Provides-Extra: docs
+Requires-Dist: diskcache (>=5.6.3,<6.0.0)
+Requires-Dist: flask (>=3.0.3,<4.0.0)
+Requires-Dist: langchain (>=0.2.14,<0.3.0)
+Requires-Dist: ollama (>=0.3.1,<0.4.0)
+Requires-Dist: openai (>=1.42.0,<2.0.0)
+Requires-Dist: paramiko (>=3.4.1,<4.0.0)
+Requires-Dist: pydantic (>=2.8.2,<3.0.0)
+Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
+Project-URL: Documentation, https://planai.readthedocs.io
+Project-URL: Repository, https://github.com/provos/planai
+Description-Content-Type: text/markdown
+
+# PlanAI
+
+[![PyPI version](https://badge.fury.io/py/planai.svg)](https://badge.fury.io/py/planai)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Python Versions](https://img.shields.io/pypi/pyversions/planai.svg)](https://pypi.org/project/planai/)
+[![Documentation Status](https://readthedocs.org/projects/planai/badge/?version=latest)](https://planai.readthedocs.io/en/latest/?badge=latest)
+
+
+**PlanAI** is an innovative system designed for complex task automation through a sophisticated graph-based architecture. It integrates traditional computations and cutting-edge AI technologies to enable versatile and efficient workflow management.
+
+## Table of Contents
+- [Key Features](#key-features)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Advanced Features](#advanced-features)
+- [Documentation](#documentation)
+- [Contributing](#contributing)
+- [License](#license)
+- [Roadmap](#roadmap)
+
+## Key Features
+
+- **Graph-Based Architecture**: Construct dynamic workflows comprising interconnected TaskWorkers for highly customizable automation.
+- **Hybrid TaskWorkers**: Combine conventional computations (e.g., API calls) with powerful LLM-driven operations, leveraging Retrieval-Augmented Generation (RAG) capabilities.
+- **Type Safety with Pydantic**: Ensure data integrity and type consistency across workflows with Pydantic-validated input and output.
+- **Intelligent Data Routing**: Utilize type-aware routing to efficiently manage data flow between nodes, adapting to multiple downstream consumers.
+
+## Requirements
+
+- Python 3.10+
+- Poetry (for development)
+
+## Installation
+
+You can install PlanAI using pip:
+
+```bash
+pip install planai
+```
+
+For development, clone the repository and install dependencies:
+
+```bash
+git clone https://github.com/provos/planai.git
+cd planai
+poetry install
+```
+
+## Usage
+
+PlanAI allows you to create complex, AI-enhanced workflows using a graph-based architecture. Here's a basic example:
+
+```python
+from planai import Graph, TaskWorker, LLMTaskWorker, llm_from_config
+from planai.task import TaskWorkItem
+from pydantic import Field
+
+# Define custom TaskWorkers
+class CustomDataProcessor(TaskWorker):
+    output_types = [ProcessedData]
+    
+    def consume_work(self, task: RawData):
+        processed_data = self.process(task.data)
+        self.publish_work(ProcessedData(data=processed_data))
+
+# Define an LLM-powered task
+class AIAnalyzer(LLMTaskWorker):
+    output_types = [AnalysisResult]
+    
+    def consume_work(self, task: ProcessedData):
+        super().consume_work(task)
+
+# Create and run the workflow
+graph = Graph(name="Data Analysis Workflow")
+data_processor = CustomDataProcessor()
+ai_analyzer = AIAnalyzer(
+   llm=llm_from_config(provider="openai", model_name="gpt-4"),
+   prompt="Analyze the provided data and derive insights")
+
+graph.add_workers(data_processor, ai_analyzer)
+graph.set_dependency(data_processor, ai_analyzer)
+
+initial_data = RawData(data="Some raw data")
+graph.run(initial_tasks=[(data_processor, initial_data)])
+```
+
+## Advanced Features
+
+PlanAI supports advanced features like:
+
+- Caching results with `CachedTaskWorker`
+- Joining multiple task results with `JoinedTaskWorker`
+- Integrating with various LLM providers (OpenAI, Ollama, etc.)
+
+For more detailed examples and advanced usage, please refer to the `examples/` directory in the repository.
+
+## Documentation
+
+Full documentation for PlanAI is available at [https://planai.readthedocs.io/](https://planai.readthedocs.io/)
+
+## Contributing
+
+We welcome contributions to PlanAI! Please see our [Contributing Guide](CONTRIBUTING.md) for more details on how to get started.
+
+## License
+
+This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
+
+For any questions or support, please open an issue on our [GitHub issue tracker](https://github.com/provos/planai/issues).

planai-0.1.0/README.md

@@ -0,0 +1,110 @@
+# PlanAI
+
+[![PyPI version](https://badge.fury.io/py/planai.svg)](https://badge.fury.io/py/planai)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Python Versions](https://img.shields.io/pypi/pyversions/planai.svg)](https://pypi.org/project/planai/)
+[![Documentation Status](https://readthedocs.org/projects/planai/badge/?version=latest)](https://planai.readthedocs.io/en/latest/?badge=latest)
+
+
+**PlanAI** is an innovative system designed for complex task automation through a sophisticated graph-based architecture. It integrates traditional computations and cutting-edge AI technologies to enable versatile and efficient workflow management.
+
+## Table of Contents
+- [Key Features](#key-features)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Advanced Features](#advanced-features)
+- [Documentation](#documentation)
+- [Contributing](#contributing)
+- [License](#license)
+- [Roadmap](#roadmap)
+
+## Key Features
+
+- **Graph-Based Architecture**: Construct dynamic workflows comprising interconnected TaskWorkers for highly customizable automation.
+- **Hybrid TaskWorkers**: Combine conventional computations (e.g., API calls) with powerful LLM-driven operations, leveraging Retrieval-Augmented Generation (RAG) capabilities.
+- **Type Safety with Pydantic**: Ensure data integrity and type consistency across workflows with Pydantic-validated input and output.
+- **Intelligent Data Routing**: Utilize type-aware routing to efficiently manage data flow between nodes, adapting to multiple downstream consumers.
+
+## Requirements
+
+- Python 3.10+
+- Poetry (for development)
+
+## Installation
+
+You can install PlanAI using pip:
+
+```bash
+pip install planai
+```
+
+For development, clone the repository and install dependencies:
+
+```bash
+git clone https://github.com/provos/planai.git
+cd planai
+poetry install
+```
+
+## Usage
+
+PlanAI allows you to create complex, AI-enhanced workflows using a graph-based architecture. Here's a basic example:
+
+```python
+from planai import Graph, TaskWorker, LLMTaskWorker, llm_from_config
+from planai.task import TaskWorkItem
+from pydantic import Field
+
+# Define custom TaskWorkers
+class CustomDataProcessor(TaskWorker):
+    output_types = [ProcessedData]
+    
+    def consume_work(self, task: RawData):
+        processed_data = self.process(task.data)
+        self.publish_work(ProcessedData(data=processed_data))
+
+# Define an LLM-powered task
+class AIAnalyzer(LLMTaskWorker):
+    output_types = [AnalysisResult]
+    
+    def consume_work(self, task: ProcessedData):
+        super().consume_work(task)
+
+# Create and run the workflow
+graph = Graph(name="Data Analysis Workflow")
+data_processor = CustomDataProcessor()
+ai_analyzer = AIAnalyzer(
+   llm=llm_from_config(provider="openai", model_name="gpt-4"),
+   prompt="Analyze the provided data and derive insights")
+
+graph.add_workers(data_processor, ai_analyzer)
+graph.set_dependency(data_processor, ai_analyzer)
+
+initial_data = RawData(data="Some raw data")
+graph.run(initial_tasks=[(data_processor, initial_data)])
+```
+
+## Advanced Features
+
+PlanAI supports advanced features like:
+
+- Caching results with `CachedTaskWorker`
+- Joining multiple task results with `JoinedTaskWorker`
+- Integrating with various LLM providers (OpenAI, Ollama, etc.)
+
+For more detailed examples and advanced usage, please refer to the `examples/` directory in the repository.
+
+## Documentation
+
+Full documentation for PlanAI is available at [https://planai.readthedocs.io/](https://planai.readthedocs.io/)
+
+## Contributing
+
+We welcome contributions to PlanAI! Please see our [Contributing Guide](CONTRIBUTING.md) for more details on how to get started.
+
+## License
+
+This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
+
+For any questions or support, please open an issue on our [GitHub issue tracker](https://github.com/provos/planai/issues).

planai-0.1.0/pyproject.toml

@@ -0,0 +1,55 @@
+[tool.poetry]
+name = "planai"
+version = "0.1.0"
+description = "A simple framework for coordinating classical compute and LLM-based tasks."
+authors = ["Niels Provos <planai@provos.org>"]
+license = "Apache-2.0"
+readme = "README.md"
+repository = "https://github.com/provos/planai"
+documentation = "https://planai.readthedocs.io"
+keywords = ["ai", "automation", "workflow", "llm"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: Apache Software License",
+    "Operating System :: OS Independent",
+    "Intended Audience :: Developers",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+packages = [
+    { include = "planai", from = "src" },
+]
+
+[tool.poetry.dependencies]
+python = "^3.10"
+pydantic = "^2.8.2"
+langchain = "^0.2.14"
+python-dotenv = "^1.0.1"
+ollama = "^0.3.1"
+diskcache = "^5.6.3"
+paramiko = "^3.4.1"
+openai = "^1.42.0"
+flask = "^3.0.3"
+
+[tool.poetry.dev-dependencies]
+pytest = "^8.3.2"
+
+[tool.poetry.group.dev.dependencies]
+sphinx = "^7.0.0"
+sphinx-rtd-theme = "^1.3.0"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.poetry.scripts]
+planai = "planai.cli:main"
+
+[tool.poetry.extras]
+docs = ["sphinx", "sphinx_rtd_theme"]
+
+[tool.poetry.group.docs]
+optional = true
+
+[tool.poetry.group.docs.dependencies]
+sphinx = "^7.0.0"
+sphinx_rtd_theme = "^1.3.0"

planai-0.1.0/src/planai/__init__.py

@@ -0,0 +1,33 @@
+# Copyright 2024 Niels Provos
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# src/planai/__init__.py
+
+from .cached_task import CachedTaskWorker
+from .graph import Graph
+from .joined_task import JoinedTaskWorker
+from .llm_interface import llm_from_config
+from .llm_task import CachedLLMTaskWorker, LLMTaskWorker
+from .task import TaskWorker, TaskWorkItem
+
+# If you want to control what gets imported with "from planai import *"
+__all__ = [
+    "Graph",
+    "llm_from_config",
+    "LLMTaskWorker",
+    "TaskWorker",
+    "TaskWorkItem",
+    "CachedTaskWorker",
+    "CachedLLMTaskWorker",
+    "JoinedTaskWorker",
+]

planai-0.1.0/src/planai/cached_task.py

@@ -0,0 +1,113 @@
+# Copyright 2024 Niels Provos
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import hashlib
+import logging
+import threading
+from typing import List, Type
+
+from diskcache import Cache
+from pydantic import Field, PrivateAttr
+
+from .task import TaskWorker, TaskWorkItem
+
+
+class CachedTaskWorker(TaskWorker):
+    cache_dir: str = Field("./cache", description="Directory to store the cache")
+    cache_size_limit: int = Field(
+        25_000_000_000, description="Cache size limit in bytes"
+    )
+    _cache: Cache = PrivateAttr()
+    _lock: threading.Lock = PrivateAttr(default_factory=threading.Lock)
+
+    def __init__(self, **data):
+        super().__init__(**data)
+        self._cache = Cache(self.cache_dir, size_limit=self.cache_size_limit)
+
+    def _pre_consume_work(self, task: TaskWorkItem):
+        self.pre_consume_work(task)
+
+        cache_key = self._get_cache_key(task)
+        cached_results = self._cache.get(cache_key)
+
+        if cached_results is not None:
+            logging.info("Cache hit for %s with key: %s", self.name, cache_key)
+            self._publish_cached_results(cached_results, task)
+        else:
+            logging.info("Cache miss for %s with key: %s", self.name, cache_key)
+            self.consume_work(task)
+            
+        self.post_consume_work(task)
+
+    def pre_consume_work(self, task: TaskWorkItem):
+        """
+        This method is called before consuming the work item. It will be called even if the task has been cached.
+        It can be used for state manipulation, e.g. changing state for a class specific cache key.
+
+        Args:
+            task (TaskWorkItem): The work item to be consumed.
+
+        Returns:
+            None
+        """
+        pass
+
+    def post_consume_work(self, task: TaskWorkItem):
+        """
+        This method is called after consuming a work item in the task. It will be called even if the task has been cached.
+        It can be used for state manipulation, e.g. changing state for a class specific cache key.
+
+        Args:
+            task (TaskWorkItem): The work item that was consumed.
+
+        Returns:
+            None
+        """
+        pass
+
+    def _get_cache_key(self, task: TaskWorkItem) -> str:
+        """Generate a unique cache key for the input task."""
+        task_dict = task.model_dump()
+        task_str = str(sorted(task_dict.items()))  # Ensure consistent ordering
+        task_str += f" - {self.name}"  # Include the task name to avoid collisions
+        for output_type in self.output_types:
+            task_str += f" - {output_type.__name__}"
+        if extra_key := self.extra_cache_key(task):
+            task_str += f" - {extra_key}"
+        return hashlib.sha1(task_str.encode()).hexdigest()
+
+    def extra_cache_key(self, task: TaskWorkItem) -> str:
+        """Can be implemented by subclasses to provide additional cache key information."""
+        return ""
+
+    def _publish_cached_results(
+        self, cached_results: List[TaskWorkItem], input_task: TaskWorkItem
+    ):
+        """Publish cached results."""
+        for result in cached_results:
+            super().publish_work(result, input_task=input_task)
+
+    def publish_work(self, task: TaskWorkItem, input_task: TaskWorkItem):
+        """Publish work and cache the results."""
+        super().publish_work(task, input_task=input_task)
+
+        cache_key = self._get_cache_key(input_task)
+        try:
+            # since the cache key includes the name of this task worker, we can use the lock in this class
+            with self._lock:
+                cached_results = self._cache.get(cache_key, default=[])
+                cached_results.append(task)
+                self._cache.set(cache_key, cached_results)
+            logging.info("Task %s cached results for key: %s", self.name, cache_key)
+        except Exception as e:
+            logging.error("Error caching results for key %s: %s", cache_key, str(e))