apify-haystack 0.0.1a6__tar.gz → 0.1.0a2__tar.gz

{apify_haystack-0.0.1a6 → apify_haystack-0.1.0a2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: apify_haystack
-Version: 0.0.1a6
+Version: 0.1.0a2
 Summary: Apify-haystack integration
 Home-page: https://apify.com
 License: Apache-2.0
@@ -38,7 +38,7 @@ Haystack offers an ecosystem of tools for building, managing, and deploying sear
 
 ## Installation
 
-Apify-haystack is available as the [`apify-haystack`](https://pypi.org/project/apify-haystack/) PyPI package.
+Apify-haystack is available at the [`apify-haystack`](https://pypi.org/project/apify-haystack/) PyPI package.
 
 ```sh
 pip install apify-haystack
@@ -46,11 +46,51 @@ pip install apify-haystack
 
 ## Examples
 
-See the [examples directory](https://github.com/apify/apify-haystack/blob/master/src/apify_haystack/examples) for more examples, here is a list of few of them
+Crawl website using Apify Website Content Crawler and convert it to Haystack Documents.
 
-- Load a dataset from Apify and convert it to Haystack Documents
-- Call Apify Actor and load a dataset to convert it to Haystack Documents
-- Crawl website, scrape text content, and store it in the InMemoryDocumentStore
+You need to have an Apify account and API token to run this example.
+You can start with a free account at [Apify](https://apify.com/) and get your [API token](https://docs.apify.com/platform/integrations/api).
+
+In the example below, specify `apify_api_token` and run the script:
+
+```python
+from dotenv import load_dotenv
+from haystack import Document
+
+from apify_haystack import ApifyDatasetFromActorCall
+
+# Sey APIFY-API-TOKEN here or load it from .env file
+apify_api_token = "" or load_dotenv()
+
+actor_id = "apify/website-content-crawler"
+run_input = {
+    "maxCrawlPages": 3,  # limit the number of pages to crawl
+    "startUrls": [{"url": "https://haystack.deepset.ai/"}],
+}
+
+
+def dataset_mapping_function(dataset_item: dict) -> Document:
+    return Document(content=dataset_item.get("text"), meta={"url": dataset_item.get("url")})
+
+
+actor = ApifyDatasetFromActorCall(
+    actor_id=actor_id, run_input=run_input, dataset_mapping_function=dataset_mapping_function
+)
+print(f"Calling the Apify actor {actor_id} ... crawling will take some time ...")
+print("You can monitor the progress at: https://console.apify.com/actors/runs")
+
+dataset = actor.run().get("documents")
+
+print(f"Loaded {len(dataset)} documents from the Apify Actor {actor_id}:")
+for d in dataset:
+    print(d)
+```
+
+See other examples in the [examples directory](https://github.com/apify/apify-haystack/blob/master/src/apify_haystack/examples) for more examples, here is a list of few of them
+
+- Load a dataset from Apify and convert it to Haystack Document
+- Call [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor and convert the data into the Haystack Documents
+- Crawl website, retrieve text content, and store it in the `InMemoryDocumentStore`
 - Retrieval-Augmented Generation (RAG): Extracting text from a website & question answering
 
 ## Support

apify_haystack-0.1.0a2/README.md

@@ -0,0 +1,80 @@
+# Apify-Haystack integration
+
+The Apify-Haystack integration allows easy interaction between the Apify platform and Haystack.
+
+Apify is a platform for web scraping, data extraction, and web automation tasks.
+It provides serverless applications called Actors for different tasks, like crawling websites, and scraping Facebook, Instagram, and Google results, etc.
+
+Haystack offers an ecosystem of tools for building, managing, and deploying search engines and LLM applications.
+
+## Installation
+
+Apify-haystack is available at the [`apify-haystack`](https://pypi.org/project/apify-haystack/) PyPI package.
+
+```sh
+pip install apify-haystack
+```
+
+## Examples
+
+Crawl website using Apify Website Content Crawler and convert it to Haystack Documents.
+
+You need to have an Apify account and API token to run this example.
+You can start with a free account at [Apify](https://apify.com/) and get your [API token](https://docs.apify.com/platform/integrations/api).
+
+In the example below, specify `apify_api_token` and run the script:
+
+```python
+from dotenv import load_dotenv
+from haystack import Document
+
+from apify_haystack import ApifyDatasetFromActorCall
+
+# Sey APIFY-API-TOKEN here or load it from .env file
+apify_api_token = "" or load_dotenv()
+
+actor_id = "apify/website-content-crawler"
+run_input = {
+    "maxCrawlPages": 3,  # limit the number of pages to crawl
+    "startUrls": [{"url": "https://haystack.deepset.ai/"}],
+}
+
+
+def dataset_mapping_function(dataset_item: dict) -> Document:
+    return Document(content=dataset_item.get("text"), meta={"url": dataset_item.get("url")})
+
+
+actor = ApifyDatasetFromActorCall(
+    actor_id=actor_id, run_input=run_input, dataset_mapping_function=dataset_mapping_function
+)
+print(f"Calling the Apify actor {actor_id} ... crawling will take some time ...")
+print("You can monitor the progress at: https://console.apify.com/actors/runs")
+
+dataset = actor.run().get("documents")
+
+print(f"Loaded {len(dataset)} documents from the Apify Actor {actor_id}:")
+for d in dataset:
+    print(d)
+```
+
+See other examples in the [examples directory](https://github.com/apify/apify-haystack/blob/master/src/apify_haystack/examples) for more examples, here is a list of few of them
+
+- Load a dataset from Apify and convert it to Haystack Document
+- Call [Website Content Crawler](https://apify.com/apify/website-content-crawler) Actor and convert the data into the Haystack Documents
+- Crawl website, retrieve text content, and store it in the `InMemoryDocumentStore`
+- Retrieval-Augmented Generation (RAG): Extracting text from a website & question answering
+
+## Support
+
+If you find any bug or issue, please [submit an issue on GitHub](https://github.com/apify/apify-haystack/issues).
+For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).
+
+## Contributing
+
+Your code contributions are welcome.
+If you have any ideas for improvements, either submit an issue or create a pull request.
+For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/apify-haystack/blob/master/CONTRIBUTING.md).
+
+## License
+
+This project is licensed under the Apache License 2.0 - see the [LICENSE](https://github.com/apify/apify-haystack/blob/master/LICENSE) file for details.

{apify_haystack-0.0.1a6 → apify_haystack-0.1.0a2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
 
 [tool.poetry]
 name = "apify_haystack"
-version = "0.0.1a6"
+version = "0.1.0a2"
 description = "Apify-haystack integration"
 authors = ["Apify Technologies s.r.o. <support@apify.com>"]
 homepage = "https://apify.com"

{apify_haystack-0.0.1a6 → apify_haystack-0.1.0a2}/src/apify_haystack/__init__.py

@@ -2,6 +2,6 @@ from importlib import metadata
 
 __version__ = metadata.version("apify-haystack")
 
-from apify_haystack.apify_dataset import ApifyDatasetFromActorCall, ApifyDatasetLoader
+from apify_haystack.apify_dataset import ApifyDatasetFromActorCall, ApifyDatasetFromTaskCall, ApifyDatasetLoader
 
-__all__ = ["ApifyDatasetLoader", "ApifyDatasetFromActorCall", "ApifyDatasetFromActorCall"]
+__all__ = ["ApifyDatasetLoader", "ApifyDatasetFromActorCall", "ApifyDatasetFromTaskCall"]

{apify_haystack-0.0.1a6 → apify_haystack-0.1.0a2}/src/apify_haystack/examples/apify_actor_call.py

@@ -19,8 +19,8 @@ from haystack import Document
 
 from apify_haystack import ApifyDatasetFromActorCall
 
-# Sey APIFY-API-TOKEN here or load it from .env file
-apify_token = "" or load_dotenv()
+# Set APIFY-API-TOKEN here or load it from .env file
+apify_api_token = "" or load_dotenv()
 
 actor_id = "apify/website-content-crawler"
 run_input = {
@@ -34,9 +34,14 @@ def dataset_mapping_function(dataset_item: dict) -> Document:
 
 
 actor = ApifyDatasetFromActorCall(
-    actor_id=actor_id, run_input=run_input, dataset_mapping_function=dataset_mapping_function
+    actor_id=actor_id,
+    run_input=run_input,
+    dataset_mapping_function=dataset_mapping_function,
+    apify_api_token=str(apify_api_token),
 )
-print(f"Calling Apify actor {actor_id} ... crawling will take some time ...")
+print(f"Calling the Apify actor {actor_id} ... crawling will take some time ...")
+print("You can monitor the progress at: https://console.apify.com/actors/runs")
+
 dataset = actor.run().get("documents")
 
 print(f"Loaded {len(dataset)} documents from the Apify Actor {actor_id}:")

{apify_haystack-0.0.1a6 → apify_haystack-0.1.0a2}/src/apify_haystack/examples/crawl_and_process_data.py

@@ -31,7 +31,7 @@ from apify_haystack import ApifyDatasetFromActorCall
 load_dotenv()
 
 # Set APIFY-API-TOKEN here or use it from .env file
-apify_token = "" or os.getenv("APIFY_API_TOKEN")
+apify_api_token = "" or os.getenv("APIFY_API_TOKEN")
 openai_api_key = "" or os.getenv("OPENAI_API_KEY")
 
 actor_id = "apify/website-content-crawler"
@@ -46,7 +46,10 @@ def dataset_mapping_function(dataset_item: dict) -> Document:
 
 
 document_loader = ApifyDatasetFromActorCall(
-    actor_id=actor_id, run_input=run_input, dataset_mapping_function=dataset_mapping_function
+    actor_id=actor_id,
+    run_input=run_input,
+    dataset_mapping_function=dataset_mapping_function,
+    apify_api_token=apify_api_token,
 )
 
 document_store = InMemoryDocumentStore()

{apify_haystack-0.0.1a6 → apify_haystack-0.1.0a2}/src/apify_haystack/examples/rag_with_crawled_website.py

@@ -33,7 +33,7 @@ from apify_haystack import ApifyDatasetFromActorCall
 
 # Set APIFY-API-TOKEN here or use it from .env file
 load_dotenv()
-apify_token = "" or os.getenv("APIFY_API_TOKEN")
+apify_api_token = "" or os.getenv("APIFY_API_TOKEN")
 openai_api_key = "" or os.getenv("OPENAI_API_KEY")
 
 actor_id = "apify/website-content-crawler"
@@ -48,7 +48,10 @@ def dataset_mapping_function(dataset_item: dict) -> Document:
 
 
 apify_dataset_loader = ApifyDatasetFromActorCall(
-    actor_id=actor_id, run_input=run_input, dataset_mapping_function=dataset_mapping_function
+    actor_id=actor_id,
+    run_input=run_input,
+    dataset_mapping_function=dataset_mapping_function,
+    apify_api_token=apify_api_token,
 )
 
 # Components

apify_haystack-0.0.1a6/README.md

@@ -1,40 +0,0 @@
-# Apify-Haystack integration
-
-The Apify-Haystack integration allows easy interaction between the Apify platform and Haystack.
-
-Apify is a platform for web scraping, data extraction, and web automation tasks.
-It provides serverless applications called Actors for different tasks, like crawling websites, and scraping Facebook, Instagram, and Google results, etc.
-
-Haystack offers an ecosystem of tools for building, managing, and deploying search engines and LLM applications.
-
-## Installation
-
-Apify-haystack is available as the [`apify-haystack`](https://pypi.org/project/apify-haystack/) PyPI package.
-
-```sh
-pip install apify-haystack
-```
-
-## Examples
-
-See the [examples directory](https://github.com/apify/apify-haystack/blob/master/src/apify_haystack/examples) for more examples, here is a list of few of them
-
-- Load a dataset from Apify and convert it to Haystack Documents
-- Call Apify Actor and load a dataset to convert it to Haystack Documents
-- Crawl website, scrape text content, and store it in the InMemoryDocumentStore
-- Retrieval-Augmented Generation (RAG): Extracting text from a website & question answering
-
-## Support
-
-If you find any bug or issue, please [submit an issue on GitHub](https://github.com/apify/apify-haystack/issues).
-For questions, you can ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/apify), in GitHub Discussions or you can join our [Discord server](https://discord.com/invite/jyEM2PRvMU).
-
-## Contributing
-
-Your code contributions are welcome.
-If you have any ideas for improvements, either submit an issue or create a pull request.
-For contribution guidelines and the code of conduct, see [CONTRIBUTING.md](https://github.com/apify/apify-haystack/blob/master/CONTRIBUTING.md).
-
-## License
-
-This project is licensed under the Apache License 2.0 - see the [LICENSE](https://github.com/apify/apify-haystack/blob/master/LICENSE) file for details.

apify_haystack-0.0.1a6/src/apify_haystack/main.py

@@ -1,5 +0,0 @@
-from __future__ import annotations
-
-
-def main() -> None:
-    """Main function."""