indexify 0.0.43__tar.gz → 0.2__tar.gz

indexify-0.2/PKG-INFO

@@ -0,0 +1,151 @@
+Metadata-Version: 2.1
+Name: indexify
+Version: 0.2
+Summary: Python Client for Indexify
+Home-page: https://github.com/tensorlakeai/indexify
+License: Apache 2.0
+Author: Tensorlake Inc.
+Author-email: support@tensorlake.ai
+Requires-Python: >=3.9,<4.0
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: cloudpickle (>=3,<4)
+Requires-Dist: docker (>=7.1.0,<8.0.0)
+Requires-Dist: httpx-sse (>=0.4.0,<0.5.0)
+Requires-Dist: httpx[http2] (>=0,<1)
+Requires-Dist: msgpack (>=1.1.0,<2.0.0)
+Requires-Dist: nanoid (>=2.0.0,<3.0.0)
+Requires-Dist: pydantic (>=2.9.2,<3.0.0)
+Requires-Dist: pyyaml (>=6,<7)
+Requires-Dist: rich (>=13,<14)
+Requires-Dist: typer (>=0.12.5,<0.13.0)
+Project-URL: Repository, https://github.com/tensorlakeai/indexify
+Description-Content-Type: text/markdown
+
+# Indexify Python SDK
+
+[![PyPI version](https://badge.fury.io/py/indexify.svg)](https://badge.fury.io/py/indexify)
+[![Discord](https://dcbadge.vercel.app/api/server/VXkY7zVmTD?style=flat&compact=true)](https://discord.gg/VXkY7zVmTD)
+
+This is the Python SDK to build real-time continuously running unstructured data processing pipelines with Indexify.
+
+Start by writing and testing your pipelines locally using your data, then deploy them into the Indexify service to process data in real-time at scale.
+
+## Installation
+
+```shell
+pip install indexify
+```
+
+## Examples 
+**[PDF Document Extraction](./examples/pdf_document_extraction/workflow.py)**
+1. Extracts text, tables and images from an ingested PDF file
+2. Indexes the text using MiniLM-L6-v2, the images with CLIP
+3. Writes the results into a vector database.
+
+**[Youtube Transcription Summarizer](./examples/video_summarization/workflow.py)**
+1. Downloads Youtube Video
+2. Extracts audio from the video and transcribes using `Faster Whisper` 
+3. Uses Llama 3.1 backed by `Llama.cpp` to understand and classify the nature of the video.
+4. Routes the transcription dynamically to one of the transcription summarizer to retain specific summarization attributes.
+5. Finally the entire transcription is embedded and stored in a vector database for retrieval.
+
+## Quick Start
+1. Write data processing functions in Python and use Pydantic objects for returning complex data types from functions
+2. Connect functions using a graph interface. Indexify automatically stores function outputs and passes them along to downstream functions. 
+3. If a function returns a list, the downstream functions will be called with each item in the list in **parallel**.
+4. The input of the first function becomes the input to the HTTP endpoint of the Graph.
+
+## Functional Features
+1. There is **NO** limit to volume of data being ingested since we use blob stores for storing metadata and objects
+2. The server can handle 10s of 1000s of files being ingested into the graphs in parallel. 
+3. The scheduler reacts under 8 microseconds to ingestion events, so it's suitable for workflows which needs to run in realtime.
+4. Batch ingestion is handled gracefully by batching ingested data and scheduling for high throughput in production settings.
+
+```python
+from pydantic import BaseModel
+from indexify import indexify_function
+from typing import Dict, Any, Optional, List
+
+# Define function inputs and outputs
+class Document(BaseModel):
+    text: str
+    metadata: Dict[str, Any]
+
+class TextChunk(BaseModel):
+    text: str
+    metadata: Dict[str, Any]
+    embedding: Optional[List[float]] = None
+
+
+# Decorate a function which is going to be part of your data processing graph
+@indexify_function()
+def split_text(doc: Document) -> List[TextChunk]:
+    midpoint = len(doc.text) // 2
+    first_half = TextChunk(text=doc.text[:midpoint], metadata=doc.metadata)
+    second_half = TextChunk(text=doc.text[midpoint:], metadata=doc.metadata)
+    return [first_half, second_half]
+
+# Any requirements specified is automatically installed in production clusters
+@indexify_function(requirements=["langchain_text_splitter"])
+def compute_embedding(chunk: TextChunk) -> TextChunk:
+    chunk.embedding = [0.1, 0.2, 0.3]
+    return chunk
+
+# You can constrain functions to run on specific executors 
+@indexify_function(executor_runtime_name="postgres-driver-image")
+def write_to_db(chunk: TextChunk):
+    # Write to your favorite vector database
+    ...
+
+## Create a graph
+from indexify import Graph
+
+g = Graph(name="my_graph", start_node=split_text)
+g.add_edge(split_text, compute_embedding)
+g.add_edge(embed_text, write_to_db)
+```
+
+## Graph Execution
+Every time the Graph is invoked, Indexify will provide an `Invocation Id` which can be used to know about the status of the processing and any outputs from the Graph.
+
+## Run the Graph Locally
+```python
+from indexify import IndexifyClient
+
+client = IndexifyClient(local=True)
+client.register_graph(g)
+invocation_id = client.invoke_graph_with_object(g.name, Document(text="Hello, world!", metadata={"source": "test"}))
+graph_outputs = client.graph_outputs(g.name, invocation_id)
+```
+
+## Deploy the Graph to Indexify Server for Production
+> Work In Progress - The version of server that works with python based graphs haven't been released yet. It will be shortly released. Join discord for development updates. 
+```python
+from indexify import IndexifyClient
+
+client = IndexifyClient(service_url="http://localhost:8900")
+client.register_graph(g)
+```
+
+#### Ingestion into the Service
+Extraction Graphs continuously run on the Indexify Service like any other web service. Indexify Server runs the extraction graphs in parallel and in real-time when new data is ingested into the service.
+
+```python
+output_id = client.invoke_graph_with_object(g.name, Document(text="Hello, world!", metadata={"source": "test"}))
+```
+
+#### Retrieve Graph Outputs for a given ingestion object
+```python
+graph_outputs = client.graph_outputs(g.name, output_id)
+```
+
+#### Retrieve All Graph Inputs 
+```python
+graph_inputs = client.graph_inputs(g.name)
+```
+

indexify-0.2/README.md

@@ -0,0 +1,122 @@
+# Indexify Python SDK
+
+[![PyPI version](https://badge.fury.io/py/indexify.svg)](https://badge.fury.io/py/indexify)
+[![Discord](https://dcbadge.vercel.app/api/server/VXkY7zVmTD?style=flat&compact=true)](https://discord.gg/VXkY7zVmTD)
+
+This is the Python SDK to build real-time continuously running unstructured data processing pipelines with Indexify.
+
+Start by writing and testing your pipelines locally using your data, then deploy them into the Indexify service to process data in real-time at scale.
+
+## Installation
+
+```shell
+pip install indexify
+```
+
+## Examples 
+**[PDF Document Extraction](./examples/pdf_document_extraction/workflow.py)**
+1. Extracts text, tables and images from an ingested PDF file
+2. Indexes the text using MiniLM-L6-v2, the images with CLIP
+3. Writes the results into a vector database.
+
+**[Youtube Transcription Summarizer](./examples/video_summarization/workflow.py)**
+1. Downloads Youtube Video
+2. Extracts audio from the video and transcribes using `Faster Whisper` 
+3. Uses Llama 3.1 backed by `Llama.cpp` to understand and classify the nature of the video.
+4. Routes the transcription dynamically to one of the transcription summarizer to retain specific summarization attributes.
+5. Finally the entire transcription is embedded and stored in a vector database for retrieval.
+
+## Quick Start
+1. Write data processing functions in Python and use Pydantic objects for returning complex data types from functions
+2. Connect functions using a graph interface. Indexify automatically stores function outputs and passes them along to downstream functions. 
+3. If a function returns a list, the downstream functions will be called with each item in the list in **parallel**.
+4. The input of the first function becomes the input to the HTTP endpoint of the Graph.
+
+## Functional Features
+1. There is **NO** limit to volume of data being ingested since we use blob stores for storing metadata and objects
+2. The server can handle 10s of 1000s of files being ingested into the graphs in parallel. 
+3. The scheduler reacts under 8 microseconds to ingestion events, so it's suitable for workflows which needs to run in realtime.
+4. Batch ingestion is handled gracefully by batching ingested data and scheduling for high throughput in production settings.
+
+```python
+from pydantic import BaseModel
+from indexify import indexify_function
+from typing import Dict, Any, Optional, List
+
+# Define function inputs and outputs
+class Document(BaseModel):
+    text: str
+    metadata: Dict[str, Any]
+
+class TextChunk(BaseModel):
+    text: str
+    metadata: Dict[str, Any]
+    embedding: Optional[List[float]] = None
+
+
+# Decorate a function which is going to be part of your data processing graph
+@indexify_function()
+def split_text(doc: Document) -> List[TextChunk]:
+    midpoint = len(doc.text) // 2
+    first_half = TextChunk(text=doc.text[:midpoint], metadata=doc.metadata)
+    second_half = TextChunk(text=doc.text[midpoint:], metadata=doc.metadata)
+    return [first_half, second_half]
+
+# Any requirements specified is automatically installed in production clusters
+@indexify_function(requirements=["langchain_text_splitter"])
+def compute_embedding(chunk: TextChunk) -> TextChunk:
+    chunk.embedding = [0.1, 0.2, 0.3]
+    return chunk
+
+# You can constrain functions to run on specific executors 
+@indexify_function(executor_runtime_name="postgres-driver-image")
+def write_to_db(chunk: TextChunk):
+    # Write to your favorite vector database
+    ...
+
+## Create a graph
+from indexify import Graph
+
+g = Graph(name="my_graph", start_node=split_text)
+g.add_edge(split_text, compute_embedding)
+g.add_edge(embed_text, write_to_db)
+```
+
+## Graph Execution
+Every time the Graph is invoked, Indexify will provide an `Invocation Id` which can be used to know about the status of the processing and any outputs from the Graph.
+
+## Run the Graph Locally
+```python
+from indexify import IndexifyClient
+
+client = IndexifyClient(local=True)
+client.register_graph(g)
+invocation_id = client.invoke_graph_with_object(g.name, Document(text="Hello, world!", metadata={"source": "test"}))
+graph_outputs = client.graph_outputs(g.name, invocation_id)
+```
+
+## Deploy the Graph to Indexify Server for Production
+> Work In Progress - The version of server that works with python based graphs haven't been released yet. It will be shortly released. Join discord for development updates. 
+```python
+from indexify import IndexifyClient
+
+client = IndexifyClient(service_url="http://localhost:8900")
+client.register_graph(g)
+```
+
+#### Ingestion into the Service
+Extraction Graphs continuously run on the Indexify Service like any other web service. Indexify Server runs the extraction graphs in parallel and in real-time when new data is ingested into the service.
+
+```python
+output_id = client.invoke_graph_with_object(g.name, Document(text="Hello, world!", metadata={"source": "test"}))
+```
+
+#### Retrieve Graph Outputs for a given ingestion object
+```python
+graph_outputs = client.graph_outputs(g.name, output_id)
+```
+
+#### Retrieve All Graph Inputs 
+```python
+graph_inputs = client.graph_inputs(g.name)
+```

indexify-0.2/indexify/__init__.py

@@ -0,0 +1,21 @@
+from . import data_loaders
+from .client import create_client
+from .functions_sdk.graph import Graph
+from .functions_sdk.indexify_functions import (
+    indexify_function,
+    indexify_router,
+)
+from .local_client import LocalClient
+from .remote_client import RemoteClient
+from .settings import DEFAULT_SERVICE_URL
+
+__all__ = [
+    "data_loaders",
+    "Graph",
+    "indexify_function",
+    "indexify_router",
+    "DEFAULT_SERVICE_URL",
+    "RemoteClient",
+    "LocalClient",
+    "create_client",
+]

{indexify-0.0.43 → indexify-0.2}/indexify/base_client.py

@@ -1,41 +1,67 @@
 from abc import ABC, abstractmethod
 from typing import Any, Dict, List, Optional, Union
 
-from .extractor_sdk import Feature, Graph
+from pydantic import Json
 
+from indexify.functions_sdk.graph import Graph
 
-class BaseClient(ABC):
+
+class IndexifyClient(ABC):
 
     ### Operational APIs
     @abstractmethod
-    def register_extraction_graph(self, graph: Graph):
+    def register_compute_graph(self, graph: Graph):
+        """
+        Register a compute graph.
+        graph: Graph: The graph to be registered
+        """
         pass
 
     @abstractmethod
-    def graphs(self) -> str:
+    def graphs(self) -> List[str]:
+        """
+        Get the graphs.
+        return: List[str]: The graphs
+        """
         pass
 
     @abstractmethod
-    def namespaces(self) -> str:
+    def namespaces(self) -> List[str]:
+        """
+        Get the namespaces.
+        return: List[str]: The namespaces
+        """
         pass
 
     @abstractmethod
     def create_namespace(self, namespace: str):
+        """
+        Create a namespace.
+        namespace: str: The name of the namespace to be created
+        """
         pass
 
     ### Ingestion APIs
     @abstractmethod
-    def invoke_graph_with_object(self, graph: str, object: Any) -> str:
+    def invoke_graph_with_object(
+        self, graph: str, block_until_done: bool = False, **kwargs
+    ) -> str:
         """
         Invokes a graph with an input object.
         graph: str: The name of the graph to invoke
-        object: Any: The input object to the graph. It should be JSON serializable
+        kwargs: Any: Named arguments to be passed to the graph. Example: url="https://www.google.com", web_page_text="Hello world!"
         return: str: The ID of the ingested object
         """
         pass
 
     @abstractmethod
-    def invoke_graph_with_file(self, graph: str, path: str) -> str:
+    def invoke_graph_with_file(
+        self,
+        graph: str,
+        path: str,
+        metadata: Optional[Dict[str, Json]] = None,
+        block_until_done: bool = False,
+    ) -> str:
         """
         Invokes a graph with an input file. The file's mimetype is appropriately detected.
         graph: str: The name of the graph to invoke
@@ -44,10 +70,21 @@ class BaseClient(ABC):
         """
         pass
 
+    @abstractmethod
+    def rerun_graph(self, graph: str):
+        """
+        Rerun a graph.
+        graph: str: The name of the graph to rerun
+        """
+        pass
+
     ### Retrieval APIs
     @abstractmethod
-    def extracted_objects(
-        self, graph: str, ingested_object_id: str, extractor_name: Optional[str]
+    def graph_outputs(
+        self,
+        graph: str,
+        invocation_id: str,
+        fn_name: Optional[str],
     ) -> Union[Dict[str, List[Any]], List[Any]]:
         """
         Returns the extracted objects by a graph for an ingested object. If the extractor name is provided, only the objects extracted by that extractor are returned.
@@ -55,17 +92,7 @@ class BaseClient(ABC):
         graph: str: The name of the graph
         ingested_object_id: str: The ID of the ingested object
         extractor_name: Optional[str]: The name of the extractor whose output is to be returned if provided
+        block_until_done: bool = True: If True, the method will block until the extraction is done. If False, the method will return immediately.
         return: Union[Dict[str, List[Any]], List[Any]]: The extracted objects. If the extractor name is provided, the output is a list of extracted objects by the extractor. If the extractor name is not provided, the output is a dictionary with the extractor name as the key and the extracted objects as the value. If no objects are found, an empty list is returned.
         """
         pass
-
-    @abstractmethod
-    def features(
-        self, object_id: str, graph: Optional[str]
-    ) -> Union[Dict[str, List[Feature]], List[Feature]]:
-        """
-        Returns the features of an object.
-        object_id: str: The ID of the object
-        return: List[Feature]: The features associated with the object that were extracted. If a graph name is provided, only the features extracted by that graph are returned.
-        """
-        pass

indexify-0.2/indexify/cli.py

@@ -0,0 +1,235 @@
+import asyncio
+import io
+import os
+import shutil
+import signal
+import subprocess
+import sys
+import threading
+import time
+from typing import Annotated, List, Optional
+
+import docker
+import nanoid
+import typer
+from rich.console import Console
+from rich.panel import Panel
+from rich.text import Text
+from rich.theme import Theme
+
+from indexify.executor.agent import ExtractorAgent
+from indexify.executor.function_worker import FunctionWorker
+from indexify.functions_sdk.image import Image
+
+custom_theme = Theme(
+    {
+        "info": "cyan",
+        "warning": "yellow",
+        "error": "red",
+        "highlight": "magenta",
+    }
+)
+
+console = Console(theme=custom_theme)
+
+app = typer.Typer(pretty_exceptions_enable=False, no_args_is_help=True)
+
+
+@app.command(
+    help="Run server and executor in dev mode (Not recommended for production.)"
+)
+def server_dev_mode():
+    indexify_server_path = os.path.expanduser("~/.indexify/indexify-server")
+    if not os.path.exists(indexify_server_path):
+        print("indexify-server not found. Downloading...")
+        try:
+            download_command = subprocess.check_output(
+                ["curl", "-s", "https://getindexify.ai"], universal_newlines=True
+            )
+            subprocess.run(download_command, shell=True, check=True)
+        except subprocess.CalledProcessError as e:
+            print(f"failed to download indexify-server: {e}")
+            exit(1)
+        try:
+            os.makedirs(os.path.dirname(indexify_server_path), exist_ok=True)
+            shutil.move("indexify-server", indexify_server_path)
+        except Exception as e:
+            print(f"failed to move indexify-server to {indexify_server_path}: {e}")
+            exit(1)
+    print("starting indexify server and executor in dev mode...")
+    print("press Ctrl+C to stop the server and executor.")
+    print(f"server binary path: {indexify_server_path}")
+    commands = [indexify_server_path, "indexify-cli executor"]
+
+    processes = []
+    stop_event = threading.Event()
+
+    def handle_output(process):
+        for line in iter(process.stdout.readline, ""):
+            sys.stdout.write(line)
+            sys.stdout.flush()
+
+    def terminate_processes():
+        print("Terminating processes...")
+        stop_event.set()
+        for process in processes:
+            if process.poll() is None:
+                try:
+                    process.terminate()
+                    process.wait(timeout=5)
+                except subprocess.TimeoutExpired:
+                    print(f"Force killing process {process.pid}")
+                    process.kill()
+
+    def signal_handler(sig, frame):
+        print("\nCtrl+C pressed. Shutting down...")
+        terminate_processes()
+        sys.exit(0)
+
+    signal.signal(signal.SIGINT, signal_handler)
+    signal.signal(signal.SIGTERM, signal_handler)
+
+    for cmd in commands:
+        process = subprocess.Popen(
+            cmd.split(),
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+            bufsize=1,
+            universal_newlines=True,
+            preexec_fn=os.setsid if os.name != "nt" else None,
+        )
+        processes.append(process)
+
+        thread = threading.Thread(target=handle_output, args=(process,))
+        thread.daemon = True
+        thread.start()
+
+    try:
+        while True:
+            time.sleep(1)
+            if all(process.poll() is not None for process in processes):
+                print("All processes have finished.")
+                break
+    except KeyboardInterrupt:
+        signal_handler(None, None)
+    finally:
+        terminate_processes()
+
+    print("Script execution completed.")
+
+
+@app.command(help="Build image for function names")
+def build_image(workflow_file_path: str, func_names: List[str]):
+    globals_dict = {}
+
+    try:
+        exec(open(workflow_file_path).read(), globals_dict)
+    except FileNotFoundError as e:
+        raise Exception(
+            f"Could not find workflow file to execute at: " f"`{workflow_file_path}`"
+        )
+
+    found_funcs = []
+    graph = None
+    for name, obj in globals_dict.items():
+        for func_name in func_names:
+            if name == func_name:
+                found_funcs.append(name)
+                _create_image_for_func(func_name=func_name, func_obj=obj)
+
+    console.print(
+        Text(f"Processed functions: ", style="cyan"),
+        Text(f"{found_funcs}", style="green"),
+    )
+
+
+@app.command(help="Joins the extractors to the coordinator server")
+def executor(
+    server_addr: str = "localhost:8900",
+    workers: Annotated[
+        int, typer.Option(help="number of worker processes for extraction")
+    ] = 1,
+    config_path: Optional[str] = typer.Option(
+        None, help="Path to the TLS configuration file"
+    ),
+    executor_cache: Optional[str] = typer.Option(
+        "~/.indexify/executor_cache", help="Path to the executor cache directory"
+    ),
+):
+    id = nanoid.generate()
+    console.print(
+        Panel(
+            f"Number of workers: {workers}\n"
+            f"Config path: {config_path}\n"
+            f"Server address: {server_addr}\n"
+            f"Executor ID: {id}\n"
+            f"Executor cache: {executor_cache}",
+            title="Agent Configuration",
+            border_style="info",
+        )
+    )
+
+    function_worker = FunctionWorker(workers=workers)
+    from pathlib import Path
+
+    executor_cache = Path(executor_cache).expanduser().absolute()
+    if os.path.exists(executor_cache):
+        shutil.rmtree(executor_cache)
+    Path(executor_cache).mkdir(parents=True, exist_ok=True)
+
+    agent = ExtractorAgent(
+        id,
+        num_workers=workers,
+        function_worker=function_worker,
+        server_addr=server_addr,
+        config_path=config_path,
+        code_path=executor_cache,
+    )
+
+    try:
+        asyncio.get_event_loop().run_until_complete(agent.run())
+    except asyncio.CancelledError as ex:
+        console.print(Text(f"Exiting gracefully: {ex}", style="bold yellow"))
+
+
+def _create_image_for_func(func_name, func_obj):
+    console.print(
+        Text("Creating container for ", style="cyan"),
+        Text(f"`{func_name}`", style="cyan bold"),
+    )
+    _build_image(image=func_obj.image, func_name=func_name)
+
+
+def _build_image(image: Image, func_name: str = None):
+    try:
+        client = docker.from_env()
+        client.ping()
+    except Exception as e:
+        console.print(
+            Text("Unable to connect with docker: ", style="red bold"),
+            Text(f"{e}", style="red"),
+        )
+        exit(-1)
+
+    docker_file_str_template = """
+FROM {base_image}
+
+WORKDIR /app
+
+"""
+
+    docker_file_str = docker_file_str_template.format(base_image=image._base_image)
+
+    run_strs = ["RUN " + i for i in image._run_strs]
+
+    docker_file_str += "\n".join(run_strs)
+
+    console.print("Creating image using Dockerfile contents:", style="cyan bold")
+    console.print(f"{docker_file_str}", style="magenta")
+
+    client = docker.from_env()
+    client.images.build(
+        fileobj=io.BytesIO(docker_file_str.encode()),
+        tag=f"{image._image_name}:{image._tag}",
+        rm=True,
+    )

indexify-0.2/indexify/client.py

@@ -0,0 +1,18 @@
+from typing import Optional
+
+from .base_client import IndexifyClient
+from .local_client import LocalClient
+from .remote_client import RemoteClient
+from .settings import DEFAULT_SERVICE_URL
+
+
+def create_client(
+    service_url: str = DEFAULT_SERVICE_URL,
+    config_path: Optional[str] = None,
+    local: bool = False,
+    *args,
+    **kwargs,
+) -> IndexifyClient:
+    if local:
+        return LocalClient()
+    return RemoteClient(config_path=config_path, service_url=service_url, **kwargs)