trustgraph 0.2.2__tar.gz → 0.2.4__tar.gz

trustgraph-0.2.4/PKG-INFO

@@ -0,0 +1,136 @@
+Metadata-Version: 2.1
+Name: trustgraph
+Version: 0.2.4
+Summary: TrustGraph provides a means to run a pipeline of flexible AI processing components in a flexible means to achieve a processing pipeline.
+Home-page: https://github.com/trustgraph-ai/trustgraph
+Download-URL: https://github.com/trustgraph-ai/trustgraph/archive/refs/tags/v0.2.4.tar.gz
+Author: trustgraph.ai
+Author-email: security@trustgraph.ai
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch
+Requires-Dist: urllib3
+Requires-Dist: transformers
+Requires-Dist: sentence-transformers
+Requires-Dist: rdflib
+Requires-Dist: pymilvus
+Requires-Dist: langchain
+Requires-Dist: langchain-core
+Requires-Dist: langchain-huggingface
+Requires-Dist: langchain-text-splitters
+Requires-Dist: langchain-community
+Requires-Dist: huggingface-hub
+Requires-Dist: requests
+Requires-Dist: cassandra-driver
+Requires-Dist: pulsar-client
+Requires-Dist: pypdf
+Requires-Dist: anthropic
+Requires-Dist: google-cloud-aiplatform
+Requires-Dist: pyyaml
+
+
+# TrustGraph
+
+## Introduction
+
+TrustGraph is a true end-to-end (e2e) knowledge pipeline that performs a `naive extraction` on a text corpus
+to build a RDF style knowledge graph coupled with a `RAG` service compatible with cloud LLMs and open-source
+SLMs (Small Language Models).
+
+The pipeline processing components are interconnected with a pub/sub engine to
+maximize modularity and enable new knowledge processing functions. The core processing components decode documents, 
+chunk text, perform embeddings, apply a local SLM/LLM, call a LLM API, and generate LM predictions.
+
+The processing showcases the reliability and efficiences of Graph RAG algorithms which can capture
+contextual language flags that are missed in conventional RAG approaches. Graph querying algorithms enable retrieving
+not just relevant knowledge but language cues essential to understanding semantic uses unique to a text corpus.
+
+Processing modules are executed in containers.  Processing can be scaled-up
+by deploying multiple containers.
+
+### Features
+
+- PDF decoding
+- Text chunking
+- Inference of LMs deployed with [Ollama](https://ollama.com)
+- Inference of LLMs: Claude, VertexAI and AzureAI serverless endpoints
+- Application of a [HuggingFace](https://hf.co) embeddings models
+- [RDF](https://www.w3.org/TR/rdf12-schema/)-aligned Knowledge Graph extraction
+- Graph edge loading into [Apache Cassandra](https://github.com/apache/cassandra)
+- Storing embeddings in [Milvus](https://github.com/milvus-io/milvus)
+- Embedding query service
+- Graph RAG query service
+- All procesing integrates with [Apache Pulsar](https://github.com/apache/pulsar/)
+- Containers, so can be deployed using Docker Compose or Kubernetes
+- Plug'n'play architecture: switch different LLM modules to suit your needs
+
+## Architecture
+
+![architecture](architecture.png)
+
+TrustGraph is designed to be modular to support as many Language Models and environments as possible. A natural
+fit for a modular architecture is to decompose functions into a set modules connected through a pub/sub backbone.
+[Apache Pulsar](https://github.com/apache/pulsar/) serves as this pub/sub backbone. Pulsar acts as the data broker
+managing inputs and outputs between modules.
+
+**Pulsar Workflows**:
+- For processing flows, Pulsar accepts the output of a processing module
+  and queues it for input to the next subscribed module.
+- For services such as LLMs and embeddings, Pulsar provides a client/server
+  model.  A Pulsar queue is used as the input to the service.  When
+  processed, the output is then delivered to a separate queue where a client
+  subscriber can request that output.
+
+The entire architecture, the pub/sub backbone and set of modules, is bundled into a single Python package. A container image with the
+package installed can also run the entire architecture.
+
+## Core Modules
+
+- `chunker-recursive` - Accepts text documents and uses LangChain recursive
+  chunking algorithm to produce smaller text chunks.
+- `embeddings-hf` - A service which analyses text and returns a vector
+  embedding using one of the HuggingFace embeddings models.
+- `embeddings-vectorize` - Uses an embeddings service to get a vector
+  embedding which is added to the processor payload.
+- `graph-rag` - A query service which applies a Graph RAG algorithm to
+  provide a response to a text prompt.
+- `graph-write-cassandra` - Takes knowledge graph edges and writes them to
+  a Cassandra store.
+- `kg-extract-definitions` - knowledge extractor - examines text and
+  produces graph edges.
+  describing discovered terms and also their defintions.  Definitions are
+  derived using the input  documents.
+- `kg-extract-relationships` - knowledge extractor - examines text and
+  produces graph edges describing the relationships between discovered
+  terms.
+- `loader` - Takes a document and loads into the processing pipeline.  Used
+  e.g. to add PDF documents.
+- `pdf-decoder` - Takes a PDF doc and emits text extracted from the document.
+  Text extraction from PDF is not a perfect science as PDF is a printable
+  format.  For instance, the wrapping of text between lines in a PDF document
+  is not semantically encoded, so the decoder will see wrapped lines as
+  space-separated.
+- `vector-write-milvus` - Takes vector-entity mappings and records them
+  in the vector embeddings store.
+
+## LM Specific Modules
+
+- `llm-azure-text` - Sends request to AzureAI serverless endpoint
+- `llm-claude-text` - Sends request to Anthropic's API
+- `llm-ollama-text` -  Sends request to LM running using Ollama
+- `llm-vertexai-text` -  Sends request to model available through VertexAI API
+
+## Quickstart Guide
+
+See [Quickstart on Docker Compose](docs/README.quickstart-docker-compose.md)
+
+## Development Guide
+
+See [Development on trustgraph](docs/README.development.md)
+
+
+

trustgraph-0.2.4/README.md

@@ -0,0 +1,102 @@
+
+# TrustGraph
+
+## Introduction
+
+TrustGraph is a true end-to-end (e2e) knowledge pipeline that performs a `naive extraction` on a text corpus
+to build a RDF style knowledge graph coupled with a `RAG` service compatible with cloud LLMs and open-source
+SLMs (Small Language Models).
+
+The pipeline processing components are interconnected with a pub/sub engine to
+maximize modularity and enable new knowledge processing functions. The core processing components decode documents, 
+chunk text, perform embeddings, apply a local SLM/LLM, call a LLM API, and generate LM predictions.
+
+The processing showcases the reliability and efficiences of Graph RAG algorithms which can capture
+contextual language flags that are missed in conventional RAG approaches. Graph querying algorithms enable retrieving
+not just relevant knowledge but language cues essential to understanding semantic uses unique to a text corpus.
+
+Processing modules are executed in containers.  Processing can be scaled-up
+by deploying multiple containers.
+
+### Features
+
+- PDF decoding
+- Text chunking
+- Inference of LMs deployed with [Ollama](https://ollama.com)
+- Inference of LLMs: Claude, VertexAI and AzureAI serverless endpoints
+- Application of a [HuggingFace](https://hf.co) embeddings models
+- [RDF](https://www.w3.org/TR/rdf12-schema/)-aligned Knowledge Graph extraction
+- Graph edge loading into [Apache Cassandra](https://github.com/apache/cassandra)
+- Storing embeddings in [Milvus](https://github.com/milvus-io/milvus)
+- Embedding query service
+- Graph RAG query service
+- All procesing integrates with [Apache Pulsar](https://github.com/apache/pulsar/)
+- Containers, so can be deployed using Docker Compose or Kubernetes
+- Plug'n'play architecture: switch different LLM modules to suit your needs
+
+## Architecture
+
+![architecture](architecture.png)
+
+TrustGraph is designed to be modular to support as many Language Models and environments as possible. A natural
+fit for a modular architecture is to decompose functions into a set modules connected through a pub/sub backbone.
+[Apache Pulsar](https://github.com/apache/pulsar/) serves as this pub/sub backbone. Pulsar acts as the data broker
+managing inputs and outputs between modules.
+
+**Pulsar Workflows**:
+- For processing flows, Pulsar accepts the output of a processing module
+  and queues it for input to the next subscribed module.
+- For services such as LLMs and embeddings, Pulsar provides a client/server
+  model.  A Pulsar queue is used as the input to the service.  When
+  processed, the output is then delivered to a separate queue where a client
+  subscriber can request that output.
+
+The entire architecture, the pub/sub backbone and set of modules, is bundled into a single Python package. A container image with the
+package installed can also run the entire architecture.
+
+## Core Modules
+
+- `chunker-recursive` - Accepts text documents and uses LangChain recursive
+  chunking algorithm to produce smaller text chunks.
+- `embeddings-hf` - A service which analyses text and returns a vector
+  embedding using one of the HuggingFace embeddings models.
+- `embeddings-vectorize` - Uses an embeddings service to get a vector
+  embedding which is added to the processor payload.
+- `graph-rag` - A query service which applies a Graph RAG algorithm to
+  provide a response to a text prompt.
+- `graph-write-cassandra` - Takes knowledge graph edges and writes them to
+  a Cassandra store.
+- `kg-extract-definitions` - knowledge extractor - examines text and
+  produces graph edges.
+  describing discovered terms and also their defintions.  Definitions are
+  derived using the input  documents.
+- `kg-extract-relationships` - knowledge extractor - examines text and
+  produces graph edges describing the relationships between discovered
+  terms.
+- `loader` - Takes a document and loads into the processing pipeline.  Used
+  e.g. to add PDF documents.
+- `pdf-decoder` - Takes a PDF doc and emits text extracted from the document.
+  Text extraction from PDF is not a perfect science as PDF is a printable
+  format.  For instance, the wrapping of text between lines in a PDF document
+  is not semantically encoded, so the decoder will see wrapped lines as
+  space-separated.
+- `vector-write-milvus` - Takes vector-entity mappings and records them
+  in the vector embeddings store.
+
+## LM Specific Modules
+
+- `llm-azure-text` - Sends request to AzureAI serverless endpoint
+- `llm-claude-text` - Sends request to Anthropic's API
+- `llm-ollama-text` -  Sends request to LM running using Ollama
+- `llm-vertexai-text` -  Sends request to model available through VertexAI API
+
+## Quickstart Guide
+
+See [Quickstart on Docker Compose](docs/README.quickstart-docker-compose.md)
+
+## Development Guide
+
+See [Development on trustgraph](docs/README.development.md)
+
+
+

trustgraph-0.2.4/scripts/embeddings-ollama

@@ -0,0 +1,6 @@
+#!/usr/bin/env python3
+
+from trustgraph.embeddings.ollama import run
+
+run()
+

trustgraph-0.2.4/scripts/init-pulsar-manager

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)
+
+curl \
+    -H "X-XSRF-TOKEN: $CSRF_TOKEN" \
+    -H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" \
+    -H 'Content-Type: application/json' \
+    -X PUT \
+    http://localhost:7750/pulsar-manager/users/superuser \
+    -d '{"name": "admin", "password": "apachepulsar", "description": "test", "email": "username@test.org"}'

{trustgraph-0.2.2 → trustgraph-0.2.4}/setup.py

@@ -4,7 +4,7 @@ import os
 with open("README.md", "r") as fh:
     long_description = fh.read()
 
-version = "0.2.2"
+version = "0.2.4"
 
 setuptools.setup(
     name="trustgraph",
@@ -46,22 +46,24 @@ setuptools.setup(
     ],
     scripts=[
         "scripts/chunker-recursive",
+        "scripts/embeddings-hf",
+        "scripts/embeddings-ollama",
+        "scripts/embeddings-vectorize",
+        "scripts/graph-rag",
         "scripts/graph-show",
         "scripts/graph-to-turtle",
         "scripts/graph-write-cassandra",
+        "scripts/init-pulsar-manager",
         "scripts/kg-extract-definitions",
         "scripts/kg-extract-relationships",
+        "scripts/llm-azure-text",
+        "scripts/llm-claude-text",
         "scripts/llm-ollama-text",
         "scripts/llm-vertexai-text",
-        "scripts/llm-claude-text",
-        "scripts/llm-azure-text",
-        "scripts/run-processing",
         "scripts/loader",
         "scripts/pdf-decoder",
         "scripts/query",
-        "scripts/embeddings-vectorize",
-        "scripts/embeddings-hf",
+        "scripts/run-processing",
         "scripts/vector-write-milvus",
-        "scripts/graph-rag",
     ]
 )

{trustgraph-0.2.2 → trustgraph-0.2.4}/trustgraph/chunker/recursive/chunker.py

@@ -30,6 +30,8 @@ class Processor:
             output_queue=default_output_queue,
             subscriber=default_subscriber,
             log_level=LogLevel.INFO,
+            chunk_size=2000,
+            chunk_overlap=100,
     ):
 
         self.client = None
@@ -50,8 +52,8 @@ class Processor:
         )
 
         self.text_splitter = RecursiveCharacterTextSplitter(
-            chunk_size=1000,
-            chunk_overlap=20,
+            chunk_size=chunk_size,
+            chunk_overlap=chunk_overlap,
             length_function=len,
             is_separator_regex=False,
         )
@@ -146,6 +148,20 @@ def run():
         help=f'Output queue (default: info)'
     )
 
+    parser.add_argument(
+        '-z', '--chunk-size',
+        type=int,
+        default=2000,
+        help=f'Chunk size (default: 2000)'
+    )
+
+    parser.add_argument(
+        '-v', '--chunk-overlap',
+        type=int,
+        default=100,
+        help=f'Chunk overlap (default: 100)'
+    )
+
     args = parser.parse_args()
 
 
@@ -159,6 +175,8 @@ def run():
                 output_queue=args.output_queue,
                 subscriber=args.subscriber,
                 log_level=args.log_level,
+                chunk_size=args.chunk_size,
+                chunk_overlap=args.chunk_overlap,
             )
 
             p.run()

trustgraph-0.2.4/trustgraph/embeddings/ollama/__init__.py

@@ -0,0 +1,3 @@
+
+from . processor import *
+

trustgraph-0.2.4/trustgraph/embeddings/ollama/__main__.py

@@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+
+from . processor import run
+
+if __name__ == '__main__':
+    run()
+

trustgraph-0.2.4/trustgraph/embeddings/ollama/processor.py

@@ -0,0 +1,175 @@
+
+"""
+Embeddings service, applies an embeddings model selected from HuggingFace.
+Input is text, output is embeddings vector.
+"""
+
+import pulsar
+from pulsar.schema import JsonSchema
+import tempfile
+import base64
+import os
+import argparse
+from langchain_community.embeddings import OllamaEmbeddings
+import time
+
+from ... schema import EmbeddingsRequest, EmbeddingsResponse
+from ... log_level import LogLevel
+
+default_pulsar_host = os.getenv("PULSAR_HOST", 'pulsar://pulsar:6650')
+default_input_queue = 'embeddings'
+default_output_queue = 'embeddings-response'
+default_subscriber = 'embeddings-ollama'
+default_model="mxbai-embed-large"
+default_ollama = 'http://localhost:11434'
+
+class Processor:
+
+    def __init__(
+            self,
+            pulsar_host=default_pulsar_host,
+            input_queue=default_input_queue,
+            output_queue=default_output_queue,
+            subscriber=default_subscriber,
+            log_level=LogLevel.INFO,
+            model=default_model,
+            ollama=default_ollama,
+    ):
+
+        self.client = None
+
+        self.client = pulsar.Client(
+            pulsar_host,
+            logger=pulsar.ConsoleLogger(log_level.to_pulsar())
+        )
+
+        self.consumer = self.client.subscribe(
+            input_queue, subscriber,
+            schema=JsonSchema(EmbeddingsRequest),
+        )
+
+        self.producer = self.client.create_producer(
+            topic=output_queue,
+            schema=JsonSchema(EmbeddingsResponse),
+        )
+
+        self.embeddings = OllamaEmbeddings(base_url=ollama, model=model)
+
+    def run(self):
+
+        while True:
+
+            msg = self.consumer.receive()
+
+            try:
+
+                v = msg.value()
+
+	        # Sender-produced ID
+
+                id = msg.properties()["id"]
+
+                print(f"Handling input {id}...", flush=True)
+
+                text = v.text
+                embeds = self.embeddings.embed_query([text])
+
+                print("Send response...", flush=True)
+                r = EmbeddingsResponse(vectors=[embeds])
+
+                self.producer.send(r, properties={"id": id})
+
+                print("Done.", flush=True)
+
+                # Acknowledge successful processing of the message
+                self.consumer.acknowledge(msg)
+
+            except Exception as e:
+
+                print("Exception:", e, flush=True)
+
+                # Message failed to be processed
+                self.consumer.negative_acknowledge(msg)
+
+    def __del__(self):
+
+        if self.client:
+            self.client.close()
+
+def run():
+
+    parser = argparse.ArgumentParser(
+        prog='embeddings-ollama',
+        description=__doc__,
+    )
+
+    parser.add_argument(
+        '-p', '--pulsar-host',
+        default=default_pulsar_host,
+        help=f'Pulsar host (default: {default_pulsar_host})',
+    )
+
+    parser.add_argument(
+        '-i', '--input-queue',
+        default=default_input_queue,
+        help=f'Input queue (default: {default_input_queue})'
+    )
+
+    parser.add_argument(
+        '-s', '--subscriber',
+        default=default_subscriber,
+        help=f'Queue subscriber name (default: {default_subscriber})'
+    )
+
+    parser.add_argument(
+        '-o', '--output-queue',
+        default=default_output_queue,
+        help=f'Output queue (default: {default_output_queue})'
+    )
+
+    parser.add_argument(
+        '-l', '--log-level',
+        type=LogLevel,
+        default=LogLevel.INFO,
+        choices=list(LogLevel),
+        help=f'Output queue (default: info)'
+    )
+
+    parser.add_argument(
+        '-m', '--model',
+        default=default_model,
+        help=f'Embeddings model (default: {default_model})'
+    )
+
+    parser.add_argument(
+        '-r', '--ollama',
+        default=default_ollama,
+        help=f'ollama (default: {default_ollama})'
+    )
+
+    args = parser.parse_args()
+
+    
+    while True:
+
+        try:
+
+            p = Processor(
+                pulsar_host=args.pulsar_host,
+                input_queue=args.input_queue,
+                output_queue=args.output_queue,
+                subscriber=args.subscriber,
+                log_level=args.log_level,
+                model=args.model,
+                ollama=args.ollama,
+            )
+
+            p.run()
+
+        except Exception as e:
+
+            print("Exception:", e, flush=True)
+            print("Will retry...", flush=True)
+
+        time.sleep(10)
+

{trustgraph-0.2.2 → trustgraph-0.2.4}/trustgraph/graph_rag.py

@@ -1,6 +1,6 @@
 
 from trustgraph.trustgraph import TrustGraph
-from trustgraph.edge_map import VectorStore
+from trustgraph.triple_vectors import TripleVectors
 from trustgraph.trustgraph import TrustGraph
 from trustgraph.llm_client import LlmClient
 from trustgraph.embeddings_client import EmbeddingsClient
@@ -15,7 +15,10 @@ class GraphRag:
             graph_hosts=None,
             pulsar_host="pulsar://pulsar:6650",
             vector_store="http://milvus:19530",
-            verbose=False
+            verbose=False,
+            entity_limit=50,
+            triple_limit=30,
+            max_sg_size=3000,
     ):
 
         self.verbose=verbose
@@ -30,11 +33,11 @@ class GraphRag:
 
         self.embeddings = EmbeddingsClient(pulsar_host=pulsar_host)
 
-        self.vecstore = VectorStore(vector_store)
+        self.vecstore = TripleVectors(vector_store)
 
-        self.entity_limit=50
-        self.query_limit=30
-        self.max_sg_size=3000
+        self.entity_limit=entity_limit
+        self.query_limit=triple_limit
+        self.max_sg_size=max_sg_size
 
         self.label_cache = {}
 
@@ -71,6 +74,8 @@ class GraphRag:
                 limit=self.entity_limit
             )
 
+            print("Obtained", len(res), "entities")
+
             entities = set([
                 item["entity"]["entity"]
                 for item in res

{trustgraph-0.2.2 → trustgraph-0.2.4}/trustgraph/llm/ollama_text/llm.py

@@ -142,8 +142,8 @@ def run():
 
     parser.add_argument(
         '-r', '--ollama',
-        default="http://localhost:11434",
-        help=f'ollama (default: http://localhost:11434)'
+        default=default_ollama,
+        help=f'ollama (default: {default_ollama})'
     )
 
     args = parser.parse_args()

{trustgraph-0.2.2 → trustgraph-0.2.4}/trustgraph/rag/graph/rag.py

@@ -34,6 +34,9 @@ class Processor:
             log_level=LogLevel.INFO,
             graph_hosts=default_graph_hosts,
             vector_store=default_vector_store,
+            entity_limit=50,
+            triple_limit=30,
+            max_sg_size=3000,
     ):
 
         self.client = None
@@ -58,6 +61,9 @@ class Processor:
             graph_hosts=graph_hosts,
             vector_store=vector_store,
             verbose=True,
+            entity_limit=entity_limit,
+            triple_limit=triple_limit,
+            max_sg_size=max_sg_size,
         )
 
     def run(self):
@@ -102,7 +108,7 @@ class Processor:
 def run():
 
     parser = argparse.ArgumentParser(
-        prog='llm-ollama-text',
+        prog='graph-rag',
         description=__doc__,
     )
 
@@ -150,6 +156,27 @@ def run():
         help=f'Vector host (default: http://milvus:19530)'
     )
 
+    parser.add_argument(
+        '-e', '--entity-limit',
+        type=int,
+        default=50,
+        help=f'Entity vector fetch limit (default: 50)'
+    )
+
+    parser.add_argument(
+        '-t', '--triple-limit',
+        type=int,
+        default=30,
+        help=f'Triple query limit, per query (default: 30)'
+    )
+
+    parser.add_argument(
+        '-u', '--max-subgraph-size',
+        type=int,
+        default=3000,
+        help=f'Max subgraph size (default: 3000)'
+    )
+
     args = parser.parse_args()
     
     while True:
@@ -164,6 +191,9 @@ def run():
                 log_level=args.log_level,
                 graph_hosts=args.graph_hosts.split(","),
                 vector_store=args.vector_store,
+                entity_limit=args.entity_limit,
+                triple_limit=args.triple_limit,
+                max_sg_size=args.max_subgraph_size,
             )
 
             p.run()