grt-lexical-graph 3.17.2.dev0__tar.gz

grt_lexical_graph-3.17.2.dev0/.gitignore

@@ -0,0 +1,33 @@
+# ignore virtual environment directories
+.venv/
+
+# ignore generated files and directories from version control
+/examples/lexical-graph/notebooks/extracted/
+/examples/lexical-graph/notebooks/output/
+/examples/lexical-graph-local-dev/notebooks/output/
+/examples/lexical-graph-hybrid-dev/notebooks/output/
+/examples/lexical-graph-hybrid-dev/notebooks/extracted/
+
+# ignore IDE files
+.vscode/
+.idea/
+
+# Jupyter Notebook checkpoints
+.ipynb_checkpoints/
+
+# cache files
+__pycache__/
+
+# Byte-compiled / optimized / DLL files
+*.pyc
+
+# hypothesis artifacts
+.hypothesis/
+
+# coverage artifact
+.coverage
+
+# kiro config
+.config.kiro
+/temp/
+/temp/

grt_lexical_graph-3.17.2.dev0/PKG-INFO

@@ -0,0 +1,190 @@
+Metadata-Version: 2.4
+Name: grt-lexical-graph
+Version: 3.17.2.dev0
+Summary: AWS GraphRAG Toolkit, lexical graph
+License-Expression: Apache-2.0
+Requires-Python: >=3.10
+Requires-Dist: anthropic-bedrock==0.8.0
+Requires-Dist: boto3>=1.40.61
+Requires-Dist: botocore>=1.40.61
+Requires-Dist: json2xml==5.2.0
+Requires-Dist: llama-index-core==0.14.17
+Requires-Dist: llama-index-embeddings-bedrock==0.8.0
+Requires-Dist: llama-index-llms-anthropic==0.11.0
+Requires-Dist: llama-index-llms-bedrock-converse==0.14.0
+Requires-Dist: lru-dict==1.3.0
+Requires-Dist: pipe==2.2
+Requires-Dist: python-dotenv==1.1.1
+Requires-Dist: smart-open==7.1.0
+Requires-Dist: spacy==3.8.7
+Requires-Dist: tfidf-matcher==0.3.0
+Description-Content-Type: text/markdown
+
+## Lexical Graph
+
+The lexical-graph package provides a framework for automating the construction of a [hierarchical lexical graph](../docs/lexical-graph/graph-model.md) from unstructured data, and composing question-answering strategies that query this graph when answering user questions.
+
+### Features
+
+  - Built-in graph store support for [Amazon Neptune Analytics](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/what-is-neptune-analytics.html), [Amazon Neptune Database](https://docs.aws.amazon.com/neptune/latest/userguide/intro.html), and [Neo4j](https://neo4j.com/docs/).
+  - Built-in vector store support for Neptune Analytics, [Amazon OpenSearch Serverless](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless.html), [Amazon S3 Vectors](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html) and Postgres with the pgvector extension.
+  - Built-in support for foundation models (LLMs and embedding models) on [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/).
+  - Easily extended to support additional graph and vector stores and model backends.
+  - [Multi-tenancy](../docs/lexical-graph/multi-tenancy.md) – multiple separate lexical graphs in the same underlying graph and vector stores.
+  - Continuous ingest and [batch extraction](../docs/lexical-graph/batch-extraction.md) (using [Bedrock batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html)) modes.
+  - [Versioned updates](../docs/lexical-graph/versioned-updates.md) for updating source documents and querying the state of the graph and vector stores at a point in time.
+  - Quickstart [AWS CloudFormation templates](../examples/lexical-graph/cloudformation-templates/) for Neptune Database, OpenSearch Serverless, and Amazon Aurora Postgres.
+
+## Installation
+
+The lexical-graph requires Python 3.10 or greater and [pip](http://www.pip-installer.org/en/latest/).
+
+Install from the latest release tag:
+
+```
+$ pip install https://github.com/awslabs/graphrag-toolkit/archive/refs/tags/v3.16.2.zip#subdirectory=lexical-graph
+```
+
+Or install from the `main` branch to get the latest changes:
+
+```
+$ pip install https://github.com/awslabs/graphrag-toolkit/archive/refs/heads/main.zip#subdirectory=lexical-graph
+```
+
+If you're running on AWS, you must run your application in an AWS region containing the Amazon Bedrock foundation models used by the lexical graph (see the [configuration](../docs/lexical-graph/configuration.md#graphragconfig) section in the documentation for details on the default models used), and must [enable access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) to these models before running any part of the solution.
+
+### Additional dependencies
+
+You will need to install additional dependencies for specific graph and vector store backends:
+
+#### Amazon OpenSearch Serverless
+
+```bash
+$ pip install opensearch-py llama-index-vector-stores-opensearch
+```
+
+#### Postgres with pgvector
+
+```bash
+$ pip install psycopg2-binary pgvector
+```
+
+#### Neo4j
+
+``` bash
+$ pip install neo4j
+```
+
+### Connection strings
+
+Pass a connection string to `GraphStoreFactory.for_graph_store()` or `VectorStoreFactory.for_vector_store()` to select a backend:
+
+| Store | Connection string |
+| --- | --- |
+| Neptune Analytics (graph) | `neptune-graph://<graph-id>` |
+| Neptune Database (graph) | `neptune-db://<hostname>` or any hostname ending `.neptune.amazonaws.com` |
+| Neo4j (graph) | `bolt://`, `bolt+ssc://`, `bolt+s://`, `neo4j://`, `neo4j+ssc://`, or `neo4j+s://` URLs |
+| OpenSearch Serverless (vector) | `aoss://<url>` |
+| Neptune Analytics (vector) | `neptune-graph://<graph-id>` |
+| pgvector (vector) | constructed via `PGVectorIndexFactory` |
+| S3 Vectors (vector) | constructed via `S3VectorIndexFactory` |
+| Dummy / no-op | `None` or any unrecognised string — falls back to `DummyGraphStore` / `DummyVectorIndex` |
+
+## Example of use
+
+### Indexing
+
+```python
+from graphrag_toolkit.lexical_graph import LexicalGraphIndex
+from graphrag_toolkit.lexical_graph.storage import GraphStoreFactory
+from graphrag_toolkit.lexical_graph.storage import VectorStoreFactory
+
+# requires pip install llama-index-readers-web
+from llama_index.readers.web import SimpleWebPageReader
+
+def run_extract_and_build():
+
+    with (
+        GraphStoreFactory.for_graph_store(
+            'neptune-db://my-graph.cluster-abcdefghijkl.us-east-1.neptune.amazonaws.com'
+        ) as graph_store,
+        VectorStoreFactory.for_vector_store(
+            'aoss://https://abcdefghijkl.us-east-1.aoss.amazonaws.com'
+        ) as vector_store
+    ):
+
+        graph_index = LexicalGraphIndex(
+            graph_store,
+            vector_store
+        )
+
+        doc_urls = [
+            'https://docs.aws.amazon.com/neptune/latest/userguide/intro.html',
+            'https://docs.aws.amazon.com/neptune-analytics/latest/userguide/what-is-neptune-analytics.html',
+            'https://docs.aws.amazon.com/neptune-analytics/latest/userguide/neptune-analytics-features.html',
+            'https://docs.aws.amazon.com/neptune-analytics/latest/userguide/neptune-analytics-vs-neptune-database.html'
+        ]
+
+        docs = SimpleWebPageReader(
+            html_to_text=True,
+            metadata_fn=lambda url:{'url': url}
+        ).load_data(doc_urls)
+
+        graph_index.extract_and_build(docs, show_progress=True)
+
+if __name__ == '__main__':
+    run_extract_and_build()
+```
+
+### Querying
+
+```python
+from graphrag_toolkit.lexical_graph import LexicalGraphQueryEngine
+from graphrag_toolkit.lexical_graph.storage import GraphStoreFactory
+from graphrag_toolkit.lexical_graph.storage import VectorStoreFactory
+
+def run_query():
+
+    with (
+        GraphStoreFactory.for_graph_store(
+            'neptune-db://my-graph.cluster-abcdefghijkl.us-east-1.neptune.amazonaws.com'
+        ) as graph_store,
+        VectorStoreFactory.for_vector_store(
+            'aoss://https://abcdefghijkl.us-east-1.aoss.amazonaws.com'
+        ) as vector_store
+    ):
+
+        query_engine = LexicalGraphQueryEngine.for_traversal_based_search(
+            graph_store,
+            vector_store
+        )
+
+        response = query_engine.query('''What are the differences between Neptune Database
+                                         and Neptune Analytics?''')
+
+        print(response.response)
+
+if __name__ == '__main__':
+    run_query()
+```
+
+## Documentation
+
+  - [Overview](../docs/lexical-graph/overview.md)
+  - [Graph Model](../docs/lexical-graph/graph-model.md)
+  - [Storage Model](../docs/lexical-graph/storage-model.md)
+  - [Indexing](../docs/lexical-graph/indexing.md)
+    - [Batch Extraction](../docs/lexical-graph/batch-extraction.md)
+    - [Configuring Batch Extraction](../docs/lexical-graph/configuring-batch-extraction.md)
+    - [Versioned Updates](../docs/lexical-graph/versioned-updates.md)
+  - [Querying](../docs/lexical-graph/querying.md)
+    - [Traversal-Based Search](../docs/lexical-graph/traversal-based-search.md)
+    - [Traversal-Based Search Configuration](../docs/lexical-graph/traversal-based-search-configuration.md)
+  - [Configuration](../docs/lexical-graph/configuration.md)
+  - [Security](../docs/lexical-graph/security.md)
+  - [FAQ](../docs/lexical-graph/faq.md)
+
+
+## License
+
+This project is licensed under the Apache-2.0 License.

grt_lexical_graph-3.17.2.dev0/README.md

@@ -0,0 +1,168 @@
+## Lexical Graph
+
+The lexical-graph package provides a framework for automating the construction of a [hierarchical lexical graph](../docs/lexical-graph/graph-model.md) from unstructured data, and composing question-answering strategies that query this graph when answering user questions.
+
+### Features
+
+  - Built-in graph store support for [Amazon Neptune Analytics](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/what-is-neptune-analytics.html), [Amazon Neptune Database](https://docs.aws.amazon.com/neptune/latest/userguide/intro.html), and [Neo4j](https://neo4j.com/docs/).
+  - Built-in vector store support for Neptune Analytics, [Amazon OpenSearch Serverless](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless.html), [Amazon S3 Vectors](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html) and Postgres with the pgvector extension.
+  - Built-in support for foundation models (LLMs and embedding models) on [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/).
+  - Easily extended to support additional graph and vector stores and model backends.
+  - [Multi-tenancy](../docs/lexical-graph/multi-tenancy.md) – multiple separate lexical graphs in the same underlying graph and vector stores.
+  - Continuous ingest and [batch extraction](../docs/lexical-graph/batch-extraction.md) (using [Bedrock batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html)) modes.
+  - [Versioned updates](../docs/lexical-graph/versioned-updates.md) for updating source documents and querying the state of the graph and vector stores at a point in time.
+  - Quickstart [AWS CloudFormation templates](../examples/lexical-graph/cloudformation-templates/) for Neptune Database, OpenSearch Serverless, and Amazon Aurora Postgres.
+
+## Installation
+
+The lexical-graph requires Python 3.10 or greater and [pip](http://www.pip-installer.org/en/latest/).
+
+Install from the latest release tag:
+
+```
+$ pip install https://github.com/awslabs/graphrag-toolkit/archive/refs/tags/v3.16.2.zip#subdirectory=lexical-graph
+```
+
+Or install from the `main` branch to get the latest changes:
+
+```
+$ pip install https://github.com/awslabs/graphrag-toolkit/archive/refs/heads/main.zip#subdirectory=lexical-graph
+```
+
+If you're running on AWS, you must run your application in an AWS region containing the Amazon Bedrock foundation models used by the lexical graph (see the [configuration](../docs/lexical-graph/configuration.md#graphragconfig) section in the documentation for details on the default models used), and must [enable access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) to these models before running any part of the solution.
+
+### Additional dependencies
+
+You will need to install additional dependencies for specific graph and vector store backends:
+
+#### Amazon OpenSearch Serverless
+
+```bash
+$ pip install opensearch-py llama-index-vector-stores-opensearch
+```
+
+#### Postgres with pgvector
+
+```bash
+$ pip install psycopg2-binary pgvector
+```
+
+#### Neo4j
+
+``` bash
+$ pip install neo4j
+```
+
+### Connection strings
+
+Pass a connection string to `GraphStoreFactory.for_graph_store()` or `VectorStoreFactory.for_vector_store()` to select a backend:
+
+| Store | Connection string |
+| --- | --- |
+| Neptune Analytics (graph) | `neptune-graph://<graph-id>` |
+| Neptune Database (graph) | `neptune-db://<hostname>` or any hostname ending `.neptune.amazonaws.com` |
+| Neo4j (graph) | `bolt://`, `bolt+ssc://`, `bolt+s://`, `neo4j://`, `neo4j+ssc://`, or `neo4j+s://` URLs |
+| OpenSearch Serverless (vector) | `aoss://<url>` |
+| Neptune Analytics (vector) | `neptune-graph://<graph-id>` |
+| pgvector (vector) | constructed via `PGVectorIndexFactory` |
+| S3 Vectors (vector) | constructed via `S3VectorIndexFactory` |
+| Dummy / no-op | `None` or any unrecognised string — falls back to `DummyGraphStore` / `DummyVectorIndex` |
+
+## Example of use
+
+### Indexing
+
+```python
+from graphrag_toolkit.lexical_graph import LexicalGraphIndex
+from graphrag_toolkit.lexical_graph.storage import GraphStoreFactory
+from graphrag_toolkit.lexical_graph.storage import VectorStoreFactory
+
+# requires pip install llama-index-readers-web
+from llama_index.readers.web import SimpleWebPageReader
+
+def run_extract_and_build():
+
+    with (
+        GraphStoreFactory.for_graph_store(
+            'neptune-db://my-graph.cluster-abcdefghijkl.us-east-1.neptune.amazonaws.com'
+        ) as graph_store,
+        VectorStoreFactory.for_vector_store(
+            'aoss://https://abcdefghijkl.us-east-1.aoss.amazonaws.com'
+        ) as vector_store
+    ):
+
+        graph_index = LexicalGraphIndex(
+            graph_store,
+            vector_store
+        )
+
+        doc_urls = [
+            'https://docs.aws.amazon.com/neptune/latest/userguide/intro.html',
+            'https://docs.aws.amazon.com/neptune-analytics/latest/userguide/what-is-neptune-analytics.html',
+            'https://docs.aws.amazon.com/neptune-analytics/latest/userguide/neptune-analytics-features.html',
+            'https://docs.aws.amazon.com/neptune-analytics/latest/userguide/neptune-analytics-vs-neptune-database.html'
+        ]
+
+        docs = SimpleWebPageReader(
+            html_to_text=True,
+            metadata_fn=lambda url:{'url': url}
+        ).load_data(doc_urls)
+
+        graph_index.extract_and_build(docs, show_progress=True)
+
+if __name__ == '__main__':
+    run_extract_and_build()
+```
+
+### Querying
+
+```python
+from graphrag_toolkit.lexical_graph import LexicalGraphQueryEngine
+from graphrag_toolkit.lexical_graph.storage import GraphStoreFactory
+from graphrag_toolkit.lexical_graph.storage import VectorStoreFactory
+
+def run_query():
+
+    with (
+        GraphStoreFactory.for_graph_store(
+            'neptune-db://my-graph.cluster-abcdefghijkl.us-east-1.neptune.amazonaws.com'
+        ) as graph_store,
+        VectorStoreFactory.for_vector_store(
+            'aoss://https://abcdefghijkl.us-east-1.aoss.amazonaws.com'
+        ) as vector_store
+    ):
+
+        query_engine = LexicalGraphQueryEngine.for_traversal_based_search(
+            graph_store,
+            vector_store
+        )
+
+        response = query_engine.query('''What are the differences between Neptune Database
+                                         and Neptune Analytics?''')
+
+        print(response.response)
+
+if __name__ == '__main__':
+    run_query()
+```
+
+## Documentation
+
+  - [Overview](../docs/lexical-graph/overview.md)
+  - [Graph Model](../docs/lexical-graph/graph-model.md)
+  - [Storage Model](../docs/lexical-graph/storage-model.md)
+  - [Indexing](../docs/lexical-graph/indexing.md)
+    - [Batch Extraction](../docs/lexical-graph/batch-extraction.md)
+    - [Configuring Batch Extraction](../docs/lexical-graph/configuring-batch-extraction.md)
+    - [Versioned Updates](../docs/lexical-graph/versioned-updates.md)
+  - [Querying](../docs/lexical-graph/querying.md)
+    - [Traversal-Based Search](../docs/lexical-graph/traversal-based-search.md)
+    - [Traversal-Based Search Configuration](../docs/lexical-graph/traversal-based-search-configuration.md)
+  - [Configuration](../docs/lexical-graph/configuration.md)
+  - [Security](../docs/lexical-graph/security.md)
+  - [FAQ](../docs/lexical-graph/faq.md)
+
+
+## License
+
+This project is licensed under the Apache-2.0 License.

grt_lexical_graph-3.17.2.dev0/pyproject.toml

@@ -0,0 +1,18 @@
+[build-system]
+requires = ["hatchling", "hatch-requirements-txt"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/graphrag_toolkit"]
+
+[project]
+name = "grt-lexical-graph"
+version = "3.17.2.dev0"
+description = "AWS GraphRAG Toolkit, lexical graph"
+readme = "README.md"
+requires-python = ">=3.10"
+dynamic = ["dependencies"]
+license = "Apache-2.0"
+
+[tool.hatch.metadata.hooks.requirements_txt]
+files = ["src/graphrag_toolkit/lexical_graph/requirements.txt"]

grt_lexical_graph-3.17.2.dev0/src/graphrag_toolkit/lexical_graph/__init__.py

@@ -0,0 +1,56 @@
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+import warnings
+from pydantic.warnings import UnsupportedFieldAttributeWarning
+
+warnings.filterwarnings('ignore', category=UnsupportedFieldAttributeWarning)
+warnings.filterwarnings('ignore', message="Can't initialize NVML")
+
+import asyncio
+import llama_index.core.async_utils
+import logging as l
+
+logger = l.getLogger(__name__)
+
+def _asyncio_run(coro):
+    
+    l.debug('Patching asyncio_run() to run coroutine on existing event loop to support notebooks.')   
+        
+    try:
+        loop = asyncio.get_event_loop()
+
+        return loop.run_until_complete(coro)
+
+    except RuntimeError as e:
+        try:
+            return asyncio.run(coro)
+        except RuntimeError as e:
+            raise RuntimeError(
+                "Detected nested async. Please use nest_asyncio.apply() to allow nested event loops."
+                "Or, use async entry methods like `aquery()`, `aretriever`, `achat`, etc."
+            )
+
+try:
+    loop = asyncio.get_event_loop()
+    if loop.is_running:
+        llama_index.core.async_utils.asyncio_run = _asyncio_run
+except RuntimeError as e:
+    pass  
+
+from .tenant_id import TenantId, DEFAULT_TENANT_ID, DEFAULT_TENANT_NAME, TenantIdType, to_tenant_id
+from .config import GraphRAGConfig as GraphRAGConfig, LLMType, EmbeddingType
+from .errors import ModelError, BatchJobError, IndexError, GraphQueryError, ConfigurationError
+from .logging import set_logging_config, set_advanced_logging_config
+from .lexical_graph_query_engine import LexicalGraphQueryEngine
+from .lexical_graph_index import LexicalGraphIndex
+from .lexical_graph_index import ExtractionConfig, BuildConfig, IndexingConfig
+from .metadata import to_metadata_filter
+from .versioning import add_versioning_info, VersioningConfig, VersioningMode
+from . import utils
+from . import indexing
+from . import retrieval
+from . import storage
+
+
+