wizit-open-rag 0.0.1__tar.gz

wizit_open_rag-0.0.1/PKG-INFO

@@ -0,0 +1,250 @@
+Metadata-Version: 2.3
+Name: wizit-open-rag
+Version: 0.0.1
+Summary: AI-powered document transcription and semantic chunking for RAG pipelines
+Keywords: rag,retrieval-augmented-generation,llm,chunking,transcription,weaviate,bedrock,langchain,langgraph,pdf,semantic-search
+Author: Restebance
+Author-email: Restebance <restebance@gmail.com>
+License: Apache-2.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Dist: boto3>=1.40.23
+Requires-Dist: langchain>=1.2.10
+Requires-Dist: langchain-aws>=1.3.0
+Requires-Dist: langchain-classic>=1.0.7
+Requires-Dist: langchain-community>=0.4.1
+Requires-Dist: langchain-core>=1.2.16
+Requires-Dist: langchain-experimental>=0.4.1
+Requires-Dist: langchain-text-splitters>=1.1.1
+Requires-Dist: langgraph>=1.0.9
+Requires-Dist: pillow>=11.3.0
+Requires-Dist: pymupdf>=1.27.1
+Requires-Dist: anthropic>=0.84.0
+Requires-Dist: psycopg2-binary>=2.9.11
+Requires-Dist: sqlalchemy[asyncio]>=2.0.43
+Requires-Dist: langchain-postgres>=0.0.17
+Requires-Dist: weaviate-client>=4.0.0
+Requires-Dist: langchain-weaviate>=0.0.3
+Requires-Python: >=3.12
+Project-URL: Bug Tracker, https://github.com/Restebance/open_rag/issues
+Project-URL: Repository, https://github.com/Restebance/open_rag
+Description-Content-Type: text/markdown
+
+# open_rag
+
+A Python library for AI-powered document transcription and semantic chunking with RAG (Retrieval-Augmented Generation). It processes PDFs through LLMs (Claude via **AWS Bedrock**), chunks the resulting Markdown semantically, enriches each chunk with surrounding context, and returns ready-to-index `Document` objects for PostgreSQL pgvector.
+
+**Version**: 0.0.1 | **Python**: >=3.12 | **Build**: uv
+
+---
+
+## Features
+
+- PDF-to-Markdown transcription powered by Claude via AWS Bedrock
+- LangGraph-based transcription workflow with configurable retry logic and accuracy thresholds
+- Semantic chunking with 85th-percentile breakpoints (plus recursive and Markdown-header strategies)
+- Per-chunk context enrichment via a dedicated LangGraph workflow — each chunk is wrapped with `<context>` and `<content>` tags
+- Pluggable storage backends: local filesystem or AWS S3
+- Vector indexing into PostgreSQL pgvector via LangChain `PGVectorStore`
+- LangSmith tracing support
+
+---
+
+## Prerequisites
+
+- Python 3.12 or higher
+- [uv](https://docs.astral.sh/uv/) for dependency management
+- AWS credentials configured (standard boto3 credential chain — env vars, `~/.aws/credentials`, or instance profile)
+- PostgreSQL database with the [pgvector](https://github.com/pgvector/pgvector) extension enabled
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install open_rag
+```
+
+For development (clone + install with dev tools):
+
+```bash
+git clone https://github.com/Restebance/open_rag.git
+cd open_rag
+uv sync --group dev
+cp example.env .env
+```
+
+Fill in `.env` with your credentials (see [Environment Variables](#environment-variables) below).
+
+---
+
+## Usage
+
+### Document Transcription
+
+`OpenRagTranscriber` accepts the raw bytes of a single PDF page and returns a `ParsedDocPage` containing the Markdown transcription.
+
+```python
+import asyncio
+from open_rag import OpenRagTranscriber
+
+transcriber = OpenRagTranscriber(
+    langsmith_project_name="my-project",   # required
+    langsmith_api_key="lsv2_...",          # required
+    llm_model_id="global.anthropic.claude-sonnet-4-6",
+    target_language="es-CO",
+    transcription_accuracy_threshold=0.90,
+    max_transcription_retries=2,
+)
+
+with open("page.pdf", "rb") as f:
+    page_bytes = f.read()
+
+result = asyncio.run(transcriber.transcribe_document(page_bytes))
+print(result.page_text)  # Markdown string
+```
+
+### Semantic Chunking with Context
+
+`ChunksManager` takes a pre-loaded Markdown string and returns a list of LangChain `Document` objects, each enriched with a contextual summary.
+
+```python
+import asyncio
+from open_rag import ChunksManager
+
+manager = ChunksManager(
+    langsmith_project_name="my-project",   # required
+    langsmith_api_key="lsv2_...",          # required
+)
+
+with open("document.md") as f:
+    markdown_content = f.read()
+
+docs = asyncio.run(manager.gen_context_chunks(
+    file_key="document.md",
+    file_markdown_content=markdown_content,
+    file_tags={"category": "hr", "department": "onboarding"},
+))
+
+# docs is a List[Document]; index to pgvector as needed
+for doc in docs:
+    print(doc.page_content)
+```
+
+> **Note:** `gen_context_chunks` does not load files from storage — the caller must pass the content as a string. Indexing to pgvector is the caller's responsibility.
+
+---
+
+## Architecture
+
+The codebase follows a clean layered architecture. Dependency direction: `transcription.py / chunks.py → application → domain ↔ infra ← workflows`.
+
+```
+open_rag/                     # installable package (src/open_rag/)
+├── transcription.py          # Public API — OpenRagTranscriber
+├── chunks.py                 # Public API — ChunksManager
+├── domain/                   # Core data models (PageToTranscribe, ParsedDocPage, ParsedDoc)
+├── application/              # Orchestration + abstract interfaces (ABCs)
+├── data/                     # Shared enums and prompt strings
+├── infra/
+│   ├── llms/                 # AWS Bedrock chat (ChatBedrockConverse)
+│   ├── embeddings/           # AWS Bedrock embeddings (BedrockEmbeddings)
+│   ├── persistence/          # Local filesystem, AWS S3, PostgreSQL managers
+│   ├── rag/                  # SemanticChunks, RecursiveChunks, MarkdownHeadersChunks, PGVectorStore, WeaviateEmbeddingsManager
+│   └── secrets/              # AWS Secrets Manager helper
+├── utils/                    # validate_file_name_format
+└── workflows/                # LangGraph state machines (transcription + context)
+tests/                        # pytest suite
+data/                         # Sample / test documents
+example.env
+pyproject.toml
+```
+
+### Key Data Flow
+
+```
+PDF bytes
+  → ParseDocModelService  (PyMuPDF → base64 pages)
+  → TranscriptionWorkflow (LangGraph → Claude via AWS Bedrock → Markdown)
+
+Markdown string + tags
+  → SemanticChunks        (AWS Bedrock embeddings, 85th-percentile breakpoints)
+  → ContextWorkflow       (LangGraph → Claude adds surrounding context per chunk)
+  → List[Document]        (each chunk wrapped in <context> / <content> tags)
+  → Caller indexes to pgvector
+```
+
+---
+
+## Environment Variables
+
+Copy `example.env` to `.env` and fill in the values:
+
+| Variable | Purpose |
+|---|---|
+| `VECTOR_STORE_CONNECTION` | PostgreSQL connection string (pgvector) |
+| `VECTOR_STORE_TABLE` | pgvector table name |
+| `LANGSMITH_API_KEY` | LangSmith API key for tracing |
+| `LANGCHAIN_PROJECT` | LangSmith project name |
+| `LANGSMITH_TRACING` | Enable LangSmith tracing (`true` / `false`) |
+| `SUPABASE_KEY` / `SUPABASE_URL` | Supabase credentials (optional) |
+
+AWS credentials are read from the standard **boto3 credential chain** and are not set in `.env`.
+
+---
+
+## Development
+
+### Running tests
+
+```bash
+# Unit tests (mocked — no AWS credentials required)
+uv run pytest
+
+# Transcription integration test (requires live AWS credentials)
+uv run python src/open_rag/transcription.py
+
+# Chunking integration test (requires live AWS credentials)
+uv run python src/open_rag/chunks.py
+```
+
+### Profiling
+
+```bash
+# CPU profiling
+uv run pyinstrument test.py transcribe <file.pdf> <source_dir> <target_dir>
+
+# Memory profiling
+uv run python -m memray run test.py transcribe <file.pdf> <source_dir> <target_dir>
+```
+
+### Building the package
+
+```bash
+uv build
+```
+
+---
+
+## Gotchas
+
+- `SemanticChunks` calls AWS Bedrock **at construction time** (via `SemanticChunker`) — not just at index time. Make sure credentials are available before instantiating `ChunksManager`.
+- Both `transcribe_document` and `gen_context_chunks` are `async`; wrap them in `asyncio.run(...)` from synchronous code.
+- `OpenRagTranscriber` and `ChunksManager` require `langsmith_project_name` and `langsmith_api_key` as **constructor arguments** — they are not read from environment variables.
+- `ParseDocModelService.parse_document_to_base64_pages` iterates `range(0, page_count)` — pages are zero-indexed (`page_number=0` is the first page).
+- AWS Bedrock cross-region model IDs use the `global.` prefix (e.g. `global.anthropic.claude-sonnet-4-6`).
+
+---
+
+## License
+
+Licensed under the [Apache License 2.0](LICENSE.md).

wizit_open_rag-0.0.1/README.md

@@ -0,0 +1,211 @@
+# open_rag
+
+A Python library for AI-powered document transcription and semantic chunking with RAG (Retrieval-Augmented Generation). It processes PDFs through LLMs (Claude via **AWS Bedrock**), chunks the resulting Markdown semantically, enriches each chunk with surrounding context, and returns ready-to-index `Document` objects for PostgreSQL pgvector.
+
+**Version**: 0.0.1 | **Python**: >=3.12 | **Build**: uv
+
+---
+
+## Features
+
+- PDF-to-Markdown transcription powered by Claude via AWS Bedrock
+- LangGraph-based transcription workflow with configurable retry logic and accuracy thresholds
+- Semantic chunking with 85th-percentile breakpoints (plus recursive and Markdown-header strategies)
+- Per-chunk context enrichment via a dedicated LangGraph workflow — each chunk is wrapped with `<context>` and `<content>` tags
+- Pluggable storage backends: local filesystem or AWS S3
+- Vector indexing into PostgreSQL pgvector via LangChain `PGVectorStore`
+- LangSmith tracing support
+
+---
+
+## Prerequisites
+
+- Python 3.12 or higher
+- [uv](https://docs.astral.sh/uv/) for dependency management
+- AWS credentials configured (standard boto3 credential chain — env vars, `~/.aws/credentials`, or instance profile)
+- PostgreSQL database with the [pgvector](https://github.com/pgvector/pgvector) extension enabled
+
+---
+
+## Installation
+
+Install from PyPI:
+
+```bash
+pip install open_rag
+```
+
+For development (clone + install with dev tools):
+
+```bash
+git clone https://github.com/Restebance/open_rag.git
+cd open_rag
+uv sync --group dev
+cp example.env .env
+```
+
+Fill in `.env` with your credentials (see [Environment Variables](#environment-variables) below).
+
+---
+
+## Usage
+
+### Document Transcription
+
+`OpenRagTranscriber` accepts the raw bytes of a single PDF page and returns a `ParsedDocPage` containing the Markdown transcription.
+
+```python
+import asyncio
+from open_rag import OpenRagTranscriber
+
+transcriber = OpenRagTranscriber(
+    langsmith_project_name="my-project",   # required
+    langsmith_api_key="lsv2_...",          # required
+    llm_model_id="global.anthropic.claude-sonnet-4-6",
+    target_language="es-CO",
+    transcription_accuracy_threshold=0.90,
+    max_transcription_retries=2,
+)
+
+with open("page.pdf", "rb") as f:
+    page_bytes = f.read()
+
+result = asyncio.run(transcriber.transcribe_document(page_bytes))
+print(result.page_text)  # Markdown string
+```
+
+### Semantic Chunking with Context
+
+`ChunksManager` takes a pre-loaded Markdown string and returns a list of LangChain `Document` objects, each enriched with a contextual summary.
+
+```python
+import asyncio
+from open_rag import ChunksManager
+
+manager = ChunksManager(
+    langsmith_project_name="my-project",   # required
+    langsmith_api_key="lsv2_...",          # required
+)
+
+with open("document.md") as f:
+    markdown_content = f.read()
+
+docs = asyncio.run(manager.gen_context_chunks(
+    file_key="document.md",
+    file_markdown_content=markdown_content,
+    file_tags={"category": "hr", "department": "onboarding"},
+))
+
+# docs is a List[Document]; index to pgvector as needed
+for doc in docs:
+    print(doc.page_content)
+```
+
+> **Note:** `gen_context_chunks` does not load files from storage — the caller must pass the content as a string. Indexing to pgvector is the caller's responsibility.
+
+---
+
+## Architecture
+
+The codebase follows a clean layered architecture. Dependency direction: `transcription.py / chunks.py → application → domain ↔ infra ← workflows`.
+
+```
+open_rag/                     # installable package (src/open_rag/)
+├── transcription.py          # Public API — OpenRagTranscriber
+├── chunks.py                 # Public API — ChunksManager
+├── domain/                   # Core data models (PageToTranscribe, ParsedDocPage, ParsedDoc)
+├── application/              # Orchestration + abstract interfaces (ABCs)
+├── data/                     # Shared enums and prompt strings
+├── infra/
+│   ├── llms/                 # AWS Bedrock chat (ChatBedrockConverse)
+│   ├── embeddings/           # AWS Bedrock embeddings (BedrockEmbeddings)
+│   ├── persistence/          # Local filesystem, AWS S3, PostgreSQL managers
+│   ├── rag/                  # SemanticChunks, RecursiveChunks, MarkdownHeadersChunks, PGVectorStore, WeaviateEmbeddingsManager
+│   └── secrets/              # AWS Secrets Manager helper
+├── utils/                    # validate_file_name_format
+└── workflows/                # LangGraph state machines (transcription + context)
+tests/                        # pytest suite
+data/                         # Sample / test documents
+example.env
+pyproject.toml
+```
+
+### Key Data Flow
+
+```
+PDF bytes
+  → ParseDocModelService  (PyMuPDF → base64 pages)
+  → TranscriptionWorkflow (LangGraph → Claude via AWS Bedrock → Markdown)
+
+Markdown string + tags
+  → SemanticChunks        (AWS Bedrock embeddings, 85th-percentile breakpoints)
+  → ContextWorkflow       (LangGraph → Claude adds surrounding context per chunk)
+  → List[Document]        (each chunk wrapped in <context> / <content> tags)
+  → Caller indexes to pgvector
+```
+
+---
+
+## Environment Variables
+
+Copy `example.env` to `.env` and fill in the values:
+
+| Variable | Purpose |
+|---|---|
+| `VECTOR_STORE_CONNECTION` | PostgreSQL connection string (pgvector) |
+| `VECTOR_STORE_TABLE` | pgvector table name |
+| `LANGSMITH_API_KEY` | LangSmith API key for tracing |
+| `LANGCHAIN_PROJECT` | LangSmith project name |
+| `LANGSMITH_TRACING` | Enable LangSmith tracing (`true` / `false`) |
+| `SUPABASE_KEY` / `SUPABASE_URL` | Supabase credentials (optional) |
+
+AWS credentials are read from the standard **boto3 credential chain** and are not set in `.env`.
+
+---
+
+## Development
+
+### Running tests
+
+```bash
+# Unit tests (mocked — no AWS credentials required)
+uv run pytest
+
+# Transcription integration test (requires live AWS credentials)
+uv run python src/open_rag/transcription.py
+
+# Chunking integration test (requires live AWS credentials)
+uv run python src/open_rag/chunks.py
+```
+
+### Profiling
+
+```bash
+# CPU profiling
+uv run pyinstrument test.py transcribe <file.pdf> <source_dir> <target_dir>
+
+# Memory profiling
+uv run python -m memray run test.py transcribe <file.pdf> <source_dir> <target_dir>
+```
+
+### Building the package
+
+```bash
+uv build
+```
+
+---
+
+## Gotchas
+
+- `SemanticChunks` calls AWS Bedrock **at construction time** (via `SemanticChunker`) — not just at index time. Make sure credentials are available before instantiating `ChunksManager`.
+- Both `transcribe_document` and `gen_context_chunks` are `async`; wrap them in `asyncio.run(...)` from synchronous code.
+- `OpenRagTranscriber` and `ChunksManager` require `langsmith_project_name` and `langsmith_api_key` as **constructor arguments** — they are not read from environment variables.
+- `ParseDocModelService.parse_document_to_base64_pages` iterates `range(0, page_count)` — pages are zero-indexed (`page_number=0` is the first page).
+- AWS Bedrock cross-region model IDs use the `global.` prefix (e.g. `global.anthropic.claude-sonnet-4-6`).
+
+---
+
+## License
+
+Licensed under the [Apache License 2.0](LICENSE.md).

wizit_open_rag-0.0.1/pyproject.toml

@@ -0,0 +1,70 @@
+[project]
+name = "wizit_open_rag"
+version = "0.0.1"
+description = "AI-powered document transcription and semantic chunking for RAG pipelines"
+readme = "README.md"
+requires-python = ">=3.12"
+license = { text = "Apache-2.0" }
+authors = [
+    { name = "Restebance", email = "restebance@gmail.com" },
+]
+keywords = [
+    "rag",
+    "retrieval-augmented-generation",
+    "llm",
+    "chunking",
+    "transcription",
+    "weaviate",
+    "bedrock",
+    "langchain",
+    "langgraph",
+    "pdf",
+    "semantic-search",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: Apache Software License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "Typing :: Typed",
+]
+dependencies = [
+    "boto3>=1.40.23",
+    "langchain>=1.2.10",
+    "langchain-aws>=1.3.0",
+    "langchain-classic>=1.0.7",
+    "langchain-community>=0.4.1",
+    "langchain-core>=1.2.16",
+    "langchain-experimental>=0.4.1",
+    "langchain-text-splitters>=1.1.1",
+    "langgraph>=1.0.9",
+    "pillow>=11.3.0",
+    "pymupdf>=1.27.1",
+    "anthropic>=0.84.0",
+    "psycopg2-binary>=2.9.11",
+    "sqlalchemy[asyncio]>=2.0.43",
+    "langchain-postgres>=0.0.17",
+    "weaviate-client>=4.0.0",
+    "langchain-weaviate>=0.0.3",
+]
+
+[project.urls]
+Repository = "https://github.com/Restebance/open_rag"
+"Bug Tracker" = "https://github.com/Restebance/open_rag/issues"
+
+[dependency-groups]
+dev = [
+    "memray>=1.18.0",
+    "pyinstrument>=5.1.1",
+    "python-dotenv>=1.1.1",
+    "pytest>=8.3.5",
+    "pytest-asyncio>=0.25.0",
+]
+
+[build-system]
+requires = ["uv_build>=0.8.15,<0.9.0"]
+build-backend = "uv_build"

wizit_open_rag-0.0.1/src/wizit_open_rag/__init__.py

@@ -0,0 +1,13 @@
+from open_rag.application.interfaces import EmbeddingsManager
+from open_rag.chunks import ChunksManager
+from open_rag.infra.rag.pg_embeddings import PgEmbeddingsManager
+from open_rag.infra.rag.weaviate_embeddings import WeaviateEmbeddingsManager
+from open_rag.transcription import OpenRagTranscriber
+
+__all__ = [
+    "OpenRagTranscriber",
+    "ChunksManager",
+    "EmbeddingsManager",
+    "PgEmbeddingsManager",
+    "WeaviateEmbeddingsManager",
+]

wizit_open_rag-0.0.1/src/wizit_open_rag/application/context_chunk_app.py

@@ -0,0 +1,150 @@
+import asyncio
+import logging
+from typing import Any
+
+from langchain_core.documents import Document
+from langchain_core.messages.human import HumanMessage
+from langsmith import Client, tracing_context
+
+from open_rag.workflows.context_workflow import ContextWorkflow
+
+from .interfaces import (
+    AiApplicationService,
+    RagChunker,
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ContextChunksInDocumentApp:
+    """
+    Service for chunking documents.
+    """
+
+    def __init__(
+        self,
+        ai_application_service: AiApplicationService,
+        rag_chunker: RagChunker,
+        langsmith_api_key: str,
+        langsmith_project_name: str,
+        target_language: str = "es",
+    ):
+        """
+        Initialize the ChunkerService.
+        """
+        self.ai_application_service = ai_application_service
+        self.rag_chunker = rag_chunker
+        self.target_language = target_language
+        self.chat_model = self.ai_application_service.load_chat_model()
+        # TODO
+        self.context_additional_instructions = ""
+        self.metadata_source = "source"
+
+        context_workflow = ContextWorkflow(
+            self.chat_model, self.context_additional_instructions
+        )
+        self.compiled_context_workflow = context_workflow.gen_workflow()
+        self.compiled_context_workflow = self.compiled_context_workflow.compile()
+        # TRACING
+        self.langsmith_project_name = langsmith_project_name
+        self.langsmith_client = Client(api_key=langsmith_api_key)
+
+    async def _retrieve_context_chunk_in_document_with_workflow(
+        self,
+        workflow,
+        markdown_content: str,
+        chunk: Document,
+        chunk_metadata: dict[str, Any] | None = None,
+    ) -> Document:
+        """Retrieve context chunks in document."""
+        try:
+            with tracing_context(
+                enabled=True,
+                project_name=self.langsmith_project_name,
+                client=self.langsmith_client,
+            ):
+                result = await workflow.ainvoke(
+                    {
+                        "messages": [
+                            HumanMessage(
+                                content=[
+                                    {
+                                        "type": "text",
+                                        "text": f"Retrieve a complete context for the following chunk: <chunk>{chunk.page_content}</chunk>,  ensure all content chunks are generated with the same document's language.",
+                                    },
+                                ]
+                            )
+                        ],
+                        "document_content": markdown_content,
+                    },
+                    {
+                        "configurable": {
+                            "transcription_accuracy_threshold": 0.95,
+                            "max_transcription_retries": 2,
+                        }
+                    },
+                )
+                chunk.page_content = f"<context>\n{result['context']}\n</context>\n <content>\n{chunk.page_content}\n</content>"
+                if chunk_metadata is not None:
+                    for key, value in chunk_metadata.items():
+                        chunk.metadata[key] = value
+                return chunk
+        except Exception as e:
+            logger.error(f"Failed to retrieve context chunks in document: {str(e)}")
+            raise
+
+    async def _retrieve_context_chunks_in_document_with_workflow(
+        self,
+        markdown_content: str,
+        chunks: list[Document],
+        chunks_metadata: dict[str, Any] | None = None,
+    ) -> list[Document]:
+        """Retrieve context chunks in document."""
+        try:
+            context_chunks_workflow_invocations = list(
+                map(
+                    lambda chunk: self._retrieve_context_chunk_in_document_with_workflow(
+                        self.compiled_context_workflow,
+                        markdown_content,
+                        chunk,
+                        chunks_metadata,
+                    ),
+                    chunks,
+                )
+            )
+            context_chunks = await asyncio.gather(*context_chunks_workflow_invocations)
+            return context_chunks
+        except Exception as e:
+            logger.error(f"Failed to retrieve context chunks in document: {str(e)}")
+            raise
+
+    async def get_context_chunks_in_document(
+        self, file_key: str, file_markdown_content: str, file_tags: dict | None = None
+    ):
+        """
+        Get the context chunks in a document.
+        """
+        try:
+            langchain_rag_document = Document(
+                id=file_key,
+                page_content=file_markdown_content,
+                metadata={
+                    self.metadata_source: file_key,
+                    "source_url": "",
+                    **(file_tags or {}),
+                },
+            )
+            logger.info(f"Document loaded:{file_key}")
+            chunks = self.rag_chunker.gen_chunks_for_document(langchain_rag_document)
+            breakpoint()
+            logger.info(f"Chunks generated:{len(chunks)}")
+            context_chunks = (
+                await self._retrieve_context_chunks_in_document_with_workflow(
+                    file_markdown_content, chunks, file_tags
+                )
+            )
+            logger.info(f"Context chunks generated:{len(context_chunks)}")
+            return context_chunks
+        except Exception as e:
+            logger.error(f"Error: {str(e)}")
+            raise e