docmesh-kbms 0.1.0__tar.gz

docmesh_kbms-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 kyundae-kim
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

docmesh_kbms-0.1.0/PKG-INFO

@@ -0,0 +1,71 @@
+Metadata-Version: 2.4
+Name: docmesh-kbms
+Version: 0.1.0
+Summary: Document knowledge management system that extracts, chunks, and embeds documents for semantic search with dms-core, Ollama, and Milvus
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiosqlite>=0.22.1
+Requires-Dist: dms-core>=0.11.0
+Requires-Dist: ollama>=0.6.2
+Requires-Dist: psycopg[binary]>=3.2.3
+Requires-Dist: pymilvus[milvus-lite]>=3.0.1
+Requires-Dist: sqlalchemy>=2.0.52
+Dynamic: license-file
+
+# docmesh-kbms
+
+문서 원본과 메타데이터는 `dms-core`가 관리하고, 이 프로젝트는 문서 내용을
+추출·chunking·embedding하여 Milvus에 저장하는 지식 저장 관리 시스템이다.
+
+외부 공개 진입점은 `kbms.KnowledgeManagement` 하나다.
+
+## 계층
+
+- Domain: Python
+- Document management: dms-core
+- ORM/RDB: SQLAlchemy with SQLite or PostgreSQL
+- Vector store: Milvus
+- Embedding provider: Ollama
+
+`KnowledgeManagement`의 문서 작업은 모두 async API이며 `await`로 호출한다.
+`upload_document()`은 dms-core에 원본 문서와 메타데이터를 저장한 뒤,
+텍스트 문서만 추출·chunking하여 Ollama/Milvus로 지식화한다. SQLAlchemy engine과
+dms-core, Ollama, Milvus client의 생성·종료 및 commit/rollback은 host가 맡는다.
+비동기 경로를 사용하려면 `AsyncEngine`(`sqlite+aiosqlite` 또는 async PostgreSQL),
+Ollama `AsyncClient`, pymilvus `AsyncMilvusClient`를 주입한다. 이 경우
+dms-core의 `AsyncDocumentManagementSDK`와 async SQLAlchemy/Milvus 호출을 사용하며,
+동기 클라이언트를 주입하면 기존 호환 경로를 사용한다.
+host 권한 정책은 생성자에 `access_policy`로 주입하고, 각 문서 작업에는 dms-core의
+`AccessContext`를 전달해 사용자·그룹 권한을 적용한다.
+
+## 검증
+
+```bash
+uv run pytest -q
+uv run ruff check kbms test_kbms
+```
+
+SQLite memory/disk와 Milvus local 검사는 항상 실행된다. PostgreSQL, Milvus server,
+Ollama 연결 검사는
+다음 환경변수가 설정된 경우 실행된다.
+
+```bash
+export KBMS_POSTGRES_URL='postgresql+psycopg://docmesh:postgres@postgres:5432/kbms'
+export KBMS_MILVUS_URI='http://milvus:19530'
+export KBMS_OLLAMA_HOST='http://192.168.219.106:11434'
+export KBMS_OLLAMA_EMBEDDING_MODEL='bge-m3'
+export KBMS_OLLAMA_GENERATION_MODEL='llama3.2'
+uv run pytest -q -m integration
+```
+
+실환경 전체 흐름(DMS + MinIO + PostgreSQL + Ollama + Milvus)은 다음 테스트가
+검증한다. MinIO 설정을 생략하면 devcontainer 기본값을 사용한다.
+
+```bash
+export KBMS_MINIO_ENDPOINT='milvus-minio:9000'
+export KBMS_MINIO_ACCESS_KEY='minioadmin'
+export KBMS_MINIO_SECRET_KEY='minioadmin'
+export KBMS_MINIO_BUCKET='kbms-e2e'
+uv run pytest -q -m real_integration test_kbms/test_real_integration.py
+```

docmesh_kbms-0.1.0/README.md

@@ -0,0 +1,56 @@
+# docmesh-kbms
+
+문서 원본과 메타데이터는 `dms-core`가 관리하고, 이 프로젝트는 문서 내용을
+추출·chunking·embedding하여 Milvus에 저장하는 지식 저장 관리 시스템이다.
+
+외부 공개 진입점은 `kbms.KnowledgeManagement` 하나다.
+
+## 계층
+
+- Domain: Python
+- Document management: dms-core
+- ORM/RDB: SQLAlchemy with SQLite or PostgreSQL
+- Vector store: Milvus
+- Embedding provider: Ollama
+
+`KnowledgeManagement`의 문서 작업은 모두 async API이며 `await`로 호출한다.
+`upload_document()`은 dms-core에 원본 문서와 메타데이터를 저장한 뒤,
+텍스트 문서만 추출·chunking하여 Ollama/Milvus로 지식화한다. SQLAlchemy engine과
+dms-core, Ollama, Milvus client의 생성·종료 및 commit/rollback은 host가 맡는다.
+비동기 경로를 사용하려면 `AsyncEngine`(`sqlite+aiosqlite` 또는 async PostgreSQL),
+Ollama `AsyncClient`, pymilvus `AsyncMilvusClient`를 주입한다. 이 경우
+dms-core의 `AsyncDocumentManagementSDK`와 async SQLAlchemy/Milvus 호출을 사용하며,
+동기 클라이언트를 주입하면 기존 호환 경로를 사용한다.
+host 권한 정책은 생성자에 `access_policy`로 주입하고, 각 문서 작업에는 dms-core의
+`AccessContext`를 전달해 사용자·그룹 권한을 적용한다.
+
+## 검증
+
+```bash
+uv run pytest -q
+uv run ruff check kbms test_kbms
+```
+
+SQLite memory/disk와 Milvus local 검사는 항상 실행된다. PostgreSQL, Milvus server,
+Ollama 연결 검사는
+다음 환경변수가 설정된 경우 실행된다.
+
+```bash
+export KBMS_POSTGRES_URL='postgresql+psycopg://docmesh:postgres@postgres:5432/kbms'
+export KBMS_MILVUS_URI='http://milvus:19530'
+export KBMS_OLLAMA_HOST='http://192.168.219.106:11434'
+export KBMS_OLLAMA_EMBEDDING_MODEL='bge-m3'
+export KBMS_OLLAMA_GENERATION_MODEL='llama3.2'
+uv run pytest -q -m integration
+```
+
+실환경 전체 흐름(DMS + MinIO + PostgreSQL + Ollama + Milvus)은 다음 테스트가
+검증한다. MinIO 설정을 생략하면 devcontainer 기본값을 사용한다.
+
+```bash
+export KBMS_MINIO_ENDPOINT='milvus-minio:9000'
+export KBMS_MINIO_ACCESS_KEY='minioadmin'
+export KBMS_MINIO_SECRET_KEY='minioadmin'
+export KBMS_MINIO_BUCKET='kbms-e2e'
+uv run pytest -q -m real_integration test_kbms/test_real_integration.py
+```

docmesh_kbms-0.1.0/docmesh_kbms.egg-info/PKG-INFO

@@ -0,0 +1,71 @@
+Metadata-Version: 2.4
+Name: docmesh-kbms
+Version: 0.1.0
+Summary: Document knowledge management system that extracts, chunks, and embeds documents for semantic search with dms-core, Ollama, and Milvus
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiosqlite>=0.22.1
+Requires-Dist: dms-core>=0.11.0
+Requires-Dist: ollama>=0.6.2
+Requires-Dist: psycopg[binary]>=3.2.3
+Requires-Dist: pymilvus[milvus-lite]>=3.0.1
+Requires-Dist: sqlalchemy>=2.0.52
+Dynamic: license-file
+
+# docmesh-kbms
+
+문서 원본과 메타데이터는 `dms-core`가 관리하고, 이 프로젝트는 문서 내용을
+추출·chunking·embedding하여 Milvus에 저장하는 지식 저장 관리 시스템이다.
+
+외부 공개 진입점은 `kbms.KnowledgeManagement` 하나다.
+
+## 계층
+
+- Domain: Python
+- Document management: dms-core
+- ORM/RDB: SQLAlchemy with SQLite or PostgreSQL
+- Vector store: Milvus
+- Embedding provider: Ollama
+
+`KnowledgeManagement`의 문서 작업은 모두 async API이며 `await`로 호출한다.
+`upload_document()`은 dms-core에 원본 문서와 메타데이터를 저장한 뒤,
+텍스트 문서만 추출·chunking하여 Ollama/Milvus로 지식화한다. SQLAlchemy engine과
+dms-core, Ollama, Milvus client의 생성·종료 및 commit/rollback은 host가 맡는다.
+비동기 경로를 사용하려면 `AsyncEngine`(`sqlite+aiosqlite` 또는 async PostgreSQL),
+Ollama `AsyncClient`, pymilvus `AsyncMilvusClient`를 주입한다. 이 경우
+dms-core의 `AsyncDocumentManagementSDK`와 async SQLAlchemy/Milvus 호출을 사용하며,
+동기 클라이언트를 주입하면 기존 호환 경로를 사용한다.
+host 권한 정책은 생성자에 `access_policy`로 주입하고, 각 문서 작업에는 dms-core의
+`AccessContext`를 전달해 사용자·그룹 권한을 적용한다.
+
+## 검증
+
+```bash
+uv run pytest -q
+uv run ruff check kbms test_kbms
+```
+
+SQLite memory/disk와 Milvus local 검사는 항상 실행된다. PostgreSQL, Milvus server,
+Ollama 연결 검사는
+다음 환경변수가 설정된 경우 실행된다.
+
+```bash
+export KBMS_POSTGRES_URL='postgresql+psycopg://docmesh:postgres@postgres:5432/kbms'
+export KBMS_MILVUS_URI='http://milvus:19530'
+export KBMS_OLLAMA_HOST='http://192.168.219.106:11434'
+export KBMS_OLLAMA_EMBEDDING_MODEL='bge-m3'
+export KBMS_OLLAMA_GENERATION_MODEL='llama3.2'
+uv run pytest -q -m integration
+```
+
+실환경 전체 흐름(DMS + MinIO + PostgreSQL + Ollama + Milvus)은 다음 테스트가
+검증한다. MinIO 설정을 생략하면 devcontainer 기본값을 사용한다.
+
+```bash
+export KBMS_MINIO_ENDPOINT='milvus-minio:9000'
+export KBMS_MINIO_ACCESS_KEY='minioadmin'
+export KBMS_MINIO_SECRET_KEY='minioadmin'
+export KBMS_MINIO_BUCKET='kbms-e2e'
+uv run pytest -q -m real_integration test_kbms/test_real_integration.py
+```

docmesh_kbms-0.1.0/docmesh_kbms.egg-info/SOURCES.txt

@@ -0,0 +1,19 @@
+LICENSE
+README.md
+pyproject.toml
+docmesh_kbms.egg-info/PKG-INFO
+docmesh_kbms.egg-info/SOURCES.txt
+docmesh_kbms.egg-info/dependency_links.txt
+docmesh_kbms.egg-info/requires.txt
+docmesh_kbms.egg-info/top_level.txt
+kbms/__init__.py
+kbms/facade.py
+kbms/dms/__init__.py
+kbms/dms/dms_core.py
+kbms/dms/dto.py
+kbms/dms/embedding.py
+kbms/dms/ingestion.py
+kbms/dms/knowledge.py
+kbms/dms/pipeline.py
+kbms/dms/retrieval.py
+kbms/dms/vectors.py

docmesh_kbms-0.1.0/docmesh_kbms.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

docmesh_kbms-0.1.0/docmesh_kbms.egg-info/requires.txt

@@ -0,0 +1,6 @@
+aiosqlite>=0.22.1
+dms-core>=0.11.0
+ollama>=0.6.2
+psycopg[binary]>=3.2.3
+pymilvus[milvus-lite]>=3.0.1
+sqlalchemy>=2.0.52

docmesh_kbms-0.1.0/docmesh_kbms.egg-info/top_level.txt

@@ -0,0 +1 @@
+kbms

docmesh_kbms-0.1.0/kbms/__init__.py

@@ -0,0 +1,4 @@
+
+from .facade import KnowledgeManagement
+
+__all__ = ["KnowledgeManagement"]

docmesh_kbms-0.1.0/kbms/dms/__init__.py

@@ -0,0 +1,40 @@
+"""Document management and knowledgeization primitives."""
+
+from .dms_core import DmsCoreClient, DmsCoreDocumentManager
+from .dto import KnowledgeDocument, KnowledgeDocumentPage
+from .embedding import OllamaEmbeddingProvider
+from .ingestion import DocumentIndexer, DocumentIngestionService
+from .knowledge import (
+    EmbeddingProvider,
+    KnowledgeIndexer,
+    TextChunk,
+    TextChunker,
+    VectorStore,
+)
+from .pipeline import PipelineBase, PipelineState, PipelineStateRepository
+from .retrieval import KnowledgeSearchService, SearchHit
+from .vectors import MilvusVectorStore
+
+__all__ = [
+
+    "DmsCoreClient",
+    "DmsCoreDocumentManager",
+
+    "DocumentIndexer",
+    "DocumentIngestionService",
+
+    "EmbeddingProvider",
+    "KnowledgeDocument",
+    "KnowledgeDocumentPage",
+    "KnowledgeIndexer",
+    "KnowledgeSearchService",
+    "MilvusVectorStore",
+    "OllamaEmbeddingProvider",
+    "PipelineBase",
+    "PipelineState",
+    "PipelineStateRepository",
+    "SearchHit",
+    "TextChunk",
+    "TextChunker",
+    "VectorStore",
+]

docmesh_kbms-0.1.0/kbms/dms/dms_core.py

@@ -0,0 +1,225 @@
+from __future__ import annotations
+
+import hashlib
+from typing import Protocol
+
+from dms import (
+    AccessContext,
+    DocumentContent,
+    DocumentManagementClient,
+    DocumentPage,
+    DocumentPartition,
+    PublicDocumentMetadata,
+    UploadDocumentRequest,
+    UploadDocumentResult,
+)
+
+
+class DmsCoreClient(Protocol):
+    def list_documents(
+        self,
+        *,
+        partition: DocumentPartition,
+        cursor: str | None = None,
+        limit: int = 100,
+        status: object | None = None,
+        access_context: AccessContext | None = None,
+    ) -> DocumentPage: ...
+
+    def get_document_metadata(
+        self,
+        document_id: str,
+        *,
+        partition: DocumentPartition,
+        access_context: AccessContext | None = None,
+    ) -> PublicDocumentMetadata: ...
+
+    def upload_document(
+        self,
+        request: UploadDocumentRequest,
+        *,
+        partition: DocumentPartition,
+        access_context: AccessContext | None = None,
+    ) -> UploadDocumentResult: ...
+
+    def get_document_content(
+        self,
+        document_id: str,
+        *,
+        partition: DocumentPartition,
+        access_context: AccessContext | None = None,
+    ) -> DocumentContent: ...
+
+    def delete_document(
+        self,
+        document_id: str,
+        *,
+        partition: DocumentPartition,
+        hard_delete: bool = False,
+        access_context: AccessContext | None = None,
+    ) -> object: ...
+
+
+class DmsCoreDocumentManager:
+    """Document-management boundary backed by the dms-core SDK."""
+
+    def __init__(self, client: DmsCoreClient | DocumentManagementClient) -> None:
+        self.client = client
+
+    def upload(
+        self,
+        *,
+        content: bytes,
+        filename: str,
+        content_type: str,
+        partition: DocumentPartition,
+        document_id: str | None = None,
+        created_by: str | None = None,
+        metadata: dict[str, object] | None = None,
+        access_context: AccessContext | None = None,
+    ) -> UploadDocumentResult:
+        if not content:
+            raise ValueError("content must not be empty")
+        if not filename.strip():
+            raise ValueError("filename must not be blank")
+        if not content_type.strip():
+            raise ValueError("content_type must not be blank")
+
+        request = UploadDocumentRequest(
+            content=content,
+            filename=filename,
+            content_type=content_type,
+            document_id=document_id,
+            created_by=created_by,
+            metadata=metadata,
+            checksum=hashlib.sha256(content).hexdigest(),
+        )
+        return self.client.upload_document(
+            request, partition=partition, access_context=access_context
+        )
+
+    def list(
+        self,
+        *,
+        partition: DocumentPartition,
+        cursor: str | None = None,
+        limit: int = 100,
+        access_context: AccessContext | None = None,
+    ) -> DocumentPage:
+        if limit <= 0:
+            raise ValueError("limit must be positive")
+        return self.client.list_documents(
+            partition=partition,
+            cursor=cursor,
+            limit=limit,
+            access_context=access_context,
+        )
+
+    def metadata(
+        self,
+        document_id: str,
+        *,
+        partition: DocumentPartition,
+        access_context: AccessContext | None = None,
+    ) -> PublicDocumentMetadata:
+        if not document_id.strip():
+            raise ValueError("document_id must not be blank")
+        return self.client.get_document_metadata(
+            document_id, partition=partition, access_context=access_context
+        )
+
+    def read(
+        self,
+        document_id: str,
+        *,
+        partition: DocumentPartition,
+        access_context: AccessContext | None = None,
+    ) -> DocumentContent:
+        if not document_id.strip():
+            raise ValueError("document_id must not be blank")
+        return self.client.get_document_content(
+            document_id, partition=partition, access_context=access_context
+        )
+
+    def delete(
+        self,
+        document_id: str,
+        *,
+        partition: DocumentPartition,
+        hard_delete: bool = False,
+        access_context: AccessContext | None = None,
+    ) -> object:
+        if not document_id.strip():
+            raise ValueError("document_id must not be blank")
+        return self.client.delete_document(
+            document_id,
+            partition=partition,
+            hard_delete=hard_delete,
+            access_context=access_context,
+        )
+
+    async def aupload(self, **kwargs: object) -> UploadDocumentResult:
+        self._validate_upload_kwargs(kwargs)
+        request = UploadDocumentRequest(
+            content=kwargs["content"],
+            filename=kwargs["filename"],
+            content_type=kwargs["content_type"],
+            document_id=kwargs.get("document_id"),
+            created_by=kwargs.get("created_by"),
+            metadata=kwargs.get("metadata"),
+            checksum=hashlib.sha256(kwargs["content"]).hexdigest(),
+        )
+        return await self.client.upload_document(
+            request,
+            partition=kwargs["partition"],
+            access_context=kwargs.get("access_context"),
+        )
+
+    async def alist(self, **kwargs: object) -> DocumentPage:
+        limit = kwargs.get("limit", 100)
+        if limit <= 0:
+            raise ValueError("limit must be positive")
+        return await self.client.list_documents(
+            partition=kwargs["partition"],
+            cursor=kwargs.get("cursor"),
+            limit=limit,
+            access_context=kwargs.get("access_context"),
+        )
+
+    async def ametadata(self, document_id: str, **kwargs: object) -> PublicDocumentMetadata:
+        if not document_id.strip():
+            raise ValueError("document_id must not be blank")
+        return await self.client.get_document_metadata(
+            document_id,
+            partition=kwargs["partition"],
+            access_context=kwargs.get("access_context"),
+        )
+
+    async def aread(self, document_id: str, **kwargs: object) -> DocumentContent:
+        if not document_id.strip():
+            raise ValueError("document_id must not be blank")
+        return await self.client.get_document_content(
+            document_id,
+            partition=kwargs["partition"],
+            access_context=kwargs.get("access_context"),
+        )
+
+    async def adelete(self, document_id: str, **kwargs: object) -> object:
+        if not document_id.strip():
+            raise ValueError("document_id must not be blank")
+        return await self.client.delete_document(
+            document_id,
+            partition=kwargs["partition"],
+            hard_delete=kwargs.get("hard_delete", False),
+            access_context=kwargs.get("access_context"),
+        )
+
+    @staticmethod
+    def _validate_upload_kwargs(kwargs: dict[str, object]) -> None:
+        content = kwargs["content"]
+        if not content:
+            raise ValueError("content must not be empty")
+        if not kwargs["filename"].strip():
+            raise ValueError("filename must not be blank")
+        if not kwargs["content_type"].strip():
+            raise ValueError("content_type must not be blank")

docmesh_kbms-0.1.0/kbms/dms/dto.py

@@ -0,0 +1,30 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime
+from typing import Any
+
+
+@dataclass(frozen=True, slots=True)
+class KnowledgeDocument:
+    """KMS document projection that hides dms-core-specific types."""
+
+    document_id: str
+    filename: str
+    content_type: str
+    file_size: int
+    status: str
+    created_at: datetime
+    updated_at: datetime
+    partition_kind: str
+    partition_id: str
+    checksum: str | None
+    created_by: str | None
+    metadata: dict[str, Any]
+
+
+@dataclass(frozen=True, slots=True)
+class KnowledgeDocumentPage:
+    items: list[KnowledgeDocument]
+    next_cursor: str | None
+    has_more: bool

docmesh_kbms-0.1.0/kbms/dms/embedding.py

@@ -0,0 +1,42 @@
+from __future__ import annotations
+
+from collections.abc import Sequence
+from typing import Any
+
+
+class OllamaEmbeddingProvider:
+    """Small adapter around Ollama's synchronous ``embed`` API."""
+
+    def __init__(self, client: Any, model: str) -> None:
+        if not isinstance(model, str) or not model.strip():
+            raise ValueError("model must not be blank")
+        self.client = client
+        self.model = model
+
+    def embed(self, text: str) -> Sequence[float]:
+        if not isinstance(text, str) or not text.strip():
+            raise ValueError("text must not be blank")
+        response = self.client.embed(model=self.model, input=text)
+        embeddings = response.get("embeddings") if isinstance(response, dict) else getattr(response, "embeddings", None)
+        if not embeddings or not isinstance(embeddings, (list, tuple)):
+            raise ValueError("invalid embedding response")
+        vector = embeddings[0] if embeddings and isinstance(embeddings[0], (list, tuple)) else embeddings
+        if not vector:
+            raise ValueError("invalid embedding response")
+        return vector
+
+    async def aembed(self, text: str) -> Sequence[float]:
+        if not isinstance(text, str) or not text.strip():
+            raise ValueError("text must not be blank")
+        response = await self.client.embed(model=self.model, input=text)
+        return self._parse_response(response)
+
+    @staticmethod
+    def _parse_response(response: Any) -> Sequence[float]:
+        embeddings = response.get("embeddings") if isinstance(response, dict) else getattr(response, "embeddings", None)
+        if not embeddings or not isinstance(embeddings, (list, tuple)):
+            raise ValueError("invalid embedding response")
+        vector = embeddings[0] if isinstance(embeddings[0], (list, tuple)) else embeddings
+        if not vector:
+            raise ValueError("invalid embedding response")
+        return vector