gap-finder 0.1.0__tar.gz

gap_finder-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 KI.SH
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gap_finder-0.1.0/PKG-INFO

@@ -0,0 +1,240 @@
+Metadata-Version: 2.4
+Name: gap-finder
+Version: 0.1.0
+Summary: Identify potential knowledge gaps in document collections using language models and topic modeling
+Author: Ben Biesenbach, Josef Küstner
+Author-email: Ben Biesenbach <bbi@informatik.uni-kiel.de>, Josef Küstner <josef.kuestner@informatik.uni-kiel.de>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: License :: OSI Approved :: MIT License
+Requires-Dist: bertopic>=0.17.4
+Requires-Dist: faiss-cpu>=1.14.3
+Requires-Dist: jinja2>=3.1.6
+Requires-Dist: langchain-text-splitters>=1.1.2
+Requires-Dist: openai>=2.46.0
+Requires-Dist: sentence-transformers>=5.6.0
+Requires-Dist: tqdm>=4.68.3
+Requires-Dist: unstructured[pdf]>=0.18.32,<0.23.1 ; python_full_version < '3.11'
+Requires-Dist: unstructured[pdf]>=0.23.1 ; python_full_version >= '3.11'
+Requires-Python: >=3.10
+Project-URL: Homepage, https://github.com/KI-Schleswig-Holstein/gap-finder
+Project-URL: Repository, https://github.com/KI-Schleswig-Holstein/gap-finder
+Project-URL: Issues, https://github.com/KI-Schleswig-Holstein/gap-finder/issues
+Description-Content-Type: text/markdown
+
+# Gap Finder
+
+Gap Finder identifies potential knowledge gaps in a vector store, a collection of PDF documents,
+or text chunks. It generates questions that the source material does not answer,
+groups them into topics with BERTopic, and ranks those topics by how well the
+indexed corpus covers them.
+
+The project was created during a feasibility study at KI.SH in
+Schleswig-Holstein. Its pipeline is based on `GapView`, described in
+[Vanessa Klotzmann's dissertation](https://escholarship.org/uc/item/2zs4w991).
+
+## Installation
+
+Install Gap Finder from PyPI:
+
+```bash
+python -m pip install gap-finder
+```
+
+For development, clone the repository and install the project with
+[uv](https://docs.astral.sh/uv/):
+
+```bash
+git clone https://github.com/KI-Schleswig-Holstein/gap-finder.git
+cd gap-finder
+uv sync
+```
+
+## Quick start
+
+The following example indexes all PDFs below `documents/` and uses an
+OpenAI-compatible chat API for question generation and topic explanations:
+
+```bash
+export GAP_FINDER_API_KEY="your-api-key"
+```
+
+```python
+from gap_finder import GapFinder
+
+gap_finder = GapFinder.from_pdf(
+    folder_path="documents",
+    language_model="your-model-name",
+    base_url="https://your-provider.example/v1",
+)
+
+topics = gap_finder.find()
+print(topics[["title", "avg_similarity", "explanation"]])
+```
+
+For a local OpenAI-compatible server such as Ollama, set its `/v1` endpoint as
+`base_url`. If the server does not require authentication, pass a placeholder
+such as `api_key="-"`.
+
+The default corpus and topic embedding models run locally and may be downloaded
+on first use. Question generation is normally the most computationally
+expensive part of the pipeline. Using an API offloads that work, but embedding
+the corpus and fitting BERTopic still require local CPU or GPU time and memory.
+
+### Use a local language model
+
+Pass a Hugging Face text-generation pipeline to run the language model on your
+machine. A GPU is recommended for all but very small models.
+
+```python
+from transformers import pipeline
+
+from gap_finder import GapFinder
+
+language_model = pipeline(
+    "text-generation",
+    model="Qwen/Qwen3-0.6B",
+    device="cuda",
+    max_new_tokens=1024,
+)
+
+gap_finder = GapFinder.from_pdf(
+    folder_path="documents",
+    language_model=language_model,
+    question_generation_batch_size=8,
+)
+topics = gap_finder.find()
+```
+
+### Generate or provide questions separately
+
+Questions can be generated before the rest of the pipeline. Because
+`create_questions()` stores its result on the instance, the following call to
+`find()` reuses them instead of generating them again:
+
+```python
+questions = gap_finder.create_questions()
+# Inspect or persist questions here.
+topics = gap_finder.find()
+```
+
+To skip automatic question generation, provide existing questions when the
+instance is created or set them later:
+
+```python
+gap_finder = GapFinder.from_pdf(
+    folder_path="documents",
+    language_model="your-model-name",
+    base_url="https://your-provider.example/v1",
+    questions=[
+        "Which populations are not represented in the study?",
+        "How does the method behave outside the tested conditions?",
+    ],
+)
+
+# Equivalent after construction:
+gap_finder.set_questions(["Which limitations remain unexamined?"])
+```
+
+A questions DataFrame must contain a `question` column. Additional columns are
+preserved as the pipeline adds topic and ranking information.
+
+## Configuration
+
+Use `GapFinder.from_pdf()` for a folder of PDFs, `GapFinder.from_chunks()` for
+pre-split text, or construct `GapFinder` directly with a populated vector store.
+
+## Vector stores
+
+Two vector-store implementations are included:
+
+- `NumpyVectorStore` is the default and performs exact cosine-similarity search
+  with NumPy.
+- `FaissVectorStore` performs exact cosine-similarity search with a FAISS
+  `IndexFlatIP` index.
+
+Select FAISS while building the index:
+
+```python
+from gap_finder import GapFinder
+from gap_finder.vector_store import FaissVectorStore
+
+gap_finder = GapFinder.from_pdf(
+    folder_path="documents",
+    vector_store_cls=FaissVectorStore,
+    language_model="your-model-name",
+    base_url="https://your-provider.example/v1",
+)
+```
+
+Custom backends can subclass `VectorStore`.
+
+## Pipeline
+
+`find()` runs the following steps:
+
+1. Load chunks and embeddings from the vector store.
+2. Reuse configured questions or generate questions with the language model.
+3. Group questions into topics with BERTopic.
+4. Compare questions with the corpus and aggregate coverage scores per topic.
+5. Add titles and explanations to the lowest-coverage topics.
+6. Score those topics for domain relevance when `domain_description` is set.
+7. Create a Markdown report section for each explained topic.
+
+Lower coverage means the source corpus covers a topic less well and therefore
+indicates a stronger potential knowledge gap.
+
+## Results
+
+`find()` returns the final topics DataFrame and stores the same object in
+`gap_finder.topics`. Its columns are:
+
+| Column | Description |
+| --- | --- |
+| `id` | BERTopic topic identifier; `-1` represents outliers. |
+| `topic_words` | Representative `(word, score)` pairs. |
+| `question_count` | Number of questions assigned to the topic. |
+| `avg_similarity` | Mean question-to-corpus similarity and primary coverage score. |
+| `std_similarity` | Standard deviation of question similarity. |
+| `min_similarity`, `max_similarity` | Similarity range within the topic. |
+| `title`, `explanation` | LLM-generated description for the configured lowest-coverage topics. |
+| `relevance_score`, `relevance_reason` | Optional domain-relevance result, present when `domain_description` is set. |
+| `report` | Markdown summary for an explained topic. |
+
+After `find()`, `gap_finder.questions` contains the question records enriched
+with `topic_id`, an optional `topic_prob`, an `embedding`, and a
+`similarity_score`. Automatically generated questions also contain `id` and
+`chunk_id`; caller-provided columns are retained.
+
+`gap_finder.topic_model` is the fitted `BERTopic` object, not a table. Use its
+standard API for deeper inspection, for example:
+
+```python
+topic_info = gap_finder.topic_model.get_topic_info()
+```
+
+## Development
+
+Install the project and development dependencies with:
+
+```bash
+uv sync
+```
+
+Run the test suite:
+
+```bash
+uv run pytest
+```
+
+The repository also has opt-in integration tests for real embedding models,
+BERTopic, local language models, and OpenAI-compatible APIs. See the
+[integration test guide](test/integration/README.md) for configuration and
+commands. These tests may download models and can require a GPU or a configured
+API endpoint.
+
+To contribute, open an issue/ticket or submit a pull/merge request.
+
+## License
+
+Gap Finder is available under the [MIT License](LICENSE).

gap_finder-0.1.0/README.md

@@ -0,0 +1,216 @@
+# Gap Finder
+
+Gap Finder identifies potential knowledge gaps in a vector store, a collection of PDF documents,
+or text chunks. It generates questions that the source material does not answer,
+groups them into topics with BERTopic, and ranks those topics by how well the
+indexed corpus covers them.
+
+The project was created during a feasibility study at KI.SH in
+Schleswig-Holstein. Its pipeline is based on `GapView`, described in
+[Vanessa Klotzmann's dissertation](https://escholarship.org/uc/item/2zs4w991).
+
+## Installation
+
+Install Gap Finder from PyPI:
+
+```bash
+python -m pip install gap-finder
+```
+
+For development, clone the repository and install the project with
+[uv](https://docs.astral.sh/uv/):
+
+```bash
+git clone https://github.com/KI-Schleswig-Holstein/gap-finder.git
+cd gap-finder
+uv sync
+```
+
+## Quick start
+
+The following example indexes all PDFs below `documents/` and uses an
+OpenAI-compatible chat API for question generation and topic explanations:
+
+```bash
+export GAP_FINDER_API_KEY="your-api-key"
+```
+
+```python
+from gap_finder import GapFinder
+
+gap_finder = GapFinder.from_pdf(
+    folder_path="documents",
+    language_model="your-model-name",
+    base_url="https://your-provider.example/v1",
+)
+
+topics = gap_finder.find()
+print(topics[["title", "avg_similarity", "explanation"]])
+```
+
+For a local OpenAI-compatible server such as Ollama, set its `/v1` endpoint as
+`base_url`. If the server does not require authentication, pass a placeholder
+such as `api_key="-"`.
+
+The default corpus and topic embedding models run locally and may be downloaded
+on first use. Question generation is normally the most computationally
+expensive part of the pipeline. Using an API offloads that work, but embedding
+the corpus and fitting BERTopic still require local CPU or GPU time and memory.
+
+### Use a local language model
+
+Pass a Hugging Face text-generation pipeline to run the language model on your
+machine. A GPU is recommended for all but very small models.
+
+```python
+from transformers import pipeline
+
+from gap_finder import GapFinder
+
+language_model = pipeline(
+    "text-generation",
+    model="Qwen/Qwen3-0.6B",
+    device="cuda",
+    max_new_tokens=1024,
+)
+
+gap_finder = GapFinder.from_pdf(
+    folder_path="documents",
+    language_model=language_model,
+    question_generation_batch_size=8,
+)
+topics = gap_finder.find()
+```
+
+### Generate or provide questions separately
+
+Questions can be generated before the rest of the pipeline. Because
+`create_questions()` stores its result on the instance, the following call to
+`find()` reuses them instead of generating them again:
+
+```python
+questions = gap_finder.create_questions()
+# Inspect or persist questions here.
+topics = gap_finder.find()
+```
+
+To skip automatic question generation, provide existing questions when the
+instance is created or set them later:
+
+```python
+gap_finder = GapFinder.from_pdf(
+    folder_path="documents",
+    language_model="your-model-name",
+    base_url="https://your-provider.example/v1",
+    questions=[
+        "Which populations are not represented in the study?",
+        "How does the method behave outside the tested conditions?",
+    ],
+)
+
+# Equivalent after construction:
+gap_finder.set_questions(["Which limitations remain unexamined?"])
+```
+
+A questions DataFrame must contain a `question` column. Additional columns are
+preserved as the pipeline adds topic and ranking information.
+
+## Configuration
+
+Use `GapFinder.from_pdf()` for a folder of PDFs, `GapFinder.from_chunks()` for
+pre-split text, or construct `GapFinder` directly with a populated vector store.
+
+## Vector stores
+
+Two vector-store implementations are included:
+
+- `NumpyVectorStore` is the default and performs exact cosine-similarity search
+  with NumPy.
+- `FaissVectorStore` performs exact cosine-similarity search with a FAISS
+  `IndexFlatIP` index.
+
+Select FAISS while building the index:
+
+```python
+from gap_finder import GapFinder
+from gap_finder.vector_store import FaissVectorStore
+
+gap_finder = GapFinder.from_pdf(
+    folder_path="documents",
+    vector_store_cls=FaissVectorStore,
+    language_model="your-model-name",
+    base_url="https://your-provider.example/v1",
+)
+```
+
+Custom backends can subclass `VectorStore`.
+
+## Pipeline
+
+`find()` runs the following steps:
+
+1. Load chunks and embeddings from the vector store.
+2. Reuse configured questions or generate questions with the language model.
+3. Group questions into topics with BERTopic.
+4. Compare questions with the corpus and aggregate coverage scores per topic.
+5. Add titles and explanations to the lowest-coverage topics.
+6. Score those topics for domain relevance when `domain_description` is set.
+7. Create a Markdown report section for each explained topic.
+
+Lower coverage means the source corpus covers a topic less well and therefore
+indicates a stronger potential knowledge gap.
+
+## Results
+
+`find()` returns the final topics DataFrame and stores the same object in
+`gap_finder.topics`. Its columns are:
+
+| Column | Description |
+| --- | --- |
+| `id` | BERTopic topic identifier; `-1` represents outliers. |
+| `topic_words` | Representative `(word, score)` pairs. |
+| `question_count` | Number of questions assigned to the topic. |
+| `avg_similarity` | Mean question-to-corpus similarity and primary coverage score. |
+| `std_similarity` | Standard deviation of question similarity. |
+| `min_similarity`, `max_similarity` | Similarity range within the topic. |
+| `title`, `explanation` | LLM-generated description for the configured lowest-coverage topics. |
+| `relevance_score`, `relevance_reason` | Optional domain-relevance result, present when `domain_description` is set. |
+| `report` | Markdown summary for an explained topic. |
+
+After `find()`, `gap_finder.questions` contains the question records enriched
+with `topic_id`, an optional `topic_prob`, an `embedding`, and a
+`similarity_score`. Automatically generated questions also contain `id` and
+`chunk_id`; caller-provided columns are retained.
+
+`gap_finder.topic_model` is the fitted `BERTopic` object, not a table. Use its
+standard API for deeper inspection, for example:
+
+```python
+topic_info = gap_finder.topic_model.get_topic_info()
+```
+
+## Development
+
+Install the project and development dependencies with:
+
+```bash
+uv sync
+```
+
+Run the test suite:
+
+```bash
+uv run pytest
+```
+
+The repository also has opt-in integration tests for real embedding models,
+BERTopic, local language models, and OpenAI-compatible APIs. See the
+[integration test guide](test/integration/README.md) for configuration and
+commands. These tests may download models and can require a GPU or a configured
+API endpoint.
+
+To contribute, open an issue/ticket or submit a pull/merge request.
+
+## License
+
+Gap Finder is available under the [MIT License](LICENSE).

gap_finder-0.1.0/pyproject.toml

@@ -0,0 +1,48 @@
+[project]
+name = "gap-finder"
+version = "0.1.0"
+description = "Identify potential knowledge gaps in document collections using language models and topic modeling"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    { name = "Ben Biesenbach", email = "bbi@informatik.uni-kiel.de" },
+    { name = "Josef Küstner", email = "josef.kuestner@informatik.uni-kiel.de" }
+]
+classifiers = [
+    "License :: OSI Approved :: MIT License",
+]
+requires-python = ">=3.10"
+dependencies = [
+    "bertopic>=0.17.4",
+    "faiss-cpu>=1.14.3",
+    "jinja2>=3.1.6",
+    "langchain-text-splitters>=1.1.2",
+    "openai>=2.46.0",
+    "sentence-transformers>=5.6.0",
+    "tqdm>=4.68.3",
+    "unstructured[pdf]>=0.18.32,<0.23.1; python_version < '3.11'",
+    "unstructured[pdf]>=0.23.1; python_version >= '3.11'",
+]
+
+[project.urls]
+Homepage = "https://github.com/KI-Schleswig-Holstein/gap-finder"
+Repository = "https://github.com/KI-Schleswig-Holstein/gap-finder"
+Issues = "https://github.com/KI-Schleswig-Holstein/gap-finder/issues"
+
+[build-system]
+requires = ["uv_build>=0.11.16,<0.12.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "python-dotenv>=1.2.1",
+    "pytest>=9.1.1",
+]
+
+[tool.pytest.ini_options]
+markers = [
+    "vs: vector-store integration tests (pdf -> chunk -> embed -> FaissVectorStore); downloads and runs a real embedding model (slow, needs network)",
+    "llm: real chat-model integration tests; requires GAP_FINDER_RUN_LLM_INTEGRATION=1 and configured local/API models",
+    "embedding: real embedding-model integration tests; requires GAP_FINDER_RUN_EMBEDDING_INTEGRATION=1 and configured local/API models",
+]

gap_finder-0.1.0/src/gap_finder/__init__.py

@@ -0,0 +1,31 @@
+from typing import TYPE_CHECKING
+
+from gap_finder._auth import MissingAPIKeyError
+from gap_finder.model_types import (
+    EmbeddingModelInput,
+    LanguageModelInput,
+    SentenceTransformerLike,
+    TextGenerationPipelineLike,
+    TopicEmbeddingModelInput,
+)
+
+if TYPE_CHECKING:
+    from gap_finder.main import GapFinder
+
+__all__ = [
+    "EmbeddingModelInput",
+    "GapFinder",
+    "LanguageModelInput",
+    "MissingAPIKeyError",
+    "SentenceTransformerLike",
+    "TextGenerationPipelineLike",
+    "TopicEmbeddingModelInput",
+]
+
+
+def __getattr__(name: str):
+    if name == "GapFinder":
+        from gap_finder.main import GapFinder
+
+        return GapFinder
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

gap_finder-0.1.0/src/gap_finder/_auth.py

@@ -0,0 +1,18 @@
+import os
+
+
+class MissingAPIKeyError(RuntimeError):
+    """Raised when no API key was provided or configured."""
+
+
+def resolve_api_key(
+    explicit: str | None = None,
+    env_var: str = "GAP_FINDER_API_KEY",
+) -> str:
+    key = explicit or os.getenv(env_var)
+    if not key:
+        raise MissingAPIKeyError(
+            f"No API key found. Pass api_key=... or set the environment "
+            f"variable {env_var}."
+        )
+    return key

gap_finder-0.1.0/src/gap_finder/embedding_model/__init__.py

@@ -0,0 +1,40 @@
+from importlib import import_module
+from typing import TYPE_CHECKING
+
+from gap_finder.embedding_model.client import OpenAIEmbeddingClient
+
+if TYPE_CHECKING:
+    from gap_finder.embedding_model.embedding_model import EmbeddingModel
+    from gap_finder.embedding_model.factory import create_embedding_model
+    from gap_finder.embedding_model.local_embedding_model import LocalEmbeddingModel
+    from gap_finder.embedding_model.openai_embedding_model import OpenAIEmbeddingModel
+
+__all__ = [
+    "EmbeddingModel",
+    "LocalEmbeddingModel",
+    "OpenAIEmbeddingClient",
+    "OpenAIEmbeddingModel",
+    "create_embedding_model",
+]
+
+
+def __getattr__(name: str):
+    if name == "EmbeddingModel":
+        from gap_finder.embedding_model.embedding_model import EmbeddingModel
+
+        return EmbeddingModel
+    if name == "create_embedding_model":
+        from gap_finder.embedding_model.factory import create_embedding_model
+
+        return create_embedding_model
+    if name == "LocalEmbeddingModel":
+        from gap_finder.embedding_model.local_embedding_model import LocalEmbeddingModel
+
+        return LocalEmbeddingModel
+    if name == "OpenAIEmbeddingModel":
+        from gap_finder.embedding_model.openai_embedding_model import OpenAIEmbeddingModel
+
+        return OpenAIEmbeddingModel
+    if name == "factory":
+        return import_module("gap_finder.embedding_model.factory")
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

gap_finder-0.1.0/src/gap_finder/embedding_model/client.py

@@ -0,0 +1,24 @@
+from openai import OpenAI
+
+from gap_finder._auth import resolve_api_key
+
+
+class OpenAIEmbeddingClient:
+    """An OpenAI client paired with the embedding model it should call."""
+
+    def __init__(
+        self,
+        model_name: str,
+        base_url: str | None = None,
+        api_key: str | None = None,
+        client: OpenAI | None = None,
+    ) -> None:
+        self.model_name = model_name
+        self.client = (
+            client
+            if client is not None
+            else OpenAI(
+                base_url=base_url,
+                api_key=resolve_api_key(api_key),
+            )
+        )

gap_finder-0.1.0/src/gap_finder/embedding_model/embedding_model.py

@@ -0,0 +1,37 @@
+from abc import ABC, abstractmethod
+
+import numpy as np
+
+
+class EmbeddingModel(ABC):
+    """Common interface for local and API-backed text embedding models."""
+
+    @abstractmethod
+    def encode(
+        self,
+        texts: list[str],
+        show_progress_bar: bool = False,
+        batch_size: int = 32,
+    ) -> np.ndarray:
+        """Encode text into an embedding matrix."""
+        raise NotImplementedError
+
+    def embed(self, documents: list[str], verbose: bool = False) -> np.ndarray:
+        """Embed documents using BERTopic-compatible arguments."""
+        return self.encode(
+            documents,
+            show_progress_bar=verbose,
+            batch_size=32,
+        )
+
+    def activate(self) -> None:
+        """Make internally owned resources ready for use."""
+        return
+
+    def deactivate(self) -> None:
+        """Optionally offload internally owned resources between steps."""
+        return
+
+    def cleanup(self) -> None:
+        """Release internally owned resources after the pipeline finishes."""
+        return