logseq-retriever 0.4.1__tar.gz

logseq_retriever-0.4.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 David Ge Liu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

logseq_retriever-0.4.1/PKG-INFO

@@ -0,0 +1,102 @@
+Metadata-Version: 2.4
+Name: logseq-retriever
+Version: 0.4.1
+Summary: Python library for loading and retrieving Logseq documents
+Author-email: DL <v49t9zpqd@mozmail.com>
+License: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: langchain<0.4.0,>=0.3.24
+Requires-Dist: pydantic<3.0,>=2.11
+Requires-Dist: pgvector-template>=0.3.4
+Provides-Extra: scripts
+Requires-Dist: python-dotenv; extra == "scripts"
+Requires-Dist: psycopg2-binary>=2.9.0; extra == "scripts"
+Requires-Dist: langchain-aws>=0.2.0; extra == "scripts"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+Requires-Dist: python-dotenv; extra == "test"
+Requires-Dist: boto3; extra == "test"
+Requires-Dist: boto3-stubs; extra == "test"
+Requires-Dist: langchain-aws>=0.2.0; extra == "test"
+Requires-Dist: psycopg[binary]>=3.0.0; extra == "test"
+Requires-Dist: ty; extra == "test"
+Requires-Dist: ruff; extra == "test"
+Provides-Extra: dist
+Requires-Dist: build>=1.2.2; extra == "dist"
+Requires-Dist: twine>=6.1.0; extra == "dist"
+Dynamic: license-file
+
+# Logseq Retriever
+Python library for loading and retrieving Logseq documents.
+
+---
+## Components
+This section provides an overview of the components provided, listed by type
+
+
+### Retrievers
+Retrievers inject context into a conversation. Works in tandem with a Contextualizer and `Document` Loader.
+- **Input**:
+  - natural-language user-input, usually query-like
+  - (optional) chat history
+- **Output**:
+  - list of `Document`s to provide context for an LLM to answer the user-input
+
+#### Implementations
+- `LogseqJournalDateRangeRetriever`
+  - retrieve Logseq journal `Document`s, intended for queries that require context from a date range
+  - required to set up:
+    - `RetrieverContextualizer`
+    - `LogseqJournalLoader`
+  - examples:
+    - "What did I do over Christmas break 2024?"
+    - "How did I spend the last Independence Day?"
+
+
+### Contextualizers
+Contextualizers serve as the bridge between natural-language input and a downstream component that
+handles fetching of relevant `Document`s.
+- **Input**:
+  - natural-language user-input, usually query-like
+  - (optional) chat history
+- **Output**:
+  - structured downstream query, based on
+
+In this library, an instance of `RetrieverContextualizer` is provided directly to 
+`Retriever`s during the latter's instantiation. To set up the `RetrieverContextualizer`, provide
+`RetrieverContextualizerProps`, which includes:
+- `llm` - this is the backbone of the contextualizer
+- `prompt` - instructions provided to the LLM
+- `output_schema` - (optional) structured schema used to fetch relevant `Document`s
+  - if no schema is provided, a string shall be returned instead
+- other flags and settings
+
+
+### Loaders
+Loaders are one type of component that can fetch relevant `Document`s. Loaders are typically specific to
+a corresponding `Retriever` component.
+- **Input**:
+  - each loader specifies its own schema
+    - the Contextualizer is usually responsible for creating an instance of the query obj to act upon
+- **Output**:
+  - `list[Document]`
+
+#### Implementations
+- `LogseqJournalFilesystemLoader`
+  - loads from the filesystem, where journal files are expected to be present at specified path
+
+
+---
+## Scripts
+
+### PGVector
+
+#### `upload_journal`
+
+usage: `python scripts/upload_journal_to_pgvector.py [-h] [-p PATH] from_date to_date`

logseq_retriever-0.4.1/README.md

@@ -0,0 +1,68 @@
+# Logseq Retriever
+Python library for loading and retrieving Logseq documents.
+
+---
+## Components
+This section provides an overview of the components provided, listed by type
+
+
+### Retrievers
+Retrievers inject context into a conversation. Works in tandem with a Contextualizer and `Document` Loader.
+- **Input**:
+  - natural-language user-input, usually query-like
+  - (optional) chat history
+- **Output**:
+  - list of `Document`s to provide context for an LLM to answer the user-input
+
+#### Implementations
+- `LogseqJournalDateRangeRetriever`
+  - retrieve Logseq journal `Document`s, intended for queries that require context from a date range
+  - required to set up:
+    - `RetrieverContextualizer`
+    - `LogseqJournalLoader`
+  - examples:
+    - "What did I do over Christmas break 2024?"
+    - "How did I spend the last Independence Day?"
+
+
+### Contextualizers
+Contextualizers serve as the bridge between natural-language input and a downstream component that
+handles fetching of relevant `Document`s.
+- **Input**:
+  - natural-language user-input, usually query-like
+  - (optional) chat history
+- **Output**:
+  - structured downstream query, based on
+
+In this library, an instance of `RetrieverContextualizer` is provided directly to 
+`Retriever`s during the latter's instantiation. To set up the `RetrieverContextualizer`, provide
+`RetrieverContextualizerProps`, which includes:
+- `llm` - this is the backbone of the contextualizer
+- `prompt` - instructions provided to the LLM
+- `output_schema` - (optional) structured schema used to fetch relevant `Document`s
+  - if no schema is provided, a string shall be returned instead
+- other flags and settings
+
+
+### Loaders
+Loaders are one type of component that can fetch relevant `Document`s. Loaders are typically specific to
+a corresponding `Retriever` component.
+- **Input**:
+  - each loader specifies its own schema
+    - the Contextualizer is usually responsible for creating an instance of the query obj to act upon
+- **Output**:
+  - `list[Document]`
+
+#### Implementations
+- `LogseqJournalFilesystemLoader`
+  - loads from the filesystem, where journal files are expected to be present at specified path
+
+
+---
+## Scripts
+
+### PGVector
+
+#### `upload_journal`
+
+usage: `python scripts/upload_journal_to_pgvector.py [-h] [-p PATH] from_date to_date`

logseq_retriever-0.4.1/logseq_retriever/loaders/__init__.py

@@ -0,0 +1,15 @@
+from logseq_retriever.loaders.journal_document_metadata import (
+    LogseqJournalDocumentMetadata,
+)
+from logseq_retriever.loaders.journal_filesystem_loader import (
+    LogseqJournalFilesystemLoader,
+)
+from logseq_retriever.loaders.journal_loader_input import LogseqJournalLoaderInput
+from logseq_retriever.loaders.journal_loader import LogseqJournalLoader
+
+__all__ = [
+    "LogseqJournalDocumentMetadata",
+    "LogseqJournalFilesystemLoader",
+    "LogseqJournalLoaderInput",
+    "LogseqJournalLoader",
+]

logseq_retriever-0.4.1/logseq_retriever/loaders/journal_document_metadata.py

@@ -0,0 +1,32 @@
+from typing import Annotated
+
+from pydantic import BaseModel, Field
+
+
+class LogseqJournalDocumentMetadata(BaseModel):
+    """
+    Metadata for a Logseq journal `Document`.
+    """
+
+    journal_date: Annotated[
+        str,
+        Field(
+            description="The date of the journal entry, in YYYY-MM-DD format.",
+            examples=["2023-01-01", "2025-06-09"],
+        ),
+    ]
+
+    journal_tags: Annotated[
+        list[str],
+        Field(
+            description="The tags associated with the journal entry.",
+            examples=[["tag1", "tag2"], ["tag3"]],
+        ),
+    ]
+
+    journal_char_count: Annotated[
+        int,
+        Field(
+            description="The number of characters in the journal entry.",
+        ),
+    ]

logseq_retriever-0.4.1/logseq_retriever/loaders/journal_filesystem_loader.py

@@ -0,0 +1,157 @@
+from datetime import date, datetime
+from logging import getLogger
+from pathlib import Path
+
+from langchain_core.documents import Document
+from logseq_retriever.loaders.journal_loader import LogseqJournalLoader
+from logseq_retriever.loaders.journal_loader_input import LogseqJournalLoaderInput
+from logseq_retriever.loaders.journal_document_metadata import (
+    LogseqJournalDocumentMetadata,
+)
+import os
+
+
+logger = getLogger(__name__)
+
+
+class LogseqJournalFilesystemLoader(LogseqJournalLoader):
+    """
+    Based on input, load a collection of Logseq journal files from the filesystem, as
+    Langchain `Document`s.
+    """
+
+    def __init__(
+        self,
+        logseq_journal_path: str,
+        **kwargs,
+    ):
+        """
+        Initialize the loader with the path to the Logseq journal directory.
+        `logseq_journal_path` should be contain Logesq journal files, such as `2025_03_27.md`
+        """
+        self.logseq_journal_path = logseq_journal_path
+        self._validate_logseq_journal_path()
+
+    def load(  # type: ignore[override]
+        self,
+        input: LogseqJournalLoaderInput,
+    ) -> list[Document]:
+        """
+        Synchronously load the documents from the Logseq journal directory, according to the input.
+        """
+        # Convert dates to datetime objects once
+        if input.start_date > input.end_date:
+            raise ValueError("journal_end_date must be after journal_start_date")
+
+        documents: list[Document] = []
+        # TODO this glob pattern can be improved by analyzing start_date & end_date to provide fewer matches
+        for path in Path(self.logseq_journal_path).glob("*.md"):
+            filename = path.name
+            if self._match_journal(filename, input.start_date, input.end_date):
+                file_path = os.path.join(self.logseq_journal_path, filename)
+                with open(file_path, "r") as file:
+                    content = file.read()
+                    documents.extend(
+                        self.__class__.parse_journal_markdown_file(
+                            content, filename, input.enable_splitting
+                        )
+                    )
+        return documents
+
+    def _validate_logseq_journal_path(self):
+        """
+        Validate the path to the Logseq journal directory. Check that the directory exists.
+        If the directory is empty, or does not contain files with the expected format, log a warning.
+        """
+        # verify that the path exist, and is a directory
+        if not os.path.exists(self.logseq_journal_path):
+            raise ValueError(
+                f"Logseq journal path does not exist: {self.logseq_journal_path}"
+            )
+        if not os.path.isdir(self.logseq_journal_path):
+            raise ValueError(
+                f"Logseq journal path is not a directory: {self.logseq_journal_path}"
+            )
+
+        # verify that the directory contains files with the expected format
+        files = os.listdir(self.logseq_journal_path)
+        if len(files) == 0:
+            logger.warning(
+                f"Logseq journal directory is empty: {self.logseq_journal_path}"
+            )
+        files = Path(self.logseq_journal_path).glob("*.md")
+        if not len(list(files)) > 0:
+            logger.warning(
+                f"No files with .md extension found in {self.logseq_journal_path}"
+            )
+
+    def _match_journal(self, filename: str, start_date: date, end_date: date) -> bool:
+        """
+        Return `True` if journal date is between `start_date` & `end_date`.
+
+        Args:
+            filename: The journal filename (e.g., "2025_03_27.md")
+            start_date: The start date as a datetime object
+            end_date: The end date as a datetime object
+
+        Returns:
+            bool: True if the file's date is within the range, False otherwise
+        """
+        if not filename.endswith(".md"):
+            return False
+
+        try:
+            # Convert filename to date object
+            file_date = datetime.strptime(filename[:-3], "%Y_%m_%d").date()
+            return start_date <= file_date <= end_date
+        except ValueError:
+            # If there's any issue parsing the date from filename, skip this file
+            return False
+
+    @staticmethod
+    def parse_journal_markdown_file(
+        content: str, filename: str, enable_splitting: bool = True
+    ) -> list[Document]:
+        """
+        Generate `Document`s from a file's contents. If necessary, split content into digestible
+        `Document`s, and attach metadata.
+        This function can potentially be augmented by calling Logseq APIs, rather than simply parsing markdown files.
+        """
+        sections = content.split("\n- ") if enable_splitting else [content]
+        docs = []
+        for section in sections:
+            if section_content := section.strip():
+                # Create a Document
+                # first, check that the content length (char count) is acceptable
+                # if longer than acceptable, then call recursively
+                # TODO: use self.p.max_char_count below instead
+                metadata = (
+                    LogseqJournalFilesystemLoader.parse_journal_markdown_file_metadata(
+                        section_content, filename
+                    )
+                )
+                docs.append(
+                    Document(
+                        page_content=section_content, metadata=metadata.model_dump()
+                    )
+                )
+        return docs
+
+    @staticmethod
+    def parse_journal_markdown_file_metadata(
+        section: str, filename: str
+    ) -> LogseqJournalDocumentMetadata:
+        """
+        Parse metadata from a journal markdown file. Return `LogseqMarkdownDocumentMetadata`.
+        This function can potentially be augmented by calling Logseq APIs, rather than simply parsing markdown files.
+        """
+        # Extract date from filename
+        date_str = filename.replace(".md", "").replace("_", "-")
+        char_count = len(section)
+
+        return LogseqJournalDocumentMetadata(
+            journal_date=date_str,
+            # TODO get tags from Document's contents
+            journal_tags=[],
+            journal_char_count=char_count,
+        )

logseq_retriever-0.4.1/logseq_retriever/loaders/journal_loader.py

@@ -0,0 +1,13 @@
+from typing import Any
+
+from langchain_core.document_loaders import BaseLoader
+from langchain_core.documents import Document
+
+
+class LogseqJournalLoader(BaseLoader):
+    """
+    Base class for loading Logseq journal files.
+    """
+
+    def load(self, input: Any) -> list[Document]:  # type: ignore[override]  # ty: ignore[invalid-method-override]
+        raise NotImplementedError("This method should be implemented by subclasses.")

logseq_retriever-0.4.1/logseq_retriever/loaders/journal_loader_input.py

@@ -0,0 +1,108 @@
+from datetime import datetime, date
+from typing import Annotated
+
+from pydantic import (
+    BaseModel,
+    Field,
+    AfterValidator,
+    computed_field,
+    PrivateAttr,
+    model_validator,
+)
+
+
+def _validate_date_format(value: str) -> str:
+    """
+    Normalize date string to ISO format (YYYY-MM-DD).
+    Accepts dates like '2023-3-1' and converts to '2023-03-01'.
+    """
+    try:
+        _parse_date(value)
+        return value
+    except ValueError:
+        raise ValueError(
+            f"Invalid date: '{value}'. Expecting ISO-8601 format: YYYY-MM-DD"
+        )
+
+
+def _parse_date(date_str: str) -> date:
+    """
+    Parse a date string into a datetime.date object.
+    """
+    return datetime.strptime(date_str, "%Y-%m-%d").date()
+
+
+class LogseqJournalLoaderInput(BaseModel):
+    """
+    Input for a Logseq journal `Document` loader, to invoke a load.
+    """
+
+    journal_start_date: Annotated[
+        str,
+        Field(
+            description="The start date of the journal to load, in YYYY-MM-DD format.",
+            examples=["2023-01-01", "2025-06-09"],
+        ),
+        AfterValidator(_validate_date_format),
+    ]
+    journal_end_date: Annotated[
+        str,
+        Field(
+            description="The end date of the journal to load, in YYYY-MM-DD format.",
+            examples=["2023-01-01", "2025-06-09"],
+        ),
+        AfterValidator(_validate_date_format),
+    ]
+    max_char_length: Annotated[
+        int,
+        Field(
+            description="The maximum number of characters to include in a single `Document`.",
+            examples=[8196, 2000],
+            default=1024 * 8,
+        ),
+    ] = 1024 * 8
+    enable_splitting: Annotated[
+        bool,
+        Field(
+            description="Whether to split the journal file into multiple `Document`s.",
+            examples=[True, False],
+            default=True,
+        ),
+    ] = True
+
+    # Private attributes that won't be included in model_dump
+    _start_date: date = PrivateAttr()
+    _end_date: date = PrivateAttr()
+
+    @model_validator(mode="after")
+    def _parse_dates(self) -> "LogseqJournalLoaderInput":
+        """Parse date strings into date objects after validation."""
+        self._start_date = _parse_date(self.journal_start_date)
+        self._end_date = _parse_date(self.journal_end_date)
+        return self
+
+    @computed_field
+    @property
+    def start_date(self) -> date:
+        """Get `journal_start_date` as a date object."""
+        return self._start_date
+
+    @computed_field
+    @property
+    def end_date(self) -> date:
+        """Get `journal_end_date` as a date object."""
+        return self._end_date
+
+
+# debugging only
+if __name__ == "__main__":
+    from pprint import pprint
+
+    pprint(LogseqJournalLoaderInput.model_json_schema())
+
+    example = LogseqJournalLoaderInput(
+        journal_start_date="2023-01-01",
+        journal_end_date="2023-01-02",
+        max_char_length=1024 * 4,
+    )
+    print(example.model_dump())

logseq_retriever-0.4.1/logseq_retriever/models/__init__.py

@@ -0,0 +1,15 @@
+from logseq_retriever.models.journal_pgvector import (
+    JournalDocument,
+    JournalCorpusMetadata,
+    JournalDocumentMetadata,
+    JournalSearchClientConfig,
+    JournalSearchQuery,
+)
+
+__all__ = [
+    "JournalDocument",
+    "JournalCorpusMetadata",
+    "JournalDocumentMetadata",
+    "JournalSearchClientConfig",
+    "JournalSearchQuery",
+]

logseq_retriever-0.4.1/logseq_retriever/models/journal_pgvector.py

@@ -0,0 +1,107 @@
+from typing import Type
+
+from pgvector.sqlalchemy import Vector
+from pydantic import Field
+from sqlalchemy import Column, String
+
+from pgvector_template.core import (
+    BaseDocument,
+    BaseDocumentMetadata,
+    BaseSearchClientConfig,
+)
+from pgvector_template.models.search import (
+    SearchQuery,
+    MetadataFilter,
+)
+
+
+class JournalDocument(BaseDocument):
+    """
+    Each `Corpus` is the entire entry for a given date. A corpus may consist of 1 or more chunks of `Document`s.
+    Each `Corpus` has a set of metadata, and each `Document` chunk has all of those, plus more.
+    """
+
+    __abstract__ = False
+    __tablename__ = "logseq_journal"
+
+    corpus_id = Column(String(len("2025-06-09")), index=True)
+    """Length of ISO date string"""
+    embedding = Column(Vector(1024))
+    """Embedding vector"""
+
+
+class JournalCorpusMetadata(BaseDocumentMetadata):
+    """Metadata schema for Logseq journal corpora. Consist of 1-or-more chunks, called `Document`s."""
+
+    # corpus
+    date_str: str = Field(
+        pattern=r"^\d{4}-\d{2}-\d{2}$",
+        description="Date in ISO format, e.g. `2025-04-20`",
+    )
+
+    # defaults
+    document_type: str = Field(default="logseq_journal")
+    schema_version: str = Field(default="2025-07-10")
+
+
+class JournalDocumentMetadata(JournalCorpusMetadata):
+    """Metadata schema for Logseq journal `Document`s. 1-or-more `Document`s make up a corpus."""
+
+    # chunk/document
+    chunk_len: int = Field()
+    """Length of the content in characters"""
+    word_count: int | None = Field()
+    """Length of the content in words"""
+    references: list[str] = Field(default=[])
+    """List of references to other Logseq documents, or journal dates"""
+    anchor_ids: list[str] = Field(default=[])
+    """Blocks in the document can have UUID anchors, which are referenced elsewhere. This is a list of all present"""
+
+
+class JournalSearchClientConfig(BaseSearchClientConfig):
+    """Configuration for the Logseq journal search client."""
+
+    document_cls: Type[BaseDocument] = JournalDocument
+    """The document type to use for the search client."""
+    document_metadata_cls: Type[BaseDocumentMetadata] = JournalDocumentMetadata
+    """The document metadata type to use for the search client."""
+    # embedding_provider
+
+
+class JournalSearchQuery(SearchQuery):
+    """
+    Standardized search query structure, specifically for searching Logseq `JournalDocument`s.
+    At least 1 search criterion is required (text, keywords, metadata_filters), but multiple are allowed.
+    Types are the same as in `SearchQuery`.
+    Descriptions are customized to better suit Logseq `JournalDocument`'s.
+    """
+
+    text: str | None = None
+    """
+    String to match against using in a semantic search, i.e. using vector distance.
+    Instead of passing in a question, rephrase the question to be a string/phrase matching closer
+    to the content expected to be found.
+    """
+
+    keywords: list[str] = []
+    """
+    List of keywords to **exact-match**.
+    If any keywords are provided, at least 1 keyword must appear in the content,
+    so use only if certain that the word will appear.
+    Do not include keywords that can be covered in metadata_filters, e.g. dates, document type.
+    If you are not certain that a word will appear, try using `text` for a semantic search instead.
+    """
+
+    metadata_filters: list[MetadataFilter] = Field(
+        default=[],
+        json_schema_extra={
+            "metadata_schema": JournalDocumentMetadata.model_json_schema()
+        },
+    )
+    """
+    List of metadata conditions that must be matched.
+    Refer to `metadata_schema` for the expected schema, as it exists in the database.
+    """
+
+    limit: int = Field(20, ge=3)
+    """Maximum number of results to return."""

logseq_retriever-0.4.1/logseq_retriever/retrievers/__init__.py

@@ -0,0 +1,19 @@
+from logseq_retriever.retrievers.contextualizer import (
+    RetrieverContextualizerProps,
+    RetrieverContextualizer,
+)
+from logseq_retriever.retrievers.journal_retriever import LogseqJournalRetriever
+from logseq_retriever.retrievers.journal_date_range_retriever import (
+    LogseqJournalDateRangeRetriever,
+)
+from logseq_retriever.retrievers.pgvector_journal_retriever import (
+    PGVectorJournalRetriever,
+)
+
+__all__ = [
+    "RetrieverContextualizerProps",
+    "RetrieverContextualizer",
+    "LogseqJournalRetriever",
+    "LogseqJournalDateRangeRetriever",
+    "PGVectorJournalRetriever",
+]