talkpipe-vault 0.0.1__tar.gz

talkpipe_vault-0.0.1/.containerignore

@@ -0,0 +1,50 @@
+# Git files (keep .git for setuptools-scm, but ignore other git files)
+.gitignore
+
+# Python cache and virtual environments
+__pycache__
+*.pyc
+*.pyo
+*.pyd
+.Python
+.venv
+venv/
+env/
+ENV/
+
+# Build artifacts
+build/
+dist/
+*.egg-info/
+
+# Test files
+tests/
+.pytest_cache/
+.coverage
+htmlcov/
+*.cover
+
+# Documentation
+docs/
+*.md
+!README.md
+
+# IDE files
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# CI/CD
+.github/
+
+# Container files (don't copy into container)
+Containerfile
+.dockerignore
+.containerignore
+
+# Other
+*.log
+.DS_Store
+

talkpipe_vault-0.0.1/.cursor/commands/comprehensive-code-review.md

@@ -0,0 +1,27 @@
+# Code Review Checklist
+
+## Overview
+Comprehensive checklist for conducting thorough code reviews to ensure quality, security, and maintainability.
+
+## Review Categories
+
+### Functionality
+- [ ] Code does what it's supposed to do
+- [ ] Edge cases are handled
+- [ ] Error handling is appropriate
+- [ ] No obvious bugs or logic errors
+
+### Code Quality
+- [ ] Code is readable and well-structured
+- [ ] Functions are small and focused
+- [ ] Variable names are descriptive
+- [ ] No code duplication
+- [ ] Follows project conventions
+- [ ] Parameters to sources and segments do not document parameters in the code comments.
+- [ ] Parameters to sources and segments use Annotated syntax as documentation.
+
+### Security
+- [ ] No obvious security vulnerabilities
+- [ ] Input validation is present
+- [ ] Sensitive data is handled properly
+- [ ] No hardcoded secrets

talkpipe_vault-0.0.1/.env.example

@@ -0,0 +1,59 @@
+# Example environment file for TalkPipe Vault
+# Copy this to .env and customize for your deployment
+
+# Web server configuration
+VAULT_HOST=0.0.0.0
+VAULT_PORT=8002
+
+# Host folder mounted read-only at /documents in the container (compose only);
+# browse to it in the folder picker when indexing. Must exist on the host.
+# VAULT_DOCUMENTS_DIR=~/Documents
+
+# Path fences enforced by the web app itself (the container image defaults to
+# these values; outside a container both are unset, i.e. unrestricted):
+# - TALKPIPE_VAULT_ROOT: vaults may only be created/opened/deleted under this
+#   single directory.
+# - TALKPIPE_DOCUMENT_ROOTS: folder browsing and indexing are limited to these
+#   directories (separate several with ":", the os.pathsep character).
+# Set a variable to empty to lift that restriction.
+# TALKPIPE_VAULT_ROOT=/app/data
+# TALKPIPE_DOCUMENT_ROOTS=/documents
+
+# LanceDB directory path (same semantics as makevectordatabase --path);
+# Whoosh index is stored under VAULT_PATH/fulltext_vault
+VAULT_PATH=/app/data/vault
+
+# Where web-interface settings (recent vaults, model choices) are persisted
+TALKPIPE_VAULT_HOME=/app/data/vault-home
+
+# Model configuration. Defaults: embeddings use model2vec
+# (minishlab/potion-retrieval-32M), which runs fully in-process — no server
+# or API key; chat uses ollama/mistral-small. Uncomment to override:
+# TALKPIPE_EMBEDDING_SOURCE=model2vec
+# TALKPIPE_EMBEDDING_MODEL=minishlab/potion-retrieval-32M
+# TALKPIPE_CHAT_SOURCE=ollama
+# TALKPIPE_CHAT_MODEL=mistral-small
+
+# Firewalled / air-gapped deployments (model2vec embeddings):
+# model2vec resolves its model through Hugging Face, which needs outbound
+# access on first use. Without it, loading can hang on connection timeouts —
+# even for an already-cached model, because online mode still makes an etag
+# check. For a restricted network, set offline mode so loads use only the
+# local cache (fast failure instead of a hang), and make the model available
+# locally by either:
+#   (a) pointing the embedding model at a downloaded directory (most reliable —
+#       no Hugging Face lookup at all), e.g.
+#       TALKPIPE_EMBEDDING_MODEL=/app/data/models/potion-retrieval-32M
+#   (b) pre-populating the Hugging Face cache on a connected machine / at image
+#       build time.
+# HF_HUB_OFFLINE=1
+# HF_HOME is already set to /app/data/hf-cache in the Containerfile, so
+# downloaded models persist in the data volume; override only to relocate it.
+# HF_HOME=/app/data/hf-cache
+
+# Optional: provider connection settings
+# OPENAI_API_KEY=sk-your-key-here
+# TalkPipe reads the Ollama server URL from TALKPIPE_OLLAMA_SERVER_URL.
+# Use http://host.containers.internal:11434 when Ollama runs on the container host.
+# Use a network hostname or IP when Ollama runs elsewhere, for example:
+TALKPIPE_OLLAMA_SERVER_URL=http://your-ollama-host:11434

talkpipe_vault-0.0.1/.flake8

@@ -0,0 +1,13 @@
+[flake8]
+# black is the source of truth for formatting: E203/W503 conflict with its
+# style, and E501 is left to black (it enforces 88 for code but deliberately
+# does not reflow long strings, comments, or docstrings).
+max-line-length = 88
+extend-ignore = E203, W503, E501
+exclude =
+    .git,
+    __pycache__,
+    build,
+    dist,
+    .venv,
+    .eggs

talkpipe_vault-0.0.1/.github/dependabot.yml

@@ -0,0 +1,48 @@
+version: 2
+updates:
+  # Enable version updates for Python dependencies
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 10
+    reviewers:
+      - "tlbauer2"
+    assignees:
+      - "tlbauer2"
+    commit-message:
+      prefix: "deps"
+      prefix-development: "dev-deps"
+      include: "scope"
+
+  # Enable version updates for Docker
+  - package-ecosystem: "docker"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 5
+    reviewers:
+      - "tlbauer2"
+    assignees:
+      - "tlbauer2"
+    commit-message:
+      prefix: "docker"
+      include: "scope"
+    # Ignore Python 3.14+ until numba supports it
+    ignore:
+      - dependency-name: "python"
+        versions: [">=3.14"]
+
+  # Enable version updates for GitHub Actions
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 5
+    reviewers:
+      - "tlbauer2"
+    assignees:
+      - "tlbauer2"
+    commit-message:
+      prefix: "ci"
+      include: "scope"

talkpipe_vault-0.0.1/.github/workflows/ci-cd.yml

@@ -0,0 +1,238 @@
+name: CI/CD Pipeline
+
+on:
+  push:
+    branches: [ main, develop, master, 'bugfix/**', 'feature/**', 'chore/**' ]
+  pull_request:
+    branches: [ main, develop, master, 'bugfix/**', 'feature/**', 'chore/**' ]
+  release:
+    types: [ published ]
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.11', '3.12', '3.13']
+
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        fetch-depth: 0
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v7
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Cache pip dependencies
+      uses: actions/cache@v6
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.os }}-pip-${{ hashFiles('pyproject.toml') }}
+        restore-keys: |
+          ${{ runner.os }}-pip-
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e .[dev]
+
+    - name: Lint with flake8
+      run: |
+        flake8 src/ tests/ --count --select=E9,F63,F7,F82 --show-source --statistics
+        flake8 src/ tests/ --count --exit-zero --max-complexity=10 --max-line-length=88 --statistics
+
+    - name: Format check with black
+      run: |
+        black --check src/ tests/
+
+    - name: Import sort check with isort
+      run: |
+        isort --check-only src/ tests/
+
+    - name: Type check with mypy
+      run: |
+        mypy src/ || true  # Allow mypy to fail for now
+
+    - name: Run tests with coverage
+      run: |
+        pytest --cov=src --cov-report=xml --cov-report=html
+
+    - name: Upload coverage to Codecov
+      if: matrix.python-version == '3.11'
+      uses: codecov/codecov-action@v7
+      with:
+        files: ./coverage.xml
+        flags: unittests
+
+  security-scan:
+    runs-on: ubuntu-latest
+    needs: test
+    env:
+      SAFETY_API_KEY: ${{ secrets.SAFETY_API_KEY }}
+
+    steps:
+    - uses: actions/checkout@v7
+
+    - name: Set up Python
+      uses: actions/setup-python@v7
+      with:
+        python-version: '3.11'
+
+    - name: Install dependencies for security scan
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e .[dev,security]
+
+    - name: Run Bandit security scan
+      run: |
+        bandit -r src/ -f json -o bandit-report.json || true
+        bandit -r src/ -f txt
+
+    - name: Run Safety dependency scan
+      run: |
+        safety scan --save-as json safety-report.json || true
+        safety scan
+
+    - name: Upload security scan results
+      uses: actions/upload-artifact@v7
+      if: always()
+      with:
+        name: security-reports
+        path: |
+          bandit-report.json
+          safety-report.json
+
+  build-container:
+    runs-on: ubuntu-latest
+    needs: [test, security-scan]
+    permissions:
+      contents: read
+      packages: write
+      security-events: write
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v7
+
+    - name: Log in to Container Registry
+      uses: docker/login-action@v4
+      with:
+        registry: ${{ env.REGISTRY }}
+        username: ${{ github.actor }}
+        password: ${{ secrets.GITHUB_TOKEN }}
+
+    - name: Set up QEMU
+      if: github.event_name == 'release'
+      uses: docker/setup-qemu-action@v4
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v4
+
+    - name: Extract metadata
+      id: meta
+      uses: docker/metadata-action@v6
+      with:
+        images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+        tags: |
+          type=ref,event=branch
+          type=ref,event=pr
+          type=semver,pattern={{version}}
+          type=semver,pattern={{major}}.{{minor}}
+          type=sha
+          type=raw,value=latest,enable=${{ github.event_name == 'release' && !contains(github.ref_name, 'alpha') && !contains(github.ref_name, 'beta') && !contains(github.ref_name, 'a') && !contains(github.ref_name, 'b') }}
+          type=raw,value=experimental,enable=${{ github.event_name == 'release' && (contains(github.ref_name, 'alpha') || contains(github.ref_name, 'beta') || contains(github.ref_name, 'a') || contains(github.ref_name, 'b')) }}
+
+    - name: Build and push Docker image
+      uses: docker/build-push-action@v7
+      with:
+        context: .
+        file: ./Containerfile
+        platforms: ${{ github.event_name == 'release' && 'linux/amd64,linux/arm64' || 'linux/amd64' }}
+        push: true
+        tags: ${{ steps.meta.outputs.tags }}
+        labels: ${{ steps.meta.outputs.labels }}
+        cache-from: type=gha
+        cache-to: type=gha,mode=max
+
+    - name: Extract first tag for Trivy
+      id: first-tag
+      run: echo "tag=$(echo '${{ steps.meta.outputs.tags }}' | head -n1)" >> $GITHUB_OUTPUT
+
+    - name: Run container security scan with Trivy
+      uses: aquasecurity/trivy-action@master
+      with:
+        image-ref: ${{ steps.first-tag.outputs.tag }}
+        format: 'sarif'
+        output: 'trivy-results.sarif'
+
+    - name: Upload Trivy scan results to GitHub Security tab
+      uses: github/codeql-action/upload-sarif@v4
+      if: always()
+      with:
+        sarif_file: 'trivy-results.sarif'
+
+  codeql-analysis:
+    name: CodeQL Analysis
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'python' ]
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v7
+
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v4
+      with:
+        languages: ${{ matrix.language }}
+
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v4
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v4
+
+  publish-package:
+    runs-on: ubuntu-latest
+    needs: [test, security-scan, build-container]
+    if: github.event_name == 'release' && github.event.action == 'published'
+
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        fetch-depth: 0
+
+    - name: Set up Python
+      uses: actions/setup-python@v7
+      with:
+        python-version: '3.11'
+
+    - name: Install build dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install build twine
+
+    - name: Build package
+      run: python -m build
+
+    - name: Check package
+      run: twine check dist/*
+
+    - name: Publish to PyPI
+      env:
+        TWINE_USERNAME: __token__
+        TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+      run: twine upload dist/*

talkpipe_vault-0.0.1/.gitignore

@@ -0,0 +1,180 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Custom 
+*NOCOMMIT* 
+.aider*
+.secret
+__manifest/

talkpipe_vault-0.0.1/.safety-project.ini

@@ -0,0 +1,5 @@
+[project]
+id = talkpipe-vault
+url = /codebases/talkpipe-vault/findings
+name = talkpipe-vault
+

talkpipe_vault-0.0.1/.vscode/settings.json

@@ -0,0 +1,7 @@
+{
+    "python.testing.pytestArgs": [
+        "tests"
+    ],
+    "python.testing.unittestEnabled": false,
+    "python.testing.pytestEnabled": true
+}

talkpipe_vault-0.0.1/AGENTS.md

@@ -0,0 +1,45 @@
+## Dev environment tips
+- When running code or pytest, use the .venv environment in the root of the project.
+- If the .venv environment doesn't exist and you need to run code or tests, abort and ask the user to create the environment.
+
+## Project Overview
+TalkPipe Vault (this repo) is a Python toolkit built on talkpipe that provides easy pipelines for 
+building vector databases and search engines, along with a demonstration application for unifying them. 
+The underlying pipelines include segments and sources for extracting text from files, monitoring 
+directories for changing files, doing RAG searches, and creating both full text and vector database
+indexing in a unified pipeline.  The library is intended to be used stand-along with the provided 
+application, but constituent parts can also be used in other applications.
+
+
+## Project Structure
+- `src/talkpipe_vault/` - Main library source code
+ - `apps/` - Application code (query interface, vault server)
+ - `pipelines/` - Pipeline implementations (building vector DBs, searching, watching)
+ - `segments.py` - Text extraction segments
+ - `watchdog.py` - File monitoring source
+ - `initialize_plugin.py` - TalkPipe plugin initialization
+- `tests/talkpipe_vault/` - Unit tests mirroring source structure
+- `docs/` - Documentation files
+- Root directory contains configuration files (pyproject.toml, docker files, etc.)
+
+## Development Guidelines 
+- When a markdown or source code file contains a line with "AGENT_TODO", replace that line with the requested code.
+- Do not automatically agree with what the user suggests.  Question assumptions and point out weaknesses in logic or inconsistences with the rest of the codebase.
+- pytest unit tests are under tests/
+- the code for the library is under src/
+- After implementing a new feature, update CHANGELOG.md. If the development section of the changelog already
+has a relevant section, update that section rather than creating a new one. 
+- When making a change, write a unit test that fails without the change or update an existing test, verify that the unit test fails, make the change and then verity that the unit test now passes.  Notify the user if it is not reasonable to write the unit test. 
+- When writing unit tests, update and augment existing unit tests is possible and appropriate rather than writing new unit tests.
+- Unit tests should be as short as possible while still testing the functionality as completely as possible.
+- User interfaces should be visually appealing and maintain consistency with each other.
+- When writing or updating pipeline examples in documentation, write examples with stand-alone code and 
+    test the examples by running them.
+- When writing documentation for segments or sources, do not document the parameters.
+- When writing new sources or segments, use Annotated in the parameter lists to explain the type and purpose of each parameter.
+- When writing a report or script that is not intended to be part of the repository, prefix the filename with "NOCOMMIT_".
+
+
+## Important Commands
+python -m build for creating a new release
+pytest --cov=src for unit test coverage