knowledge-assistant 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 jycd
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,117 @@
+# Knowledge Assistant
+
+A local-first personal knowledge base. Import PDFs and notes, search them by meaning and
+keyword, and ask questions that are answered with citations — all from one SQLite file on
+your machine. Indexing and search never touch the network; an LLM (local Ollama or a cloud
+API) is only needed for answers and AI note tidying.
+
+```
+node launcher/bin/ka.js
+```
+
+That installs a private Python environment on first run, downloads a small embedding model
+(~130 MB), and opens the app in your browser. The launcher is not published to npm.
+
+## Features
+
+- **Import** — drop PDFs or paste text. Documents are split into passages, embedded locally,
+  and indexed in the background with live progress.
+- **Search** — hybrid retrieval: vector similarity (sqlite-vec) fused with BM25 keyword
+  search (FTS5). Scope by category or topic.
+- **Ask** — answers stream in with `[n]` citations that link to the exact passages used.
+- **Library** — organise entries into categories and topics. Edits re-index automatically.
+- **Notes** — write, tidy into a structured note (rule-based or AI), promote to the library.
+- **Email** — sync a mailbox over IMAP; each message becomes a searchable entry, deduplicated by Message-ID, resumable by UID.
+- **Templates** — five built-in structures plus your own.
+- **Preferences** — describe how you like notes formatted in plain language.
+
+## Requirements
+
+- Python 3.10+ (the npm launcher finds it; `pip` users install directly)
+- Node 18+ only if using the npm launcher
+- Optional: [Ollama](https://ollama.com) for fully offline answers, or an OpenAI/Anthropic key
+
+## Install without npm
+
+```bash
+pip install knowledge-assistant[ollama]   # or [openai] / [anthropic] / [all]
+ka serve --open
+```
+
+## Configuration
+
+Environment variables (or a `.env` file in the working directory), all prefixed `KA_`:
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `KA_DATA_DIR` | `~/.knowledge-assistant` | Where the database, uploads and models live |
+| `KA_PORT` | `8765` | HTTP port |
+| `KA_LLM_PROVIDER` | `ollama` | `ollama`, `openai`, `anthropic`, or `none` |
+| `KA_LLM_MODEL` | `llama3.2` | Model name for the chosen provider |
+| `KA_OLLAMA_HOST` | `http://127.0.0.1:11434` | |
+| `KA_OPENAI_API_KEY` / `KA_ANTHROPIC_API_KEY` | | Only for cloud providers |
+| `KA_EMBEDDING_MODEL` | `BAAI/bge-small-en-v1.5` | Local embedding model (fastembed) |
+| `KA_CHUNK_TOKENS` | `512` | Passage size |
+| `KA_SEARCH_MAX_DISTANCE` | `0.45` | Vector cutoff (cosine distance); re-tune if you change the embedding model |
+| `KA_WORKER_THREADS` | `2` | Background import workers |
+| `KA_IMAP_HOST` / `KA_IMAP_USER` / `KA_IMAP_PASSWORD` | | Set all three to enable the email source (Gmail: use an app password) |
+| `KA_IMAP_FOLDER` | `INBOX` | Folder to sync |
+| `KA_IMAP_MAX_PER_SYNC` | `200` | Cap per sync run |
+
+## CLI
+
+```
+ka serve [--open] [--port N]   start API, worker and web UI
+ka ingest FILE                 import a PDF or text file without the server
+ka search "query"              search from the terminal
+ka db path | ka db reset       locate or wipe the database
+```
+
+## Architecture
+
+One Python process: FastAPI serves the API and the built React app; a thread pool consumes a
+job table in the same SQLite database; PDF parsing runs in a subprocess so a parser crash
+fails one job instead of the server.
+
+```
+npm launcher ─► ka serve
+                 ├─ FastAPI  /api/v1  + static SPA
+                 ├─ Worker pool ◄── jobs table
+                 │     └─ pdf_extract (subprocess)
+                 ├─ Embedder: fastembed (ONNX, local)
+                 ├─ LLM: ollama | openai | anthropic | none
+                 └─ knowledge.db: tables + vec0 + fts5
+```
+
+The job queue is the producer/consumer seam: producers call `JobQueue.enqueue`, handlers are
+registered by kind. Backing it with Redis or a separate worker process later means replacing
+`jobs/queue.py` only. New sources (e.g. an email connector) are new job kinds.
+
+```
+src/knowledge_assistant/
+  core/        models, repositories, chunking, embeddings, search, llm/, notes, preferences
+  jobs/        queue, worker, handlers, pdf_extract
+  api/         FastAPI routers and schemas
+  cli.py       `ka`
+web/           React 19 + Vite + Tailwind 4 (builds into src/knowledge_assistant/static)
+launcher/      npm package: finds Python, creates a venv, runs `ka serve`
+tests/         pytest (core, jobs, API) — no network, no model download
+```
+
+## Development
+
+```bash
+python -m venv .venv && . .venv/bin/activate
+pip install -e ".[dev,all]"
+pytest
+
+cd web && npm install && npm run dev      # UI with API proxy to :8765
+ka serve                                   # in another terminal
+```
+
+Build for local use: `cd web && npm run build`, then `./scripts/build-launcher.sh` to produce
+the wheel and stage it in `launcher/` for the Node launcher.
+
+## License
+
+MIT

package/bin/ka.js

@@ -0,0 +1,78 @@
+#!/usr/bin/env node
+/**
+ * npm launcher for Knowledge Assistant.
+ *
+ * The app is a Python service; this script finds a Python >= 3.10, creates a private
+ * virtualenv under ~/.knowledge-assistant/venv, installs the bundled wheel (or the pinned
+ * PyPI release), and execs `ka <args>`. Re-running is fast: the venv is reused.
+ *
+ * No native Node modules, no network beyond pip. Works on macOS, Linux, Windows.
+ */
+"use strict";
+const { spawnSync, spawn } = require("node:child_process");
+const fs = require("node:fs");
+const os = require("node:os");
+const path = require("node:path");
+
+const PKG = require("../package.json");
+const HOME = process.env.KA_DATA_DIR || path.join(os.homedir(), ".knowledge-assistant");
+const VENV = path.join(HOME, "venv");
+const WIN = process.platform === "win32";
+const VENV_PY = WIN ? path.join(VENV, "Scripts", "python.exe") : path.join(VENV, "bin", "python");
+const STAMP = path.join(VENV, ".ka-version");
+
+function log(msg) { process.stderr.write(`[knowledge-assistant] ${msg}\n`); }
+
+function pythonVersion(bin) {
+  const r = spawnSync(bin, ["-c", "import sys;print('%d.%d'%sys.version_info[:2])"], { encoding: "utf8" });
+  if (r.status !== 0) return null;
+  const [maj, min] = r.stdout.trim().split(".").map(Number);
+  return maj > 3 || (maj === 3 && min >= 10) ? `${maj}.${min}` : null;
+}
+
+function findPython() {
+  const candidates = [process.env.KA_PYTHON, "python3.12", "python3.11", "python3.13", "python3.10", "python3", "python", WIN ? "py" : null].filter(Boolean);
+  for (const c of candidates) {
+    const args = c === "py" ? ["-3"] : [];
+    const r = spawnSync(c, [...args, "-c", "import sys;print('%d.%d'%sys.version_info[:2])"], { encoding: "utf8" });
+    if (r.status === 0) {
+      const [maj, min] = r.stdout.trim().split(".").map(Number);
+      if (maj === 3 && min >= 10) return c === "py" ? ["py", "-3"] : [c];
+    }
+  }
+  return null;
+}
+
+function ensureVenv() {
+  const upToDate = fs.existsSync(VENV_PY) && fs.existsSync(STAMP) && fs.readFileSync(STAMP, "utf8").trim() === PKG.version;
+  if (upToDate) return;
+  const py = findPython();
+  if (!py) {
+    log("Python 3.10+ is required but was not found. Install it from https://python.org and re-run.");
+    process.exit(1);
+  }
+  fs.mkdirSync(HOME, { recursive: true });
+  if (!fs.existsSync(VENV_PY)) {
+    log(`creating virtualenv in ${VENV} (one-time)`);
+    const r = spawnSync(py[0], [...py.slice(1), "-m", "venv", VENV], { stdio: "inherit" });
+    if (r.status !== 0) process.exit(r.status ?? 1);
+  }
+  const wheelDir = path.join(__dirname, "..", "python");
+  const wheel = fs.existsSync(wheelDir) ? fs.readdirSync(wheelDir).find((f) => f.endsWith(".whl")) : null;
+  const target = wheel ? path.join(wheelDir, wheel) : `knowledge-assistant==${PKG.version}`;
+  log(`installing ${wheel || target} (this downloads the embedding runtime; a few minutes the first time)`);
+  const r = spawnSync(VENV_PY, ["-m", "pip", "install", "--quiet", "--upgrade", "pip", target], { stdio: "inherit" });
+  if (r.status !== 0) process.exit(r.status ?? 1);
+  fs.writeFileSync(STAMP, PKG.version);
+}
+
+function main() {
+  ensureVenv();
+  const args = process.argv.slice(2);
+  const finalArgs = args.length ? args : ["serve", "--open"];
+  const child = spawn(VENV_PY, ["-m", "knowledge_assistant.cli", ...finalArgs], { stdio: "inherit", env: { ...process.env, KA_DATA_DIR: HOME } });
+  for (const sig of ["SIGINT", "SIGTERM"]) process.on(sig, () => child.kill(sig));
+  child.on("exit", (code) => process.exit(code ?? 0));
+}
+
+main();

package/package.json

@@ -0,0 +1,38 @@
+{
+  "name": "knowledge-assistant",
+  "version": "0.1.0",
+  "description": "Local-first personal knowledge base: PDF import, hybrid search, RAG Q&A, notes. Runs entirely on your machine.",
+  "license": "MIT",
+  "bin": {
+    "knowledge-assistant": "bin/ka.js",
+    "ka": "bin/ka.js"
+  },
+  "files": [
+    "bin",
+    "python/*.whl",
+    "README.md",
+    "LICENSE"
+  ],
+  "engines": {
+    "node": ">=18"
+  },
+  "keywords": [
+    "knowledge-base",
+    "rag",
+    "local-first",
+    "sqlite",
+    "vector-search",
+    "pdf"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/jycd25/knowledge-assistant"
+  },
+  "homepage": "https://github.com/jycd25/knowledge-assistant#readme",
+  "bugs": {
+    "url": "https://github.com/jycd25/knowledge-assistant/issues"
+  },
+  "publishConfig": {
+    "access": "public"
+  }
+}