openkb 0.0.1__tar.gz → 0.1.0.dev0__tar.gz

openkb-0.1.0.dev0/LICENSE

@@ -0,0 +1,190 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to the Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by the Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding any notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2026 Vectify AI
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

openkb-0.1.0.dev0/PKG-INFO

@@ -0,0 +1,275 @@
+Metadata-Version: 2.3
+Name: openkb
+Version: 0.1.0.dev0
+Summary: Karpathy's LLM Knowledge Base workflow — powered by PageIndex
+License: Apache-2.0
+Keywords: ai,rag,retrieval,knowledge-base,llm,pageindex,agents,document
+Author: Kylin
+Author-email: quanqi@pageindex.ai
+Requires-Python: >=3.10
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Dist: click (>=8.0)
+Requires-Dist: litellm
+Requires-Dist: markitdown[all]
+Requires-Dist: openai-agents
+Requires-Dist: pageindex (==0.3.0.dev0)
+Requires-Dist: python-dotenv
+Requires-Dist: pyyaml
+Requires-Dist: watchdog (>=3.0)
+Project-URL: Homepage, https://github.com/VectifyAI/OpenKB
+Project-URL: Issues, https://github.com/VectifyAI/OpenKB/issues
+Project-URL: Repository, https://github.com/VectifyAI/OpenKB
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+<a href="https://openkb.ai">
+  <img src="https://docs.pageindex.ai/images/openkb.png" alt="OpenKB (by PageIndex)" />
+</a>
+
+# OpenKB — Open LLM Knowledge Base
+
+<p align="center"><i>Scale to long documents&nbsp; • &nbsp;Reasoning-based retrieval&nbsp; • &nbsp;Native multi-modality&nbsp; • &nbsp;No Vector DB</i></p>
+
+</div>
+
+---
+
+# 📑 What is OpenKB
+
+**OpenKB (Open Knowledge Base)** is an open-source system (in CLI) that compiles raw documents into a structured, interlinked wiki-style knowledge base using LLMs, powered by [**PageIndex**](https://github.com/VectifyAI/PageIndex) for vectorless long document retrieval.
+
+The idea is based on a [concept](https://x.com/karpathy/status/2039805659525644595) described by Andrej Karpathy: LLMs generate summaries, concept pages, and cross-references, all maintained automatically. Knowledge compounds over time instead of being re-derived on every query.
+
+### Why not just traditional RAG?
+
+Traditional RAG rediscovers knowledge from scratch on every query. Nothing accumulates. OpenKB compiles knowledge once into a persistent wiki, then keeps it current. Cross-references already exist. Contradictions are flagged. Synthesis reflects everything consumed.
+
+### Features
+
+- **Any format** — PDF, Word, PowerPoint, Excel, HTML, Markdown, text, CSV, and more via markitdown
+- **Scale to long documents** — Long and complex documents are handled via [PageIndex](https://github.com/VectifyAI/PageIndex) tree indexing, enabling accurate, vectorless long-context retrieval
+- **Native multi-modality** — Retrieves and understands figures, tables, and images, not just text
+- **Auto wiki** — LLM generates summaries, concept pages, and cross-links. You curate sources; the LLM does the rest
+- **Query** — Ask questions against your wiki. The LLM navigates your compiled knowledge to answer
+- **Lint** — Health checks find contradictions, gaps, orphans, and stale content
+- **Watch mode** — Drop files into `raw/`, wiki updates automatically
+- **Obsidian compatible** — Wiki is plain `.md` files with `[[wikilinks]]`. Open in Obsidian for graph view and browsing
+
+# 🚀 Getting Started
+
+### Install
+
+```bash
+pip install openkb
+```
+
+### Quick start
+
+```bash
+# 1. Create a directory for your knowledge base
+mkdir my-kb && cd my-kb
+
+# 2. Initialize the knowledge base
+openkb init
+
+# 3. Add documents
+openkb add paper.pdf
+openkb add ~/papers/                   # Add a whole directory
+openkb add article.html
+
+# 4. Ask questions
+openkb query "What are the main findings?"
+
+# 5. Check wiki health
+openkb lint
+```
+
+### Set up your LLM
+
+OpenKB comes with [multi-LLM support](https://docs.litellm.ai/docs/providers) (e.g., OpenAI, Claude, Gemini) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).
+
+Set your model during `openkb init`, or in [`.openkb/config.yaml`](#configuration), using `provider/model` LiteLLM format (like `anthropic/claude-sonnet-4-6`). OpenAI models can omit the prefix (like `gpt-5.4`).
+
+Create a `.env` file with your LLM API key:
+
+```bash
+LLM_API_KEY=your_llm_api_key
+```
+
+# 🧩 How It Works
+
+### Architecture
+
+```
+raw/                              You drop files here
+ │
+ ├─ Short docs ──→ markitdown ──→ LLM reads full text
+ │                                     │
+ ├─ Long PDFs ──→ PageIndex ────→ LLM reads document trees
+ │                                     │
+ │                                     ▼
+ │                         Wiki Compilation (using LLM)
+ │                                     │
+ ▼                                     ▼
+wiki/
+ ├── index.md            Knowledge base overview
+ ├── log.md              Operations timeline
+ ├── AGENTS.md           Wiki schema (LLM instructions)
+ ├── sources/            Full-text conversions
+ ├── summaries/          Per-document summaries
+ ├── concepts/           Cross-document synthesis ← the good stuff
+ ├── explorations/       Saved query results
+ └── reports/            Lint reports
+```
+
+### Short vs. long document handling
+
+| | Short documents | Long documents (PDF ≥ 20 pages) |
+|---|---|---|
+| **Convert** | markitdown → Markdown | PageIndex → tree index + summaries |
+| **Images** | Extracted inline (pymupdf) | Extracted by PageIndex |
+| **LLM reads** | Full text | Document trees |
+| **Result** | summary + concepts | summary + concepts |
+
+Short docs are read in full by the LLM. Long PDFs are indexed by PageIndex into a hierarchical tree with summaries. The LLM reads the tree instead of the full text, enabling better retrieval from long documents.
+
+### The wiki compiles knowledge
+
+When you add a document, the LLM:
+
+1. Generates a **summary** page
+2. Reads existing **concept** pages
+3. Creates or updates concepts with cross-document synthesis
+4. Updates the **index** and **log**
+
+A single source might touch 10-15 wiki pages. Knowledge accumulates: each document enriches the existing wiki rather than sitting in isolation.
+
+# 📦 Usage
+
+### Commands
+
+| Command | Description |
+|---|---|
+| `openkb init` | Initialize a new knowledge base (interactive) |
+| `openkb add <file_or_dir>` | Add documents and compile to wiki |
+| `openkb query "question"` | Ask a question against the knowledge base |
+| `openkb query "question" --save` | Ask and save the answer to `wiki/explorations/` |
+| `openkb watch` | Watch `raw/` and auto-compile new files |
+| `openkb lint` | Run structural + knowledge health checks |
+| `openkb list` | List indexed documents and concepts |
+| `openkb status` | Show knowledge base stats |
+
+<!-- | `openkb lint --fix` | Auto-fix what it can | -->
+
+### Configuration
+
+Settings are initialized by `openkb init`, and stored in `.openkb/config.yaml`:
+
+```yaml
+model: gpt-5.4                   # LLM model (any LiteLLM-supported provider)
+language: en                     # Wiki output language
+pageindex_threshold: 20          # PDF pages threshold for PageIndex
+```
+
+Model names use `provider/model` LiteLLM [format](https://docs.litellm.ai/docs/providers) (OpenAI models can omit the prefix):
+
+| Provider | Model example |
+|---|---|
+| OpenAI | `gpt-5.4` |
+| Anthropic | `anthropic/claude-sonnet-4-6` |
+| Gemini | `gemini/gemini-3.1-pro-preview` |
+
+### PageIndex integration
+
+Long documents are challenging for LLMs due to context limits, context rot, and summarization loss.
+[PageIndex](https://github.com/VectifyAI/PageIndex) solves this with vectorless, reasoning-based retrieval — building a hierarchical tree index that lets LLMs reason over the index for context-aware retrieval.
+
+PageIndex runs locally by default using the [open-source version](https://github.com/VectifyAI/PageIndex), with no external dependencies required.
+
+#### Optional: Cloud Support
+
+For large or complex PDFs, [PageIndex Cloud](https://docs.pageindex.ai/) can be used to access additional capabilities, including:
+
+- OCR support for scanned PDFs (via hosted VLM models)
+- Faster structure generation
+- Scalable indexing for large documents
+
+Set `PAGEINDEX_API_KEY` in your `.env` to enable cloud features:
+
+```
+PAGEINDEX_API_KEY=your_pageindex_api_key
+```
+
+### AGENTS.md
+
+The `wiki/AGENTS.md` file defines wiki structure and conventions. It's the LLM's instruction manual for maintaining the wiki. Customize it to change how your wiki is organized.
+
+At runtime, the LLM reads `AGENTS.md` from disk, so your edits take effect immediately.
+
+### Using with Obsidian
+
+OpenKB's wiki is a directory of Markdown files with `[[wikilinks]]`. Obsidian renders it natively.
+
+1. Open `wiki/` as an Obsidian vault
+2. Browse summaries, concepts, and explorations
+3. Use graph view to see knowledge connections
+4. Use Obsidian Web Clipper to add web articles to `raw/`
+
+# 🧭 Learn More
+
+### Compared to Karpathy's Approach
+
+| | Karpathy's workflow | OpenKB |
+|---|---|---|
+| Short documents | LLM reads directly | markitdown → LLM reads |
+| Long documents | Context limits, context rot | PageIndex tree index |
+| Supported formats | Web clipper → .md | PDF, Word, PPT, Excel, HTML, text, CSV, .md |
+| Wiki compilation | LLM agent | LLM agent (same) |
+| Q&A | Query over wiki | Wiki + PageIndex retrieval |
+
+### Tech Stack
+
+- [PageIndex](https://github.com/VectifyAI/PageIndex) — Vectorless, reasoning-based document indexing and retrieval
+- [markitdown](https://github.com/microsoft/markitdown) — Universal file-to-markdown conversion
+- [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) — Agent framework (supports non-OpenAI models via LiteLLM)
+- [LiteLLM](https://github.com/BerriAI/litellm) — Multi-provider LLM gateway
+- [Click](https://click.palletsprojects.com/) — CLI framework
+- [watchdog](https://github.com/gorakhargosh/watchdog) — Filesystem monitoring
+
+### Roadmap
+
+- [ ] Extend long document handling to non-PDF formats
+- [ ] Scale to large document collections with nested folder support
+- [ ] Hierarchical concept (topic) indexing for massive knowledge bases
+- [ ] Database-backed storage engine
+- [ ] Web UI for browsing and managing wikis
+
+### Contributing
+
+Contributions are welcome! Please submit a pull request, or open an [issue](https://github.com/VectifyAI/OpenKB/issues) for bugs or feature requests. For larger changes, consider opening an issue first to discuss the approach.
+
+### License
+
+Apache 2.0. See [LICENSE](LICENSE).
+
+### Support Us
+
+If you find OpenKB useful, give us a star 🌟 — and check out [PageIndex](https://github.com/VectifyAI/PageIndex) too!  
+
+<div>
+
+[![Twitter](https://img.shields.io/badge/Twitter-000000?style=for-the-badge&logo=x&logoColor=white)](https://x.com/PageIndexAI)&ensp;
+[![LinkedIn](https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white)](https://www.linkedin.com/company/vectify-ai/)&ensp;
+[![Contact Us](https://img.shields.io/badge/Contact_Us-3B82F6?style=for-the-badge&logo=envelope&logoColor=white)](https://ii2abc2jejf.typeform.com/to/tK3AXl8T)
+
+</div>
+