minichat-studio 0.3.0__tar.gz

minichat_studio-0.3.0/LICENSE

@@ -0,0 +1,7 @@
+Copyright 2026 Jason Jiménez Cruz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

minichat_studio-0.3.0/PKG-INFO

@@ -0,0 +1,363 @@
+Metadata-Version: 2.4
+Name: minichat-studio
+Version: 0.3.0
+Summary: MiniCS Lite (Mini ChatML Studio Lite) — a local-first studio for building high quality LLM datasets with RAG + graph grounding.
+Author-email: Jason Jimenez Cruz <jasonjimenezcruz@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/jasonjimnz/minics
+Project-URL: Repository, https://github.com/jasonjimnz/minics
+Project-URL: Issues, https://github.com/jasonjimnz/minics/issues
+Project-URL: Changelog, https://github.com/jasonjimnz/minics/releases
+Keywords: llm,dataset,rag,graph,chromadb,ladybug,flask,pywebview
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Web Environment
+Classifier: Framework :: Flask
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: flask>=3.0
+Requires-Dist: pywebview>=5.0
+Requires-Dist: pydantic>=2.6
+Requires-Dist: openai>=1.40
+Requires-Dist: langchain>=0.3
+Requires-Dist: langchain-core>=0.3
+Requires-Dist: langchain-openai>=0.2
+Requires-Dist: langchain-community>=0.3
+Requires-Dist: langchain-text-splitters>=0.3
+Requires-Dist: chromadb<2,>=0.5
+Requires-Dist: ladybug<0.19,>=0.17
+Requires-Dist: pypdf>=4.2
+Requires-Dist: python-docx>=1.1
+Requires-Dist: mammoth>=1.8
+Requires-Dist: markdownify>=0.12
+Requires-Dist: pylatexenc>=2.10
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Dynamic: license-file
+
+# MiniCS Lite
+
+**Mini ChatML Studio Lite** — a local-first studio for building high quality
+LLM datasets. MiniCS combines a ChatML-aware dataset library, a hybrid RAG +
+knowledge-graph grounding engine, an LLM authoring toolbox and a markdown
+editor into a single desktop application (Flask + PyWebView). Everything lives
+inside your own `~/.minics` folder — no cloud, no accounts, no lock-in.
+
+> **MiniCS is a mixed project.** It combines two sibling codebases: **ChatML
+> Studio**, a platform for handling datasets and entries, and **Tyness**, a
+> *tiny documental harness* for enhancing LLM capabilities with embed
+> databases. MiniCS Lite distills both into one small, local-first tool — the
+> root projects will keep growing with more capabilities.
+>
+> **MiniCS is OpenAI-protocol first.** Any OpenAI-compatible endpoint works:
+> OpenAI, Azure-style gateways, vLLM, llama.cpp (`llama-server`), LM Studio,
+> Ollama's OpenAI shim, or anything else speaking `/v1/chat/completions` and
+> `/v1/embeddings`.
+
+---
+
+## Highlights
+
+| Area | What you get |
+| --- | --- |
+| **Dataset library** | Datasets, ChatML entries with system/user/assistant roles, draft → review → approved workflow, automatic versioning on every message edit, structural ChatML validation. |
+| **Grounding engine** | Import **PDF, DOCX, TXT, Markdown, LaTeX** — every file is converted to cleaned markdown (optionally repaired by the LLM), then chunked and indexed. |
+| **Hybrid retrieval** | ChromaDB vector search **plus** Ladybug graph topology, fused with Reciprocal Rank Fusion (RRF). Each side degrades gracefully when the other is unavailable. |
+| **LLM everywhere** | Enhance fields, suggest tags, evaluate entries against 6 quality dimensions, generate grounded new entries, clean markdown, extract graph entities. Structured output with 3 fallback strategies. |
+| **Background-first** | Long operations (import, indexing, evaluation) run on a worker thread pool with live progress streamed to the UI over **Server-Sent Events**, backed by a concurrency-safe SQLite writer thread. |
+| **Chat with switches** | Per-conversation toggles for RAG, graph and grounding — retrieval and injection are separate, so you can *preview* what would be used without forcing it into the prompt. |
+| **Export anywhere** | ChatML, JSONL, Alpaca, ShareGPT and Markdown — byte-identical from UI, CLI or Python. |
+| **Themable UI** | 15 hand-tuned themes — System, Light, Dark, Ocean, Forest, Sunset, Nord, Rose, Midnight, Paper, Mono, Grape, Dracula, Catppuccin Mocha and Solarized — with native dark form controls, tinted shadows and themed scrollbars. Themes are pure CSS-variable swaps. |
+
+---
+
+## A walkthrough of the web app
+
+The screenshots below (from [`docs/screenshots/`](docs/screenshots/)) follow a
+real end-to-end session against two local model servers: a **llama.cpp
+`llama-server`** instance serving the chat model and an **Ollama** instance
+serving the embedding model — exactly the split-endpoint setup MiniCS is
+designed for.
+
+### 1. Home / Dashboard
+
+![Dashboard](docs/screenshots/01_home.png)
+
+The landing screen. Live counters for datasets, entries, approved entries,
+documents, indexed chunks and graph entities, a *Getting started* checklist
+that ticks itself off as you complete each step, and a recent-activity feed.
+Everything in the sidebar — Datasets, Entries, Collections, Documents, Chat,
+Graph, Activity, Settings — is one click away.
+
+### 2. Settings — wiring up the endpoints
+
+![Settings](docs/screenshots/02_settings.png)
+
+Here the app is pointed at the two local servers: the **LLM endpoint** is a
+`llama-server` instance (`qwen-3-8` on `:8000/v1`) and the **embeddings
+endpoint** is Ollama (`bge-m3:latest` on `:11434/v1`, 1024 dimensions — hit
+*Detect* to sniff it automatically, since it drives the Chroma vector store).
+The Retrieval panel tunes the hybrid search defaults (Top K, RRF k, graph
+hops), and the Application panel picks one of the 15 themes (Catppuccin Mocha
+here) and the background-worker pool size. *Test connection* validates both
+endpoints before *Finish setup* locks them in.
+
+### 3. Creating a dataset
+
+![New dataset](docs/screenshots/03_new_dataset.png)
+
+Datasets are the top-level containers for training examples. Give one a name,
+a short description and optional comma-separated tags — that's all it takes.
+
+### 4. Creating an entry
+
+![New entry](docs/screenshots/04_new_entry.png)
+
+Each entry is a ChatML conversation: **system**, **user** and **assistant**
+messages. Fill them by hand, or type a topic into *Generate with LLM
+(grounded)* and let the model draft the whole entry using your imported
+documents as context.
+
+### 5. The entry editor
+
+![Entry editor](docs/screenshots/05_created_entry.png)
+
+The generated entry lands in the editor. Messages can be reordered, deleted
+and switched between roles; a *Valid ChatML* banner confirms the structure in
+real time. The LLM helpers are one click away — *Enhance assistant*, *Suggest
+tags* (in action here) and *Evaluate* against the six quality dimensions —
+and every message edit bumps the entry version automatically.
+
+### 6. Importing a markdown document
+
+![Document viewer](docs/screenshots/06_adding_document.png)
+
+Dropping a markdown file (or PDF/DOCX/TXT/LaTeX — it gets converted) opens the
+document viewer: a split-pane markdown editor with live preview and formatting
+toolbar. This one has been chunked into **127 chunks**, ready to index. From
+here you can *Clean with LLM*, *Enrich graph* and *Reindex* without leaving
+the dialog.
+
+### 7. Querying the knowledge graph
+
+![Graph](docs/screenshots/07_searching_graph.png)
+
+The Graph screen shows the entity topology extracted from your documents —
+923 entities, 1492 mentions and 4779 co-occurrences from a single import.
+Type a query (`Sorting` here) and hit *Explore* to render the matching
+sub-graph; *Rebuild* regenerates the whole topology from the index.
+
+### 8. Testing the LLM
+
+![Chat](docs/screenshots/08_chat_with_model.png)
+
+Chat is where the model gets smoke-tested. Each conversation carries its own
+**RAG**, **Graph** and **Grounding** toggles — you can preview what retrieval
+*would* inject without forcing it into the prompt. This call is a plain
+sanity-check prompt with all toggles off.
+
+### 9. Creating a collection
+
+![New collection](docs/screenshots/09_new_collection.png)
+
+Collections are curated sets of entries destined for export — a name and a
+description and you're in.
+
+### 10. Curating and exporting
+
+![Collection](docs/screenshots/10_adding_entries.png)
+
+Search your entries and *Add* them to the collection, then download the whole
+set in one click — **ChatML, JSONL, Alpaca, ShareGPT or Markdown** —
+byte-identical whether exported from the UI, the CLI or Python.
+
+That's the full loop: **configure → create → author → import → ground → chat
+→ curate → export.**
+
+---
+
+## Quick start
+
+```bash
+pip install -e ".[dev]"
+
+minics init                     # create ~/.minics
+minics setup                    # interactive LLM + embedding wizard
+minics                          # launch the desktop app (default command)
+```
+
+Point the wizard at any OpenAI-compatible chat endpoint and any embedding
+endpoint (they may be different servers). The embedding dimension is detected
+automatically and required — Chroma collections are created with it.
+
+Other entry points:
+
+```bash
+minics serve --port 8800        # Flask only, opened in a browser
+minics models                   # list models from the endpoint
+minics search "what is RRF?"    # hybrid retrieval from the terminal
+minics export <collection> -o out.jsonl --format jsonl
+minics documents --import paper.pdf --index
+minics reindex                  # rebuild the vector index + graph
+python -m minics --help         # everything else
+```
+
+### Run as a server (Docker)
+
+The repo ships a server container that keeps the whole store in a volume and
+auto-creates it on first start — no setup steps on the host:
+
+```bash
+docker compose up -d --build   # or: docker build -t minics-lite . && docker run -d -p 8765:8765 -v minics-data:/data minics-lite
+```
+
+Open `http://localhost:8765`, finish setup in the UI. Details in
+[docs/docker.md](docs/docker.md).
+
+### Use it as a library
+
+MiniCS is a library first; the Flask API and the desktop shell are thin layers
+on top of a single application context (`minics.services.context.AppContext`).
+
+```python
+from minics.services.context import get_context
+
+ctx = get_context()
+dataset = ctx.datasets.create("Support QA", tags=["support"])
+entry = ctx.entries.create(dataset.id, user="What is RRF?", assistant="A fusion method.")
+ctx.indexer.index_document(ctx.documents.import_file("handbook.pdf").id)
+result = ctx.retriever.retrieve("how does fusion work?")
+print(result.context_text())
+```
+
+---
+
+## What's inside
+
+```
+~/.minics/
+├── config.json        # LLM / embedding / retrieval / app settings
+├── minics.sqlite3     # datasets, entries, collections, documents, chats (WAL)
+├── vectordb/          # ChromaDB persistent store (cosine space)
+├── graphdb/           # Ladybug graph database (minics.graph)
+├── documents/
+│   ├── originals/     # untouched uploads, kept for provenance/re-extraction
+│   └── markdown/      # cleaned markdown used for chunking + grounding
+├── logs/
+└── cache/
+```
+
+Override the location with the `MINICS_HOME` environment variable (used by the
+test suite to run against throwaway homes).
+
+---
+
+## Architecture
+
+```
+CLI / PyWebView ──► Flask API ──► services (library API) ──┬─► SQLite (single writer thread)
+                                                           ├─► ChromaDB (vectors)
+                                                           ├─► Ladybug (graph)
+                                                           └─► OpenAI-compatible LLM + embeddings
+```
+
+- **`minics.core`** — paths, config, SQLite (WAL + serialised writer thread),
+  in-process event bus, background job manager, generic pydantic repository.
+- **`minics.domain`** — pydantic entities (Dataset, Entry, Collection, Document,
+  Conversation, …). Storage-agnostic: serialised to rows *and* to JSON.
+- **`minics.llm`** — OpenAI/LangChain client factory, embeddings, structured
+  output with fallbacks, all authoring helpers and prompts.
+- **`minics.rag`** — markdown-aware chunking, heuristic entity extraction,
+  Chroma store, Ladybug graph, RRF fusion, hybrid retriever.
+- **`minics.documents`** — PDF/DOCX/TXT/Markdown/LaTeX → clean markdown.
+- **`minics.services`** — the library API: datasets, entries, collections,
+  documents, authoring, chat, export.
+- **`minics.server`** — Flask app factory, REST API blueprints, SSE event
+  stream, and the no-build static SPA.
+- **`minics.desktop` / `minics.cli`** — the PyWebView desktop shell and the CLI.
+
+Every layer is documented in depth in [`docs/`](docs/index.md):
+
+| Document | Contents |
+| --- | --- |
+| [Documentation index](docs/index.md) | Reading order and full map of the docs. |
+| [Architecture](docs/architecture.md) | Layers, request lifecycle, threading model, design decisions. |
+| [Libraries](docs/libraries.md) | Every dependency, why it's there and where it's used. |
+| [Configuration](docs/configuration.md) | Every config key, the setup wizard, `MINICS_HOME`, secrets masking. |
+| [Core & backend](docs/backend.md) | SQLite writer, event bus, job manager, repositories, services, CLI, desktop shell. |
+| [LLM layer](docs/llm.md) | OpenAI protocol client, embeddings, structured output + truncation handling, prompts, authoring helpers. |
+| [RAG & graph](docs/rag.md) | Conversion, chunking, entity extraction, Ladybug schema, Chroma, indexing pipeline, RRF hybrid retrieval. |
+| [HTTP API](docs/api.md) | Full REST reference, SSE event stream, job workflow, error mapping. |
+| [Docker](docs/docker.md) | Server container: build, run, Compose, volumes, first-run setup, troubleshooting. |
+| [Frontend](docs/frontend.md) | The no-build SPA: modules, hash routing, SSE, themes, markdown, graph view, every screen. |
+| [Development](docs/development.md) | Environment, tests, fixtures, style, migrations, how to contribute. |
+
+---
+
+## The 60-second tour
+
+1. **Settings** → point MiniCS at your chat + embedding endpoints, hit
+   *Test connection*, *Finish setup*.
+2. **Documents** → drop a PDF/DOCX/TXT/MD/TeX file. It is converted to markdown,
+   cleaned (heuristically, plus the LLM if enabled), chunked along headings,
+   embedded into Chroma, and wired into the entity graph.
+3. **Datasets / Entries** → author ChatML entries by hand or generate them from
+   a topic grounded in your documents. Enhance, tag and evaluate with the LLM;
+   every message edit bumps the entry version.
+4. **Collections** → curate approved entries and export to ChatML / JSONL /
+   Alpaca / ShareGPT / Markdown.
+5. **Chat** → talk to your model. Toggle RAG / graph / grounding per
+   conversation; answers cite the chunks they used as `[n]` references.
+6. **Graph** → explore the entity topology extracted from your documents.
+7. **Activity** → every long operation is a background job with live progress,
+   cancellable at any time.
+
+---
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest -q          # tests (no model server required — embeddings are faked)
+ruff check src tests
+```
+
+The test suite runs against an isolated `MINICS_HOME` with deterministic fake
+embeddings, so it needs no network access. See
+[docs/development.md](docs/development.md).
+
+## Contributing
+
+Contributions are welcome! MiniCS is a small project maintained in tiny,
+versioned increments, so keep pull requests focused. Contributors must follow
+the project's core principles:
+
+- **Runs everywhere.** The application must work on any OS — Linux, macOS and
+  Windows — using **Python 3.12 or newer**. Avoid OS-specific code paths
+  unless they are guarded and tested.
+- **Local-first.** No cloud, no accounts, no telemetry; everything stays in
+  `~/.minics`.
+- **OpenAI-protocol only.** Talk to models exclusively through the
+  OpenAI-compatible HTTP protocol.
+- **Lite by design.** MiniCS Lite is the compact edition: the root projects
+  (ChatML Studio and Tyness) are the ones that will grow bigger and gain more
+  capabilities over time.
+
+See [docs/development.md](docs/development.md) for environment setup, tests
+and style conventions.
+
+## Status
+
+**Version 0.3.0 is ready.** The project is still a **beta** and intentionally
+a **small project** — a lite blend of ChatML Studio and Tyness — built in
+small, versioned increments (see `git log`).
+
+## License
+
+MIT — see [pyproject.toml](pyproject.toml).