adaptron 0.1.0__tar.gz

adaptron-0.1.0/.gitignore

@@ -0,0 +1,211 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Node.js
+node_modules/
+.next/

adaptron-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 GIS-DHSIT
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

adaptron-0.1.0/PKG-INFO

@@ -0,0 +1,333 @@
+Metadata-Version: 2.4
+Name: adaptron
+Version: 0.1.0
+Summary: End-to-end LLM Fine-tuning Framework
+Project-URL: Homepage, https://github.com/MuthuSubramanian/Adaptron
+Project-URL: Repository, https://github.com/MuthuSubramanian/Adaptron
+Project-URL: Bug Tracker, https://github.com/MuthuSubramanian/Adaptron/issues
+Author: Muthu Subramanian G
+License-Expression: MIT
+License-File: LICENSE
+Keywords: fine-tuning,llm,machine-learning,nlp,training,transformers
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: typer>=0.12
+Provides-Extra: all
+Requires-Dist: accelerate>=0.28; extra == 'all'
+Requires-Dist: apscheduler>=3.10; extra == 'all'
+Requires-Dist: bitsandbytes>=0.43; extra == 'all'
+Requires-Dist: boto3>=1.34; extra == 'all'
+Requires-Dist: cassandra-driver>=3.29; extra == 'all'
+Requires-Dist: chromadb>=0.5; extra == 'all'
+Requires-Dist: confluent-kafka>=2.3; extra == 'all'
+Requires-Dist: datasets>=2.18; extra == 'all'
+Requires-Dist: elasticsearch>=8.12; extra == 'all'
+Requires-Dist: fastapi>=0.110; extra == 'all'
+Requires-Dist: google-cloud-bigquery>=3.17; extra == 'all'
+Requires-Dist: httpx>=0.27; extra == 'all'
+Requires-Dist: huggingface-hub>=0.22; extra == 'all'
+Requires-Dist: langchain-community>=0.2; extra == 'all'
+Requires-Dist: openpyxl>=3.1; extra == 'all'
+Requires-Dist: pandas>=2.2; extra == 'all'
+Requires-Dist: peft>=0.10; extra == 'all'
+Requires-Dist: pymongo>=4.6; extra == 'all'
+Requires-Dist: pypdf>=4.0; extra == 'all'
+Requires-Dist: python-docx>=1.1; extra == 'all'
+Requires-Dist: redis>=5.0; extra == 'all'
+Requires-Dist: sentence-transformers>=2.6; extra == 'all'
+Requires-Dist: snowflake-connector-python>=3.7; extra == 'all'
+Requires-Dist: spacy>=3.7; extra == 'all'
+Requires-Dist: sqlalchemy>=2.0; extra == 'all'
+Requires-Dist: torch>=2.1; extra == 'all'
+Requires-Dist: transformers>=4.40; extra == 'all'
+Requires-Dist: trl>=0.8; extra == 'all'
+Requires-Dist: uvicorn>=0.29; extra == 'all'
+Requires-Dist: websockets>=12.0; extra == 'all'
+Provides-Extra: api
+Requires-Dist: fastapi>=0.110; extra == 'api'
+Requires-Dist: uvicorn>=0.29; extra == 'api'
+Requires-Dist: websockets>=12.0; extra == 'api'
+Provides-Extra: connectors
+Requires-Dist: apscheduler>=3.10; extra == 'connectors'
+Requires-Dist: boto3>=1.34; extra == 'connectors'
+Requires-Dist: cassandra-driver>=3.29; extra == 'connectors'
+Requires-Dist: confluent-kafka>=2.3; extra == 'connectors'
+Requires-Dist: elasticsearch>=8.12; extra == 'connectors'
+Requires-Dist: google-cloud-bigquery>=3.17; extra == 'connectors'
+Requires-Dist: httpx>=0.27; extra == 'connectors'
+Requires-Dist: pymongo>=4.6; extra == 'connectors'
+Requires-Dist: redis>=5.0; extra == 'connectors'
+Requires-Dist: snowflake-connector-python>=3.7; extra == 'connectors'
+Provides-Extra: deploy
+Requires-Dist: huggingface-hub>=0.22; extra == 'deploy'
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest-cov>=5.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.3; extra == 'dev'
+Provides-Extra: ingest
+Requires-Dist: langchain-community>=0.2; extra == 'ingest'
+Requires-Dist: openpyxl>=3.1; extra == 'ingest'
+Requires-Dist: pandas>=2.2; extra == 'ingest'
+Requires-Dist: pypdf>=4.0; extra == 'ingest'
+Requires-Dist: python-docx>=1.1; extra == 'ingest'
+Requires-Dist: sqlalchemy>=2.0; extra == 'ingest'
+Provides-Extra: rag
+Requires-Dist: chromadb>=0.5; extra == 'rag'
+Requires-Dist: sentence-transformers>=2.6; extra == 'rag'
+Provides-Extra: train
+Requires-Dist: accelerate>=0.28; extra == 'train'
+Requires-Dist: bitsandbytes>=0.43; extra == 'train'
+Requires-Dist: datasets>=2.18; extra == 'train'
+Requires-Dist: peft>=0.10; extra == 'train'
+Requires-Dist: torch>=2.1; extra == 'train'
+Requires-Dist: transformers>=4.40; extra == 'train'
+Requires-Dist: trl>=0.8; extra == 'train'
+Provides-Extra: understand
+Requires-Dist: sentence-transformers>=2.6; extra == 'understand'
+Requires-Dist: spacy>=3.7; extra == 'understand'
+Description-Content-Type: text/markdown
+
+# Adaptron
+
+**End-to-end LLM Fine-tuning Framework**
+
+<!-- Badges placeholder -->
+<!-- ![PyPI](https://img.shields.io/pypi/v/adaptron) -->
+<!-- ![Python](https://img.shields.io/pypi/pyversions/adaptron) -->
+<!-- ![License](https://img.shields.io/github/license/GIS-DHSIT/adaptron) -->
+
+---
+
+## Overview
+
+Adaptron is a plugin-based framework that takes you from raw documents to a deployed, fine-tuned language model. It orchestrates six pipeline stages -- **Ingest, Understand, Synthesize, Train, Evaluate, Deploy** -- and exposes the entire workflow through a Python API, a CLI, a FastAPI backend, and a Next.js web UI.
+
+## Features
+
+- **Multi-format ingestion** -- PDF, DOCX, CSV, and SQL data sources
+- **Semantic understanding** -- chunking, entity extraction, quality scoring, schema inference
+- **Instruction synthesis** -- template-based training data generation
+- **Multiple training strategies** -- QLoRA (Unsloth+PEFT), Full Fine-Tuning, Continual Pre-Training, Distillation, DPO alignment
+- **Domain evaluation** -- automated scoring of fine-tuned models
+- **One-click deployment** -- Ollama, GGUF export, HuggingFace Hub push
+- **Training Strategy Wizard** -- answers seven questions, picks the best training mode and base model automatically
+- **Playground** -- chat with deployed models, compare outputs side-by-side, toggle RAG augmentation
+- **Plugin system** -- register custom ingesters, trainers, deployers, or any stage component
+- **Event-driven pipeline** -- real-time progress via EventBus and WebSocket
+
+## Quick Start
+
+```bash
+# Install with all optional dependencies
+pip install -e ".[all]"
+
+# Initialize a project
+adaptron init --project-dir my-project
+cd my-project
+
+# Edit adaptron.yaml, then run the pipeline
+adaptron run
+```
+
+## Installation
+
+Requires **Python 3.11+**.
+
+```bash
+# Core only (config, CLI, pipeline orchestrator)
+pip install -e .
+
+# With specific extras
+pip install -e ".[train]"       # torch, transformers, PEFT, TRL, bitsandbytes
+pip install -e ".[ingest]"      # pypdf, python-docx, pandas, sqlalchemy
+pip install -e ".[understand]"  # spacy, sentence-transformers
+pip install -e ".[rag]"         # chromadb, sentence-transformers
+pip install -e ".[api]"         # fastapi, uvicorn, websockets
+pip install -e ".[deploy]"      # huggingface-hub
+
+# Everything
+pip install -e ".[all]"
+
+# Development (pytest, ruff, coverage)
+pip install -e ".[dev]"
+```
+
+## Usage
+
+### Python API
+
+```python
+from adaptron.core.config import PipelineConfig, WizardAnswers
+
+# Let the wizard pick the best strategy
+answers = WizardAnswers(
+    primary_goal="qa_docs",
+    data_sources=["docs"],
+    data_freshness="static",
+    hardware="mid",
+    timeline="medium",
+    accuracy="professional",
+    model_size="small",
+)
+config = PipelineConfig.from_wizard(answers)
+
+print(config.base_model)       # e.g. Qwen/Qwen2.5-7B-Instruct
+print(config.training_modes)   # e.g. ['qlora', 'rag']
+
+# Or load from YAML
+config = PipelineConfig.from_yaml("adaptron.yaml")
+```
+
+### CLI Commands
+
+```bash
+adaptron version              # Show version
+adaptron init                 # Create adaptron.yaml with defaults
+adaptron run                  # Execute the pipeline
+adaptron wizard               # Interactive training strategy wizard
+adaptron playground           # Chat with a finetuned model via Ollama
+adaptron playground --rag     # Chat with RAG context augmentation
+```
+
+### Web UI
+
+```bash
+# Start the FastAPI backend
+uvicorn adaptron.api.main:create_app --factory --reload
+
+# In a separate terminal, start the Next.js frontend
+cd web && npm install && npm run dev
+```
+
+Then open [http://localhost:3000](http://localhost:3000) to access the Wizard, Dashboard, and Playground.
+
+## Architecture
+
+```
+                         adaptron.yaml
+                              |
+                              v
+  +-------+  +------------+  +------------+  +-------+  +----------+  +--------+
+  |Ingest | ->| Understand | ->| Synthesize | ->| Train | ->| Evaluate | ->| Deploy |
+  +-------+  +------------+  +------------+  +-------+  +----------+  +--------+
+   PDF,DOCX   Chunker,        Instruction     QLoRA,     Domain        Ollama,
+   CSV,SQL    Entities,        templates      FullFT,    scoring       GGUF,
+              Quality,                        CPT,DPO,                 HF Hub
+              Schema                          Distill
+
+  All stages are plugins registered in the global PluginRegistry.
+  The PipelineOrchestrator runs them sequentially, emitting events
+  via EventBus that the WebSocket API streams to the frontend.
+```
+
+## Plugin System
+
+Every pipeline component is a plugin. Register your own with the `@register_plugin` decorator:
+
+```python
+from adaptron.core.registry import register_plugin
+from adaptron.ingest.base import BaseIngester
+from adaptron.ingest.models import DataSource, RawDocument
+
+@register_plugin("ingester", "my_custom")
+class MyCustomIngester(BaseIngester):
+    def ingest(self, source: DataSource) -> list[RawDocument]:
+        # Your custom ingestion logic
+        ...
+
+    def supported_types(self) -> list[str]:
+        return ["my_custom"]
+```
+
+Retrieve plugins at runtime:
+
+```python
+from adaptron.core.registry import global_registry
+
+ingester_cls = global_registry.get("ingest", "my_custom")
+```
+
+## Pipeline Stages
+
+| Stage | Module | Plugins | Description |
+|-------|--------|---------|-------------|
+| **Ingest** | `adaptron.ingest` | `pdf`, `docx`, `csv`, `sql` | Extract text and metadata from data sources |
+| **Understand** | `adaptron.understand` | `chunker`, `entities`, `quality`, `schema` | Semantic chunking, entity extraction, quality scoring, schema inference |
+| **Synthesize** | `adaptron.synthesize` | `instruction` | Generate instruction-response training pairs from chunks |
+| **Train** | `adaptron.train` | `qlora`, `full_ft`, `cpt`, `distill`, `alignment` (DPO) | Fine-tune or align the base model |
+| **Evaluate** | `adaptron.evaluate` | `domain` | Score model outputs against domain-specific criteria |
+| **Deploy** | `adaptron.deploy` | `ollama`, `gguf`, `huggingface` | Export and deploy the fine-tuned model |
+
+## Playground
+
+The playground lets you interact with deployed models:
+
+- **Chat mode** -- streaming conversation with any Ollama-hosted model
+- **RAG toggle** -- augment prompts with relevant context from ChromaDB
+- **Comparison mode** -- send the same prompt to two models side-by-side (web UI)
+- **CLI access** -- `adaptron playground --model adaptron-mymodel --rag`
+
+## Configuration
+
+Adaptron uses a YAML configuration file (`adaptron.yaml`):
+
+```yaml
+# Wizard answers -- drive automatic strategy selection
+wizard:
+  primary_goal: qa_docs          # qa_docs | erp_edw | report_gen | specialist
+  data_sources:
+    - docs                       # docs | erp | edw
+  data_freshness: static         # static | monthly | daily | realtime
+  hardware: mid                  # low | mid | high | cloud
+  timeline: medium               # fast | medium | long | unlimited
+  accuracy: professional         # professional | enterprise | mission
+  model_size: small              # tiny | small | medium | large
+
+# Manual overrides
+overrides:
+  epochs: 3
+  learning_rate: 0.0002
+  batch_size: 4
+  lora_rank: 64
+  max_seq_length: 2048
+  quantization: Q4_K_M
+
+data:
+  input_dir: ./data
+  output_dir: ./output
+
+deploy:
+  targets:
+    - gguf
+    - ollama
+```
+
+## Development
+
+```bash
+# Clone and install in dev mode
+git clone <repo-url> && cd Adaptron
+pip install -e ".[all,dev]"
+
+# Run the test suite
+pytest --tb=short -q
+
+# Lint
+ruff check adaptron/ tests/
+```
+
+## License
+
+MIT License. See [LICENSE](LICENSE) for details.