anthracite-ai 1.5.0__tar.gz

anthracite_ai-1.5.0/LICENSE

@@ -0,0 +1,15 @@
+Apache License 2.0
+
+Copyright (c) 2026 Nebulix Labs
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

anthracite_ai-1.5.0/MANIFEST.in

@@ -0,0 +1,3 @@
+include README.md
+include LICENSE
+recursive-include anthracite *.py *.typed

anthracite_ai-1.5.0/PKG-INFO

@@ -0,0 +1,465 @@
+Metadata-Version: 2.4
+Name: anthracite-ai
+Version: 1.5.0
+Summary: Universal AI training & fine-tuning framework with smart SFT, external tokenizer support, and HF streaming.
+License: Apache License 2.0
+        
+        Copyright (c) 2026 Nebulix Labs
+        
+        Licensed under the Apache License, Version 2.0 (the "License");
+        you may not use this file except in compliance with the License.
+        You may obtain a copy of the License at
+        
+            http://www.apache.org/licenses/LICENSE-2.0
+        
+        Unless required by applicable law or agreed to in writing, software
+        distributed under the License is distributed on an "AS IS" BASIS,
+        WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+        See the License for the specific language governing permissions and
+        limitations under the License.
+        
+Project-URL: Homepage, https://github.com/your-org/anthracite
+Project-URL: Documentation, https://github.com/your-org/anthracite#readme
+Project-URL: Bug Tracker, https://github.com/your-org/anthracite/issues
+Keywords: ai,deep-learning,llm,training,fine-tuning,tokenizer,nlp
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: safetensors>=0.4
+Requires-Dist: tqdm>=4.65
+Provides-Extra: hf
+Requires-Dist: datasets>=2.14; extra == "hf"
+Requires-Dist: huggingface_hub>=0.20; extra == "hf"
+Requires-Dist: tokenizers>=0.15; extra == "hf"
+Requires-Dist: transformers>=4.36; extra == "hf"
+Provides-Extra: all
+Requires-Dist: datasets>=2.14; extra == "all"
+Requires-Dist: huggingface_hub>=0.20; extra == "all"
+Requires-Dist: tokenizers>=0.15; extra == "all"
+Requires-Dist: transformers>=4.36; extra == "all"
+Dynamic: license-file
+
+# Anthracite v1.5
+
+**Universal AI training & fine-tuning framework** — train text generation and embedding models from scratch or fine-tune them with one Python call.
+
+---
+
+## Table of Contents
+
+1. [Installation](#installation)
+2. [Quick Start](#quick-start)
+3. [Tokenizer Options](#tokenizer-options-new-in-v15)
+4. [Supervised Fine-Tuning (SFT)](#supervised-fine-tuning-sft)
+5. [Dataset Formats](#dataset-formats--smart-key-detection)
+6. [HF Dataset Streaming](#hf-dataset-streaming-new-in-v15)
+7. [Embedding Models](#embedding-models)
+8. [Generation & Inference](#generation--inference)
+9. [Fine-Tuning an Existing Model](#fine-tuning-an-existing-model)
+10. [Configuration Reference](#configuration-reference)
+11. [PyPI Upload Guide](#pypi-upload-guide)
+12. [What Changed in v1.5](#whats-new-in-v15)
+
+---
+
+## Installation
+
+```bash
+pip install anthracite-ai                    # core (PyTorch required separately)
+pip install anthracite-ai[hf]               # + HuggingFace tokenizers, datasets, hub
+pip install anthracite-ai[all]              # everything above
+```
+
+> **PyTorch** is not bundled (too large).  Install it from https://pytorch.org first.
+
+---
+
+## Quick Start
+
+```python
+from anthracite import train, generate
+
+# 1. Train a 20 M parameter text model
+train(
+    model_name  = "MyGPT-20M",
+    dataset     = "my_data.jsonl",   # local file, HF dataset id, or directory
+    tokens      = 100_000_000,        # training token budget
+    params      = "20M",             # target parameter count
+    context_length = 512,
+    tokenizer   = "bundled",         # ← use built-in GPT-2 tokenizer (new v1.5)
+    device      = "auto",
+    precision   = "auto",
+)
+
+# 2. Generate text
+text = generate("./models/MyGPT-20M", "Once upon a time")
+print(text)
+```
+
+---
+
+## Tokenizer Options (new in v1.5)
+
+Anthracite v1.5 gives you **full control** over the tokenizer.  Four modes:
+
+### 1. Train from scratch (default, v1.0 behaviour)
+```python
+train(..., tokenizer=None)   # trains a byte-level BPE tokenizer on your dataset
+```
+
+### 2. Built-in bundled tokenizer
+A GPT-2 style 50 k BPE tokenizer ships with Anthracite.  Use it to skip the tokenizer training step:
+```python
+train(..., tokenizer="bundled")
+```
+
+### 3. HuggingFace tokenizer by repo id
+```python
+# Load by HF Hub repo id
+train(..., tokenizer="gpt2")
+train(..., tokenizer="mistralai/Mistral-7B-v0.1")
+train(..., tokenizer="meta-llama/Llama-2-7b-hf")
+
+# Load a specific file: repo/model/filename
+train(..., tokenizer="bert-base-uncased/tokenizer.json")
+```
+Requires: `pip install transformers` (or `tokenizers` + `huggingface_hub`)
+
+### 4. Local tokenizer directory / file
+```python
+train(..., tokenizer="./my_tokenizer/")     # directory with tokenizer.json
+train(..., tokenizer="./tokenizer.json")    # exact file path
+```
+
+### 5. Live tokenizer object (any library)
+```python
+from transformers import AutoTokenizer
+tok = AutoTokenizer.from_pretrained("gpt2")
+train(..., tokenizer=tok)
+
+# SentencePiece, tiktoken, tokenizers, etc. all work via duck-typing:
+import tiktoken
+enc = tiktoken.get_encoding("cl100k_base")
+train(..., tokenizer=enc)
+```
+
+### Standalone tokenizer loading
+```python
+from anthracite import load_tokenizer
+
+tok = load_tokenizer("bundled")            # built-in
+tok = load_tokenizer("gpt2")              # HF Hub
+tok = load_tokenizer("./my_tokenizer/")   # local path
+
+ids = tok.encode("Hello world!")
+text = tok.decode(ids)
+print(text)   # "Hello world!"
+```
+
+---
+
+## Supervised Fine-Tuning (SFT)
+
+SFT mode trains the model to generate the *output* given the *input*, and masks input tokens from the loss so the model doesn't just learn to repeat the prompt.
+
+```python
+from anthracite import train
+
+train(
+    model_name     = "MyChat",
+    dataset        = "instructions.jsonl",
+    tokens         = 50_000_000,
+    params         = "20M",
+    objective      = "sft",           # ← enable SFT mode
+    sft_mask_input = True,            # mask prompt tokens (default True)
+    sft_format     = "auto",          # auto-detect format (default)
+    tokenizer      = "bundled",
+)
+```
+
+#### Explicit format
+```python
+train(..., sft_format="alpaca")      # Alpaca instruction format
+train(..., sft_format="sharegpt")    # ShareGPT multi-turn
+train(..., sft_format="chatML")      # ChatML messages format
+train(..., sft_format="openai")      # prompt / completion
+train(..., sft_format="qa")          # question / answer
+train(..., sft_format="text")        # raw text, no masking
+```
+
+---
+
+## Dataset Formats & Smart Key Detection
+
+Anthracite v1.5 **automatically detects** your dataset format.  You don't need to specify column names — the `SmartKeyExtractor` peeks at a sample of records, scores every known format, and picks the best match.
+
+### Supported formats
+
+| Format | Keys detected | Input | Output |
+|---|---|---|---|
+| **Alpaca** | `instruction`, `input`, `output` | instruction + input | output |
+| **Alpaca-short** | `instruction`, `output` | instruction | output |
+| **OpenAI** | `prompt`, `completion` | prompt | completion |
+| **Prompt-Response** | `prompt`, `response` | prompt | response |
+| **QA** | `question`, `answer` | question | answer |
+| **Generic IO** | `input`, `output` | input | output |
+| **AI-User** | `user`, `ai` | user | ai |
+| **ShareGPT** | `conversations` list | full conversation | (no mask) |
+| **ChatML** | `messages` list | full conversation | (no mask) |
+| **Raw text** | `text`, `content`, `body` | full text | (no mask) |
+| **2-column generic** | any 2 string keys | first key | second key |
+
+### Alpaca format example
+```jsonl
+{"instruction": "Translate to French.", "input": "Hello world", "output": "Bonjour le monde"}
+{"instruction": "Summarise this.", "input": "Long text...", "output": "Short summary."}
+```
+
+### ShareGPT / ChatML format example
+```jsonl
+{"conversations": [{"from": "human", "value": "Hi"}, {"from": "gpt", "value": "Hello!"}]}
+{"messages": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello!"}]}
+```
+
+### OpenAI / prompt-completion format
+```jsonl
+{"prompt": "Tell me a joke.", "completion": "Why did the chicken..."}
+```
+
+### Raw text format
+```jsonl
+{"text": "Once upon a time in a land far away..."}
+```
+
+---
+
+## HF Dataset Streaming (new in v1.5)
+
+When you pass a Hugging Face dataset id, Anthracite now **streams** the data instead of downloading the whole dataset first.  This means:
+
+- Training starts **immediately** (no multi-GB download wait)
+- Large datasets (100 GB+) fit on machines with little disk space
+- Memory usage stays constant regardless of dataset size
+
+```python
+train(
+    model_name  = "MyGPT",
+    dataset     = "HuggingFaceFW/fineweb",   # 15 TB — streams fine!
+    tokens      = 1_000_000_000,
+    params      = "100M",
+    hf_streaming = True,    # default in v1.5
+)
+```
+
+To download fully (v1.0 behaviour):
+```python
+train(..., hf_streaming=False)
+```
+
+---
+
+## Embedding Models
+
+```python
+from anthracite import train, embed, similarity
+
+# Train a sentence embedding model
+train(
+    model_name  = "MyEmbedder",
+    model_type  = "embedding",
+    dataset     = "corpus.txt",
+    tokens      = 20_000_000,
+    params      = "30M",
+    context_length = 256,
+    objective   = "contrastive",  # or "mlm" for pre-training stage
+    tokenizer   = "bundled",
+)
+
+# Use it
+vecs = embed("./models/MyEmbedder", ["sentence A", "sentence B"])
+score = similarity(vecs[0], vecs[1])
+print(f"Similarity: {score:.3f}")
+```
+
+---
+
+## Generation & Inference
+
+```python
+from anthracite import generate
+
+# Simple generation
+text = generate("./models/MyGPT-20M", "The quick brown fox")
+print(text)
+
+# With options
+text = generate(
+    "./models/MyGPT-20M",
+    "Tell me about space",
+    max_new_tokens = 300,
+    temperature    = 0.8,
+    top_p          = 0.9,
+)
+```
+
+---
+
+## Fine-Tuning an Existing Model
+
+```python
+from anthracite import finetune
+
+finetune(
+    model      = "./models/MyGPT-20M",     # base model path or HF repo id
+    dataset    = "instructions.jsonl",
+    tokens     = 10_000_000,
+    objective  = "sft",
+    tokenizer  = None,   # None = use base model's tokenizer (default)
+    output_dir = "./models/MyGPT-20M-SFT",
+)
+```
+
+You can also fine-tune from a HuggingFace Hub model:
+```python
+finetune(
+    model   = "your-username/MyGPT-20M",   # HF Hub repo id
+    dataset = "instructions.jsonl",
+    tokens  = 5_000_000,
+)
+```
+
+---
+
+## Configuration Reference
+
+All parameters for `train()` and `finetune()`:
+
+| Parameter | Default | Description |
+|---|---|---|
+| `model_name` | `"anthracite-model"` | Output model name / directory |
+| `model_type` | `"text_gen"` | `"text_gen"` or `"embedding"` |
+| `dataset` | — | File path, HF dataset id, or list of strings |
+| `tokens` | `10_000_000` | Training token budget (int or `"100M"`) |
+| `params` | `"20M"` | Target parameters (int or `"20M"`) |
+| `context_length` | `512` | Sequence length |
+| `vocab_size` | `32768` | Vocabulary size (when training tokenizer from scratch) |
+| `tokenizer` | `None` | `None` / `"bundled"` / HF id / path / object |
+| `device` | `"auto"` | `"auto"` / `"cuda"` / `"mps"` / `"cpu"` |
+| `precision` | `"auto"` | `"auto"` / `"fp32"` / `"bf16"` / `"fp16"` |
+| `batch_size` | `"auto"` | Effective batch size or `"auto"` |
+| `learning_rate` | `3e-4` | Peak learning rate |
+| `output_dir` | `./models/<name>` | Where to save the model |
+| `objective` | `"auto"` | `"auto"` / `"causal_lm"` / `"mlm"` / `"contrastive"` / `"sft"` |
+| `sft_mask_input` | `True` | Mask prompt tokens from loss (SFT only) |
+| `sft_format` | `"auto"` | Dataset format hint (SFT only) |
+| `hf_streaming` | `True` | Stream HF datasets (don't download fully) |
+| `checkpoint_interval` | `5000` | Save checkpoint every N steps |
+| `keep_last_checkpoints` | `3` | Number of recent checkpoints to keep |
+| `seed` | `42` | Random seed |
+| `val_split` | `0.01` | Fraction of data held out for validation |
+| `gradient_accumulation` | `None` | Accumulation steps (auto if `None`) |
+| `weight_decay` | `0.1` | AdamW weight decay |
+| `grad_clip` | `1.0` | Gradient clipping norm |
+| `warmup_ratio` | `0.02` | Fraction of steps used for LR warm-up |
+| `num_workers` | `0` | DataLoader worker processes |
+
+---
+
+## PyPI Upload Guide
+
+Follow these steps to publish your own fork or a new release to PyPI.
+
+### 1. Install build tools
+```bash
+pip install build twine
+```
+
+### 2. Build the package
+```bash
+cd anthracite-pkg        # the directory containing pyproject.toml
+python -m build          # creates dist/anthracite_ai-1.5.0.tar.gz and .whl
+```
+
+### 3. Test on TestPyPI first (recommended)
+```bash
+# Upload to TestPyPI
+twine upload --repository testpypi dist/*
+
+# Install from TestPyPI to verify
+pip install --index-url https://test.pypi.org/simple/ anthracite-ai
+```
+
+### 4. Upload to PyPI
+```bash
+twine upload dist/*
+# Enter your PyPI username and password (or API token)
+```
+
+### 5. Using an API token (safer than password)
+```bash
+# Create a token at https://pypi.org/manage/account/token/
+twine upload dist/* -u __token__ -p pypi-<your-token-here>
+```
+
+### 6. Automate with GitHub Actions
+Create `.github/workflows/publish.yml`:
+```yaml
+name: Publish to PyPI
+on:
+  release:
+    types: [published]
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: { python-version: "3.11" }
+      - run: pip install build twine
+      - run: python -m build
+      - run: twine upload dist/*
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+```
+
+---
+
+## What's New in v1.5
+
+### Tokenizer overhaul
+- **`tokenizer="bundled"`** — GPT-2 tokenizer ships with Anthracite, no training needed.
+- **External tokenizer support** — load from HF Hub, local path, or pass any live tokenizer object (HuggingFace, tiktoken, SentencePiece, etc.).
+- **Bug fix** — `encode()` and `decode()` no longer require `self` to be passed manually; the tokenizer works correctly as a standalone object.
+
+### Smart SFT format detection (`SmartKeyExtractor`)
+- Automatically detects Alpaca, ShareGPT, ChatML, OpenAI, QA, and 10+ other formats.
+- Correctly identifies `input` (prompt) vs `output` (answer) fields.
+- Masks input tokens from the training loss so the model learns to generate answers, not repeat prompts.
+
+### HF Dataset streaming
+- Hugging Face datasets stream by default (`hf_streaming=True`).
+- Training starts immediately without downloading multi-GB datasets.
+- Works with any HF dataset, including terabyte-scale ones.
+
+### PyPI-ready packaging
+- `pyproject.toml` with proper metadata, optional dependencies, and entry points.
+- `pip install anthracite-ai` and `pip install anthracite-ai[hf]` work out of the box.
+
+---
+
+## License
+
+MIT License — see `LICENSE` for details.