inferkit 0.1.0__tar.gz

inferkit-0.1.0/.gitignore

@@ -0,0 +1,41 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.eggs/
+
+# Virtual environments
+.venv/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Caches and temp files
+.cache/
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+*.tmp
+*.bak
+
+# Environment
+.env
+
+# Artifacts
+*.slm
+
+# Development notes
+.dev/
+
+# Github Copilot
+.github/instructions/
+.github/copilot-instructions.md

inferkit-0.1.0/PKG-INFO

@@ -0,0 +1,191 @@
+Metadata-Version: 2.4
+Name: inferkit
+Version: 0.1.0
+Summary: Embeddable inference engine for small language models — the SQLite for AI
+Author-email: Milad Olad <milad.olad@gmail.com>
+License-Expression: MIT
+Keywords: ai,edge,embedding,inference,llm,local,offline,slm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: huggingface-hub>=0.25
+Requires-Dist: llama-cpp-python>=0.3
+Requires-Dist: tokenizers>=0.21
+Requires-Dist: typer>=0.15
+Provides-Extra: mlx
+Requires-Dist: mlx-lm>=0.31; extra == 'mlx'
+Description-Content-Type: text/markdown
+
+# slmkit
+
+Package small language models into self-contained, embeddable `.slm` artifacts.
+
+**SQLite for AI** — one file, zero config, zero internet, 5ms inference.
+
+```python
+import slmkit
+
+classifier = slmkit.load("sentiment.slm")
+result = classifier("The aurora was breathtaking")
+# → {"label": "positive", "confidence": 1.0}
+```
+
+## Install
+
+```bash
+pip install slmkit
+```
+
+Requires Python 3.11+. The `llama-cpp-python` dependency compiles C++ on first install, which may take a few minutes.
+
+## Quickstart
+
+### 1. Compile a model
+
+Package a pre-quantized GGUF model from HuggingFace into a `.slm` artifact:
+
+```bash
+slm compile \
+  --source Qwen/Qwen2.5-0.5B-Instruct-GGUF \
+  --task classification \
+  --labels "positive,negative,neutral" \
+  --quantize q4_k_m \
+  --output sentiment.slm
+```
+
+This downloads the model (~490 MB), bakes in a task-specific prompt, and packages everything into a single file.
+
+You can also pass a local `.gguf` file as `--source` instead of a HuggingFace repo.
+
+### 2. Load and run
+
+```python
+import slmkit
+
+with slmkit.load("sentiment.slm") as model:
+    result = model("The midnight sun over Tromsø was unforgettable")
+    print(result)
+    # {"label": "positive", "confidence": 1.0}
+
+    results = model.batch([
+        "The recipe was a disaster",
+        "I finally fixed the timing belt myself",
+    ])
+    # [{"label": "negative", ...}, {"label": "positive", ...}]
+```
+
+### 3. Test and inspect
+
+```bash
+# Run inference from the CLI
+slm test sentiment.slm --input "The coral reef was thriving"
+
+# View artifact metadata
+slm info sentiment.slm
+```
+
+## Validation
+
+Bake test cases into the artifact at compile time:
+
+```bash
+slm compile \
+  --source Qwen/Qwen2.5-0.5B-Instruct-GGUF \
+  --task classification \
+  --labels "positive,negative,neutral" \
+  --quantize q4_k_m \
+  --validation tests.json \
+  --output sentiment.slm
+```
+
+Where `tests.json` contains:
+
+```json
+[
+  {"input": "The view from the summit was breathtaking", "expected_label": "positive"},
+  {"input": "The bridge collapsed during rush hour", "expected_label": "negative"}
+]
+```
+
+Then run the validation suite:
+
+```bash
+slm test sentiment.slm
+```
+
+## The `.slm` format
+
+A `.slm` file is a ZIP archive with a fixed structure:
+
+```
+sentiment.slm (ZIP)
+├── model.gguf            # Quantized model weights
+├── manifest.json         # Task type, backend, labels, source info
+├── prompt_template.txt   # Baked-in system prompt
+├── tokenizer.json        # Tokenizer config (optional)
+└── validation.json       # Test cases (optional)
+```
+
+One file. Everything included. Ship it, load it, call it.
+
+## Current scope
+
+This is an early release focused on proving the core workflow:
+
+- **Backends**: llama-cpp-python (default), MLX (Apple Silicon, `pip install slmkit[mlx]`)
+- **Task**: classification
+- **Models**: pre-quantized GGUF or MLX-format models from HuggingFace
+
+### MLX on Apple Silicon
+
+For Metal-accelerated inference on M-series Macs:
+
+```bash
+pip install slmkit[mlx]
+
+slm compile \
+  --source mlx-community/Qwen2.5-0.5B-Instruct-4bit \
+  --task classification \
+  --labels "positive,negative,neutral" \
+  --backend mlx \
+  --output sentiment-mlx.slm
+```
+
+The backend is auto-detected from the model source, or set explicitly with `--backend`.
+
+## Examples
+
+Runnable examples in [`examples/`](examples/):
+
+### Classification
+
+| Example | Model | Size | Description |
+|---------|-------|------|-------------|
+| [sentiment.py](examples/classification/sentiment.py) | Qwen2.5-0.5B | ~490 MB | Restaurant review sentiment |
+| [moderation.py](examples/classification/moderation.py) | Qwen2.5-0.5B | ~490 MB | Content moderation with validation |
+| [spam_detection.py](examples/classification/spam_detection.py) | SmolLM2-360M | ~300 MB | Binary spam vs ham |
+| [urgency_triage.py](examples/classification/urgency_triage.py) | Llama-3.2-1B | ~700 MB | Priority routing |
+| [language_detection.py](examples/classification/language_detection.py) | Qwen2.5-1.5B | ~1 GB | Six-language detection |
+| [topic_categorization.py](examples/classification/topic_categorization.py) | Gemma-2-2B | ~1.5 GB | Article topic sorting |
+| [tone_detection.py](examples/classification/tone_detection.py) | Phi-3.5-Mini | ~2.2 GB | Tone/intent classification |
+
+### MLX (Apple Silicon)
+
+| Example | Model | Size | Description |
+|---------|-------|------|-------------|
+| [sentiment.py](examples/mlx/sentiment.py) | Qwen2.5-0.5B-4bit | ~243 MB | Metal-accelerated sentiment |
+
+```bash
+uv run python examples/classification/sentiment.py
+```
+
+## License
+
+MIT

inferkit-0.1.0/README.md

@@ -0,0 +1,166 @@
+# slmkit
+
+Package small language models into self-contained, embeddable `.slm` artifacts.
+
+**SQLite for AI** — one file, zero config, zero internet, 5ms inference.
+
+```python
+import slmkit
+
+classifier = slmkit.load("sentiment.slm")
+result = classifier("The aurora was breathtaking")
+# → {"label": "positive", "confidence": 1.0}
+```
+
+## Install
+
+```bash
+pip install slmkit
+```
+
+Requires Python 3.11+. The `llama-cpp-python` dependency compiles C++ on first install, which may take a few minutes.
+
+## Quickstart
+
+### 1. Compile a model
+
+Package a pre-quantized GGUF model from HuggingFace into a `.slm` artifact:
+
+```bash
+slm compile \
+  --source Qwen/Qwen2.5-0.5B-Instruct-GGUF \
+  --task classification \
+  --labels "positive,negative,neutral" \
+  --quantize q4_k_m \
+  --output sentiment.slm
+```
+
+This downloads the model (~490 MB), bakes in a task-specific prompt, and packages everything into a single file.
+
+You can also pass a local `.gguf` file as `--source` instead of a HuggingFace repo.
+
+### 2. Load and run
+
+```python
+import slmkit
+
+with slmkit.load("sentiment.slm") as model:
+    result = model("The midnight sun over Tromsø was unforgettable")
+    print(result)
+    # {"label": "positive", "confidence": 1.0}
+
+    results = model.batch([
+        "The recipe was a disaster",
+        "I finally fixed the timing belt myself",
+    ])
+    # [{"label": "negative", ...}, {"label": "positive", ...}]
+```
+
+### 3. Test and inspect
+
+```bash
+# Run inference from the CLI
+slm test sentiment.slm --input "The coral reef was thriving"
+
+# View artifact metadata
+slm info sentiment.slm
+```
+
+## Validation
+
+Bake test cases into the artifact at compile time:
+
+```bash
+slm compile \
+  --source Qwen/Qwen2.5-0.5B-Instruct-GGUF \
+  --task classification \
+  --labels "positive,negative,neutral" \
+  --quantize q4_k_m \
+  --validation tests.json \
+  --output sentiment.slm
+```
+
+Where `tests.json` contains:
+
+```json
+[
+  {"input": "The view from the summit was breathtaking", "expected_label": "positive"},
+  {"input": "The bridge collapsed during rush hour", "expected_label": "negative"}
+]
+```
+
+Then run the validation suite:
+
+```bash
+slm test sentiment.slm
+```
+
+## The `.slm` format
+
+A `.slm` file is a ZIP archive with a fixed structure:
+
+```
+sentiment.slm (ZIP)
+├── model.gguf            # Quantized model weights
+├── manifest.json         # Task type, backend, labels, source info
+├── prompt_template.txt   # Baked-in system prompt
+├── tokenizer.json        # Tokenizer config (optional)
+└── validation.json       # Test cases (optional)
+```
+
+One file. Everything included. Ship it, load it, call it.
+
+## Current scope
+
+This is an early release focused on proving the core workflow:
+
+- **Backends**: llama-cpp-python (default), MLX (Apple Silicon, `pip install slmkit[mlx]`)
+- **Task**: classification
+- **Models**: pre-quantized GGUF or MLX-format models from HuggingFace
+
+### MLX on Apple Silicon
+
+For Metal-accelerated inference on M-series Macs:
+
+```bash
+pip install slmkit[mlx]
+
+slm compile \
+  --source mlx-community/Qwen2.5-0.5B-Instruct-4bit \
+  --task classification \
+  --labels "positive,negative,neutral" \
+  --backend mlx \
+  --output sentiment-mlx.slm
+```
+
+The backend is auto-detected from the model source, or set explicitly with `--backend`.
+
+## Examples
+
+Runnable examples in [`examples/`](examples/):
+
+### Classification
+
+| Example | Model | Size | Description |
+|---------|-------|------|-------------|
+| [sentiment.py](examples/classification/sentiment.py) | Qwen2.5-0.5B | ~490 MB | Restaurant review sentiment |
+| [moderation.py](examples/classification/moderation.py) | Qwen2.5-0.5B | ~490 MB | Content moderation with validation |
+| [spam_detection.py](examples/classification/spam_detection.py) | SmolLM2-360M | ~300 MB | Binary spam vs ham |
+| [urgency_triage.py](examples/classification/urgency_triage.py) | Llama-3.2-1B | ~700 MB | Priority routing |
+| [language_detection.py](examples/classification/language_detection.py) | Qwen2.5-1.5B | ~1 GB | Six-language detection |
+| [topic_categorization.py](examples/classification/topic_categorization.py) | Gemma-2-2B | ~1.5 GB | Article topic sorting |
+| [tone_detection.py](examples/classification/tone_detection.py) | Phi-3.5-Mini | ~2.2 GB | Tone/intent classification |
+
+### MLX (Apple Silicon)
+
+| Example | Model | Size | Description |
+|---------|-------|------|-------------|
+| [sentiment.py](examples/mlx/sentiment.py) | Qwen2.5-0.5B-4bit | ~243 MB | Metal-accelerated sentiment |
+
+```bash
+uv run python examples/classification/sentiment.py
+```
+
+## License
+
+MIT

inferkit-0.1.0/examples/.gitignore

@@ -0,0 +1 @@
+*.slm

inferkit-0.1.0/examples/classification/language_detection.py

@@ -0,0 +1,59 @@
+"""Compile and run a language detection classifier.
+
+Identifies the language of short text snippets across six languages.
+Demonstrates multi-class classification with non-English inputs.
+Uses Qwen2.5-1.5B for stronger multilingual understanding.
+
+Usage:
+    uv run python examples/classification/language_detection.py
+"""
+
+from pathlib import Path
+
+import slmkit
+from slmkit.compiler import compile_model
+
+_HERE = Path(__file__).parent
+ARTIFACT = _HERE / "qwen1.5b-language.slm"
+SOURCE = "Qwen/Qwen2.5-1.5B-Instruct-GGUF"  # ~1 GB — good multilingual coverage
+LABELS = ["english", "spanish", "french", "german", "italian", "portuguese"]
+
+SAMPLES = [
+    "The James Webb telescope captured infrared light from a galaxy 13 billion years old",
+    "El volcán Popocatépetl lanzó cenizas sobre los pueblos cercanos esta mañana",
+    "Les vendanges en Bourgogne ont commencé plus tôt que prévu cette année",
+    "Der Biergarten am Englischen Garten war trotz Regen gut besucht",
+    "La gondola scivolava silenziosa lungo il Canal Grande al tramonto",
+    "O carnaval de Olinda atrai milhões de foliões todos os anos",
+    "A herd of elephants crossed the Zambezi at dawn",
+    "La receta original del mole poblano lleva más de veinte ingredientes",
+]
+
+
+def main() -> None:
+    if not ARTIFACT.exists():
+        print(f"Compiling {SOURCE} → {ARTIFACT} ...")
+        compile_model(
+            SOURCE,
+            task="classification",
+            output=str(ARTIFACT),
+            labels=LABELS,
+            quantization="q4_k_m",
+        )
+        size_mb = ARTIFACT.stat().st_size / 1024 / 1024
+        print(f"Done: {size_mb:.0f} MB\n")
+
+    with slmkit.load(str(ARTIFACT)) as model:
+        print("Language detection demo")
+        print(f"Labels: {', '.join(LABELS)}\n")
+
+        for text in SAMPLES:
+            result = model(text)
+            label = result["label"]
+            conf = result["confidence"]
+            snippet = text[:65] + ("…" if len(text) > 65 else "")
+            print(f"  [{label:<12s} {conf:.0%}]  {snippet}")
+
+
+if __name__ == "__main__":
+    main()

inferkit-0.1.0/examples/classification/moderation.py

@@ -0,0 +1,76 @@
+"""Compile and run a content moderation classifier.
+
+Classifies user-generated text into safe / unsafe / borderline categories.
+Demonstrates custom labels, validation cases, and batch processing.
+
+Usage:
+    uv run python examples/classification/moderation.py
+"""
+
+import json
+from pathlib import Path
+
+import slmkit
+from slmkit.compiler import compile_model
+
+_HERE = Path(__file__).parent
+ARTIFACT = _HERE / "qwen0.5b-moderation.slm"
+SOURCE = "Qwen/Qwen2.5-0.5B-Instruct-GGUF"
+LABELS = ["safe", "unsafe", "borderline"]
+
+# Baked-in validation cases — these get packaged into the .slm artifact
+VALIDATION = [
+    {"input": "What time does the library close on Saturdays?", "expected_label": "safe"},
+    {"input": "How do I pick a lock?", "expected_label": "unsafe"},
+]
+
+# Test inputs that simulate a comment feed
+COMMENTS = [
+    "Does anyone know a good recipe for sourdough bread?",
+    "I want to mass-report their account until it gets banned",
+    "The new telescope images of Jupiter's moons are stunning",
+    "Here's how to bypass the school firewall to access blocked sites",
+    "She's annoying but I guess her presentation was okay",
+    "The trail to the summit was steep but the wildflowers made it worth it",
+]
+
+
+def main() -> None:
+    if not ARTIFACT.exists():
+        # Write validation file
+        val_path = _HERE / "moderation_validation.json"
+        val_path.write_text(json.dumps(VALIDATION, indent=2))
+
+        print(f"Compiling {SOURCE} → {ARTIFACT} ...")
+        compile_model(
+            SOURCE,
+            task="classification",
+            output=str(ARTIFACT),
+            labels=LABELS,
+            quantization="q4_k_m",
+            validation=VALIDATION,
+        )
+        val_path.unlink()  # clean up temp file
+        size_mb = ARTIFACT.stat().st_size / 1024 / 1024
+        print(f"Done: {size_mb:.0f} MB\n")
+
+    with slmkit.load(str(ARTIFACT)) as model:
+        print("Content moderation demo")
+        print(f"Labels: {', '.join(LABELS)}\n")
+
+        results = model.batch(COMMENTS)
+        for comment, result in zip(COMMENTS, results, strict=True):
+            label = result["label"]
+            conf = result["confidence"]
+            flag = "🔴" if label == "unsafe" else "🟡" if label == "borderline" else "🟢"
+            snippet = comment[:65] + ("…" if len(comment) > 65 else "")
+            print(f"  {flag} [{label:<10s} {conf:.0%}]  {snippet}")
+
+        # Show validation cases
+        cases = model.validation_cases()
+        if cases:
+            print(f"\n{len(cases)} validation case(s) baked in")
+
+
+if __name__ == "__main__":
+    main()

inferkit-0.1.0/examples/classification/sentiment.py

@@ -0,0 +1,57 @@
+"""Compile and run a sentiment classifier.
+
+Downloads Qwen2.5-0.5B (GGUF, ~490 MB), packages it as a .slm artifact,
+then classifies a batch of restaurant review snippets.
+
+Usage:
+    uv run python examples/classification/sentiment.py
+"""
+
+from pathlib import Path
+
+import slmkit
+from slmkit.compiler import compile_model
+
+_HERE = Path(__file__).parent
+ARTIFACT = _HERE / "qwen0.5b-sentiment.slm"
+SOURCE = "Qwen/Qwen2.5-0.5B-Instruct-GGUF"
+LABELS = ["positive", "negative", "neutral"]
+
+REVIEWS = [
+    "The wood-fired pizza had a perfect char and the burrata was impossibly fresh",
+    "We waited 45 minutes for cold pasta that tasted like it came from a freezer",
+    "Decent sushi, nothing wrong with it but nothing memorable either",
+    "The sommelier paired a 2019 Barolo with our osso buco — genuinely transcendent",
+    "They forgot our reservation, seated us by the kitchen, and overcharged us",
+]
+
+
+def main() -> None:
+    # Compile (skips download if already cached by HuggingFace)
+    if not ARTIFACT.exists():
+        print(f"Compiling {SOURCE} → {ARTIFACT} ...")
+        compile_model(
+            SOURCE,
+            task="classification",
+            output=str(ARTIFACT),
+            labels=LABELS,
+            quantization="q4_k_m",
+        )
+        size_mb = ARTIFACT.stat().st_size / 1024 / 1024
+        print(f"Done: {size_mb:.0f} MB\n")
+
+    # Load and classify
+    with slmkit.load(str(ARTIFACT)) as model:
+        print(f"Model: {model.info()['source']}")
+        print(f"Labels: {', '.join(LABELS)}\n")
+
+        for review in REVIEWS:
+            result = model(review)
+            label = result["label"]
+            conf = result["confidence"]
+            snippet = review[:72] + ("…" if len(review) > 72 else "")
+            print(f"  [{label:<8s} {conf:.0%}]  {snippet}")
+
+
+if __name__ == "__main__":
+    main()

inferkit-0.1.0/examples/classification/spam_detection.py

@@ -0,0 +1,60 @@
+"""Compile and run a spam detection classifier.
+
+Binary spam vs ham classifier for short messages — demonstrates the
+simplest possible label set (two classes).
+Uses Qwen2.5-0.5B — reliable for binary tasks.
+
+Usage:
+    uv run python examples/classification/spam_detection.py
+"""
+
+from pathlib import Path
+
+import slmkit
+from slmkit.compiler import compile_model
+
+_HERE = Path(__file__).parent
+ARTIFACT = _HERE / "qwen0.5b-spam.slm"
+SOURCE = "Qwen/Qwen2.5-0.5B-Instruct-GGUF"  # ~490 MB — reliable for binary tasks
+LABELS = ["ham", "spam"]
+
+MESSAGES = [
+    "Hey, are we still meeting at the trailhead at 7am tomorrow?",
+    "CONGRATULATIONS! You've been selected to receive a $1,000 gift card. Click here NOW!",
+    "The quarterly review has been moved to Thursday — same room, same time",
+    "Make $5,000/week from home with this ONE WEIRD TRICK. Limited spots available!!!",
+    "Can you pick up olive oil and sourdough on your way home?",
+    "URGENT: Your account will be suspended unless you verify your identity immediately",
+    "The new batch of honey from the rooftop hives is ready for bottling",
+    "You have (1) unread message from a secret admirer. Open now before it expires!",
+]
+
+
+def main() -> None:
+    if not ARTIFACT.exists():
+        print(f"Compiling {SOURCE} → {ARTIFACT} ...")
+        compile_model(
+            SOURCE,
+            task="classification",
+            output=str(ARTIFACT),
+            labels=LABELS,
+            quantization="q4_k_m",
+        )
+        size_mb = ARTIFACT.stat().st_size / 1024 / 1024
+        print(f"Done: {size_mb:.0f} MB\n")
+
+    with slmkit.load(str(ARTIFACT)) as model:
+        print("Spam detection demo")
+        print(f"Labels: {', '.join(LABELS)}\n")
+
+        for msg in MESSAGES:
+            result = model(msg)
+            label = result["label"]
+            conf = result["confidence"]
+            icon = "🚫" if label == "spam" else "✉️"
+            snippet = msg[:65] + ("…" if len(msg) > 65 else "")
+            print(f"  {icon} [{label:<4s} {conf:.0%}]  {snippet}")
+
+
+if __name__ == "__main__":
+    main()