pylolm 0.1.0__tar.gz

pylolm-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 AuraWay
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pylolm-0.1.0/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.4
+Name: pylolm
+Version: 0.1.0
+Summary: LoLm: run GGUF language models on-device on Android from Python (llama.cpp bindings for Kivy, Flet and Termux).
+Author-email: AuraWay <auraway.dev@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/aura-way/pylolm
+Project-URL: Issues, https://github.com/aura-way/pylolm/issues
+Project-URL: Changelog, https://github.com/aura-way/pylolm/blob/main/CHANGELOG.md
+Keywords: llm,llama.cpp,gguf,android,termux,kivy,flet,on-device,local-llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: Android
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: C
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# pylolm (LoLm)
+
+**Local LLMs on Android, from Python.** An Android-friendly take on `llama-cpp-python`, built on
+[llama.cpp](https://github.com/ggml-org/llama.cpp) and designed for **Kivy**, **Flet** and **Termux**.
+
+- Zero Python dependencies: a thin `ctypes` layer over a small C shim.
+- `llama-cpp-python`-style API: `Llama(...)`, `create_chat_completion(...)`, `create_completion(...)`.
+- Streaming generators, stop strings, a `stop()` button for your UI.
+- Multi-turn chat that reuses the KV cache, so long conversations stay fast.
+- Aimed at small GGUF models (0.5B-2B) on phones.
+
+> **Status: alpha (0.1.0).** Text generation only. TTS / STT / OCR are on the roadmap.
+
+## Install
+
+```bash
+pip install pylolm
+pylolm get-libs      # downloads the prebuilt native libraries (seconds)
+pylolm doctor        # checks that everything loads
+```
+
+`get-libs` fetches the Android arm64 libraries attached to the matching GitHub Release, checks their
+SHA-256 checksum, and installs them to `~/.pylolm/lib`. These are built by CI to run on a wide range
+of phones.
+
+Prefer to compile for your exact device, or `get-libs` is not available for your setup?
+
+```bash
+pkg install cmake clang git
+pylolm build-native  # clones llama.cpp and builds on the device (slow: ~30 min)
+```
+
+Libraries are searched in `$PYLOLM_LIB_DIR`, then `pylolm/lib/`, then `~/.pylolm/lib/`.
+
+## Quick start
+
+```python
+from pylolm import Llama
+
+llm = Llama("/sdcard/models/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf", n_ctx=2048)
+
+messages = [{"role": "user", "content": "Explain GGUF in two sentences."}]
+
+# whole answer
+out = llm.create_chat_completion(messages, max_tokens=200)
+print(out["choices"][0]["message"]["content"])
+
+# streaming
+for chunk in llm.create_chat_completion(messages, max_tokens=200, stream=True):
+    piece = chunk["choices"][0]["delta"].get("content")
+    if piece:
+        print(piece, end="", flush=True)
+```
+
+Try it from the terminal: `pylolm chat /path/to/model.gguf`.
+
+## API
+
+| Call | Notes |
+|---|---|
+| `Llama(model_path, n_ctx=2048, n_threads=None, verbose=False)` | `n_threads` defaults to `min(4, cpu_count)` |
+| `create_chat_completion(messages, max_tokens=256, temperature=0.7, top_p=0.95, top_k=40, min_p=0.05, repeat_penalty=1.0, seed=None, stop=None, stream=False, chat_template=None)` | OpenAI-style dict, or chunk iterator with `stream=True` |
+| `create_completion(prompt, ...)` / `llm(prompt, ...)` | raw text completion |
+| `stop()` | stop the running generation from another thread |
+| `reset()` | clear the KV cache (forget the conversation) |
+| `count_tokens(text)` | tokens used by a string |
+| `close()` / `with Llama(...) as llm:` | free the model |
+
+Errors: `LibraryNotFoundError`, `ModelLoadError`, `ContextOverflowError`, `GenerationError`
+(all subclass `PyLoLmError`).
+
+`usage` in a non-streaming result includes `cached_prompt_tokens`: how many prompt tokens were
+reused from the previous turn.
+
+**Chat templates.** The model's built-in template is applied through llama.cpp's template
+support, which covers the common families (ChatML/Qwen, Llama 3, Gemma, ...) but is not a full
+Jinja engine. If a newer model's template is rejected, pass `chat_template="chatml"` (or another
+known name) or build the prompt yourself and call `create_completion`.
+
+## Using it in apps
+
+**UI threads.** Generation blocks, so run it off the UI thread. A streaming generator is easiest:
+
+```python
+import threading
+
+def worker():
+    for chunk in llm.create_chat_completion(messages, stream=True):
+        text = chunk["choices"][0]["delta"].get("content")
+        if text:
+            Clock.schedule_once(lambda dt, t=text: append_to_label(t))  # Kivy
+            # page.run_thread / page.update(...) for Flet
+
+threading.Thread(target=worker, daemon=True).start()
+```
+
+Call `llm.stop()` from your Stop button; breaking out of the loop also stops generation.
+
+**Flet.** `flet build apk` installs packages from PyPI using the Android platform tag, so the
+`android_24_arm64_v8a` wheel is what ships the native libraries. This path needs on-device
+verification for your Flet/Python version.
+
+**Kivy / python-for-android.** p4a does not consume Android wheels yet. For now, bundle the
+`.so` files with your app and point `PYLOLM_LIB_DIR` at them before importing pylolm:
+
+```python
+import os
+os.environ["PYLOLM_LIB_DIR"] = os.path.join(os.path.dirname(__file__), "pylolm_libs")
+import pylolm
+```
+
+A proper p4a recipe is on the roadmap.
+
+## Building from source
+
+```bash
+git clone https://github.com/auraway/pylolm && cd pylolm
+pip install -e ".[dev]"
+python -m pytest                                   # pure-Python tests, no model needed
+PYLOLM_MODEL=/path/model.gguf python -m pytest     # + integration tests
+```
+
+Native libraries: `src/pylolm/native/build_termux.sh` (on a phone) or `build_android_ndk.sh`
+(cross-compile). The llama.cpp version is pinned in `src/pylolm/native/LLAMA_CPP_REF`.
+
+## Releasing to PyPI
+
+1. Set `LLAMA_CPP_REF` to a llama.cpp tag/commit you have tested.
+2. Bump `version` in `pyproject.toml` and add a `CHANGELOG.md` entry.
+3. On PyPI, add a [trusted publisher](https://docs.pypi.org/trusted-publishers/) for this repo
+   (workflow `release.yml`, environment `pypi`).
+4. `git tag v0.1.0 && git push --tags`. CI builds the sdist, a pure wheel, and the Android arm64
+   wheel, publishes them to PyPI, and attaches `pylolm-libs-android-arm64-v8a.tar.gz` (+ `.sha256`)
+   to the GitHub Release for `pylolm get-libs`. The tag must match the version in `pyproject.toml`.
+
+## Roadmap
+
+- p4a recipe for Kivy; verified Flet build
+- whisper.cpp (speech-to-text) module
+- TTS and OCR modules
+- GPU backends (Vulkan/OpenCL) as an opt-in
+
+## License
+
+MIT. llama.cpp is MIT-licensed too; model licenses are your responsibility.

pylolm-0.1.0/README.md

@@ -0,0 +1,145 @@
+# pylolm (LoLm)
+
+**Local LLMs on Android, from Python.** An Android-friendly take on `llama-cpp-python`, built on
+[llama.cpp](https://github.com/ggml-org/llama.cpp) and designed for **Kivy**, **Flet** and **Termux**.
+
+- Zero Python dependencies: a thin `ctypes` layer over a small C shim.
+- `llama-cpp-python`-style API: `Llama(...)`, `create_chat_completion(...)`, `create_completion(...)`.
+- Streaming generators, stop strings, a `stop()` button for your UI.
+- Multi-turn chat that reuses the KV cache, so long conversations stay fast.
+- Aimed at small GGUF models (0.5B-2B) on phones.
+
+> **Status: alpha (0.1.0).** Text generation only. TTS / STT / OCR are on the roadmap.
+
+## Install
+
+```bash
+pip install pylolm
+pylolm get-libs      # downloads the prebuilt native libraries (seconds)
+pylolm doctor        # checks that everything loads
+```
+
+`get-libs` fetches the Android arm64 libraries attached to the matching GitHub Release, checks their
+SHA-256 checksum, and installs them to `~/.pylolm/lib`. These are built by CI to run on a wide range
+of phones.
+
+Prefer to compile for your exact device, or `get-libs` is not available for your setup?
+
+```bash
+pkg install cmake clang git
+pylolm build-native  # clones llama.cpp and builds on the device (slow: ~30 min)
+```
+
+Libraries are searched in `$PYLOLM_LIB_DIR`, then `pylolm/lib/`, then `~/.pylolm/lib/`.
+
+## Quick start
+
+```python
+from pylolm import Llama
+
+llm = Llama("/sdcard/models/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf", n_ctx=2048)
+
+messages = [{"role": "user", "content": "Explain GGUF in two sentences."}]
+
+# whole answer
+out = llm.create_chat_completion(messages, max_tokens=200)
+print(out["choices"][0]["message"]["content"])
+
+# streaming
+for chunk in llm.create_chat_completion(messages, max_tokens=200, stream=True):
+    piece = chunk["choices"][0]["delta"].get("content")
+    if piece:
+        print(piece, end="", flush=True)
+```
+
+Try it from the terminal: `pylolm chat /path/to/model.gguf`.
+
+## API
+
+| Call | Notes |
+|---|---|
+| `Llama(model_path, n_ctx=2048, n_threads=None, verbose=False)` | `n_threads` defaults to `min(4, cpu_count)` |
+| `create_chat_completion(messages, max_tokens=256, temperature=0.7, top_p=0.95, top_k=40, min_p=0.05, repeat_penalty=1.0, seed=None, stop=None, stream=False, chat_template=None)` | OpenAI-style dict, or chunk iterator with `stream=True` |
+| `create_completion(prompt, ...)` / `llm(prompt, ...)` | raw text completion |
+| `stop()` | stop the running generation from another thread |
+| `reset()` | clear the KV cache (forget the conversation) |
+| `count_tokens(text)` | tokens used by a string |
+| `close()` / `with Llama(...) as llm:` | free the model |
+
+Errors: `LibraryNotFoundError`, `ModelLoadError`, `ContextOverflowError`, `GenerationError`
+(all subclass `PyLoLmError`).
+
+`usage` in a non-streaming result includes `cached_prompt_tokens`: how many prompt tokens were
+reused from the previous turn.
+
+**Chat templates.** The model's built-in template is applied through llama.cpp's template
+support, which covers the common families (ChatML/Qwen, Llama 3, Gemma, ...) but is not a full
+Jinja engine. If a newer model's template is rejected, pass `chat_template="chatml"` (or another
+known name) or build the prompt yourself and call `create_completion`.
+
+## Using it in apps
+
+**UI threads.** Generation blocks, so run it off the UI thread. A streaming generator is easiest:
+
+```python
+import threading
+
+def worker():
+    for chunk in llm.create_chat_completion(messages, stream=True):
+        text = chunk["choices"][0]["delta"].get("content")
+        if text:
+            Clock.schedule_once(lambda dt, t=text: append_to_label(t))  # Kivy
+            # page.run_thread / page.update(...) for Flet
+
+threading.Thread(target=worker, daemon=True).start()
+```
+
+Call `llm.stop()` from your Stop button; breaking out of the loop also stops generation.
+
+**Flet.** `flet build apk` installs packages from PyPI using the Android platform tag, so the
+`android_24_arm64_v8a` wheel is what ships the native libraries. This path needs on-device
+verification for your Flet/Python version.
+
+**Kivy / python-for-android.** p4a does not consume Android wheels yet. For now, bundle the
+`.so` files with your app and point `PYLOLM_LIB_DIR` at them before importing pylolm:
+
+```python
+import os
+os.environ["PYLOLM_LIB_DIR"] = os.path.join(os.path.dirname(__file__), "pylolm_libs")
+import pylolm
+```
+
+A proper p4a recipe is on the roadmap.
+
+## Building from source
+
+```bash
+git clone https://github.com/auraway/pylolm && cd pylolm
+pip install -e ".[dev]"
+python -m pytest                                   # pure-Python tests, no model needed
+PYLOLM_MODEL=/path/model.gguf python -m pytest     # + integration tests
+```
+
+Native libraries: `src/pylolm/native/build_termux.sh` (on a phone) or `build_android_ndk.sh`
+(cross-compile). The llama.cpp version is pinned in `src/pylolm/native/LLAMA_CPP_REF`.
+
+## Releasing to PyPI
+
+1. Set `LLAMA_CPP_REF` to a llama.cpp tag/commit you have tested.
+2. Bump `version` in `pyproject.toml` and add a `CHANGELOG.md` entry.
+3. On PyPI, add a [trusted publisher](https://docs.pypi.org/trusted-publishers/) for this repo
+   (workflow `release.yml`, environment `pypi`).
+4. `git tag v0.1.0 && git push --tags`. CI builds the sdist, a pure wheel, and the Android arm64
+   wheel, publishes them to PyPI, and attaches `pylolm-libs-android-arm64-v8a.tar.gz` (+ `.sha256`)
+   to the GitHub Release for `pylolm get-libs`. The tag must match the version in `pyproject.toml`.
+
+## Roadmap
+
+- p4a recipe for Kivy; verified Flet build
+- whisper.cpp (speech-to-text) module
+- TTS and OCR modules
+- GPU backends (Vulkan/OpenCL) as an opt-in
+
+## License
+
+MIT. llama.cpp is MIT-licensed too; model licenses are your responsibility.

pylolm-0.1.0/pyproject.toml

@@ -0,0 +1,53 @@
+[build-system]
+requires = ["setuptools>=77", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "pylolm"
+version = "0.1.0"
+description = "LoLm: run GGUF language models on-device on Android from Python (llama.cpp bindings for Kivy, Flet and Termux)."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.9"
+authors = [{ name = "AuraWay", email = "auraway.dev@gmail.com" }]
+keywords = ["llm", "llama.cpp", "gguf", "android", "termux", "kivy", "flet", "on-device", "local-llm"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Operating System :: Android",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: C",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Typing :: Typed",
+]
+dependencies = []
+
+[project.optional-dependencies]
+dev = ["pytest>=7", "ruff", "build", "twine"]
+
+[project.scripts]
+pylolm = "pylolm.__main__:main"
+
+[project.urls]
+Homepage = "https://github.com/aura-way/pylolm"
+Issues = "https://github.com/aura-way/pylolm/issues"
+Changelog = "https://github.com/aura-way/pylolm/blob/main/CHANGELOG.md"
+
+[tool.setuptools]
+package-dir = { "" = "src" }
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+pylolm = ["py.typed", "lib/*.so", "native/*"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py39"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

pylolm-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

pylolm-0.1.0/src/pylolm/__init__.py

@@ -0,0 +1,36 @@
+"""pylolm (LoLm): run GGUF language models on-device on Android, from Python.
+
+An Android-friendly take on llama-cpp-python, built on llama.cpp and designed for
+Kivy, Flet and Termux.
+"""
+
+from .errors import (
+    ContextOverflowError,
+    DownloadError,
+    GenerationError,
+    LibraryNotFoundError,
+    ModelLoadError,
+    PyLoLmError,
+)
+from .llama import Llama
+
+try:
+    from importlib.metadata import PackageNotFoundError, version as _version
+
+    try:
+        __version__ = _version("pylolm")
+    except PackageNotFoundError:  # running from a source checkout
+        __version__ = "0.0.0+local"
+except ImportError:  # pragma: no cover
+    __version__ = "0.0.0+local"
+
+__all__ = [
+    "Llama",
+    "PyLoLmError",
+    "LibraryNotFoundError",
+    "ModelLoadError",
+    "ContextOverflowError",
+    "DownloadError",
+    "GenerationError",
+    "__version__",
+]

pylolm-0.1.0/src/pylolm/__main__.py

@@ -0,0 +1,168 @@
+"""Command line: ``python -m pylolm {doctor,get-libs,build-native,chat}`` (also ``pylolm``)."""
+
+import argparse
+import platform
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+from . import __version__, _lib
+from .errors import DownloadError, PyLoLmError
+
+
+def cmd_doctor(_args) -> int:
+    print("pylolm %s" % __version__)
+    print("python %s on %s (%s)" % (platform.python_version(), platform.system(), platform.machine()))
+    print("search order for native libraries:")
+    for d in _lib.candidate_dirs():
+        found = (d / _lib.SHIM_NAME).is_file()
+        print("  [%s] %s" % ("x" if found else " ", d))
+    try:
+        _lib.load()
+    except PyLoLmError as exc:
+        print("\nNOT READY: %s" % exc)
+        return 1
+    print("\nOK: native libraries loaded from %s" % _lib.loaded_dir())
+    return 0
+
+
+def cmd_get_libs(args) -> int:
+    from . import _download
+
+    pending = [False]  # a progress line is on screen without a trailing newline
+
+    def progress(done, total):
+        pending[0] = True
+        mb = done / 1e6
+        text = "  %.1f / %.1f MB" % (mb, total / 1e6) if total else "  %.1f MB" % mb
+        print("\r" + text, end="", flush=True)
+
+    def log(message):
+        if pending[0]:
+            print()
+            pending[0] = False
+        print(message)
+
+    try:
+        _download.get_libs(
+            version=args.version,
+            repo=args.repo,
+            base_url=args.base_url,
+            force=args.force,
+            progress=progress,
+            log=log,
+        )
+    except DownloadError as exc:
+        if pending[0]:
+            print()
+        print("error: %s" % exc, file=sys.stderr)
+        return 1
+    print("\nNext: pylolm doctor")
+    return 0
+
+
+def cmd_build_native(_args) -> int:
+    script = Path(__file__).resolve().parent / "native" / "build_termux.sh"
+    if not script.is_file():
+        print("build script not found: %s" % script, file=sys.stderr)
+        return 1
+    return subprocess.call(["bash", str(script)])
+
+
+def cmd_chat(args) -> int:
+    from .llama import Llama
+
+    try:
+        llm = Llama(args.model, n_ctx=args.n_ctx, n_threads=args.threads)
+    except (PyLoLmError, FileNotFoundError) as exc:
+        print("error: %s" % exc, file=sys.stderr)
+        return 1
+
+    history = []
+    if args.system:
+        history.append({"role": "system", "content": args.system})
+    print("pylolm chat - /reset clears history, /exit quits.\n")
+
+    while True:
+        try:
+            user = input("you> ").strip()
+        except (EOFError, KeyboardInterrupt):
+            print()
+            break
+        if not user:
+            continue
+        if user == "/exit":
+            break
+        if user == "/reset":
+            history = history[:1] if args.system else []
+            llm.reset()
+            print("(history cleared)")
+            continue
+
+        history.append({"role": "user", "content": user})
+        reply = []
+        t0 = time.time()
+        n = 0
+        try:
+            print("bot> ", end="", flush=True)
+            for chunk in llm.create_chat_completion(
+                history, max_tokens=args.max_tokens, temperature=args.temp, stream=True
+            ):
+                text = chunk["choices"][0]["delta"].get("content")
+                if text:
+                    n += 1
+                    reply.append(text)
+                    print(text, end="", flush=True)
+        except KeyboardInterrupt:
+            llm.stop()
+            print("\n(interrupted)")
+        except PyLoLmError as exc:
+            print("\nerror: %s" % exc)
+            history.pop()
+            continue
+        dt = max(time.time() - t0, 1e-6)
+        print("\n[%d chunks, %.1f/s]\n" % (n, n / dt))
+        history.append({"role": "assistant", "content": "".join(reply)})
+
+    llm.close()
+    return 0
+
+
+def main(argv=None) -> int:
+    parser = argparse.ArgumentParser(prog="pylolm", description="Local LLMs on Android, from Python")
+    parser.add_argument("--version", action="version", version="pylolm " + __version__)
+    sub = parser.add_subparsers(dest="command")
+
+    sub.add_parser("doctor", help="check that the native libraries can be found and loaded")
+    sub.add_parser("build-native", help="build llama.cpp + the shim on this device (slow, Termux)")
+
+    get = sub.add_parser("get-libs", help="download the prebuilt native libraries (fast)")
+    get.add_argument("--version", default=None, help="release to download (default: this pylolm)")
+    get.add_argument("--repo", default=None, help="GitHub repo as owner/name")
+    get.add_argument("--base-url", default=None, help="download from this URL instead of GitHub")
+    get.add_argument("--force", action="store_true", help="replace existing libraries")
+
+    chat = sub.add_parser("chat", help="interactive chat in the terminal")
+    chat.add_argument("model", help="path to a .gguf file")
+    chat.add_argument("--n-ctx", type=int, default=2048)
+    chat.add_argument("--threads", type=int, default=None)
+    chat.add_argument("--system", default=None, help="system prompt")
+    chat.add_argument("--max-tokens", type=int, default=512)
+    chat.add_argument("--temp", type=float, default=0.7)
+
+    args = parser.parse_args(argv)
+    handlers = {
+        "doctor": cmd_doctor,
+        "get-libs": cmd_get_libs,
+        "build-native": cmd_build_native,
+        "chat": cmd_chat,
+    }
+    if args.command not in handlers:
+        parser.print_help()
+        return 0
+    return handlers[args.command](args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())