dataspires 0.1.0__tar.gz

dataspires-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 DataSpires
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dataspires-0.1.0/MANIFEST.in

@@ -0,0 +1,9 @@
+include LICENSE
+include README.md
+include pyproject.toml
+
+recursive-include afrilink *.py
+
+prune container_build
+prune examples
+prune .git

dataspires-0.1.0/PKG-INFO

@@ -0,0 +1,451 @@
+Metadata-Version: 2.4
+Name: dataspires
+Version: 0.1.0
+Summary: DataSpires SDK — Train, finetune, and generate on GPUs from your notebook
+Home-page: https://github.com/DataSpires/backend-server
+Author: DataSpires
+Author-email: DataSpires <info@dataspires.com>
+License: MIT
+Project-URL: Homepage, https://dataspires.com
+Project-URL: Documentation, https://www.dataspires.com/docs
+Project-URL: Repository, https://github.com/DataSpires/afrilink-sdk
+Project-URL: Bug Tracker, https://github.com/DataSpires/afrilink-sdk/issues
+Keywords: hpc,high-performance-computing,finetuning,pretrain,generate,vlm,llm,lora,notebook,gpu,dataspires,afrilink
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.28.0
+Provides-Extra: full
+Requires-Dist: psutil>=5.9.0; extra == "full"
+Provides-Extra: build
+Requires-Dist: cryptography>=41.0.0; extra == "build"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Dynamic: author
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-python
+
+# DataSpires SDK
+
+**Version:** 0.1.0
+
+Train, finetune, and run inference on DataSpires GPUs from any notebook.
+
+```bash
+pip install dataspires
+```
+
+**Works in:** Google Colab · Kaggle · Jupyter · VS Code · any Python 3.8+ environment
+
+---
+
+## Contents
+
+1. [60-second quickstart](#60-second-quickstart)
+2. [Authentication](#authentication)
+3. [Which method should I use?](#which-method-should-i-use)
+4. [Backend](#backend)
+5. [Guides by task](#guides-by-task)
+   — [Pretrain](#pretrain) · [Finetune](#finetune) · [Generate](#generate)
+   — [Checkpointing](#checkpointing-and-resuming)
+6. [Working with your output model](#working-with-your-model)
+7. [API reference](#api-reference)
+8. [Hardware & billing](#hardware--billing)
+9. [Model & dataset registry](#model--dataset-registry)
+10. [Troubleshooting](#troubleshooting)
+11. [Built-in help](#built-in-help)
+
+---
+
+## 60-second quickstart
+
+```python
+from dataspires import DataSpiresClient
+
+client = DataSpiresClient()
+client.authenticate(api_key="afk_live_…")  # or set API_KEY
+
+job = client.pretrain(
+    architecture="cnn",
+    kind="resnet18",
+    init="pretrained",
+    data="./images/",
+    config={"n_epoch": 5, "num_classes": 2, "seed": 42},
+    gpus=1,
+)
+result = job.run(wait=True)
+print(result["status"])  # "completed" | "failed" | "timeout" | …
+if result["status"] != "completed":
+    print(result.get("error") or result.get("reason"))
+```
+
+`finetune()` and `generate()` use the same `client → job → job.run()` pattern.
+
+---
+
+## Authentication
+
+### Get a key
+
+1. Sign up at [dataspires.com](https://dataspires.com).
+2. **Profile → DataSpires SDK keys → Create new key.** Copy the `afk_live_…` value — it's shown once.
+3. Store it as `API_KEY`.
+
+### Set the key
+
+| Environment | How |
+| --- | --- |
+| Google Colab | Secrets → `API_KEY` |
+| Kaggle | Secrets → `API_KEY` |
+| Local | `os.environ["API_KEY"] = "afk_live_…"` |
+| Anywhere | `client.authenticate(api_key="afk_live_…")` |
+
+```python
+client = DataSpiresClient()
+client.authenticate()
+```
+
+`authenticate()` raises on a bad or revoked key. Rotate a key by revoking it on the dashboard and creating a new one.
+
+---
+
+## Which method should I use?
+
+| Your goal | Call | You provide |
+| --- | --- | --- |
+| Detector, CNN, or scratch transformer | `client.pretrain()` | `architecture`, `kind`, `data`, optional `init` / `config` |
+| Fine-tune an LLM / VLM foundation model | `client.finetune()` | `model`, `training_mode`, `data`, optional `task=` |
+| Run a model without changing weights | `client.generate()` | `model`, prompts / image+question manifest |
+
+> Train from zero → `pretrain()`. Adapt existing weights → `finetune()`. Frozen answers → `generate()`.
+
+---
+
+## Backend
+
+Jobs run on the DataSpires GPU cluster by default.
+
+| | |
+| --- | --- |
+| **Job model** | Async — submit, then poll or `list_jobs()` |
+| **Outputs** | `job.download()` and `client.download_model()` |
+| **Multi-GPU** | Supported when the cluster has capacity |
+| **Wall-clock max** | `time_limit` capped at **10 hours** |
+
+### `status` vs `state`
+
+| API | Field | Example values |
+| --- | --- | --- |
+| `job.run()` return dict | **`status`** | `submitted`, `completed`, `failed`, `timeout`, `cancelled` |
+| `client.list_jobs()` / `client.get_job_status()` | **`state`** | same lifecycle strings; may also include `queued`, `running` |
+
+```python
+result = job.run(wait=True)
+print(result["status"])
+
+for entry in client.list_jobs():
+    print(entry["job_id"], entry["state"], entry.get("reason"))
+```
+
+---
+
+## Guides by task
+
+### Pretrain
+
+```python
+job = client.pretrain(
+    architecture="cnn",
+    kind="resnet18",
+    init="pretrained",
+    data="./images/",
+    config={"n_epoch": 5, "num_classes": 2, "batch_size": 32, "seed": 42},
+    gpus=1,
+    time_limit="02:00:00",
+)
+result = job.run(wait=True)
+rows = job.get_metrics()
+job.export_metrics("./run.csv")
+```
+
+**Detector.** Metrics are loss only in v1.
+
+```python
+job = client.pretrain(
+    architecture="detector",
+    kind="detect",          # or segment, pose
+    init="pretrained",
+    weights="yolo11n.pt",
+    data="./dataset/",
+    data_config="data.yaml",
+    config={"n_epoch": 100, "imgsz": 640, "batch_size": 16, "seed": 42},
+)
+```
+
+**Decoder from scratch.** `tokenizer` is required for decoder kinds only.
+
+```python
+job = client.pretrain(
+    architecture="transformer",
+    kind="gpt-decoder-tiny",
+    init="scratch",
+    tokenizer="gpt2",
+    data="./corpus/",
+    config={"max_steps": 1000, "n_epoch": 1, "seed": 42},
+)
+```
+
+| `architecture` | `kind` |
+| --- | --- |
+| `detector` | `detect`, `segment`, `pose` |
+| `cnn` | `resnet18`, `resnet50` |
+| `transformer` | `gpt-decoder-tiny`, `qwen-decoder-tiny`, `llama-3-decoder-tiny`, `vit-tiny` |
+
+`time_limit` is an `HH:MM:SS` string.
+
+```python
+job.download("./my-outputs")
+job.download("./ckpts", include="checkpoints")
+client.download_model(job.job_id, "./my-outputs")
+```
+
+### Finetune
+
+```python
+import pandas as pd
+
+data = pd.DataFrame({"text": ["Below is an instruction...\n\n### Response:\n..."]})
+
+job = client.finetune(
+    model="qwen2.5-0.5b",
+    training_mode="low",     # low | medium | high | custom
+    data=data,
+    gpus=1,
+    time_limit="01:00:00",
+)
+result = job.run(wait=True)
+if result["status"] == "completed":
+    client.download_model(result["job_id"], "./my-model")
+```
+
+| Mode | Strategy | Precision |
+| --- | --- | --- |
+| `low` | QLoRA (rank 8) | 4-bit |
+| `medium` | LoRA (rank 16) | 8-bit on 1 GPU, bf16 otherwise |
+| `high` | High-rank LoRA (rank 64) | bf16 |
+| `custom` | User-defined (see `config=`) | `bf16`, `fp16`, `4bit`, or `8bit` |
+
+`training_mode="custom"` requires `config=`. Presets reject `config`. `gpus` and `time_limit` stay outside `config`.
+
+```python
+job = client.finetune(
+    model="qwen2.5-0.5b",
+    training_mode="custom",
+    data=data,
+    config={
+        "batch_size": 2,
+        "learning_rate": 2e-4,
+        "scheduler": "cosine",
+        "n_epoch": 3,
+        "precision": "bf16",
+        "lora": {"r": 16, "alpha": 32, "dropout": 0.05},
+        "max_seq_length": 2048,
+        "optimizer": "adamw_torch",
+        "n_checkpoint": 100,
+        "n_evals": 0,
+        "seed": 42,
+    },
+)
+rows = job.get_metrics()
+job.export_metrics("./run_metrics.csv")
+```
+
+Causal-LM accuracy is token accuracy, not a task score. If `n_evals` is above 0 and there is no eval split, the SDK fails before submit.
+
+For VLMs, pass `task=` (captioning / VQA) with image+text data. See `dataspires.docs("finetune")`.
+
+Optional: `checkpoint={"every": 200, "keep": 2}` and `resume_from=` (prior job, job id, or `{job_id, step}`).
+
+### Checkpointing and resuming
+
+Jobs honour `time_limit` (capped at 10 hours). A stopped run can be continued with `resume_from=`.
+
+```python
+job = client.finetune(
+    model="qwen2.5-0.5b",
+    training_mode="low",
+    data=data,
+    time_limit="08:00:00",
+    checkpoint={"every": 200, "keep": 2},
+)
+result = job.run(wait=True)
+
+if result["status"] != "completed":
+    job2 = client.finetune(
+        model="qwen2.5-0.5b",
+        training_mode="low",
+        data=data,
+        resume_from=job,
+    )
+    job2.run(wait=True)
+```
+
+| Path | Meaning |
+| --- | --- |
+| `output/checkpoints/checkpoint-N/` | Mid-run snapshots |
+| `output/final/` | Weights after a clean exit |
+
+`client.list_checkpoints(job_id)` lists prefixes. See `dataspires.docs("checkpoints")`.
+
+### Generate
+
+*(Frozen inference — no weight updates)*
+
+```python
+job = client.generate(
+    model="smolvlm-256m",
+    data="./manifest_dir/",
+    outputs=["text", "confidence"],
+    max_new_tokens=64,
+    temperature=0.0,
+)
+result = job.run(wait=True)
+```
+
+### Job failures
+
+`job.run(wait=True)` returns a dict on failure instead of raising. Check `result["status"] == "completed"` before downloading. Auth and submit-time validation errors do raise.
+
+---
+
+## Working with your model
+
+### Convert to GGUF for Ollama / llama.cpp
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+
+base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
+model = PeftModel.from_pretrained(base, "./my-model")
+merged = model.merge_and_unload()
+merged.save_pretrained("./my-model-merged")
+AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").save_pretrained("./my-model-merged")
+```
+
+Then convert with [llama.cpp](https://github.com/ggerganov/llama.cpp) `convert_hf_to_gguf.py`.
+
+### Publish to Hugging Face Hub
+
+```python
+from huggingface_hub import HfApi
+api = HfApi()
+api.upload_folder(folder_path="./my-model", repo_id="you/my-model", repo_type="model")
+```
+
+---
+
+## API reference
+
+### `DataSpiresClient`
+
+| Method | Description |
+| --- | --- |
+| `authenticate(api_key=None)` | Resolve key; raises on failure |
+| `pretrain(architecture, kind, init, config, …)` | Returns a job — call `.run()` |
+| `finetune(model, training_mode, data, config, …)` | Returns `FinetuneJob` — call `.run()` |
+| `generate(model, data, outputs, …)` | Returns `GenerateJob` — call `.run()` |
+| `download_model(job_id, local_dir, include=("final",))` | Download outputs |
+| `list_checkpoints(job_id)` | List checkpoint prefixes |
+| `upload_dataset(local_path, dataset_name=None)` | Optional separate staging |
+| `list_available_models()` / `list_available_datasets()` | Registry lookups |
+| `get_model_requirements(model, training_mode)` | GPU/memory recommendation |
+| `cancel_job(job_id)` | Stop a job |
+| `list_jobs(include_completed=False)` | List jobs → dicts with **`state`** |
+| `get_job_status(job_id)` | Poll by ID → dict with **`state`** |
+
+### Job objects
+
+| Member | Description |
+| --- | --- |
+| `run(wait=True, poll_interval=30)` | Submit; with `wait=True`, return a result **dict** with **`status`** |
+| `cancel()` | Stop the job |
+| `get_logs(tail=100)` | Recent log lines |
+| `get_metrics()` | Loss/acc rows |
+| `export_metrics(path)` | Write metrics to `.csv` or `.json` |
+| `download(local_dir, include=("final",))` | Download outputs |
+| `checkpoints()` | List checkpoint prefixes |
+| `estimated_cost_usd()` | Planning ceiling from `gpus × time_limit` |
+| `status`, `job_id` | Current tracked state / job ID |
+
+---
+
+## Hardware & billing
+
+Prefer small models and `training_mode="low"` for lighter runs.
+
+| Model size | Mode | Fits on 1 GPU? |
+| --- | --- | --- |
+| 0.5B – 3B | low / medium | yes |
+| 3B – 7B | low (QLoRA 4-bit) | yes |
+| 7B | medium | yes |
+| 13B | low | tight |
+| 30B+ | low | unlikely |
+
+**Limits:** **10h** `time_limit`; hard max **32 GPUs** (SDK warns above 8).
+
+**Billing:** invoices at [dataspires.com/dashboard/billing](https://dataspires.com/dashboard/billing). GPU time is **$0.60/GPU-hr for an L4**. `job.estimated_cost_usd()` is a planning ceiling from `time_limit`, not the invoice.
+
+---
+
+## Model & dataset registry
+
+```python
+client.list_available_models()
+client.list_available_models(size="tiny")
+client.list_available_datasets()
+client.get_model_requirements("qwen2.5-0.5b", "low")
+```
+
+---
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+| --- | --- | --- |
+| `API key auth failed` | Bad / revoked key | Create a new key; set `API_KEY` |
+| Job stuck in `queued` | Cluster at capacity | `client.list_jobs()`; wait or try later |
+| `KeyError: 'status'` on `list_jobs()` | Wrong field | Use `entry["state"]` for list/status APIs |
+| `KeyError: 'state'` on `job.run()` result | Wrong field | Use `result["status"]` for `job.run()` |
+| Gated HF model fails | Missing token | Add `HUGGINGFACE_TOKEN` |
+| `download_model()` returns nothing | No finals written | Confirm the run completed; try `include="checkpoints"` |
+| `reason` is `DeadlineExceeded` | Hit `time_limit` | `resume_from=job` |
+
+---
+
+## Built-in help
+
+```python
+import dataspires
+dataspires.docs("help")
+dataspires.docs("finetune")
+dataspires.docs("pretrain")
+dataspires.docs("checkpoints")
+```
+
+`AfriLinkClient` and `import afrilink` remain available as aliases for existing notebooks.