tensorless 0.1.0__tar.gz → 0.2.0__tar.gz

tensorless-0.2.0/PKG-INFO

@@ -0,0 +1,13 @@
+Metadata-Version: 2.4
+Name: tensorless
+Version: 0.2.0
+Summary: ML with maximum automation and minimum setup.
+Author: Tensorless Contributors
+License: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Dynamic: license-file

{tensorless-0.1.0 → tensorless-0.2.0}/docs/architecture.md

@@ -22,7 +22,8 @@ tensorless/
 │   ├── detector.py               Dataset -> task string
 │   └── config.py                   Dataset + TrainConfig -> ResolvedConfig
 ├── tokenization/
-│   └── char_tokenizer.py          CharTokenizer: vocab build/encode/decode/save/load
+│   ├── char_tokenizer.py          CharTokenizer: vocab build/encode/decode/save/load
+│   └── bpe_tokenizer.py           BPETokenizer: corpus-trained subword encoding
 ├── models/
 │   ├── transformer.py              TinyTransformer (GPT-style decoder)
 │   ├── mlp.py                       TabularMLP

{tensorless-0.1.0 → tensorless-0.2.0}/docs/configuration.md

@@ -26,6 +26,8 @@ defaults.
 | `ff_mult` | auto | Feed-forward expansion multiplier (transformer only) |
 | `dropout` | `0.1` | Dropout probability |
 | `max_seq_len` | `256` (text) / `1` (tabular) | Max sequence length in tokens (text tasks) |
+| `tokenizer` | `"bpe"` | Text tokenizer: `"bpe"` or `"char"` |
+| `bpe_vocab_size` | `1000` | Maximum vocabulary size when `tokenizer="bpe"` |
 
 ## Optimization
 

{tensorless-0.1.0 → tensorless-0.2.0}/docs/installation.md

@@ -7,26 +7,6 @@
 - Optional: a CUDA-capable GPU, or a TPU with `torch_xla` installed, for
   faster training. Tensorless works fine on CPU-only machines too.
 
-## Install from source
-
-Tensorless isn't published to PyPI yet. Install it directly from a
-checkout of this repository:
-
-```bash
-git clone https://github.com/tensorless/tensorless.git
-cd tensorless
-pip install -e .
-```
-
-The `-e` (editable) flag means changes to the source are picked up
-immediately without reinstalling — useful if you're also contributing.
-
-If you plan to run the test suite, install the dev extras too:
-
-```bash
-pip install -e ".[dev]"
-```
-
 ## Verify your installation
 
 ```bash

{tensorless-0.1.0 → tensorless-0.2.0}/docs/limitations.md

@@ -6,12 +6,10 @@ current limits:
 
 ## Models
 
-- The default text tokenizer is **character-level**, not subword/BPE.
-  This means no vocabulary file dependency and it works on any language
-  out of the box, but it's less token-efficient than production
-  tokenizers (GPT-style BPE, SentencePiece), so very long documents take
-  more sequence positions to represent, and generation quality per
-  parameter is lower than a comparably-sized BPE-tokenized model.
+- The default text tokenizer is the dependency-free **BPE** tokenizer, which
+  learns merges from the training corpus. It is more token-efficient than the
+  optional `tokenizer="char"` alternative, but still simpler than production
+  tokenizers such as GPT-style BPE or SentencePiece.
 - Models are trained **from scratch** every time — there's no
   fine-tuning of pretrained checkpoints. This keeps things dependency-
   free and fast to set up, but means text-generation quality on small

{tensorless-0.1.0 → tensorless-0.2.0}/examples/text_generation_example.py

@@ -32,12 +32,12 @@ def main():
     model = tl.train(
         CORPUS_PATH,
         out=os.path.join(DATA_DIR, "text_gen_example.tl"),
-        epochs=5,
-        d_model=64,
-        layers=2,
-        heads=4,
-        batch_size=16,
-        max_seq_len=64,
+        # epochs=5,
+        # d_model=64,
+        # layers=2,
+        # heads=4,
+        # batch_size=16,
+        # max_seq_len=64,
     )
 
     print("\nGenerated text:")

{tensorless-0.1.0 → tensorless-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "tensorless"
-version = "0.1.0"
+version = "0.2.0"
 description = "ML with maximum automation and minimum setup."
 readme = "README.md"
 requires-python = ">=3.9"

{tensorless-0.1.0 → tensorless-0.2.0}/tensorless/auto/config.py

@@ -71,6 +71,9 @@ def resolve_config(ds: Dataset, user: TrainConfig) -> ResolvedConfig:
     model_type = user.model_type or (
         "transformer" if task in ("text-generation", "text-classification") else "mlp"
     )
+    tokenizer = user.tokenizer or "bpe"
+    if tokenizer not in ("char", "bpe"):
+        raise ValueError("tokenizer must be 'char' or 'bpe'")
 
     d_model, layers, heads, ff_mult = _auto_model_size(n, ds.kind)
 
@@ -92,6 +95,8 @@ def resolve_config(ds: Dataset, user: TrainConfig) -> ResolvedConfig:
         ff_mult=user.ff_mult or ff_mult,
         dropout=user.dropout if user.dropout is not None else 0.1,
         max_seq_len=user.max_seq_len or (256 if ds.kind in ("text", "text_labeled") else 1),
+        tokenizer=tokenizer,
+        bpe_vocab_size=user.bpe_vocab_size or 1000,
         optimizer=user.optimizer or "adamw",
         learning_rate=user.learning_rate or (3e-4 if model_type == "transformer" else 1e-3),
         weight_decay=user.weight_decay if user.weight_decay is not None else 0.01,

{tensorless-0.1.0 → tensorless-0.2.0}/tensorless/config.py

@@ -33,6 +33,8 @@ class TrainConfig:
     ff_mult: Optional[int] = None
     dropout: Optional[float] = None
     max_seq_len: Optional[int] = None
+    tokenizer: Optional[str] = None           # "char" or "bpe"
+    bpe_vocab_size: Optional[int] = None
 
     # --- optimization ---
     optimizer: Optional[str] = None             # "adamw", "adam", "sgd"
@@ -89,6 +91,8 @@ class ResolvedConfig:
     ff_mult: int
     dropout: float
     max_seq_len: int
+    tokenizer: str
+    bpe_vocab_size: int
 
     optimizer: str
     learning_rate: float

{tensorless-0.1.0 → tensorless-0.2.0}/tensorless/runtime.py

@@ -16,7 +16,7 @@ from typing import Any, Dict, List, Union
 import torch
 
 from .models.registry import build_model
-from .tokenization.char_tokenizer import CharTokenizer
+from .tokenization import tokenizer_from_state_dict
 from .data.tabular import TabularPreprocessor
 from .devices.device import get_torch_device
 from .errors import ModelError
@@ -35,7 +35,7 @@ class LoadedModel:
 
         self.tokenizer = None
         if payload.get("tokenizer_state") is not None:
-            self.tokenizer = CharTokenizer.from_state_dict(payload["tokenizer_state"])
+            self.tokenizer = tokenizer_from_state_dict(payload["tokenizer_state"])
 
         self.preprocessor = None
         if payload.get("preprocessor_state") is not None:

tensorless-0.2.0/tensorless/tokenization/__init__.py

@@ -0,0 +1,11 @@
+from .bpe_tokenizer import BPETokenizer
+from .char_tokenizer import CharTokenizer
+
+
+def tokenizer_from_state_dict(state):
+	if state.get("tokenizer_type", "char") == "bpe":
+		return BPETokenizer.from_state_dict(state)
+	return CharTokenizer.from_state_dict(state)
+
+
+__all__ = ["CharTokenizer", "BPETokenizer", "tokenizer_from_state_dict"]

tensorless-0.2.0/tensorless/tokenization/bpe_tokenizer.py

@@ -0,0 +1,123 @@
+"""A small, dependency-free byte-pair encoding tokenizer."""
+
+from __future__ import annotations
+
+from collections import Counter
+from typing import Dict, List, Sequence, Tuple
+
+from .char_tokenizer import BOS, EOS, PAD, SPECIAL_TOKENS, UNK
+
+
+class BPETokenizer:
+    """Character-initialized BPE tokenizer suitable for small local corpora.
+
+    BPE operates on Unicode characters rather than bytes so the resulting
+    vocabulary remains portable and decoding preserves the original text.
+    """
+
+    def __init__(self, vocab: List[str], merges: List[Tuple[str, str]]):
+        self.vocab = list(vocab)
+        self.merges = [tuple(pair) for pair in merges]
+        self._stoi: Dict[str, int] = {token: i for i, token in enumerate(self.vocab)}
+
+    @property
+    def vocab_size(self) -> int:
+        return len(self.vocab)
+
+    @property
+    def pad_id(self) -> int:
+        return self._stoi[PAD]
+
+    @property
+    def unk_id(self) -> int:
+        return self._stoi[UNK]
+
+    @property
+    def bos_id(self) -> int:
+        return self._stoi[BOS]
+
+    @property
+    def eos_id(self) -> int:
+        return self._stoi[EOS]
+
+    @classmethod
+    def build(cls, texts: Sequence[str], vocab_size: int = 1000) -> "BPETokenizer":
+        if vocab_size < len(SPECIAL_TOKENS):
+            raise ValueError(f"vocab_size must be at least {len(SPECIAL_TOKENS)}")
+
+        symbols = set(char for text in texts for char in text)
+        vocab = list(SPECIAL_TOKENS) + sorted(symbols - set(SPECIAL_TOKENS))
+        sequences = [list(text) for text in texts]
+        merges: List[Tuple[str, str]] = []
+
+        while len(vocab) < vocab_size:
+            counts = Counter(
+                (sequence[index], sequence[index + 1])
+                for sequence in sequences
+                for index in range(len(sequence) - 1)
+            )
+            if not counts:
+                break
+            best_pair, best_count = min(
+                counts.items(), key=lambda item: (-item[1], item[0])
+            )
+            if best_count < 2:
+                break
+            merged = best_pair[0] + best_pair[1]
+            if merged in vocab:
+                break
+            merges.append(best_pair)
+            vocab.append(merged)
+            sequences = [cls._apply_merge(sequence, best_pair, merged) for sequence in sequences]
+
+        return cls(vocab, merges)
+
+    @staticmethod
+    def _apply_merge(sequence: List[str], pair: Tuple[str, str], merged: str) -> List[str]:
+        result: List[str] = []
+        index = 0
+        while index < len(sequence):
+            if index + 1 < len(sequence) and (sequence[index], sequence[index + 1]) == pair:
+                result.append(merged)
+                index += 2
+            else:
+                result.append(sequence[index])
+                index += 1
+        return result
+
+    def _tokenize(self, text: str) -> List[str]:
+        tokens = list(text)
+        for pair in self.merges:
+            tokens = self._apply_merge(tokens, pair, pair[0] + pair[1])
+        return tokens
+
+    def encode(self, text: str, add_special_tokens: bool = True) -> List[int]:
+        ids = [self._stoi.get(token, self.unk_id) for token in self._tokenize(text)]
+        if add_special_tokens:
+            ids = [self.bos_id] + ids + [self.eos_id]
+        return ids
+
+    def decode(self, ids: Sequence[int], skip_special_tokens: bool = True) -> str:
+        special_ids = {self.pad_id, self.bos_id, self.eos_id} if skip_special_tokens else set()
+        result = []
+        for index in ids:
+            if index in special_ids:
+                continue
+            if 0 <= index < len(self.vocab):
+                token = self.vocab[index]
+                if token == UNK:
+                    result.append("\ufffd")
+                elif token not in SPECIAL_TOKENS:
+                    result.append(token)
+        return "".join(result)
+
+    def state_dict(self) -> Dict:
+        return {
+            "tokenizer_type": "bpe",
+            "vocab": self.vocab,
+            "merges": [list(pair) for pair in self.merges],
+        }
+
+    @classmethod
+    def from_state_dict(cls, state: Dict) -> "BPETokenizer":
+        return cls(list(state["vocab"]), [tuple(pair) for pair in state.get("merges", [])])

{tensorless-0.1.0 → tensorless-0.2.0}/tensorless/tokenization/char_tokenizer.py

@@ -77,7 +77,7 @@ class CharTokenizer:
         return "".join(chars)
 
     def state_dict(self) -> Dict:
-        return {"vocab": self.vocab}
+        return {"tokenizer_type": "char", "vocab": self.vocab}
 
     @classmethod
     def from_state_dict(cls, state: Dict) -> "CharTokenizer":

{tensorless-0.1.0 → tensorless-0.2.0}/tensorless/training/data_prep.py

@@ -16,6 +16,7 @@ from torch.utils.data import DataLoader, Dataset as TorchDataset
 
 from ..data.loader import Dataset
 from ..data.tabular import TabularPreprocessor
+from ..tokenization.bpe_tokenizer import BPETokenizer
 from ..tokenization.char_tokenizer import CharTokenizer
 from ..auto.detector import target_column
 from ..errors import DataError
@@ -87,10 +88,16 @@ def _split_indices(n: int, val_split: float, seed: int) -> Tuple[List[int], List
     return train_idx, val_idx
 
 
+def _build_tokenizer(ds: Dataset, cfg: Dict[str, Any]):
+    if cfg.get("tokenizer", "bpe") == "bpe":
+        return BPETokenizer.build(ds.texts, vocab_size=cfg.get("bpe_vocab_size", 1000))
+    return CharTokenizer.build(ds.texts)
+
+
 def prepare_text_generation(
-    ds: Dataset, cfg: Dict[str, Any], tokenizer: Optional[CharTokenizer] = None
+    ds: Dataset, cfg: Dict[str, Any], tokenizer=None
 ) -> PreparedData:
-    tokenizer = tokenizer or CharTokenizer.build(ds.texts)
+    tokenizer = tokenizer or _build_tokenizer(ds, cfg)
     all_ids: List[int] = []
     for t in ds.texts:
         all_ids.extend(tokenizer.encode(t, add_special_tokens=True))
@@ -129,10 +136,10 @@ def prepare_text_generation(
 def prepare_text_classification(
     ds: Dataset,
     cfg: Dict[str, Any],
-    tokenizer: Optional[CharTokenizer] = None,
+    tokenizer=None,
     classes: Optional[List[str]] = None,
 ) -> PreparedData:
-    tokenizer = tokenizer or CharTokenizer.build(ds.texts)
+    tokenizer = tokenizer or _build_tokenizer(ds, cfg)
     classes = classes or sorted(set(ds.labels))
     label2id = {c: i for i, c in enumerate(classes)}
 

{tensorless-0.1.0 → tensorless-0.2.0}/tensorless/training/trainer.py

@@ -19,7 +19,7 @@ from ..data.loader import Dataset
 from ..devices.device import get_torch_device
 from ..checkpoint.manager import CheckpointManager
 from ..models.registry import build_model
-from ..tokenization.char_tokenizer import CharTokenizer
+from ..tokenization import tokenizer_from_state_dict
 from ..data.tabular import TabularPreprocessor
 from .early_stopping import EarlyStopping
 from . import data_prep as dp
@@ -97,7 +97,7 @@ def run_training(
     preprocessor = None
     if resume_state is not None:
         if resume_state.get("tokenizer_state") is not None:
-            tokenizer = CharTokenizer.from_state_dict(resume_state["tokenizer_state"])
+            tokenizer = tokenizer_from_state_dict(resume_state["tokenizer_state"])
         if resume_state.get("preprocessor_state") is not None:
             preprocessor = TabularPreprocessor.from_state_dict(resume_state["preprocessor_state"])
 

tensorless-0.2.0/tensorless.egg-info/PKG-INFO

@@ -0,0 +1,13 @@
+Metadata-Version: 2.4
+Name: tensorless
+Version: 0.2.0
+Summary: ML with maximum automation and minimum setup.
+Author: Tensorless Contributors
+License: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Dynamic: license-file

{tensorless-0.1.0 → tensorless-0.2.0}/tensorless.egg-info/SOURCES.txt

@@ -1,6 +1,5 @@
 LICENSE
 MANIFEST.in
-README.md
 pyproject.toml
 docs/api_reference.md
 docs/architecture.md
@@ -56,6 +55,7 @@ tensorless/models/transformer.py
 tensorless/serialization/__init__.py
 tensorless/serialization/tl_format.py
 tensorless/tokenization/__init__.py
+tensorless/tokenization/bpe_tokenizer.py
 tensorless/tokenization/char_tokenizer.py
 tensorless/training/__init__.py
 tensorless/training/data_prep.py

{tensorless-0.1.0 → tensorless-0.2.0}/tests/test_train_text_generation.py

@@ -1,6 +1,7 @@
 import os
 
 import tensorless as tl
+from tensorless.tokenization.bpe_tokenizer import BPETokenizer
 from tensorless.serialization.tl_format import load_tl
 
 from .conftest import TINY_TEXT_KWARGS
@@ -10,6 +11,7 @@ def test_train_text_generation_creates_tl_file(text_corpus, workdir):
     model = tl.train(text_corpus, out="model.tl", **TINY_TEXT_KWARGS)
     assert os.path.isfile("model.tl")
     assert model.task == "text-generation"
+    assert isinstance(model.tokenizer, BPETokenizer)
     assert model.info()["training_complete"] is True
 
 
@@ -21,6 +23,22 @@ def test_generate_after_reload(text_corpus, workdir):
     assert len(text) > 0
 
 
+def test_bpe_tokenizer_trains_and_reloads(text_corpus, workdir):
+    model = tl.train(
+        text_corpus,
+        out="bpe.tl",
+        tokenizer="bpe",
+        bpe_vocab_size=64,
+        **TINY_TEXT_KWARGS,
+    )
+    assert isinstance(model.tokenizer, BPETokenizer)
+    assert model.tokenizer.decode(model.tokenizer.encode("the quick")) == "the quick"
+
+    reloaded = tl.load("bpe.tl")
+    assert isinstance(reloaded.tokenizer, BPETokenizer)
+    assert reloaded.tokenizer.decode(reloaded.tokenizer.encode("the quick")) == "the quick"
+
+
 def test_cpu_training_works(text_corpus, workdir):
     model = tl.train(text_corpus, out="model.tl", device="cpu", **TINY_TEXT_KWARGS)
     assert model.config["device"] == "cpu"

tensorless-0.1.0/PKG-INFO

@@ -1,111 +0,0 @@
-Metadata-Version: 2.4
-Name: tensorless
-Version: 0.1.0
-Summary: ML with maximum automation and minimum setup.
-Author: Tensorless Contributors
-License: MIT
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: torch>=2.0
-Provides-Extra: dev
-Requires-Dist: pytest>=7.0; extra == "dev"
-Dynamic: license-file
-
-# Tensorless
-
-**ML with maximum automation and minimum setup.**
-
-```python
-import tensorless as tl
-
-tl.train("./data")
-```
-
-That's it. Tensorless inspects your dataset, figures out what kind of
-task you're trying to solve, builds and configures a model, trains it,
-validates it, checkpoints it, and saves a single portable `model.tl`
-file you can move anywhere.
-
-```python
-model = tl.run("model.tl")   # interactive chat, if it's a text model
-# or
-model = tl.load("model.tl")
-model.predict(...)
-```
-
-Simple by default. Powerful when you need it:
-
-```python
-tl.train(
-    "./data",
-    d_model=512,
-    layers=6,
-    learning_rate=3e-4,
-    batch_size=32,
-)
-```
-
-## Why Tensorless
-
-Most ML frameworks assume you already know what model you want, how big
-it should be, which optimizer and learning rate to use, and how to wire
-up checkpointing and resumption yourself. Tensorless flips that: it
-makes a reasonable, working choice for all of that automatically, and
-lets you override exactly the parts you care about.
-
-It also remembers what it already did. Run `tl.train("./data")` twice on
-the same dataset and it won't retrain — it'll just hand you back the
-model it already trained. Change the data, and it retrains. Get
-interrupted partway through a long run, and the next call resumes right
-where it left off. This is the **Smart Auto Check**, and it's the core
-idea the whole framework is built around.
-
-## Install
-
-```bash
-pip install -e .
-```
-
-See [docs/installation.md](docs/installation.md) for details and
-requirements.
-
-## Documentation
-
-| Doc | What's in it |
-|---|---|
-| [Installation](docs/installation.md) | Requirements, install steps, verifying your setup |
-| [Quick Start](docs/quickstart.md) | The fastest path to a trained model |
-| [Beginner Tutorial](docs/tutorial.md) | A guided, from-scratch walkthrough |
-| [Automatic Mode](docs/automatic_mode.md) | How auto-detection and auto-configuration work, and the Smart Auto Check |
-| [Training](docs/training.md) | `tl.train()` in depth, all supported tasks and data formats |
-| [Inference](docs/inference.md) | `tl.run()`, `tl.load()`, and the prediction API |
-| [Checkpointing & Resume](docs/checkpointing.md) | How checkpoints work and how resumption is decided |
-| [The `.tl` Format](docs/tl_format.md) | What's inside a `.tl` file and why it's portable |
-| [Configuration](docs/configuration.md) | Every override you can pass, and what it does |
-| [CLI](docs/cli.md) | `tensorless train / run / inspect / info` |
-| [API Reference](docs/api_reference.md) | Full function/class signatures |
-| [Examples](docs/examples.md) | Worked examples for each supported task |
-| [Troubleshooting](docs/troubleshooting.md) | Common errors and what to do about them |
-| [Architecture](docs/architecture.md) | How the codebase is organized, for contributors |
-| [Contributing](docs/contributing.md) | How to add models, backends, or data formats |
-| [Roadmap](docs/roadmap.md) | What's planned |
-| [Limitations](docs/limitations.md) | What Tensorless deliberately doesn't do (yet) |
-
-## Supported today
-
-- **Text generation** (language modeling) from `.txt`/`.md` files or JSON/JSONL with a `text` field
-- **Text classification** from a directory of class subfolders (`positive/`, `negative/`, ...) or labeled JSON/JSONL
-- **Tabular classification and regression** from CSV/TSV/JSON/JSONL with a target column
-
-## Project status
-
-Tensorless is an early-stage, actively developed framework. The core
-loop — inspect, auto-configure, train, checkpoint, save, reload, infer —
-is real and tested end-to-end (see [tests/](tests/)). See
-[docs/limitations.md](docs/limitations.md) for what's intentionally out
-of scope right now, and [docs/roadmap.md](docs/roadmap.md) for what's next.
-
-## License
-
-MIT

tensorless-0.1.0/README.md

@@ -1,97 +0,0 @@
-# Tensorless
-
-**ML with maximum automation and minimum setup.**
-
-```python
-import tensorless as tl
-
-tl.train("./data")
-```
-
-That's it. Tensorless inspects your dataset, figures out what kind of
-task you're trying to solve, builds and configures a model, trains it,
-validates it, checkpoints it, and saves a single portable `model.tl`
-file you can move anywhere.
-
-```python
-model = tl.run("model.tl")   # interactive chat, if it's a text model
-# or
-model = tl.load("model.tl")
-model.predict(...)
-```
-
-Simple by default. Powerful when you need it:
-
-```python
-tl.train(
-    "./data",
-    d_model=512,
-    layers=6,
-    learning_rate=3e-4,
-    batch_size=32,
-)
-```
-
-## Why Tensorless
-
-Most ML frameworks assume you already know what model you want, how big
-it should be, which optimizer and learning rate to use, and how to wire
-up checkpointing and resumption yourself. Tensorless flips that: it
-makes a reasonable, working choice for all of that automatically, and
-lets you override exactly the parts you care about.
-
-It also remembers what it already did. Run `tl.train("./data")` twice on
-the same dataset and it won't retrain — it'll just hand you back the
-model it already trained. Change the data, and it retrains. Get
-interrupted partway through a long run, and the next call resumes right
-where it left off. This is the **Smart Auto Check**, and it's the core
-idea the whole framework is built around.
-
-## Install
-
-```bash
-pip install -e .
-```
-
-See [docs/installation.md](docs/installation.md) for details and
-requirements.
-
-## Documentation
-
-| Doc | What's in it |
-|---|---|
-| [Installation](docs/installation.md) | Requirements, install steps, verifying your setup |
-| [Quick Start](docs/quickstart.md) | The fastest path to a trained model |
-| [Beginner Tutorial](docs/tutorial.md) | A guided, from-scratch walkthrough |
-| [Automatic Mode](docs/automatic_mode.md) | How auto-detection and auto-configuration work, and the Smart Auto Check |
-| [Training](docs/training.md) | `tl.train()` in depth, all supported tasks and data formats |
-| [Inference](docs/inference.md) | `tl.run()`, `tl.load()`, and the prediction API |
-| [Checkpointing & Resume](docs/checkpointing.md) | How checkpoints work and how resumption is decided |
-| [The `.tl` Format](docs/tl_format.md) | What's inside a `.tl` file and why it's portable |
-| [Configuration](docs/configuration.md) | Every override you can pass, and what it does |
-| [CLI](docs/cli.md) | `tensorless train / run / inspect / info` |
-| [API Reference](docs/api_reference.md) | Full function/class signatures |
-| [Examples](docs/examples.md) | Worked examples for each supported task |
-| [Troubleshooting](docs/troubleshooting.md) | Common errors and what to do about them |
-| [Architecture](docs/architecture.md) | How the codebase is organized, for contributors |
-| [Contributing](docs/contributing.md) | How to add models, backends, or data formats |
-| [Roadmap](docs/roadmap.md) | What's planned |
-| [Limitations](docs/limitations.md) | What Tensorless deliberately doesn't do (yet) |
-
-## Supported today
-
-- **Text generation** (language modeling) from `.txt`/`.md` files or JSON/JSONL with a `text` field
-- **Text classification** from a directory of class subfolders (`positive/`, `negative/`, ...) or labeled JSON/JSONL
-- **Tabular classification and regression** from CSV/TSV/JSON/JSONL with a target column
-
-## Project status
-
-Tensorless is an early-stage, actively developed framework. The core
-loop — inspect, auto-configure, train, checkpoint, save, reload, infer —
-is real and tested end-to-end (see [tests/](tests/)). See
-[docs/limitations.md](docs/limitations.md) for what's intentionally out
-of scope right now, and [docs/roadmap.md](docs/roadmap.md) for what's next.
-
-## License
-
-MIT

tensorless-0.1.0/tensorless/tokenization/__init__.py

@@ -1,3 +0,0 @@
-from .char_tokenizer import CharTokenizer
-
-__all__ = ["CharTokenizer"]

tensorless-0.1.0/tensorless.egg-info/PKG-INFO

@@ -1,111 +0,0 @@
-Metadata-Version: 2.4
-Name: tensorless
-Version: 0.1.0
-Summary: ML with maximum automation and minimum setup.
-Author: Tensorless Contributors
-License: MIT
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: torch>=2.0
-Provides-Extra: dev
-Requires-Dist: pytest>=7.0; extra == "dev"
-Dynamic: license-file
-
-# Tensorless
-
-**ML with maximum automation and minimum setup.**
-
-```python
-import tensorless as tl
-
-tl.train("./data")
-```
-
-That's it. Tensorless inspects your dataset, figures out what kind of
-task you're trying to solve, builds and configures a model, trains it,
-validates it, checkpoints it, and saves a single portable `model.tl`
-file you can move anywhere.
-
-```python
-model = tl.run("model.tl")   # interactive chat, if it's a text model
-# or
-model = tl.load("model.tl")
-model.predict(...)
-```
-
-Simple by default. Powerful when you need it:
-
-```python
-tl.train(
-    "./data",
-    d_model=512,
-    layers=6,
-    learning_rate=3e-4,
-    batch_size=32,
-)
-```
-
-## Why Tensorless
-
-Most ML frameworks assume you already know what model you want, how big
-it should be, which optimizer and learning rate to use, and how to wire
-up checkpointing and resumption yourself. Tensorless flips that: it
-makes a reasonable, working choice for all of that automatically, and
-lets you override exactly the parts you care about.
-
-It also remembers what it already did. Run `tl.train("./data")` twice on
-the same dataset and it won't retrain — it'll just hand you back the
-model it already trained. Change the data, and it retrains. Get
-interrupted partway through a long run, and the next call resumes right
-where it left off. This is the **Smart Auto Check**, and it's the core
-idea the whole framework is built around.
-
-## Install
-
-```bash
-pip install -e .
-```
-
-See [docs/installation.md](docs/installation.md) for details and
-requirements.
-
-## Documentation
-
-| Doc | What's in it |
-|---|---|
-| [Installation](docs/installation.md) | Requirements, install steps, verifying your setup |
-| [Quick Start](docs/quickstart.md) | The fastest path to a trained model |
-| [Beginner Tutorial](docs/tutorial.md) | A guided, from-scratch walkthrough |
-| [Automatic Mode](docs/automatic_mode.md) | How auto-detection and auto-configuration work, and the Smart Auto Check |
-| [Training](docs/training.md) | `tl.train()` in depth, all supported tasks and data formats |
-| [Inference](docs/inference.md) | `tl.run()`, `tl.load()`, and the prediction API |
-| [Checkpointing & Resume](docs/checkpointing.md) | How checkpoints work and how resumption is decided |
-| [The `.tl` Format](docs/tl_format.md) | What's inside a `.tl` file and why it's portable |
-| [Configuration](docs/configuration.md) | Every override you can pass, and what it does |
-| [CLI](docs/cli.md) | `tensorless train / run / inspect / info` |
-| [API Reference](docs/api_reference.md) | Full function/class signatures |
-| [Examples](docs/examples.md) | Worked examples for each supported task |
-| [Troubleshooting](docs/troubleshooting.md) | Common errors and what to do about them |
-| [Architecture](docs/architecture.md) | How the codebase is organized, for contributors |
-| [Contributing](docs/contributing.md) | How to add models, backends, or data formats |
-| [Roadmap](docs/roadmap.md) | What's planned |
-| [Limitations](docs/limitations.md) | What Tensorless deliberately doesn't do (yet) |
-
-## Supported today
-
-- **Text generation** (language modeling) from `.txt`/`.md` files or JSON/JSONL with a `text` field
-- **Text classification** from a directory of class subfolders (`positive/`, `negative/`, ...) or labeled JSON/JSONL
-- **Tabular classification and regression** from CSV/TSV/JSON/JSONL with a target column
-
-## Project status
-
-Tensorless is an early-stage, actively developed framework. The core
-loop — inspect, auto-configure, train, checkpoint, save, reload, infer —
-is real and tested end-to-end (see [tests/](tests/)). See
-[docs/limitations.md](docs/limitations.md) for what's intentionally out
-of scope right now, and [docs/roadmap.md](docs/roadmap.md) for what's next.
-
-## License
-
-MIT