build-slm 1.0.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,126 @@
1
+ Metadata-Version: 2.1
2
+ Name: build-slm
3
+ Version: 1.0.0
4
+ Summary: Research-Grade TPU-Native Language Model Training and Inference Framework
5
+ Author: Niranjan Kumar K.
6
+ License: Apache-2.0
7
+ Project-URL: Homepage, https://github.com/niranjankumark/slm
8
+ Project-URL: Documentation, https://github.com/niranjankumark/slm/blob/main/docs/USER_GUIDE.md
9
+ Project-URL: Bug Tracker, https://github.com/niranjankumark/slm/issues
10
+ Classifier: Development Status :: 5 - Production/Stable
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: Intended Audience :: Science/Research
13
+ Classifier: License :: OSI Approved :: Apache Software License
14
+ Classifier: Operating System :: OS Independent
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Programming Language :: Python :: 3.10
17
+ Classifier: Programming Language :: Python :: 3.11
18
+ Classifier: Programming Language :: Python :: 3.12
19
+ Classifier: Programming Language :: Python :: 3.13
20
+ Classifier: Programming Language :: Python :: 3.14
21
+ Classifier: Programming Language :: Python :: 3.15
22
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
23
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
24
+ Requires-Python: >=3.10
25
+ Description-Content-Type: text/markdown
26
+ Provides-Extra: tpu
27
+ Requires-Dist: jax[tpu] ; extra == 'tpu'
28
+ Provides-Extra: gpu
29
+ Requires-Dist: jax[cuda] ; extra == 'gpu'
30
+ Provides-Extra: cpu
31
+ Requires-Dist: jax ; extra == 'cpu'
32
+ Provides-Extra: dev
33
+ Requires-Dist: twine ; extra == 'dev'
34
+ Requires-Dist: build ; extra == 'dev'
35
+ Requires-Dist: wheel ; extra == 'dev'
36
+ Requires-Dist: pytest ; extra == 'dev'
37
+
38
+ # KLM: Research-Grade TPU-Native Language Model Framework
39
+
40
+ **Package name:** `klm`  |  **Author:** Niranjan Kumar K.  |  **Version:** 1.0.0  |  **License:** Apache-2.0
41
+
42
+ An advanced, scalable, research-grade language model training and inference framework optimized exclusively for Google Colab TPU accelerators using JAX and XLA, featuring a 100% backward-compatible pure-Python CPU reference implementation.
43
+
44
+ > **Backward compatibility:** `import slm` and `from slm import *` continue to work — the `slm` shim re-exports everything from `klm`.
45
+
46
+ See [the user guide](docs/USER_GUIDE.md), [the Google Colab TPU guide](docs/COLAB_TPU_GUIDE.md), [the benchmark report](docs/BENCHMARK_REPORT.md), and [the automated test report](docs/TEST_REPORT.md).
47
+
48
+ ---
49
+
50
+ ## Highlights
51
+
52
+ - **TPU-Native JAX/XLA Engine**: High-throughput training on Google Colab TPU (v2-8, v3-8, v4-8) with bfloat16 mixed precision and float32 master weights.
53
+ - **Advanced Architecture**: Causal multi-head attention, vectorized RoPE, numerically stable RMSNorm in float32, SwiGLU feed-forward networks, and pre-normalization.
54
+ - **Sequence Packing**: Static-shape sequence multi-packing with 2D block-diagonal causal masks and padding-aware loss masks.
55
+ - **Multi-Device TPU Scaling**: Automatic topology inspection, JAX device mesh, and `NamedSharding` across TPU cores.
56
+ - **Dedicated Instruction Tuning**: SFT pipeline supporting QA, Alpaca-style instruction-response, multi-turn, reasoning, and math datasets with **assistant-only loss masking**.
57
+ - **Inference Optimization**: Stateful KV-caching for O(1) decoding, greedy search, temperature, top-k, nucleus top-p, and repetition penalties.
58
+ - **Data Quality Intelligence**: Multi-stage cleaning, unicode NFKC, mojibake repair, HTML/boilerplate stripping, SimHash deduplication, 10-domain classification, and zero-leakage disjoint splitting.
59
+ - **Comprehensive Evaluation**: Top-1, Top-5, Top-10 token accuracy, word accuracy, perplexity, Expected Calibration Error (ECE), and long-context position-bucket evaluation.
60
+ - **Safe Migration & Recovery**: Atomic checkpoint saving with SHA-256 parameter checksums and backward-compatible v4-to-v5 `.kni` migration.
61
+
62
+ ---
63
+
64
+ ## Installation
65
+
66
+ ```bash
67
+ # Base install (pure Python stdlib — no external dependencies)
68
+ pip install klm
69
+
70
+ # With TPU acceleration (Google Colab TPU)
71
+ pip install "klm[tpu]"
72
+
73
+ # With GPU acceleration (CUDA)
74
+ pip install "klm[gpu]"
75
+
76
+ # With CPU-only JAX
77
+ pip install "klm[cpu]"
78
+ ```
79
+
80
+ ---
81
+
82
+ ## Quick Start
83
+
84
+ ```python
85
+ from klm import *
86
+
87
+ # Initialize model (backend="auto" uses TPU if present, falls back to CPU)
88
+ model = klm(
89
+ vocab_size=512,
90
+ context_length=128,
91
+ d_model=64,
92
+ heads=4,
93
+ layers=2,
94
+ ff_dim=128,
95
+ learning_rate=3e-4,
96
+ weight_decay=0.01,
97
+ )
98
+
99
+ # Train on a folder of documents
100
+ report = model.train(
101
+ "folder_path",
102
+ epochs=3,
103
+ batch_size=4,
104
+ gradient_accumulation_steps=2,
105
+ checkpoint_path="models/my_model.kni",
106
+ use_advanced_cleaning=True,
107
+ )
108
+
109
+ # Fast KV-cached text generation
110
+ output = model.predict("The research showed that", max_tokens=80, temperature=0.7, top_p=0.9)
111
+ print(output)
112
+ ```
113
+
114
+ Use `model.save("models/name")` and `model.load("models/name")` to store or restore a model. The only accepted model extension is `.kni`; it is added automatically when omitted.
115
+
116
+ ---
117
+
118
+ ## Backward Compatibility
119
+
120
+ The `slm` package name is preserved as a shim. Existing code using `from slm import *` or `import slm` continues to work without any changes.
121
+
122
+ ```python
123
+ # Legacy code — still works unchanged
124
+ from slm import *
125
+ model = slm(vocab_size=512)
126
+ ```
@@ -0,0 +1,24 @@
1
+ slm/__init__.py,sha256=IaWrHuqJDpTUKGE3-t1lYCxgt_6WLFrjil9W9wiNSho,1735
2
+ slm/benchmarks.py,sha256=LII6QDQNrZiXmcTSkNHnhHUNqhknnV05VTpWNc-rD_4,8105
3
+ slm/core.py,sha256=MyNh7WsThC7B6rvwy7Y_c-4PLUwD0tYw09j8j5TlHIo,25689
4
+ slm/data_pipeline.py,sha256=Jn09cYsImgo6Ji2DXrC_ncKWMzgkJtbGf7X4yLj38Bk,21335
5
+ slm/dataset.py,sha256=1kExFxQJsErTe_k7oYzanOZTF6Btiz70Nvzr_EEST2I,6577
6
+ slm/evaluation.py,sha256=vVXyfeK3GC0KGeBDbcEGEf8zBj9IYW3A-JMcRanX72Q,10949
7
+ slm/experiments.py,sha256=n6fMMlu5cnQiJopA7hSHVvhspm2ufyBVl5HGip9ZvLg,5990
8
+ slm/extractors.py,sha256=POfcVRyZET3tT6UyDHqlQaX47F9wZ9tMgm44PJ2oniA,2820
9
+ slm/inference.py,sha256=Gcg_8bMdPJixAjtPuJESZgeqs52YI4onLuxZAuwnrOQ,6510
10
+ slm/instruction.py,sha256=aThJ8IpX8Fs7X88dABMK8_g4UNlEPvmPQvZLzmaTeDk,14173
11
+ slm/jax_engine.py,sha256=-MqsIGCPIKAQr_xVE0yGwX48801_MpTMqaMyzirlSQU,27758
12
+ slm/migration.py,sha256=2X2TSiG5RKSHiaCRmF1JC98YS5DKSrhmw_JTfCpnn3c,6477
13
+ slm/packing.py,sha256=vGBuhmtJm0DnvPY7Vf_3fMSVlJiP0D0yvVKt6rioINc,8597
14
+ slm/progress.py,sha256=PgQbuSmHXA9N-Iekwl3p2MtS6fsiyfVhgl9frtObIZQ,711
15
+ slm/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
16
+ slm/reports/slm_training_report.json,sha256=GxAgG-ldWVrgiq_hpp_Zw6ytRVv1tFgTNw8KMSTYR-g,2503
17
+ slm/reports/slm_training_report.txt,sha256=zMTxotQjTN15deEUyJvVKhomRU8d2EF1C4hcgkJxsU4,647
18
+ slm/text.py,sha256=1GpH5IdewwpIhs5mTzkSNM3LVj4zHQ3aWgVu_K3kI_c,600
19
+ slm/tokenizer.py,sha256=sfxOhyQiCqMqW1lGewqJR_I4V2nZfoF_It06aCUYZQY,5255
20
+ slm/transformer.py,sha256=MR7iPfP1jKvbp0VT9lyDf1kObqfrOC3AG3nc_9HTPNM,43941
21
+ build_slm-1.0.0.dist-info/WHEEL,sha256=rbPMe3Bi7hBFBUdZHQOjytr-gFSmLaY6TSSyI6ziUCA,86
22
+ build_slm-1.0.0.dist-info/METADATA,sha256=3Ux3AA5gbrDQlQ9UUODLvSg5v9ZOODbvIKAvmAj1aRU,5183
23
+ build_slm-1.0.0.dist-info/licenses/LICENSE,sha256=0y5Px4xchjUxCeGinVtZXbG56HZ2GcWetlk3FLG93Nc,746
24
+ build_slm-1.0.0.dist-info/RECORD,,
@@ -0,0 +1,4 @@
1
+ Wheel-Version: 1.0
2
+ Generator: klm-build-1.0.0
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
@@ -0,0 +1,17 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ Copyright 2026 Niranjan Kumar K.
6
+
7
+ Licensed under the Apache License, Version 2.0 (the "License");
8
+ you may not use this file except in compliance with the License.
9
+ You may obtain a copy of the License at
10
+
11
+ http://www.apache.org/licenses/LICENSE-2.0
12
+
13
+ Unless required by applicable law or agreed to in writing, software
14
+ distributed under the License is distributed on an "AS IS" BASIS,
15
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16
+ See the License for the specific language governing permissions and
17
+ limitations under the License.
slm/__init__.py ADDED
@@ -0,0 +1,55 @@
1
+ from .core import SLM, slm
2
+ from .core import help
3
+ from .extractors import extract_file, extract_text
4
+ from .tokenizer import BPETokenizer, normalize, tokenize
5
+ from .transformer import TinyLanguageModel
6
+
7
+ # Research-Grade TPU-Native Extensions
8
+ from .data_pipeline import process_dataset, filter_quality, DOMAINS
9
+ from .packing import pack_dataset, PackedBatch
10
+ from .instruction import InstructionTuner, InstructionExample, parse_instruction_data
11
+ from .evaluation import evaluate_comprehensive, evaluate_generation_quality
12
+ from .inference import generate_with_kv_cache, batch_generate, KVCache
13
+ from .benchmarks import BenchmarkProfiler, evaluate_benchmarks, DEFAULT_BENCHMARKS
14
+ from .experiments import ExperimentTracker
15
+ from .migration import migrate_checkpoint, validate_checkpoint_integrity, safe_load_checkpoint, atomic_save_checkpoint
16
+ from .jax_engine import is_jax_available, get_tpu_info, JAXTrainingEngine
17
+
18
+ __all__ = [
19
+ "SLM",
20
+ "slm",
21
+ "help",
22
+ "extract_file",
23
+ "extract_text",
24
+ "normalize",
25
+ "tokenize",
26
+ "BPETokenizer",
27
+ "TinyLanguageModel",
28
+ "process_dataset",
29
+ "filter_quality",
30
+ "DOMAINS",
31
+ "pack_dataset",
32
+ "PackedBatch",
33
+ "InstructionTuner",
34
+ "InstructionExample",
35
+ "parse_instruction_data",
36
+ "evaluate_comprehensive",
37
+ "evaluate_generation_quality",
38
+ "generate_with_kv_cache",
39
+ "batch_generate",
40
+ "KVCache",
41
+ "BenchmarkProfiler",
42
+ "evaluate_benchmarks",
43
+ "DEFAULT_BENCHMARKS",
44
+ "ExperimentTracker",
45
+ "migrate_checkpoint",
46
+ "validate_checkpoint_integrity",
47
+ "safe_load_checkpoint",
48
+ "atomic_save_checkpoint",
49
+ "is_jax_available",
50
+ "get_tpu_info",
51
+ "JAXTrainingEngine",
52
+ ]
53
+
54
+ __version__ = "1.0.0"
55
+ __author__ = "Niranjan Kumar K."
slm/benchmarks.py ADDED
@@ -0,0 +1,169 @@
1
+ """
2
+ SLM Benchmarking and Observability Dashboard
3
+ Measures:
4
+ - TPU initialization latency
5
+ - XLA compilation latency vs steady-state latency
6
+ - First-step vs steady-state step latency
7
+ - Tokens-per-second (TPS) and batches-per-second (BPS)
8
+ - Forward-pass, backward-pass, and optimizer execution times
9
+ - TPU memory consumption (where available)
10
+ - Preserves existing DEFAULT_BENCHMARKS and evaluate_benchmarks API
11
+ """
12
+
13
+ import json
14
+ import time
15
+ from typing import Any, Callable, Dict, List, Optional, Tuple, Union
16
+
17
+ DEFAULT_BENCHMARKS = {
18
+ "unseen_literary_passages": [
19
+ "At dawn, Mira crossed the quiet bridge while the river carried leaves toward the sea.",
20
+ "The old clock stopped during the storm, yet everyone in the house remembered its song.",
21
+ ],
22
+ "general_english_prediction": [
23
+ "A careful reader compares each claim with the evidence before reaching a conclusion.",
24
+ "Rain softened the garden soil, and new green shoots appeared beside the stone path.",
25
+ ],
26
+ "grammar_sentence_completion": [
27
+ "Although the train was delayed, the passengers remained calm and waited on the platform.",
28
+ "She placed the keys beside the lamp so that she could find them in the morning.",
29
+ ],
30
+ "technical_text": [
31
+ "A cache stores frequently used values so later requests can avoid repeating expensive work.",
32
+ "The sensor samples voltage at fixed intervals and records each measurement with a timestamp.",
33
+ ],
34
+ }
35
+
36
+
37
+ def evaluate_benchmarks(model, corpora=None, instruction_tuned=False):
38
+ """Preserved backward-compatible evaluation benchmark function."""
39
+ model._require_trained()
40
+ corpus_set = DEFAULT_BENCHMARKS if corpora is None else corpora
41
+ results = {}
42
+ for name, documents in corpus_set.items():
43
+ texts = [str(item) for item in documents if str(item).strip()]
44
+ sequences = [model.language_model.tokenizer.encode(text) for text in texts]
45
+ results[name] = model.language_model._evaluate(sequences)
46
+ if instruction_tuned and "question_answering" in corpus_set:
47
+ texts = [str(item) for item in corpus_set["question_answering"] if str(item).strip()]
48
+ sequences = [model.language_model.tokenizer.encode(text) for text in texts]
49
+ results["question_answering"] = model.language_model._evaluate(sequences)
50
+ elif instruction_tuned:
51
+ results["question_answering"] = {
52
+ "status": "not_evaluated",
53
+ "reason": "Provide a held-out question-answer corpus under question_answering.",
54
+ }
55
+ else:
56
+ results["question_answering"] = {
57
+ "status": "not_evaluated",
58
+ "reason": "Instruction tuning has not been performed.",
59
+ }
60
+ return {
61
+ "note": "These are reproducible corpus prediction metrics, not a human quality score. Supply fixed held-out corpora for comparable reports.",
62
+ "benchmarks": results,
63
+ }
64
+
65
+
66
+ class BenchmarkProfiler:
67
+ """
68
+ Detailed hardware and runtime profiler for training and inference.
69
+ Separates one-off XLA compilation latency from steady-state step latency.
70
+ """
71
+ def __init__(self):
72
+ self.events: List[Dict[str, Any]] = []
73
+ self.step_durations: List[float] = []
74
+ self.forward_times: List[float] = []
75
+ self.backward_times: List[float] = []
76
+ self.optimizer_times: List[float] = []
77
+ self.data_loading_times: List[float] = []
78
+ self.tpu_init_time: float = 0.0
79
+ self.compilation_time: float = 0.0
80
+
81
+ def record_tpu_init(self, duration: float):
82
+ self.tpu_init_time = duration
83
+
84
+ def record_step(
85
+ self,
86
+ step_idx: int,
87
+ duration: float,
88
+ tokens: int,
89
+ forward_time: Optional[float] = None,
90
+ backward_time: Optional[float] = None,
91
+ opt_time: Optional[float] = None,
92
+ data_time: Optional[float] = None,
93
+ ):
94
+ if step_idx == 1:
95
+ # Step 1 incorporates initial graph tracing and XLA compilation
96
+ self.compilation_time = duration
97
+ else:
98
+ self.step_durations.append(duration)
99
+
100
+ if forward_time is not None:
101
+ self.forward_times.append(forward_time)
102
+ if backward_time is not None:
103
+ self.backward_times.append(backward_time)
104
+ if opt_time is not None:
105
+ self.optimizer_times.append(opt_time)
106
+ if data_time is not None:
107
+ self.data_loading_times.append(data_time)
108
+
109
+ self.events.append({
110
+ "step": step_idx,
111
+ "duration": duration,
112
+ "tokens": tokens,
113
+ "tokens_per_second": (tokens / max(1e-9, duration)),
114
+ })
115
+
116
+ def summary(self, tpu_info: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
117
+ steady_state = self.step_durations if self.step_durations else [self.compilation_time]
118
+ mean_step_sec = sum(steady_state) / max(1, len(steady_state))
119
+ total_tokens = sum(e["tokens"] for e in self.events)
120
+ total_steady_tokens = sum(e["tokens"] for e in self.events if e["step"] > 1) or total_tokens
121
+ total_steady_time = sum(steady_state)
122
+
123
+ steady_tps = total_steady_tokens / max(1e-9, total_steady_time)
124
+ steady_bps = len(steady_state) / max(1e-9, total_steady_time)
125
+
126
+ report = {
127
+ "hardware": tpu_info or {},
128
+ "tpu_initialization_seconds": round(self.tpu_init_time, 4),
129
+ "xla_compilation_seconds": round(self.compilation_time, 4),
130
+ "first_step_latency_seconds": round(self.compilation_time, 4),
131
+ "steady_state_step_latency_seconds": round(mean_step_sec, 6),
132
+ "steady_state_tokens_per_second": round(steady_tps, 2),
133
+ "steady_state_batches_per_second": round(steady_bps, 2),
134
+ "measured_steps": len(self.events),
135
+ "breakdown": {
136
+ "mean_forward_seconds": round(sum(self.forward_times) / max(1, len(self.forward_times)), 6) if self.forward_times else None,
137
+ "mean_backward_seconds": round(sum(self.backward_times) / max(1, len(self.backward_times)), 6) if self.backward_times else None,
138
+ "mean_optimizer_seconds": round(sum(self.optimizer_times) / max(1, len(self.optimizer_times)), 6) if self.optimizer_times else None,
139
+ "mean_data_loading_seconds": round(sum(self.data_loading_times) / max(1, len(self.data_loading_times)), 6) if self.data_loading_times else None,
140
+ },
141
+ }
142
+ return report
143
+
144
+ def format_text_report(self, tpu_info: Optional[Dict[str, Any]] = None) -> str:
145
+ s = self.summary(tpu_info)
146
+ lines = [
147
+ "===========================================================",
148
+ "SLM TPU PERFORMANCE AND OBSERVABILITY REPORT",
149
+ "===========================================================",
150
+ f"Hardware Platform: {s['hardware'].get('platform', 'CPU/Reference')}",
151
+ f"Device Count: {s['hardware'].get('device_count', 1)}",
152
+ f"TPU Detected: {s['hardware'].get('tpu_detected', False)}",
153
+ "-----------------------------------------------------------",
154
+ f"TPU Initialization: {s['tpu_initialization_seconds']:.4f} s",
155
+ f"XLA Compilation (Step 1): {s['xla_compilation_seconds']:.4f} s",
156
+ f"Steady-State Step Latency: {s['steady_state_step_latency_seconds']:.6f} s",
157
+ f"Steady-State Throughput: {s['steady_state_tokens_per_second']:.2f} tokens/s",
158
+ f"Steady-State Batch Rate: {s['steady_state_batches_per_second']:.2f} batches/s",
159
+ "-----------------------------------------------------------",
160
+ ]
161
+ bd = s["breakdown"]
162
+ if any(v is not None for v in bd.values()):
163
+ lines.append("Stage Breakdown (Steady-State):")
164
+ for k, v in bd.items():
165
+ if v is not None:
166
+ lines.append(f" {k}: {v:.6f} s")
167
+ lines.append("-----------------------------------------------------------")
168
+ lines.append("Note: Step 1 compilation overhead is strictly isolated from steady-state throughput.")
169
+ return "\n".join(lines)