easymodd 0.1.0__tar.gz → 0.2.4__tar.gz

easymodd-0.2.4/PKG-INFO

@@ -0,0 +1,466 @@
+Metadata-Version: 2.4
+Name: easymodd
+Version: 0.2.4
+Summary: Composable model training, artifact analysis, secure workflows, and efficient data tooling
+Author: Vnex Lab
+Maintainer: Vnex Lab
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/vnex-lab/EasyMoDD
+Project-URL: Repository, https://github.com/vnex-lab/EasyMoDD
+Project-URL: Issues, https://github.com/vnex-lab/EasyMoDD/issues
+Project-URL: Documentation, https://github.com/vnex-lab/EasyMoDD#readme
+Project-URL: Support, https://ko-fi.com/vnexlab
+Keywords: machine-learning,deep-learning,numpy,cuda,training,transformers,artifact-analysis,decompiler,compiler,encryption
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=2.0
+Provides-Extra: cuda
+Requires-Dist: cupy-cuda12x>=13.6; extra == "cuda"
+Provides-Extra: data
+Requires-Dist: pandas>=2.0; extra == "data"
+Requires-Dist: pyarrow>=15.0; extra == "data"
+Requires-Dist: scikit-learn>=1.4; extra == "data"
+Provides-Extra: security
+Requires-Dist: cryptography>=44; extra == "security"
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+# EasyMoDD
+
+EasyMoDD is a configurable, training-first machine-learning library. It is
+programmable from Python like a numerical library, with reusable tensors,
+modules, datasets, losses, optimizers, trainers, model families, and checkpoint
+policies.
+
+Models initialize randomly unless a consuming project explicitly resumes from
+its own checkpoint. EasyMoDD does not include a chatbot, web application,
+built-in corpus, project `config.txt`, generated checkpoints, or demo data.
+
+## Install
+
+```powershell
+python -m pip install easymodd
+```
+
+For a local checkout:
+
+```powershell
+python -m pip install -e .
+```
+
+The core distribution is source-only plus NumPy; it does not bundle model
+weights, corpora, JDKs, native toolchains, browsers, or decompiler binaries.
+The web console uses Python's standard library. CUDA, tabular-data tooling,
+and AES-GCM encryption are optional extras, keeping a normal install far below
+2 GB. Large datasets, generated apps, models, and user-selected toolchains stay
+outside the library package.
+
+
+The package version is controlled by `project.version` in `pyproject.toml`.
+Increase it for every PyPI release, and do not reuse an already-uploaded
+version.
+
+Optional integrations:
+
+```powershell
+python -m pip install easymodd[cuda]
+python -m pip install easymodd[data]
+python -m pip install easymodd[dev]
+```
+
+## Python API
+
+```python
+import numpy as np
+from easymodd import ArrayDataset, MSELoss, SGD, Trainer, build_mlp
+
+model = build_mlp(input_size=1, hidden_sizes=[8], output_size=1)
+dataset = ArrayDataset(
+    np.array([[1.0], [2.0], [3.0]], dtype=np.float32),
+    np.array([[2.0], [4.0], [6.0]], dtype=np.float32),
+)
+history = Trainer(
+    SGD(model.parameters(), learning_rate=0.01),
+    MSELoss(),
+    max_steps=100,
+).fit(model, dataset)
+print(history.values["loss"][-1])
+```
+
+The programmable foundation includes:
+
+- NumPy and explicit CUDA backend helpers
+- `Module`, `Parameter`, `Linear`, `ReLU`, and `Sequential`
+- GELU, sigmoid, tanh, and dropout layers
+- MLP model factory
+- Model registries for custom factories
+- `ArrayDataset` and packed `TextDataset`
+- `BatchLoader`, deterministic splits, and `Subset`
+- Composable transforms and batch collation
+- MSE and cross-entropy loss contracts
+- L1 and Huber losses
+- SGD, Adam, and AdamW optimizers
+- Cosine and warmup learning-rate schedulers
+- Reusable `Trainer` and `TrainingHistory`
+- Accuracy, MAE, and perplexity metrics
+- JSONL metrics logging and early stopping callbacks
+- Local experiment tracking and dependency-free summaries
+- Versioned model/optimizer/scheduler serialization
+- Environment-only secrets for optional integrations
+- Config loading and validation for consuming projects
+- Checkpoint enable/disable, retention, and compression policy
+- Separate text Transformer and Vision Transformer configuration modules
+- Streaming corpus fingerprints and bounded-memory duplicate audits
+- Memory-mapped text windows for large byte-token corpora
+
+### Corpus audit before training
+
+The streaming auditor fingerprints the exact source file, estimates words and
+tokens, reports blank or malformed records, and detects normalized duplicates.
+It stores only record hashes in a temporary SQLite file rather than loading the
+corpus or all hashes into RAM. Optionally write a deduplicated copy:
+
+```powershell
+easymodd audit-corpus --source corpus.jsonl --format jsonl `
+  --text-field response.text --deduplicated-output cleaned.jsonl --json
+```
+
+Plain UTF-8 text is treated as newline-delimited records. JSONL is validated
+one record at a time. The SHA-256 fingerprint and audit counts can be saved
+with an experiment to make dataset changes visible before expensive training.
+
+For training on a corpus larger than available RAM, use `MMapTextDataset`:
+
+```python
+from easymodd import MMapTextDataset
+
+with MMapTextDataset("corpus.txt", sequence_length=512, stride=512) as dataset:
+  inputs, next_tokens = dataset[0]
+  print(len(dataset), inputs.shape, next_tokens.shape)
+```
+
+It uses raw UTF-8 bytes as token IDs 0-255, keeps the source file memory-mapped,
+and materializes only the requested sequence window. Use `stride < sequence_length`
+for overlapping windows; this increases training examples and I/O.
+
+The restored prototype scripts remain available as compatibility programs for
+existing custom checkpoints. New projects should import EasyMoDD directly.
+
+## Consumer Project Configuration
+
+EasyMoDD does not force users into one project layout. A consuming project can
+create its own `config.txt` using TOML syntax and load it with:
+
+```python
+from easymodd import load_config
+config = load_config("config.txt")
+```
+
+The config loader supports project-defined paths for datasets, runs, models,
+logs, and checkpoints. The library never assumes a filename such as
+`best.bin`.
+
+A consumer project can configure checkpoint behavior like this:
+
+```toml
+[checkpoints]
+enabled = true
+keep_recent = 2
+compress_old = true
+```
+
+When disabled, `CheckpointManager` creates no directory or files. When enabled,
+older numbered checkpoints can be compressed while recent checkpoints remain
+available.
+
+## Package Layout
+
+```text
+src/easymodd/
+  models/
+    mlp.py
+    text_transformer.py
+    vision_transformer.py
+  data.py
+  losses.py
+  nn.py
+  tensor.py
+  optim.py
+  training.py
+  checkpoints.py
+  config.py
+  cli.py
+```
+
+Each model family has its own module. The shared training contracts are designed
+so new model types can be added without changing dataset or checkpoint code.
+
+## Extending EasyMoDD
+
+Register a custom model factory:
+
+```python
+from easymodd import ModelRegistry
+
+registry = ModelRegistry()
+registry.register("my_model", lambda width=8: build_my_model(width))
+model = registry.create("my_model", width=32)
+```
+
+Optional integrations can read environment variables without storing secrets:
+
+```python
+from easymodd import SecretStore
+
+secrets = SecretStore(prefix="MY_APP_")
+token = secrets.get("TRACKING_TOKEN")
+```
+
+EasyMoDD never writes API keys to configs, checkpoints, experiment files, or
+logs. Provider-specific integrations can be built on top of this helper.
+
+## Data and Training
+
+The library does not ship a general-purpose training database. Users provide
+legally obtained text, image, or tabular data in their own projects. EasyMoDD
+provides dataset contracts and validation helpers; it does not pretend that a
+small demo corpus can produce a general assistant.
+
+## Development Status
+
+The tensor/module API, programmable MLP training loop, dataset contracts,
+checkpoint policy, model-family layout, and config loader are implemented.
+Text Transformer training, Vision Transformer training, full resume manifests,
+evaluation, schedulers, callbacks, and additional model families are active
+parts of the public-library roadmap.
+
+Support development on [Ko-fi](https://ko-fi.com/vnexlab).
+
+## Tests and examples
+
+The `tests/` directory contains individual subsystem tests plus a full-stack
+integration test. `examples/multi_file_project/` demonstrates a real consumer
+project layout with TOML config, multiple Python plugin files, custom web
+routes/pages, C++ source, a CMake shared-library recipe, and a custom DLL
+metadata backend.
+
+The multi-file test loads that consumer config, discovers the C++ source,
+registers the custom backend/API/page, and inspects a synthetic PE DLL without
+loading or executing it. A C++ compiler command is dry-run only and is skipped
+when no compatible compiler is installed. Tests also cover Java JAR/class,
+Python bytecode, WebAssembly, training, corpus auditing, encryption, web auth,
+simulation, math helpers, and local forwarding.
+
+Run the suite with:
+
+```powershell
+python -m pip install -e ".[dev,security]"
+python -m pytest -q
+```
+
+If a native C++ compiler is installed, the suite also verifies the compiler
+adapter's dry-run command. It does not execute generated DLL/EXE fixtures.
+
+## EasyModel artifact tooling
+
+This distribution also includes the separate `EasyModel` namespace for safe,
+configurable artifact inspection, simulations, app scaffolding, networking, and
+compiler/decompiler adapters. It does not replace the `easymodd` training API.
+
+The namespaces connect through `EasyModel.Workbench` and a bounded,
+thread-safe event bus. Training, simulations, corpus audits, artifact
+inspection, encryption, and custom tools publish redacted events to one local
+run history. `EzMHandler` is the root facade for coordinated operations.
+
+```python
+from EasyModel import SecurityPolicy, detect_artifact
+from EzDecompiler import EzDecompiler
+
+handler = EzDecompiler(policy=SecurityPolicy(allow_native_tools=False))
+artifact = handler.detect("application.jar")
+report = handler.inspect("application.jar")
+print(artifact.kind, report.to_json())
+```
+
+From a source checkout:
+
+```powershell
+python EzDecompiler.py detect application.jar --json
+python EzDecompiler.py inspect application.jar --json
+python EzDecompiler.py decompile application.jar --dry-run --json
+```
+
+After installation:
+
+```powershell
+easymodel detect application.jar --json
+easymodel inspect application.jar --json
+easymodel passport application.jar --json
+easymodel compare original.jar --against rebuilt.jar --json
+easymodel web
+easymodel lan
+easymodel app --name my_toolbox --directory ./my_toolbox --template toolbox
+```
+
+An Artifact Passport adds a streaming SHA-256 identity plus safe structural
+metadata: JAR entry and manifest inventory, archive traversal/duplicate-name
+warnings, PE sections and .NET marker detection, Python bytecode header details,
+and WebAssembly section sizes. It never extracts or executes the artifact.
+Passports can be compared to see whether rebuilt or transformed outputs changed.
+
+### Local web workbench
+
+`easymodel web` starts a standard-library web console at `127.0.0.1:8765`. It
+combines artifact passports, corpus audits, file encryption, and recent shared
+events. The server prints a random session token; enter it in the page. Remote
+binding is rejected unless explicitly enabled with a token, and remote use must
+be behind HTTPS. Keyboard shortcuts only operate inside the open browser page;
+the library does not capture system-wide keystrokes.
+
+Trusted plugins can add isolated browser pages and authenticated POST API
+routes. A custom page can call registered routes through the parent console
+bridge without receiving the session token. Plugins execute Python code, so
+load only modules you trust.
+
+### Custom pages and API plugins
+
+Consumer-owned TOML controls plugin loading and the web address:
+
+```toml
+[plugins]
+enabled = true
+modules = ["my_project.easy_plugins"]
+
+[web]
+host = "127.0.0.1"
+port = 8765
+allow_remote = false
+```
+
+Each configured module exposes `register(workbench)`:
+
+```python
+from EasyModel import ApiRoute, ArtifactKind, WebPage
+from EasyModel.compilers import CompilerBackend
+from EasyModel.decompilers import ExternalDecompiler
+
+def register(workbench):
+  workbench.register_component("tools", "hello", lambda: {"ready": True})
+  workbench.register_compiler_backend(CompilerBackend("cc", "cc"))
+  workbench.register_decompiler_backend(ExternalDecompiler(
+    "my-java-decompiler", "my-decompiler", {ArtifactKind.JAR, ArtifactKind.CLASS}
+  ))
+  workbench.register_page(WebPage(
+    "/pages/status", "Project status",
+    "<main><h1>Project status</h1><p>Custom page content.</p></main>",
+  ))
+  workbench.register_api_route(ApiRoute(
+    "/api/custom/status", lambda body, hub: {"ready": True}
+  ))
+```
+
+The page runs in a sandboxed frame. It can ask the parent workbench to call
+only registered custom POST routes:
+
+```javascript
+window.parent.postMessage({
+  type: "easymodel-api",
+  path: "/api/custom/status",
+  payload: {}
+}, "*");
+```
+
+The browser hosts custom page HTML in a sandboxed frame. Custom API routes are
+POST-only and require the same session authentication as built-in operations.
+
+### App builder, simulation, and math
+
+Scaffold a separate consumer project without overwriting existing files:
+
+```powershell
+easymodel app --name lab_app --directory ./lab_app --template web
+easymodel app --name queue_sim --directory ./queue_sim --template simulation
+easymodel app --name model_lab --directory ./model_lab --template training
+```
+
+The `DiscreteEventSimulator` provides deterministic event scheduling, seeded
+randomness, and a maximum event count. Register simulation factories under
+the `simulations` component group and run them with `easymodel simulate --config
+config.toml --component your_simulator`. `EasyModel.mathx` provides stable
+softmax, blockwise pairwise distances, linear solves, and Simpson integration
+using NumPy's optimized kernels, finite-difference gradients, and deterministic memory-bounded Monte Carlo integration.
+
+The shared component registry includes starter providers for MLP models,
+discrete-event simulations, JAR inspection, project scaffolding, corpus audit,
+and artifact passports. Custom plugins may register more models, simulators,
+compilers, decompilers, apps, browser pages, and authenticated API routes.
+
+Compiler commands are blocked unless execution is explicitly allowed. Inspect
+the structured command first with `--dry-run`; only execute a toolchain and
+source you trust:
+
+```powershell
+easymodel compile main.c --backend cc --output app --dry-run --config config.toml
+easymodel compile main.c --backend cc --output app --allow-execution --config config.toml
+```
+
+### LAN hosting and TCP forwarding
+
+`easymodel lan` reports private interface addresses. To intentionally make the
+console reachable on a trusted LAN, use an environment token and explicitly
+enable remote binding:
+
+```powershell
+$env:EASYMODEL_WEB_TOKEN = "a-long-random-token"
+easymodel web --host 0.0.0.0 --allow-remote
+```
+
+`easymodel forward --port 9000 --target-host 127.0.0.1 --target-port 8000`
+forwards a local TCP port. Non-loopback bind addresses require
+`--allow-remote`. Forwarding does not edit router/firewall rules, configure
+UPnP, or create an Internet tunnel. External reachability remains the network
+owner's responsibility; never expose an unauthenticated service.
+
+### Streaming file encryption
+
+Install the optional security extra and keep passphrases in the environment,
+not shell history:
+
+```powershell
+python -m pip install "easymodd[security]"
+$env:EASYMODEL_PASSPHRASE = "use-a-long-unique-passphrase"
+easymodel encrypt confidential.bin --output confidential.bin.emenc
+easymodel decrypt confidential.bin.emenc --output restored.bin
+```
+
+Encryption uses chunked AES-256-GCM with authenticated headers/chunks,
+truncation detection, and atomic output. Files stream through fixed-size
+buffers rather than loading entirely into RAM. Passphrases are never saved in
+configs, event logs, or file metadata. Losing the passphrase means the file
+cannot be recovered.
+
+Supported artifact families include Java JAR/class, Windows PE EXE/DLL,
+.NET assemblies, Python bytecode, and WebAssembly detection. The first
+release provides safe built-in JAR inspection plus optional external-tool
+adapters. Native binaries can produce headers, imports, assembly, or
+pseudocode, but cannot reliably restore original optimized source code.
+
+Compiler adapters use installed toolchains such as `javac`, `.NET`, Python,
+GCC/Clang/MSVC, or WebAssembly tools. Tool execution is policy-controlled and
+disabled by default unless explicitly enabled by the consumer project.
+
+Only analyze software you own or have permission to inspect. EasyModel does
+not provide DRM, anti-tamper, or access-control bypass functionality.