initium-deeplearning 0.1.0__tar.gz → 0.3.0__tar.gz

{initium_deeplearning-0.1.0 → initium_deeplearning-0.3.0}/.gitignore

@@ -216,3 +216,8 @@ __marimo__/
 
 # Streamlit
 .streamlit/secrets.toml
+
+# Experimental documents for context, not to be committed
+documents/*
+dnc-context/*
+tmp/*

initium_deeplearning-0.3.0/PKG-INFO

@@ -0,0 +1,411 @@
+Metadata-Version: 2.5
+Name: initium-deeplearning
+Version: 0.3.0
+Summary: Differentiable Neural Computer research framework with pluggable LSTM/Mamba/CfC controllers, MoE, split-graph and stochastic write heads.
+Project-URL: Homepage, https://github.com/samueleternity/initium
+Project-URL: Issues, https://github.com/samueleternity/initium/issues
+Author: samueleternity
+License-Expression: GPL-3.0
+License-File: LICENSE
+Keywords: differentiable-neural-computer,dnc,liquid-neural-networks,mamba,pytorch
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: dnc>=1.1.0
+Requires-Dist: ncps>=0.0.7
+Requires-Dist: numpy>=1.26
+Requires-Dist: torch>=2.4
+Provides-Extra: real-data
+Requires-Dist: opencv-python>=4.8; extra == 'real-data'
+Requires-Dist: tokenizers>=0.19; extra == 'real-data'
+Provides-Extra: text
+Requires-Dist: tokenizers>=0.19; extra == 'text'
+Provides-Extra: video
+Requires-Dist: opencv-python>=4.8; extra == 'video'
+Description-Content-Type: text/markdown
+
+# Initium
+
+Initium is an experimental research framework for combining a Differentiable Neural Computer (DNC)/augmented memory with interchangeable sequence controllers and task/data pipelines. Its central idea is to keep an explicit, addressable external memory while exploring controllers such as LSTM, Mamba-family state-space models, and closed-form continuous-time (CfC) networks. The code also includes stochastic memory-write regularization, sparse expert layers, split-graph models, memory-structure experiments, and checkpoint-driven inference.
+
+> **Research status:** Alpha and actively changing. Results below are experiment-specific observations, not guarantees that a configuration will reproduce on another machine, code revision, or dataset. This README is deliberately maintained as a working guide: update it when the code, default configuration, or evidence changes.
+
+## What the project is investigating
+
+The project studies how a neural controller can interact with a differentiable external memory on sequential tasks, especially graph traversal. It asks questions at several levels:
+
+1. **Memory and controller:** Can a controller process a sequence and use DNC read/write operations to store and retrieve information needed later?
+2. **Generalization:** How does performance change between the training curriculum and a held-out or out-of-distribution (OOD) evaluation distribution?
+3. **Write regularization:** Does placing a probabilistic prior on each memory write improve generalization, and what are the costs or seed-to-seed variations?
+4. **Compute structure:** Can the controller's sequence computation be parallelized or sparsified while preserving the sequential memory interface?
+5. **Task breadth and deployment:** Can trained checkpoints be evaluated through a common inference interface across graph, text, audio, video, and multimodal tasks?
+
+These are research questions, not all established project claims. In particular, supported dataset interfaces do not imply that every modality has been trained and validated to the same standard as graph traversal.
+
+## Launch
+
+You can either fork/clone the repository and customize everything you need or you can use pip/pdm/uv for ready to use installation.
+
+Guide for pip usage:
+
+To install the whole framework: pip install initium-deeplearning
+
+For ANY mamba related operation you need: python -m initium.setup_wheels 
+
+Optionally you can try:
+pip install "initium-deeplearning[text]"
+pip install "initium-deeplearning[video]"
+pip install "initium-deeplearning[real-data]"
+
+To see all CLI optionality:
+initium-infer --help
+python initium.core_training --help
+
+Example usage:
+
+python -m initium.core_training 0.0 --controller lstm
+python -m initium.core_training 0.0 --split-graph --split-graph-variant mamba1 ...
+
+Inference runs need a checkpoint first. The built-in tool analyzes it and refuses unsupported task types or dimensions. Dataset preparation-only mode does not need a checkpoint.
+
+### Reusing prepared datasets
+
+Training and inference can save the processed dataset state so the raw source is not parsed or tokenized again. `--save DIRECTORY` writes to `DIRECTORY/<type>/<source> [prepared]/dataset.pkl`; `--load-prepared` accepts that prepared folder (or a save root containing exactly one prepared dataset of the requested type). When loading, `--dataset-link` is optional. Use `--prepare-only` with `--save` to prepare and save a dataset without starting a run:
+
+```bash
+python -m initium.core_training 0.0 --dataset-type text --dataset-link ./corpus.txt --save ./prepared --prepare-only
+python -m initium.core_training 0.0 --dataset-type text --load-prepared "./prepared/text/corpus [prepared]"
+initium-infer checkpoint.pt --dataset-type text --load-prepared "./prepared/text/corpus [prepared]"
+initium-infer --dataset-type text --dataset-link ./corpus.txt --save ./prepared --prepare-only
+```
+
+Prepared artifacts retain the resolved train/test pools or graph mappings, including the split used when they were created. They are Python pickle files and should only be loaded from sources you trust.
+
+## High-level flow
+
+```text
+Configuration and random seed
+        │
+        ▼
+Dataset registry ──► task episodes / curriculum ──► batch and sequence tensors
+                                                          │
+                                                          ▼
+                                            Controller + DNC external memory
+                                            ├─ controller processes sequence
+                                            ├─ write head updates memory
+                                            └─ read heads return memory content
+                                                          │
+                                                          ▼
+                                   task loss + optional KL / MoE auxiliary losses
+                                                          │
+                                                          ▼
+                           optimizer, evaluation, logs, checkpoints, curriculum
+                                                          │
+                                                          ▼
+                      checkpoint loader ──► task inference ──► metrics and run logs
+```
+
+The training core delegates task construction and task-specific behavior to a dataset interface. The model combines a selected controller with DNC memory operations. During training, the loop computes task and optional auxiliary objectives, updates model parameters, periodically evaluates ID and OOD performance, advances the curriculum when its criteria are met, writes diagnostics, and saves resumable checkpoints. Inference reconstructs a model from a checkpoint, checks that the requested task and dimensions are supported, creates the task runner, and reports episode metrics. Optional inference caches can reuse eligible results or shared prefixes.
+
+## Architecture and major components
+
+### Differentiable external memory
+
+The DNC supplies a memory matrix and differentiable read/write interface. The memory can retain information beyond a controller's immediate hidden state. Content-based addressing, read heads, and temporal-link structures are part of the underlying DNC implementation. The temporal link matrix has quadratic cost in the number of memory cells, which motivates the project's link-matrix ablations and memory-size investigations.
+
+### Read/write mechanism roadmap
+
+The split-graph memory exposes independent read and write variants while retaining the DNC usage, allocation, write addressing, read modes, and temporal-link machinery:
+
+- `--dnc-read-variant cosine` is the DNC baseline. `relational-mlp` replaces cosine similarity with a learned score for each `(query, memory row)` pair. `relational-residual` adds a scaled learned pairwise score to cosine and zero-initializes its final layer, so it starts exactly at the DNC baseline. `--dnc-read-residual-scale` controls that correction's strength (default `1.0`). Track A logs the correction-to-cosine RMS ratio to show whether it contributes meaningfully. Write content addressing remains cosine-based for both learned readers.
+- `--dnc-write-variant learned` is the DNC baseline. `kanerva-closed-form` turns the existing write-value projection into an observation and applies a per-row online isotropic Gaussian posterior update. It does not import the Kanerva Machine's exchangeable episode objective or claim to reproduce its full generative model.
+- The variants compose, so the four baseline / Track A / Track B / combined cells are expressible through the same training and checkpoint pipeline. The closed-form writer uses a deterministic observation projection; the stochastic write head is not installed for that variant.
+- The graph curriculum emits ID, OOD, and OOD-offset metrics as before. Graph runs also write `run_<id>_relational_probe.csv`, with cosine top-1, selected read-head top-1, and held-out-composition accuracy. `data/graph_traversal/order_probe.py` supplies paired prefix-permutation helpers for the Q99 order-sensitivity evaluation.
+- Split-graph graph runs also write `run_<id>_track_a.csv` and `run_<id>_track_b.csv`. Track A records ID/OOD accuracy, relational stress and held-out-composition accuracy, ordinary-input read-score/cosine correlation, and read-content entropy/maximum weight. Track B records the Q99 order-sensitivity gate, ID/OOD posterior variance and write-update diagnostics, and training-window loss, gradient norm, AMP scale, and learning rate. These are experiment logs; no training or evaluation results are checked in.
+
+The DNC implementation defaults to `--base-dnc`. `--dnc-fast-allocation` uses the exact allocation implementation that reuses the stock sorted usage permutation and removes its second full sort; `--dnc-compile-memory` compiles the per-timestep memory forward with `torch.compile`. These preserve DNC equations and state-dict keys. `--dnc-allocation-topk K` is an independent approximation that allocates among only the K least-used cells. It can reduce selection work, but omits allocation mass outside the selected cells and may affect learning. `--dnc-link-sharpness [EXPONENT]` is a parameter-free learning variant inspired by arXiv:1904.10278 that sharpens the shape of forward/backward temporal-link reads while preserving their original total mass; it defaults to exponent 2. Both functional variants can be started from a lesson-14 checkpoint. `--optimized-dnc` enables fast allocation and compiled memory forward; for split-graph runs it also defaults to top-64 allocation and exponent-2 link sharpening. Compilation is lazy, so compare sustained steps/s after warm-up. The exact execution options do not remove dense temporal-link cost.
+
+`--link-matrix-mode ablated` bypasses temporal-link updates and traversal products, which saves dense link work but removes ordered recall; `sparse_topk` currently masks a dense update and therefore is not a speed optimization. The paper evidence is task-dependent: rsDNC reports that temporal links contribute little on its QA workload, while the original DNC reports sparse links working on copy but link removal failing there. Split-graph traversal chains multiple hops, so link removal stays an isolated comparison, never part of `--optimized-dnc`. Use `--batch-size` to sweep larger episode batches; this can improve utilization but increases memory use and changes gradient noise. Compare equal seeds and optimizer-step budgets, and report both steps/s and steps-to-target accuracy.
+
+Gradient checkpointing is omitted from the speed bundle because it saves activation memory by recomputing work in backward. Truncated BPTT is omitted because graph facts needed for a later query can precede it by many steps; detaching DNC state can remove that learning signal. A hybrid scan over guessed addressing weights is plausible mathematically, since memory updates become affine once weights are fixed, but the addressing depends on the evolving memory. It would need an iterative solver and convergence/accuracy evidence; at the current short lesson-1 sequences, its repeated passes may cost more than the sequential loop. These are follow-up research directions rather than robust drop-in optimizations.
+
+The paper review supports this staged approach. Sparse Access Memory reports much faster sparse reads/writes, including a temporal-association variant, but it changes the addressing/indexing mechanism and is not a drop-in exact DNC optimization. BPPSA and parallel-scan work apply to linear recurrences; the DNC loop here has a nonlinear read-to-next-write dependency, so parallelizing timesteps would require a changed model. The rsDNC work's slimmer memory and link changes were developed for question answering, and the local link ablation already gives us a direct task-specific control. For now, compilation and exact allocation are the faithful speed tests; sparse addressing remains a distinct research branch.
+
+Use the selected split-graph baseline with either supported controller combiner:
+
+```bash
+python -m initium.core_training 0.0 --split-graph --split-graph-variant mamba1 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc
+python -m initium.core_training 0.0 --split-graph --split-graph-variant mamba1 --split-graph-combiner-mode controller --split-graph-combiner-variant mamba+cfc
+```
+
+### Phase 0 shared workspace (MoE broadcast)
+
+The optional `--workspace` path adds a persistent shared slot matrix to a split-graph model with MoE enabled. The final parallel-backbone MoE bank supplies the specialist candidates and its existing noisy top-k routes. At each sequence timestep, selected expert outputs compete for slot writes; a learned RMC-style gate blends proposed slot content with the previous matrix. Every expert in that bank then attends to the updated matrix, and the existing sparse MoE routes combine the context-enriched expert outputs. The workspace resets at the beginning of each episode and persists across its timesteps. The `--moe-backbone-residual-scale` setting scales the complete final-bank delta, including its broadcast contribution, just as it scales the ordinary MoE residuals in the selection-only backbone. The DNC memory uses the same fp32/autocast-disabled path with or without workspace. This is a shared-object broadcast implementation; it does not implement per-recipient translation.
+
+Use the Phase 1 matched split-graph/CfC-combiner setup with MoE and the workspace enabled as follows:
+
+```bash
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix phase2-workspace-coadapted --split-graph --split-graph-variant mamba1 --split-graph-num-blocks 2 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement all --moe-backbone-residual-scale 0.1 --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --workspace --workspace-num-slots 5 --workspace-train-mode coadapted --total-steps 30000
+```
+
+`--workspace` requires both `--split-graph` and `--moe`. The default is 5 slots, based on Goyal et al.'s shared-workspace reference result; it is a starting configuration, not a claimed optimum for graph traversal. `--workspace-train-mode coadapted` trains the full model. Choose `frozen` to train only workspace parameters while freezing the backbone, MoE/combiner, DNC, stochastic write heads, and task output projection. `--workspace-no-selection` replaces competitive selected writes with random unweighted writes for the no-Selection control. `--workspace-reset-each-step` resets slots before each timestep to test persistence. `--workspace-num-slots 0` bypasses workspace writes and readback, providing a plumbing comparison against the existing MoE model. Run IDs encode these conditions to avoid log/checkpoint collisions.
+
+For a frozen-workspace experiment, pass a compatible workspace-disabled baseline checkpoint through `--resume`. Initium detects this case, verifies the saved architecture/settings and shared tensor shapes, loads the matching backbone, MoE, combiner, DNC, stochastic-write, and task-head weights, and keeps the newly added workspace parameters at their fresh initialization. It then resets the optimizer, scaler, RNG streams, training step, and curriculum to step 0 / lesson 1 so the workspace learns across the full curriculum. The checkpoint's final memory size is used to load its weights, then memory is resized to the first lesson's configured size. This warm-start accepts only matching beta/model settings and rejects missing or mismatched non-workspace weights. A checkpoint that already includes workspace state follows the ordinary exact-resume path, including optimizer and progress restoration. Example: add `--resume checkpoints/<baseline-checkpoint>.pt` to the frozen-workspace command above and use a distinct `--run-id-suffix` for the new experiment.
+
+The console prints a startup line that reports whether the workspace is active or bypassed, its capacity, write-selection mode, persistence setting, train mode, and active compute precision. Active workspace attention and slot-state updates run in fp32 outside CUDA autocast; zero-slot/closed-gate bypass retains the selection-only path. At the normal training log cadence, `workspace[...]` reports selected write mass and active-write fraction, the learned write-gate mean, slot-state RMS, readback-to-expert-output RMS ratio, and read-gate value. These values show whether the path is active and the relative scale of its write and broadcast signals; they do not by themselves establish a task benefit.
+
+The shared object operates on the final backbone MoE bank; earlier backbone MoE layers retain their existing behavior. This is the Phase 0 plumbing target, not evidence that Broadcast improves task performance. The first short run should compare the zero-slot bypass against the same MoE configuration without `--workspace`; only after this plumbing gate should the frozen/co-adapted and no-Selection comparisons begin. Record seed, full command, commit, hardware/software environment, curriculum completion, ID/OOD metrics, convergence steps, and any failures in `07-Experiment/Broadcasting/Progress/Phase-0-Implementation.md`. All current evidence is graph-traversal-specific; transfer across task families remains untested.
+
+### Phase 4 experimental bistable write gate
+
+The default `--workspace-gate-variant continuous` preserves the Phase 0–3 sigmoid update. The opt-in `bistable` variant treats each workspace slot as a competing write candidate. It combines the workspace's content-selection score and the MoE route confidence with independent Gaussian background noise, then iterates recurrent self-excitation and lateral inhibition before making a hard write/no-write decision. A straight-through estimator carries gradients through the pre-commit probabilities. This is an experimental adaptation of GNW ignition, not a mechanism implemented by the source neuroscience papers.
+
+For the matched gate-only comparison, resume the same five-slot frozen-workspace checkpoint twice. The frozen control updates only the existing continuous `write_gate`; the bistable condition updates only the new recurrent gate parameters. The gate-variant warm start loads shared model/workspace weights, initializes only the new gate, and starts with a fresh optimizer. `--workspace-train-mode coadapted` remains available to train the full model from that same checkpoint as a separate condition. Bistable background noise uses an independent checkpointed RNG stream, so zero-noise and selection-strength ablations do not change the graph episode RNG stream.
+
+```bash
+python -m src.initium.core_training 0.0 --seed 0 --resume <step-8000-workspace-checkpoint.pt> --run-id-suffix phase4-continuous-gate-pilot --split-graph --split-graph-variant mamba1 --split-graph-num-blocks 2 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement all --moe-backbone-residual-scale 0.1 --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --workspace --workspace-num-slots 5 --workspace-train-mode frozen --workspace-train-scope gate --workspace-gate-variant continuous --total-steps 8100
+```
+
+Use the same command, checkpoint, seed, and step budget for the bistable condition, changing the suffix and adding `--workspace-gate-variant bistable`. Gate-only runs use a safety tripwire that aborts before backward/update if any instrumented segment exceeds `--workspace-stability-max-activation` (default `100000`). Segment diagnostics report activation/gradient RMS and maxima; bistable diagnostics additionally report the pre-commit probability histogram (10 bins), low/middle/high occupancy, hard commit frequency, writer frequencies, slot RMS, and readback scale during training and held-out ID/OOD evaluation. A zero-noise ablation uses `--workspace-bistable-noise-scale 0`; selection-strength sweeps use `--workspace-bistable-selection-scale` while holding `--eval-seed` fixed. The gate still samples noise during evaluation unless `--workspace-bistable-deterministic` is supplied.
+
+The preregistered operational bimodality check is at least 10% of held-out candidate probabilities below 0.1, at least 10% above 0.9, and at most 20% in [0.1, 0.9], sustained across episodes. Also require finite, nonzero gate gradients, no repeated skipped optimizer updates, and no task or paired-memory-dependency collapse. Run the short stability/pilot budget first; do not start a long Phase 4 run unless the histogram and stability gates pass. Results remain specific to graph traversal and the single available trained model seed.
+
+After saving a pilot checkpoint, evaluate that same checkpoint and `--eval-seed` under each `--eval-workspace-mode checkpoint`, `reset-each-step`, and `bypass`. For noise and selection ablations, use the same `--eval-seed` and change only the corresponding bistable scale. The gate RNG is separately checkpointed, so these changes keep the episode stream fixed; use distinct run suffixes when recording each condition.
+
+### MoE numerical and memory-use diagnostics
+
+MoE experts now initialize as a zero-output residual branch: each expert's final projection starts at zero, so enabling MoE preserves the original controller signal at step 0 while the output projections learn. Top-1 routing retains the selected expert's probability from the full expert softmax; normalizing a one-expert selected set would turn every gate into 1 and eliminate task-loss gradients to the router. Top-k greater than one continues to normalize over selected experts. The CfC multi-source combiner likewise retains its original affine projection over the concatenated backbone and previous-read inputs; its source-aware experts contribute an additive correction around that baseline. Capacity slots go to the highest-confidence routes rather than the earliest rows in the flattened token batch, avoiding a systematic advantage for the first source when source batches share an expert's capacity. CfC source diagnostics report accepted and dropped route fractions separately for each source at training-log cadence.
+
+The SwitchMoE router projection and expert MLPs run in FP32 even when the surrounding model uses CUDA autocast; low-precision inputs receive FP32 MoE outputs so large finite expert values are not converted into fp16 infinities at the residual boundary. Ordinary routing dispatches activations into fixed-capacity per-expert buffers and applies each expert's original Linear parameters directly. It does not expand or gather full expert weight matrices per token. This matters for recurrent controllers: the intermediate batched-weight route materialized a 512×1536 projection matrix for every route and autograd retained those copies across timesteps. With 32 source tokens, one such projection copy was 96 MiB; input and output projections were both affected. Each expert now evaluates its capacity buffer, including any unused capacity rows. Route-expert and route-slot indices are both shaped `(tokens, top_k)` before gathering; this avoids advanced-index broadcasting across unrelated tokens. The workspace's specialist-candidate path intentionally evaluates every expert because broadcast requires all candidates, but it shares the expert parameters instead of copying them per token. Training logs include the fraction of selected routes dropped by capacity. Router and per-source diagnostics stay on-device during recurrent steps and synchronize only when needed for log output. The load-balancing loss is averaged across repeated calls to a recurrent MoE layer (for example, one call per CfC timestep), then summed across distinct MoE layers; its weight no longer scales with episode length.
+
+Periodic memory-dependency checks replay the exact random episode batch used by the corresponding memory-on ID evaluation, then restore the RNG streams. This makes `memory on - memory off` a paired functional ablation rather than a difference between two independently sampled batches. The training loop also clips all trainable optimizer parameters, including the task output projection, stops before backward on non-finite forward outputs/losses, reports the first parameter with a non-finite unscaled gradient, logs skipped non-finite-gradient updates, and aborts after eight consecutive such steps instead of continuing a run that is no longer updating safely.
+
+The supplied Phase 1/2 traces are pre-fix evidence. One Phase 2 trace first reports a non-finite task loss at step 11,665, recurrent outputs become non-finite at step 11,666, and the process continues to step 14,106 with an AMP scale of zero before manual interruption. Memory-on/off checks in those logs used separate random episode batches, so their small positive or negative differences are not reliable paired evidence. See `07-Experiment/Broadcasting/Progress/Phase-1-MoE-Failure-Analysis.md` for the causal breakdown and re-run gate. Re-run the Phase 1 selection-only condition before Phase 2; keep the proposed 8-expert, top-1 configuration and matched seed/settings fixed while checking finite gradients, per-bank routing, and paired memory dependency.
+
+Training console logs now show **one MoE-bank record per 100-step logging window**: route and router-importance fractions by expert, router entropy and maximum probability, input/expert/MoE-delta activation RMS and maximum magnitude, capacity-drop fraction, and accepted gate mass. The `moe_residual:` segment records the complete MoE block output after its skip connection, separately from `moe_delta_rms/max`. A separate `[segment-diag:train]` line reports each recognized module's call count, input/output RMS and max, non-finite fraction, boundary input/output gradient RMS and non-finite fraction, and hierarchical parameter-gradient norms. Boundary gradients are collected during training backward and during failure traces; evaluation remains forward-only. The collector recognizes Mamba and CfC backbones/blocks, Mamba/CfC controller wrappers/blocks/cells, MoE blocks, chained controllers, DNC memory, workspace, and task output; names include their module paths, so the same logger works across supported controller combinations.
+
+Split-graph DNC interface transforms, cosine content addressing, usage/allocation, write/read weighting, temporal links, and recurrent memory updates now run in fp32 even when the outer controller uses CUDA autocast. The stock cosine rule divides by the memory/key norm product plus `1e-6`; the instrumented trace showed finite score-side gradients but a non-finite key-side gradient, so fp32 protects that path from low-precision intermediate-gradient overflow. The startup log and checkpoint metadata identify this precision choice. The equations remain unchanged.
+
+To inspect DNC internals, add `--dnc-backward-trace` to a split-graph training command. The flag watches gradients at read/write interface values, cosine-addressing scores/logits/softmax, usage, allocation, write weights, temporal links, precedence, recurrent memory, and read vectors, plus the patched CUDA `torch.prod`/`torch.cumprod` outputs and their inputs. It reports every distinct hooked boundary with a non-finite gradient on the first affected optimizer step, rather than stopping after the first hook label, and includes affected batch rows. At the regular logging interval and on failure, `[dnc-numerics:...]` reports memory/read/link magnitudes, read-weight mass violations, and per-product input min/max, fraction below `1e-6`, negative/non-finite fraction, and call count. The flag synchronizes at watched backward boundaries, so use a short diagnostic capture around a known first event. Earlier captures reported their first bad memory-input gradients at steps 1,120 and 626; the matched fp32 validation run below completed 5,000 steps without a bad DNC hook or non-finite-gradient skip.
+
+```bash
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix dnc-backward-trace --split-graph --split-graph-variant mamba1 --split-graph-num-blocks 2 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement all --moe-backbone-residual-scale 0.1 --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --total-steps 1800 --dnc-backward-trace
+```
+
+Keep other flags exactly matched to the run being investigated. The fp32 DNC path removed the observed bad-key-gradient pattern in the 5,000-step matched validation capture. That run also acquired strong paired ID memory dependence by step 2,500; see `AI-Architecture-Research/07-Experiment/Broadcasting/Progress/Phase-1-MoE-Failure-Analysis.md` for the complete result and remaining generalization limits. Repeat across seeds before treating the numerical and learning outcomes as robust.
+
+Evaluation emits separate channel lines for ID, OOD, memory-off, combiner-stage ablations, and final ID/OOD checks. These summarize the evaluations independently from training windows. If a forward value or gradient becomes non-finite, the current segment window and MoE-bank window are printed immediately before training stops or skips the unsafe update. The summaries aggregate detached device-side scalars and synchronize at log/evaluation boundaries; they do not retain activation graphs.
+
+To isolate where split-graph MoE changes the memory-learning transition, `--moe-placement` selects which component receives expert banks while keeping the model, seed, and router settings matched. `all` (the default) preserves the existing behavior; `backbone` enables MoE only in the parallel backbone; `combiner` enables MoE only in the sequential controller combiner. For the CfC combiner, the latter is its source-aware multi-source MoE bank. Component placement requires `--moe --split-graph`; combiner-only additionally requires controller combiner mode. The workspace requires backbone MoE and therefore cannot be combined with combiner-only placement.
+
+Run these short, seed-matched diagnostics on the same source revision. Each is capped at 5,000 steps so the usual early memory-use transition can be compared before committing to the full Phase 1 horizon:
+
+```bash
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix phase1-smoke-no-moe --split-graph --split-graph-variant mamba1 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --total-steps 5000
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix phase1-smoke-moe-backbone --split-graph --split-graph-variant mamba1 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement backbone --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --total-steps 5000
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix phase1-smoke-moe-combiner --split-graph --split-graph-variant mamba1 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement combiner --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --total-steps 5000
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix phase1-smoke-moe-all --split-graph --split-graph-variant mamba1 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement all --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --total-steps 5000
+```
+
+Compare the paired memory-on/off dependency, ID/OOD metrics, curriculum progress, per-component segment diagnostics, and per-bank routing/activation summaries. The placement runs distinguish backbone-MoE effects from the CfC multi-source path; the `all` condition checks their interaction. These are diagnostic smoke runs, not evidence of a completed Phase 1 result.
+
+For a backbone-only run, `--moe-backbone-block-isolation INDEX` installs an MoE bank after only the selected zero-based parallel-backbone block. The other backbone blocks still run their Mamba/CfC computation but have no MoE; the sequential combiner must also be excluded with `--moe-placement backbone`. The flag is range-checked against `--split-graph-num-blocks`, included in run IDs and checkpoint metadata, and cannot be combined with the workspace broadcast path. For a two-block Mamba backbone, run one isolated condition per block:
+
+```bash
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix phase1-moe-backbone-block0 --split-graph --split-graph-variant mamba1 --split-graph-num-blocks 2 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement backbone --moe-backbone-block-isolation 0 --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --total-steps 5000
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix phase1-moe-backbone-block1 --split-graph --split-graph-variant mamba1 --split-graph-num-blocks 2 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement backbone --moe-backbone-block-isolation 1 --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --total-steps 5000
+```
+
+`--moe-backbone-residual-scale SCALE` multiplies the active parallel-backbone MoE delta before its residual addition (`backbone_output + SCALE * moe_delta`). Its default is `1.0`, preserving prior behavior. The scale must be finite and non-negative, requires split-graph MoE with `--moe-placement backbone` or `all`, and is included in run IDs/checkpoint metadata. With `all`, this changes only the parallel-backbone MoE residuals; combiner MoE residuals remain at 1.0. It scales the task-loss path through the backbone MoE branch while leaving routing and the load-balancing auxiliary loss active. When workspace broadcast is enabled, the scale applies to the complete final-bank delta after expert readback, including both the selected expert outputs and their broadcast contribution. Workspace slot writes and router/load-balancing calculations remain active at their existing scales. Active workspace attention and slot updates run in fp32 outside CUDA autocast, and split-graph DNC memory math uses its fp32 path in both workspace and selection-only runs. In an isolated-block run, only that block has an MoE residual to scale. For the current block-1 follow-up, use a deliberately conservative 0.1 scale:
+
+```bash
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix phase1-block1-residual-scale-0p1 --split-graph --split-graph-variant mamba1 --split-graph-num-blocks 2 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement backbone --moe-backbone-block-isolation 1 --moe-backbone-residual-scale 0.1 --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --total-steps 5000
+```
+
+To test backbone and combiner MoE together after the backbone-only scale control, omit block isolation and select `--moe-placement all`. The residual scale still applies only to the backbone; the combiner MoE keeps its default residual scale of 1.0:
+
+```bash
+python -m src.initium.core_training 0.0 --seed 0 --run-id-suffix phase1-backbone-combiner-backbone-scale-0p1 --split-graph --split-graph-variant mamba1 --split-graph-num-blocks 2 --split-graph-combiner-mode controller --split-graph-combiner-variant cfc --moe --moe-placement all --moe-backbone-residual-scale 0.1 --moe-num-experts 8 --moe-top-k 1 --moe-capacity-factor 1.5 --moe-load-balance-alpha 0.01 --total-steps 5000
+```
+
+Each `moe-bank[backbone.moe_blocks.INDEX ...]` line reports router-selected expert load, router importance, entropy, maximum probability, and capacity-drop rate separately for valid graph timesteps and all-zero padded timesteps. The valid-token mask is derived from the graph input encoding and is diagnostic-only in this experiment: routing, shared capacity, balancing loss, and outputs still include padding, so the run measures the current behavior without applying a padding fix. The per-block isolation changes which banks are active but leaves the routing policy otherwise unchanged. Compare the two blocks' valid/padded distributions and memory-dependency curves against the non-isolated backbone-only trace before deciding whether to test mask-aware routing.
+
+Add `--dnc-read-variant relational-mlp` to reproduce the original Track A reader, or `--dnc-read-variant relational-residual --dnc-read-residual-scale 1.0` to use the cosine-anchored Track A reader. Lower scales such as `0.5` are available for follow-up comparisons. Use `--dnc-write-variant kanerva-closed-form` for Track B. Add a reader and writer variant together only for the combined cell after each track is characterized. Dense temporal links remain the default. The short-run baseline reproduction gate and the Q99 paired-order evaluation are deliberate user-run experiment steps; no run results are assumed by this repository change.
+
+### Controllers
+
+The training core exposes controller choices through its command-line interface and configuration:
+
+- **LSTM:** baseline recurrent controller.
+- **Mamba:** selective state-space controller.
+- **Mamba-2 and Mamba-3:** additional state-space variants; current experiment notes report that these configurations underperformed in earlier tests and need further investigation.
+- **CfC:** closed-form continuous-time controller.
+- **Hybrid chains:** compositions of supported controller stages, such as a Mamba/CfC sequence.
+- **Split-graph:** a parallel sequence backbone plus a separate combiner for the sequential memory-addressing work. It is an experimental way to expose more of the controller computation to parallel execution while retaining the memory interface and currently performs the best out of all options.
+
+Some options require optional or hardware-sensitive dependencies, particularly `mamba-ssm` and `causal-conv1d`. A controller being selectable in code should not be read as evidence that it is equally mature or portable on every platform.
+
+### Stochastic writes and KL regularization
+
+The write-head extension models a write vector as a diagonal Gaussian. It can sample a write using the reparameterization trick and add a KL penalty to a prior distribution. The implementation supports a fixed standard-normal prior and a learned prior updated from recent write statistics on a periodic, detached snapshot schedule. The loss can include KL annealing and free bits; the learned-prior path logs its snapshot state separately.
+
+The intended locality distinction is important: the per-write KL is computed from the current write distribution and a prior snapshot. In the learned-prior variant, the prior itself is fitted from recent writes periodically, outside the per-step gradient path. That makes the overall mechanism periodically data-conditioned even though the per-write calculation uses the current write and frozen prior snapshot.
+
+Also KL regularization has specific, uninvestigated behaviour: it tends to stabilize grad_norms allowing the model to not collapse (detected once and needs further investigations) - it was mildly visible in graph-traversal experiments but its full strength occured in multimodal tasks where a model managed to collapse in lesson 14 with beta=0.0, meanwhile with beta=0.001(KL term active) the same model kept going with no issues.
+
+### Other experimental mechanisms
+
+- **Mixture of Experts (MoE):** sparse expert feed-forward capacity with routing and load-balancing diagnostics. The current project notes caution that this overhead may be unnecessary or harmful at smaller model sizes; its motivation is future scaling, not a demonstrated benefit for the current compact experiments.
+- **Dynamic memory size (Dynamic-N):** usage-triggered growth of the memory cell array within configured bounds.
+- **Link-matrix options:** ablations or sparsification of temporal-link computation.
+- **Dynamic beta:** an optional bounded controller for the KL weight, distinct from a fixed-beta sweep.
+- **Multimodal and real-data loaders:** interfaces for synthetic and selected real text, audio, video, graph, and multimodal inputs. Their practical coverage and validation are still evolving.
+
+## Repository map
+
+| Path | Role |
+| --- | --- |
+| `src/initium/core_training.py` | Dataset-agnostic training loop, CLI, evaluation, logging, checkpointing, and orchestration of model options. |
+| `src/initium/config/` | Training, regularization, controller, and architecture defaults. |
+| `src/initium/data/` | Dataset registry, task episodes, curriculum/data handling, codecs, and real-data utilities. |
+| `src/initium/mamba_controller/` | DNC controller integration, Mamba variants, hybrid controllers, and split-graph implementation. |
+| `src/initium/LNN_controller/` | CfC and hybrid continuous-time controller implementations. |
+| `src/initium/memory_manipulation/` | Stochastic write heads, prior snapshots, memory resizing, link-matrix ablations, and compatibility helpers. |
+| `src/initium/MoE/` | Sparse expert layer and its local usage notes. |
+| `src/initium/inference/` | Checkpoint loading, task registry, inference engine, metrics, logging, and cache support. |
+| `setup_wheels.py` | Environment/dependency setup entry point for specialized wheels. |
+| `pyproject.toml` | Package metadata, base and optional dependencies, and development tooling groups. |
+
+## Setup
+
+The PyPI distribution is named `initium-deeplearning`; its Python import package is `initium`. The project targets Python 3.10 or later. Core dependencies are declared in `pyproject.toml`; accelerated state-space controllers may need platform-specific PyTorch/CUDA wheels and additional packages. Install the project and optional task dependencies in an environment appropriate to your hardware. For specialized dependency setup, inspect the repository's `setup_wheels.py` and its help output before running it; GPU package compatibility depends on the installed PyTorch, CUDA, and Python versions.
+
+Typical editable installation for development:
+
+```bash
+python -m pip install -e .
+```
+
+Optional dependencies are grouped by modality, for example `text`, `video`, and `real-data` in the project metadata. Audio/video processing may also rely on system tools such as FFmpeg. Mamba-family components can require packages beyond the base install. The default training path is intended to be usable without those optional controller dependencies.
+
+The source tree is packaged as `initium`; internal modules use the `initium.*` namespace.
+
+## Training
+
+The main entry point is `src/initium/core_training.py`. From the repository root, inspect the available arguments with:
+
+```bash
+python -m initium.core_training --help
+```
+
+The script accepts a positional KL weight (`beta`); omitting it runs the configured beta sweep. It also accepts a random seed and controller/task/architecture options. A representative graph-task run with the Mamba controller and the dynamic-memory/split-graph experiments enabled is:
+
+```bash
+python -m initium.core_training 0.0 --seed 0 --controller mamba --dynamic-n-mode --split-graph
+```
+
+This is an example of the CLI shape, not a universal best configuration. Check `src/initium/config/train_config.py` and `src/initium/config/controller_config.py` for the defaults in the checkout you're running. In particular, training duration, model size, beta sweep, dataset, evaluation cadence, and memory options may have changed since older experiment records were produced.
+
+Training runs typically produce:
+
+- periodic and final checkpoints, including model, optimizer, random-generator, curriculum, and architecture/prior state needed to resume or evaluate;
+- step-level or periodic logs for task loss, KL, accuracy, OOD evaluation, and training health;
+- additional CSV diagnostics for learned-prior snapshots and optional architecture features;
+- a run identifier that encodes relevant settings to reduce accidental collisions.
+
+Keep seeds, source revision, full command, configuration, device/dependency versions, and evaluation protocol with any result you intend to compare. A single final metric is not enough to characterize these experiments: curriculum position, checkpoint step, sampled versus deterministic writes, and evaluation sample count can change the interpretation.
+
+## Evaluation and inference
+
+The checkpoint-based inference CLI lives at `src/initium/inference/run_inference.py`. Its help output is the current reference for arguments:
+
+```bash
+python -m initium.inference.run_inference --help
+```
+
+The command accepts a checkpoint, task type, optional data link, episode count, seed, device, and task-specific settings. It can inspect checkpoint capabilities, compare reset and persistent experience modes, report memory-ablation behavior, and optionally configure inference caches. A basic invocation has this form:
+
+```bash
+python -m initium.inference.run_inference path/to/checkpoint.pt --dataset-type graph
+```
+
+The loader checks checkpoint/task compatibility and refuses unsupported task types or dimensions. Graph, text, audio, and video task runners are registered; multimodal execution is also represented in the code but its input dimensions depend on the selected modality combination. Verify the checkpoint's recorded dataset configuration before comparing results across task types.
+
+## Experiments and current evidence
+
+The supplied experiment records describe a staged investigation of a KL penalty on DNC memory writes. They are summarized here as results rather than by the names of the individual working files, so this overview can remain useful if those files are renamed or reorganized. -> If anyone wishes to see the original files, you can create an issue and will be contacted afterwards.
+
+### Fixed-prior multi-seed study
+
+The first study compared a no-KL anchor with a small fixed-prior KL weight over multiple random seeds with --controller lstm. It found meaningful heterogeneity: several seeds showed non-collapsed OOD gains, some showed little benefit, and two seeds showed genuine reversals. The strongest observed seed improved OOD triple accuracy by roughly 11–12 percentage points over its own anchor. The study therefore supports a **conditional** positive result for the tested configuration, not a claim that the effect is universal or statistically settled. The measured OOD perfect-traversal fraction also did not track triple accuracy uniformly, so the metric definition matters.
+
+### Learned-prior study
+
+The follow-up replaced the fixed prior with a prior periodically estimated from recent write statistics. Four deliberately selected seeds were tested: two previous reversal cases and two stronger fixed-prior cases. Outcomes were mixed: the two former reversals improved, one strong case was nearly flat, and the strongest fixed-prior result regressed by about five to six OOD points. Across the four seeds, the mean OOD change was small and not statistically distinguishable from zero; ID accuracy was lower by about 1.1–2.7 points in all four comparisons. Diagnostics did not indicate KL collapse. A regression-to-the-mean explanation is plausible (the reported correlation was large at n=4), but the sample is too small to distinguish that from a true repair of the earlier reversal cases. The structural audit found the learned prior's snapshot update was detached and periodic, with no second online posterior; that is a design characterization, not proof of a generalization benefit.
+
+### Alternate Mamba-controller study
+
+A separate study compared beta 0 and beta 0.001 in a Mamba, dynamic-memory, split-graph configuration across five seeds. The logs showed no statistically clear OOD or ID-to-OOD offset improvement from the KL term; beta 0 had a small nominal OOD edge in four of five seeds. The beta 0.001 runs consistently left the first curriculum lesson later. The comparison is provisional: only one seed reached the final curriculum lesson, four runs were unfinished, and there was no wide-evaluation sweep for that batch. This configuration should not be described as a completed replication of the earlier study because its controller, memory size, training budget, and evaluation coverage differ.
+
+### Evidence boundaries and open work
+
+- Results above apply to the exact runs and evaluation procedures recorded; they do not establish broad performance across modalities, hardware, or model scales.
+- Phase 1's seed variation and Phase 2's four-seed selection limit aggregate claims. Report per-seed values and uncertainty alongside averages.
+- The learned prior's apparent variance-dampening/mean-reversion effect needs more seeds and controls to separate it from failure-mode-specific repair.
+- The alternate Mamba batch needs completion of unfinished seeds and late-curriculum evaluation before a gate-level conclusion.
+- Mamba-2/Mamba-3 performance, MoE's useful scaling regime, memory-link trade-offs, dynamic memory growth, and real-data/multimodal behavior remain active investigation areas.
+- Experiment status can change. Update this section when new runs are verified; retain the method, sample size, seed set, incompleteness, and uncertainty so summaries do not outrun evidence.
+
+## Research foundations
+
+The project draws on several complementary lines of work. These references motivate components and questions; their inclusion does not imply that Initium reproduces each paper's results.
+
+### Differentiable memory and neural sequence models
+
+- Graves et al., **“Hybrid computing using a neural network with dynamic external memory.”** DNC memory, differentiable addressing, and the graph-traversal motivation.
+- Hasani et al., **“Closed-form Continuous-time Neural Networks.”** Continuous-time controller design.
+- Gu and Dao, **“Mamba-3: Improved Sequence Modeling using State Space Principles.”** State-space sequence-modeling direction explored by controller variants.
+- Dao and Gu, **“Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality.”** Structured state-space/sequence computation background.
+
+### Delta-rule and state-space improvements
+
+- **“Gated Delta Networks: Improving Mamba 2 with Delta Rule.”** Gated delta-rule sequence processing.
+- **“Parallelizing Linear Transformers with the Delta Rule over Sequence Length.”** Parallelization ideas related to sequence-length computation.
+
+### Sparse expert models
+
+- Shazeer et al., **“Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.”** Sparse expert routing foundations.
+- Fedus et al., **“Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.”** Simplified sparse expert scaling.
+- **“Multi-Rate Mixture of Experts for Accelerating Liquid Neural Network Training.”** Motivation for combining expert computation with liquid-network training.
+- **“MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts.”** MoE and selective state-space combination.
+
+### Stochastic regularization and the DNC baseline
+
+- Higgins et al., **“beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.”** Gaussian KL regularization and reparameterized latent sampling adapted here to memory writes.
+- Graves et al., **“Hybrid computing using a neural network with dynamic external memory.”** Baseline memory architecture and task family used by the Q21 study.
+- Additional work on local credit assignment, priors, and continual-learning regularization informed the experiment design and interpretation. Consult the experiment notes for the specific comparisons and caveats.
+
+## Development principles
+
+Because the repository is new and research directions are still changing:
+
+- Prefer describing mechanisms and measured outcomes over calling a configuration “best” without a scoped comparison.
+- Distinguish implemented, exercised, and validated features. A registry entry is implementation coverage, not empirical validation.
+- Preserve negative and mixed results; record the seed, comparison anchor, metric, evaluation window, and incomplete runs.
+- Keep CLI help, configuration defaults, this README, and experiment records aligned as the project evolves.
+- Treat research papers as motivation and technical background, not as evidence that this implementation matches their methods or results unless a reproduction has been explicitly checked.
+
+## License
+
+Initium is distributed under the GNU General Public License v3.0. See [`LICENSE`](LICENSE).
+
+## Publishing to PyPI
+
+Package versions come from `__version__` in `src/__init__.py`. After merging a version update into `main`, create a GitHub Release whose tag matches that version (for example, `v0.1.0`). The `Publish to PyPI` workflow builds and validates the wheel and source distribution, then publishes them automatically. It refuses releases whose tag points to a commit outside `main` or whose tag does not match the package version.
+
+The PyPI distribution name is `initium-deeplearning`; the source directory and import namespace remain `initium`. The configured PyPI Trusted Publisher should target project `initium-deeplearning` with owner `samueleternity`, repository `initium`, workflow `publish-pypi.yml`, and GitHub environment `pypi`. Publishing uses short-lived OIDC credentials, so no PyPI API token is stored in GitHub.