pyfly-lightning 0.1.0__tar.gz

pyfly_lightning-0.1.0/CHANGELOG.md

@@ -0,0 +1,189 @@
+# Changelog
+
+## 0.1.0 (2026-09-20)
+
+### Changed (packaging)
+- **The import name is `pyfly_lightning`, not `pyfly`.** PyPI already has an unrelated
+  `pyfly` (a load-testing framework) that installs a top-level `pyfly/` package. Two
+  distributions owning the same top-level directory get merged in `site-packages`: one
+  `__init__.py` wins and uninstalling either removes the other's files. Renamed to follow
+  the `pytorch-lightning` / `pytorch_lightning` convention. Verified by installing both
+  into one environment and uninstalling each in turn -- the other survives intact.
+- The console script is `pyfly-lightning` (was `pyfly`), matching the distribution name,
+  with `python -m pyfly_lightning` as an equivalent. The other package claims no console
+  script, but keeping the command distinct avoids reviving the confusion the rename fixes.
+
+First release with a usable API. Data layer, simulator, staged pipeline, controls,
+automatic planner and CLI.
+
+**Read this before citing any number.** Every claim about the substrate's value that
+this project measured came back negative: the connectome's specific wiring is
+interchangeable with its own degree-preserving rewiring, and under training it loses
+to a permutation of its own weights. What IS supported is that the substrate plus an
+automatically optimised interface works, and that the substrate need not be the fly.
+See `docs/architecture.md` sections 11 and 22-23 and the README.
+
+### Added
+- `pyfly_lightning.auto.plan`: derives substrate size, pools, latent, T and the ES budget from the task's
+  shape, with `Plan.reason` recording the measurement behind every choice.
+- `pyfly.fit(X, y)`: one call from a task to a fitted model plus the mandatory controls and a
+  verdict.
+- `SparseConnectome.from_edgelist` and `NeuronTable.from_dataframe`: bring ANY connectome, with
+  arbitrary node labels and column names. `Substrate.build` now keeps label space and code space
+  separate, canonicalises custom column names at the boundary, and raises a named error when the
+  I/O boundary is undeclared instead of failing much later.
+- `Substrate.positions_of()`, `pyfly.load()`, `FitResult`.
+- 103 tests.
+- `pyfly_lightning.sim.probe`: `echo_probe` (with a `late_echo` metric that separates wave
+  propagation from recirculation), `separability_probe`, `broadcast_coverage`.
+- `LifTorch(differentiable=True)`: a smooth gate that keeps the autograd graph, so the
+  whole encoder -> substrate -> decoder pipeline trains end to end. Plus
+  `set_trainable_weights()`, which keeps the connectome's wiring while making every edge
+  weight a parameter.
+- `pyfly_lightning.train.pipeline.StagedPipeline`: encode (ES) -> decode (backprop) -> modulate
+  (DAN readback), with an inference-safe modulation channel.
+- `examples/emotion_pipeline.py`: the six-arm comparison, gated.
+- 91 tests.
+- `pyfly_lightning.gates.structure_function_gate`: the M1 structure->function gate with its shuffled-weight
+  control, reproducing Shiu et al.'s protocol with the target discovered from the data (their MN9
+  is absent from the public FlyWire 783 release and from `sez_neurons.pickle`).
+- `Substrate.build(extra_input_ids=)`, since stratified sensory sampling yields only a couple of
+  neurons per class -- far too few to drive a named circuit.
+- `pyfly_lightning.sim.bench` memory-ceiling reporting.
+- 73 tests.
+- `pyfly_lightning.train.outer`: `StudySpace` / `Param` / `OuterLoop`, using Vizier when installed and a
+  randomised search otherwise, recording which backend actually ran. `default_space()` covers the
+  knobs PLAN.md v0.3 lists.
+- `pyfly_lightning.train.rl.PPOAgent`: actor-critic with clipped objective, entropy bonus and normalised
+  advantage. `CueDelayChoice(act_every_step=True)` gives a genuinely multi-step trajectory.
+- `pyfly_lightning.prune_evo`: NSGA-II over group-level genomes with crowding distance and a Pareto front,
+  plus `group_labels_by_depth` / `_by_scc` / `_by_component`. Returns `front_collapsed` when the
+  front is degenerate instead of presenting it as a result.
+- 68 tests.
+- `pyfly_lightning.envs.CueDelayChoice`: cue, delay, one-shot choice. Balanced cues, ±1 reward.
+- `pyfly_lightning.train.rl`: `ReplayBuffer`, `QHead`, `DQNAgent` (epsilon schedule, target network,
+  optional `RewardCalibrator` in the reward path) and `random_policy_baseline`. `feature_fn`
+  lets the pooled-feature control reuse the identical training loop.
+- `examples/rl_dqn.py`, with random / rewired / pooled controls.
+- 59 tests, including a regression test for the random-baseline seeding bug and a positive
+  control that DQN learns when the features carry the signal.
+- `pyfly_lightning.reward`: `ObjectiveComposer` (frozen: a trainer may rescale the signal but never
+  reweight the task), `RewardCalibrator` (five methods, output bounded to [-1, +1]),
+  `CreditChannel` (multicast with latency and separate negative-valence targets),
+  `PlasticityPolicy` variants with `NoPlasticity` as the default, `PlasticPool` for
+  three-factor updates restricted to a pool's internal edges, and `run_manifest()`.
+- `examples/reward_bus.py`, gated on three controls.
+- 53 tests.
+- `pyfly_lightning.freeze`: `FreezeState` for the four distinct freeze meanings (plasticity, adapter,
+  topology, state) plus `FrozenError`. ES refuses to write a frozen adapter; masking refuses
+  while `topology` is frozen.
+- `pyfly_lightning.prune`: `MaskPruner` with a tolerance contract and rollback, multi-signal importance
+  (activity over every node, graph degree, optional causal ablation by group), protected I/O, and
+  a `PruneReport`. `FlyModule.node_activity()` records per-node spiking without the readout.
+- `examples/freeze_prune.py`: freeze -> prune -> verify, gated.
+- 37 tests.
+- `pyfly_lightning.train.es`: antithetic OpenAI-ES with rank normalisation, optimising the encoder and
+  router (the parameters upstream of the gradient firewall, which no gradient can reach).
+  Fitness is closed-form: extract spike counts, solve a ridge readout analytically, score.
+- `FlyModule.features(..., stacked=)`: feedforward and recurrent (sequence) modes, with tail steps
+  so the last frame can actually reach the outputs, and a time-resolved readout so ordering
+  survives. `forward()` is unchanged and uses `stacked_readout`.
+- `Substrate.build(include_pools=)` and `Substrate.recurrence_report()`.
+- `examples/delayed_match.py`: the decisive experiment, with all four controls.
+- 32 tests.
+- `pyfly_lightning.sim`: LIF dynamics with a numpy reference backend and a bit-identical torch backend;
+  sign-preserving L1 row normalisation (the signed-sum version silently deleted inhibition).
+- `pyfly_lightning.model.substrate`: connected substrate slices grown from sampled inputs that are
+  guaranteed to contain reachable output neurons, plus drive calibration to a target input
+  firing rate.
+- `pyfly_lightning.model.nulls`: degree-preserving rewiring (unique disjoint index sampling -- the
+  with-replacement version corrupted degree preservation), weight shuffling.
+- `pyfly_lightning.train`: `FlyModule`, a real `lightning.LightningModule` when lightning is installed
+  (plain `nn.Module` fallback otherwise), `PooledFeatureMLP`, and the four-control report
+  contract with an `UNSUPPORTED` verdict by default.
+- `examples/mnist.py`: the MNIST smoke test, with all four controls and a saved report.
+- 27 tests; the suite runs green with data and skips cleanly without it.
+- `pyfly_lightning.data`: dataset registry, sha256-pinned downloader, cache, `NeuronTable`,
+  `SparseConnectome`, and an auditable `SignReport` for the transmitter->sign assumption.
+- `pyfly_lightning.model.routing`: structured `SensorySurface` + `GroupRouter`, parameterised by
+  connectome structure (optic-lobe hex columns, olfactory glomeruli, labelled lines)
+  rather than a dense matrix. `learnable=False` gives the reservoir baseline.
+- `tools/census_wm_ports.py`: reproduces the working-memory port census.
+- `docs/census-wm-ports.md`: why the central complex is the working-memory port.
+- `docs/architecture.md`: the three-network design, training stages, gradient options.
+
+### Verified by cross-validation (section 25)
+- Against `flybrain` (github.com/alextitonis/fly.ai), an independent implementation of the
+  same connectome with sha256-pinned prebuilt matrices:
+  graph reproduced exactly (166,700 nodes / 25,582,938 edges, identical sparsity,
+  max |difference| 5.96e-08 = float32 eps); dynamics agree to 5.9e-06 of 507,598 spikes
+  over 200 steps with per-neuron rate correlation r = 1.000000.
+- `prepare_weights(normalize="input"|"output")`: the two projects normalize on opposite
+  axes. Both are L1 and sign-preserving; they are different dynamical systems, and our
+  recorded numbers are all under `"output"`.
+- `tools/crossvalidate_flybrain.py` keeps the check reproducible.
+
+### Measured, low data (section 24)
+- Iris, per-fold ES, 3 seeds x 5 folds: fly 0.7644 vs **rewired 0.7711**, crossing over
+  between seeds. CPU vs raw 4 features 0.8289.
+- A first run showed fly +5.3pp over rewired; that was a leak (ES on all 150 samples
+  before CV) plus a single seed, and the corrected run removes it. Fifth independent
+  negative, and the cleanest.
+
+### Measured, staged pipeline (section 22)
+- C1 replicated over seeds 0/1/2: +15.3, +14.5, +7.6 points; mean +12.5, positive in every
+  seed. The direct-input baseline itself moves 0.4507-0.4927, so the honest claim is "7.6 to
+  15.3 points", not one number.
+- Method 0.6087 vs direct input + PPL1 0.4560 -> **C1 holds, +15.3pp**.
+- Dopamine channel 0.6087 vs 0.4927 without -> **C2 holds, +11.6pp** (+25.2pp unfrozen).
+- Topology: fly 0.6087 vs rewired 0.6607 -> **C3 fails**; with trainable weights,
+  shuffled weights do best (0.7360), so the real weights are a worse prior than their own
+  permutation.
+- The connectome's contribution that IS positive: CX converts a slice from a wire that
+  dies in 2 steps into one that recirculates for 60+ (`late_echo` 0.0 -> 30.0).
+
+### Measured, M1 gate
+- real connectivity 1.00 vs shuffled weights 0.44 (delta +0.56) -> **PASS**. The paper reports
+  100% vs 1/100; the real arm matches, the shuffled arm is weaker for four documented reasons.
+- Ablation: with raw (un-normalised) weights both arms are 1.00 and the control is vacuous, because
+  the network saturates. Row normalisation is what makes the gate discriminating, not a
+  numerical convenience.
+
+### Measured, M6 throughput
+- Refutes risk R2: 16,384 nodes / 528,013 edges uses 50.1 MB of a 6 GB card (99.2% headroom).
+  Latency is flat at ~0.15 ms/step, so throughput (~6,750 steps/s) is the constraint, not memory.
+
+### Measured, evolutionary pruning
+- NSGA-II verified on a known synthetic front and on a synthetic objective that forces a real
+  trade-off.
+- On the real substrate the Pareto front COLLAPSES: every non-dominated point shares one behaviour
+  score, because the grouping has one giant component (99% of nodes) and the model has almost no
+  behaviour to preserve (0.2113 against 10 classes, chance 0.10). Reported as
+  `front_collapsed: true`, not presented as a result.
+
+### Measured, RL
+- DQN on cue-delay-choice: random 0.4725, fly 0.4850, rewired 0.4850 (identical), pooled 1.0000.
+  Fourth confirmation of the section 11 finding. The pooled control proves the task is solvable
+  and the loop is sound, so the substrate itself is not carrying the cue across the delay.
+
+### Measured, reward layer
+- Bounded calibration across rewards spanning +-1e5 for all five methods.
+- `NoPlasticity` is a verified no-op (`total|dW| = 0.0`); three-factor plasticity changes
+  weights (0.1629); an 8-step-delayed signal acts 256x more weakly than an aligned one
+  (0.000636 vs 0.1629) -- the empirical reason this is stimulation, not a loss.
+
+### Measured, pruning
+- Freeze then mask-prune on MNIST: edges 166,600 -> 53,974 (-67.6%) with the held-out score
+  unchanged at 0.145, 1,429 of 4,096 nodes masked out, protected positions intact
+  (`examples/reports/freeze-prune.json`).
+
+### Measured, negative
+- Delayed match-to-sample: fly 0.556 vs **rewired 0.591**, pooled MLP 1.000. The rewiring wins.
+  Third independent confirmation that the topology prior is not contributing
+  (`docs/architecture.md` sections 11 and 13).
+
+### Verified against MaleCNS v1.0
+- 165,122 `Traced` neurons; 164,587 node graph; 25,563,197 significant-only directed edges.
+- Declared I/O boundary: 15,896 inputs, 2,295 outputs.
+- Input surface: 39,616 neurons in 1,269 structural groups; router 41,877 params at
+  `latent_dim=32, rank=0` vs 1,267,712 dense (30.3x).

pyfly_lightning-0.1.0/CITATION.cff

@@ -0,0 +1,35 @@
+cff-version: 1.2.0
+message: "If you use this software, cite it and the underlying datasets."
+title: pyfly-lightning
+authors:
+  - name: NewJerseyStyle
+abstract: >-
+  A framework in which the substrate is a frozen Drosophila connectome and the
+  only trained parts are encode/decode adapters around declared ports.
+type: software
+license: MIT
+version: 0.1.0
+date-released: "2026-09-17"
+preferred-citation:
+  type: software
+  title: pyfly-lightning
+  year: 2026
+references:
+  - type: article
+    title: "Neuronal wiring diagram of an adult brain"
+    year: 2024
+    journal: Nature
+    authors:
+      - family-names: Dorkenwald
+  - type: article
+    title: "Sexual dimorphism in the complete connectome of the Drosophila male central nervous system"
+    year: 2026
+    journal: Cell
+    authors:
+      - family-names: Berg
+  - type: article
+    title: "A Drosophila computational brain model reveals sensorimotor processing"
+    year: 2024
+    journal: Nature
+    authors:
+      - family-names: Shiu

pyfly_lightning-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 pyfly-lightning contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pyfly_lightning-0.1.0/MANIFEST.in

@@ -0,0 +1,11 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+include CITATION.cff
+include PLAN.md
+recursive-include docs *.md
+recursive-include examples *.py *.json
+recursive-include tests *.py
+recursive-include tools *.py
+recursive-exclude * __pycache__
+recursive-exclude * *.py[cod]