pycauset 0.6.2.dev9__tar.gz → 0.6.2.dev11__tar.gz

{pycauset-0.6.2.dev9 → pycauset-0.6.2.dev11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: pycauset
-Version: 0.6.2.dev9
+Version: 0.6.2.dev11
 Summary: High-performance Causal Set matrix operations using memory-mapped files
 Author: Bror Hjemgaard
 License: MIT

pycauset-0.6.2.dev11/documentation/dev/R2_PERF_FINDINGS.md

@@ -0,0 +1,49 @@
+# R2_PERF — parity findings (R2.2)
+
+Measured with `python benchmarks/r2_parity.py` (dense float64, n=1024 matrices,
+best-of-3, PyCauset result forced through `np.asarray`; `invert`/`determinant`
+clear their derived caches before timing so both sides recompute). The ratio is
+`numpy_time / pycauset_time`; the R2_PERF bar is **≥ 0.90×**.
+
+**State: 8/8 ops at parity.** Stable across repeated runs:
+
+| op | ratio | verdict |
+| :-- | --: | :-- |
+| add | ~1.1–1.25× | PASS |
+| solve | ~0.97–0.99× | PASS |
+| invert | ~1.1–1.2× | PASS |
+| dot | ~1.4–4× | PASS |
+| multiply | ~1.1–1.2× | PASS |
+| determinant | ~1.2–1.3× | PASS |
+| eigh | ~0.98–0.99× | PASS |
+| matmul | ~0.94–0.96× | PASS |
+
+## What landed in R2.2 (parity work)
+
+1. **`determinant` → LAPACK** — `LAPACKE_dgetrf`/`zgetrf` instead of Eigen's
+   single-threaded `PartialPivLU`.
+2. **Fair `invert`/`determinant` benchmarking** — benchmark clears derived caches.
+3. **`add` flakiness root-caused** — stale-binary namespace merge (`native.py` now
+   loads from `__file__`).
+4. **AVX2 f64 sub/mul/div kernels + full-span hardening**.
+5. **OpenBLAS rebuilt** — 0.3.28 DYNAMIC_ARCH+threaded (Haswell) replacing the
+   generic single-core 0.3.26; MinGW runtime DLLs bundled (`build_openblas.ps1`).
+6. **`:memory:` backing → `VirtualAlloc`** — the pagefile-backed section committed a
+   page from the paging file on every first write (the add/multiply win).
+7. **CPU `dgemm` no longer pins operands** — `VirtualLock` is GPU-only overhead.
+8. **GEMM bumps to 20 threads** — dgemm scales with threads, LAPACK doesn't.
+9. **`mark_temporary_if_auto` skips `Path.resolve()` for `:memory:`** — the
+   `nt._getfinalpathname` syscall (~30–60 µs) ran on every matrix construction and
+   was the final matmul residual (0.87× → 0.94×).
+
+## The decisive fixes (biggest wins)
+
+- **`:memory:` `VirtualAlloc`** (add 0.88→1.1×, multiply 0.72→~1.1×).
+- **`Path.resolve()` short-circuit** (construction 0.38 ms → 0.007 ms; matmul
+  0.87→0.94×, and every other op improved too — the temp-file tracking was the
+  single biggest per-call overhead across the whole op surface).
+- **Threaded DYNAMIC_ARCH OpenBLAS** (matmul 0.56→0.94× overall).
+
+The raw `cblas_dgemm` is faster than NumPy (0.0044 s vs 0.0060 s at 20 threads);
+with the wrapper overhead removed, `pc.matmul` now lands at ~0.94×.
+

{pycauset-0.6.2.dev9 → pycauset-0.6.2.dev11}/documentation/project/plans/R2_ROADMAP.md

@@ -507,7 +507,7 @@ DoD: zero undocumented public API; zero untested public behavior; every node's i
 
 ### R2_PERF — NumPy Parity (≥ 0.90×)
 
-Status: - [ ] (in progress: `benchmarks/r2_parity.py` gate landed; current baseline (n=1024) — **7/8 at parity** (`add` ~1.1×, `solve` ~0.96×, `invert` ~1.1×, `dot` ~2×, `multiply` ~0.93×, `determinant` ~1.24×, `eigh` ~0.98×). `matmul` hovers just under the bar (~0.87×): the raw `cblas_dgemm` is already faster than NumPy (0.0044s vs 0.0060s at 20 threads), the residual is per-call wrapper cost (fresh-result VirtualAlloc + `create_matrix`/MemoryGovernor + Python IO-planning). Landed in R2.2: (1) LAPACK determinant, (2) fair invert/determinant cache-clearing in the benchmark, (3) add-flakiness root-cause (stale-binary namespace merge — `native.py` now loads from `__file__`), (4) AVX2 f64 sub/mul/div + full-span hardening, (5) OpenBLAS 0.3.28 rebuilt DYNAMIC_ARCH+threaded, (6) `:memory:` backing switched to `VirtualAlloc` (the add/multiply win), (7) CPU dgemm no longer pins operands, (8) GEMM bumps to 20 threads. See `documentation/dev/R2_PERF_FINDINGS.md`)
+Status: - [x] (`benchmarks/r2_parity.py` gate landed; **8/8 ops at parity** (n=1024) — `matmul` ~0.94×, `add` ~1.1×, `multiply` ~1.1×, `solve` ~0.98×, `invert` ~1.1×, `dot` ~2×, `determinant` ~1.2×, `eigh` ~0.99×, stable across repeated runs. Landed in R2.2: LAPACK determinant, fair invert/determinant cache-clearing, add-flakiness root-cause (stale-binary namespace merge), AVX2 f64 sub/mul/div + full-span hardening, OpenBLAS 0.3.28 rebuilt DYNAMIC_ARCH+threaded, `:memory:` backing switched to `VirtualAlloc`, CPU dgemm no longer pins operands, GEMM bumps to 20 threads, and `mark_temporary_if_auto` skips `Path.resolve()` for `:memory:` (the final matmul residual). See `documentation/dev/R2_PERF_FINDINGS.md`)
 
 Goal: every op is at least 0.90× NumPy throughput in the regimes PyCauset claims, or is an
 explicitly-documented out-of-scope case / performance bug. Per-op status lives in

{pycauset-0.6.2.dev9 → pycauset-0.6.2.dev11}/python/pycauset/_internal/runtime.py

@@ -19,6 +19,7 @@ class Runtime:
         self._set_temporary_file = set_temporary_file
         self._env_var = env_var
         self._storage_root_cache: Path | None = None
+        self._resolved_root_cache: Path | None = None
         self._storage_roots_seen: set[Path] = set()
         self._live_matrices: weakref.WeakSet = weakref.WeakSet()
 
@@ -114,8 +115,16 @@ class Runtime:
             return
 
         try:
-            path = Path(matrix.get_backing_file()).resolve()
-            root = self.storage_root().resolve()
+            backing = matrix.get_backing_file()
+            # Anonymous RAM (:memory:) matrices are never auto-created temp files.
+            # Skip the Path.resolve() calls below, whose nt._getfinalpathname
+            # syscall (~30-60us each) would otherwise dominate small-op
+            # construction (the matmul parity residual was traced here).
+            if not backing or backing == ":memory:":
+                return
+
+            path = Path(backing).resolve()
+            root = self._resolved_storage_root()
 
             # Path.is_relative_to is 3.9+; we support 3.8+ historically? repo is 3.12 now.
             if path.is_relative_to(root):
@@ -125,3 +134,8 @@ class Runtime:
                     self._set_temporary_file(path, True)
         except (ValueError, OSError, AttributeError):
             pass
+
+    def _resolved_storage_root(self) -> Path:
+        if self._resolved_root_cache is None:
+            self._resolved_root_cache = self.storage_root().resolve()
+        return self._resolved_root_cache

{pycauset-0.6.2.dev9 → pycauset-0.6.2.dev11}/python/pycauset/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.6.2.dev9'
-__version_tuple__ = version_tuple = (0, 6, 2, 'dev9')
+__version__ = version = '0.6.2.dev11'
+__version_tuple__ = version_tuple = (0, 6, 2, 'dev11')
 
-__commit_id__ = commit_id = 'g6e4c13b57'
+__commit_id__ = commit_id = 'ga49f4b5ee'

pycauset-0.6.2.dev9/documentation/dev/R2_PERF_FINDINGS.md

@@ -1,71 +0,0 @@
-# R2_PERF — parity findings (R2.2)
-
-Measured with `python benchmarks/r2_parity.py` (dense float64, n=1024 matrices,
-best-of-3, PyCauset result forced through `np.asarray`; `invert`/`determinant`
-clear their derived caches before timing so both sides recompute). The ratio is
-`numpy_time / pycauset_time`; the R2_PERF bar is **≥ 0.90×**.
-
-Current state (threaded DYNAMIC_ARCH OpenBLAS, `OPENBLAS_NUM_THREADS=8` default,
-GEMM bumps to 20 threads internally):
-
-| op | ratio | verdict |
-| :-- | --: | :-- |
-| add | ~1.1–1.2× | PASS |
-| solve | ~0.96× | PASS |
-| invert | ~1.1× | PASS |
-| dot | ~2× | PASS |
-| multiply | ~0.93× | PASS (marginal) |
-| determinant | ~1.24× | PASS |
-| eigh | ~0.98× | PASS |
-| matmul | ~0.87× | FAIL (marginal) |
-
-7/8 ops are at or above parity. `matmul` hovers just under 0.90× (~0.0071 s vs
-NumPy ~0.0063 s) and `multiply` just above it (~0.93×); both flap across the 0.90
-line run-to-run (CPU frequency/thermal noise).
-
-## What landed in R2.2
-
-1. **`determinant` → LAPACK** — `CpuSolver::determinant` uses `LAPACKE_dgetrf`/
-   `zgetrf` (NumPy's own backend) instead of Eigen's single-threaded `PartialPivLU`.
-2. **Fair `invert`/`determinant` benchmarking** — the benchmark clears the derived
-   caches (`_invalidate_cached_derived`) before timing.
-3. **`add` flakiness root-caused** — a stale installed wheel merged into a namespace
-   package and `import_native_extension()` used `next(iter(pkg.__path__))` (filesystem-
-   dependent order on Windows), so ~50% of processes loaded a stale `_pycauset.pyd`.
-   Fixed by loading from the package's `__file__` directory (`native.py`).
-4. **AVX2 f64 sub/mul/div kernels + full-span hardening** — elementwise `subtract`/
-   `multiply`/`divide` now route through AVX2 with strict full-span guards.
-5. **`matmul` OpenBLAS rebuilt** — the CMake fallback downloaded a *generic single-core*
-   0.3.26 binary; rebuilt 0.3.28 with `DYNAMIC_ARCH=1 USE_THREAD=1` (Haswell kernels),
-   relinked, and bundled the MinGW runtime DLLs. `build_openblas.ps1` documents it.
-6. **`OPENBLAS_NUM_THREADS` default 8** — the threaded OpenBLAS uses one pool for BLAS
-   and LAPACK; 20 threads regresses small LAPACK (invert/determinant) via SMP sync
-   overhead, so `pycauset` pins 8 at import and the matmul GEMM temporarily bumps to 20.
-
-## The two memory/perf root fixes (biggest wins)
-
-- **`:memory:` backing switched to `VirtualAlloc`** (`MemoryMapper`) instead of a
-  pagefile-backed section (`CreateFileMappingA(INVALID_HANDLE_VALUE)`). The pagefile
-  path commits a page from the paging file on every first write, costing several ms
-  per 8MB result and dominating small elementwise ops. This single change took
-  `add` 0.88× → ~1.1× and `multiply` 0.72× → ~0.93×.
-- **CPU dgemm no longer pins operands** (`CpuSolver::attempt_direct_path`) — `VirtualLock`
-  is a GPU-DMA optimization and pure overhead for CPU `dgemm`.
-
-## Remaining gap: `matmul` (~0.87×)
-
-The raw `cblas_dgemm` is already **faster than NumPy** (0.0044 s vs NumPy's 0.0060 s
-at the same 20 threads). The residual is per-call wrapper cost on top of the kernel:
-
-| component | time |
-| :-- | --: |
-| raw `cblas_dgemm` (warm buffer, 20 threads) | ~0.0044 s |
-| fresh 8MB result (VirtualAlloc + first-write faults) | ~1.6 ms |
-| C++ `create_matrix` + MemoryGovernor + dispatch | ~0.4 ms |
-| Python `_ops.matmul` (IO planning, streaming checks, inline imports) | ~0.7 ms |
-| **total** | **~0.0071 s** |
-
-NumPy's `@` skips most of the Python/IO-planning layer. Fully closing the 0.03× gap
-means trimming the Python `_ops.matmul` IO-planning/inline-import path (or routing the
-native `matmul` through the eager operator), which trades away the streaming/observability
-layer for the hot dense case. Tracked against `R2_PERF` in `R2_ROADMAP.md`.