pantheon-gpu 1.2.0__tar.gz

pantheon_gpu-1.2.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

pantheon_gpu-1.2.0/NOTICE

@@ -0,0 +1,42 @@
+Pantheon GPU
+Copyright (c) Pantheon GPU contributors
+
+This product is licensed under the Apache License, Version 2.0. See LICENSE.
+
+================================================================================
+Third-party components bundled in this repository
+================================================================================
+
+kernels/common/nvenc/nvEncodeAPI.h
+    Copyright (c) 2010-2024 NVIDIA Corporation.
+    Licensed under MIT-style terms that expressly permit use, copying,
+    modification, publication and distribution, provided the copyright notice
+    and permission notice are retained. The notice is preserved in the file.
+    Redistributable: yes.
+
+================================================================================
+Third-party components required but NOT bundled
+================================================================================
+
+NVIDIA OptiX SDK headers (optix.h, optix_stubs.h and friends)
+    Copyright (c) NVIDIA Corporation.
+    These headers state that use, reproduction, disclosure or distribution
+    without an express license agreement from NVIDIA is strictly prohibited.
+    They are therefore NOT redistributed with Pantheon.
+
+    Only the rt_virus workload needs them. The build probes for them and, when
+    they are absent, rt_virus builds a dummy kernel and reports its run as
+    skipped rather than failing the build. To build the real RT kernel, obtain
+    the OptiX SDK from NVIDIA and point the build at its include directory:
+
+        make PLATFORM=CUDA OPTIX_PATH=/path/to/optix/include
+
+AMD HIP-RT headers (hiprt/hiprt.h)
+    Copyright (c) Advanced Micro Devices, Inc.
+    Not bundled. Handled the same way: rt_virus builds its dummy kernel when
+    they are absent. See the HIP-RT vendoring target in the Makefile.
+
+================================================================================
+
+If you add a third-party file to this repository, record it above and confirm
+its license permits redistribution first.

pantheon_gpu-1.2.0/PKG-INFO

@@ -0,0 +1,270 @@
+Metadata-Version: 2.4
+Name: pantheon-gpu
+Version: 1.2.0
+Summary: GPU stress testing and diagnostics for NVIDIA CUDA and AMD ROCm
+License: Apache-2.0
+Project-URL: Homepage, https://pantheongpu.com
+Project-URL: Source, https://github.com/pantheongpu/pantheon
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: psutil
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: nvidia-ml-py
+Provides-Extra: reports
+Requires-Dist: openpyxl; extra == "reports"
+Dynamic: license-file
+
+# Pantheon: Universal GPU Stress & Diagnostics Suite
+
+Pantheon is a cross-platform (CUDA/ROCm) stress testing tool designed to isolate and hammer specific GPU subsystems. Unlike generic benchmarks (Furmark, 3DMark), Pantheon allows you to test specific silicon limits.
+
+## Requirements
+
+### Running From Source
+
+This repository is the source tree. It does not build installable packages or
+a standalone executable; run the checked-out runner directly:
+
+```bash
+python3 pantheon.py --test baseline_metrics --duration 10
+python3 pantheon.py --test tensor_virus --duration 60 --gpu 0
+```
+
+Pantheon auto-detects CUDA, ROCm/HIP, or mock mode at runtime. Do not pass
+`--platform cuda` for normal use.
+
+Kernels are compiled on first use, or explicitly:
+
+```bash
+make PLATFORM=CUDA      # NVIDIA
+make PLATFORM=HIP       # AMD
+make PLATFORM=MOCK      # CPU backend, no GPU required
+```
+
+### Runtime Dependencies
+- **Hardware Drivers:** Latest NVIDIA or AMD Linux drivers for the target GPU.
+- **Compiler:** CUDA `nvcc` for NVIDIA GPUs, or ROCm/HIP `hipcc` for AMD GPUs. Pantheon uses the local compiler to build GPU workload binaries for the current machine.
+- **System tools:** `make` and a C++ compiler such as `g++`.
+- **Linux compatibility:** Official x86-64 binaries are built in a `manylinux_2_28` environment and require GLIBC 2.28 or newer.
+
+### Compiler Requirements
+To run stress tests natively (and avoid **Mock Mode**), you must have the backend compiler for your specific GPU architecture installed:
+
+#### For NVIDIA GPUs (CUDA)
+**Linux (Ubuntu/Debian):**
+```bash
+sudo apt-get update
+sudo apt-get install nvidia-cuda-toolkit
+```
+
+#### For AMD GPUs (ROCm/HIP)
+**Linux (Ubuntu/Debian):**
+```bash
+sudo apt-get update
+sudo apt-get install hipcc
+```
+
+## Quick Start
+```bash
+python3 pantheon.py --version
+
+# Run a specific "Power Virus" test
+python3 pantheon.py --test tensor_virus --duration 60
+
+# Run a specific "VRM Cracker" test
+python3 pantheon.py --test pulse_virus --duration 60
+
+# Run the inference-path stress suite on GPU 0
+ python3 pantheon.py --test inference --duration 60 --gpu 0 --mem 50
+```
+
+## Ray tracing workload (rt_virus)
+
+NVIDIA's OptiX headers are proprietary and not redistributable, so this tree
+does not ship them and `rt_virus` builds a dummy kernel that reports
+`SKIP` by default. To run the real workload on NVIDIA hardware, download the
+[OptiX SDK](https://developer.nvidia.com/designworks/optix/download) (free
+NVIDIA developer account required) and point `OPTIX_PATH` at its `include`
+directory:
+
+```bash
+OPTIX_PATH=/opt/NVIDIA-OptiX-SDK-8.0.0/include python3 pantheon.py --test rt_virus --duration 60
+```
+
+The build cache keys on `OPTIX_PATH`, so setting or changing it triggers a
+rebuild automatically. On AMD hardware, build with `ENABLE_HIPRT=1` instead,
+which clones and builds HIP-RT itself.
+
+## AI and inference workloads
+
+The `inference`, `training` and `ai_auxiliary` suites exercise GPU execution
+and memory patterns relevant to model serving. They are diagnostic workloads,
+not model benchmarks: they load no weights, measure no model quality, and do
+not replace an end-to-end benchmark such as vLLM or TensorRT-LLM.
+
+```bash
+python3 pantheon.py --test inference --duration 60 --gpu 0 --mem 50
+```
+
+See [`docs/ai_workloads.md`](docs/ai_workloads.md) for what each workload
+stresses and what its reported rate does and does not mean.
+
+## Hardware Counter Profiling
+
+Add `--profile` to wrap each workload in the native vendor profiler:
+
+```bash
+python3 pantheon.py --test tensor_virus --duration 30 --gpu 0 --profile
+```
+
+`--profile` is an exhaustive diagnostic mode. Pantheon profiles every selected
+workload and GPU, running GPUs serially to avoid profiler contention and to
+keep the artifacts isolated. On NVIDIA it runs Nsight Compute (`ncu`) for
+hardware counters and Nsight Systems (`nsys`) for a CUDA/NVTX timeline. On AMD
+it uses `rocprofv3` to collect performance counters together with the HIP
+runtime, kernel-dispatch, and memory-operation trace.
+
+The default counter sets cover SM/CU occupancy and active blocks, instruction
+pipeline and memory-operation mix, branch and memory-access efficiency,
+scheduler stalls, L1/L2/cache behavior, DRAM traffic, and host/peer transfer
+fabric indicators where the installed profiler exposes them. Each profile also
+includes the RAS and PCIe reliability snapshots collected before and after the
+workload.
+Supported devices report ECC correctable and uncorrectable counts, memory
+retirement state, and PCIe replay, CRC, TLP/DLLP, and fatal/non-fatal counters.
+AMD profiles also collect exposed per-block RAS data and CPER firmware records.
+Linux PCIe AER counters and new AER/poison events are included when the host
+exposes them. Unsupported hardware is labeled unavailable rather than reported
+as zero. One workload/GPU bundle is written below the timestamped run directory:
+
+```text
+results/<run-id>/profiles/<workload>/gpu<id>/
+  profile_manifest.json
+  time_series.csv
+  hardware_counters.csv
+  ras.json
+  summary.json
+  trace.*
+```
+
+`summary.xlsx` links each result to its bundle and includes the flattened
+counter values. Profiling can require multiple vendor passes and is therefore
+substantially slower than a normal stress run. Pantheon refuses an explicit
+profile request when the required vendor tools are unavailable, instead of
+silently running without profiler artifacts.
+
+## Fault Maps
+
+Console fault reporting is capped at a few lines per run so a widescale
+corruption cannot flood the log. That keeps the console readable but means a
+run reports an accurate error *count* and only a handful of error *locations*.
+
+Memory workloads accept `--fault_map <file>` to record every failing address:
+
+```bash
+./build/memory_read 0 60 90 --verify --fault_map faults.csv
+```
+
+The file is self-describing CSV:
+
+```text
+# workload=memory_read gpu=0 init_pattern=2 init_pattern_name=crosstalk total_faults=3 recorded=3 capacity=4096
+index,expected,actual,xor
+1337,0xe76e5272,0xecc3ec9d,0x0badbeef
+```
+
+The `xor` column names the flipped bits, which is what distinguishes a single
+defective cell from a stuck data lane: one bit position failing across many
+unrelated addresses indicates shared structure, not a cell. Supported by
+`memory_read`, `memory_write`, `atomic_virus`, `ras_validator`,
+`memory_retention`, `march_test`, `memory_hammer`, and `galpat`
+(`memory_read_agg` and `memory_write_agg` are aliases of the first two with a
+different `--init_pattern`).
+
+## Memory Diagnostics
+
+The `diagnostics` suite looks for defective cells rather than measuring
+bandwidth: `march_test`, `memory_hammer`, `galpat`, `memory_retention` and
+`ras_validator`.
+
+```bash
+python3 pantheon.py --suite diagnostics --duration 300 --gpu 0 --mem 90
+```
+
+See [`docs/memory_diagnostics.md`](docs/memory_diagnostics.md) for what each
+one finds, how they compose as a funnel, the full `--init_pattern` table, and
+the caveats that decide whether a result means anything.
+
+## Reliability checks on every run
+
+Pantheon collects lightweight before-and-after reliability snapshots for every
+workload, even without `--profile`. Each result records a `RAS Status`, a
+human-readable `RAS Error Delta`, and the path to its JSON report under
+`results/<run-id>/reliability/<workload>/gpu<id>/ras.json`.
+
+The snapshot covers NVIDIA ECC and PCIe error counters, AMD RAS and CPER
+records where their management tools expose them, and Linux PCIe AER events.
+`CLEAN` means supported counters did not change during the workload;
+`WARNING` records new correctable or recoverable events; `ERROR` records new
+uncorrectable, fatal, or poison events; and `UNAVAILABLE` means the platform
+does not expose the relevant telemetry. Hardware counters and traces still
+require `--profile`.
+
+Profiler counter names can vary by GPU generation and driver/toolkit version.
+Override or extend the default sets without editing the source:
+
+```bash
+# Replace the NVIDIA metric list entirely
+PANTHEON_CUDA_METRICS="sm__throughput.avg.pct_of_peak_sustained_elapsed,dram__bytes_read.sum" \
+  python3 pantheon.py --test memory_read --duration 30 --profile
+
+# Add one AMD counter to Pantheon's default list
+PANTHEON_HIP_METRICS_APPEND="CUSTOM_COUNTER_NAME" \
+  python3 pantheon.py --test memory_write --duration 30 --profile
+```
+
+## Interpretation of Results
+
+The summary report (`results/<timestamp>/summary.xlsx`) contains detailed "Pro" metrics. Here is how to interpret them:
+
+* **Efficiency (MB/J):** Calculated as `Throughput / Watts`.
+    * **Healthy:** Stays relatively constant throughout the run.
+    * **Degraded:** If this drops significantly during a 1-hour burn-in, your silicon is "leaking" current (thermal runaway) or the VRMs are becoming inefficient due to heat.
+* **PCIe Link:** Verifies the physical connection speed (e.g., `Gen4 x16`).
+    * **Red Flag:** If it drops to `x8` or `Gen3` under load, check your riser cable, motherboard slot, or GPU mounting pressure.
+* **Throttle Reason:** Tells you *why* performance is limited.
+    * `[POWER]`: **Normal.** The card hit its TDP limit (expected for viruses).
+    * `[THERMAL]`: **Critical.** The core is overheating (typically >83°C Edge or >110°C Junction). Check thermal paste.
+    * `[VOLTAGE]`: The VRMs cannot supply enough stable voltage to maintain the clock.
+* **Junction (Hotspot) vs. Edge:** * On newer AMD drivers, the primary temperature reported is the **Junction**. This is the absolute hottest single point on the silicon die. 
+    * It is normal for Junction to be 15-25°C hotter than the traditional "Edge" average.
+* **Max Mem Temp:** The hottest point on your VRAM (Memory/GDDR6X). Keep this under 100°C to avoid permanent damage.
+* **Throughput (GB/s):**
+    * For `memory_read` / `memory_write`, this should be within 90% of your card's theoretical max bandwidth.
+    * Low throughput = Memory Controller instability or aggressive error correction (ECC) kicking in.
+
+## Incremental Reports
+
+For long `--test all` runs, Pantheon writes one atomic
+`database/pantheon_report_<run>_<sequence>_<workload>_gpu<id>.json` file when
+each workload finishes. These are complete workload records, not snapshots of
+an active workload, so a results publisher can safely copy them while the
+remaining queue is still running. The aggregate report is still written at the
+end of the run. Reports marked `partial`, `failed`, or `incomplete` are
+excluded by downstream importers.
+
+## Per-workload documentation
+
+Each workload has its own page under `kernels/<test>/README.md` describing what
+it stresses, how to read a failure, and what its reported metric does not mean.
+
+## Disclaimer
+
+Pantheon is an independent project. It is not generated, verified, or endorsed
+by **AMD** or **NVIDIA**, and its results are not produced by official vendor
+performance labs. Measurements reflect the driver versions, power limits and
+thermal conditions of whatever machine they were taken on, so figures from
+different environments are not directly comparable.