scope-profiler 0.2__tar.gz → 0.2.2__tar.gz

{scope_profiler-0.2 → scope_profiler-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scope-profiler
-Version: 0.2
+Version: 0.2.2
 Summary: Profile code regions in python, optionally with LIKWID markers.
 Author: Max
 Project-URL: Source, https://github.com/max-models/scope-profiler
@@ -23,6 +23,8 @@ Provides-Extra: pproc
 Requires-Dist: ipykernel; extra == "pproc"
 Requires-Dist: jupyterlab; extra == "pproc"
 Requires-Dist: matplotlib; extra == "pproc"
+Requires-Dist: maxplotlibx>=0.1.5; extra == "pproc"
+Requires-Dist: pandas; extra == "pproc"
 Provides-Extra: dev
 Requires-Dist: black[jupyter]; extra == "dev"
 Requires-Dist: isort; extra == "dev"
@@ -97,22 +99,81 @@ Execution:
 
 ```bash
 ❯ python test.py
-Region: main
-  Total Calls : 1
-  Total Time  : 0.001503709 s
-  Avg Time    : 0.001503709 s
-  Min Time    : 0.001503709 s
-  Max Time    : 0.001503709 s
-  Std Dev     : 0.0 s
-----------------------------------------
-Region: iteration
-  Total Calls : 10
-  Total Time  : 3.832e-06 s
-  Avg Time    : 3.832e-07 s
-  Min Time    : 2.08e-07 s
-  Max Time    : 8.75e-07 s
-  Std Dev     : 2.2431888016838885e-07 s
-----------------------------------------
+profiling_data.h5  (1 rank(s))
+  region     ranks  calls    total [s]      avg [s]      min [s]      max [s]      std [s]
+  ----------------------------------------------------------------------------------------
+  main           1      1   0.00150371   0.00150371   0.00150371   0.00150371            0
+  iteration      1     10    3.832e-06    3.832e-07     2.08e-07     8.75e-07  2.24319e-07
+  ----------------------------------------------------------------------------------------
+  TOTAL                11   0.00150754
+
+  Regions may nest, so the summed total can exceed the wall-clock time.
+```
+
+`finalize()` prints the same table as `scope-profiler inspect` and
+`ProfilingH5Reader.print_summary()`. Pass `verbose=False` to suppress it.
+
+## Inspecting a profiling file
+
+`scope-profiler inspect` prints what is inside an HDF5 profiling file: the
+full run metadata (host, CPU, loaded modules, Slurm job, environment) and one
+statistics line per region, with no plotting dependencies needed.
+
+```bash
+scope-profiler inspect profiling_data.h5
+```
+
+```text
+==============================================================================
+profiling_data.h5
+2 rank(s), 4 region(s), 0.18 MiB, 0.0951538 s wall clock
+==============================================================================
+
+Metadata
+  Run
+    timestamp              : 2026-07-26T18:57:49
+    user                   : mlindqvi
+    hostname               : lrdn1234
+  System
+    chip_information       : AMD EPYC 9654 96-Core Processor
+  Parallelism
+    mpi_size               : 2
+    omp_num_threads        : 8
+    total_cores            : 16
+  Slurm
+    SLURM_JOB_ID           : 9988776
+  Modules (4)
+    profile/base
+    gcc/12.3.0
+    openmpi/4.1.6--gcc--12.3.0
+    python/3.11.7
+
+Regions (4)
+  region    ranks  calls  total [s]    avg [s]     min [s]    max [s]      std [s]
+  --------------------------------------------------------------------------------
+  timestep      2      8   0.139235  0.0174044   0.0165256  0.0176325  0.000338551
+  solve         2      8  0.0991292  0.0123911   0.0115046  0.0125382  0.000335212
+  setup         2      2  0.0473326  0.0236663   0.0222938  0.0250388   0.00137254
+  assemble      2      8  0.0399496  0.0049937  0.00484729  0.0050345   5.5882e-05
+  --------------------------------------------------------------------------------
+  TOTAL               26   0.325647
+```
+
+Long values such as `PATH` are clipped unless `--full` is passed, regions can
+be filtered with `--include`/`--exclude`/`--ranks`, reordered with `--sort`,
+and either section shown alone with `--metadata-only` / `--regions-only`.
+
+The metadata can also be exported to JSON, with one entry per inspected file
+and no clipping:
+
+```bash
+scope-profiler inspect profiling_data.h5 --export-metadata metadata.json --quiet
+```
+
+```python
+from scope_profiler.inspection import write_metadata_json
+
+write_metadata_json("profiling_data.h5", "metadata.json")
 ```
 
 ## Example plots
@@ -245,6 +306,48 @@ ProfileManager.finalize()
 Both `fibonacci` and `fibonacci_ctx` will report one call per recursive
 invocation, each with correct, non-overlapping timing data.
 
+## Analysing results in Python
+
+`ProfilingH5Reader` loads a merged profiling file and behaves like an ordered
+mapping of region name to region. Every duration and timestamp it reports is in
+**seconds**:
+
+```python
+from scope_profiler import ProfilingH5Reader
+
+reader = ProfilingH5Reader("profiling_data.h5")
+reader.print_summary()
+
+# region       calls     total [s]       avg [s]       min [s]       max [s]
+# ---------------------------------------------------------------------------
+# setup            1       0.02401       0.02401       0.02401       0.02401
+# timestep         5      0.062835      0.012567     0.0087755     0.0187844
+
+solve = reader["solve"]           # an MPIRegion: the region across all ranks
+solve.num_calls                   # summed over ranks
+solve.total_duration              # seconds
+solve.average_durations()         # {rank: seconds}, for load imbalance
+solve[0].durations                # every call on rank 0, as a numpy array
+```
+
+`summary()` returns the same table as a list of dicts, and `to_dataframe()`
+returns it as a pandas DataFrame (one row per region, or per region and rank
+with `per_rank=True`):
+
+```python
+frame = reader.to_dataframe().sort_values("total_duration", ascending=False)
+per_rank = reader.to_dataframe(per_rank=True)
+```
+
+`include` / `exclude` regexes select regions in `get_regions()`, `summary()`,
+`to_dataframe()` and every `plot_*` function.
+
+The [tutorial notebooks](tutorials/) cover this in depth:
+[getting started](tutorials/01_getting_started.ipynb),
+[post-processing](tutorials/02_postprocessing.ipynb),
+[visualization](tutorials/03_visualization.ipynb) and
+[profiling modes](tutorials/04_profiling_modes.ipynb).
+
 ## Flame graphs
 
 Because each call - including recursive re-entries of the same region -
@@ -265,8 +368,7 @@ scope-profiler pproc profiling_data.h5 --show -o figures
 Or programmatically:
 
 ```python
-from scope_profiler.h5reader import ProfilingH5Reader
-from scope_profiler.plotting_scripts import plot_flame
+from scope_profiler import ProfilingH5Reader, plot_flame
 
 reader = ProfilingH5Reader("profiling_data.h5")
 plot_flame(reader, filepath="flame_plot.png")
@@ -284,3 +386,44 @@ scope-profiler pproc profiling_data.h5 --cmap viridis -o figures
 By default the flame graph covers rank 0, since it represents a single
 execution's call stack; pass `ranks=[...]` to render one flame graph per
 requested rank.
+
+## Exporting plot data
+
+Every `plot_*` function accepts a `data_filepath` argument that writes the
+exact data behind the chart to a file, so it can be re-parsed and re-plotted
+later without the original HDF5 file. `data_format` selects `"csv"` (default)
+or `"json"`:
+
+```python
+plot_gantt(reader, filepath="gantt_plot.png", data_filepath="gantt_data.csv")
+plot_gantt(
+    reader,
+    filepath="gantt_plot.png",
+    data_filepath="gantt_data.json",
+    data_format="json",
+)
+```
+
+The JSON payload additionally includes a `colors` map (region or file label
+to `#rrggbb`) matching the colors used in the matplotlib plot, so a
+JavaScript charting library like Plotly can reproduce the same look.
+
+`scope-profiler pproc --export-data` does the same for every plot in one
+run, writing `gantt_data`, `flame_data`, `durations_data`, and (for multiple
+input files) `speedup_data` alongside the PNGs. Pass `--export-data-format
+json` to get `.json` files instead of the default `.csv`:
+
+```bash
+scope-profiler pproc profiling_data.h5 -o figures --export-data
+scope-profiler pproc profiling_data.h5 -o figures --export-data --export-data-format json
+```
+
+Pass `--skip-plot-images` (requires `--export-data`) to skip rendering the
+PNGs entirely and only write the exported data plus `region_statistics.json`
+— useful when a website renders charts client-side (e.g. with Plotly)
+straight from the JSON:
+
+```bash
+scope-profiler pproc profiling_data.h5 -o figures \
+  --export-data --export-data-format json --skip-plot-images
+```

{scope_profiler-0.2 → scope_profiler-0.2.2}/README.md

@@ -54,22 +54,81 @@ Execution:
 
 ```bash
 ❯ python test.py
-Region: main
-  Total Calls : 1
-  Total Time  : 0.001503709 s
-  Avg Time    : 0.001503709 s
-  Min Time    : 0.001503709 s
-  Max Time    : 0.001503709 s
-  Std Dev     : 0.0 s
-----------------------------------------
-Region: iteration
-  Total Calls : 10
-  Total Time  : 3.832e-06 s
-  Avg Time    : 3.832e-07 s
-  Min Time    : 2.08e-07 s
-  Max Time    : 8.75e-07 s
-  Std Dev     : 2.2431888016838885e-07 s
-----------------------------------------
+profiling_data.h5  (1 rank(s))
+  region     ranks  calls    total [s]      avg [s]      min [s]      max [s]      std [s]
+  ----------------------------------------------------------------------------------------
+  main           1      1   0.00150371   0.00150371   0.00150371   0.00150371            0
+  iteration      1     10    3.832e-06    3.832e-07     2.08e-07     8.75e-07  2.24319e-07
+  ----------------------------------------------------------------------------------------
+  TOTAL                11   0.00150754
+
+  Regions may nest, so the summed total can exceed the wall-clock time.
+```
+
+`finalize()` prints the same table as `scope-profiler inspect` and
+`ProfilingH5Reader.print_summary()`. Pass `verbose=False` to suppress it.
+
+## Inspecting a profiling file
+
+`scope-profiler inspect` prints what is inside an HDF5 profiling file: the
+full run metadata (host, CPU, loaded modules, Slurm job, environment) and one
+statistics line per region, with no plotting dependencies needed.
+
+```bash
+scope-profiler inspect profiling_data.h5
+```
+
+```text
+==============================================================================
+profiling_data.h5
+2 rank(s), 4 region(s), 0.18 MiB, 0.0951538 s wall clock
+==============================================================================
+
+Metadata
+  Run
+    timestamp              : 2026-07-26T18:57:49
+    user                   : mlindqvi
+    hostname               : lrdn1234
+  System
+    chip_information       : AMD EPYC 9654 96-Core Processor
+  Parallelism
+    mpi_size               : 2
+    omp_num_threads        : 8
+    total_cores            : 16
+  Slurm
+    SLURM_JOB_ID           : 9988776
+  Modules (4)
+    profile/base
+    gcc/12.3.0
+    openmpi/4.1.6--gcc--12.3.0
+    python/3.11.7
+
+Regions (4)
+  region    ranks  calls  total [s]    avg [s]     min [s]    max [s]      std [s]
+  --------------------------------------------------------------------------------
+  timestep      2      8   0.139235  0.0174044   0.0165256  0.0176325  0.000338551
+  solve         2      8  0.0991292  0.0123911   0.0115046  0.0125382  0.000335212
+  setup         2      2  0.0473326  0.0236663   0.0222938  0.0250388   0.00137254
+  assemble      2      8  0.0399496  0.0049937  0.00484729  0.0050345   5.5882e-05
+  --------------------------------------------------------------------------------
+  TOTAL               26   0.325647
+```
+
+Long values such as `PATH` are clipped unless `--full` is passed, regions can
+be filtered with `--include`/`--exclude`/`--ranks`, reordered with `--sort`,
+and either section shown alone with `--metadata-only` / `--regions-only`.
+
+The metadata can also be exported to JSON, with one entry per inspected file
+and no clipping:
+
+```bash
+scope-profiler inspect profiling_data.h5 --export-metadata metadata.json --quiet
+```
+
+```python
+from scope_profiler.inspection import write_metadata_json
+
+write_metadata_json("profiling_data.h5", "metadata.json")
 ```
 
 ## Example plots
@@ -202,6 +261,48 @@ ProfileManager.finalize()
 Both `fibonacci` and `fibonacci_ctx` will report one call per recursive
 invocation, each with correct, non-overlapping timing data.
 
+## Analysing results in Python
+
+`ProfilingH5Reader` loads a merged profiling file and behaves like an ordered
+mapping of region name to region. Every duration and timestamp it reports is in
+**seconds**:
+
+```python
+from scope_profiler import ProfilingH5Reader
+
+reader = ProfilingH5Reader("profiling_data.h5")
+reader.print_summary()
+
+# region       calls     total [s]       avg [s]       min [s]       max [s]
+# ---------------------------------------------------------------------------
+# setup            1       0.02401       0.02401       0.02401       0.02401
+# timestep         5      0.062835      0.012567     0.0087755     0.0187844
+
+solve = reader["solve"]           # an MPIRegion: the region across all ranks
+solve.num_calls                   # summed over ranks
+solve.total_duration              # seconds
+solve.average_durations()         # {rank: seconds}, for load imbalance
+solve[0].durations                # every call on rank 0, as a numpy array
+```
+
+`summary()` returns the same table as a list of dicts, and `to_dataframe()`
+returns it as a pandas DataFrame (one row per region, or per region and rank
+with `per_rank=True`):
+
+```python
+frame = reader.to_dataframe().sort_values("total_duration", ascending=False)
+per_rank = reader.to_dataframe(per_rank=True)
+```
+
+`include` / `exclude` regexes select regions in `get_regions()`, `summary()`,
+`to_dataframe()` and every `plot_*` function.
+
+The [tutorial notebooks](tutorials/) cover this in depth:
+[getting started](tutorials/01_getting_started.ipynb),
+[post-processing](tutorials/02_postprocessing.ipynb),
+[visualization](tutorials/03_visualization.ipynb) and
+[profiling modes](tutorials/04_profiling_modes.ipynb).
+
 ## Flame graphs
 
 Because each call - including recursive re-entries of the same region -
@@ -222,8 +323,7 @@ scope-profiler pproc profiling_data.h5 --show -o figures
 Or programmatically:
 
 ```python
-from scope_profiler.h5reader import ProfilingH5Reader
-from scope_profiler.plotting_scripts import plot_flame
+from scope_profiler import ProfilingH5Reader, plot_flame
 
 reader = ProfilingH5Reader("profiling_data.h5")
 plot_flame(reader, filepath="flame_plot.png")
@@ -241,3 +341,44 @@ scope-profiler pproc profiling_data.h5 --cmap viridis -o figures
 By default the flame graph covers rank 0, since it represents a single
 execution's call stack; pass `ranks=[...]` to render one flame graph per
 requested rank.
+
+## Exporting plot data
+
+Every `plot_*` function accepts a `data_filepath` argument that writes the
+exact data behind the chart to a file, so it can be re-parsed and re-plotted
+later without the original HDF5 file. `data_format` selects `"csv"` (default)
+or `"json"`:
+
+```python
+plot_gantt(reader, filepath="gantt_plot.png", data_filepath="gantt_data.csv")
+plot_gantt(
+    reader,
+    filepath="gantt_plot.png",
+    data_filepath="gantt_data.json",
+    data_format="json",
+)
+```
+
+The JSON payload additionally includes a `colors` map (region or file label
+to `#rrggbb`) matching the colors used in the matplotlib plot, so a
+JavaScript charting library like Plotly can reproduce the same look.
+
+`scope-profiler pproc --export-data` does the same for every plot in one
+run, writing `gantt_data`, `flame_data`, `durations_data`, and (for multiple
+input files) `speedup_data` alongside the PNGs. Pass `--export-data-format
+json` to get `.json` files instead of the default `.csv`:
+
+```bash
+scope-profiler pproc profiling_data.h5 -o figures --export-data
+scope-profiler pproc profiling_data.h5 -o figures --export-data --export-data-format json
+```
+
+Pass `--skip-plot-images` (requires `--export-data`) to skip rendering the
+PNGs entirely and only write the exported data plus `region_statistics.json`
+— useful when a website renders charts client-side (e.g. with Plotly)
+straight from the JSON:
+
+```bash
+scope-profiler pproc profiling_data.h5 -o figures \
+  --export-data --export-data-format json --skip-plot-images
+```

{scope_profiler-0.2 → scope_profiler-0.2.2}/pyproject.toml

@@ -5,7 +5,7 @@ requires = [ "setuptools", "wheel" ]
 
 [project]
 name = "scope-profiler"
-version = "0.2"
+version = "0.2.2"
 description = "Profile code regions in python, optionally with LIKWID markers."
 readme = "README.md"
 keywords = [ "python" ]
@@ -35,6 +35,8 @@ optional-dependencies.pproc = [
   "ipykernel",
   "jupyterlab",
   "matplotlib",
+  "maxplotlibx >= 0.1.5",
+  "pandas",
 ]
 
 optional-dependencies.dev = [

scope_profiler-0.2.2/src/scope_profiler/__init__.py

@@ -0,0 +1,49 @@
+"""scope-profiler: lightweight region-based profiling for Python and HPC applications."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+from scope_profiler.h5reader import ProfilingH5Reader
+from scope_profiler.mpi_region import MPIRegion
+from scope_profiler.profile_manager import ProfileManager
+from scope_profiler.region import Region
+
+try:
+    __version__ = version("scope-profiler")
+except PackageNotFoundError:
+    __version__ = "unknown"
+
+# Plotting pulls in the optional maxplotlib stack, so these are resolved on
+# first access (PEP 562) rather than at import time, keeping
+# `import scope_profiler` cheap inside the applications being profiled.
+_LAZY_PLOTTING = frozenset(
+    {
+        "collect_region_statistics",
+        "plot_durations",
+        "plot_flame",
+        "plot_gantt",
+        "plot_speedup",
+        "write_region_statistics_json",
+    }
+)
+
+__all__ = [
+    "MPIRegion",
+    "ProfileManager",
+    "ProfilingH5Reader",
+    "Region",
+    *sorted(_LAZY_PLOTTING),
+]
+
+
+def __getattr__(name: str):
+    """Resolve the plotting helpers lazily; see ``_LAZY_PLOTTING``."""
+    if name in _LAZY_PLOTTING:
+        from scope_profiler import plotting_scripts
+
+        return getattr(plotting_scripts, name)
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+
+def __dir__() -> list:
+    """Include the lazily-resolved plotting helpers in ``dir()``."""
+    return sorted(set(globals()) | _LAZY_PLOTTING)

{scope_profiler-0.2 → scope_profiler-0.2.2}/src/scope_profiler/__main__.py

@@ -2,7 +2,7 @@
 
 Also runnable as ``python -m scope_profiler <command> ...``.
 
-Two subcommands:
+Three subcommands:
 
 - ``scope-profiler run script.py [args...]`` -- profiles a script's function
   calls without requiring any decorators or context managers in the script
@@ -12,6 +12,9 @@ Two subcommands:
 - ``scope-profiler pproc file.h5 [...]`` -- reads merged HDF5 profiling
   output and renders Gantt/flame/duration/speedup charts. See
   ``scope_profiler.post_processing`` for its full set of options.
+- ``scope-profiler inspect file.h5 [...]`` -- prints the run metadata and a
+  per-region statistics table for merged HDF5 profiling output, without
+  producing any plots. See ``scope_profiler.inspection``.
 """
 
 import argparse
@@ -47,8 +50,8 @@ def _parse_run_args(argv):
     parser.add_argument(
         "--buffer-limit",
         type=int,
-        default=100_000,
-        help="Max buffered calls per region before flushing to disk (default: 100000)",
+        default=1024,
+        help="Initial buffer capacity per region; grows as needed (default: 1024)",
     )
     parser.add_argument("script", help="Script to run and profile")
     parser.add_argument(
@@ -97,9 +100,17 @@ def _pproc(argv):
     return pproc_main(argv)
 
 
+def _inspect(argv):
+    """Handle ``scope-profiler inspect``: delegate to the inspection CLI."""
+    from scope_profiler.inspection import main as inspect_main
+
+    return inspect_main(argv)
+
+
 _COMMANDS = {
     "run": _run,
     "pproc": _pproc,
+    "inspect": _inspect,
 }
 
 
@@ -123,6 +134,12 @@ def main(argv=None):
         help="Post-process and plot HDF5 profiling data "
         "(see `scope-profiler pproc --help`)",
     )
+    subparsers.add_parser(
+        "inspect",
+        add_help=False,
+        help="Print metadata and region statistics of HDF5 profiling data "
+        "(see `scope-profiler inspect --help`)",
+    )
 
     if not argv:
         parser.print_help()