PyPI - mindstudio-probe - Versions diffs - 1.1.1__tar.gz → 1.2.2__tar.gz - Mend

mindstudio-probe 1.1.1tar.gz → 1.2.2tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (430) hide show

{mindstudio_probe-1.1.1 → mindstudio_probe-1.2.2}/MANIFEST.in RENAMED Viewed

@@ -2,4 +2,5 @@ include README.md
 include LICENSE
 recursive-include msprobe *
 recursive-exclude msprobe/test *
+recursive-exclude msprobe/ccsrc *

{mindstudio_probe-1.1.1 → mindstudio_probe-1.2.2}/PKG-INFO RENAMED Viewed

@@ -1,7 +1,7 @@
 Metadata-Version: 2.1
 Name: mindstudio-probe
-Version: 1.1.1
-Summary: Pytorch Ascend Probe Utils
+Version: 1.2.2
+Summary: Ascend Probe Utils
 Home-page: https://gitee.com/ascend/mstt/tree/master/debug/accuracy_tools/msprobe
 Author: Ascend Team
 Author-email: pmail_mindstudio@huawei.com
@@ -11,6 +11,7 @@ Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Education
 Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: C++
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Scientific/Engineering :: Mathematics
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence

{mindstudio_probe-1.1.1 → mindstudio_probe-1.2.2}/mindstudio_probe.egg-info/PKG-INFO RENAMED Viewed

@@ -1,7 +1,7 @@
 Metadata-Version: 2.1
 Name: mindstudio-probe
-Version: 1.1.1
-Summary: Pytorch Ascend Probe Utils
+Version: 1.2.2
+Summary: Ascend Probe Utils
 Home-page: https://gitee.com/ascend/mstt/tree/master/debug/accuracy_tools/msprobe
 Author: Ascend Team
 Author-email: pmail_mindstudio@huawei.com
@@ -11,6 +11,7 @@ Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Education
 Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: C++
 Classifier: Topic :: Scientific/Engineering
 Classifier: Topic :: Scientific/Engineering :: Mathematics
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence

{mindstudio_probe-1.1.1 → mindstudio_probe-1.2.2}/mindstudio_probe.egg-info/SOURCES.txt RENAMED Viewed

@@ -8,6 +8,7 @@ mindstudio_probe.egg-info/entry_points.txt
 mindstudio_probe.egg-info/not-zip-safe
 mindstudio_probe.egg-info/requires.txt
 mindstudio_probe.egg-info/top_level.txt
+msprobe/CMakeLists.txt
 msprobe/README.md
 msprobe/__init__.py
 msprobe/config.json
@@ -35,6 +36,9 @@ msprobe/core/compare/layer_mapping/__init__.py
 msprobe/core/compare/layer_mapping/data_scope_parser.py
 msprobe/core/compare/layer_mapping/layer_mapping.py
 msprobe/core/compare/layer_mapping/postprocess_pass.py
+msprobe/core/compare/merge_result/merge_result.py
+msprobe/core/compare/merge_result/merge_result_cli.py
+msprobe/core/compare/merge_result/utils.py
 msprobe/core/data_dump/data_collector.py
 msprobe/core/data_dump/json_writer.py
 msprobe/core/data_dump/scope.py
@@ -75,10 +79,18 @@ msprobe/docs/19.monitor.md
 msprobe/docs/20.monitor_performance_baseline.md
 msprobe/docs/21.visualization_PyTorch.md
 msprobe/docs/22.visualization_MindSpore.md
-msprobe/docs/23.tool_function_introduction.md
+msprobe/docs/23.generate_operator_PyTorch.md
+msprobe/docs/24.code_mapping_Mindspore.md
+msprobe/docs/25.tool_function_introduction.md
+msprobe/docs/26.data_dump_PyTorch_baseline.md
+msprobe/docs/27.dump_json_instruction.md
+msprobe/docs/28.debugger_save_instruction.md
+msprobe/docs/28.kernel_dump_MindSpore.md
 msprobe/docs/FAQ.md
 msprobe/docs/S02.report_free_benchmarking_validation_performance_baseline.md
-msprobe/docs/data_dump_Mindspore/dynamic_graph_quick_start_example.md
+msprobe/docs/accuracy_checker_MindSpore/accuracy_checker_MindSpore_baseline.md
+msprobe/docs/data_dump_MindSpore/data_dump_MindSpore_baseline.md
+msprobe/docs/data_dump_MindSpore/dynamic_graph_quick_start_example.md
 msprobe/docs/img/BLOOM-7B_1.png
 msprobe/docs/img/BLOOM-7B_2.png
 msprobe/docs/img/BLOOM-7B_3.png
@@ -109,11 +121,35 @@ msprobe/docs/img/grad_probe_image-2.png
 msprobe/docs/img/grad_probe_image-3.png
 msprobe/docs/img/grad_probe_image-4.png
 msprobe/docs/img/grad_probe_image.png
+msprobe/docs/img/merge_result.png
 msprobe/docs/img/module_compare.png
 msprobe/docs/img/ms_dump.png
 msprobe/docs/img/ms_layer.png
 msprobe/docs/img/pt_dump.png
 msprobe/docs/img/monitor/cpu_info.png
+msprobe/docs/img/monitor/step_count_per_record.png
+msprobe/docs/img/visualization/fuzzy_match_ms.png
+msprobe/docs/img/visualization/fuzzy_match_pt.png
+msprobe/docs/img/visualization/tensorboard_1.png
+msprobe/docs/img/visualization/tensorboard_2.png
+msprobe/docs/img/visualization/vis_browser_1.png
+msprobe/docs/img/visualization/vis_browser_2.png
+msprobe/docs/img/visualization/vis_precision_info.png
+msprobe/docs/img/visualization/vis_search_info.png
+msprobe/docs/img/visualization/vis_show_info.png
+msprobe/docs/img/visualization/vis_showcase.png
+msprobe/docs/img/visualization/vis_unmatch_info.png
+msprobe/docs/visualization/GPTModel.png
+msprobe/docs/visualization/ParallelMLP.png
+msprobe/docs/visualization/layer_mapping_example.md
+msprobe/docs/visualization/mapping.png
+msprobe/docs/visualization/mapping1.png
+msprobe/docs/visualization/module_name.png
+msprobe/docs/visualization/module_name1.png
+msprobe/docs/visualization/no_mapping.png
+msprobe/docs/visualization/no_mapping1.png
+msprobe/docs/visualization/no_mapping_analyze.png
+msprobe/docs/visualization/top_layer.png
 msprobe/mindspore/__init__.py
 msprobe/mindspore/cell_processor.py
 msprobe/mindspore/ms_config.py
@@ -132,8 +168,16 @@ msprobe/mindspore/api_accuracy_checker/data_manager.py
 msprobe/mindspore/api_accuracy_checker/main.py
 msprobe/mindspore/api_accuracy_checker/multi_api_accuracy_checker.py
 msprobe/mindspore/api_accuracy_checker/multi_data_manager.py
+msprobe/mindspore/api_accuracy_checker/torch_mindtorch_importer.py
 msprobe/mindspore/api_accuracy_checker/type_mapping.py
 msprobe/mindspore/api_accuracy_checker/utils.py
+msprobe/mindspore/code_mapping/__init__.py
+msprobe/mindspore/code_mapping/bind.py
+msprobe/mindspore/code_mapping/cmd_parser.py
+msprobe/mindspore/code_mapping/graph.py
+msprobe/mindspore/code_mapping/graph_parser.py
+msprobe/mindspore/code_mapping/main.py
+msprobe/mindspore/code_mapping/processor.py
 msprobe/mindspore/common/const.py
 msprobe/mindspore/common/log.py
 msprobe/mindspore/common/utils.py
@@ -154,6 +198,7 @@ msprobe/mindspore/dump/hook_cell/hook_cell.py
 msprobe/mindspore/dump/hook_cell/primitive_hooks.py
 msprobe/mindspore/dump/hook_cell/support_wrap_ops.yaml
 msprobe/mindspore/dump/hook_cell/wrap_api.py
+msprobe/mindspore/dump/kernel_dump/kernel_config.py
 msprobe/mindspore/dym_loader/hook_dynamic_loader.cc
 msprobe/mindspore/dym_loader/hook_dynamic_loader.h
 msprobe/mindspore/free_benchmark/__init__.py
@@ -183,12 +228,22 @@ msprobe/mindspore/grad_probe/grad_monitor.py
 msprobe/mindspore/grad_probe/grad_stat_csv.py
 msprobe/mindspore/grad_probe/hook.py
 msprobe/mindspore/grad_probe/utils.py
+msprobe/mindspore/mindtorch/__init__.py
+msprobe/mindspore/mindtorch/mindtorch_adaptor.py
+msprobe/mindspore/monitor/anomaly_detect.py
+msprobe/mindspore/monitor/features.py
+msprobe/mindspore/monitor/module_hook.py
+msprobe/mindspore/monitor/module_spec_verifier.py
+msprobe/mindspore/monitor/utils.py
+msprobe/mindspore/monitor/distributed/__init__.py
+msprobe/mindspore/monitor/distributed/distributed_ops.yaml
+msprobe/mindspore/monitor/distributed/stack_blacklist.yaml
+msprobe/mindspore/monitor/distributed/wrap_distributed.py
 msprobe/mindspore/overflow_check/__init__.py
 msprobe/mindspore/overflow_check/kernel_graph_overflow_check.py
 msprobe/mindspore/overflow_check/overflow_check_tool_factory.py
 msprobe/pytorch/__init__.py
 msprobe/pytorch/function_factory.py
-msprobe/pytorch/module_processer.py
 msprobe/pytorch/parse.py
 msprobe/pytorch/pt_config.py
 msprobe/pytorch/service.py
@@ -206,10 +261,20 @@ msprobe/pytorch/api_accuracy_checker/compare/api_precision_standard.yaml
 msprobe/pytorch/api_accuracy_checker/compare/api_precision_threshold.yaml
 msprobe/pytorch/api_accuracy_checker/compare/compare.py
 msprobe/pytorch/api_accuracy_checker/compare/compare_column.py
+msprobe/pytorch/api_accuracy_checker/compare/compare_input.py
 msprobe/pytorch/api_accuracy_checker/compare/compare_utils.py
 msprobe/pytorch/api_accuracy_checker/generate_op_script/config_op.json
 msprobe/pytorch/api_accuracy_checker/generate_op_script/op_generator.py
 msprobe/pytorch/api_accuracy_checker/generate_op_script/operator_replication.template
+msprobe/pytorch/api_accuracy_checker/precision_standard/absolute_threshold.py
+msprobe/pytorch/api_accuracy_checker/precision_standard/accumulative_error_compare.py
+msprobe/pytorch/api_accuracy_checker/precision_standard/base_standard.py
+msprobe/pytorch/api_accuracy_checker/precision_standard/benchmark_compare.py
+msprobe/pytorch/api_accuracy_checker/precision_standard/binary_consistency.py
+msprobe/pytorch/api_accuracy_checker/precision_standard/standard_config.py
+msprobe/pytorch/api_accuracy_checker/precision_standard/standard_register.py
+msprobe/pytorch/api_accuracy_checker/precision_standard/thousandth_standard.py
+msprobe/pytorch/api_accuracy_checker/precision_standard/ulp_compare.py
 msprobe/pytorch/api_accuracy_checker/run_ut/.keep
 msprobe/pytorch/api_accuracy_checker/run_ut/__init__.py
 msprobe/pytorch/api_accuracy_checker/run_ut/data_generate.py
@@ -227,16 +292,21 @@ msprobe/pytorch/api_accuracy_checker/tensor_transport_layer/server.py
 msprobe/pytorch/api_accuracy_checker/tensor_transport_layer/torch_ops_config.yaml
 msprobe/pytorch/api_accuracy_checker/tensor_transport_layer/utils.py
 msprobe/pytorch/bench_functions/__init__.py
+msprobe/pytorch/bench_functions/apply_adam.py
 msprobe/pytorch/bench_functions/apply_adam_w.py
 msprobe/pytorch/bench_functions/confusion_transpose.py
 msprobe/pytorch/bench_functions/fast_gelu.py
+msprobe/pytorch/bench_functions/group_norm_silu.py
 msprobe/pytorch/bench_functions/layer_norm_eval.py
 msprobe/pytorch/bench_functions/linear.py
 msprobe/pytorch/bench_functions/matmul_backward.py
+msprobe/pytorch/bench_functions/mish.py
+msprobe/pytorch/bench_functions/moe_gating_top_k_softmax.py
 msprobe/pytorch/bench_functions/npu_fusion_attention.py
 msprobe/pytorch/bench_functions/rms_norm.py
 msprobe/pytorch/bench_functions/rotary_mul.py
 msprobe/pytorch/bench_functions/scaled_mask_softmax.py
+msprobe/pytorch/bench_functions/sort_v2.py
 msprobe/pytorch/bench_functions/swiglu.py
 msprobe/pytorch/common/__init__.py
 msprobe/pytorch/common/compare_script.template
@@ -251,6 +321,9 @@ msprobe/pytorch/debugger/__init__.py
 msprobe/pytorch/debugger/debugger_config.py
 msprobe/pytorch/debugger/precision_debugger.py
 msprobe/pytorch/dump/kernel_dump/kernel_config.py
+msprobe/pytorch/dump/module_dump/__init__.py
+msprobe/pytorch/dump/module_dump/module_dump.py
+msprobe/pytorch/dump/module_dump/module_processer.py
 msprobe/pytorch/free_benchmark/__init__.py
 msprobe/pytorch/free_benchmark/main.py
 msprobe/pytorch/free_benchmark/common/__init__.py
@@ -278,14 +351,13 @@ msprobe/pytorch/free_benchmark/result_handlers/check_handler.py
 msprobe/pytorch/free_benchmark/result_handlers/fix_handler.py
 msprobe/pytorch/free_benchmark/result_handlers/handler_factory.py
 msprobe/pytorch/free_benchmark/result_handlers/preheat_handler.py
-msprobe/pytorch/functional/__init__.py
-msprobe/pytorch/functional/module_dump.py
 msprobe/pytorch/grad_probe/__init__.py
 msprobe/pytorch/grad_probe/grad_monitor.py
 msprobe/pytorch/grad_probe/grad_stat_csv.py
 msprobe/pytorch/hook_module/__init__.py
 msprobe/pytorch/hook_module/api_registry.py
 msprobe/pytorch/hook_module/hook_module.py
+msprobe/pytorch/hook_module/register_optimizer_hook.py
 msprobe/pytorch/hook_module/support_wrap_ops.yaml
 msprobe/pytorch/hook_module/utils.py
 msprobe/pytorch/hook_module/wrap_aten.py
@@ -298,6 +370,7 @@ msprobe/pytorch/hook_module/wrap_vf.py
 msprobe/pytorch/monitor/__init__.py
 msprobe/pytorch/monitor/anomaly_analyse.py
 msprobe/pytorch/monitor/anomaly_detect.py
+msprobe/pytorch/monitor/csv2tb.py
 msprobe/pytorch/monitor/features.py
 msprobe/pytorch/monitor/module_hook.py
 msprobe/pytorch/monitor/module_metric.py
@@ -340,6 +413,7 @@ msprobe/visualization/compare/graph_comparator.py
 msprobe/visualization/compare/mode_adapter.py
 msprobe/visualization/graph/__init__.py
 msprobe/visualization/graph/base_node.py
+msprobe/visualization/graph/distributed_analyzer.py
 msprobe/visualization/graph/graph.py
 msprobe/visualization/graph/node_colors.py
 msprobe/visualization/graph/node_op.py

mindstudio_probe-1.2.2/msprobe/CMakeLists.txt ADDED Viewed

@@ -0,0 +1,5 @@
+add_subdirectory(ccsrc)
+if (DEFINED BUILD_TEST_CASE AND "${BUILD_TEST_CASE}" STREQUAL "True")
+add_subdirectory(test)
+endif()

{mindstudio_probe-1.1.1 → mindstudio_probe-1.2.2}/msprobe/README.md RENAMED Viewed

@@ -15,7 +15,7 @@ debugger = PrecisionDebugger(config_path='./config.json')
 ...
 debugger.start() # 一般在训练循环开头启动工具
 ... # 循环体
-debugger.stop() # 一般在训练循环末尾结束工具
+debugger.stop() # 一般在训练循环末尾结束工具。必须调用，否则可能导致精度数据落盘不全
 debugger.step() # 在训练循环的最后需要重置工具，非循环场景不需要
 ```
@@ -51,15 +51,21 @@ export MSPROBE_LOG_LEVEL={x}
 **1. Pytorch 框架下，工具暂不支持 Fully Sharded Data Parallel(FSDP)。**
+**2. 工具读写的所有路径，如config_path、dump_path等，只允许包含大小写字母、数字、下划线、斜杠、点和短横线。**
 ## ⚙️ [安装](./docs/01.installation.md)
+## 🌟 新版本特性
+请参见[特性变更说明](./docs/01.installation.md#特性变更说明)。
 ## 🛠️ config.json [介绍](./docs/02.config_introduction.md) 和 [示例](./docs/03.config_examples.md)
 ## 🧰 主要功能
 ### 0 用前必看
-使用工具前，建议先浏览[**工具功能模块简介、适用场景和当前版本局限性**](./docs/23.tool_function_introduction.md)，了解功能特性。
+使用工具前，建议先浏览[**工具功能模块简介、适用场景和当前版本局限性**](./docs/25.tool_function_introduction.md)，了解功能特性。
 ### 1 数据采集
@@ -121,7 +127,7 @@ MindSpore 动态图场景的[离线预检](./docs/09.accuracy_checker_MindSpore.
 该功能收集和聚合模型训练过程中的网络层，优化器， 通信算子的中间值，帮助诊断模型训练过程中计算， 通信，优化器各部分出现的异常情况。
-[PyTorch 场景的训练状态监控](./docs/19.monitor.md)
+[兼容 PyTorch 和 MindSpore 框架的训练状态监控](./docs/19.monitor.md)
 ### 10 分级可视化构图比对
@@ -131,29 +137,18 @@ MindSpore 动态图场景的[离线预检](./docs/09.accuracy_checker_MindSpore.
 [MindSpore 场景的分级可视化构图比对](./docs/22.visualization_MindSpore.md)
-## 🌟 新版本特性
-若查看历史版本特性，请参见[安装](./docs/01.installation.md)。
+### 11 单算子API自动生成脚本
+该功能将msprobe工具dump的精度数据进行解析，自动生成单API脚本，用于复现整网中出现的算子问题，降低用户复现问题的成本，供开发分析算子问题。
-【数据采集】
-- 支持 config.json 中的 step 传入范围；
-- 优化了指定 step 的机制，指定 step 结束后工具不再采集数据，但训练会继续运行。工具结束运行后，日志提示信息如下：
-    ```bash
-    ****************************************
-    *      msprobe ends successfully.      *
-    ****************************************
-    ```
-    注：在多卡场景，每张卡进程训练到指定 step 之后都会打印一次上述信息。
+[PyTorch 单算子API自动生成脚本](./docs/23.generate_operator_PyTorch.md)
-【精度预检】
-- 在 PyTorch 场景，支持部分 NPU 融合算子预检。
+### 12 数码关联
-【精度比对】
-- 解决了使用 MindSpore 需要安装 PyTorch 的问题。
+该功能只支持 MindSpore 静态图场景，用于将IR图与dump数据进行关联，获取dump数据和代码调用栈的关联关系。
-【无标杆比对】
-- 补充在 PyTorch 场景的性能基线报告；
-- 支持 MindSpore 场景的 change_value 扰动模式。
+[MindSpore 场景的数码关联](./docs/24.code_mapping_Mindspore.md)
 ## 📑 补充材料

{mindstudio_probe-1.1.1 → mindstudio_probe-1.2.2}/msprobe/config.json RENAMED Viewed

@@ -5,6 +5,7 @@
     "step": [],
     "level": "L1",
     "enable_dataloader": false,
+    "async_dump": false,
     "tensor": {
         "scope": [],
         "list":[],

mindstudio-probe 1.1.1__tar.gz → 1.2.2__tar.gz

mindstudio-probe 1.1.1tar.gz → 1.2.2tar.gz