mindstudio-probe 1.0.1__py3-none-any.whl → 1.0.3__py3-none-any.whl

msprobe/doc/grad_probe/grad_probe.md

@@ -0,0 +1,207 @@
+# Ascend模型梯度状态监测工具
+
+梯度状态监测工具提供了两种能力：
+
+- 将模型权重的梯度数据导出。这种功能可以将模型权重的梯度值以统计量的形式采集出来，用以分析问题。
+- 将两份梯度数据进行相似度对比。在有标杆问题中，可以确认训练过程中精度问题出现的step，以及抓取反向过程中的问题。
+
+工具支持PyTorch版本：2.0/2.1/2.2；支持MindSpore版本：r2.3。
+
+## 工具特性
+
+- 使用便捷，无需在训练流程里插入代码
+- 可以精准定位问题出现的step
+
+## 使用方式
+
+### 梯度数据导出
+
+1. 创建配置文件config.json，样例如下：
+
+   ```json
+   {
+    "task": "grad_probe",
+    "dump_path": "./dump_path", 
+    "rank": [],  
+    "step": [],  
+    "grad_probe": {
+        "grad_level": "L1",
+        "param_list": [],
+        "bounds": [-1, 0, 1]
+    }
+   }
+   ```
+   > step指的是优化器被调用的次数（并非模型跑的step，某些step，例如loss为nan时，不会调用优化器）
+
+   **参数说明**
+
+   | 参数                       | 说明                          | 输入类型                     | 是否必选 |
+   |--------------------------------|-----------------------------------|-----------------|----------|
+   | task            | 填为"grad_probe"。 |  str | 是 |
+   | grad_level                  | 输出级别。决定导出数据的详细程度，级别越大导出数据越详细。可取值：L0, L1, L2|str  | 是     |
+   | param_list             | 权重名称列表，表示需要监控的权重。列表为空就表示监控所有权重。 | List[str] | 是       |
+   | rank                   | rank id列表，在多卡场景下，表示需要导出梯度数据的进程的rank id。列表为空就表示导出所有rank的数据。（MindSpore静态图模式下，当前暂不支持指定rank功能） | List[int] | 是       |
+   | step                   | step列表，表示需要导出数据的step列表。列表为空就表示导出所有step的数据。（MindSpore静态图模式下，当前暂不支持指定step功能） | List[int] | 是 |
+   | bounds                 | 区间列表，用来划分区间以统计数值的分布。需要保证由数据小到大排列。可以使用默认值[-1, 0, 1] | List[float] | 是  |
+   | dump_path            | 输出目录。如果不存在就会创建一个新目录。 |  str | 是 |
+
+   **不同级别的level的导出数据**
+
+
+   | 级别 | 特征数据表头                                                 | 是否有方向数据 |
+   | ---- | ------------------------------------------------------------ | -------------- |
+   | L0   | ("param_name", "MD5", "max", "min", "norm", "shape")         | 否             |
+   | L1   | ("param_name", "max", "min", "norm", "shape")         | 是             |
+   | L2   | ("param_name", *intervals, "=0", "max", "min", "norm", "shape") | 是             |
+   
+   intervals就是根据值分布bounds划分出的区间。
+   MindSpore静态图模式下，L0级别中暂不支持"MD5"
+   
+   **方向数据解释**
+   
+   因为模型的参数往往非常大，所以存储真实数据是不可接受的，这里折衷一下，只存储梯度数据的正负号（一个布尔值），也就是方向。
+   
+   **bounds和值分布解释**
+   
+   + 值分布：梯度数据落在各个区间的元素个数占总元素个数的比例。
+   + bounds：一个列表，用来划分出区间以统计值分布。例如传入bounds = [-10, 0, 10]，此时有一个 grad_value: Tensor = [9.3 , 5.4, -1.0, -12.3]，依据 bounds 划分出 (-inf, -10]、(-10, 0]、(0, 10]、(10, inf) 四个区间，然后统计grad_value里的数据落在每个区间内的个数，得到 1、1、2、0。如下图所示：   
+   ![Alt text](img/image-1.png)
+
+2. 插入代码。示例代码如下：
+
+- PyTorch框架：模型构造完成后，传入config.json的路径实例化一个GradientMonitor对象，然后调用gm.monitor并将`模型`作为参数传入。
+```python
+from msprobe.pytorch import PrecisionDebugger
+debugger = PrecisionDebugger("config_json_path")
+debugger.monitor(model)
+```
+- MindSpore框架：优化器构造完成后，传入config.json的路径实例化一个GradientMonitor对象，然后调用gm.monitor并将`优化器`作为参数传入。
+```python
+from msprobe.mindspore import PrecisionDebugger
+debugger = PrecisionDebugger("config_json_path")
+debugger.monitor(optimizer)
+```
+
+3. 结束监控（MindSpore静态图模式下需要）
+
+   在训练结束之后，调用stop接口
+
+```python
+gm.stop()
+```
+
+### 输出结果
+**输出目录结构**（以level配置L2为例）
+
+```bash
+{dump_path}
+      ├── rank{rank_id}
+      │        ├── grad_summary_{step}.csv
+      │        ├── step{step}
+      │        │        ├── {param_name}.npy
+```
++ {timestamp}：梯度工具导出数据的时候会在output_path下生成一个时间戳目录，然后在这个时间戳目录下输出结果。
++ rank_{rank_id}：在分布式场景下，会记录卡的rank_id。非分布式场景下，如果是CPU则记录进程号，如果是CPU或GPU则记录卡号
++ grad_summary_{step}.csv：会分step记录每一步的梯度数据统计值。
++ step_{step}：这个目录下会存放该step的梯度的方向数据。
++ {param_name}.pt(npy)：模型参数的梯度方向数据，PyTorch保存的是pt文件，MindSpore是npy文件。
+
+**grad_summary_{step}.csv**
+
+样例如下：
+
+![Alt text](img/image.png)
+
+| 字段                  | 含义                                                         |
+| --------------------- | ------------------------------------------------------------|
+| Param_name            | 模型参数名称。                                                |
+| MD5                   | 梯度数据的MD5值。                                            |
+| (-inf, -0.01]...[0.01, inf) | 梯度值落在区间内的元素个数占总元素的比例。               |
+| =0                    | 梯度为0的元素个数占总元素的比例。                             |
+| Max                   | 最大值。                                                      |
+| Min                   | 最小值。                                                      |
+| Norm                  | L2norm值。                                                    |
+| Shape                 | 形状。                                                        |
+
+### 梯度相似度比对
+
+会根据所导出的权重，分step比对梯度相似度，输出每个权重的梯度相似度和总的梯度相似度。单个权重的梯度相似度为两份方向数据的重合度，总的梯度相似度为每个权重的梯度相似度按元素个数加权。
+
+#### 前提条件
+
+- 相同配置下，以Level为L1或L2分别采集npu和gpu环境下的梯度数据。
+- 将两份梯度数据传到同一环境下。
+
+#### 使用方式
+
+
+新建如下Python脚本，传入npu和gpu的dump_path以及输出目录，比对结果输出目录不存在的话会新建：
+
+```python
+from msprobe import *
+GradComparator.compare_distributed("配置文件里写的dump_path",
+                                    "配置文件里写的dump_path",
+                                    "比对结果输出目录")
+```
+   
+
+### 比对结果
+
+**输出目录结构**
+
+如下为多卡比对结果，单卡则没有rank_{rank_id}这一级目录。
+
+```bash
+比对结果输出目录
+      ├── rank{rank_id}
+      │         ├── similarities.csv
+      │         └── similarities_picture
+      │                     ├── {param_name}.png
+      │                     └── summary_similarities.png
+```
+
+**问题界定**
+
+原则：对于任意权重，第0步的梯度相似度低于0.97，或者某一步的梯度相似度下降超过0.03，认为这一步存在精度问题。例子如下：
+
+- 第0步相似度低于0.97
+
+![Alt text](img/image-3.png)
+
+- 第3步相似度下降超过0.03
+
+![Alt text](img/image-4.png)
+
+- 正常情况
+
+![Alt text](img/image-2.png)
+
+这个原则是一个经验性的指标，并不是严格的标注，还需要结合实际情况具体分析。
+
+## 公开接口
+
+**接口说明**
+
+```python
+PrecisionDebugger.monitor(module)
+```
+
+| 参数  | 说明                 | 是否必选 |
+| ----- | -------------------- | -------- |
+| module |Pytorch框架下传入模型，必须是torch.nn.Module；MindSpore框架下传入优化器。 | 是       |
+
+
+**接口说明**
+
+```python
+GradComparator.compare_distributed(dump_path1, dump_path2, output_path)
+```
+
+| 参数  | 说明                 | 是否必选 |
+| ----- | -------------------- | -------- |
+| dump_path1 |需要比对的其中一个dump目录，也就是配置文件里写的dump_path。 | 是       |
+| dump_path2 |需要比对的其中一个dump目录，也就是配置文件里写的dump_path，与dump_path1可以互换。 | 是       |
+| output_path |输出结果目录，不存在会新建。 | 是       |
+
+
+# FAQ 

msprobe/mindspore/api_accuracy_checker/api_accuracy_checker.py

@@ -0,0 +1,246 @@
+import json
+import os
+
+from msprobe.core.common.file_check import FileOpen
+from msprobe.core.common.utils import write_csv, add_time_as_suffix
+from msprobe.core.common.const import Const, CompareConst, MsCompareConst
+from msprobe.core.common.log import logger
+from msprobe.mindspore.api_accuracy_checker.api_info import ApiInfo
+from msprobe.mindspore.api_accuracy_checker.api_runner import api_runner, ApiInputAggregation
+from msprobe.mindspore.api_accuracy_checker.base_compare_algorithm import compare_algorithms
+from msprobe.mindspore.api_accuracy_checker.utils import check_and_get_from_json_dict, global_context
+
+
+class BasicInfoAndStatus:
+    def __init__(self, api_name, bench_dtype, tested_dtype, shape, status, err_msg) -> None:
+        self.api_name = api_name
+        self.bench_dtype = bench_dtype
+        self.tested_dtype = tested_dtype
+        self.shape = shape
+        self.status = status
+        self.err_msg = err_msg
+
+class ResultCsvEntry:
+    def __init__(self) -> None:
+        self.forward_pass_status = None
+        self.backward_pass_status = None
+        self.forward_err_msg = ""
+        self.backward_err_msg = ""
+        self.overall_err_msg = None
+
+
+class ApiAccuracyChecker:
+    def __init__(self):
+        self.api_infos = dict()
+        self.results = dict()
+
+    @staticmethod
+    def run_and_compare_helper(api_info, api_name_str, api_input_aggregation, forward_or_backward):
+        '''
+        Args:
+            api_info: ApiInfo
+            api_name_str: str
+            api_input_aggregation: ApiInputAggregation
+            forward_or_backward: str: Union["forward", "backward"]
+
+        Return:
+            output_list: List[tuple(str, str, BasicInfoAndStatus, dict{str: CompareResult})]
+
+        Description:
+            get mindspore api output, run torch api and get output.
+            compare output.
+            record compare result.
+        '''
+        # get output
+        if global_context.get_is_constructed():
+            # constructed situation, need use constructed input to run mindspore api getting tested_output
+            tested_outputs = api_runner(api_input_aggregation, api_name_str, forward_or_backward, Const.MS_FRAMEWORK)
+        else:
+            tested_outputs = api_info.get_compute_element_list(forward_or_backward, Const.OUTPUT)
+        bench_outputs = api_runner(api_input_aggregation, api_name_str, forward_or_backward, Const.PT_FRAMEWORK)
+
+        # compare output
+        output_list = []
+        for i, (bench_out, tested_out) in enumerate(zip(bench_outputs, tested_outputs)):
+            api_name_with_slot = Const.SEP.join([api_name_str, forward_or_backward, Const.OUTPUT, str(i)])
+            bench_dtype = bench_out.get_dtype()
+            tested_dtype = tested_out.get_dtype()
+            shape = bench_out.get_shape()
+
+            compare_result_dict = dict()
+            for compare_algorithm_name, compare_algorithm in compare_algorithms.items():
+                compare_result = compare_algorithm(bench_out, tested_out)
+                compare_result_dict[compare_algorithm_name] = compare_result
+
+            if compare_result_dict.get(CompareConst.COSINE).pass_status == CompareConst.PASS and \
+                compare_result_dict.get(CompareConst.MAX_ABS_ERR).pass_status == CompareConst.PASS:
+                status = CompareConst.PASS
+                err_msg = ""
+            else:
+                status = CompareConst.ERROR
+                err_msg = compare_result_dict.get(CompareConst.COSINE).err_msg + \
+                    compare_result_dict.get(CompareConst.MAX_ABS_ERR).err_msg
+            basic_info_status = \
+                BasicInfoAndStatus(api_name_with_slot, bench_dtype, tested_dtype, shape, status, err_msg)
+            output_list.append(tuple([api_name_str, forward_or_backward, basic_info_status, compare_result_dict]))
+        return output_list
+
+    def parse(self, api_info_path):
+        with FileOpen(api_info_path, "r") as f:
+            api_info_dict = json.load(f)
+
+        # init global context
+        task = check_and_get_from_json_dict(api_info_dict, MsCompareConst.TASK_FIELD,
+                                            "task field in api_info.json",accepted_type=str,
+                                            accepted_value=(MsCompareConst.STATISTICS_TASK,
+                                                            MsCompareConst.TENSOR_TASK))
+        is_constructed = task == MsCompareConst.STATISTICS_TASK
+        if not is_constructed:
+            dump_data_dir = check_and_get_from_json_dict(api_info_dict, MsCompareConst.DUMP_DATA_DIR_FIELD,
+                                                         "dump_data_dir field in api_info.json", accepted_type=str)
+        else:
+            dump_data_dir = ""
+        global_context.init(is_constructed, dump_data_dir)
+
+        api_info_data = check_and_get_from_json_dict(api_info_dict, MsCompareConst.DATA_FIELD,
+                                                     "data field in api_info.json", accepted_type=dict)
+        for api_name, api_info in api_info_data.items():
+            is_mint = api_name.split(Const.SEP)[0] in \
+                (MsCompareConst.MINT, MsCompareConst.MINT_FUNCTIONAL)
+            if not is_mint:
+                continue
+            forbackward_str = api_name.split(Const.SEP)[-1]
+            if forbackward_str not in (Const.FORWARD, Const.BACKWARD):
+                logger.warning(f"api: {api_name} is not recognized as forward api or backward api, skip this.")
+            api_name = Const.SEP.join(api_name.split(Const.SEP)[:-1]) # www.xxx.yyy.zzz --> www.xxx.yyy
+            if api_name not in self.api_infos:
+                self.api_infos[api_name] = ApiInfo(api_name)
+
+            if forbackward_str == Const.FORWARD:
+                self.api_infos[api_name].load_forward_info(api_info)
+            else:
+                self.api_infos[api_name].load_backward_info(api_info)
+
+    def run_and_compare(self):
+        for api_name_str, api_info in self.api_infos.items():
+            if not api_info.check_forward_info():
+                logger.warning(f"api: {api_name_str} is lack of forward infomation, skip forward and backward check")
+                continue
+            forward_inputs = api_info.get_compute_element_list(Const.FORWARD, Const.INPUT)
+            kwargs = api_info.get_kwargs()
+            forward_inputs_aggregation = ApiInputAggregation(forward_inputs, kwargs, None)
+            forward_output_list = None
+            try:
+                forward_output_list = \
+                    self.run_and_compare_helper(api_info, api_name_str, forward_inputs_aggregation, Const.FORWARD)
+            except Exception as e:
+                logger.warning(f"exception occurs when running and comparing {api_name_str} forward api"
+                               f"detailed exception information: {e}")
+            self.record(forward_output_list)
+
+            if not api_info.check_backward_info():
+                logger.warning(f"api: {api_name_str} is lack of backward infomation, skip backward check")
+                continue
+            gradient_inputs = api_info.get_compute_element_list(Const.BACKWARD, Const.INPUT)
+            backward_inputs_aggregation = ApiInputAggregation(forward_inputs, kwargs, gradient_inputs)
+            backward_output_list = None
+            try:
+                backward_output_list = \
+                    self.run_and_compare_helper(api_info, api_name_str, backward_inputs_aggregation, Const.BACKWARD)
+            except Exception as e:
+                logger.warning(f"exception occurs when running and comparing {api_name_str} backward api"
+                               f"detailed exception information: {e}")
+            self.record(backward_output_list)
+
+    def record(self, output_list):
+        if output_list is None:
+            return
+        for output in output_list:
+            api_real_name, forward_or_backward, basic_info, compare_result_dict = output
+            key = tuple([api_real_name, forward_or_backward])
+            if key not in self.results:
+                self.results[key] = []
+            self.results[key].append(tuple([basic_info, compare_result_dict]))
+
+
+    def to_detail_csv(self, csv_dir):
+        # detail_csv
+        detail_csv = []
+        detail_csv_header_basic_info = [
+            MsCompareConst.DETAIL_CSV_API_NAME,
+            MsCompareConst.DETAIL_CSV_BENCH_DTYPE,
+            MsCompareConst.DETAIL_CSV_TESTED_DTYPE,
+            MsCompareConst.DETAIL_CSV_SHAPE,
+        ]
+        detail_csv_header_compare_result = list(compare_algorithms.keys())
+        detail_csv_header_status = [
+            MsCompareConst.DETAIL_CSV_PASS_STATUS,
+            MsCompareConst.DETAIL_CSV_MESSAGE,
+        ]
+
+        detail_csv_header = detail_csv_header_basic_info + detail_csv_header_compare_result + detail_csv_header_status
+        detail_csv.append(detail_csv_header)
+
+        for _, results in self.results.items():
+            # detail csv
+            for res in results:
+                basic_info, compare_result_dict = res
+                csv_row_basic_info = \
+                    [basic_info.api_name, basic_info.bench_dtype, basic_info.tested_dtype, basic_info.shape]
+                csv_row_compare_result = list(compare_result_dict.get(algorithm_name).compare_value \
+                                            for algorithm_name in detail_csv_header_compare_result)
+                csv_row_status = [basic_info.status, basic_info.err_msg]
+                csv_row = csv_row_basic_info  + csv_row_compare_result + csv_row_status
+                detail_csv.append(csv_row)
+
+        file_name = os.path.join(csv_dir, add_time_as_suffix(MsCompareConst.DETAIL_CSV_FILE_NAME))
+        write_csv(detail_csv, file_name, mode="w")
+
+
+    def to_result_csv(self, csv_dir):
+        result_csv_dict = dict()
+        for key, results in self.results.items():
+            api_real_name, forward_or_backward = key
+            forward_or_backward_pass_status = CompareConst.PASS
+            forward_or_backward_overall_err_msg = ""
+            # detail csv
+            for res in results:
+                basic_info, _ = res
+                if basic_info.status != CompareConst.PASS:
+                    forward_or_backward_pass_status = CompareConst.ERROR
+                forward_or_backward_overall_err_msg += basic_info.err_msg
+            forward_or_backward_overall_err_msg = \
+                "" if forward_or_backward_pass_status == CompareConst.PASS else forward_or_backward_overall_err_msg
+
+            #result_csv_dict
+            if api_real_name not in result_csv_dict:
+                result_csv_dict[api_real_name] = ResultCsvEntry()
+            if forward_or_backward == Const.FORWARD:
+                result_csv_dict[api_real_name].forward_pass_status = forward_or_backward_pass_status
+                result_csv_dict[api_real_name].forward_err_msg = forward_or_backward_overall_err_msg
+            else:
+                result_csv_dict[api_real_name].backward_pass_status = forward_or_backward_pass_status
+                result_csv_dict[api_real_name].backward_err_msg = forward_or_backward_overall_err_msg
+
+        #result_csv
+        result_csv = []
+        result_csv_header = [
+            MsCompareConst.DETAIL_CSV_API_NAME,
+            MsCompareConst.RESULT_CSV_FORWARD_TEST_SUCCESS,
+            MsCompareConst.RESULT_CSV_BACKWARD_TEST_SUCCESS,
+            MsCompareConst.DETAIL_CSV_MESSAGE,
+        ]
+        result_csv.append(result_csv_header)
+
+        for api_name, result_csv_entry in result_csv_dict.items():
+            if result_csv_entry.forward_pass_status == CompareConst.PASS and \
+                result_csv_entry.backward_pass_status == CompareConst.PASS:
+                overall_err_msg = ""
+            else:
+                overall_err_msg = result_csv_entry.forward_err_msg + result_csv_entry.backward_err_msg
+            row = [api_name, result_csv_entry.forward_pass_status,
+                   result_csv_entry.backward_pass_status, overall_err_msg]
+            result_csv.append(row)
+
+        file_name = os.path.join(csv_dir, add_time_as_suffix(MsCompareConst.RESULT_CSV_FILE_NAME))
+        write_csv(result_csv, file_name, mode="w")

msprobe/mindspore/api_accuracy_checker/api_info.py

@@ -0,0 +1,69 @@
+from msprobe.mindspore.api_accuracy_checker.compute_element import ComputeElement
+from msprobe.core.common.const import Const
+from msprobe.mindspore.api_accuracy_checker.utils import check_and_get_from_json_dict
+from msprobe.core.common.exceptions import ApiAccuracyCheckerException
+from msprobe.core.common.log import logger
+
+class ApiInfo:
+    def __init__(self, api_name):
+        self.api_name = api_name
+        self.forward_info = None
+        self.backward_info = None
+
+    def load_forward_info(self, forward_info_dict):
+        self.forward_info = forward_info_dict
+
+    def load_backward_info(self, backward_info_dict):
+        self.backward_info = backward_info_dict
+
+    def check_forward_info(self):
+        return self.forward_info is not None
+
+    def check_backward_info(self):
+        return self.backward_info is not None
+
+    def get_compute_element_list(self, forward_or_backward, input_or_output):
+        '''
+        Args:
+            forward_or_backward: str, Union["forward", "backward"]
+            input_or_output: str,  Union["input", "output"]
+
+        Return:
+            compute_element_list: List[ComputeElement]
+        '''
+        mapping = {
+            (Const.FORWARD, Const.INPUT): [self.forward_info, Const.INPUT_ARGS,
+                                           f"input_args field of {self.api_name} forward api in api_info.json"],
+            (Const.FORWARD, Const.OUTPUT): [self.forward_info, Const.OUTPUT,
+                                            f"output field of {self.api_name} forward api in api_info.json"],
+            (Const.BACKWARD, Const.INPUT): [self.backward_info, Const.INPUT,
+                                            f"input field of {self.api_name} backward api in api_info.json"],
+            (Const.BACKWARD, Const.OUTPUT): [self.backward_info, Const.OUTPUT,
+                                             f"output field of {self.api_name} backward api in api_info.json"]
+        }
+        dict_instance, key, key_desc = mapping.get((forward_or_backward, input_or_output))
+        compute_element_info_list = check_and_get_from_json_dict(dict_instance, key, key_desc, accepted_type=list)
+        compute_element_list = [ComputeElement(compute_element_info=compute_element_info)
+                                for compute_element_info in compute_element_info_list]
+        return compute_element_list
+
+    def get_kwargs(self):
+        '''
+        Return:
+            kwargs_compute_element_dict: dict{str: ComputeElement}
+        '''
+        kwargs_dict = check_and_get_from_json_dict(self.forward_info, Const.INPUT_KWARGS,
+                                                   "input_kwargs in api_info.json", accepted_type=dict)
+        for key_str, compute_element_info in kwargs_dict.items():
+            if not isinstance(key_str, str):
+                err_msg = "ApiInfo.get_kwargs failed: compute_element_dict key is not a string"
+                logger.error_log_with_exp(err_msg,
+                                          ApiAccuracyCheckerException(ApiAccuracyCheckerException.ParseJsonFailed))
+            if not isinstance(compute_element_info, (list, dict)):
+                err_msg = "ApiInfo.get_kwargs failed: compute_element_dict value is not a list or dict"
+                logger.error_log_with_exp(err_msg,
+                                          ApiAccuracyCheckerException(ApiAccuracyCheckerException.ParseJsonFailed))
+        kwargs_compute_element_dict = {key_str: ComputeElement(compute_element_info=compute_element_info)
+                                       for key_str, compute_element_info in kwargs_dict.items()}
+        return kwargs_compute_element_dict
+