mindstudio-probe 1.0.1__py3-none-any.whl → 1.0.4__py3-none-any.whl

msprobe/docs/S02.report_free_benchmarking_validation_performance_baseline.md

@@ -0,0 +1,146 @@
+# 无标杆工具场景验证和性能基线报告
+
+## 1 环境信息
+
+NPU：Atlas A2 训练系列产品
+
+CPU：
+
+![输入图片说明](img/cpu_info.png)
+
+Torch：2.1.0
+
+CANN：8.0.T5
+
+除上述环境信息影响性能外，API 的数量、种类以及 Shape 都会对性能产生影响，因此本次选取不同场景网络和不同算子进行测试。
+
+## 2 模型信息和性能基线
+
+大模型在使用 msprobe 采集数据时，建议先简化模型层数，减少采集数据量。
+
+以下场景的性能基线测试数据均为多次测试后取平均值，实际运行时性能数据可能会根据环境状态稍有浮动。
+
+### [2.1 ModelLink 模型](https://gitee.com/ascend/ModelLink)
+
+NUM_LAYER：1
+
+NPU 卡数：1
+
+主要数据类型：FLOAT16
+
+#### 2.1.1 LLaMA2-7B
+
+softmax 算子为 FLOAT32，输入输出大小均为 2G，为模型最大显存开销的 API。在该模型下、对无标杆工具处理模式、插装范围、扰动方式组合下的性能和显存基线进行验证。
+
+| 处理模式   | 前/反向  | 算子范围  | 扰动方式 | 耗时（s） | 显存峰值（GB） | 耗时膨胀倍数 | 显存峰值膨胀倍数 | 备注    |
+|-------|--------|------|-----|-------|---------|--------|-------|--------|
+| /        | /       | /           | /                 | 0.24      | 13.69          | 1            | 1                | 混合精度模式基线                   |
+| check    | 前      | ["softmax"] | improve_precision | 0.26      | 13.69          | 1.08         | 1             | softmax 本身为高精度，跳过      |
+| check    | 前      | ["softmax"] | add_noise         | 0.54      | 19.17          | 2.25         | 1.40             |                                |
+| check    | 前      | ["softmax"] | bit_noise         | 0.56      | 19.17          | 2.33         | 1.40             |                                |
+| check    | 前      | ["softmax"] | change_value      | 0.48      | 14.9           | 2            | 1.09             |                                |
+| check    | 前      | ["softmax"] | no_change         | 0.47      | 14.9           | 1.96         | 1.09             |                                |
+| check | 前 | ["softmax"] | to_cpu | 26.45 | 22.67 | 110.21 | 1.66 | 不建议整网 |
+| check    | 前      | ["matmul"]  | improve_precision | 0.57      | 13.69          | 2.38       | 1                |                                |
+| check    | 前      | ["matmul"]  | change_value      | 0.48      | 13.69          | 2            | 1                |                                |
+| check | 前 | ["matmul"] | to_cpu | 78.43 | 19.20 | 326.79 | 1.40 | 不建议整网 |
+| check | 前 | [] | improve_precision | 3.45 | 18.79 | 14.37 | 1.37 | |
+| check | 前 | [] | add_noise | 4.67 | 19.17 | 19.46 | 1.40 | |
+| check | 前 | [] | bit_noise | 16.99 | 19.17 | 70.79 | 1.40 | |
+| check | 前 | [] | no_change | 3.22 | 14.90 | 13.42 | 1.09 | |
+| check    | 反      | ["softmax"] | improve_precision | 6.23      | 25.69          | 25.96        | 1.88             | 不建议整网                     |
+| check    | 反      | ["softmax"] | change_value      | 22.76     | 25.69          | 94.83        | 1.88             | 不建议整网                     |
+| check | 反 | ["softmax"] | to_cpu | 141.71 | 26.19 | 590.46 | 1.91 | 不建议整网 |
+| fix      | 前      | ["softmax"] | to_cpu            | 9.70      | 16.67          | 40.42        | 1.22             | 不支持整网、不支持反向         |
+| fix      | 前      | ["softmax"] | improve_precision | 0.26      | 14.67          | 1.08         | 1.07             | 不支持整网、不支持反向         |
+| 预热     | 前      | []          | improve_precision | 155.07 | 24.79 | 646.13 | 1.81 | 低精度模型基线、只测预热的迭代 |
+| 预热     | 反      | []          | improve_precision | 72.29 | 22.01 | 301.21 | 1.61 | 低精度模型基线、只测预热的迭代，grad_output 为高精度的算子跳过 |
+
+#### 2.1.2 Aquila2-7B
+
+| 处理模式  | 前/反向   | 算子范围   | 扰动方式 | 耗时（s） | 显存峰值（GB） | 耗时膨胀倍数 | 显存峰值膨胀倍数 | 备注    |
+|----------|------|-----|---|----|-----|-------|------|-------------|
+| /        | /       | /           | /                 | 0.17 | 13.66 | 1            | 1                | 混合精度模式基线   |
+| check    | 前      | []          | improve_precision | 1.57 | 14.24 | 9.24 | 1.04 |                                |
+| check    | 反      | []          | add_noise         | 21.05 | 14.19 | 123.82 | 1.04 |                                |
+| fix      | 前      | []          | improve_precision | 0.95 | 15.55 | 5.59 | 1.14 |                                |
+
+#### 2.1.3 Baichuan2-7B
+
+| 处理模式  | 前/反向  | 算子范围 | 扰动方式 | 耗时（s）| 显存峰值（GB）| 耗时膨胀倍数 | 显存峰值膨胀倍数 | 备注                           |
+|----|-----|---|--|----|----|------|-------|---------|
+| /        | /       | /           | /                 | 0.26 | 12.12 | 1            | 1                | 混合精度模式基线    |
+| check    | 前      | []          | improve_precision | 1.02 | 12.27 | 3.92 | 1.01 |                                |
+| check    | 反      | []          | add_noise         | 11.15 | 12.67 | 42.88 | 1.05 |                                |
+| fix      | 前      | []          | improve_precision | 0.95 | 12.82 | 3.65 | 1.06 |                                |
+
+#### 2.1.4 Bloom-7B
+
+| 处理模式  | 前/反向  | 算子范围  | 扰动方式 | 耗时（s）| 显存峰值（GB）| 耗时膨胀倍数 | 显存峰值膨胀倍数 | 备注   |
+|-----|------|------|------|----|-----|-----|-------|----|
+| /        | /       | /           | /                 | 0.14 | 9.51 | 1            | 1                | 混合精度模式基线  |
+| check    | 前      | []          | improve_precision | 1.64 | 11.58 | 11.71 | 1.22 |                                |
+| check    | 反      | []          | add_noise         | 17.15 | 9.51 | 122.5 | 1 |                                |
+| fix      | 前      | []          | improve_precision | 0.87 | 10.62 | 6.21 | 1.12 |                                |
+
+#### 2.1.5 Interlm-7B
+
+| 处理模式   | 前/反向   | 算子范围   | 扰动方式 | 耗时（s） | 显存峰值（GB） | 耗时膨胀倍数 | 显存峰值膨胀倍数 | 备注   |
+|-------------|--------|-------|----|------|-----|------|-------|----|
+| /        | /       | /           | /                 | 0.13 | 10.76 | 1            | 1                | 混合精度模式基线   |
+| check    | 前      | []          | improve_precision | 1.19 | 11.68 | 9.15 | 1.09 |                                |
+| check    | 反      | []          | add_noise         | 11.69 | 10.89 | 89.92 | 1.01 |                                |
+| fix      | 前      | []          | improve_precision | 0.75 | 11.68 | 5.77 | 1.09 |                                |
+
+#### 2.1.6 Qwen-7B
+
+| 处理模式   | 前/反向   | 算子范围 | 扰动方式 | 耗时（s） | 显存峰值（GB） | 耗时膨胀倍数 | 显存峰值膨胀倍数 | 备注   |
+|--------|-------|-----|-----|----|------|-----|------|------|
+| /        | /       | /           | /                 | 0.28 | 18.41 | 1            | 1                | 混合精度模式基线  |
+| check    | 前      | []          | improve_precision | 2.34 | 23.18 | 8.36 | 1.26 |                                |
+| check    | 反      | []          | add_noise         | 22.07 | 19.47 | 78.82 | 1.06 |                                |
+| fix      | 前      | []          | improve_precision | 1.31 | 21.11 | 4.68 | 1.15 |                                |
+
+#### 2.1.7 Gemma-7B
+
+| 处理模式    | 前/反向   | 算子范围  | 扰动方式 | 耗时（s） | 显存峰值（GB） | 耗时膨胀倍数 | 显存峰值膨胀倍数 | 备注   |
+|--------|-------|------|---|----|-----|-----|-----|---------|
+| /        | /       | /           | /                 | 0.15 | 11.06 | 1            | 1                | 混合精度模式基线    |
+| check    | 前      | []          | improve_precision | 1.49 | 13.17 | 9.93 | 1.19 |                                |
+| check    | 反      | []          | add_noise         | 16.69 | 11.06 | 111.27 | 1 |                                |
+| fix      | 前      | []          | improve_precision | 0.87 | 12.25 | 5.8 | 1.11 |                                |
+
+### [2.2 ModelZoo-PyTorch 模型](https://gitee.com/ascend/ModelZoo-PyTorch)
+
+#### 2.2.1 ResNet50-Cifar
+
+NPU 卡数：1
+
+主要数据类型：FLOAT16
+
+主要算子为 conv2d，每个 step 有 51 个, 因此对 conv2d 进行检测。CV 模型、依赖 mmcv 实现（如果不修改 mmcv 代码、工具无法获取 step 信息和反向信息）。
+
+| 处理模式   | 前/反向   | 算子范围  | 扰动方式 | 耗时（s） | 显存峰值（GB） | 耗时膨胀倍数 | 显存峰值膨胀倍数 | 备注   |
+|------------|---------|--------|-----|------|---|--------|-------|----|
+| /        | /       | /           | /                 | 0.09      |  7.63         | 1            | 1                | 基线                   |
+| check    | 前      | ["conv2d"]  | improve_precision | 0.889      | 7.94          |  9.81       |  1.04            |                                |
+| fix      | 前      | ["conv2d"]  | improve_precision | 0.328      | 7.47          |  3.64       |  0.91            |                                |
+| fix      | 前      | ["conv2d"]  | to_cpu            | 12.23      | 7.47          |  135.88     |  0.91              |                                |
+
+#### 2.2.2 OpenSora1.0
+
+NPU 卡数：4
+
+主要数据类型：FLOAT16
+
+每张卡每个 step 中 linear 算子个数为 257 个，FA 算子个数为 83（FA 算子反向无效）。
+
+| 处理模式   | 前/反向  | 算子范围  | 扰动方式 | 耗时（s） | 显存峰值（GB） | 耗时膨胀倍数 | 显存峰值膨胀倍数 | 备注   |
+|------------|------|-------|----|----|-----|-----|------|-----|
+| /   | /  | /  | /      | 0.99 | 17.61 | 1   | 1     | 混合精度模式基线    |
+| check    | 前      | ["linear","npu_fusion_attention"] | improve_precision | 3.88 | 17.61 | 3.92 | 1 |                                |
+| check | 前 | ["linear","npu_fusion_attention"] | add_noise | 3.46 | 17.61 | 3.49 | 1 | |
+| check | 反 | ["linear"] | improve_precision | 12.61 | 17.61 | 12.74 | 1 | |
+| check    | 反      | ["linear"]  | add_noise         | 9.8 | 17.61 | 9.90 | 1 |                                |
+| fix      | 前      | ["linear"] | to_cpu            | 18.83 | 17.61 | 19.02 | 1 |                                |
+| fix | 前 | ["linear"] | improve_precision | 2.83 | 17.61 | 2.86 | 1 | |

msprobe/mindspore/__init__.py

@@ -1 +1 @@
-from msprobe.mindspore.debugger.precision_debugger import PrecisionDebugger
+from msprobe.mindspore.debugger.precision_debugger import PrecisionDebugger

msprobe/mindspore/api_accuracy_checker/api_accuracy_checker.py

@@ -0,0 +1,255 @@
+import json
+import os
+
+from msprobe.core.common.file_utils import FileOpen, create_directory, write_csv
+from msprobe.core.common.utils import add_time_as_suffix
+from msprobe.core.common.const import Const, CompareConst, MsCompareConst
+from msprobe.mindspore.common.log import logger
+from msprobe.mindspore.api_accuracy_checker.api_info import ApiInfo
+from msprobe.mindspore.api_accuracy_checker.api_runner import api_runner, ApiInputAggregation
+from msprobe.mindspore.api_accuracy_checker.base_compare_algorithm import compare_algorithms
+from msprobe.mindspore.api_accuracy_checker.utils import (check_and_get_from_json_dict, global_context,
+                                                          trim_output_compute_element_list)
+
+
+class BasicInfoAndStatus:
+    def __init__(self, api_name, bench_dtype, tested_dtype, shape, status, err_msg) -> None:
+        self.api_name = api_name
+        self.bench_dtype = bench_dtype
+        self.tested_dtype = tested_dtype
+        self.shape = shape
+        self.status = status
+        self.err_msg = err_msg
+
+class ResultCsvEntry:
+    def __init__(self) -> None:
+        self.forward_pass_status = None
+        self.backward_pass_status = None
+        self.forward_err_msg = ""
+        self.backward_err_msg = ""
+        self.overall_err_msg = None
+
+
+class ApiAccuracyChecker:
+    def __init__(self):
+        self.api_infos = dict()
+        self.results = dict()
+
+    @staticmethod
+    def run_and_compare_helper(api_info, api_name_str, api_input_aggregation, forward_or_backward):
+        '''
+        Args:
+            api_info: ApiInfo
+            api_name_str: str
+            api_input_aggregation: ApiInputAggregation
+            forward_or_backward: str: Union["forward", "backward"]
+
+        Return:
+            output_list: List[tuple(str, str, BasicInfoAndStatus, dict{str: CompareResult})]
+
+        Description:
+            get mindspore api output, run torch api and get output.
+            compare output.
+            record compare result.
+        '''
+        # get output
+        if global_context.get_is_constructed():
+            # constructed situation, need use constructed input to run mindspore api getting tested_output
+            tested_outputs = api_runner(api_input_aggregation, api_name_str, forward_or_backward, Const.MS_FRAMEWORK)
+        else:
+            tested_outputs = api_info.get_compute_element_list(forward_or_backward, Const.OUTPUT)
+        bench_outputs = api_runner(api_input_aggregation, api_name_str, forward_or_backward, Const.PT_FRAMEWORK)
+        tested_outputs = trim_output_compute_element_list(tested_outputs, forward_or_backward)
+        bench_outputs = trim_output_compute_element_list(bench_outputs, forward_or_backward)
+        if len(tested_outputs) != len(bench_outputs):
+            logger.warning(f"ApiAccuracyChecker.run_and_compare_helper: api: {api_name_str}.{forward_or_backward}, "
+                           "number of bench outputs and tested outputs is different, comparing result can be wrong. "
+                           f"tested outputs: {len(tested_outputs)}, bench outputs: {len(bench_outputs)}")
+
+        # compare output
+        output_list = []
+        for i, (bench_out, tested_out) in enumerate(zip(bench_outputs, tested_outputs)):
+            api_name_with_slot = Const.SEP.join([api_name_str, forward_or_backward, Const.OUTPUT, str(i)])
+            bench_dtype = bench_out.get_dtype()
+            tested_dtype = tested_out.get_dtype()
+            shape = bench_out.get_shape()
+
+            compare_result_dict = dict()
+            for compare_algorithm_name, compare_algorithm in compare_algorithms.items():
+                compare_result = compare_algorithm(bench_out, tested_out)
+                compare_result_dict[compare_algorithm_name] = compare_result
+
+            if compare_result_dict.get(CompareConst.COSINE).pass_status == CompareConst.PASS and \
+                compare_result_dict.get(CompareConst.MAX_ABS_ERR).pass_status == CompareConst.PASS:
+                status = CompareConst.PASS
+                err_msg = ""
+            else:
+                status = CompareConst.ERROR
+                err_msg = compare_result_dict.get(CompareConst.COSINE).err_msg + \
+                    compare_result_dict.get(CompareConst.MAX_ABS_ERR).err_msg
+            basic_info_status = \
+                BasicInfoAndStatus(api_name_with_slot, bench_dtype, tested_dtype, shape, status, err_msg)
+            output_list.append(tuple([api_name_str, forward_or_backward, basic_info_status, compare_result_dict]))
+        return output_list
+
+    def parse(self, api_info_path):
+        with FileOpen(api_info_path, "r") as f:
+            api_info_dict = json.load(f)
+
+        # init global context
+        task = check_and_get_from_json_dict(api_info_dict, MsCompareConst.TASK_FIELD,
+                                            "task field in api_info.json",accepted_type=str,
+                                            accepted_value=(MsCompareConst.STATISTICS_TASK,
+                                                            MsCompareConst.TENSOR_TASK))
+        is_constructed = task == MsCompareConst.STATISTICS_TASK
+        if not is_constructed:
+            dump_data_dir = check_and_get_from_json_dict(api_info_dict, MsCompareConst.DUMP_DATA_DIR_FIELD,
+                                                         "dump_data_dir field in api_info.json", accepted_type=str)
+        else:
+            dump_data_dir = ""
+        global_context.init(is_constructed, dump_data_dir)
+
+        api_info_data = check_and_get_from_json_dict(api_info_dict, MsCompareConst.DATA_FIELD,
+                                                     "data field in api_info.json", accepted_type=dict)
+        for api_name, api_info in api_info_data.items():
+            is_mint = api_name.split(Const.SEP)[0] in \
+                (MsCompareConst.MINT, MsCompareConst.MINT_FUNCTIONAL)
+            if not is_mint:
+                continue
+            forbackward_str = api_name.split(Const.SEP)[-1]
+            if forbackward_str not in (Const.FORWARD, Const.BACKWARD):
+                logger.warning(f"api: {api_name} is not recognized as forward api or backward api, skip this.")
+            api_name = Const.SEP.join(api_name.split(Const.SEP)[:-1]) # www.xxx.yyy.zzz --> www.xxx.yyy
+            if api_name not in self.api_infos:
+                self.api_infos[api_name] = ApiInfo(api_name)
+
+            if forbackward_str == Const.FORWARD:
+                self.api_infos[api_name].load_forward_info(api_info)
+            else:
+                self.api_infos[api_name].load_backward_info(api_info)
+
+    def run_and_compare(self):
+        for api_name_str, api_info in self.api_infos.items():
+            if not api_info.check_forward_info():
+                logger.warning(f"api: {api_name_str} is lack of forward infomation, skip forward and backward check")
+                continue
+            forward_inputs = api_info.get_compute_element_list(Const.FORWARD, Const.INPUT)
+            kwargs = api_info.get_kwargs()
+            forward_inputs_aggregation = ApiInputAggregation(forward_inputs, kwargs, None)
+            forward_output_list = None
+            try:
+                forward_output_list = \
+                    self.run_and_compare_helper(api_info, api_name_str, forward_inputs_aggregation, Const.FORWARD)
+            except Exception as e:
+                logger.warning(f"exception occurs when running and comparing {api_name_str} forward api"
+                               f"detailed exception information: {e}")
+            self.record(forward_output_list)
+
+            if not api_info.check_backward_info():
+                logger.warning(f"api: {api_name_str} is lack of backward infomation, skip backward check")
+                continue
+            gradient_inputs = api_info.get_compute_element_list(Const.BACKWARD, Const.INPUT)
+            backward_inputs_aggregation = ApiInputAggregation(forward_inputs, kwargs, gradient_inputs)
+            backward_output_list = None
+            try:
+                backward_output_list = \
+                    self.run_and_compare_helper(api_info, api_name_str, backward_inputs_aggregation, Const.BACKWARD)
+            except Exception as e:
+                logger.warning(f"exception occurs when running and comparing {api_name_str} backward api"
+                               f"detailed exception information: {e}")
+            self.record(backward_output_list)
+
+    def record(self, output_list):
+        if output_list is None:
+            return
+        for output in output_list:
+            api_real_name, forward_or_backward, basic_info, compare_result_dict = output
+            key = tuple([api_real_name, forward_or_backward])
+            if key not in self.results:
+                self.results[key] = []
+            self.results[key].append(tuple([basic_info, compare_result_dict]))
+
+
+    def to_detail_csv(self, csv_dir):
+        # detail_csv
+        detail_csv = []
+        detail_csv_header_basic_info = [
+            MsCompareConst.DETAIL_CSV_API_NAME,
+            MsCompareConst.DETAIL_CSV_BENCH_DTYPE,
+            MsCompareConst.DETAIL_CSV_TESTED_DTYPE,
+            MsCompareConst.DETAIL_CSV_SHAPE,
+        ]
+        detail_csv_header_compare_result = list(compare_algorithms.keys())
+        detail_csv_header_status = [
+            MsCompareConst.DETAIL_CSV_PASS_STATUS,
+            MsCompareConst.DETAIL_CSV_MESSAGE,
+        ]
+
+        detail_csv_header = detail_csv_header_basic_info + detail_csv_header_compare_result + detail_csv_header_status
+        detail_csv.append(detail_csv_header)
+
+        for _, results in self.results.items():
+            # detail csv
+            for res in results:
+                basic_info, compare_result_dict = res
+                csv_row_basic_info = \
+                    [basic_info.api_name, basic_info.bench_dtype, basic_info.tested_dtype, basic_info.shape]
+                csv_row_compare_result = list(compare_result_dict.get(algorithm_name).compare_value \
+                                            for algorithm_name in detail_csv_header_compare_result)
+                csv_row_status = [basic_info.status, basic_info.err_msg]
+                csv_row = csv_row_basic_info  + csv_row_compare_result + csv_row_status
+                detail_csv.append(csv_row)
+
+        file_name = os.path.join(csv_dir, add_time_as_suffix(MsCompareConst.DETAIL_CSV_FILE_NAME))
+        create_directory(csv_dir)
+        write_csv(detail_csv, file_name, mode="w")
+
+
+    def to_result_csv(self, csv_dir):
+        result_csv_dict = dict()
+        for key, results in self.results.items():
+            api_real_name, forward_or_backward = key
+            forward_or_backward_pass_status = CompareConst.PASS
+            forward_or_backward_overall_err_msg = ""
+            # detail csv
+            for res in results:
+                basic_info, _ = res
+                if basic_info.status != CompareConst.PASS:
+                    forward_or_backward_pass_status = CompareConst.ERROR
+                forward_or_backward_overall_err_msg += basic_info.err_msg
+            forward_or_backward_overall_err_msg = \
+                "" if forward_or_backward_pass_status == CompareConst.PASS else forward_or_backward_overall_err_msg
+
+            #result_csv_dict
+            if api_real_name not in result_csv_dict:
+                result_csv_dict[api_real_name] = ResultCsvEntry()
+            if forward_or_backward == Const.FORWARD:
+                result_csv_dict[api_real_name].forward_pass_status = forward_or_backward_pass_status
+                result_csv_dict[api_real_name].forward_err_msg = forward_or_backward_overall_err_msg
+            else:
+                result_csv_dict[api_real_name].backward_pass_status = forward_or_backward_pass_status
+                result_csv_dict[api_real_name].backward_err_msg = forward_or_backward_overall_err_msg
+
+        #result_csv
+        result_csv = []
+        result_csv_header = [
+            MsCompareConst.DETAIL_CSV_API_NAME,
+            MsCompareConst.RESULT_CSV_FORWARD_TEST_SUCCESS,
+            MsCompareConst.RESULT_CSV_BACKWARD_TEST_SUCCESS,
+            MsCompareConst.DETAIL_CSV_MESSAGE,
+        ]
+        result_csv.append(result_csv_header)
+
+        for api_name, result_csv_entry in result_csv_dict.items():
+            if result_csv_entry.forward_pass_status == CompareConst.PASS and \
+                result_csv_entry.backward_pass_status == CompareConst.PASS:
+                overall_err_msg = ""
+            else:
+                overall_err_msg = result_csv_entry.forward_err_msg + result_csv_entry.backward_err_msg
+            row = [api_name, result_csv_entry.forward_pass_status,
+                   result_csv_entry.backward_pass_status, overall_err_msg]
+            result_csv.append(row)
+
+        file_name = os.path.join(csv_dir, add_time_as_suffix(MsCompareConst.RESULT_CSV_FILE_NAME))
+        create_directory(csv_dir)
+        write_csv(result_csv, file_name, mode="w")

msprobe/mindspore/api_accuracy_checker/api_info.py

@@ -0,0 +1,69 @@
+from msprobe.mindspore.api_accuracy_checker.compute_element import ComputeElement
+from msprobe.core.common.const import Const
+from msprobe.mindspore.api_accuracy_checker.utils import check_and_get_from_json_dict
+from msprobe.core.common.exceptions import ApiAccuracyCheckerException
+from msprobe.mindspore.common.log import logger
+
+class ApiInfo:
+    def __init__(self, api_name):
+        self.api_name = api_name
+        self.forward_info = None
+        self.backward_info = None
+
+    def load_forward_info(self, forward_info_dict):
+        self.forward_info = forward_info_dict
+
+    def load_backward_info(self, backward_info_dict):
+        self.backward_info = backward_info_dict
+
+    def check_forward_info(self):
+        return self.forward_info is not None
+
+    def check_backward_info(self):
+        return self.backward_info is not None
+
+    def get_compute_element_list(self, forward_or_backward, input_or_output):
+        '''
+        Args:
+            forward_or_backward: str, Union["forward", "backward"]
+            input_or_output: str,  Union["input", "output"]
+
+        Return:
+            compute_element_list: List[ComputeElement]
+        '''
+        mapping = {
+            (Const.FORWARD, Const.INPUT): [self.forward_info, Const.INPUT_ARGS,
+                                           f"input_args field of {self.api_name} forward api in api_info.json"],
+            (Const.FORWARD, Const.OUTPUT): [self.forward_info, Const.OUTPUT,
+                                            f"output field of {self.api_name} forward api in api_info.json"],
+            (Const.BACKWARD, Const.INPUT): [self.backward_info, Const.INPUT,
+                                            f"input field of {self.api_name} backward api in api_info.json"],
+            (Const.BACKWARD, Const.OUTPUT): [self.backward_info, Const.OUTPUT,
+                                             f"output field of {self.api_name} backward api in api_info.json"]
+        }
+        dict_instance, key, key_desc = mapping.get((forward_or_backward, input_or_output))
+        compute_element_info_list = check_and_get_from_json_dict(dict_instance, key, key_desc, accepted_type=list)
+        compute_element_list = [ComputeElement(compute_element_info=compute_element_info)
+                                for compute_element_info in compute_element_info_list]
+        return compute_element_list
+
+    def get_kwargs(self):
+        '''
+        Return:
+            kwargs_compute_element_dict: dict{str: ComputeElement}
+        '''
+        kwargs_dict = check_and_get_from_json_dict(self.forward_info, Const.INPUT_KWARGS,
+                                                   "input_kwargs in api_info.json", accepted_type=dict)
+        for key_str, compute_element_info in kwargs_dict.items():
+            if not isinstance(key_str, str):
+                err_msg = "ApiInfo.get_kwargs failed: compute_element_dict key is not a string"
+                logger.error_log_with_exp(err_msg,
+                                          ApiAccuracyCheckerException(ApiAccuracyCheckerException.ParseJsonFailed))
+            if not isinstance(compute_element_info, (list, dict)):
+                err_msg = "ApiInfo.get_kwargs failed: compute_element_dict value is not a list or dict"
+                logger.error_log_with_exp(err_msg,
+                                          ApiAccuracyCheckerException(ApiAccuracyCheckerException.ParseJsonFailed))
+        kwargs_compute_element_dict = {key_str: ComputeElement(compute_element_info=compute_element_info)
+                                       for key_str, compute_element_info in kwargs_dict.items()}
+        return kwargs_compute_element_dict
+

msprobe/mindspore/api_accuracy_checker/api_runner.py

@@ -0,0 +1,156 @@
+
+
+import mindspore
+import torch
+from mindspore import ops
+
+from msprobe.mindspore.api_accuracy_checker.compute_element import ComputeElement
+from msprobe.core.common.const import Const, MsCompareConst
+from msprobe.core.common.exceptions import ApiAccuracyCheckerException
+from msprobe.mindspore.common.log import logger
+from msprobe.mindspore.api_accuracy_checker.utils import convert_to_tuple
+from msprobe.mindspore.api_accuracy_checker.type_mapping import float_dtype_str_list, torch_dtype_to_dtype_str
+
+
+class ApiInputAggregation:
+    def __init__(self, inputs, kwargs, gradient_inputs) -> None:
+        '''
+        Args:
+            inputs: List[ComputeElement]
+            kwargs: dict{str: ComputeElement}
+            gradient_inputs: Union[List[ComputeElement], None]
+        '''
+        self.inputs = inputs
+        self.kwargs = kwargs
+        self.gradient_inputs = gradient_inputs
+
+api_parent_module_mapping = {
+    (MsCompareConst.MINT, Const.MS_FRAMEWORK): mindspore.mint,
+    (MsCompareConst.MINT, Const.PT_FRAMEWORK): torch,
+    (MsCompareConst.MINT_FUNCTIONAL, Const.MS_FRAMEWORK): mindspore.mint.nn.functional,
+    (MsCompareConst.MINT_FUNCTIONAL, Const.PT_FRAMEWORK): torch.nn.functional
+}
+
+
+class ApiRunner:
+    def __call__(self, api_input_aggregation, api_name_str, forward_or_backward=Const.FORWARD,
+                 api_platform=Const.MS_FRAMEWORK):
+        '''
+        Args:
+            api_input_aggregation: ApiInputAggregation
+            api_name_str: str, e.g. "MintFunctional.relu.0"
+            forward_or_backward: str, Union["forward", "backward"]
+            api_platform: str, Union["mindspore", "torch"]
+
+        Return:
+            outputs: list[ComputeElement]
+
+        Description:
+            run mindspore.mint/torch api
+        '''
+        api_type_str, api_sub_name = self.get_info_from_name(api_name_str)
+        api_instance = self.get_api_instance(api_type_str, api_sub_name, api_platform)
+
+        return self.run_api(api_instance, api_input_aggregation, forward_or_backward, api_platform)
+
+    @staticmethod
+    def get_info_from_name(api_name_str):
+        '''
+        Args:
+            api_name_str: str, the trimmed key of data dict in api_info.json. e.g. "MintFunctional.relu.0"
+
+        Return:
+            api_type_str: str, Union["MintFunctional", "Mint"]
+            api_sub_name: str, e.g. "relu"
+        '''
+        api_name_list = api_name_str.split(Const.SEP)
+        if len(api_name_list) != 3:
+            err_msg = f"ApiRunner.get_info_from_name failed: api_name_str: {api_name_str} is not in defined format"
+            logger.error_log_with_exp(err_msg, ApiAccuracyCheckerException(ApiAccuracyCheckerException.WrongValue))
+        api_type_str, api_sub_name = api_name_list[0], api_name_list[1]
+        if api_type_str not in [MsCompareConst.MINT, MsCompareConst.MINT_FUNCTIONAL]:
+            err_msg = f"ApiRunner.get_info_from_name failed: not mint or mint.nn.functional api"
+            logger.error_log_with_exp(err_msg, ApiAccuracyCheckerException(ApiAccuracyCheckerException.WrongValue))
+
+        return api_type_str, api_sub_name
+
+    @staticmethod
+    def get_api_instance(api_type_str, api_sub_name, api_platform):
+        '''
+        Args:
+            api_type_str: str, Union["MintFunctional", "Mint"]
+            api_sub_name: str, e.g. "relu"
+            api_platform: str: Union["mindpore", "torch"]
+
+        Return:
+            api_instance: function object
+
+        Description:
+            get mindspore.mint/torch api fucntion
+            mindspore.mint.{api_sub_name} <--> torch.{api_sub_name}
+            mindspore.mint.nn.functional.{api_sub_name} <--> torch.nn.functional.{api_sub_name}
+        '''
+
+        api_parent_module = api_parent_module_mapping.get((api_type_str, api_platform))
+        module_str = "mindspore.mint." if api_platform == Const.MS_FRAMEWORK else "torch."
+        submodule_str = "nn.functional." if api_type_str == MsCompareConst.MINT_FUNCTIONAL else ""
+        full_api_name = module_str + submodule_str + api_sub_name
+        if not hasattr(api_parent_module, api_sub_name):
+            err_msg = f"ApiRunner.get_api_instance failed: {full_api_name} is not found"
+            logger.error_log_with_exp(err_msg, ApiAccuracyCheckerException(ApiAccuracyCheckerException.ApiWrong))
+
+        api_instance = getattr(api_parent_module, api_sub_name)
+        if not callable(api_instance):
+            err_msg = f"ApiRunner.get_api_instance failed: {full_api_name} is not callable"
+            logger.error_log_with_exp(err_msg, ApiAccuracyCheckerException(ApiAccuracyCheckerException.ApiWrong))
+
+        return api_instance
+
+    @staticmethod
+    def run_api(api_instance, api_input_aggregation, forward_or_backward, api_platform):
+        inputs = tuple(compute_element.get_parameter(get_origin=False, tensor_platform=api_platform)
+                       for compute_element in api_input_aggregation.inputs)
+        kwargs = {key: value.get_parameter(get_origin=False, tensor_platform=api_platform)
+                  for key, value in api_input_aggregation.kwargs.items()}
+        gradient_inputs = api_input_aggregation.gradient_inputs
+
+        if forward_or_backward == Const.FORWARD:
+            forward_result = api_instance(*inputs, **kwargs) # can be single tensor or tuple
+            forward_result_tuple = convert_to_tuple(forward_result)
+            res_compute_element_list = [ComputeElement(parameter=api_res) for api_res in forward_result_tuple]
+        else:
+            if gradient_inputs is None:
+                err_msg = f"ApiRunner.run_api failed: run backward api but gradient_inputs is missing"
+                logger.error_log_with_exp(err_msg, ApiAccuracyCheckerException(ApiAccuracyCheckerException.WrongValue))
+            gradient_inputs = tuple(compute_element.get_parameter(get_origin=False, tensor_platform=api_platform)
+                                    for compute_element in gradient_inputs)
+            if api_platform == Const.MS_FRAMEWORK:
+                if len(gradient_inputs) == 1:
+                    gradient_inputs = gradient_inputs[0]
+                def api_with_kwargs(*forward_inputs):
+                    return api_instance(*forward_inputs, **kwargs)
+                grad_func = ops.GradOperation(get_all=True, sens_param=True)(api_with_kwargs)
+                backward_result = grad_func(*inputs, gradient_inputs) # can be single tensor or tuple
+                backward_result_tuple = convert_to_tuple(backward_result)
+                res_compute_element_list = [ComputeElement(parameter=api_res) for api_res in backward_result_tuple]
+            else:
+                #set requires_grad
+                requires_grad_index = []
+                for index, tensor in enumerate(inputs):
+                    if isinstance(tensor, torch.Tensor) and \
+                        torch_dtype_to_dtype_str.get(tensor.dtype) in float_dtype_str_list:
+                        setattr(tensor, "requires_grad", True)
+                        requires_grad_index.append(index)
+                forward_results = api_instance(*inputs, **kwargs)
+                forward_results = convert_to_tuple(forward_results)
+                for forward_res, gradient_in in zip(forward_results, gradient_inputs):
+                    forward_res.backward(gradient_in)
+                backward_result_list = []
+                for index in requires_grad_index:
+                    backward_result_list.append(getattr(inputs[index], "grad"))
+                res_compute_element_list = [ComputeElement(parameter=api_res) for api_res in backward_result_list]
+
+        return res_compute_element_list
+
+
+api_runner = ApiRunner()