@alanwang-merit/meritdata-platform-skill-package 1.0.0

package/skills/meritdata-platform-impexp/references/async-and-validate.md

@@ -0,0 +1,353 @@
+# 异步处理与校验（async-and-validate）
+
+> 大文件导入必须异步处理；导入校验失败必须收集错误行。
+
+---
+
+## 1. 异步处理
+
+### 1.1 为什么必须异步
+
+| 文件大小 | 同步处理 | 异步处理 |
+|---------|---------|---------|
+| < 1MB | 可接受 | 推荐 |
+| 1-10MB | 风险（超时）| 必须 |
+| > 10MB | ❌ 必超时 | ✅ 必异步 |
+| > 100MB | ❌ OOM | ✅ 分片 |
+
+### 1.2 实现方案
+
+**方案 A：Spring Async**
+
+```java
+@Configuration
+@EnableAsync
+public class AsyncConfig {
+
+    @Bean("importExecutor")
+    public Executor importExecutor() {
+        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
+        executor.setCorePoolSize(2);
+        executor.setMaxPoolSize(5);
+        executor.setQueueCapacity(50);
+        executor.setThreadNamePrefix("import-");
+        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
+        executor.initialize();
+        return executor;
+    }
+}
+
+@Service
+public class ImportTaskService {
+
+    @Autowired
+    private ImportService importService;
+
+    @Async("importExecutor")
+    public Future<ImportResult> asyncImport(String taskId, MultipartFile file) {
+        ImportResult result = importService.doImport(file);
+        taskRepository.updateResult(taskId, result);
+        return new AsyncResult<>(result);
+    }
+}
+```
+
+**方案 B：消息队列（高可靠场景）**
+
+```
+用户上传 → 写入 MQ → 后台 Worker 消费 → 更新任务状态 → 通知用户
+```
+
+**方案 C：Job（详见 meritdata-platform-job）**
+
+适合定期批量导入（如每天定时导入某个外部文件）。
+
+### 1.3 任务状态查询
+
+```java
+@RestController
+@RequestMapping("/api/v1/import-tasks")
+public class ImportTaskController {
+
+    @Autowired
+    private ImportTaskRepository taskRepository;
+
+    @GetMapping("/{taskId}")
+    public ApiResponse<ImportTaskDTO> getTaskStatus(@PathVariable String taskId) {
+        ImportTask task = taskRepository.findById(taskId).orElseThrow();
+        return ApiResponse.success(convertToDTO(task));
+    }
+}
+```
+
+### 1.4 前端轮询
+
+```vue
+<template>
+  <el-progress :percentage="task.progress" :status="task.status" />
+  <el-button @click="checkStatus">刷新状态</el-button>
+</template>
+
+<script setup>
+import { ref, onMounted } from 'vue';
+
+const task = ref({ progress: 0, status: 'PROCESSING' });
+
+async function checkStatus() {
+  const data = await getImportTask(taskId);
+  task.value = data;
+  if (task.value.status === 'PROCESSING') {
+    setTimeout(checkStatus, 2000);  // 每 2 秒轮询
+  }
+}
+</script>
+```
+
+---
+
+## 2. 错误行收集
+
+### 2.1 错误模型
+
+```java
+@Data
+@AllArgsConstructor
+public class ImportError {
+    private Integer rowIndex;      // 行号（从第 2 行开始，数据行）
+    private String field;          // 出错字段
+    private String message;        // 错误消息
+    private String rowData;        // 行原始数据（方便排查）
+}
+
+@Data
+public class ImportResult {
+    private Integer totalCount;
+    private Integer successCount;
+    private Integer errorCount;
+    private List<ImportError> errors = new ArrayList<>();
+}
+```
+
+### 2.2 错误收集器
+
+```java
+public class ImportErrorCollector {
+
+    private final List<ImportError> errors = new ArrayList<>();
+
+    public void addError(int rowIndex, String message) {
+        errors.add(new ImportError(rowIndex, "", message, ""));
+    }
+
+    public void addFieldError(int rowIndex, String field, String message) {
+        errors.add(new ImportError(rowIndex, field, message, ""));
+    }
+
+    public void addRowError(int rowIndex, String rowData, String message) {
+        errors.add(new ImportError(rowIndex, "", message, rowData));
+    }
+
+    public List<ImportError> getErrors() {
+        return errors;
+    }
+
+    public boolean hasErrors() {
+        return !errors.isEmpty();
+    }
+
+    public int getErrorCount() {
+        return errors.size();
+    }
+}
+```
+
+### 2.3 监听器中使用
+
+```java
+@Slf4j
+public class OrderImportListener extends AnalysisEventListener<OrderImportVO> {
+
+    private final ImportErrorCollector errorCollector = new ImportErrorCollector();
+
+    @Override
+    public void invoke(OrderImportVO data, AnalysisContext context) {
+        int rowIndex = context.readRowHolder().getRowIndex() + 1;
+
+        // 字段校验
+        List<String> errors = new ArrayList<>();
+
+        if (StringUtils.isBlank(data.getOrderNo())) {
+            errors.add("订单号不能为空");
+        }
+
+        if (data.getAmount() == null || data.getAmount().compareTo(BigDecimal.ZERO) <= 0) {
+            errors.add("金额必须大于 0");
+        }
+
+        if (!errors.isEmpty()) {
+            String rowData = JSON.toJSONString(data);
+            for (String error : errors) {
+                errorCollector.addRowError(rowIndex, rowData, error);
+            }
+            return;  // 不入库
+        }
+
+        // 通过校验，收集到批量列表
+        cachedData.add(data);
+    }
+}
+```
+
+---
+
+## 3. 校验失败的处理策略
+
+### 3.1 模式 A：全部回滚（严格）
+
+任何一行校验失败，整批回滚。
+
+**适用**：财务、合同等高一致性场景。
+
+```java
+@Override
+public ImportResult doImport(MultipartFile file) {
+    ImportResult result = new ImportResult();
+
+    EasyExcel.read(file.getInputStream(), OrderImportVO.class, new OrderImportListener(...)).sheet().doRead();
+
+    if (result.getErrorCount() > 0) {
+        // 不入库
+        return result;
+    }
+
+    // 全部校验通过 → 入库
+    saveBatch(validData);
+    return result;
+}
+```
+
+### 3.2 模式 B：部分导入（容错）
+
+跳过错误行，成功行正常入库。
+
+**适用**：批量初始化、字典导入等容错场景。
+
+```java
+@Override
+public ImportResult doImport(MultipartFile file) {
+    ImportResult result = new ImportResult();
+
+    EasyExcel.read(file.getInputStream(), OrderImportVO.class, new OrderImportListener(...)).sheet().doRead();
+
+    // 部分导入：成功行入库，错误行返回给用户
+    if (!validData.isEmpty()) {
+        saveBatch(validData);
+        result.setSuccessCount(validData.size());
+    }
+    result.setErrorCount(errorCollector.getErrorCount());
+    result.setErrors(errorCollector.getErrors());
+    return result;
+}
+```
+
+### 3.3 错误报告导出
+
+```java
+@GetMapping("/import-tasks/{taskId}/errors")
+public void downloadErrorReport(@PathVariable String taskId, HttpServletResponse response) {
+    ImportTask task = taskRepository.findById(taskId).orElseThrow();
+    List<ImportError> errors = task.getErrors();
+
+    response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
+    response.setHeader("Content-disposition", "attachment;filename=import_errors.xlsx");
+
+    EasyExcel.write(response.getOutputStream(), ImportErrorExportVO.class)
+        .sheet("错误报告")
+        .doWrite(convertToExportVO(errors));
+}
+```
+
+---
+
+## 4. 关联校验
+
+### 4.1 场景
+
+导入的"部门名称"必须在平台存在。
+
+### 4.2 批量预加载 + Map 查询
+
+```java
+@Override
+public ImportResult doImport(MultipartFile file) {
+    // 1. 预加载所有部门（避免循环查）
+    List<OrgDTO> allOrgs = orgClient.listAll();
+    Map<String, String> orgCodeToId = allOrgs.stream()
+        .collect(Collectors.toMap(OrgDTO::getCode, OrgDTO::getId));
+
+    // 2. 导入时校验
+    EasyExcel.read(file.getInputStream(), OrderImportVO.class, new OrderImportListener(orgCodeToId, ...))
+        .sheet().doRead();
+}
+```
+
+### 4.3 ❌ 反例：循环单条查
+
+```java
+// ❌ 错：1000 条数据 = 1000 次 Feign
+@Override
+public void invoke(OrderImportVO data, AnalysisContext context) {
+    OrgDTO org = orgClient.getByCode(data.getDeptCode());  // N+1
+    // ...
+}
+```
+
+---
+
+## 5. 事务边界
+
+### 5.1 推荐：分批事务
+
+```java
+public class OrderImportListener extends AnalysisEventListener<OrderImportVO> {
+
+    private static final int BATCH_SIZE = 500;
+    private List<OrderImportVO> cachedData = new ArrayList<>(BATCH_SIZE);
+
+    @Override
+    public void invoke(OrderImportVO data, AnalysisContext context) {
+        if (validate(data)) {
+            cachedData.add(data);
+        }
+
+        if (cachedData.size() >= BATCH_SIZE) {
+            saveBatch();  // 每 500 行一个事务
+            cachedData.clear();
+        }
+    }
+
+    @Transactional
+    public void saveBatch() {
+        importService.batchSave(cachedData);
+    }
+}
+```
+
+### 5.2 ❌ 反例：单事务导入全部
+
+```java
+// ❌ 错：100 万行 → 1 个事务 → 锁等待 30 分钟
+@Transactional
+public void doImport(MultipartFile file) {
+    List<OrderImportVO> allData = readAll(file);
+    orderRepository.saveAll(allData);  // 单事务
+}
+```
+
+---
+
+## 6. 关联阅读
+
+- [excel-guide.md](excel-guide.md) — EasyExcel 接入
+- [scenarios.md](scenarios.md) — 典型场景
+- [SKILL.md](../SKILL.md)

package/skills/meritdata-platform-impexp/references/excel-guide.md

@@ -0,0 +1,326 @@
+# EasyExcel 导入导出指南（excel-guide）
+
+> 平台使用 EasyExcel 作为 Excel 处理框架。本指南说明模板设计、读写实现。
+
+---
+
+## 1. 引入依赖
+
+```xml
+<dependency>
+    <groupId>com.alibaba</groupId>
+    <artifactId>easyexcel</artifactId>
+    <version>3.3.4</version>
+</dependency>
+```
+
+---
+
+## 2. 导出 Excel
+
+### 2.1 定义数据模型
+
+```java
+@Data
+@ColumnWidth(20)
+public class OrderExportVO {
+
+    @ExcelProperty("订单号")
+    private String orderNo;
+
+    @ExcelProperty("申请人")
+    private String applicantName;
+
+    @ExcelProperty("金额")
+    private BigDecimal amount;
+
+    @ExcelProperty(value = "状态", converter = StatusConverter.class)
+    private String status;
+
+    @ExcelProperty("创建时间")
+    @DateTimeFormat("yyyy-MM-dd HH:mm:ss")
+    private LocalDateTime createTime;
+}
+```
+
+### 2.2 简单导出
+
+```java
+@GetMapping("/export")
+public void exportOrders(HttpServletResponse response) throws IOException {
+    response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
+    response.setCharacterEncoding("utf-8");
+    String fileName = URLEncoder.encode("采购订单", "UTF-8").replaceAll("\\+", "%20");
+    response.setHeader("Content-disposition", "attachment;filename*=utf-8''" + fileName + ".xlsx");
+
+    // 1. 查数据
+    List<OrderExportVO> data = orderService.listForExport(query);
+
+    // 2. 写 Excel
+    EasyExcel.write(response.getOutputStream(), OrderExportVO.class)
+        .sheet("订单列表")
+        .doWrite(data);
+}
+```
+
+### 2.3 多 Sheet 导出
+
+```java
+ExcelWriter excelWriter = null;
+try {
+    excelWriter = EasyExcel.write(response.getOutputStream()).build();
+
+    WriteSheet sheet1 = EasyExcel.writerSheet(0, "订单列表").head(OrderExportVO.class).build();
+    excelWriter.write(orderList, sheet1);
+
+    WriteSheet sheet2 = EasyExcel.writerSheet(1, "订单明细").head(OrderItemExportVO.class).build();
+    excelWriter.write(itemList, sheet2);
+} finally {
+    if (excelWriter != null) excelWriter.finish();
+}
+```
+
+### 2.4 复杂表头（合并单元格）
+
+```java
+@Data
+public class ComplexOrderExportVO {
+
+    @ExcelProperty({"订单信息", "订单号"})
+    private String orderNo;
+
+    @ExcelProperty({"订单信息", "申请人"})
+    private String applicantName;
+
+    @ExcelProperty({"金额信息", "金额"})
+    private BigDecimal amount;
+
+    @ExcelProperty({"金额信息", "税额"})
+    private BigDecimal taxAmount;
+
+    @ExcelProperty({"时间信息", "创建时间"})
+    private LocalDateTime createTime;
+}
+```
+
+---
+
+## 3. 导入 Excel
+
+### 3.1 定义导入模型
+
+```java
+@Data
+public class OrderImportVO {
+
+    @ExcelProperty("订单号")
+    @NotBlank(message = "订单号不能为空")
+    private String orderNo;
+
+    @ExcelProperty("金额")
+    @NotNull(message = "金额不能为空")
+    private BigDecimal amount;
+
+    @ExcelProperty("申请人工号")
+    private String applicantCode;
+}
+```
+
+### 3.2 监听器（处理每行数据）
+
+```java
+@Slf4j
+public class OrderImportListener extends AnalysisEventListener<OrderImportVO> {
+
+    private static final int BATCH_SIZE = 500;
+
+    private List<OrderImportVO> cachedData = new ArrayList<>(BATCH_SIZE);
+
+    private final OrderImportService importService;
+    private final ImportErrorCollector errorCollector;
+
+    @Override
+    public void invoke(OrderImportVO data, AnalysisContext context) {
+        // 1. 行级校验
+        if (data.getAmount() == null || data.getAmount().compareTo(BigDecimal.ZERO) <= 0) {
+            errorCollector.addError(context.readRowHolder().getRowIndex(), "金额必须大于 0");
+            return;
+        }
+
+        cachedData.add(data);
+
+        // 2. 批量处理
+        if (cachedData.size() >= BATCH_SIZE) {
+            importService.batchSave(cachedData);
+            cachedData.clear();
+        }
+    }
+
+    @Override
+    public void doAfterAllAnalysed(AnalysisContext context) {
+        // 处理剩余数据
+        if (!cachedData.isEmpty()) {
+            importService.batchSave(cachedData);
+            cachedData.clear();
+        }
+        log.info("导入完成，共 {} 行", context.readRowHolder().getRowIndex());
+    }
+}
+```
+
+### 3.3 Controller
+
+```java
+@PostMapping("/import")
+public ApiResponse<ImportResult> importOrders(@RequestParam MultipartFile file) {
+    ImportResult result = new ImportResult();
+
+    try {
+        EasyExcel.read(file.getInputStream(), OrderImportVO.class, new OrderImportListener(importService, result.getErrors()))
+            .sheet()
+            .doRead();
+        result.setSuccess(true);
+        return ApiResponse.success(result);
+    } catch (Exception e) {
+        log.error("导入失败", e);
+        return ApiResponse.error("导入失败：" + e.getMessage());
+    }
+}
+```
+
+---
+
+## 4. 字典转换器
+
+```java
+public class StatusConverter implements Converter<String> {
+
+    @Override
+    public Class<?> supportJavaTypeKey() {
+        return String.class;
+    }
+
+    @Override
+    public CellDataTypeEnum supportExcelTypeKey() {
+        return CellDataTypeEnum.STRING;
+    }
+
+    @Override
+    public String convertToJavaData(CellData cellData, ExcelContentProperty contentProperty, GlobalConfiguration globalConfiguration) {
+        // Excel 读 → Java 对象
+        String name = cellData.getStringValue();
+        return lookupService.getCodeByName("ORDER_STATUS", name);
+    }
+
+    @Override
+    public CellData<?> convertToExcelData(String value, ExcelContentProperty contentProperty, GlobalConfiguration globalConfiguration) {
+        // Java 对象 → Excel 写
+        String name = lookupService.getName("ORDER_STATUS", value);
+        return new CellData<>(name);
+    }
+}
+```
+
+---
+
+## 5. 模板下载
+
+### 5.1 简单模板
+
+```java
+@GetMapping("/template")
+public void downloadTemplate(HttpServletResponse response) throws IOException {
+    response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
+    response.setCharacterEncoding("utf-8");
+    String fileName = URLEncoder.encode("订单导入模板", "UTF-8");
+    response.setHeader("Content-disposition", "attachment;filename=" + fileName + ".xlsx");
+
+    // 只写表头
+    EasyExcel.write(response.getOutputStream(), OrderImportVO.class)
+        .sheet("订单列表")
+        .doWrite(Collections.emptyList());
+}
+```
+
+### 5.2 带下拉选择的模板
+
+```java
+// 创建带下拉的模板
+EasyExcel.write(response.getOutputStream(), OrderImportVO.class)
+    .registerWriteHandler(new DropdownWriteHandler("状态", Arrays.asList("草稿", "审批中", "已完成")))
+    .sheet("订单列表")
+    .doWrite(Collections.emptyList());
+```
+
+---
+
+## 6. ❌ 反模式
+
+### 6.1 ❌ 同步处理大文件
+
+```java
+// ❌ 错：用户上传 100MB 文件 → 同步处理 → 请求超时
+@PostMapping("/import")
+public ApiResponse importOrders(@RequestParam MultipartFile file) {
+    EasyExcel.read(file.getInputStream(), ...).sheet().doRead();  // 同步
+    return ApiResponse.success();
+}
+
+// ✅ 正确：异步处理
+@PostMapping("/import")
+public ApiResponse<String> importOrders(@RequestParam MultipartFile file) {
+    String taskId = importTaskService.submit(file);
+    return ApiResponse.success("导入任务已提交，taskId=" + taskId);
+}
+```
+
+详见 [async-and-validate.md](async-and-validate.md)。
+
+### 6.2 ❌ 用 POI 自己写
+
+```java
+// ❌ 错：POI 复杂、内存占用大
+XSSFWorkbook workbook = new XSSFWorkbook();
+SXSSFSheet sheet = workbook.createSheet("订单");
+for (int i = 0; i < rows.size(); i++) {
+    XSSFRow row = sheet.createRow(i);
+    // ...
+}
+```
+
+✅ 用 EasyExcel，省内存 + 简单。
+
+### 6.3 ❌ 一次性读所有数据到内存
+
+```java
+// ❌ 错：100 万行 → OOM
+List<OrderImportVO> allData = new ArrayList<>();
+EasyExcel.read(inputStream, OrderImportVO.class, new AnalysisEventListener<OrderImportVO>() {
+    @Override
+    public void invoke(OrderImportVO data, AnalysisContext context) {
+        allData.add(data);  // 不分批 → OOM
+    }
+}).sheet().doRead();
+```
+
+✅ 用监听器 + 分批处理（详见上面 §3.2）。
+
+### 6.4 ❌ 不校验导入数据
+
+```java
+// ❌ 错：直接入库
+@Override
+public void invoke(OrderImportVO data, AnalysisContext context) {
+    orderService.save(convertToOrder(data));  // 没校验
+}
+```
+
+✅ 必须校验（字段非空、长度、字典、关联存在性等）。
+
+---
+
+## 7. 关联阅读
+
+- [async-and-validate.md](async-and-validate.md) — 异步处理与校验
+- [scenarios.md](scenarios.md) — 典型场景
+- [SKILL.md](../SKILL.md)