@alanwang-merit/meritdata-platform-skill-package 1.0.10 → 1.0.11

package/skills/meritdata-platform-impexp/references/async-and-validate.md

@@ -1,353 +0,0 @@
-# 异步处理与校验（async-and-validate）
-
-> 大文件导入必须异步处理；导入校验失败必须收集错误行。
-
----
-
-## 1. 异步处理
-
-### 1.1 为什么必须异步
-
-| 文件大小 | 同步处理 | 异步处理 |
-|---------|---------|---------|
-| < 1MB | 可接受 | 推荐 |
-| 1-10MB | 风险（超时）| 必须 |
-| > 10MB | ❌ 必超时 | ✅ 必异步 |
-| > 100MB | ❌ OOM | ✅ 分片 |
-
-### 1.2 实现方案
-
-**方案 A：Spring Async**
-
-```java
-@Configuration
-@EnableAsync
-public class AsyncConfig {
-
-    @Bean("importExecutor")
-    public Executor importExecutor() {
-        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
-        executor.setCorePoolSize(2);
-        executor.setMaxPoolSize(5);
-        executor.setQueueCapacity(50);
-        executor.setThreadNamePrefix("import-");
-        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
-        executor.initialize();
-        return executor;
-    }
-}
-
-@Service
-public class ImportTaskService {
-
-    @Autowired
-    private ImportService importService;
-
-    @Async("importExecutor")
-    public Future<ImportResult> asyncImport(String taskId, MultipartFile file) {
-        ImportResult result = importService.doImport(file);
-        taskRepository.updateResult(taskId, result);
-        return new AsyncResult<>(result);
-    }
-}
-```
-
-**方案 B：消息队列（高可靠场景）**
-
-```
-用户上传 → 写入 MQ → 后台 Worker 消费 → 更新任务状态 → 通知用户
-```
-
-**方案 C：Job（详见 meritdata-platform-job）**
-
-适合定期批量导入（如每天定时导入某个外部文件）。
-
-### 1.3 任务状态查询
-
-```java
-@RestController
-@RequestMapping("/api/v1/import-tasks")
-public class ImportTaskController {
-
-    @Autowired
-    private ImportTaskRepository taskRepository;
-
-    @GetMapping("/{taskId}")
-    public ApiResponse<ImportTaskDTO> getTaskStatus(@PathVariable String taskId) {
-        ImportTask task = taskRepository.findById(taskId).orElseThrow();
-        return ApiResponse.success(convertToDTO(task));
-    }
-}
-```
-
-### 1.4 前端轮询
-
-```vue
-<template>
-  <el-progress :percentage="task.progress" :status="task.status" />
-  <el-button @click="checkStatus">刷新状态</el-button>
-</template>
-
-<script setup>
-import { ref, onMounted } from 'vue';
-
-const task = ref({ progress: 0, status: 'PROCESSING' });
-
-async function checkStatus() {
-  const data = await getImportTask(taskId);
-  task.value = data;
-  if (task.value.status === 'PROCESSING') {
-    setTimeout(checkStatus, 2000);  // 每 2 秒轮询
-  }
-}
-</script>
-```
-
----
-
-## 2. 错误行收集
-
-### 2.1 错误模型
-
-```java
-@Data
-@AllArgsConstructor
-public class ImportError {
-    private Integer rowIndex;      // 行号（从第 2 行开始，数据行）
-    private String field;          // 出错字段
-    private String message;        // 错误消息
-    private String rowData;        // 行原始数据（方便排查）
-}
-
-@Data
-public class ImportResult {
-    private Integer totalCount;
-    private Integer successCount;
-    private Integer errorCount;
-    private List<ImportError> errors = new ArrayList<>();
-}
-```
-
-### 2.2 错误收集器
-
-```java
-public class ImportErrorCollector {
-
-    private final List<ImportError> errors = new ArrayList<>();
-
-    public void addError(int rowIndex, String message) {
-        errors.add(new ImportError(rowIndex, "", message, ""));
-    }
-
-    public void addFieldError(int rowIndex, String field, String message) {
-        errors.add(new ImportError(rowIndex, field, message, ""));
-    }
-
-    public void addRowError(int rowIndex, String rowData, String message) {
-        errors.add(new ImportError(rowIndex, "", message, rowData));
-    }
-
-    public List<ImportError> getErrors() {
-        return errors;
-    }
-
-    public boolean hasErrors() {
-        return !errors.isEmpty();
-    }
-
-    public int getErrorCount() {
-        return errors.size();
-    }
-}
-```
-
-### 2.3 监听器中使用
-
-```java
-@Slf4j
-public class OrderImportListener extends AnalysisEventListener<OrderImportVO> {
-
-    private final ImportErrorCollector errorCollector = new ImportErrorCollector();
-
-    @Override
-    public void invoke(OrderImportVO data, AnalysisContext context) {
-        int rowIndex = context.readRowHolder().getRowIndex() + 1;
-
-        // 字段校验
-        List<String> errors = new ArrayList<>();
-
-        if (StringUtils.isBlank(data.getOrderNo())) {
-            errors.add("订单号不能为空");
-        }
-
-        if (data.getAmount() == null || data.getAmount().compareTo(BigDecimal.ZERO) <= 0) {
-            errors.add("金额必须大于 0");
-        }
-
-        if (!errors.isEmpty()) {
-            String rowData = JSON.toJSONString(data);
-            for (String error : errors) {
-                errorCollector.addRowError(rowIndex, rowData, error);
-            }
-            return;  // 不入库
-        }
-
-        // 通过校验，收集到批量列表
-        cachedData.add(data);
-    }
-}
-```
-
----
-
-## 3. 校验失败的处理策略
-
-### 3.1 模式 A：全部回滚（严格）
-
-任何一行校验失败，整批回滚。
-
-**适用**：财务、合同等高一致性场景。
-
-```java
-@Override
-public ImportResult doImport(MultipartFile file) {
-    ImportResult result = new ImportResult();
-
-    EasyExcel.read(file.getInputStream(), OrderImportVO.class, new OrderImportListener(...)).sheet().doRead();
-
-    if (result.getErrorCount() > 0) {
-        // 不入库
-        return result;
-    }
-
-    // 全部校验通过 → 入库
-    saveBatch(validData);
-    return result;
-}
-```
-
-### 3.2 模式 B：部分导入（容错）
-
-跳过错误行，成功行正常入库。
-
-**适用**：批量初始化、字典导入等容错场景。
-
-```java
-@Override
-public ImportResult doImport(MultipartFile file) {
-    ImportResult result = new ImportResult();
-
-    EasyExcel.read(file.getInputStream(), OrderImportVO.class, new OrderImportListener(...)).sheet().doRead();
-
-    // 部分导入：成功行入库，错误行返回给用户
-    if (!validData.isEmpty()) {
-        saveBatch(validData);
-        result.setSuccessCount(validData.size());
-    }
-    result.setErrorCount(errorCollector.getErrorCount());
-    result.setErrors(errorCollector.getErrors());
-    return result;
-}
-```
-
-### 3.3 错误报告导出
-
-```java
-@GetMapping("/import-tasks/{taskId}/errors")
-public void downloadErrorReport(@PathVariable String taskId, HttpServletResponse response) {
-    ImportTask task = taskRepository.findById(taskId).orElseThrow();
-    List<ImportError> errors = task.getErrors();
-
-    response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
-    response.setHeader("Content-disposition", "attachment;filename=import_errors.xlsx");
-
-    EasyExcel.write(response.getOutputStream(), ImportErrorExportVO.class)
-        .sheet("错误报告")
-        .doWrite(convertToExportVO(errors));
-}
-```
-
----
-
-## 4. 关联校验
-
-### 4.1 场景
-
-导入的"部门名称"必须在平台存在。
-
-### 4.2 批量预加载 + Map 查询
-
-```java
-@Override
-public ImportResult doImport(MultipartFile file) {
-    // 1. 预加载所有部门（避免循环查）
-    List<OrgDTO> allOrgs = orgClient.listAll();
-    Map<String, String> orgCodeToId = allOrgs.stream()
-        .collect(Collectors.toMap(OrgDTO::getCode, OrgDTO::getId));
-
-    // 2. 导入时校验
-    EasyExcel.read(file.getInputStream(), OrderImportVO.class, new OrderImportListener(orgCodeToId, ...))
-        .sheet().doRead();
-}
-```
-
-### 4.3 ❌ 反例：循环单条查
-
-```java
-// ❌ 错：1000 条数据 = 1000 次 Feign
-@Override
-public void invoke(OrderImportVO data, AnalysisContext context) {
-    OrgDTO org = orgClient.getByCode(data.getDeptCode());  // N+1
-    // ...
-}
-```
-
----
-
-## 5. 事务边界
-
-### 5.1 推荐：分批事务
-
-```java
-public class OrderImportListener extends AnalysisEventListener<OrderImportVO> {
-
-    private static final int BATCH_SIZE = 500;
-    private List<OrderImportVO> cachedData = new ArrayList<>(BATCH_SIZE);
-
-    @Override
-    public void invoke(OrderImportVO data, AnalysisContext context) {
-        if (validate(data)) {
-            cachedData.add(data);
-        }
-
-        if (cachedData.size() >= BATCH_SIZE) {
-            saveBatch();  // 每 500 行一个事务
-            cachedData.clear();
-        }
-    }
-
-    @Transactional
-    public void saveBatch() {
-        importService.batchSave(cachedData);
-    }
-}
-```
-
-### 5.2 ❌ 反例：单事务导入全部
-
-```java
-// ❌ 错：100 万行 → 1 个事务 → 锁等待 30 分钟
-@Transactional
-public void doImport(MultipartFile file) {
-    List<OrderImportVO> allData = readAll(file);
-    orderRepository.saveAll(allData);  // 单事务
-}
-```
-
----
-
-## 6. 关联阅读
-
-- [excel-guide.md](excel-guide.md) — EasyExcel 接入
-- [scenarios.md](scenarios.md) — 典型场景
-- [SKILL.md](../SKILL.md)

package/skills/meritdata-platform-impexp/references/excel-guide.md

@@ -1,326 +0,0 @@
-# EasyExcel 导入导出指南（excel-guide）
-
-> 平台使用 EasyExcel 作为 Excel 处理框架。本指南说明模板设计、读写实现。
-
----
-
-## 1. 引入依赖
-
-```xml
-<dependency>
-    <groupId>com.alibaba</groupId>
-    <artifactId>easyexcel</artifactId>
-    <version>3.3.4</version>
-</dependency>
-```
-
----
-
-## 2. 导出 Excel
-
-### 2.1 定义数据模型
-
-```java
-@Data
-@ColumnWidth(20)
-public class OrderExportVO {
-
-    @ExcelProperty("订单号")
-    private String orderNo;
-
-    @ExcelProperty("申请人")
-    private String applicantName;
-
-    @ExcelProperty("金额")
-    private BigDecimal amount;
-
-    @ExcelProperty(value = "状态", converter = StatusConverter.class)
-    private String status;
-
-    @ExcelProperty("创建时间")
-    @DateTimeFormat("yyyy-MM-dd HH:mm:ss")
-    private LocalDateTime createTime;
-}
-```
-
-### 2.2 简单导出
-
-```java
-@GetMapping("/export")
-public void exportOrders(HttpServletResponse response) throws IOException {
-    response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
-    response.setCharacterEncoding("utf-8");
-    String fileName = URLEncoder.encode("采购订单", "UTF-8").replaceAll("\\+", "%20");
-    response.setHeader("Content-disposition", "attachment;filename*=utf-8''" + fileName + ".xlsx");
-
-    // 1. 查数据
-    List<OrderExportVO> data = orderService.listForExport(query);
-
-    // 2. 写 Excel
-    EasyExcel.write(response.getOutputStream(), OrderExportVO.class)
-        .sheet("订单列表")
-        .doWrite(data);
-}
-```
-
-### 2.3 多 Sheet 导出
-
-```java
-ExcelWriter excelWriter = null;
-try {
-    excelWriter = EasyExcel.write(response.getOutputStream()).build();
-
-    WriteSheet sheet1 = EasyExcel.writerSheet(0, "订单列表").head(OrderExportVO.class).build();
-    excelWriter.write(orderList, sheet1);
-
-    WriteSheet sheet2 = EasyExcel.writerSheet(1, "订单明细").head(OrderItemExportVO.class).build();
-    excelWriter.write(itemList, sheet2);
-} finally {
-    if (excelWriter != null) excelWriter.finish();
-}
-```
-
-### 2.4 复杂表头（合并单元格）
-
-```java
-@Data
-public class ComplexOrderExportVO {
-
-    @ExcelProperty({"订单信息", "订单号"})
-    private String orderNo;
-
-    @ExcelProperty({"订单信息", "申请人"})
-    private String applicantName;
-
-    @ExcelProperty({"金额信息", "金额"})
-    private BigDecimal amount;
-
-    @ExcelProperty({"金额信息", "税额"})
-    private BigDecimal taxAmount;
-
-    @ExcelProperty({"时间信息", "创建时间"})
-    private LocalDateTime createTime;
-}
-```
-
----
-
-## 3. 导入 Excel
-
-### 3.1 定义导入模型
-
-```java
-@Data
-public class OrderImportVO {
-
-    @ExcelProperty("订单号")
-    @NotBlank(message = "订单号不能为空")
-    private String orderNo;
-
-    @ExcelProperty("金额")
-    @NotNull(message = "金额不能为空")
-    private BigDecimal amount;
-
-    @ExcelProperty("申请人工号")
-    private String applicantCode;
-}
-```
-
-### 3.2 监听器（处理每行数据）
-
-```java
-@Slf4j
-public class OrderImportListener extends AnalysisEventListener<OrderImportVO> {
-
-    private static final int BATCH_SIZE = 500;
-
-    private List<OrderImportVO> cachedData = new ArrayList<>(BATCH_SIZE);
-
-    private final OrderImportService importService;
-    private final ImportErrorCollector errorCollector;
-
-    @Override
-    public void invoke(OrderImportVO data, AnalysisContext context) {
-        // 1. 行级校验
-        if (data.getAmount() == null || data.getAmount().compareTo(BigDecimal.ZERO) <= 0) {
-            errorCollector.addError(context.readRowHolder().getRowIndex(), "金额必须大于 0");
-            return;
-        }
-
-        cachedData.add(data);
-
-        // 2. 批量处理
-        if (cachedData.size() >= BATCH_SIZE) {
-            importService.batchSave(cachedData);
-            cachedData.clear();
-        }
-    }
-
-    @Override
-    public void doAfterAllAnalysed(AnalysisContext context) {
-        // 处理剩余数据
-        if (!cachedData.isEmpty()) {
-            importService.batchSave(cachedData);
-            cachedData.clear();
-        }
-        log.info("导入完成，共 {} 行", context.readRowHolder().getRowIndex());
-    }
-}
-```
-
-### 3.3 Controller
-
-```java
-@PostMapping("/import")
-public ApiResponse<ImportResult> importOrders(@RequestParam MultipartFile file) {
-    ImportResult result = new ImportResult();
-
-    try {
-        EasyExcel.read(file.getInputStream(), OrderImportVO.class, new OrderImportListener(importService, result.getErrors()))
-            .sheet()
-            .doRead();
-        result.setSuccess(true);
-        return ApiResponse.success(result);
-    } catch (Exception e) {
-        log.error("导入失败", e);
-        return ApiResponse.error("导入失败：" + e.getMessage());
-    }
-}
-```
-
----
-
-## 4. 字典转换器
-
-```java
-public class StatusConverter implements Converter<String> {
-
-    @Override
-    public Class<?> supportJavaTypeKey() {
-        return String.class;
-    }
-
-    @Override
-    public CellDataTypeEnum supportExcelTypeKey() {
-        return CellDataTypeEnum.STRING;
-    }
-
-    @Override
-    public String convertToJavaData(CellData cellData, ExcelContentProperty contentProperty, GlobalConfiguration globalConfiguration) {
-        // Excel 读 → Java 对象
-        String name = cellData.getStringValue();
-        return lookupService.getCodeByName("ORDER_STATUS", name);
-    }
-
-    @Override
-    public CellData<?> convertToExcelData(String value, ExcelContentProperty contentProperty, GlobalConfiguration globalConfiguration) {
-        // Java 对象 → Excel 写
-        String name = lookupService.getName("ORDER_STATUS", value);
-        return new CellData<>(name);
-    }
-}
-```
-
----
-
-## 5. 模板下载
-
-### 5.1 简单模板
-
-```java
-@GetMapping("/template")
-public void downloadTemplate(HttpServletResponse response) throws IOException {
-    response.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
-    response.setCharacterEncoding("utf-8");
-    String fileName = URLEncoder.encode("订单导入模板", "UTF-8");
-    response.setHeader("Content-disposition", "attachment;filename=" + fileName + ".xlsx");
-
-    // 只写表头
-    EasyExcel.write(response.getOutputStream(), OrderImportVO.class)
-        .sheet("订单列表")
-        .doWrite(Collections.emptyList());
-}
-```
-
-### 5.2 带下拉选择的模板
-
-```java
-// 创建带下拉的模板
-EasyExcel.write(response.getOutputStream(), OrderImportVO.class)
-    .registerWriteHandler(new DropdownWriteHandler("状态", Arrays.asList("草稿", "审批中", "已完成")))
-    .sheet("订单列表")
-    .doWrite(Collections.emptyList());
-```
-
----
-
-## 6. ❌ 反模式
-
-### 6.1 ❌ 同步处理大文件
-
-```java
-// ❌ 错：用户上传 100MB 文件 → 同步处理 → 请求超时
-@PostMapping("/import")
-public ApiResponse importOrders(@RequestParam MultipartFile file) {
-    EasyExcel.read(file.getInputStream(), ...).sheet().doRead();  // 同步
-    return ApiResponse.success();
-}
-
-// ✅ 正确：异步处理
-@PostMapping("/import")
-public ApiResponse<String> importOrders(@RequestParam MultipartFile file) {
-    String taskId = importTaskService.submit(file);
-    return ApiResponse.success("导入任务已提交，taskId=" + taskId);
-}
-```
-
-详见 [async-and-validate.md](async-and-validate.md)。
-
-### 6.2 ❌ 用 POI 自己写
-
-```java
-// ❌ 错：POI 复杂、内存占用大
-XSSFWorkbook workbook = new XSSFWorkbook();
-SXSSFSheet sheet = workbook.createSheet("订单");
-for (int i = 0; i < rows.size(); i++) {
-    XSSFRow row = sheet.createRow(i);
-    // ...
-}
-```
-
-✅ 用 EasyExcel，省内存 + 简单。
-
-### 6.3 ❌ 一次性读所有数据到内存
-
-```java
-// ❌ 错：100 万行 → OOM
-List<OrderImportVO> allData = new ArrayList<>();
-EasyExcel.read(inputStream, OrderImportVO.class, new AnalysisEventListener<OrderImportVO>() {
-    @Override
-    public void invoke(OrderImportVO data, AnalysisContext context) {
-        allData.add(data);  // 不分批 → OOM
-    }
-}).sheet().doRead();
-```
-
-✅ 用监听器 + 分批处理（详见上面 §3.2）。
-
-### 6.4 ❌ 不校验导入数据
-
-```java
-// ❌ 错：直接入库
-@Override
-public void invoke(OrderImportVO data, AnalysisContext context) {
-    orderService.save(convertToOrder(data));  // 没校验
-}
-```
-
-✅ 必须校验（字段非空、长度、字典、关联存在性等）。
-
----
-
-## 7. 关联阅读
-
-- [async-and-validate.md](async-and-validate.md) — 异步处理与校验
-- [scenarios.md](scenarios.md) — 典型场景
-- [SKILL.md](../SKILL.md)