@rashidee/co2 1.2.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/index.js +25449 -0
- package/drizzle/0000_remarkable_charles_xavier.sql +210 -0
- package/drizzle/0001_wise_wolf_cub.sql +13 -0
- package/drizzle/0002_secret_spyke.sql +16 -0
- package/drizzle/0003_workspace_config_singleton.sql +16 -0
- package/drizzle/0004_terminal_claim_v120.sql +11 -0
- package/drizzle/0005_brainstorm_v124.sql +10 -0
- package/drizzle/0006_project_v140.sql +6 -0
- package/drizzle/0007_prd_wizard_v124.sql +14 -0
- package/drizzle/0008_workflow_artifact_v126.sql +24 -0
- package/drizzle/0009_project_import_v150.sql +7 -0
- package/drizzle/0010_quality_v151.sql +3 -0
- package/drizzle/meta/0000_snapshot.json +1335 -0
- package/drizzle/meta/0001_snapshot.json +1416 -0
- package/drizzle/meta/0002_snapshot.json +1510 -0
- package/drizzle/meta/_journal.json +83 -0
- package/package.json +41 -0
- package/plugin/.claude-plugin/marketplace.json +36 -0
- package/plugin/.claude-plugin/plugin.json +22 -0
- package/plugin/README.md +625 -0
- package/plugin/SKILLS.md +350 -0
- package/plugin/co2.svg +5 -0
- package/plugin/commands/brainstorm-loop.md +129 -0
- package/plugin/skills/conductor-defect/SKILL.md +898 -0
- package/plugin/skills/conductor-feature-develop/SKILL.md +1383 -0
- package/plugin/skills/conductor-feature-develop/references/implementation-domain-template.md +131 -0
- package/plugin/skills/conductor-feature-develop/references/implementation-master-template.md +89 -0
- package/plugin/skills/conductor-feature-develop/references/playwright-setup.md +224 -0
- package/plugin/skills/conductor-feature-prepare/SKILL.md +604 -0
- package/plugin/skills/depgen-k8s/SKILL.md +468 -0
- package/plugin/skills/depgen-k8s/references/dockerfile-laravel.md +197 -0
- package/plugin/skills/depgen-k8s/references/dockerfile-nodejs.md +163 -0
- package/plugin/skills/depgen-k8s/references/dockerfile-spring-boot.md +174 -0
- package/plugin/skills/depgen-k8s/references/k8s-patterns.md +869 -0
- package/plugin/skills/mockgen-shadcn/SKILL.md +1067 -0
- package/plugin/skills/mockgen-shadcn/references/admin-layout-template.md +1467 -0
- package/plugin/skills/mockgen-shadcn/references/mockup-hub-template.md +498 -0
- package/plugin/skills/mockgen-shadcn/references/mockup-index-template.md +216 -0
- package/plugin/skills/mockgen-tailwind/SKILL.md +904 -0
- package/plugin/skills/mockgen-tailwind/references/admin-layout-template.md +720 -0
- package/plugin/skills/mockgen-tailwind/references/mockup-hub-template.md +498 -0
- package/plugin/skills/mockgen-tailwind/references/mockup-index-template.md +190 -0
- package/plugin/skills/modelgen-nosql/SKILL.md +617 -0
- package/plugin/skills/modelgen-nosql/references/model-extraction-methodology.md +497 -0
- package/plugin/skills/modelgen-nosql/references/nosql-design-guide.md +258 -0
- package/plugin/skills/modelgen-relational/SKILL.md +707 -0
- package/plugin/skills/modelgen-relational/references/model-extraction-methodology.md +499 -0
- package/plugin/skills/specgen-flutter-riverpod/SKILL.md +898 -0
- package/plugin/skills/specgen-flutter-riverpod/references/component-patterns.md +971 -0
- package/plugin/skills/specgen-flutter-riverpod/references/notification-patterns.md +457 -0
- package/plugin/skills/specgen-flutter-riverpod/references/routing-patterns.md +362 -0
- package/plugin/skills/specgen-flutter-riverpod/references/security-patterns.md +709 -0
- package/plugin/skills/specgen-flutter-riverpod/references/spec-template.md +1566 -0
- package/plugin/skills/specgen-flutter-riverpod/references/state-patterns.md +518 -0
- package/plugin/skills/specgen-flutter-riverpod/references/storage-patterns.md +370 -0
- package/plugin/skills/specgen-laravel-eloquent-bladehtmx/SKILL.md +984 -0
- package/plugin/skills/specgen-laravel-eloquent-bladehtmx/references/batch-patterns.md +359 -0
- package/plugin/skills/specgen-laravel-eloquent-bladehtmx/references/messaging-patterns.md +468 -0
- package/plugin/skills/specgen-laravel-eloquent-bladehtmx/references/modulith-patterns.md +461 -0
- package/plugin/skills/specgen-laravel-eloquent-bladehtmx/references/reporting-patterns.md +803 -0
- package/plugin/skills/specgen-laravel-eloquent-bladehtmx/references/security-patterns.md +501 -0
- package/plugin/skills/specgen-laravel-eloquent-bladehtmx/references/spec-template.md +1268 -0
- package/plugin/skills/specgen-laravel-eloquent-bladehtmx/references/web-patterns.md +301 -0
- package/plugin/skills/specgen-node-cli-web/SKILL.md +561 -0
- package/plugin/skills/specgen-node-cli-web/references/auth-patterns.md +293 -0
- package/plugin/skills/specgen-node-cli-web/references/cli-packaging-patterns.md +250 -0
- package/plugin/skills/specgen-node-cli-web/references/server-patterns.md +247 -0
- package/plugin/skills/specgen-node-cli-web/references/spec-template.md +401 -0
- package/plugin/skills/specgen-node-cli-web/references/testing-patterns.md +174 -0
- package/plugin/skills/specgen-react-mui/SKILL.md +789 -0
- package/plugin/skills/specgen-react-mui/references/component-patterns.md +1077 -0
- package/plugin/skills/specgen-react-mui/references/reporting-patterns.md +1998 -0
- package/plugin/skills/specgen-react-mui/references/routing-patterns.md +461 -0
- package/plugin/skills/specgen-react-mui/references/security-patterns.md +599 -0
- package/plugin/skills/specgen-react-mui/references/spec-template.md +1030 -0
- package/plugin/skills/specgen-react-mui/references/state-patterns.md +327 -0
- package/plugin/skills/specgen-react-tailwind/SKILL.md +859 -0
- package/plugin/skills/specgen-react-tailwind/references/component-patterns.md +1212 -0
- package/plugin/skills/specgen-react-tailwind/references/reporting-patterns.md +1529 -0
- package/plugin/skills/specgen-react-tailwind/references/routing-patterns.md +669 -0
- package/plugin/skills/specgen-react-tailwind/references/security-patterns.md +656 -0
- package/plugin/skills/specgen-react-tailwind/references/spec-template.md +1237 -0
- package/plugin/skills/specgen-react-tailwind/references/state-patterns.md +327 -0
- package/plugin/skills/specgen-sdk-java/SKILL.md +933 -0
- package/plugin/skills/specgen-sdk-java/references/api-extraction-patterns.md +247 -0
- package/plugin/skills/specgen-sdk-java/references/http-client-patterns.md +495 -0
- package/plugin/skills/specgen-sdk-java/references/packaging-patterns.md +366 -0
- package/plugin/skills/specgen-sdk-java/references/spec-template.md +956 -0
- package/plugin/skills/specgen-spring-jpa-jtehtmx/SKILL.md +1284 -0
- package/plugin/skills/specgen-spring-jpa-jtehtmx/references/batch-patterns.md +1224 -0
- package/plugin/skills/specgen-spring-jpa-jtehtmx/references/jasper-patterns.md +1638 -0
- package/plugin/skills/specgen-spring-jpa-jtehtmx/references/messaging-patterns.md +512 -0
- package/plugin/skills/specgen-spring-jpa-jtehtmx/references/modulith-patterns.md +354 -0
- package/plugin/skills/specgen-spring-jpa-jtehtmx/references/security-patterns.md +551 -0
- package/plugin/skills/specgen-spring-jpa-jtehtmx/references/spec-template.md +2502 -0
- package/plugin/skills/specgen-spring-jpa-jtehtmx/references/web-patterns.md +503 -0
- package/plugin/skills/specgen-spring-jpa-restapi/SKILL.md +990 -0
- package/plugin/skills/specgen-spring-jpa-restapi/references/batch-patterns.md +1224 -0
- package/plugin/skills/specgen-spring-jpa-restapi/references/messaging-patterns.md +512 -0
- package/plugin/skills/specgen-spring-jpa-restapi/references/modulith-patterns.md +279 -0
- package/plugin/skills/specgen-spring-jpa-restapi/references/reporting-patterns.md +1536 -0
- package/plugin/skills/specgen-spring-jpa-restapi/references/restapi-patterns.md +722 -0
- package/plugin/skills/specgen-spring-jpa-restapi/references/security-patterns.md +351 -0
- package/plugin/skills/specgen-spring-jpa-restapi/references/spec-template.md +553 -0
- package/plugin/skills/specgen-ts-cli/SKILL.md +861 -0
- package/plugin/skills/specgen-ts-cli/references/async-patterns.md +1101 -0
- package/plugin/skills/specgen-ts-cli/references/command-patterns.md +383 -0
- package/plugin/skills/specgen-ts-cli/references/config-patterns.md +430 -0
- package/plugin/skills/specgen-ts-cli/references/database-patterns.md +750 -0
- package/plugin/skills/specgen-ts-cli/references/packaging-patterns.md +426 -0
- package/plugin/skills/specgen-ts-cli/references/spec-template.md +1391 -0
- package/plugin/skills/specgen-ts-cli/references/testing-patterns.md +909 -0
- package/plugin/skills/testgen-functional/SKILL.md +1381 -0
- package/plugin/skills/tracegen-matrix/SKILL.md +243 -0
- package/plugin/skills/util-preparek8senv/SKILL.md +422 -0
- package/plugin/skills/util-projectinit/SKILL.md +307 -0
- package/plugin/skills/util-projectinit/reference/brainstorm-loop.md +129 -0
- package/plugin/skills/util-projectinit/reference/brainstorm-protocol.md +144 -0
- package/plugin/skills/util-projectinit/reference/brainstorm-setup-guide.md +153 -0
- package/plugin/skills/util-projectinit/reference/brainstorm-template-appdev.md +87 -0
- package/plugin/skills/util-projectinit/reference/co2-context-generation-guide.md +761 -0
- package/plugin/skills/util-projectsync/SKILL.md +501 -0
- package/plugin/skills/util-updprd/SKILL.md +324 -0
- package/plugin/skills/util-usanalyzer/SKILL.md +307 -0
- package/plugin/skills/util-ustagger/SKILL.md +154 -0
- package/static/assets/abnfDiagram-VRR7QNED-DvPuwyCV.js +1 -0
- package/static/assets/arc-DRKw7sjs.js +1 -0
- package/static/assets/architectureDiagram-ZJ3FMSHR-CaNcxUbC.js +36 -0
- package/static/assets/blockDiagram-677ZJIJ3-Demd4AOK.js +132 -0
- package/static/assets/c4Diagram-LMCZKHZV-PVKUfc2x.js +10 -0
- package/static/assets/channel-C2nVbuwv.js +1 -0
- package/static/assets/chunk-2Q5K7J3B-CExf8ww_.js +1 -0
- package/static/assets/chunk-32BRIVSS-CFPaoONF.js +1 -0
- package/static/assets/chunk-5VM5RSS4-DE2esM67.js +15 -0
- package/static/assets/chunk-EX3LRPZG-SON_YdDG.js +231 -0
- package/static/assets/chunk-JWPE2WC7-Qxah8LNr.js +1 -0
- package/static/assets/chunk-MOJQB5TN-BcWGQV3j.js +88 -0
- package/static/assets/chunk-RYQCIY6F-DkDj-OR5.js +1 -0
- package/static/assets/chunk-V7JOEXUC-BlI1Mrah.js +206 -0
- package/static/assets/chunk-VR4S4FIN-Agf__duj.js +1 -0
- package/static/assets/chunk-XXDRQBXY-B5CfAh85.js +1 -0
- package/static/assets/classDiagram-OUVF2IWQ-DXJIljsR.js +1 -0
- package/static/assets/classDiagram-v2-EOCWNBFH-DXJIljsR.js +1 -0
- package/static/assets/cose-bilkent-JH36ORCC-DO0cQkcy.js +1 -0
- package/static/assets/cynefin-VYW2F7L2-CfOnsnXc.js +178 -0
- package/static/assets/cynefinDiagram-TSTJHNR4-BJ8tULAv.js +62 -0
- package/static/assets/cytoscape.esm-DTSO7Bv0.js +331 -0
- package/static/assets/dagre-VKFMJZFB-DfjwzHqP.js +4 -0
- package/static/assets/defaultLocale-DX6XiGOO.js +1 -0
- package/static/assets/diagram-FQU43EPY-CndSOIIM.js +3 -0
- package/static/assets/diagram-G47NLZAW-Czf5hN0Z.js +24 -0
- package/static/assets/diagram-NH7WQ7WH-COAyojGh.js +24 -0
- package/static/assets/diagram-OA4YK3LP-H8PoiN2m.js +30 -0
- package/static/assets/diagram-WEI45ONY-DzcaVzjp.js +41 -0
- package/static/assets/ebnfDiagram-CCIWWBDH-DPazGYLJ.js +1 -0
- package/static/assets/erDiagram-Q63AITRT-Bo-fubgF.js +85 -0
- package/static/assets/flowDiagram-23GEKE2U-CSvxUDwh.js +156 -0
- package/static/assets/ganttDiagram-NO4QXBWP-DnBveWX6.js +292 -0
- package/static/assets/gitGraphDiagram-IHSO6WYX-B3rVL8c-.js +106 -0
- package/static/assets/graph-C9eacEi8.js +1 -0
- package/static/assets/index-LZSQkLE2.css +41 -0
- package/static/assets/index-y7XDSgMG.js +495 -0
- package/static/assets/infoDiagram-FWYZ7A6U-CvWV9yy2.js +2 -0
- package/static/assets/init-Gi6I4Gst.js +1 -0
- package/static/assets/ishikawaDiagram-FXEZZL3T-DF28a1N6.js +70 -0
- package/static/assets/journeyDiagram-5HDEW3XC-CB6CugYk.js +139 -0
- package/static/assets/kanban-definition-HUTT4EX6-ResSLF6d.js +89 -0
- package/static/assets/katex-C5jXJg4s.js +257 -0
- package/static/assets/layout-DEXfKzaS.js +1 -0
- package/static/assets/linear-CLO--ryk.js +1 -0
- package/static/assets/map-Czzmt4hB.js +1 -0
- package/static/assets/mindmap-definition-LN4V7U3C-BvW3EbxX.js +96 -0
- package/static/assets/ordinal-Cboi1Yqb.js +1 -0
- package/static/assets/pegDiagram-2B236MQR-BHCgqrrM.js +1 -0
- package/static/assets/pieDiagram-ENE6RG2P-DVmCQ5FA.js +39 -0
- package/static/assets/quadrantDiagram-ABIIQ3AL-B3e5KR52.js +7 -0
- package/static/assets/railroadDiagram-RFXS5EU6-Csv5ysLc.js +1 -0
- package/static/assets/requirementDiagram-TGXJPOKE-DOMVzvnB.js +84 -0
- package/static/assets/sankeyDiagram-HTMAVEWB-B1dcTOvy.js +40 -0
- package/static/assets/sequenceDiagram-DBY2YBRQ-BY414yeO.js +162 -0
- package/static/assets/sizeCapture-X5ZJPWSS-CuJfrfBN.js +1 -0
- package/static/assets/stateDiagram-2N3HPSRC-FX4rJkzV.js +1 -0
- package/static/assets/stateDiagram-v2-6OUMAXLB-BVufbO_H.js +1 -0
- package/static/assets/swimlanes-5IMT3BWC-Sfy58mmv.js +2 -0
- package/static/assets/swimlanesDiagram-G3AALYLV-wDCDw08H.js +8 -0
- package/static/assets/timeline-definition-FHXFAJF6-CmV7by8p.js +120 -0
- package/static/assets/vennDiagram-L72KCM5P-FqIyFZFt.js +34 -0
- package/static/assets/wardleyDiagram-EHGQE667-BZ06O45i.js +78 -0
- package/static/assets/xychartDiagram-FW5EYKEG-BOkffR06.js +7 -0
- package/static/index.html +24 -0
|
@@ -0,0 +1,1224 @@
|
|
|
1
|
+
# Scheduling Patterns — Quartz Scheduling + Optional Spring Batch Processing
|
|
2
|
+
|
|
3
|
+
This reference describes the scheduling and batch processing scaffolding for the spec.
|
|
4
|
+
Include this content in Section 18 of the generated specification. All code samples must
|
|
5
|
+
be reproduced in the generated spec with full import statements and constructor injection.
|
|
6
|
+
|
|
7
|
+
The scheduling feature has two tiers:
|
|
8
|
+
- **Scheduling = yes**: Quartz scheduler for cron-triggered jobs (always included)
|
|
9
|
+
- **Scheduling = yes AND Spring Batch = yes**: Adds Spring Batch reader/processor/writer
|
|
10
|
+
pipeline for chunk-oriented data processing, triggered by Quartz
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Architecture Overview
|
|
15
|
+
|
|
16
|
+
### Quartz-Only Mode (Scheduling = yes, Spring Batch = no)
|
|
17
|
+
|
|
18
|
+
When Spring Batch is not selected, Quartz runs standalone. Jobs extend `QuartzJobBean`
|
|
19
|
+
and execute application logic directly (service calls, API calls, cleanup tasks, etc.)
|
|
20
|
+
without the Batch reader/processor/writer pipeline.
|
|
21
|
+
|
|
22
|
+
```
|
|
23
|
+
Quartz Scheduler (job store matches selected database)
|
|
24
|
+
└── Trigger (cron: "0 0 2 * * ?")
|
|
25
|
+
└── QuartzJobBean
|
|
26
|
+
└── executes business logic directly (service calls, etc.)
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
### Quartz + Spring Batch Mode (Scheduling = yes, Spring Batch = yes)
|
|
30
|
+
|
|
31
|
+
When Spring Batch is selected, Quartz triggers fire Spring Batch jobs. This gives the
|
|
32
|
+
application cron-based scheduling with robust, restartable, chunk-oriented processing.
|
|
33
|
+
|
|
34
|
+
- **Spring Quartz**: Manages job scheduling — cron triggers, intervals, fire-and-forget.
|
|
35
|
+
Quartz decides WHEN jobs run.
|
|
36
|
+
- **Spring Batch**: Manages job execution — reading, processing, writing data in chunks.
|
|
37
|
+
Batch decides HOW jobs run.
|
|
38
|
+
|
|
39
|
+
```
|
|
40
|
+
Quartz Scheduler (job store matches selected database)
|
|
41
|
+
└── Trigger (cron: "0 0 2 * * ?")
|
|
42
|
+
└── QuartzJobBean
|
|
43
|
+
└── launches Spring Batch Job
|
|
44
|
+
└── Step 1: Reader → Processor → Writer
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Maven Dependency
|
|
50
|
+
|
|
51
|
+
Include in `pom.xml`:
|
|
52
|
+
|
|
53
|
+
**[If Database = MongoDB]:**
|
|
54
|
+
```xml
|
|
55
|
+
<!-- Quartz MongoDB Job Store -->
|
|
56
|
+
<dependency>
|
|
57
|
+
<groupId>io.fluidsonic.mirror</groupId>
|
|
58
|
+
<artifactId>fluidsonic-mirror-quartz</artifactId>
|
|
59
|
+
<version>${fluidsonic-mirror.version}</version>
|
|
60
|
+
</dependency>
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## Quartz Configuration
|
|
66
|
+
|
|
67
|
+
### application.yml Quartz section
|
|
68
|
+
|
|
69
|
+
```yaml
|
|
70
|
+
spring:
|
|
71
|
+
quartz:
|
|
72
|
+
job-store-type: mongodb
|
|
73
|
+
properties:
|
|
74
|
+
org.quartz.threadPool.threadCount: 5
|
|
75
|
+
org.quartz.scheduler.instanceName: {{ARTIFACT_ID}}-scheduler
|
|
76
|
+
org.quartz.scheduler.instanceId: AUTO
|
|
77
|
+
org.quartz.jobStore.class: io.fluidsonic.mirror.quartz.MongoDBJobStore
|
|
78
|
+
org.quartz.jobStore.mongoUri: mongodb://localhost:27017/{{DB_NAME}}
|
|
79
|
+
org.quartz.jobStore.dbName: {{DB_NAME}}
|
|
80
|
+
org.quartz.jobStore.collectionPrefix: qrtz_
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
The `fluidsonic-mirror` MongoDB job store persists job details, triggers, and scheduler
|
|
84
|
+
state directly in MongoDB collections prefixed with `qrtz_`. This eliminates the need
|
|
85
|
+
for a relational database and supports clustered Quartz instances with `instanceId: AUTO`.
|
|
86
|
+
|
|
87
|
+
Adapt the `job-store-type` and properties based on the selected database:
|
|
88
|
+
- **MongoDB**: Use `mongodb` job store type with `fluidsonic-mirror-quartz` (as shown above)
|
|
89
|
+
- **PostgreSQL/MySQL**: Use `jdbc` job store type with built-in Quartz JDBC support
|
|
90
|
+
- **none**: Use `memory` job store type (no persistence, lost on restart)
|
|
91
|
+
|
|
92
|
+
### QuartzConfig.java
|
|
93
|
+
|
|
94
|
+
```java
|
|
95
|
+
package {{BASE_PACKAGE}}.config;
|
|
96
|
+
|
|
97
|
+
import {{BASE_PACKAGE}}.scheduling.internal.job.SampleQuartzJob;
|
|
98
|
+
import org.quartz.CronScheduleBuilder;
|
|
99
|
+
import org.quartz.JobBuilder;
|
|
100
|
+
import org.quartz.JobDetail;
|
|
101
|
+
import org.quartz.Trigger;
|
|
102
|
+
import org.quartz.TriggerBuilder;
|
|
103
|
+
import org.springframework.context.annotation.Bean;
|
|
104
|
+
import org.springframework.context.annotation.Configuration;
|
|
105
|
+
|
|
106
|
+
@Configuration
|
|
107
|
+
public class QuartzConfig {
|
|
108
|
+
|
|
109
|
+
@Bean
|
|
110
|
+
public JobDetail sampleJobDetail() {
|
|
111
|
+
return JobBuilder.newJob(SampleQuartzJob.class)
|
|
112
|
+
.withIdentity("sampleJob", "default-group")
|
|
113
|
+
.withDescription("Sample scheduled job that triggers a Spring Batch job")
|
|
114
|
+
.storeDurably()
|
|
115
|
+
.build();
|
|
116
|
+
}
|
|
117
|
+
|
|
118
|
+
@Bean
|
|
119
|
+
public Trigger sampleJobTrigger(JobDetail sampleJobDetail) {
|
|
120
|
+
return TriggerBuilder.newTrigger()
|
|
121
|
+
.forJob(sampleJobDetail)
|
|
122
|
+
.withIdentity("sampleTrigger", "default-group")
|
|
123
|
+
.withDescription("Fires daily at 2 AM")
|
|
124
|
+
.withSchedule(CronScheduleBuilder.cronSchedule("0 0 2 * * ?"))
|
|
125
|
+
.build();
|
|
126
|
+
}
|
|
127
|
+
}
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
## Quartz Job Class (Quartz-Only — Spring Batch = no)
|
|
133
|
+
|
|
134
|
+
When Spring Batch is not selected, Quartz jobs execute business logic directly via
|
|
135
|
+
service calls. This is the default when `Scheduling = yes` but `Spring Batch = no`.
|
|
136
|
+
|
|
137
|
+
### SampleQuartzJob.java (Quartz-only)
|
|
138
|
+
|
|
139
|
+
```java
|
|
140
|
+
package {{BASE_PACKAGE}}.scheduling.internal.job;
|
|
141
|
+
|
|
142
|
+
import lombok.RequiredArgsConstructor;
|
|
143
|
+
import lombok.extern.slf4j.Slf4j;
|
|
144
|
+
import org.quartz.JobExecutionContext;
|
|
145
|
+
import org.quartz.JobExecutionException;
|
|
146
|
+
import org.springframework.scheduling.quartz.QuartzJobBean;
|
|
147
|
+
import org.springframework.stereotype.Component;
|
|
148
|
+
|
|
149
|
+
@Component
|
|
150
|
+
@RequiredArgsConstructor
|
|
151
|
+
@Slf4j
|
|
152
|
+
public class SampleQuartzJob extends QuartzJobBean {
|
|
153
|
+
|
|
154
|
+
// Inject your application services here
|
|
155
|
+
// private final SomeService someService;
|
|
156
|
+
|
|
157
|
+
@Override
|
|
158
|
+
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
|
|
159
|
+
try {
|
|
160
|
+
log.info("Executing scheduled job: sampleJob");
|
|
161
|
+
|
|
162
|
+
// Execute business logic directly — no Spring Batch pipeline
|
|
163
|
+
// someService.performScheduledTask();
|
|
164
|
+
|
|
165
|
+
log.info("Scheduled job sampleJob completed successfully");
|
|
166
|
+
} catch (Exception e) {
|
|
167
|
+
log.error("Failed to execute scheduled job: sampleJob", e);
|
|
168
|
+
throw new JobExecutionException("Scheduled job execution failed", e);
|
|
169
|
+
}
|
|
170
|
+
}
|
|
171
|
+
}
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
---
|
|
175
|
+
|
|
176
|
+
## Quartz Job Class (Spring Batch = yes)
|
|
177
|
+
|
|
178
|
+
When Spring Batch is selected, the Quartz job launches a Spring Batch job instead of
|
|
179
|
+
executing logic directly. Include this version **only when Spring Batch = yes**.
|
|
180
|
+
|
|
181
|
+
### SampleQuartzJob.java (Batch launcher)
|
|
182
|
+
|
|
183
|
+
```java
|
|
184
|
+
package {{BASE_PACKAGE}}.scheduling.internal.job;
|
|
185
|
+
|
|
186
|
+
import lombok.RequiredArgsConstructor;
|
|
187
|
+
import lombok.extern.slf4j.Slf4j;
|
|
188
|
+
import org.quartz.JobExecutionContext;
|
|
189
|
+
import org.quartz.JobExecutionException;
|
|
190
|
+
import org.springframework.batch.core.Job;
|
|
191
|
+
import org.springframework.batch.core.JobParameters;
|
|
192
|
+
import org.springframework.batch.core.JobParametersBuilder;
|
|
193
|
+
import org.springframework.batch.core.launch.JobLauncher;
|
|
194
|
+
import org.springframework.scheduling.quartz.QuartzJobBean;
|
|
195
|
+
import org.springframework.stereotype.Component;
|
|
196
|
+
|
|
197
|
+
@Component
|
|
198
|
+
@RequiredArgsConstructor
|
|
199
|
+
@Slf4j
|
|
200
|
+
public class SampleQuartzJob extends QuartzJobBean {
|
|
201
|
+
|
|
202
|
+
private final JobLauncher jobLauncher;
|
|
203
|
+
private final Job sampleBatchJob;
|
|
204
|
+
|
|
205
|
+
@Override
|
|
206
|
+
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
|
|
207
|
+
try {
|
|
208
|
+
JobParameters params = new JobParametersBuilder()
|
|
209
|
+
.addLong("timestamp", System.currentTimeMillis())
|
|
210
|
+
.toJobParameters();
|
|
211
|
+
|
|
212
|
+
log.info("Launching batch job: sampleBatchJob at {}", params);
|
|
213
|
+
jobLauncher.run(sampleBatchJob, params);
|
|
214
|
+
log.info("Batch job sampleBatchJob completed successfully");
|
|
215
|
+
} catch (Exception e) {
|
|
216
|
+
log.error("Failed to launch batch job: sampleBatchJob", e);
|
|
217
|
+
throw new JobExecutionException("Batch job launch failed", e);
|
|
218
|
+
}
|
|
219
|
+
}
|
|
220
|
+
}
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
The `timestamp` parameter ensures each execution is unique (Spring Batch requires
|
|
224
|
+
unique parameters per run by default).
|
|
225
|
+
|
|
226
|
+
---
|
|
227
|
+
|
|
228
|
+
## Spring Batch Configuration [Include only if Spring Batch = yes]
|
|
229
|
+
|
|
230
|
+
### SampleBatchConfig.java
|
|
231
|
+
|
|
232
|
+
```java
|
|
233
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch;
|
|
234
|
+
|
|
235
|
+
import lombok.RequiredArgsConstructor;
|
|
236
|
+
import org.springframework.batch.core.Job;
|
|
237
|
+
import org.springframework.batch.core.Step;
|
|
238
|
+
import org.springframework.batch.core.job.builder.JobBuilder;
|
|
239
|
+
import org.springframework.batch.core.repository.JobRepository;
|
|
240
|
+
import org.springframework.batch.core.step.builder.StepBuilder;
|
|
241
|
+
import org.springframework.batch.item.ItemProcessor;
|
|
242
|
+
import org.springframework.batch.item.ItemReader;
|
|
243
|
+
import org.springframework.batch.item.ItemWriter;
|
|
244
|
+
import org.springframework.context.annotation.Bean;
|
|
245
|
+
import org.springframework.context.annotation.Configuration;
|
|
246
|
+
import org.springframework.transaction.PlatformTransactionManager;
|
|
247
|
+
|
|
248
|
+
@Configuration
|
|
249
|
+
@RequiredArgsConstructor
|
|
250
|
+
public class SampleBatchConfig {
|
|
251
|
+
|
|
252
|
+
private final JobRepository jobRepository;
|
|
253
|
+
private final PlatformTransactionManager transactionManager;
|
|
254
|
+
|
|
255
|
+
@Bean
|
|
256
|
+
public Job sampleBatchJob(Step sampleStep) {
|
|
257
|
+
return new JobBuilder("sampleBatchJob", jobRepository)
|
|
258
|
+
.start(sampleStep)
|
|
259
|
+
.build();
|
|
260
|
+
}
|
|
261
|
+
|
|
262
|
+
@Bean
|
|
263
|
+
public Step sampleStep(
|
|
264
|
+
ItemReader<SampleSourceDocument> reader,
|
|
265
|
+
ItemProcessor<SampleSourceDocument, SampleTargetDocument> processor,
|
|
266
|
+
ItemWriter<SampleTargetDocument> writer) {
|
|
267
|
+
return new StepBuilder("sampleStep", jobRepository)
|
|
268
|
+
.<SampleSourceDocument, SampleTargetDocument>chunk(100, transactionManager)
|
|
269
|
+
.reader(reader)
|
|
270
|
+
.processor(processor)
|
|
271
|
+
.writer(writer)
|
|
272
|
+
.faultTolerant()
|
|
273
|
+
.skipLimit(10)
|
|
274
|
+
.skip(Exception.class)
|
|
275
|
+
.build();
|
|
276
|
+
}
|
|
277
|
+
}
|
|
278
|
+
```
|
|
279
|
+
|
|
280
|
+
Key configuration points:
|
|
281
|
+
- **Chunk size**: 100 items per transaction commit (tunable)
|
|
282
|
+
- **Fault tolerance**: Skip up to 10 failures before aborting
|
|
283
|
+
- **Transaction manager**: Each chunk is a single transaction
|
|
284
|
+
|
|
285
|
+
### Batch YAML configuration [Include only if Spring Batch = yes]
|
|
286
|
+
|
|
287
|
+
```yaml
|
|
288
|
+
spring:
|
|
289
|
+
batch:
|
|
290
|
+
jdbc:
|
|
291
|
+
initialize-schema: never # MongoDB-backed app, no JDBC schema needed
|
|
292
|
+
job:
|
|
293
|
+
enabled: false # Prevent auto-run on startup; Quartz triggers jobs
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
---
|
|
297
|
+
|
|
298
|
+
## ItemReader — MongoDB Source [Include only if Spring Batch = yes]
|
|
299
|
+
|
|
300
|
+
### SampleItemReader.java
|
|
301
|
+
|
|
302
|
+
```java
|
|
303
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch;
|
|
304
|
+
|
|
305
|
+
import lombok.RequiredArgsConstructor;
|
|
306
|
+
import org.springframework.batch.core.configuration.annotation.StepScope;
|
|
307
|
+
import org.springframework.batch.item.data.MongoPagingItemReader;
|
|
308
|
+
import org.springframework.batch.item.data.builder.MongoPagingItemReaderBuilder;
|
|
309
|
+
import org.springframework.context.annotation.Bean;
|
|
310
|
+
import org.springframework.context.annotation.Configuration;
|
|
311
|
+
import org.springframework.data.domain.Sort;
|
|
312
|
+
import org.springframework.data.mongodb.core.MongoTemplate;
|
|
313
|
+
import java.util.Map;
|
|
314
|
+
|
|
315
|
+
@Configuration
|
|
316
|
+
@RequiredArgsConstructor
|
|
317
|
+
public class SampleItemReaderConfig {
|
|
318
|
+
|
|
319
|
+
private final MongoTemplate mongoTemplate;
|
|
320
|
+
|
|
321
|
+
@StepScope
|
|
322
|
+
@Bean
|
|
323
|
+
public MongoPagingItemReader<SampleSourceDocument> sampleReader() {
|
|
324
|
+
return new MongoPagingItemReaderBuilder<SampleSourceDocument>()
|
|
325
|
+
.name("sampleReader")
|
|
326
|
+
.template(mongoTemplate)
|
|
327
|
+
.targetType(SampleSourceDocument.class)
|
|
328
|
+
.jsonQuery("{}")
|
|
329
|
+
.sorts(Map.of("_id", Sort.Direction.ASC))
|
|
330
|
+
.pageSize(100)
|
|
331
|
+
.build();
|
|
332
|
+
}
|
|
333
|
+
}
|
|
334
|
+
```
|
|
335
|
+
|
|
336
|
+
`@StepScope` makes the reader step-scoped, allowing late binding of parameters
|
|
337
|
+
(e.g., date ranges passed from Quartz job parameters).
|
|
338
|
+
|
|
339
|
+
---
|
|
340
|
+
|
|
341
|
+
## ItemProcessor — Transformation [Include only if Spring Batch = yes]
|
|
342
|
+
|
|
343
|
+
### SampleItemProcessor.java
|
|
344
|
+
|
|
345
|
+
```java
|
|
346
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch;
|
|
347
|
+
|
|
348
|
+
import lombok.extern.slf4j.Slf4j;
|
|
349
|
+
import org.springframework.batch.item.ItemProcessor;
|
|
350
|
+
import org.springframework.stereotype.Component;
|
|
351
|
+
|
|
352
|
+
@Component
|
|
353
|
+
@Slf4j
|
|
354
|
+
public class SampleItemProcessor implements ItemProcessor<SampleSourceDocument, SampleTargetDocument> {
|
|
355
|
+
|
|
356
|
+
@Override
|
|
357
|
+
public SampleTargetDocument process(SampleSourceDocument item) {
|
|
358
|
+
log.debug("Processing item: {}", item.getId());
|
|
359
|
+
|
|
360
|
+
// Return null to skip/filter this item
|
|
361
|
+
if (item.getFieldOne() == null) {
|
|
362
|
+
log.warn("Skipping item with null fieldOne: {}", item.getId());
|
|
363
|
+
return null;
|
|
364
|
+
}
|
|
365
|
+
|
|
366
|
+
return SampleTargetDocument.builder()
|
|
367
|
+
.sourceId(item.getId())
|
|
368
|
+
.transformedField(item.getFieldOne().toUpperCase())
|
|
369
|
+
.processedAt(java.time.Instant.now())
|
|
370
|
+
.build();
|
|
371
|
+
}
|
|
372
|
+
}
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
---
|
|
376
|
+
|
|
377
|
+
## ItemWriter — MongoDB Sink [Include only if Spring Batch = yes]
|
|
378
|
+
|
|
379
|
+
### SampleItemWriter.java
|
|
380
|
+
|
|
381
|
+
```java
|
|
382
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch;
|
|
383
|
+
|
|
384
|
+
import lombok.RequiredArgsConstructor;
|
|
385
|
+
import lombok.extern.slf4j.Slf4j;
|
|
386
|
+
import org.springframework.batch.item.Chunk;
|
|
387
|
+
import org.springframework.batch.item.ItemWriter;
|
|
388
|
+
import org.springframework.data.mongodb.core.MongoTemplate;
|
|
389
|
+
import org.springframework.stereotype.Component;
|
|
390
|
+
|
|
391
|
+
@Component
|
|
392
|
+
@RequiredArgsConstructor
|
|
393
|
+
@Slf4j
|
|
394
|
+
public class SampleItemWriter implements ItemWriter<SampleTargetDocument> {
|
|
395
|
+
|
|
396
|
+
private final MongoTemplate mongoTemplate;
|
|
397
|
+
|
|
398
|
+
@Override
|
|
399
|
+
public void write(Chunk<? extends SampleTargetDocument> chunk) {
|
|
400
|
+
log.info("Writing batch of {} items", chunk.size());
|
|
401
|
+
mongoTemplate.insertAll(chunk.getItems());
|
|
402
|
+
}
|
|
403
|
+
}
|
|
404
|
+
```
|
|
405
|
+
|
|
406
|
+
For updates to existing documents, use `chunk.getItems().forEach(mongoTemplate::save)`.
|
|
407
|
+
For inserts of new documents, `mongoTemplate.insertAll()` is more efficient as it uses
|
|
408
|
+
a bulk insert operation.
|
|
409
|
+
|
|
410
|
+
---
|
|
411
|
+
|
|
412
|
+
## Sample Document Classes [Include only if Spring Batch = yes]
|
|
413
|
+
|
|
414
|
+
These placeholder document classes are used by the reader, processor, and writer.
|
|
415
|
+
Replace with actual module types in the real implementation.
|
|
416
|
+
|
|
417
|
+
```java
|
|
418
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch;
|
|
419
|
+
|
|
420
|
+
import lombok.AllArgsConstructor;
|
|
421
|
+
import lombok.Getter;
|
|
422
|
+
import lombok.NoArgsConstructor;
|
|
423
|
+
import lombok.Setter;
|
|
424
|
+
import org.springframework.data.annotation.Id;
|
|
425
|
+
import org.springframework.data.mongodb.core.mapping.Document;
|
|
426
|
+
|
|
427
|
+
@Getter
|
|
428
|
+
@Setter
|
|
429
|
+
@NoArgsConstructor
|
|
430
|
+
@AllArgsConstructor
|
|
431
|
+
@Document(collection = "sample_source")
|
|
432
|
+
public class SampleSourceDocument {
|
|
433
|
+
|
|
434
|
+
@Id
|
|
435
|
+
private String id;
|
|
436
|
+
private String fieldOne;
|
|
437
|
+
private String fieldTwo;
|
|
438
|
+
}
|
|
439
|
+
```
|
|
440
|
+
|
|
441
|
+
```java
|
|
442
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch;
|
|
443
|
+
|
|
444
|
+
import lombok.AllArgsConstructor;
|
|
445
|
+
import lombok.Builder;
|
|
446
|
+
import lombok.Getter;
|
|
447
|
+
import lombok.NoArgsConstructor;
|
|
448
|
+
import lombok.Setter;
|
|
449
|
+
import org.springframework.data.annotation.Id;
|
|
450
|
+
import org.springframework.data.mongodb.core.mapping.Document;
|
|
451
|
+
import java.time.Instant;
|
|
452
|
+
|
|
453
|
+
@Getter
|
|
454
|
+
@Setter
|
|
455
|
+
@Builder
|
|
456
|
+
@NoArgsConstructor
|
|
457
|
+
@AllArgsConstructor
|
|
458
|
+
@Document(collection = "sample_target")
|
|
459
|
+
public class SampleTargetDocument {
|
|
460
|
+
|
|
461
|
+
@Id
|
|
462
|
+
private String id;
|
|
463
|
+
private String sourceId;
|
|
464
|
+
private String transformedField;
|
|
465
|
+
private Instant processedAt;
|
|
466
|
+
}
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
---
|
|
470
|
+
|
|
471
|
+
## Job Metadata Storage
|
|
472
|
+
|
|
473
|
+
Quartz job metadata (triggers, job details, scheduler state) is persisted in MongoDB
|
|
474
|
+
via `io.fluidsonic.mirror`. This is configured entirely through `application.yml` Quartz
|
|
475
|
+
properties — no additional Java configuration is needed.
|
|
476
|
+
|
|
477
|
+
Spring Batch's `JobRepository` (which tracks job execution history, step status, etc.)
|
|
478
|
+
uses an in-memory `MapJobRepositoryFactoryBean` by default in this skeleton. For
|
|
479
|
+
production use where job restart and execution history matter, replace with a MongoDB-backed
|
|
480
|
+
`JobRepository` implementation.
|
|
481
|
+
|
|
482
|
+
---
|
|
483
|
+
|
|
484
|
+
## Scheduling Module Package Structure
|
|
485
|
+
|
|
486
|
+
### Quartz-Only (Scheduling = yes, Spring Batch = no)
|
|
487
|
+
|
|
488
|
+
```
|
|
489
|
+
scheduling/
|
|
490
|
+
├── SchedulingService.java # Public interface (if other modules need to trigger jobs)
|
|
491
|
+
└── internal/
|
|
492
|
+
└── job/
|
|
493
|
+
└── SampleQuartzJob.java # Quartz job — executes logic directly
|
|
494
|
+
```
|
|
495
|
+
|
|
496
|
+
### Quartz + Spring Batch (Scheduling = yes, Spring Batch = yes)
|
|
497
|
+
|
|
498
|
+
```
|
|
499
|
+
scheduling/
|
|
500
|
+
├── SchedulingService.java # Public interface (if other modules need to trigger jobs)
|
|
501
|
+
└── internal/
|
|
502
|
+
├── job/
|
|
503
|
+
│ └── SampleQuartzJob.java # Quartz job that launches batch
|
|
504
|
+
└── batch/
|
|
505
|
+
├── SampleBatchConfig.java # Job, Step, chunk config
|
|
506
|
+
├── SampleItemReader.java # MongoDB reader
|
|
507
|
+
├── SampleItemProcessor.java # Transformation logic
|
|
508
|
+
└── SampleItemWriter.java # MongoDB writer
|
|
509
|
+
```
|
|
510
|
+
|
|
511
|
+
This module follows the same public/internal encapsulation as modules. The Quartz
|
|
512
|
+
jobs and Batch configuration are implementation details hidden in `internal/`. If other
|
|
513
|
+
modules need to trigger a job programmatically, they use the `SchedulingService` interface
|
|
514
|
+
in the module root. Otherwise, jobs are triggered purely by Quartz cron schedules and the
|
|
515
|
+
public interface can be omitted.
|
|
516
|
+
|
|
517
|
+
---
|
|
518
|
+
|
|
519
|
+
# Remote Partitioning with RabbitMQ
|
|
520
|
+
|
|
521
|
+
This section covers **Spring Batch remote partitioning** using **RabbitMQ** as the
|
|
522
|
+
messaging middleware. Remote partitioning enables horizontal scaling of batch jobs across
|
|
523
|
+
multiple worker nodes. Include this content in Section 18 of the generated specification
|
|
524
|
+
only when **Scheduling = yes AND Remote Partitioning = yes**. All code samples must be
|
|
525
|
+
reproduced in the generated spec with full import statements and constructor injection.
|
|
526
|
+
|
|
527
|
+
---
|
|
528
|
+
|
|
529
|
+
## Remote Partitioning Architecture
|
|
530
|
+
|
|
531
|
+
Remote partitioning splits a batch step's data across multiple worker processes. A
|
|
532
|
+
**manager** node partitions the input and sends partition metadata via RabbitMQ queues.
|
|
533
|
+
**Worker** nodes pick up partitions, execute the step locally, and report completion back
|
|
534
|
+
via a reply queue.
|
|
535
|
+
|
|
536
|
+
```
|
|
537
|
+
┌─────────────────────────────────────────────────────────────────────┐
|
|
538
|
+
│ Manager Node │
|
|
539
|
+
│ │
|
|
540
|
+
│ Quartz Trigger ─► QuartzJobBean ─► Spring Batch Job │
|
|
541
|
+
│ │ │
|
|
542
|
+
│ Manager Step │
|
|
543
|
+
│ (Partitioner) │
|
|
544
|
+
│ │ │
|
|
545
|
+
│ ┌────────────┴────────────┐ │
|
|
546
|
+
│ ▼ ▼ │
|
|
547
|
+
│ Partition 1 metadata Partition N metadata │
|
|
548
|
+
│ │ │ │
|
|
549
|
+
│ ▼ ▼ │
|
|
550
|
+
│ ┌─── Outbound Channel (request) ───┐ │
|
|
551
|
+
│ │ AmqpOutboundChannelAdapter │ │
|
|
552
|
+
│ └──────────────┬───────────────────┘ │
|
|
553
|
+
│ │ │
|
|
554
|
+
└──────────────────────────────┼──────────────────────────────────────┘
|
|
555
|
+
│
|
|
556
|
+
▼
|
|
557
|
+
┌──────────────────────────────┐
|
|
558
|
+
│ RabbitMQ │
|
|
559
|
+
│ │
|
|
560
|
+
│ batch.partition.requests │ (Direct Exchange)
|
|
561
|
+
│ batch.partition.replies │
|
|
562
|
+
│ batch.partition.requests.dlq │ (Dead Letter Queue)
|
|
563
|
+
└──────────┬───────────────────┘
|
|
564
|
+
│
|
|
565
|
+
┌────────────┼────────────────┐
|
|
566
|
+
▼ ▼ ▼
|
|
567
|
+
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
|
|
568
|
+
│ Worker Node 1 │ │ Worker Node 2 │ │ Worker Node N │
|
|
569
|
+
│ │ │ │ │ │
|
|
570
|
+
│ Inbound Channel │ │ Inbound Channel │ │ Inbound Channel │
|
|
571
|
+
│ (request) │ │ (request) │ │ (request) │
|
|
572
|
+
│ │ │ │ │ │ │ │ │
|
|
573
|
+
│ ▼ │ │ ▼ │ │ ▼ │
|
|
574
|
+
│ Worker Step │ │ Worker Step │ │ Worker Step │
|
|
575
|
+
│ Reader→ │ │ Reader→ │ │ Reader→ │
|
|
576
|
+
│ Processor→ │ │ Processor→ │ │ Processor→ │
|
|
577
|
+
│ Writer │ │ Writer │ │ Writer │
|
|
578
|
+
│ │ │ │ │ │ │ │ │
|
|
579
|
+
│ ▼ │ │ ▼ │ │ ▼ │
|
|
580
|
+
│ Outbound Channel│ │ Outbound Channel│ │ Outbound Channel│
|
|
581
|
+
│ (reply) │ │ (reply) │ │ (reply) │
|
|
582
|
+
└──────────────────┘ └──────────────────┘ └──────────────────┘
|
|
583
|
+
```
|
|
584
|
+
|
|
585
|
+
Key points:
|
|
586
|
+
- The **manager** only partitions and delegates — it does NOT read/process/write data itself
|
|
587
|
+
- Each **worker** receives a `StepExecution` with partition-specific `ExecutionContext`
|
|
588
|
+
(e.g., `minId`, `maxId`) and runs a full Reader → Processor → Writer pipeline locally
|
|
589
|
+
- All nodes share the **same `JobRepository` database** so the manager can track partition
|
|
590
|
+
completion; this is a hard requirement for remote partitioning
|
|
591
|
+
- Workers are deployed as **separate JVM instances** of the same application JAR, activated
|
|
592
|
+
with a different Spring profile (e.g., `--spring.profiles.active=worker`)
|
|
593
|
+
|
|
594
|
+
---
|
|
595
|
+
|
|
596
|
+
## Maven Dependencies (Remote Partitioning)
|
|
597
|
+
|
|
598
|
+
Add these dependencies **in addition to** the base scheduling dependencies
|
|
599
|
+
(`spring-boot-starter-quartz`, `spring-boot-starter-batch`):
|
|
600
|
+
|
|
601
|
+
```xml
|
|
602
|
+
<!-- Spring Batch Integration — remote partitioning support -->
|
|
603
|
+
<dependency>
|
|
604
|
+
<groupId>org.springframework.batch</groupId>
|
|
605
|
+
<artifactId>spring-batch-integration</artifactId>
|
|
606
|
+
</dependency>
|
|
607
|
+
|
|
608
|
+
<!-- Spring Integration AMQP — RabbitMQ channel adapters -->
|
|
609
|
+
<dependency>
|
|
610
|
+
<groupId>org.springframework.integration</groupId>
|
|
611
|
+
<artifactId>spring-integration-amqp</artifactId>
|
|
612
|
+
</dependency>
|
|
613
|
+
|
|
614
|
+
<!-- Spring Boot AMQP Starter — RabbitMQ auto-configuration -->
|
|
615
|
+
<dependency>
|
|
616
|
+
<groupId>org.springframework.boot</groupId>
|
|
617
|
+
<artifactId>spring-boot-starter-amqp</artifactId>
|
|
618
|
+
</dependency>
|
|
619
|
+
```
|
|
620
|
+
|
|
621
|
+
These are managed by the Spring Boot BOM — no explicit version needed.
|
|
622
|
+
|
|
623
|
+
---
|
|
624
|
+
|
|
625
|
+
## Application Configuration (Remote Partitioning)
|
|
626
|
+
|
|
627
|
+
### application.yml — RabbitMQ connection
|
|
628
|
+
|
|
629
|
+
```yaml
|
|
630
|
+
spring:
|
|
631
|
+
rabbitmq:
|
|
632
|
+
host: localhost
|
|
633
|
+
port: 5672
|
|
634
|
+
username: guest
|
|
635
|
+
password: guest
|
|
636
|
+
virtual-host: /
|
|
637
|
+
|
|
638
|
+
app:
|
|
639
|
+
batch:
|
|
640
|
+
partition:
|
|
641
|
+
grid-size: 4
|
|
642
|
+
request-queue: batch.partition.requests
|
|
643
|
+
reply-queue: batch.partition.replies
|
|
644
|
+
request-dlq: batch.partition.requests.dlq
|
|
645
|
+
exchange: batch.partition.exchange
|
|
646
|
+
```
|
|
647
|
+
|
|
648
|
+
### application-worker.yml — Worker profile
|
|
649
|
+
|
|
650
|
+
Workers disable the Quartz scheduler and web server since they only process partitions:
|
|
651
|
+
|
|
652
|
+
```yaml
|
|
653
|
+
spring:
|
|
654
|
+
quartz:
|
|
655
|
+
auto-startup: false
|
|
656
|
+
main:
|
|
657
|
+
web-application-type: none
|
|
658
|
+
|
|
659
|
+
server:
|
|
660
|
+
port: 0
|
|
661
|
+
```
|
|
662
|
+
|
|
663
|
+
Activate with `--spring.profiles.active=worker`.
|
|
664
|
+
|
|
665
|
+
---
|
|
666
|
+
|
|
667
|
+
## RabbitMQ Infrastructure Configuration
|
|
668
|
+
|
|
669
|
+
### RabbitMQPartitionConfig.java
|
|
670
|
+
|
|
671
|
+
```java
|
|
672
|
+
package {{BASE_PACKAGE}}.config;
|
|
673
|
+
|
|
674
|
+
import org.springframework.amqp.core.Binding;
|
|
675
|
+
import org.springframework.amqp.core.BindingBuilder;
|
|
676
|
+
import org.springframework.amqp.core.DirectExchange;
|
|
677
|
+
import org.springframework.amqp.core.Queue;
|
|
678
|
+
import org.springframework.amqp.core.QueueBuilder;
|
|
679
|
+
import org.springframework.beans.factory.annotation.Value;
|
|
680
|
+
import org.springframework.context.annotation.Bean;
|
|
681
|
+
import org.springframework.context.annotation.Configuration;
|
|
682
|
+
|
|
683
|
+
@Configuration
|
|
684
|
+
public class RabbitMQPartitionConfig {
|
|
685
|
+
|
|
686
|
+
@Value("${app.batch.partition.exchange}")
|
|
687
|
+
private String exchangeName;
|
|
688
|
+
|
|
689
|
+
@Value("${app.batch.partition.request-queue}")
|
|
690
|
+
private String requestQueueName;
|
|
691
|
+
|
|
692
|
+
@Value("${app.batch.partition.reply-queue}")
|
|
693
|
+
private String replyQueueName;
|
|
694
|
+
|
|
695
|
+
@Value("${app.batch.partition.request-dlq}")
|
|
696
|
+
private String requestDlqName;
|
|
697
|
+
|
|
698
|
+
@Bean
|
|
699
|
+
public DirectExchange partitionExchange() {
|
|
700
|
+
return new DirectExchange(exchangeName);
|
|
701
|
+
}
|
|
702
|
+
|
|
703
|
+
@Bean
|
|
704
|
+
public Queue partitionRequestQueue() {
|
|
705
|
+
return QueueBuilder.durable(requestQueueName)
|
|
706
|
+
.withArgument("x-dead-letter-exchange", "")
|
|
707
|
+
.withArgument("x-dead-letter-routing-key", requestDlqName)
|
|
708
|
+
.build();
|
|
709
|
+
}
|
|
710
|
+
|
|
711
|
+
@Bean
|
|
712
|
+
public Queue partitionReplyQueue() {
|
|
713
|
+
return QueueBuilder.durable(replyQueueName).build();
|
|
714
|
+
}
|
|
715
|
+
|
|
716
|
+
@Bean
|
|
717
|
+
public Queue partitionRequestDlq() {
|
|
718
|
+
return QueueBuilder.durable(requestDlqName).build();
|
|
719
|
+
}
|
|
720
|
+
|
|
721
|
+
@Bean
|
|
722
|
+
public Binding requestBinding(Queue partitionRequestQueue, DirectExchange partitionExchange) {
|
|
723
|
+
return BindingBuilder.bind(partitionRequestQueue)
|
|
724
|
+
.to(partitionExchange)
|
|
725
|
+
.with(requestQueueName);
|
|
726
|
+
}
|
|
727
|
+
|
|
728
|
+
@Bean
|
|
729
|
+
public Binding replyBinding(Queue partitionReplyQueue, DirectExchange partitionExchange) {
|
|
730
|
+
return BindingBuilder.bind(partitionReplyQueue)
|
|
731
|
+
.to(partitionExchange)
|
|
732
|
+
.with(replyQueueName);
|
|
733
|
+
}
|
|
734
|
+
}
|
|
735
|
+
```
|
|
736
|
+
|
|
737
|
+
---
|
|
738
|
+
|
|
739
|
+
## Partitioner — Splitting Data into Ranges
|
|
740
|
+
|
|
741
|
+
### SamplePartitioner.java
|
|
742
|
+
|
|
743
|
+
The Partitioner creates a `Map<String, ExecutionContext>` where each entry represents one
|
|
744
|
+
partition. Workers receive these execution contexts with range parameters to scope their
|
|
745
|
+
data reading.
|
|
746
|
+
|
|
747
|
+
```java
|
|
748
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch.partition;
|
|
749
|
+
|
|
750
|
+
import lombok.RequiredArgsConstructor;
|
|
751
|
+
import lombok.extern.slf4j.Slf4j;
|
|
752
|
+
import org.springframework.batch.core.partition.support.Partitioner;
|
|
753
|
+
import org.springframework.batch.item.ExecutionContext;
|
|
754
|
+
import org.springframework.data.mongodb.core.MongoTemplate;
|
|
755
|
+
import org.springframework.data.mongodb.core.query.Query;
|
|
756
|
+
import org.springframework.stereotype.Component;
|
|
757
|
+
import java.util.HashMap;
|
|
758
|
+
import java.util.Map;
|
|
759
|
+
|
|
760
|
+
@Component
|
|
761
|
+
@RequiredArgsConstructor
|
|
762
|
+
@Slf4j
|
|
763
|
+
public class SamplePartitioner implements Partitioner {
|
|
764
|
+
|
|
765
|
+
private final MongoTemplate mongoTemplate;
|
|
766
|
+
|
|
767
|
+
@Override
|
|
768
|
+
public Map<String, ExecutionContext> partition(int gridSize) {
|
|
769
|
+
long totalCount = mongoTemplate.count(new Query(), "sample_source");
|
|
770
|
+
long partitionSize = (totalCount + gridSize - 1) / gridSize;
|
|
771
|
+
|
|
772
|
+
Map<String, ExecutionContext> partitions = new HashMap<>();
|
|
773
|
+
for (int i = 0; i < gridSize; i++) {
|
|
774
|
+
long minIndex = i * partitionSize;
|
|
775
|
+
long maxIndex = Math.min((i + 1) * partitionSize - 1, totalCount - 1);
|
|
776
|
+
|
|
777
|
+
ExecutionContext ctx = new ExecutionContext();
|
|
778
|
+
ctx.putLong("minIndex", minIndex);
|
|
779
|
+
ctx.putLong("maxIndex", maxIndex);
|
|
780
|
+
ctx.putString("partitionName", "partition-" + i);
|
|
781
|
+
|
|
782
|
+
partitions.put("partition-" + i, ctx);
|
|
783
|
+
log.info("Created partition-{}: minIndex={}, maxIndex={}", i, minIndex, maxIndex);
|
|
784
|
+
}
|
|
785
|
+
return partitions;
|
|
786
|
+
}
|
|
787
|
+
}
|
|
788
|
+
```
|
|
789
|
+
|
|
790
|
+
For relational databases (PostgreSQL/MySQL), partition by primary key ID ranges instead:
|
|
791
|
+
|
|
792
|
+
```java
|
|
793
|
+
// Alternative for JPA-backed data
|
|
794
|
+
@Override
|
|
795
|
+
public Map<String, ExecutionContext> partition(int gridSize) {
|
|
796
|
+
Long minId = jdbcTemplate.queryForObject("SELECT MIN(id) FROM sample_source", Long.class);
|
|
797
|
+
Long maxId = jdbcTemplate.queryForObject("SELECT MAX(id) FROM sample_source", Long.class);
|
|
798
|
+
long range = (maxId - minId + gridSize) / gridSize;
|
|
799
|
+
|
|
800
|
+
Map<String, ExecutionContext> partitions = new HashMap<>();
|
|
801
|
+
for (int i = 0; i < gridSize; i++) {
|
|
802
|
+
ExecutionContext ctx = new ExecutionContext();
|
|
803
|
+
ctx.putLong("minId", minId + (i * range));
|
|
804
|
+
ctx.putLong("maxId", Math.min(minId + ((i + 1) * range) - 1, maxId));
|
|
805
|
+
partitions.put("partition-" + i, ctx);
|
|
806
|
+
}
|
|
807
|
+
return partitions;
|
|
808
|
+
}
|
|
809
|
+
```
|
|
810
|
+
|
|
811
|
+
---
|
|
812
|
+
|
|
813
|
+
## Spring Integration Channels
|
|
814
|
+
|
|
815
|
+
### PartitionChannelConfig.java
|
|
816
|
+
|
|
817
|
+
Defines the `DirectChannel` beans and AMQP adapters that bridge Spring Batch remote
|
|
818
|
+
partitioning with RabbitMQ.
|
|
819
|
+
|
|
820
|
+
```java
|
|
821
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch.partition;
|
|
822
|
+
|
|
823
|
+
import lombok.RequiredArgsConstructor;
|
|
824
|
+
import org.springframework.amqp.core.AmqpTemplate;
|
|
825
|
+
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
|
|
826
|
+
import org.springframework.beans.factory.annotation.Value;
|
|
827
|
+
import org.springframework.context.annotation.Bean;
|
|
828
|
+
import org.springframework.context.annotation.Configuration;
|
|
829
|
+
import org.springframework.integration.amqp.dsl.Amqp;
|
|
830
|
+
import org.springframework.integration.channel.DirectChannel;
|
|
831
|
+
import org.springframework.integration.dsl.IntegrationFlow;
|
|
832
|
+
import org.springframework.messaging.MessageChannel;
|
|
833
|
+
|
|
834
|
+
@Configuration
|
|
835
|
+
@RequiredArgsConstructor
|
|
836
|
+
public class PartitionChannelConfig {
|
|
837
|
+
|
|
838
|
+
private final ConnectionFactory connectionFactory;
|
|
839
|
+
private final AmqpTemplate amqpTemplate;
|
|
840
|
+
|
|
841
|
+
@Value("${app.batch.partition.request-queue}")
|
|
842
|
+
private String requestQueueName;
|
|
843
|
+
|
|
844
|
+
@Value("${app.batch.partition.reply-queue}")
|
|
845
|
+
private String replyQueueName;
|
|
846
|
+
|
|
847
|
+
@Value("${app.batch.partition.exchange}")
|
|
848
|
+
private String exchangeName;
|
|
849
|
+
|
|
850
|
+
// --- Channels ---
|
|
851
|
+
|
|
852
|
+
@Bean
|
|
853
|
+
public MessageChannel partitionRequestChannel() {
|
|
854
|
+
return new DirectChannel();
|
|
855
|
+
}
|
|
856
|
+
|
|
857
|
+
@Bean
|
|
858
|
+
public MessageChannel partitionReplyChannel() {
|
|
859
|
+
return new DirectChannel();
|
|
860
|
+
}
|
|
861
|
+
|
|
862
|
+
// --- Outbound: Manager sends partitions to RabbitMQ request queue ---
|
|
863
|
+
|
|
864
|
+
@Bean
|
|
865
|
+
public IntegrationFlow outboundRequestFlow() {
|
|
866
|
+
return IntegrationFlow.from(partitionRequestChannel())
|
|
867
|
+
.handle(Amqp.outboundAdapter(amqpTemplate)
|
|
868
|
+
.exchangeName(exchangeName)
|
|
869
|
+
.routingKey(requestQueueName))
|
|
870
|
+
.get();
|
|
871
|
+
}
|
|
872
|
+
|
|
873
|
+
// --- Inbound: Manager receives replies from RabbitMQ reply queue ---
|
|
874
|
+
|
|
875
|
+
@Bean
|
|
876
|
+
public IntegrationFlow inboundReplyFlow() {
|
|
877
|
+
return IntegrationFlow.from(
|
|
878
|
+
Amqp.inboundAdapter(connectionFactory, replyQueueName))
|
|
879
|
+
.channel(partitionReplyChannel())
|
|
880
|
+
.get();
|
|
881
|
+
}
|
|
882
|
+
|
|
883
|
+
// --- Inbound: Worker receives partitions from RabbitMQ request queue ---
|
|
884
|
+
|
|
885
|
+
@Bean
|
|
886
|
+
public IntegrationFlow inboundRequestFlow() {
|
|
887
|
+
return IntegrationFlow.from(
|
|
888
|
+
Amqp.inboundAdapter(connectionFactory, requestQueueName))
|
|
889
|
+
.channel(partitionRequestChannel())
|
|
890
|
+
.get();
|
|
891
|
+
}
|
|
892
|
+
|
|
893
|
+
// --- Outbound: Worker sends replies to RabbitMQ reply queue ---
|
|
894
|
+
|
|
895
|
+
@Bean
|
|
896
|
+
public IntegrationFlow outboundReplyFlow() {
|
|
897
|
+
return IntegrationFlow.from(partitionReplyChannel())
|
|
898
|
+
.handle(Amqp.outboundAdapter(amqpTemplate)
|
|
899
|
+
.exchangeName(exchangeName)
|
|
900
|
+
.routingKey(replyQueueName))
|
|
901
|
+
.get();
|
|
902
|
+
}
|
|
903
|
+
}
|
|
904
|
+
```
|
|
905
|
+
|
|
906
|
+
**Important:** In a real deployment, the manager and worker channel beans must be
|
|
907
|
+
separated using Spring profiles (`@Profile("manager")` / `@Profile("worker")`) so that
|
|
908
|
+
manager-only flows (outbound request, inbound reply) and worker-only flows (inbound
|
|
909
|
+
request, outbound reply) are activated on the correct node. The sample above shows all
|
|
910
|
+
flows in one class for clarity.
|
|
911
|
+
|
|
912
|
+
---
|
|
913
|
+
|
|
914
|
+
## Manager Step Configuration
|
|
915
|
+
|
|
916
|
+
### PartitionedBatchConfig.java (Manager)
|
|
917
|
+
|
|
918
|
+
The manager step uses `RemotePartitioningManagerStepBuilder` to send partition metadata
|
|
919
|
+
to workers via RabbitMQ and aggregate their results.
|
|
920
|
+
|
|
921
|
+
```java
|
|
922
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch.partition;
|
|
923
|
+
|
|
924
|
+
import lombok.RequiredArgsConstructor;
|
|
925
|
+
import org.springframework.batch.core.Job;
|
|
926
|
+
import org.springframework.batch.core.Step;
|
|
927
|
+
import org.springframework.batch.core.job.builder.JobBuilder;
|
|
928
|
+
import org.springframework.batch.core.partition.support.Partitioner;
|
|
929
|
+
import org.springframework.batch.core.repository.JobRepository;
|
|
930
|
+
import org.springframework.batch.integration.partition.RemotePartitioningManagerStepBuilderFactory;
|
|
931
|
+
import org.springframework.beans.factory.annotation.Value;
|
|
932
|
+
import org.springframework.context.annotation.Bean;
|
|
933
|
+
import org.springframework.context.annotation.Configuration;
|
|
934
|
+
import org.springframework.context.annotation.Profile;
|
|
935
|
+
import org.springframework.integration.channel.DirectChannel;
|
|
936
|
+
import org.springframework.messaging.MessageChannel;
|
|
937
|
+
|
|
938
|
+
@Configuration
|
|
939
|
+
@Profile("!worker")
|
|
940
|
+
@RequiredArgsConstructor
|
|
941
|
+
public class PartitionedBatchConfig {
|
|
942
|
+
|
|
943
|
+
private final JobRepository jobRepository;
|
|
944
|
+
private final RemotePartitioningManagerStepBuilderFactory managerStepBuilderFactory;
|
|
945
|
+
|
|
946
|
+
@Value("${app.batch.partition.grid-size:4}")
|
|
947
|
+
private int gridSize;
|
|
948
|
+
|
|
949
|
+
@Bean
|
|
950
|
+
public Job partitionedBatchJob(Step managerStep) {
|
|
951
|
+
return new JobBuilder("partitionedBatchJob", jobRepository)
|
|
952
|
+
.start(managerStep)
|
|
953
|
+
.build();
|
|
954
|
+
}
|
|
955
|
+
|
|
956
|
+
@Bean
|
|
957
|
+
public Step managerStep(
|
|
958
|
+
Partitioner samplePartitioner,
|
|
959
|
+
MessageChannel partitionRequestChannel,
|
|
960
|
+
MessageChannel partitionReplyChannel) {
|
|
961
|
+
return managerStepBuilderFactory.get("managerStep")
|
|
962
|
+
.partitioner("workerStep", samplePartitioner)
|
|
963
|
+
.gridSize(gridSize)
|
|
964
|
+
.outputChannel(partitionRequestChannel)
|
|
965
|
+
.inputChannel(partitionReplyChannel)
|
|
966
|
+
.build();
|
|
967
|
+
}
|
|
968
|
+
}
|
|
969
|
+
```
|
|
970
|
+
|
|
971
|
+
**Note:** `@EnableBatchIntegration` must be placed on a `@Configuration` class (or the
|
|
972
|
+
main application class) to make `RemotePartitioningManagerStepBuilderFactory` and
|
|
973
|
+
`RemotePartitioningWorkerStepBuilderFactory` available for injection.
|
|
974
|
+
|
|
975
|
+
```java
|
|
976
|
+
package {{BASE_PACKAGE}}.config;
|
|
977
|
+
|
|
978
|
+
import org.springframework.batch.integration.config.annotation.EnableBatchIntegration;
|
|
979
|
+
import org.springframework.context.annotation.Configuration;
|
|
980
|
+
|
|
981
|
+
@Configuration
|
|
982
|
+
@EnableBatchIntegration
|
|
983
|
+
public class BatchIntegrationConfig {
|
|
984
|
+
}
|
|
985
|
+
```
|
|
986
|
+
|
|
987
|
+
---
|
|
988
|
+
|
|
989
|
+
## Worker Step Configuration
|
|
990
|
+
|
|
991
|
+
### WorkerBatchConfig.java
|
|
992
|
+
|
|
993
|
+
The worker step uses `RemotePartitioningWorkerStepBuilderFactory` to receive partitions
|
|
994
|
+
and execute the local Reader → Processor → Writer pipeline.
|
|
995
|
+
|
|
996
|
+
```java
|
|
997
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch.partition;
|
|
998
|
+
|
|
999
|
+
import lombok.RequiredArgsConstructor;
|
|
1000
|
+
import org.springframework.batch.core.Step;
|
|
1001
|
+
import org.springframework.batch.integration.partition.RemotePartitioningWorkerStepBuilderFactory;
|
|
1002
|
+
import org.springframework.batch.item.ItemProcessor;
|
|
1003
|
+
import org.springframework.batch.item.ItemReader;
|
|
1004
|
+
import org.springframework.batch.item.ItemWriter;
|
|
1005
|
+
import org.springframework.context.annotation.Bean;
|
|
1006
|
+
import org.springframework.context.annotation.Configuration;
|
|
1007
|
+
import org.springframework.context.annotation.Profile;
|
|
1008
|
+
import org.springframework.messaging.MessageChannel;
|
|
1009
|
+
import org.springframework.transaction.PlatformTransactionManager;
|
|
1010
|
+
|
|
1011
|
+
@Configuration
|
|
1012
|
+
@Profile("worker")
|
|
1013
|
+
@RequiredArgsConstructor
|
|
1014
|
+
public class WorkerBatchConfig {
|
|
1015
|
+
|
|
1016
|
+
private final RemotePartitioningWorkerStepBuilderFactory workerStepBuilderFactory;
|
|
1017
|
+
private final PlatformTransactionManager transactionManager;
|
|
1018
|
+
|
|
1019
|
+
@Bean
|
|
1020
|
+
public Step workerStep(
|
|
1021
|
+
MessageChannel partitionRequestChannel,
|
|
1022
|
+
MessageChannel partitionReplyChannel,
|
|
1023
|
+
ItemReader<SampleSourceDocument> partitionedReader,
|
|
1024
|
+
ItemProcessor<SampleSourceDocument, SampleTargetDocument> processor,
|
|
1025
|
+
ItemWriter<SampleTargetDocument> writer) {
|
|
1026
|
+
return workerStepBuilderFactory.get("workerStep")
|
|
1027
|
+
.inputChannel(partitionRequestChannel)
|
|
1028
|
+
.outputChannel(partitionReplyChannel)
|
|
1029
|
+
.<SampleSourceDocument, SampleTargetDocument>chunk(100, transactionManager)
|
|
1030
|
+
.reader(partitionedReader)
|
|
1031
|
+
.processor(processor)
|
|
1032
|
+
.writer(writer)
|
|
1033
|
+
.build();
|
|
1034
|
+
}
|
|
1035
|
+
}
|
|
1036
|
+
```
|
|
1037
|
+
|
|
1038
|
+
The `partitionedReader` is a `@StepScope` bean that reads `minIndex`/`maxIndex` (or
|
|
1039
|
+
`minId`/`maxId`) from the step's `ExecutionContext` to scope its query:
|
|
1040
|
+
|
|
1041
|
+
```java
|
|
1042
|
+
package {{BASE_PACKAGE}}.scheduling.internal.batch.partition;
|
|
1043
|
+
|
|
1044
|
+
import lombok.RequiredArgsConstructor;
|
|
1045
|
+
import org.springframework.batch.core.configuration.annotation.StepScope;
|
|
1046
|
+
import org.springframework.batch.item.data.MongoPagingItemReader;
|
|
1047
|
+
import org.springframework.batch.item.data.builder.MongoPagingItemReaderBuilder;
|
|
1048
|
+
import org.springframework.beans.factory.annotation.Value;
|
|
1049
|
+
import org.springframework.context.annotation.Bean;
|
|
1050
|
+
import org.springframework.context.annotation.Configuration;
|
|
1051
|
+
import org.springframework.data.domain.Sort;
|
|
1052
|
+
import org.springframework.data.mongodb.core.MongoTemplate;
|
|
1053
|
+
import java.util.Map;
|
|
1054
|
+
|
|
1055
|
+
@Configuration
|
|
1056
|
+
@RequiredArgsConstructor
|
|
1057
|
+
public class PartitionedReaderConfig {
|
|
1058
|
+
|
|
1059
|
+
private final MongoTemplate mongoTemplate;
|
|
1060
|
+
|
|
1061
|
+
@StepScope
|
|
1062
|
+
@Bean
|
|
1063
|
+
public MongoPagingItemReader<SampleSourceDocument> partitionedReader(
|
|
1064
|
+
@Value("#{stepExecutionContext['minIndex']}") Long minIndex,
|
|
1065
|
+
@Value("#{stepExecutionContext['maxIndex']}") Long maxIndex) {
|
|
1066
|
+
return new MongoPagingItemReaderBuilder<SampleSourceDocument>()
|
|
1067
|
+
.name("partitionedReader")
|
|
1068
|
+
.template(mongoTemplate)
|
|
1069
|
+
.targetType(SampleSourceDocument.class)
|
|
1070
|
+
.jsonQuery("{}")
|
|
1071
|
+
.sorts(Map.of("_id", Sort.Direction.ASC))
|
|
1072
|
+
.currentItemCount(minIndex.intValue())
|
|
1073
|
+
.maxItemCount(maxIndex.intValue() + 1)
|
|
1074
|
+
.pageSize(100)
|
|
1075
|
+
.build();
|
|
1076
|
+
}
|
|
1077
|
+
}
|
|
1078
|
+
```
|
|
1079
|
+
|
|
1080
|
+
---
|
|
1081
|
+
|
|
1082
|
+
## Quartz Integration with Partitioned Jobs
|
|
1083
|
+
|
|
1084
|
+
When remote partitioning is enabled, the Quartz job launches the **partitioned** batch
|
|
1085
|
+
job instead of the simple single-step job. Update the Quartz job bean to inject the
|
|
1086
|
+
partitioned job:
|
|
1087
|
+
|
|
1088
|
+
### PartitionedQuartzJob.java
|
|
1089
|
+
|
|
1090
|
+
```java
|
|
1091
|
+
package {{BASE_PACKAGE}}.scheduling.internal.job;
|
|
1092
|
+
|
|
1093
|
+
import lombok.RequiredArgsConstructor;
|
|
1094
|
+
import lombok.extern.slf4j.Slf4j;
|
|
1095
|
+
import org.quartz.JobExecutionContext;
|
|
1096
|
+
import org.quartz.JobExecutionException;
|
|
1097
|
+
import org.springframework.batch.core.Job;
|
|
1098
|
+
import org.springframework.batch.core.JobParameters;
|
|
1099
|
+
import org.springframework.batch.core.JobParametersBuilder;
|
|
1100
|
+
import org.springframework.batch.core.launch.JobLauncher;
|
|
1101
|
+
import org.springframework.beans.factory.annotation.Qualifier;
|
|
1102
|
+
import org.springframework.scheduling.quartz.QuartzJobBean;
|
|
1103
|
+
import org.springframework.stereotype.Component;
|
|
1104
|
+
|
|
1105
|
+
@Component
|
|
1106
|
+
@RequiredArgsConstructor
|
|
1107
|
+
@Slf4j
|
|
1108
|
+
public class PartitionedQuartzJob extends QuartzJobBean {
|
|
1109
|
+
|
|
1110
|
+
private final JobLauncher jobLauncher;
|
|
1111
|
+
|
|
1112
|
+
@Qualifier("partitionedBatchJob")
|
|
1113
|
+
private final Job partitionedBatchJob;
|
|
1114
|
+
|
|
1115
|
+
@Override
|
|
1116
|
+
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
|
|
1117
|
+
try {
|
|
1118
|
+
JobParameters params = new JobParametersBuilder()
|
|
1119
|
+
.addLong("timestamp", System.currentTimeMillis())
|
|
1120
|
+
.toJobParameters();
|
|
1121
|
+
|
|
1122
|
+
log.info("Launching partitioned batch job at {}", params);
|
|
1123
|
+
jobLauncher.run(partitionedBatchJob, params);
|
|
1124
|
+
log.info("Partitioned batch job completed successfully");
|
|
1125
|
+
} catch (Exception e) {
|
|
1126
|
+
log.error("Failed to launch partitioned batch job", e);
|
|
1127
|
+
throw new JobExecutionException("Partitioned batch job launch failed", e);
|
|
1128
|
+
}
|
|
1129
|
+
}
|
|
1130
|
+
}
|
|
1131
|
+
```
|
|
1132
|
+
|
|
1133
|
+
The `QuartzConfig` registers this job with its trigger in the same manner as
|
|
1134
|
+
`SampleQuartzJob` (see the single-process Quartz section above). Both the simple and
|
|
1135
|
+
partitioned Quartz jobs can coexist — use separate job identities and triggers.
|
|
1136
|
+
|
|
1137
|
+
---
|
|
1138
|
+
|
|
1139
|
+
## Job Metadata Storage Note
|
|
1140
|
+
|
|
1141
|
+
Remote partitioning has a **hard requirement**: all manager and worker nodes must share
|
|
1142
|
+
the same `JobRepository` database. The manager writes partition metadata (`StepExecution`
|
|
1143
|
+
entries) and workers update them on completion. Without a shared database, the manager
|
|
1144
|
+
cannot track partition status.
|
|
1145
|
+
|
|
1146
|
+
- **If Database = MongoDB**: Use a shared MongoDB instance. Spring Batch's `JobRepository`
|
|
1147
|
+
must be configured with a MongoDB-backed implementation or fall back to an embedded
|
|
1148
|
+
JDBC `JobRepository` (e.g., H2) accessible to all nodes via a shared JDBC datasource.
|
|
1149
|
+
- **If Database = PostgreSQL/MySQL**: The shared relational database naturally satisfies
|
|
1150
|
+
this requirement. Use `spring.batch.jdbc.initialize-schema: always` on first run.
|
|
1151
|
+
- **If Database = none**: Remote partitioning **cannot be used** without a shared database.
|
|
1152
|
+
A shared database is required. The spec must warn the user that selecting
|
|
1153
|
+
`Remote Partitioning = yes` with `Database = none` is an invalid combination.
|
|
1154
|
+
|
|
1155
|
+
---
|
|
1156
|
+
|
|
1157
|
+
## Updated Package Structure (with Remote Partitioning)
|
|
1158
|
+
|
|
1159
|
+
When Remote Partitioning is enabled, add a `partition/` subpackage:
|
|
1160
|
+
|
|
1161
|
+
```
|
|
1162
|
+
scheduling/
|
|
1163
|
+
├── SchedulingService.java
|
|
1164
|
+
└── internal/
|
|
1165
|
+
├── job/
|
|
1166
|
+
│ ├── SampleQuartzJob.java
|
|
1167
|
+
│ └── PartitionedQuartzJob.java
|
|
1168
|
+
└── batch/
|
|
1169
|
+
├── SampleBatchConfig.java
|
|
1170
|
+
├── SampleItemReader.java
|
|
1171
|
+
├── SampleItemProcessor.java
|
|
1172
|
+
├── SampleItemWriter.java
|
|
1173
|
+
└── partition/
|
|
1174
|
+
├── SamplePartitioner.java
|
|
1175
|
+
├── PartitionChannelConfig.java
|
|
1176
|
+
├── PartitionedBatchConfig.java # Manager step (profile: !worker)
|
|
1177
|
+
├── WorkerBatchConfig.java # Worker step (profile: worker)
|
|
1178
|
+
└── PartitionedReaderConfig.java # @StepScope partitioned reader
|
|
1179
|
+
```
|
|
1180
|
+
|
|
1181
|
+
---
|
|
1182
|
+
|
|
1183
|
+
## Operational Notes
|
|
1184
|
+
|
|
1185
|
+
### Worker Deployment
|
|
1186
|
+
|
|
1187
|
+
Workers are deployed using the **same application JAR** with a different Spring profile:
|
|
1188
|
+
|
|
1189
|
+
```bash
|
|
1190
|
+
# Manager node (default — runs Quartz scheduler + web server)
|
|
1191
|
+
java -jar app.jar --spring.profiles.active=default
|
|
1192
|
+
|
|
1193
|
+
# Worker node (no web server, no Quartz — only listens for partitions)
|
|
1194
|
+
java -jar app.jar --spring.profiles.active=worker
|
|
1195
|
+
```
|
|
1196
|
+
|
|
1197
|
+
Scale workers horizontally by running more instances. RabbitMQ distributes partitions
|
|
1198
|
+
across all connected workers via competing consumers on the request queue.
|
|
1199
|
+
|
|
1200
|
+
### Failure and Retry Semantics
|
|
1201
|
+
|
|
1202
|
+
- If a worker fails mid-partition, its `StepExecution` is marked `FAILED` in the
|
|
1203
|
+
`JobRepository`. The manager step also fails, and the entire job can be **restarted** —
|
|
1204
|
+
Spring Batch only re-executes failed partitions on restart.
|
|
1205
|
+
- Configure RabbitMQ message acknowledgment to `AUTO` (default) so messages are only
|
|
1206
|
+
acknowledged after the worker finishes processing.
|
|
1207
|
+
- Set reasonable RabbitMQ consumer prefetch (`spring.rabbitmq.listener.simple.prefetch=1`)
|
|
1208
|
+
to prevent a single worker from hoarding all partitions.
|
|
1209
|
+
|
|
1210
|
+
### Dead Letter Queue (DLQ)
|
|
1211
|
+
|
|
1212
|
+
The request queue is configured with a DLQ (`batch.partition.requests.dlq`). Messages
|
|
1213
|
+
that are rejected (e.g., deserialization failure, poison messages) are routed to the DLQ
|
|
1214
|
+
for inspection rather than being lost or infinitely retried.
|
|
1215
|
+
|
|
1216
|
+
Monitor the DLQ in production and alert on message accumulation.
|
|
1217
|
+
|
|
1218
|
+
### Monitoring
|
|
1219
|
+
|
|
1220
|
+
- Use RabbitMQ Management UI (port `15672`) to inspect queue depths, consumer counts,
|
|
1221
|
+
and message rates.
|
|
1222
|
+
- Use Spring Batch's `JobExplorer` to query partition execution status programmatically.
|
|
1223
|
+
- Consider exposing a health indicator that checks RabbitMQ connectivity
|
|
1224
|
+
(`management.health.rabbit.enabled: true`).
|