@groupby/ai-dev 0.5.13 → 0.5.15
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/teams/brain-studio/skills/arch-context/SKILL.md +135 -0
- package/teams/brain-studio/skills/draft-plan/SKILL.md +222 -0
- package/teams/brain-studio/skills/draft-pr/SKILL.md +291 -0
- package/teams/brain-studio/skills/git-context/SKILL.md +159 -0
- package/teams/brain-studio/skills/html/SKILL.md +233 -0
- package/teams/brain-studio/skills/javascript/SKILL.md +680 -0
- package/teams/brain-studio/skills/jira-context/SKILL.md +113 -0
- package/teams/brain-studio/skills/jira-spec/SKILL.md +247 -0
- package/teams/brain-studio/skills/pr-review/SKILL.md +310 -0
- package/teams/brain-studio/skills/react/SKILL.md +282 -0
- package/teams/brain-studio/skills/styled-components/SKILL.md +336 -0
- package/teams/brain-studio/skills/tdd-implement/SKILL.md +278 -0
- package/teams/brain-studio/skills/typescript/SKILL.md +491 -0
- package/teams/fhr-andromeda/github/PULL_REQUEST_TEMPLATE.md +51 -0
- package/teams/fhr-andromeda/github/copilot-instructions.md +47 -0
- package/teams/fhr-andromeda/instructions/CLAUDE.md +66 -0
- package/teams/fhr-andromeda/instructions/ai-instructions.md +312 -0
- package/teams/fhr-andromeda/instructions/architecture.md +181 -0
- package/teams/fhr-andromeda/instructions/code-review.md +127 -0
- package/teams/fhr-andromeda/instructions/coding-guidelines.md +139 -0
- package/teams/fhr-andromeda/instructions/domain.md +125 -0
- package/teams/fhr-andromeda/instructions/jira-ticket-template.md +55 -0
- package/teams/fhr-andromeda/instructions/modules/data-quality-frontend.md +57 -0
- package/teams/fhr-andromeda/instructions/modules/data-quality.md +62 -0
- package/teams/fhr-andromeda/instructions/modules/fhr-custom-transforms.md +59 -0
- package/teams/fhr-andromeda/instructions/modules/hop-dev-platform-frontend.md +60 -0
- package/teams/fhr-andromeda/instructions/modules/hop-dev-platform.md +62 -0
- package/teams/fhr-andromeda/instructions/modules/workflow-scripts.md +119 -0
- package/teams/fhr-andromeda/skills/fhr-pr/SKILL.md +71 -0
- package/teams/fhr-andromeda/skills/fhr-review/SKILL.md +78 -0
- package/teams/fhr-andromeda/skills/fhr-spec/SKILL.md +96 -0
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# CLAUDE.md - Repository Context
|
|
2
|
+
|
|
3
|
+
This file is the repository entrypoint for AI agents. It explains what this repo contains and
|
|
4
|
+
which harness file owns each kind of guidance.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Repository Purpose
|
|
9
|
+
|
|
10
|
+
`fhr-apache-hop` contains application code for Fredhopper ETL 2.0, a Kubernetes-native data
|
|
11
|
+
pipeline platform built around Apache Hop.
|
|
12
|
+
|
|
13
|
+
Infrastructure definitions live in `andromeda-argo-deployments`. See `.ai/architecture.md` for
|
|
14
|
+
the repository boundary definition.
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Harness File Ownership
|
|
19
|
+
|
|
20
|
+
| File | Owns |
|
|
21
|
+
|---|---|
|
|
22
|
+
| `.github/copilot-instructions.md` | Copilot bootstrap and context-loading order |
|
|
23
|
+
| `CLAUDE.md` | Repository map and harness ownership |
|
|
24
|
+
| `.ai/architecture.md` | Stable system boundaries, ETL workflow structure, and module responsibilities |
|
|
25
|
+
| `.ai/domain.md` | Business terminology and domain concepts |
|
|
26
|
+
| `.ai/ai-instructions.md` | AI workflow, clarification, TDD, stop conditions, verification |
|
|
27
|
+
| `.ai/coding-guidelines.md` | Coding conventions and implementation rules |
|
|
28
|
+
| `.ai/code-review.md` | Review severity and checklist |
|
|
29
|
+
| `.ai/jira-ticket-template.md` | Full Jira ticket spec template |
|
|
30
|
+
| `<module>/CLAUDE.md` | Module-specific constraints and integration context |
|
|
31
|
+
|
|
32
|
+
Keep each file focused on its owner topic. Do not duplicate detailed rules across files.
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Systems in This Repository
|
|
37
|
+
|
|
38
|
+
| System | Modules | Purpose |
|
|
39
|
+
|---|---|---|
|
|
40
|
+
| ETL Engine | `fhr-custom-transforms`, `standard-etl-transformations`, `fhr-hop-server` | Apache Hop extensions, packaged ETL flows, and runtime image assembly |
|
|
41
|
+
| Workflow Scripts | `workflow-scripts` | Python logic executed by Argo Workflow steps |
|
|
42
|
+
| HDSys | `hop-dev-platform`, `hop-dev-platform-frontend` | Browser-based Hop workspace platform for ETL customisation |
|
|
43
|
+
| DQSys | `data-quality`, `data-quality-frontend` | Data-quality reporting service and UI |
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## Working in a Module
|
|
48
|
+
|
|
49
|
+
Before changing a module:
|
|
50
|
+
|
|
51
|
+
1. Read this file and the root context files listed above.
|
|
52
|
+
2. Read the module's `CLAUDE.md`.
|
|
53
|
+
3. Inspect existing code in the target area instead of relying on stale inventories.
|
|
54
|
+
4. Follow `.ai/ai-instructions.md` for workflow and `.ai/coding-guidelines.md` for
|
|
55
|
+
implementation style.
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## Build Entry Points
|
|
60
|
+
|
|
61
|
+
Use the root Maven project for Java and packaging modules. Use each Angular module's package
|
|
62
|
+
scripts for frontend modules. Prefer module-scoped builds unless a shared parent or cross-module
|
|
63
|
+
contract changed.
|
|
64
|
+
|
|
65
|
+
Exact commands are in `.ai/ai-instructions.md` §8. Use the module's current build files as the
|
|
66
|
+
authoritative source for profiles and flags.
|
|
@@ -0,0 +1,312 @@
|
|
|
1
|
+
# AI Instructions - fhr-apache-hop
|
|
2
|
+
|
|
3
|
+
This file owns AI operating rules: how to start, clarify, plan, generate, verify, and stop.
|
|
4
|
+
Coding style belongs in `.ai/coding-guidelines.md`; architecture belongs in `.ai/architecture.md`;
|
|
5
|
+
module-specific constraints belong in the relevant module `CLAUDE.md`.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## 1. Operating Priority
|
|
10
|
+
|
|
11
|
+
Apply instructions in this order:
|
|
12
|
+
|
|
13
|
+
1. System/developer instructions.
|
|
14
|
+
2. `.github/copilot-instructions.md`.
|
|
15
|
+
3. This file.
|
|
16
|
+
4. Relevant module `CLAUDE.md`.
|
|
17
|
+
5. `.ai/coding-guidelines.md`.
|
|
18
|
+
6. Existing local code patterns.
|
|
19
|
+
|
|
20
|
+
If conflict remains, stop and ask.
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## 2. Mandatory Pre-Flight
|
|
25
|
+
|
|
26
|
+
Before technical work:
|
|
27
|
+
|
|
28
|
+
1. Read root `CLAUDE.md`.
|
|
29
|
+
2. Read `.ai/architecture.md`.
|
|
30
|
+
3. Read `.ai/domain.md`.
|
|
31
|
+
4. Read `.ai/coding-guidelines.md`.
|
|
32
|
+
5. Read the relevant module `CLAUDE.md` when a module is in scope.
|
|
33
|
+
6. Identify acceptance criteria and missing context.
|
|
34
|
+
|
|
35
|
+
Do not infer contracts from stale examples. Inspect current source where needed.
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## 3. Task Context File
|
|
40
|
+
|
|
41
|
+
For functional code-generation tasks, create and maintain a task context file:
|
|
42
|
+
|
|
43
|
+
```
|
|
44
|
+
tasks/TASK-<TICKET-ID>.md
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Add `tasks/` to `.gitignore` — these are working files, not source artefacts.
|
|
48
|
+
|
|
49
|
+
The file should track: task title, ticket, module, and agent type; current phase;
|
|
50
|
+
clarification Q&A; TDD plan; files changed; build/verification result; blockers or stop
|
|
51
|
+
conditions.
|
|
52
|
+
|
|
53
|
+
Minimal heading structure:
|
|
54
|
+
|
|
55
|
+
```markdown
|
|
56
|
+
# Task: <title>
|
|
57
|
+
Ticket: <id> | Module: <module> | Agent: <type> | Phase: <current>
|
|
58
|
+
|
|
59
|
+
## Clarifications
|
|
60
|
+
## TDD Plan
|
|
61
|
+
## Files
|
|
62
|
+
## Build Result
|
|
63
|
+
## Blockers
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
If no ticket exists, ask whether a task context file is required before creating one.
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## 4. Clarification Protocol
|
|
71
|
+
|
|
72
|
+
Ask one batched clarification round before code when:
|
|
73
|
+
|
|
74
|
+
- acceptance criteria cannot become concrete tests without guessing;
|
|
75
|
+
- module scope is unclear;
|
|
76
|
+
- an API, data, environment, or configuration contract is missing;
|
|
77
|
+
- the task needs a new dependency, module, migration, or cross-repo change;
|
|
78
|
+
- the request conflicts with documented architecture or coding rules.
|
|
79
|
+
|
|
80
|
+
Write questions clearly, grouped by topic. Wait for answers before implementation.
|
|
81
|
+
|
|
82
|
+
If no clarification is needed, state that and proceed to the TDD plan.
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## 5. TDD Workflow for Functional Changes
|
|
87
|
+
|
|
88
|
+
Functional code tasks must follow this order.
|
|
89
|
+
|
|
90
|
+
### Phase 1 - TDD Plan
|
|
91
|
+
|
|
92
|
+
Before writing code, output:
|
|
93
|
+
|
|
94
|
+
1. Numbered test cases with name, given/when/then, and type.
|
|
95
|
+
2. Files to create or modify, labelled by role.
|
|
96
|
+
3. Output order.
|
|
97
|
+
4. Mermaid diagram showing test, production, migration, and config dependencies.
|
|
98
|
+
|
|
99
|
+
End with:
|
|
100
|
+
|
|
101
|
+
> **Awaiting approval.** Reply with `APPROVED`, `lgtm`, or `proceed` to continue to Phase 2.
|
|
102
|
+
|
|
103
|
+
Stop until approval is received.
|
|
104
|
+
|
|
105
|
+
### Phase 2 - Tests Red
|
|
106
|
+
|
|
107
|
+
Write complete tests first. They must compile and be expected to fail because production
|
|
108
|
+
behaviour is not implemented yet.
|
|
109
|
+
|
|
110
|
+
State:
|
|
111
|
+
|
|
112
|
+
> Tests written. These will fail (red) until Phase 3 is complete.
|
|
113
|
+
|
|
114
|
+
### Phase 3 - Production Green
|
|
115
|
+
|
|
116
|
+
Write the minimum production code needed to satisfy the tests.
|
|
117
|
+
|
|
118
|
+
### Phase 4 - Refactor
|
|
119
|
+
|
|
120
|
+
Clean up without changing behaviour. Remove duplication, unused imports, dead methods,
|
|
121
|
+
placeholders, and stale tests.
|
|
122
|
+
|
|
123
|
+
State:
|
|
124
|
+
|
|
125
|
+
> Refactor complete. All tests remain green.
|
|
126
|
+
|
|
127
|
+
### Phase 5 - Build Verification
|
|
128
|
+
|
|
129
|
+
Run the appropriate module build and fix failures before declaring done. Report the result
|
|
130
|
+
clearly.
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
## 6. Output Order
|
|
135
|
+
|
|
136
|
+
For functional changes, produce output in this order:
|
|
137
|
+
|
|
138
|
+
1. TDD plan.
|
|
139
|
+
2. Test files.
|
|
140
|
+
3. Production files.
|
|
141
|
+
4. Migration files, if required.
|
|
142
|
+
5. Config/property changes, if required.
|
|
143
|
+
6. Brief review note.
|
|
144
|
+
|
|
145
|
+
Do not interleave test and production output.
|
|
146
|
+
|
|
147
|
+
---
|
|
148
|
+
|
|
149
|
+
## 7. Stop Conditions
|
|
150
|
+
|
|
151
|
+
Stop and notify the developer when:
|
|
152
|
+
|
|
153
|
+
- Phase 1 approval has not been received for a functional code task.
|
|
154
|
+
- The task would modify an existing Flyway migration.
|
|
155
|
+
- The task would create a new Maven/module boundary.
|
|
156
|
+
- The task requires an unapproved dependency or framework.
|
|
157
|
+
- The task requires infrastructure or workflow template changes outside this repo.
|
|
158
|
+
- Acceptance criteria contradict documented architecture or API contracts.
|
|
159
|
+
- A red-phase test unexpectedly passes before production code exists.
|
|
160
|
+
- Compilation or build verification still fails after three focused fix attempts.
|
|
161
|
+
- Required business, API, data, security, or deployment context is missing.
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
165
|
+
## 8. Verification Rules
|
|
166
|
+
|
|
167
|
+
After modifying runnable code:
|
|
168
|
+
|
|
169
|
+
- Check edited files for IDE/type errors where possible.
|
|
170
|
+
- Run the smallest relevant build command that proves the change:
|
|
171
|
+
- Java: `mvn clean install -pl <module> -am --no-transfer-progress`
|
|
172
|
+
- Angular: `ng build --configuration development` (from the module directory)
|
|
173
|
+
- Use a full build when parent configuration, shared code, or cross-module contracts changed.
|
|
174
|
+
- Do not skip failing tests or weaken assertions to pass verification.
|
|
175
|
+
- If tooling is unavailable, state exactly what could not be verified.
|
|
176
|
+
|
|
177
|
+
Commit messages must follow conventional-commit format: `feat(scope): description` or
|
|
178
|
+
`fix(scope): description`.
|
|
179
|
+
|
|
180
|
+
Documentation-only changes do not require application builds, but should still be checked for
|
|
181
|
+
Markdown or obvious consistency issues.
|
|
182
|
+
|
|
183
|
+
---
|
|
184
|
+
|
|
185
|
+
## 9. Harness Gap Protocol
|
|
186
|
+
|
|
187
|
+
If correct generation requires missing context:
|
|
188
|
+
|
|
189
|
+
1. Do not guess.
|
|
190
|
+
2. State exactly what is missing.
|
|
191
|
+
3. Identify which harness or module document should own that missing context.
|
|
192
|
+
4. Ask for clarification before implementing.
|
|
193
|
+
|
|
194
|
+
Harness gaps should be fixed in context files, not bypassed with low-confidence code.
|
|
195
|
+
|
|
196
|
+
---
|
|
197
|
+
|
|
198
|
+
## 10. Task Sizing Guidance
|
|
199
|
+
|
|
200
|
+
Prefer tasks that:
|
|
201
|
+
|
|
202
|
+
- touch one module;
|
|
203
|
+
- have concrete acceptance criteria;
|
|
204
|
+
- avoid unrelated refactoring;
|
|
205
|
+
- separate API, persistence, frontend, and infrastructure concerns when practical.
|
|
206
|
+
|
|
207
|
+
If a task is too broad, recommend splitting it before implementation.
|
|
208
|
+
|
|
209
|
+
---
|
|
210
|
+
|
|
211
|
+
## 11. Autonomous Jira Workflow
|
|
212
|
+
|
|
213
|
+
This section describes the end-to-end flow when work starts from a Jira ticket ID.
|
|
214
|
+
|
|
215
|
+
### Step 1 — Ticket Ingestion
|
|
216
|
+
|
|
217
|
+
When given a Jira ticket ID:
|
|
218
|
+
|
|
219
|
+
1. If the Atlassian MCP is available, fetch the ticket using `getJiraIssue`. If no Jira MCP
|
|
220
|
+
is configured, ask the developer to paste the ticket title, description, and acceptance
|
|
221
|
+
criteria before continuing.
|
|
222
|
+
2. Read the title, description, and any existing acceptance criteria.
|
|
223
|
+
3. Run a single batched interrogation round — ask only what cannot be derived from the
|
|
224
|
+
ticket, harness files, or current source.
|
|
225
|
+
4. Once interrogation is complete, produce a structured AI Spec (see format below) and:
|
|
226
|
+
- Present it in the conversation for review.
|
|
227
|
+
- If the Atlassian MCP is available, append it to the Jira ticket description as a new
|
|
228
|
+
`## AI Spec` section using `editJiraIssue` (append; do not overwrite the original
|
|
229
|
+
description). Skip this sub-step when no MCP is configured.
|
|
230
|
+
|
|
231
|
+
### Step 2 — Spec Approval
|
|
232
|
+
|
|
233
|
+
Do not write any code or create a branch until the spec is explicitly approved. End the
|
|
234
|
+
spec presentation with:
|
|
235
|
+
|
|
236
|
+
> **Awaiting spec approval.** Reply with `APPROVED`, `lgtm`, or `proceed` to continue.
|
|
237
|
+
|
|
238
|
+
Stop until approval is received. Spec approval is Phase 1 approval for the TDD workflow.
|
|
239
|
+
|
|
240
|
+
### Step 3 — Branch Creation
|
|
241
|
+
|
|
242
|
+
After spec approval, create the working branch from `master`:
|
|
243
|
+
|
|
244
|
+
```bash
|
|
245
|
+
git checkout -b <TICKET-ID>-<short-kebab-description>
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
The branch name must include the Jira ticket key (e.g. `FHR-7108`) so Jira auto-links the
|
|
249
|
+
branch in its development panel. No MCP call is needed — Jira detects the key automatically.
|
|
250
|
+
|
|
251
|
+
### Step 4 — Implementation (TDD Phases 2–5)
|
|
252
|
+
|
|
253
|
+
Follow the TDD workflow in §5. Commit after each meaningful unit:
|
|
254
|
+
|
|
255
|
+
- A passing test suite for a new behaviour.
|
|
256
|
+
- A completed service method with its tests.
|
|
257
|
+
- A migration paired with the model change it supports.
|
|
258
|
+
|
|
259
|
+
Do not batch unrelated changes into one commit. Each commit message must follow
|
|
260
|
+
conventional-commit format: `feat(scope): description` or `fix(scope): description`.
|
|
261
|
+
|
|
262
|
+
### Step 5 — PR Creation
|
|
263
|
+
|
|
264
|
+
After Phase 5 build verification passes:
|
|
265
|
+
|
|
266
|
+
1. Push the branch and open a PR against `master` using `gh pr create`.
|
|
267
|
+
2. PR title: `<TICKET-ID>: <short description>`.
|
|
268
|
+
3. PR body must include:
|
|
269
|
+
- Summary of what changed compared to `master` and why (from the spec objective).
|
|
270
|
+
- Notable implementation decisions or constraints applied.
|
|
271
|
+
- Testing evidence (build result, test count or coverage delta).
|
|
272
|
+
- Link to the Jira ticket.
|
|
273
|
+
4. If the Atlassian MCP is available, post the PR URL as a comment on the Jira ticket using
|
|
274
|
+
`addCommentToJiraIssue`. Skip this step when no MCP is configured.
|
|
275
|
+
|
|
276
|
+
### AI Spec Format
|
|
277
|
+
|
|
278
|
+
```markdown
|
|
279
|
+
## AI Spec
|
|
280
|
+
|
|
281
|
+
**Ticket**: <id> **Module**: <module> **Agent**: <type>
|
|
282
|
+
|
|
283
|
+
### Objective
|
|
284
|
+
One or two sentences — what this achieves and why.
|
|
285
|
+
|
|
286
|
+
### Out of Scope
|
|
287
|
+
|
|
288
|
+
### Functional Requirements
|
|
289
|
+
1. Given … when … then …
|
|
290
|
+
|
|
291
|
+
### Technical Constraints
|
|
292
|
+
Architecture pattern, security requirements, cross-repo impact, new dependencies
|
|
293
|
+
(require explicit approval).
|
|
294
|
+
|
|
295
|
+
### Acceptance Criteria
|
|
296
|
+
- [ ] Given … when … then …
|
|
297
|
+
- [ ] All existing tests pass.
|
|
298
|
+
- [ ] BUILD SUCCESS confirmed locally.
|
|
299
|
+
- [ ] No TODO, FIXME, placeholder comments, or commented-out code.
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
---
|
|
303
|
+
|
|
304
|
+
## 12. Jira Spec Expectations
|
|
305
|
+
|
|
306
|
+
A useful ticket for AI generation should include: agent type and module; objective and
|
|
307
|
+
out-of-scope items; functional requirements in given/when/then form; API, data, security, and
|
|
308
|
+
config contracts when relevant; dependency and cross-repo impact notes; acceptance criteria and
|
|
309
|
+
definition of done.
|
|
310
|
+
|
|
311
|
+
Use the AI Spec Format in §11 as the canonical format when writing or consuming specs. The full
|
|
312
|
+
human-facing ticket template is in `.ai/jira-ticket-template.md`.
|
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
# Architecture - fhr-apache-hop
|
|
2
|
+
|
|
3
|
+
This file owns stable system architecture: what each system is responsible for, how modules
|
|
4
|
+
interact, and where this repository's boundaries are.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Repository Boundary
|
|
9
|
+
|
|
10
|
+
This repository contains application code for ETL 2.0. Kubernetes manifests, Helm charts, Argo
|
|
11
|
+
Workflow templates, and cluster deployment definitions are maintained in the separate
|
|
12
|
+
`andromeda-argo-deployments` repository.
|
|
13
|
+
|
|
14
|
+
If a task changes runtime orchestration, workflow templates, cluster permissions, or deployment
|
|
15
|
+
shape, flag the cross-repo impact instead of generating infrastructure changes here.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## System Overview
|
|
20
|
+
|
|
21
|
+
```mermaid
|
|
22
|
+
flowchart LR
|
|
23
|
+
subgraph HDSys[HDSys]
|
|
24
|
+
HDFE[HDSys UI]
|
|
25
|
+
HDBE[HDSys Backend]
|
|
26
|
+
HDFE --> HDBE
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
subgraph ETL[ETL Engine]
|
|
30
|
+
Hop[Hop Runtime]
|
|
31
|
+
Plugins[Custom Transforms]
|
|
32
|
+
Packages[Standard ETL Packages]
|
|
33
|
+
Hop --> Plugins
|
|
34
|
+
Hop --> Packages
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
subgraph WF[Workflow Scripts]
|
|
38
|
+
Steps[Argo Step Logic]
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
subgraph DQSys[DQSys]
|
|
42
|
+
DQFE[DQSys UI]
|
|
43
|
+
DQBE[DQSys Backend]
|
|
44
|
+
DB[(PostgreSQL)]
|
|
45
|
+
DQFE --> DQBE --> DB
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
Storage[(Object Storage)]
|
|
49
|
+
Events[External Workflow Orchestration]
|
|
50
|
+
|
|
51
|
+
Events --> Steps
|
|
52
|
+
Steps --> Hop
|
|
53
|
+
Hop --> Storage
|
|
54
|
+
Steps --> DQBE
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## System Responsibilities
|
|
60
|
+
|
|
61
|
+
### ETL Engine
|
|
62
|
+
|
|
63
|
+
Owns Apache Hop runtime extensions and standard ETL packages. It transforms customer catalogue
|
|
64
|
+
inputs into outputs consumed by downstream Fredhopper systems (FAS XML files, suggestion
|
|
65
|
+
indices, upload packages).
|
|
66
|
+
|
|
67
|
+
### Workflow Scripts
|
|
68
|
+
|
|
69
|
+
Own Python logic executed by Argo workflow steps. Scripts orchestrate job setup, data movement,
|
|
70
|
+
runtime invocation, result publication, metrics, and DQ integration. Workflow definitions live
|
|
71
|
+
in `andromeda-argo-deployments`.
|
|
72
|
+
|
|
73
|
+
### HDSys
|
|
74
|
+
|
|
75
|
+
Owns the development platform for ETL customisation. It provisions browser-based Hop workspaces
|
|
76
|
+
for consultants and manages the GitOps workflow for customer ETL packages. It is a
|
|
77
|
+
development-time system and does not participate in ETL run execution.
|
|
78
|
+
|
|
79
|
+
### DQSys
|
|
80
|
+
|
|
81
|
+
Owns data-quality reporting and ETL Dashboard data. Workflow steps send quality and operational
|
|
82
|
+
signals after each run, and the DQSys UI presents them to users.
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## Module Responsibilities
|
|
87
|
+
|
|
88
|
+
| Module | Responsibility |
|
|
89
|
+
|---|---|
|
|
90
|
+
| `fhr-custom-transforms` | Fredhopper-specific Hop transform implementations |
|
|
91
|
+
| `standard-etl-transformations` | Standard reusable ETL packages |
|
|
92
|
+
| `fhr-hop-server` | Hop runtime packaging and assembly |
|
|
93
|
+
| `workflow-scripts` | Workflow-step application logic |
|
|
94
|
+
| `hop-dev-platform` | HDSys backend APIs and integrations |
|
|
95
|
+
| `hop-dev-platform-frontend` | HDSys user interface |
|
|
96
|
+
| `data-quality` | DQSys backend APIs and persistence |
|
|
97
|
+
| `data-quality-frontend` | DQSys user interface |
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## ETL Workflow Structure
|
|
102
|
+
|
|
103
|
+
An ETL run is an Argo workflow triggered by a `trigger.json` file placed in S3. Steps run in
|
|
104
|
+
this order:
|
|
105
|
+
|
|
106
|
+
1. **extract-and-dispatch** — determines whether input is a zip (batch) or stream, extracts
|
|
107
|
+
if needed, and resolves the Hop project name from service type and data format.
|
|
108
|
+
2. **initialize-job** — assembles the `hop_job_config` JSON blob and writes it for all
|
|
109
|
+
downstream steps.
|
|
110
|
+
3. **create-pvc** + **metrics-daemon** — run in parallel; the PVC provides shared storage
|
|
111
|
+
between the data and XML Hop server pods.
|
|
112
|
+
4. **run-etl** — launches the Hop server pod. Init container: `prepare_data.py`. Main
|
|
113
|
+
container: `run_and_monitor.py`. Sidecar: hop-server process.
|
|
114
|
+
5. **generate-xml** — same pod structure with the XML project config; produces FAS XML files.
|
|
115
|
+
Runs for `fas` service type only when `generate_xml` is true.
|
|
116
|
+
6. **submit-dq-workflow** — submits a fire-and-forget child workflow (`dq_execution.py`).
|
|
117
|
+
Runs for `fas` only; does not block the main DAG.
|
|
118
|
+
7. **publish-results** — uploads outputs and logs to S3; sends completion signals to Kafka.
|
|
119
|
+
8. **configure-kafka** — creates Kafka topics; runs for streaming (`load-data-stream`) only.
|
|
120
|
+
9. **submit-populate-dashboard** — submits a fire-and-forget child workflow
|
|
121
|
+
(`populate_dashboard.py`).
|
|
122
|
+
|
|
123
|
+
---
|
|
124
|
+
|
|
125
|
+
## hop_job_config — Central Run Contract
|
|
126
|
+
|
|
127
|
+
`initialize_job.py` produces a `hop_job_config` JSON blob passed to every downstream step.
|
|
128
|
+
It is the single source of truth for per-run identity (customer, instance, service_type,
|
|
129
|
+
trigger_id), S3 paths, resource sizing, Hop project configuration, database and Kafka
|
|
130
|
+
coordinates, and incremental state.
|
|
131
|
+
|
|
132
|
+
All workflow scripts consume it via `apply_hop_job_config()` from `arg_utils.py`, which merges
|
|
133
|
+
its fields onto the script's parsed argument namespace. Scripts access fields directly as
|
|
134
|
+
`args.customer`, `args.service_type`, etc.
|
|
135
|
+
|
|
136
|
+
`initialize_job.py` is the authoritative definition of all available fields. Scripts must not
|
|
137
|
+
re-derive what is already present in the blob.
|
|
138
|
+
|
|
139
|
+
---
|
|
140
|
+
|
|
141
|
+
## DQSys Population Paths
|
|
142
|
+
|
|
143
|
+
DQSys is populated through three distinct paths with different triggers and targets:
|
|
144
|
+
|
|
145
|
+
| Path | Script | Trigger | Writes |
|
|
146
|
+
|---|---|---|---|
|
|
147
|
+
| DQ checks | `dq_execution.py` | After data ETL, `fas` only | `check_registry`, `check_issues`, `run_log` in DQ DB |
|
|
148
|
+
| Run metadata | `populate_dashboard.py` | After publish-results, all service types | One row in `{customer}_{instance}_etl_dashboard` in DQ DB |
|
|
149
|
+
| Reindex completion | `update_etl_dashboard.py` (Deployment) | Kafka: FAS `load-xml` or suggest `load-idx` | Updates `total_run_time` and `complete_status` on the existing row |
|
|
150
|
+
|
|
151
|
+
`dq_execution.py` uses a two-database connection (`db_utils.DatabaseConnection`): it reads SQL
|
|
152
|
+
checks against the **ETL database** (schema `{customer}_{instance}`) and writes results to the
|
|
153
|
+
**DQ database** (`data_quality`). The other two paths write only to the DQ database.
|
|
154
|
+
|
|
155
|
+
---
|
|
156
|
+
|
|
157
|
+
## prepare_data — PVC Setup
|
|
158
|
+
|
|
159
|
+
`prepare_data.py` runs as the init container of every Hop server pod and chains three steps:
|
|
160
|
+
|
|
161
|
+
1. `deploy_standard_etl_package.py` — deploys the standard ETL ZIP for the Hop project.
|
|
162
|
+
2. `deploy_custom_etl_package.py` — sparse-clones the customer's directory from the
|
|
163
|
+
`custom-etl` GitHub repository, using `hop_job_config.custom_etl_branch` if set.
|
|
164
|
+
3. `map_etl_directories.py` — writes S3 path mappings into the Hop project config on the PVC.
|
|
165
|
+
|
|
166
|
+
Customer customisations merged into `custom-etl` take effect on the next ETL run; there is no
|
|
167
|
+
active sync between HDSys and the ETL runtime.
|
|
168
|
+
|
|
169
|
+
---
|
|
170
|
+
|
|
171
|
+
## Stable Architecture Constraints
|
|
172
|
+
|
|
173
|
+
- `data-quality` is reactive (Spring WebFlux + R2DBC) and must remain non-blocking.
|
|
174
|
+
- `hop-dev-platform` is synchronous (Spring MVC) and must not adopt reactive APIs.
|
|
175
|
+
- Frontend modules communicate through backend APIs only; no direct database or infrastructure
|
|
176
|
+
access.
|
|
177
|
+
- Workflow scripts are entrypoint logic for orchestration steps, not workflow definitions.
|
|
178
|
+
- Infrastructure and workflow template changes belong in `andromeda-argo-deployments`.
|
|
179
|
+
- Cross-module API changes must consider their consumers in the same change plan.
|
|
180
|
+
- `hop_job_config` is the single source of truth for per-run state; scripts must not
|
|
181
|
+
re-derive what is already in it.
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
# Code Review Checklist - fhr-apache-hop
|
|
2
|
+
|
|
3
|
+
This file owns review criteria. It operationalizes `.ai/coding-guidelines.md`,
|
|
4
|
+
`.ai/architecture.md`, and the relevant module `CLAUDE.md`.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Severity
|
|
9
|
+
|
|
10
|
+
- **BLOCK** - must be fixed before merge; violates safety, architecture, or hard repository
|
|
11
|
+
rules.
|
|
12
|
+
- **WARN** - should be fixed; convention drift or maintainability risk.
|
|
13
|
+
- **NOTE** - optional suggestion or observation.
|
|
14
|
+
|
|
15
|
+
Every finding should include severity, location, impact, and a concise recommendation.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Universal BLOCK Checks
|
|
20
|
+
|
|
21
|
+
- Secrets, credentials, environment-specific URLs, or local paths are committed.
|
|
22
|
+
- Existing Flyway migrations are modified.
|
|
23
|
+
- New dependencies, frameworks, modules, or infrastructure scope appear without approval.
|
|
24
|
+
- Placeholders, `TODO`, `FIXME`, or commented-out code remain.
|
|
25
|
+
- Tests are missing for new or changed behaviour.
|
|
26
|
+
- Code contradicts documented architecture for the target module.
|
|
27
|
+
- Error handling hides failures or creates silent data loss.
|
|
28
|
+
- Security or authorization checks are bypassed.
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## Universal WARN Checks
|
|
33
|
+
|
|
34
|
+
- Code does not match local naming, formatting, or structure conventions.
|
|
35
|
+
- Duplicated logic or literals that should be extracted or reused.
|
|
36
|
+
- Comments explain obvious code instead of non-obvious constraints.
|
|
37
|
+
- Public contracts changed without updating consumers or documenting impact.
|
|
38
|
+
- Build, test, or verification evidence is missing.
|
|
39
|
+
- New code is broader than the task scope.
|
|
40
|
+
- `split()` with magic numeric indices used instead of constants, regex, or `pathlib`.
|
|
41
|
+
- Unused imports, dead methods, or unreachable branches left after refactoring.
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## Java Review Checks
|
|
46
|
+
|
|
47
|
+
- Lombok is used for boilerplate; no manually written getters, setters, or constructors.
|
|
48
|
+
**BLOCK** if absent.
|
|
49
|
+
- Spring beans use constructor injection (`@RequiredArgsConstructor`). **BLOCK** if
|
|
50
|
+
`@Autowired` field injection is used.
|
|
51
|
+
- Logging uses `@Slf4j` and `log.*`; no `System.out.println`. **BLOCK** if violated.
|
|
52
|
+
- Domain exceptions and centralized HTTP error mapping used where appropriate.
|
|
53
|
+
- Database integration tests use real Testcontainers, not in-memory substitutes. **BLOCK** if
|
|
54
|
+
violated.
|
|
55
|
+
|
|
56
|
+
### data-quality (reactive)
|
|
57
|
+
|
|
58
|
+
- All service and repository methods return `Mono<T>` or `Flux<T>`. **BLOCK** if synchronous
|
|
59
|
+
return types are introduced.
|
|
60
|
+
- No `.block()` calls anywhere in production or test code. **BLOCK**.
|
|
61
|
+
- Reactive tests use `StepVerifier`. **BLOCK** if `.block()` is used in tests instead.
|
|
62
|
+
|
|
63
|
+
### hop-dev-platform (synchronous)
|
|
64
|
+
|
|
65
|
+
- No `Mono`, `Flux`, or WebFlux patterns introduced. **BLOCK** if reactive types appear in
|
|
66
|
+
production code.
|
|
67
|
+
|
|
68
|
+
### fhr-custom-transforms (Hop plugins)
|
|
69
|
+
|
|
70
|
+
- Every new transform has both a `Meta` class and a `Data` class. **BLOCK** if either is
|
|
71
|
+
missing.
|
|
72
|
+
- `Meta` class is annotated with `@Transform(...)`. **BLOCK** if annotation is absent.
|
|
73
|
+
- Dialog/UI code uses SWT only; no Swing or JavaFX. **BLOCK** if violated.
|
|
74
|
+
- No SWT display-dependent code outside dialog classes (breaks headless CI). **BLOCK**.
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## Python Review Checks
|
|
79
|
+
|
|
80
|
+
- Function signatures are typed. **WARN** if absent.
|
|
81
|
+
- Scripts use `logger_config.py` for logging; no `print()` in production code. **BLOCK**.
|
|
82
|
+
- Scripts use `arg_utils.py` for argument parsing; no raw `argparse` or `sys.argv`. **BLOCK**.
|
|
83
|
+
- Fatal errors are logged with context (`exc_info=True`) and exit non-zero.
|
|
84
|
+
- No bare `except:` — specific exceptions only. **BLOCK**.
|
|
85
|
+
- External systems (S3, Kafka, Kubernetes, PostgreSQL) are mocked in unit tests.
|
|
86
|
+
- New scripts follow the mandatory structure: imports → logger → constants → helpers →
|
|
87
|
+
`main()` → `if __name__ == "__main__"`.
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
## Angular Review Checks
|
|
92
|
+
|
|
93
|
+
- UI components use PrimeNG only; no other UI libraries. **BLOCK**.
|
|
94
|
+
- HTTP calls are kept in services; `HttpClient` never called directly in components. **BLOCK**.
|
|
95
|
+
- Observable subscriptions include cleanup (`takeUntilDestroyed` or `async` pipe). **BLOCK**
|
|
96
|
+
if a `.subscribe()` call has no cleanup.
|
|
97
|
+
- API responses have reusable typed interfaces; `any` is avoided.
|
|
98
|
+
- Component and service specs cover changed behaviour.
|
|
99
|
+
|
|
100
|
+
---
|
|
101
|
+
|
|
102
|
+
## Database and Migration Checks
|
|
103
|
+
|
|
104
|
+
- Schema changes are represented by new migrations only; existing migrations are untouched.
|
|
105
|
+
- Migration names follow the Flyway convention (`V<major>_<minor>__<description>.sql`) with a
|
|
106
|
+
sequential version number — inspect existing migrations before choosing the next number.
|
|
107
|
+
- Persistence model changes and API contracts stay aligned.
|
|
108
|
+
- Migrations are backward-compatible where practical.
|
|
109
|
+
|
|
110
|
+
---
|
|
111
|
+
|
|
112
|
+
## Cross-Module Impact Checks
|
|
113
|
+
|
|
114
|
+
Flag when a change affects:
|
|
115
|
+
|
|
116
|
+
- backend API consumers;
|
|
117
|
+
- workflow step inputs, outputs, or environment contracts;
|
|
118
|
+
- frontend API models;
|
|
119
|
+
- database schema used outside one module;
|
|
120
|
+
- infrastructure or workflow definitions outside this repo.
|
|
121
|
+
|
|
122
|
+
When the `data-quality` REST API contract changes (endpoint paths, request or response fields),
|
|
123
|
+
explicitly check whether `dq_execution.py`, `dq_custom_checks.py`, and `populate_dashboard.py`
|
|
124
|
+
in `workflow-scripts` need to be updated in the same PR.
|
|
125
|
+
|
|
126
|
+
Do not require same-PR fixes for every impact, but require the impact to be explicit and
|
|
127
|
+
tracked.
|