@groupby/ai-dev 0.5.13 → 0.5.14

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@groupby/ai-dev",
-  "version": "0.5.13",
+  "version": "0.5.14",
   "description": "Interactive installer for Rezolve Ai development content",
   "type": "module",
   "bin": {

package/teams/fhr-andromeda/github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,51 @@
+## AI Generation Info
+
+- **Jira ticket:** <!-- e.g. ETL-1234 -->
+- **Agent type:** <!-- e.g. Java Dev - data-quality | Angular Dev - frontend | Python Dev - workflow-scripts -->
+- **AI tool:** <!-- e.g. Claude (claude-sonnet-4-6) | GitHub Copilot -->
+- **Spec referenced:** <!-- link to Jira spec or paste inline -->
+
+---
+
+## TDD Summary
+
+- **Tests written first:** <!-- yes / no — if no, explain why -->
+- **Test files:**
+  - <!-- src/test/java/... or src/integration-test/groovy/... -->
+- **Production files:**
+  - <!-- src/main/java/... -->
+
+---
+
+## What was generated
+
+<!-- Brief description of what the AI produced -->
+
+---
+
+## What was reviewed / changed after generation
+
+<!-- Any manual corrections made post-generation, and the reason -->
+<!-- If nothing was changed: "None — generated output accepted as-is" -->
+
+---
+
+## Review notes
+
+<!-- Areas that warrant careful human review — edge cases, security, reactive chains, migrations, etc. -->
+<!-- If none: "None identified" -->
+
+---
+
+## Checklist
+
+- [ ] Jira ticket referenced above
+- [ ] AI tool and agent type declared
+- [ ] Tests written before or alongside production code (TDD order)
+- [ ] Unit tests ≥ 90% coverage of new code, positive + negative cases
+- [ ] `BUILD SUCCESS` confirmed locally (`mvn clean install -pl <module> -am` or `ng build`)
+- [ ] No `TODO`, `FIXME`, or placeholder comments
+- [ ] No secrets, hardcoded URLs, or environment-specific values
+- [ ] Lombok used for all Java boilerplate; no manual getters/setters/constructors
+- [ ] Constructor injection used in all Spring beans (no `@Autowired` field injection)
+- [ ] Cross-repo impact checked — if `andromeda-argo-deployments` is affected, linked ticket exists

package/teams/fhr-andromeda/github/copilot-instructions.md

@@ -0,0 +1,47 @@
+# GitHub Copilot Repository Instructions
+
+This file is the Copilot bootstrap for this repository. It should stay short and only explain
+which context files to load and how to resolve conflicts.
+
+---
+
+## Mandatory Context
+
+Before answering technical questions or generating changes, read and apply:
+
+| File | Owns |
+|---|---|
+| [`CLAUDE.md`](../CLAUDE.md) | Repository map and harness file ownership |
+| [`.ai/architecture.md`](../.ai/architecture.md) | System boundaries, ETL workflow structure, and module responsibilities |
+| [`.ai/domain.md`](../.ai/domain.md) | Business terminology and ETL domain concepts |
+| [`.ai/ai-instructions.md`](../.ai/ai-instructions.md) | AI workflow, TDD protocol, stop conditions |
+| [`.ai/coding-guidelines.md`](../.ai/coding-guidelines.md) | Coding conventions and implementation rules |
+| Relevant module `CLAUDE.md` | Module-specific context and constraints |
+| [`.ai/jira-ticket-template.md`](../.ai/jira-ticket-template.md) | Full Jira ticket spec template |
+
+Use [`.ai/code-review.md`](../.ai/code-review.md) when reviewing code or diffs.
+
+---
+
+## Instruction Priority
+
+If guidance conflicts, apply this order:
+
+1. System/developer instructions.
+2. This bootstrap file.
+3. `.ai/ai-instructions.md`.
+4. Relevant module `CLAUDE.md`.
+5. `.ai/coding-guidelines.md`.
+6. Existing local code patterns.
+
+If a conflict still remains, stop and ask instead of guessing.
+
+---
+
+## Non-Negotiable Bootstrap Rules
+
+- For functional code tasks, follow the TDD workflow in `.ai/ai-instructions.md`.
+- Do not generate test or production code before Phase 1 approval.
+- Do not modify existing Flyway migrations.
+- Do not introduce new dependencies, modules, or infrastructure scope without explicit approval.
+- If required context is missing, declare a harness gap and wait for clarification.

package/teams/fhr-andromeda/instructions/CLAUDE.md

@@ -0,0 +1,66 @@
+# CLAUDE.md - Repository Context
+
+This file is the repository entrypoint for AI agents. It explains what this repo contains and
+which harness file owns each kind of guidance.
+
+---
+
+## Repository Purpose
+
+`fhr-apache-hop` contains application code for Fredhopper ETL 2.0, a Kubernetes-native data
+pipeline platform built around Apache Hop.
+
+Infrastructure definitions live in `andromeda-argo-deployments`. See `.ai/architecture.md` for
+the repository boundary definition.
+
+---
+
+## Harness File Ownership
+
+| File | Owns |
+|---|---|
+| `.github/copilot-instructions.md` | Copilot bootstrap and context-loading order |
+| `CLAUDE.md` | Repository map and harness ownership |
+| `.ai/architecture.md` | Stable system boundaries, ETL workflow structure, and module responsibilities |
+| `.ai/domain.md` | Business terminology and domain concepts |
+| `.ai/ai-instructions.md` | AI workflow, clarification, TDD, stop conditions, verification |
+| `.ai/coding-guidelines.md` | Coding conventions and implementation rules |
+| `.ai/code-review.md` | Review severity and checklist |
+| `.ai/jira-ticket-template.md` | Full Jira ticket spec template |
+| `<module>/CLAUDE.md` | Module-specific constraints and integration context |
+
+Keep each file focused on its owner topic. Do not duplicate detailed rules across files.
+
+---
+
+## Systems in This Repository
+
+| System | Modules | Purpose |
+|---|---|---|
+| ETL Engine | `fhr-custom-transforms`, `standard-etl-transformations`, `fhr-hop-server` | Apache Hop extensions, packaged ETL flows, and runtime image assembly |
+| Workflow Scripts | `workflow-scripts` | Python logic executed by Argo Workflow steps |
+| HDSys | `hop-dev-platform`, `hop-dev-platform-frontend` | Browser-based Hop workspace platform for ETL customisation |
+| DQSys | `data-quality`, `data-quality-frontend` | Data-quality reporting service and UI |
+
+---
+
+## Working in a Module
+
+Before changing a module:
+
+1. Read this file and the root context files listed above.
+2. Read the module's `CLAUDE.md`.
+3. Inspect existing code in the target area instead of relying on stale inventories.
+4. Follow `.ai/ai-instructions.md` for workflow and `.ai/coding-guidelines.md` for
+   implementation style.
+
+---
+
+## Build Entry Points
+
+Use the root Maven project for Java and packaging modules. Use each Angular module's package
+scripts for frontend modules. Prefer module-scoped builds unless a shared parent or cross-module
+contract changed.
+
+Exact commands are in `.ai/ai-instructions.md` §8. Use the module's current build files as the
+authoritative source for profiles and flags.

package/teams/fhr-andromeda/instructions/ai-instructions.md

@@ -0,0 +1,312 @@
+# AI Instructions - fhr-apache-hop
+
+This file owns AI operating rules: how to start, clarify, plan, generate, verify, and stop.
+Coding style belongs in `.ai/coding-guidelines.md`; architecture belongs in `.ai/architecture.md`;
+module-specific constraints belong in the relevant module `CLAUDE.md`.
+
+---
+
+## 1. Operating Priority
+
+Apply instructions in this order:
+
+1. System/developer instructions.
+2. `.github/copilot-instructions.md`.
+3. This file.
+4. Relevant module `CLAUDE.md`.
+5. `.ai/coding-guidelines.md`.
+6. Existing local code patterns.
+
+If conflict remains, stop and ask.
+
+---
+
+## 2. Mandatory Pre-Flight
+
+Before technical work:
+
+1. Read root `CLAUDE.md`.
+2. Read `.ai/architecture.md`.
+3. Read `.ai/domain.md`.
+4. Read `.ai/coding-guidelines.md`.
+5. Read the relevant module `CLAUDE.md` when a module is in scope.
+6. Identify acceptance criteria and missing context.
+
+Do not infer contracts from stale examples. Inspect current source where needed.
+
+---
+
+## 3. Task Context File
+
+For functional code-generation tasks, create and maintain a task context file:
+
+```
+tasks/TASK-<TICKET-ID>.md
+```
+
+Add `tasks/` to `.gitignore` — these are working files, not source artefacts.
+
+The file should track: task title, ticket, module, and agent type; current phase;
+clarification Q&A; TDD plan; files changed; build/verification result; blockers or stop
+conditions.
+
+Minimal heading structure:
+
+```markdown
+# Task: <title>
+Ticket: <id> | Module: <module> | Agent: <type> | Phase: <current>
+
+## Clarifications
+## TDD Plan
+## Files
+## Build Result
+## Blockers
+```
+
+If no ticket exists, ask whether a task context file is required before creating one.
+
+---
+
+## 4. Clarification Protocol
+
+Ask one batched clarification round before code when:
+
+- acceptance criteria cannot become concrete tests without guessing;
+- module scope is unclear;
+- an API, data, environment, or configuration contract is missing;
+- the task needs a new dependency, module, migration, or cross-repo change;
+- the request conflicts with documented architecture or coding rules.
+
+Write questions clearly, grouped by topic. Wait for answers before implementation.
+
+If no clarification is needed, state that and proceed to the TDD plan.
+
+---
+
+## 5. TDD Workflow for Functional Changes
+
+Functional code tasks must follow this order.
+
+### Phase 1 - TDD Plan
+
+Before writing code, output:
+
+1. Numbered test cases with name, given/when/then, and type.
+2. Files to create or modify, labelled by role.
+3. Output order.
+4. Mermaid diagram showing test, production, migration, and config dependencies.
+
+End with:
+
+> **Awaiting approval.** Reply with `APPROVED`, `lgtm`, or `proceed` to continue to Phase 2.
+
+Stop until approval is received.
+
+### Phase 2 - Tests Red
+
+Write complete tests first. They must compile and be expected to fail because production
+behaviour is not implemented yet.
+
+State:
+
+> Tests written. These will fail (red) until Phase 3 is complete.
+
+### Phase 3 - Production Green
+
+Write the minimum production code needed to satisfy the tests.
+
+### Phase 4 - Refactor
+
+Clean up without changing behaviour. Remove duplication, unused imports, dead methods,
+placeholders, and stale tests.
+
+State:
+
+> Refactor complete. All tests remain green.
+
+### Phase 5 - Build Verification
+
+Run the appropriate module build and fix failures before declaring done. Report the result
+clearly.
+
+---
+
+## 6. Output Order
+
+For functional changes, produce output in this order:
+
+1. TDD plan.
+2. Test files.
+3. Production files.
+4. Migration files, if required.
+5. Config/property changes, if required.
+6. Brief review note.
+
+Do not interleave test and production output.
+
+---
+
+## 7. Stop Conditions
+
+Stop and notify the developer when:
+
+- Phase 1 approval has not been received for a functional code task.
+- The task would modify an existing Flyway migration.
+- The task would create a new Maven/module boundary.
+- The task requires an unapproved dependency or framework.
+- The task requires infrastructure or workflow template changes outside this repo.
+- Acceptance criteria contradict documented architecture or API contracts.
+- A red-phase test unexpectedly passes before production code exists.
+- Compilation or build verification still fails after three focused fix attempts.
+- Required business, API, data, security, or deployment context is missing.
+
+---
+
+## 8. Verification Rules
+
+After modifying runnable code:
+
+- Check edited files for IDE/type errors where possible.
+- Run the smallest relevant build command that proves the change:
+  - Java: `mvn clean install -pl <module> -am --no-transfer-progress`
+  - Angular: `ng build --configuration development` (from the module directory)
+- Use a full build when parent configuration, shared code, or cross-module contracts changed.
+- Do not skip failing tests or weaken assertions to pass verification.
+- If tooling is unavailable, state exactly what could not be verified.
+
+Commit messages must follow conventional-commit format: `feat(scope): description` or
+`fix(scope): description`.
+
+Documentation-only changes do not require application builds, but should still be checked for
+Markdown or obvious consistency issues.
+
+---
+
+## 9. Harness Gap Protocol
+
+If correct generation requires missing context:
+
+1. Do not guess.
+2. State exactly what is missing.
+3. Identify which harness or module document should own that missing context.
+4. Ask for clarification before implementing.
+
+Harness gaps should be fixed in context files, not bypassed with low-confidence code.
+
+---
+
+## 10. Task Sizing Guidance
+
+Prefer tasks that:
+
+- touch one module;
+- have concrete acceptance criteria;
+- avoid unrelated refactoring;
+- separate API, persistence, frontend, and infrastructure concerns when practical.
+
+If a task is too broad, recommend splitting it before implementation.
+
+---
+
+## 11. Autonomous Jira Workflow
+
+This section describes the end-to-end flow when work starts from a Jira ticket ID.
+
+### Step 1 — Ticket Ingestion
+
+When given a Jira ticket ID:
+
+1. If the Atlassian MCP is available, fetch the ticket using `getJiraIssue`. If no Jira MCP
+   is configured, ask the developer to paste the ticket title, description, and acceptance
+   criteria before continuing.
+2. Read the title, description, and any existing acceptance criteria.
+3. Run a single batched interrogation round — ask only what cannot be derived from the
+   ticket, harness files, or current source.
+4. Once interrogation is complete, produce a structured AI Spec (see format below) and:
+   - Present it in the conversation for review.
+   - If the Atlassian MCP is available, append it to the Jira ticket description as a new
+     `## AI Spec` section using `editJiraIssue` (append; do not overwrite the original
+     description). Skip this sub-step when no MCP is configured.
+
+### Step 2 — Spec Approval
+
+Do not write any code or create a branch until the spec is explicitly approved. End the
+spec presentation with:
+
+> **Awaiting spec approval.** Reply with `APPROVED`, `lgtm`, or `proceed` to continue.
+
+Stop until approval is received. Spec approval is Phase 1 approval for the TDD workflow.
+
+### Step 3 — Branch Creation
+
+After spec approval, create the working branch from `master`:
+
+```bash
+git checkout -b <TICKET-ID>-<short-kebab-description>
+```
+
+The branch name must include the Jira ticket key (e.g. `FHR-7108`) so Jira auto-links the
+branch in its development panel. No MCP call is needed — Jira detects the key automatically.
+
+### Step 4 — Implementation (TDD Phases 2–5)
+
+Follow the TDD workflow in §5. Commit after each meaningful unit:
+
+- A passing test suite for a new behaviour.
+- A completed service method with its tests.
+- A migration paired with the model change it supports.
+
+Do not batch unrelated changes into one commit. Each commit message must follow
+conventional-commit format: `feat(scope): description` or `fix(scope): description`.
+
+### Step 5 — PR Creation
+
+After Phase 5 build verification passes:
+
+1. Push the branch and open a PR against `master` using `gh pr create`.
+2. PR title: `<TICKET-ID>: <short description>`.
+3. PR body must include:
+   - Summary of what changed compared to `master` and why (from the spec objective).
+   - Notable implementation decisions or constraints applied.
+   - Testing evidence (build result, test count or coverage delta).
+   - Link to the Jira ticket.
+4. If the Atlassian MCP is available, post the PR URL as a comment on the Jira ticket using
+   `addCommentToJiraIssue`. Skip this step when no MCP is configured.
+
+### AI Spec Format
+
+```markdown
+## AI Spec
+
+**Ticket**: <id>  **Module**: <module>  **Agent**: <type>
+
+### Objective
+One or two sentences — what this achieves and why.
+
+### Out of Scope
+
+### Functional Requirements
+1. Given … when … then …
+
+### Technical Constraints
+Architecture pattern, security requirements, cross-repo impact, new dependencies
+(require explicit approval).
+
+### Acceptance Criteria
+- [ ] Given … when … then …
+- [ ] All existing tests pass.
+- [ ] BUILD SUCCESS confirmed locally.
+- [ ] No TODO, FIXME, placeholder comments, or commented-out code.
+```
+
+---
+
+## 12. Jira Spec Expectations
+
+A useful ticket for AI generation should include: agent type and module; objective and
+out-of-scope items; functional requirements in given/when/then form; API, data, security, and
+config contracts when relevant; dependency and cross-repo impact notes; acceptance criteria and
+definition of done.
+
+Use the AI Spec Format in §11 as the canonical format when writing or consuming specs. The full
+human-facing ticket template is in `.ai/jira-ticket-template.md`.

package/teams/fhr-andromeda/instructions/architecture.md

@@ -0,0 +1,181 @@
+# Architecture - fhr-apache-hop
+
+This file owns stable system architecture: what each system is responsible for, how modules
+interact, and where this repository's boundaries are.
+
+---
+
+## Repository Boundary
+
+This repository contains application code for ETL 2.0. Kubernetes manifests, Helm charts, Argo
+Workflow templates, and cluster deployment definitions are maintained in the separate
+`andromeda-argo-deployments` repository.
+
+If a task changes runtime orchestration, workflow templates, cluster permissions, or deployment
+shape, flag the cross-repo impact instead of generating infrastructure changes here.
+
+---
+
+## System Overview
+
+```mermaid
+flowchart LR
+    subgraph HDSys[HDSys]
+        HDFE[HDSys UI]
+        HDBE[HDSys Backend]
+        HDFE --> HDBE
+    end
+
+    subgraph ETL[ETL Engine]
+        Hop[Hop Runtime]
+        Plugins[Custom Transforms]
+        Packages[Standard ETL Packages]
+        Hop --> Plugins
+        Hop --> Packages
+    end
+
+    subgraph WF[Workflow Scripts]
+        Steps[Argo Step Logic]
+    end
+
+    subgraph DQSys[DQSys]
+        DQFE[DQSys UI]
+        DQBE[DQSys Backend]
+        DB[(PostgreSQL)]
+        DQFE --> DQBE --> DB
+    end
+
+    Storage[(Object Storage)]
+    Events[External Workflow Orchestration]
+
+    Events --> Steps
+    Steps --> Hop
+    Hop --> Storage
+    Steps --> DQBE
+```
+
+---
+
+## System Responsibilities
+
+### ETL Engine
+
+Owns Apache Hop runtime extensions and standard ETL packages. It transforms customer catalogue
+inputs into outputs consumed by downstream Fredhopper systems (FAS XML files, suggestion
+indices, upload packages).
+
+### Workflow Scripts
+
+Own Python logic executed by Argo workflow steps. Scripts orchestrate job setup, data movement,
+runtime invocation, result publication, metrics, and DQ integration. Workflow definitions live
+in `andromeda-argo-deployments`.
+
+### HDSys
+
+Owns the development platform for ETL customisation. It provisions browser-based Hop workspaces
+for consultants and manages the GitOps workflow for customer ETL packages. It is a
+development-time system and does not participate in ETL run execution.
+
+### DQSys
+
+Owns data-quality reporting and ETL Dashboard data. Workflow steps send quality and operational
+signals after each run, and the DQSys UI presents them to users.
+
+---
+
+## Module Responsibilities
+
+| Module | Responsibility |
+|---|---|
+| `fhr-custom-transforms` | Fredhopper-specific Hop transform implementations |
+| `standard-etl-transformations` | Standard reusable ETL packages |
+| `fhr-hop-server` | Hop runtime packaging and assembly |
+| `workflow-scripts` | Workflow-step application logic |
+| `hop-dev-platform` | HDSys backend APIs and integrations |
+| `hop-dev-platform-frontend` | HDSys user interface |
+| `data-quality` | DQSys backend APIs and persistence |
+| `data-quality-frontend` | DQSys user interface |
+
+---
+
+## ETL Workflow Structure
+
+An ETL run is an Argo workflow triggered by a `trigger.json` file placed in S3. Steps run in
+this order:
+
+1. **extract-and-dispatch** — determines whether input is a zip (batch) or stream, extracts
+   if needed, and resolves the Hop project name from service type and data format.
+2. **initialize-job** — assembles the `hop_job_config` JSON blob and writes it for all
+   downstream steps.
+3. **create-pvc** + **metrics-daemon** — run in parallel; the PVC provides shared storage
+   between the data and XML Hop server pods.
+4. **run-etl** — launches the Hop server pod. Init container: `prepare_data.py`. Main
+   container: `run_and_monitor.py`. Sidecar: hop-server process.
+5. **generate-xml** — same pod structure with the XML project config; produces FAS XML files.
+   Runs for `fas` service type only when `generate_xml` is true.
+6. **submit-dq-workflow** — submits a fire-and-forget child workflow (`dq_execution.py`).
+   Runs for `fas` only; does not block the main DAG.
+7. **publish-results** — uploads outputs and logs to S3; sends completion signals to Kafka.
+8. **configure-kafka** — creates Kafka topics; runs for streaming (`load-data-stream`) only.
+9. **submit-populate-dashboard** — submits a fire-and-forget child workflow
+   (`populate_dashboard.py`).
+
+---
+
+## hop_job_config — Central Run Contract
+
+`initialize_job.py` produces a `hop_job_config` JSON blob passed to every downstream step.
+It is the single source of truth for per-run identity (customer, instance, service_type,
+trigger_id), S3 paths, resource sizing, Hop project configuration, database and Kafka
+coordinates, and incremental state.
+
+All workflow scripts consume it via `apply_hop_job_config()` from `arg_utils.py`, which merges
+its fields onto the script's parsed argument namespace. Scripts access fields directly as
+`args.customer`, `args.service_type`, etc.
+
+`initialize_job.py` is the authoritative definition of all available fields. Scripts must not
+re-derive what is already present in the blob.
+
+---
+
+## DQSys Population Paths
+
+DQSys is populated through three distinct paths with different triggers and targets:
+
+| Path | Script | Trigger | Writes |
+|---|---|---|---|
+| DQ checks | `dq_execution.py` | After data ETL, `fas` only | `check_registry`, `check_issues`, `run_log` in DQ DB |
+| Run metadata | `populate_dashboard.py` | After publish-results, all service types | One row in `{customer}_{instance}_etl_dashboard` in DQ DB |
+| Reindex completion | `update_etl_dashboard.py` (Deployment) | Kafka: FAS `load-xml` or suggest `load-idx` | Updates `total_run_time` and `complete_status` on the existing row |
+
+`dq_execution.py` uses a two-database connection (`db_utils.DatabaseConnection`): it reads SQL
+checks against the **ETL database** (schema `{customer}_{instance}`) and writes results to the
+**DQ database** (`data_quality`). The other two paths write only to the DQ database.
+
+---
+
+## prepare_data — PVC Setup
+
+`prepare_data.py` runs as the init container of every Hop server pod and chains three steps:
+
+1. `deploy_standard_etl_package.py` — deploys the standard ETL ZIP for the Hop project.
+2. `deploy_custom_etl_package.py` — sparse-clones the customer's directory from the
+   `custom-etl` GitHub repository, using `hop_job_config.custom_etl_branch` if set.
+3. `map_etl_directories.py` — writes S3 path mappings into the Hop project config on the PVC.
+
+Customer customisations merged into `custom-etl` take effect on the next ETL run; there is no
+active sync between HDSys and the ETL runtime.
+
+---
+
+## Stable Architecture Constraints
+
+- `data-quality` is reactive (Spring WebFlux + R2DBC) and must remain non-blocking.
+- `hop-dev-platform` is synchronous (Spring MVC) and must not adopt reactive APIs.
+- Frontend modules communicate through backend APIs only; no direct database or infrastructure
+  access.
+- Workflow scripts are entrypoint logic for orchestration steps, not workflow definitions.
+- Infrastructure and workflow template changes belong in `andromeda-argo-deployments`.
+- Cross-module API changes must consider their consumers in the same change plan.
+- `hop_job_config` is the single source of truth for per-run state; scripts must not
+  re-derive what is already in it.