@groupby/ai-dev 0.5.12 → 0.5.14

package/teams/fhr-andromeda/instructions/coding-guidelines.md

@@ -0,0 +1,139 @@
+# Coding Guidelines - fhr-apache-hop
+
+This file owns implementation conventions. It should define stable coding rules, not
+architecture, workflow process, or module inventories.
+
+---
+
+## 1. How to Apply These Guidelines
+
+Apply rules in this order:
+
+1. Safety and workflow rules in `.ai/ai-instructions.md`.
+2. Architecture and module ownership from `.ai/architecture.md` and module-level `CLAUDE.md` files.
+3. Existing local patterns in the target module.
+4. Language and framework conventions in this file.
+5. General clean-code judgement.
+
+If local code conflicts with this file, follow the local pattern and mention the conflict in the
+review note.
+
+---
+
+## 2. Repository-Wide Hard Rules
+
+- Do not introduce dependencies, frameworks, tools, or modules without explicit approval.
+- Do not hardcode secrets, credentials, connection strings, customer-specific URLs, or local
+  paths.
+- Do not leave `TODO`, `FIXME`, placeholder comments, or commented-out code.
+- Match existing naming, formatting, structure, and test style in the target module.
+- Generate tests for new or changed behaviour.
+- Prefer small, focused changes over broad rewrites.
+- Fail fast with clear, specific errors when required input or configuration is missing.
+- Avoid duplication; before writing new code check whether the logic already exists somewhere
+  it logically belongs, and extend or reuse it.
+- Use constants for repeated string literals and magic values; never use `split()` with magic
+  numeric indices — use named regex groups, string templates, or `pathlib` so the structure is
+  readable and resilient.
+- Comments explain only the non-obvious *why* — a hidden constraint, a workaround, or a
+  non-obvious invariant. Never comment what the code already says.
+- After removing or moving code, remove unused imports, dead methods, unreachable branches, and
+  tests that no longer cover live behaviour.
+
+---
+
+## 3. Module Decision Table
+
+| Module | Must do | Must not do |
+|---|---|---|
+| `data-quality` (reactive) | Stay non-blocking; compose reactive flows; test with `StepVerifier` | Call `.block()`; introduce synchronous persistence |
+| `hop-dev-platform` (synchronous) | Use normal MVC/service/repository flow | Introduce `Mono`, `Flux`, or WebFlux patterns |
+| `fhr-custom-transforms` (Hop plugins) | Follow existing Hop plugin lifecycle and metadata patterns | Treat plugins like Spring services |
+| `workflow-scripts` (Python) | Keep scripts typed, logged, env-driven, and single-purpose | Use production `print()`, bare exceptions, or direct argument parsing |
+| `hop-dev-platform-frontend`, `data-quality-frontend` (Angular) | Typed models, services for HTTP, PrimeNG UI, safe subscription cleanup | Call HTTP directly in components; introduce other UI libraries |
+
+---
+
+## 4. Java Guidelines
+
+- Use Lombok for boilerplate in DTOs, entities, configuration objects, and Spring beans. Do not
+  manually write getters, setters, constructors, or equality methods.
+- Use constructor injection for Spring beans (`@RequiredArgsConstructor`).
+- Use structured logging (`@Slf4j` and `log.*`); never print to stdout.
+- Keep validation at boundaries and throw meaningful domain exceptions.
+- Keep mapping, persistence, transport, and business logic separated.
+- Put transactions at the service boundary in synchronous persistence modules.
+- For reactive modules, return reactive types from service and repository flows and never block.
+- Java package roots: `com.fredhopper.*` for plugin and HDSys code; `com.data_quality.*` for
+  DQSys code.
+
+### Java Tests
+
+- Unit tests: JUnit 5 with `@ExtendWith(MockitoExtension.class)`.
+- Integration tests: Spock/Groovy in `src/integration-test/groovy/`.
+- Database integration tests: Testcontainers (real PostgreSQL containers, never in-memory
+  substitutes).
+- Reactive tests: `StepVerifier` — never `.block()` in test assertions.
+
+---
+
+## 5. Python Guidelines
+
+- Follow PEP 8 and annotate all function signatures with types.
+- Keep scripts single-purpose and structured consistently: imports → logger → constants →
+  helpers → `main()` → `if __name__ == "__main__"`.
+- Use `logger_config.py` for logging; never use production `print()`.
+- Use `arg_utils.py` for argument parsing; never parse raw command-line arguments directly.
+- Read configuration from environment variables with sensible defaults; required variables must
+  fail fast with a clear error naming the missing variable.
+- Catch specific exceptions, log with exception context (`exc_info=True`), and exit non-zero on
+  fatal workflow-step errors.
+- Unit tests live in `src/test/python/test_<script_name>.py`; mock all external systems
+  (S3, Kafka, Kubernetes, PostgreSQL).
+
+---
+
+## 6. Angular Guidelines
+
+- Use standalone components unless existing local code requires otherwise.
+- Use PrimeNG only for UI components.
+- One component per file; follow Angular naming conventions.
+- Keep HTTP calls in services, never directly in components.
+- Prefer `async` pipe for rendered observable state.
+- Use manual subscriptions only for imperative side effects and always clean them up
+  (e.g. `takeUntilDestroyed`).
+- Define reusable interfaces for API responses; avoid `any`.
+- Component and service specs live alongside the source file.
+
+---
+
+## 7. Database Guidelines
+
+- Schema changes require new migrations; never modify existing migrations.
+- Inspect current migrations before choosing a new migration name.
+- Keep migrations backward-compatible where practical.
+- Keep persistence models, migration changes, and API contracts aligned.
+
+---
+
+## 8. Docker Guidelines
+
+- Use minimal images and multi-stage builds where useful.
+- Copy only required files; avoid broad context copies.
+- Do not bake secrets, local paths, or environment-specific values into images.
+- Match existing image naming and registry conventions.
+
+---
+
+## 9. Final Code Cleanup Checklist
+
+Before finishing any code change:
+
+- No unused imports.
+- No dead code or unreachable branches.
+- No placeholder comments or commented-out code.
+- No duplicated literals that should be constants.
+- No `split()` with magic array indices.
+- No direct logging or printing violations.
+- No new unapproved dependencies.
+- Tests cover changed behaviour and relevant error paths.

package/teams/fhr-andromeda/instructions/domain.md

@@ -0,0 +1,125 @@
+# Domain Context - fhr-apache-hop
+
+This file owns business terminology and domain concepts. It should explain what the system means,
+not how the code is structured.
+
+---
+
+## Fredhopper Context
+
+Fredhopper powers product discovery for e-commerce, including search, navigation,
+personalisation, recommendations, and catalogue indexing.
+
+ETL 2.0 is the data pipeline platform that prepares customer catalogue data for Fredhopper
+services. It replaces legacy ETL processes with a Kubernetes-native, event-driven model.
+
+---
+
+## ETL 2.0
+
+ETL 2.0 processes customer catalogue data through extraction, transformation, quality checks,
+and publication. It is designed around:
+
+- Ephemeral runtime execution instead of long-lived per-customer process servers.
+- Shared standard platform code with customer-specific configuration and customisation.
+- Event-driven or scheduled execution.
+- Observability and data-quality reporting as first-class outputs.
+
+---
+
+## Core Systems
+
+### HDSys
+
+HDSys is the Hop Development System. It gives consultants browser-based workspaces for
+developing, testing, and managing customer ETL customisations.
+
+Key concepts:
+
+- Workspace: an isolated browser-based ETL development environment.
+- Standard package: shared ETL logic available to all customers.
+- Customer customisation: customer-specific ETL changes managed through GitOps in `custom-etl`.
+- Workspace lifecycle: creation, use, expiry, and cleanup of development environments.
+
+### DQSys
+
+DQSys is the Data Quality System. It stores and presents quality and operational signals from
+ETL runs.
+
+Key concepts:
+
+- DQ check: a SQL validation rule applied to ETL output in the ETL database.
+- DQ report: the results of one or more checks for an ETL run.
+- ETL Dashboard: operational view of run status, timing, input type, and log links.
+- Accumulated reporting: quality history across incremental and repeated runs.
+
+---
+
+## ETL Run Lifecycle
+
+1. A customer catalogue change or schedule places a `trigger.json` file in S3.
+2. An Argo event sensor detects the file and submits an Argo workflow.
+3. Runtime context is prepared: input data is extracted, job configuration is assembled.
+4. ETL packages (standard + customer-specific) are deployed onto a PVC.
+5. Hop server runs the data transformation and writes results to the ETL database.
+6. FAS XML files are generated from the transformation output.
+7. Outputs and logs are published to S3; status signals are sent to Kafka.
+8. DQ checks run against the ETL database output; results are written to DQSys.
+9. ETL Dashboard data is populated in DQSys.
+10. FAS loads the XML files (reindex), then signals completion via Kafka to DQSys.
+
+---
+
+## Hop Project Names
+
+The `hop_project_name` identifies the specific ETL pipeline to run. It is derived from
+`service_type` and the input format by `extract_and_dispatch.py`:
+
+| hop_project_name | service_type | Description |
+|---|---|---|
+| `load-data-file` | fas | Full run, batch ZIP input |
+| `load-data-stream` | fas | Full run, streaming (CIDP) input |
+| `load-data-file-incremental` | fas | Incremental run, batch ZIP |
+| `load-data-stream-incremental` | fas | Incremental run, streaming |
+| `xml-writer` | fas | XML generation phase (derived from full runs) |
+| `xml-writer-incremental` | fas | Incremental XML generation |
+| `hop-generate` | suggest | Suggestion index generation |
+| `hop-finalize` | upload | Upload finalisation |
+
+The XML phase projects are never triggered directly; they are derived from the data-phase project
+name by `initialize_job.py`. Suggest and upload service types do not have an XML phase.
+
+---
+
+## Domain Terms
+
+| Term | Meaning |
+|---|---|
+| Customer | Fredhopper client whose catalogue is processed |
+| Instance | Environment or deployment context for a customer (e.g. `fas:live1`) |
+| Service type | Top-level ETL variant: `fas`, `suggest`, or `upload` |
+| Instance name | The environment part of the instance field (e.g. `live1` from `fas:live1`) |
+| Catalogue | Product data provided by a customer |
+| FAS | Fredhopper Asset Search, the core search and indexing service |
+| FAS XML | XML output format consumed by FAS |
+| Reindex | The process by which FAS loads and indexes the XML files produced by an ETL run |
+| Full ETL | Run that processes the complete catalogue |
+| Incremental ETL | Run that processes only changes since the last full run |
+| Trigger ID | Stable identifier for a single ETL run; derived from `trigger.json` or the workflow timestamp; used as the primary key across DQ tables |
+| Data ID | The trigger folder identifier (catalogue ID), as extracted from the S3 key |
+| CIDP | Customer Integration Data Platform; the streaming input path (`load-data-stream`) |
+| Argo | Workflow orchestration layer used by ETL 2.0 |
+| Hop | Apache Hop runtime used for ETL transformation |
+| Suggest | Search suggestion/typeahead data flow |
+| Smart Suggest | Newer suggestion implementation |
+| GitOps | Branch, pull request, and merge workflow for ETL customisations in `custom-etl` |
+| TC/DC | Consultant roles that use HDSys for customer delivery |
+| hop_job_config | The central per-run JSON contract produced by `initialize_job.py` and consumed by all downstream workflow steps |
+
+---
+
+## Guidance for AI Agents
+
+Use this file to understand vocabulary and business intent. For module ownership and system
+structure, use `.ai/architecture.md`. For coding rules, use `.ai/coding-guidelines.md`. For
+workflow rules, use `.ai/ai-instructions.md`.

package/teams/fhr-andromeda/instructions/jira-ticket-template.md

@@ -0,0 +1,55 @@
+# Jira Ticket Template — ETL 2.0 AI-First Development
+
+Paste the block below into the Jira ticket description. Fill in every section.
+Hand the ticket URL (or paste the description) directly to the agent — it will read the harness and start.
+
+---
+
+---
+
+## Ticket
+
+**Module**: `data-quality` | `hop-dev-platform` | `workflow-scripts` | `frontend` | `data-quality-frontend` | `fhr-custom-transforms`
+**Agent**: *(from the module `CLAUDE.md` header — e.g. `Java Dev - data-quality`)*
+**Cross-repo**: No | Yes — andromeda-argo-deployments also changes
+
+---
+
+## What to Build
+
+*2–4 sentences. What the feature does and why it is needed. No implementation detail — just the goal.*
+
+---
+
+## Requirements
+
+*Each line is one concrete behaviour. Use Given / When / Then. Be exact: field names, endpoint paths, status codes, env var names.*
+
+- Given …, when …, then …
+- Given …, when …, then …
+- Given …, when …, then *(edge case / error path)*
+
+---
+
+## Do Not Touch
+
+*Explicitly name files, classes, endpoints, or areas the agent must leave unchanged.*
+
+- …
+
+---
+
+## Acceptance Criteria
+
+*Verifiable by a reviewer who hasn't read the code. Tick these off before raising the PR.*
+
+- [ ] …
+- [ ] …
+- [ ] All existing tests pass — no regressions.
+
+---
+
+## Extra Context
+
+*Only fill this in if the requirement cannot be derived from the harness files.*
+*Leave blank if everything the agent needs is already in `CLAUDE.md`, `architecture.md`, `coding-guidelines.md`, or the module `CLAUDE.md`.*

package/teams/fhr-andromeda/instructions/modules/data-quality-frontend.md

@@ -0,0 +1,57 @@
+# CLAUDE.md - data-quality-frontend
+
+**Agent type:** `Angular Dev - data-quality-frontend`
+
+This file owns module-specific context for the DQSys frontend. Read it after the root harness
+files. Inspect current source for exact routes, services, models, and component structure.
+
+---
+
+## What This Module Is
+
+`data-quality-frontend` is the DQSys web UI. It presents data-quality reports and ETL Dashboard
+views backed by the `data-quality` service.
+
+It should remain a client of backend APIs and must not duplicate backend business rules.
+
+---
+
+## Architecture Constraints
+
+- Use the module's existing Angular standalone-component pattern.
+- Use PrimeNG for UI components only.
+- Keep HTTP calls behind the module's existing API/service abstraction.
+- Keep cross-cutting HTTP behaviour (auth headers, error handling) in the existing interceptor
+  pattern.
+- Model API responses with typed interfaces; avoid `any`.
+- Do not introduce parallel global state mechanisms when one already exists.
+
+---
+
+## Integration Boundaries
+
+This frontend depends on the `data-quality` API contract. Any backend API change must be
+reflected in models, services, and tests.
+
+When adding views that depend on customer, instance, report, or dashboard context, inspect the
+current state-management pattern before implementing new state.
+
+---
+
+## Testing Expectations
+
+- Component and service tests use the module's existing Angular test setup (Karma/Jasmine).
+- HTTP-facing tests mock backend calls through the service/API abstraction.
+- Tests should cover loading, empty, error, and permission-relevant states where applicable.
+
+---
+
+## Context to Confirm When Needed
+
+Stop and ask, or inspect current source, when a task depends on:
+
+- exact backend API response fields;
+- current runtime configuration mechanism;
+- authentication/session handling;
+- shared instance or report-selection state;
+- route structure or navigation contracts.

package/teams/fhr-andromeda/instructions/modules/data-quality.md

@@ -0,0 +1,62 @@
+# CLAUDE.md - data-quality
+
+**Agent type:** `Java Dev - data-quality`
+
+This file owns module-specific context for the DQSys backend. Read it after the root harness
+files. Inspect current source for exact classes, fields, and paths.
+
+---
+
+## What This Module Is
+
+`data-quality` is the DQSys backend. It receives data-quality and operational signals from ETL
+workflow steps, persists report and dashboard data, and exposes APIs used by the DQSys frontend.
+
+It is part of the reporting and control plane. It does not execute ETL transformations directly.
+
+---
+
+## Architecture Constraints
+
+- This module is fully reactive (Spring WebFlux + R2DBC).
+- No `.block()` calls anywhere — not in production code, not in tests.
+- All service and repository methods must return `Mono<T>` or `Flux<T>`.
+- Schema changes are managed through new Flyway migrations only; inspect existing migrations
+  before choosing the next version number.
+- Authentication and customer scoping must not be bypassed. New endpoints must inject
+  `AccountContext` via `@AuthenticationPrincipal` and scope queries to the authenticated
+  customer.
+
+---
+
+## Integration Boundaries
+
+This module is consumed by:
+
+- DQSys frontend users through backend APIs.
+- Workflow scripts that publish DQ and dashboard signals: `dq_execution.py`,
+  `dq_custom_checks.py`, and `populate_dashboard.py`.
+
+When changing API or schema contracts, identify frontend and workflow-script impact explicitly.
+See `.ai/code-review.md` for the cross-module caller checklist.
+
+---
+
+## Testing Expectations
+
+- Unit tests: JUnit 5 with `StepVerifier` for reactive assertions; never `.block()`.
+- Integration tests: Spock/Groovy in `src/integration-test/groovy/`.
+- Database integration tests: real PostgreSQL via Testcontainers.
+- Security and customer-scope behaviour must be tested when endpoint access changes.
+
+---
+
+## Context to Confirm When Needed
+
+Stop and ask, or inspect current source, when a task depends on:
+
+- exact API field names or response shapes;
+- current migration state and next migration version number;
+- customer/account scoping rules;
+- check-template or custom-check configuration contracts;
+- whether a workflow script consumes the changed contract.

package/teams/fhr-andromeda/instructions/modules/fhr-custom-transforms.md

@@ -0,0 +1,59 @@
+# CLAUDE.md - fhr-custom-transforms
+
+**Agent type:** `Java Dev - fhr-custom-transforms`
+
+This file owns module-specific context for Fredhopper Apache Hop plugins. Read it after the
+root harness files. Inspect current source for exact plugin names, metadata, and registration
+patterns.
+
+---
+
+## What This Module Is
+
+`fhr-custom-transforms` provides Fredhopper-specific Apache Hop transforms used by ETL
+packages. The module is packaged for use by the Hop runtime.
+
+It owns transformation behaviour, plugin metadata, and plugin-level tests. It does not own
+runtime or workflow orchestration.
+
+---
+
+## Architecture Constraints
+
+- Follow the existing Hop plugin lifecycle and base-class patterns.
+- Keep configuration metadata separate from runtime processing state.
+- Keep row-processing logic independent from UI/dialog code.
+- Use Hop metadata APIs for row and field handling.
+- Use existing factories and type abstractions instead of bypassing them.
+- Dialog/UI code must use SWT only (no Swing or JavaFX) and must stay compatible with headless
+  test execution.
+
+---
+
+## Integration Boundaries
+
+This module is consumed by standard ETL packages and the Hop runtime image assembly. Changes can
+affect pipeline compatibility, serialized Hop metadata, and customer ETL behaviour.
+
+When changing plugin metadata, row schema, or output shape, consider existing pipeline
+definitions and downstream consumers.
+
+---
+
+## Testing Expectations
+
+- Unit tests: JUnit 5; focus on transformation behaviour and metadata handling.
+- Tests must avoid display-dependent SWT setup unless explicitly testing dialog-only code.
+- Add regression tests for pipeline compatibility when changing serialized metadata or row
+  shape.
+
+---
+
+## Context to Confirm When Needed
+
+Stop and ask, or inspect current source, when a task depends on:
+
+- exact Hop annotation or registration requirements;
+- serialized plugin metadata compatibility;
+- supported attribute/value type semantics;
+- whether a change affects packaged standard ETL definitions.

package/teams/fhr-andromeda/instructions/modules/hop-dev-platform-frontend.md

@@ -0,0 +1,60 @@
+# CLAUDE.md - hop-dev-platform-frontend
+
+**Agent type:** `Angular Dev - hop-dev-platform-frontend`
+
+This file owns module-specific context for the HDSys frontend. Read it after the root harness
+files. Inspect current source for exact routes, services, guards, models, and component
+structure.
+
+---
+
+## What This Module Is
+
+`hop-dev-platform-frontend` is the HDSys web UI. It lets consultants create and manage Hop
+workspaces, provision customer data, and work through the GitOps flow for ETL customisations.
+
+It should remain a client of `hop-dev-platform` APIs and must not duplicate backend workspace
+or authorization rules.
+
+---
+
+## Architecture Constraints
+
+- Use the module's existing standalone-component pattern.
+- Use PrimeNG for UI components only.
+- Keep HTTP calls in Angular services.
+- Keep route protection aligned with the existing authentication guard pattern.
+- Model API responses with typed interfaces; avoid `any`.
+- Use the existing state-management approach instead of creating parallel state stores.
+- Keep API URLs compatible with the current dev and deployment proxy approach.
+
+---
+
+## Integration Boundaries
+
+This frontend depends on `hop-dev-platform` API, authentication, workspace lifecycle, and
+GitOps contracts. Backend contract changes must be reflected in services, models, routes, and
+tests.
+
+When changing workspace or Git workflows, consider both user journey and backend authorization.
+
+---
+
+## Testing Expectations
+
+- Component and service tests use the module's existing Angular test setup (Karma/Jasmine).
+- HTTP-facing tests use request-mocking utilities.
+- Tests should cover loading, empty, error, and permission-relevant states where applicable.
+- Route/auth changes need guard or navigation coverage.
+
+---
+
+## Context to Confirm When Needed
+
+Stop and ask, or inspect current source, when a task depends on:
+
+- exact backend API response fields;
+- current authentication and route-guard flow;
+- workspace state-management behaviour;
+- GitOps workflow states and actions;
+- runtime configuration or API base-path handling.

package/teams/fhr-andromeda/instructions/modules/hop-dev-platform.md

@@ -0,0 +1,62 @@
+# CLAUDE.md - hop-dev-platform
+
+**Agent type:** `Java Dev - hop-dev-platform`
+
+This file owns module-specific context for the HDSys backend. Read it after the root harness
+files. Inspect current source for exact controllers, services, configuration classes, and DTOs.
+
+---
+
+## What This Module Is
+
+`hop-dev-platform` is the HDSys backend. It provisions and manages browser-based Hop workspaces
+for ETL consultants and supports the GitOps workflow for customer customisations.
+
+It owns backend APIs, workspace lifecycle orchestration, Kubernetes integration,
+authentication, and supporting persistence for HDSys.
+
+---
+
+## Architecture Constraints
+
+- This module is synchronous Spring MVC — not reactive. No `Mono`, `Flux`, or WebFlux patterns.
+- Kubernetes operations must go through the configured `KubernetesClient` bean; do not
+  instantiate a new client.
+- All protected endpoints must go through the existing OAuth2 security filter chain; do not
+  add unprotected endpoints without explicit instruction.
+- Use typed configuration for external systems and credentials.
+- Put transactional write behaviour at the service boundary.
+
+---
+
+## Integration Boundaries
+
+This module integrates with:
+
+- Kubernetes for workspace runtime resources;
+- Git hosting for customer customisation workflows;
+- the HDSys frontend through backend APIs.
+
+When changing API, security, workspace lifecycle, or resource contracts, identify frontend and
+runtime impact explicitly.
+
+---
+
+## Testing Expectations
+
+- Unit tests: JUnit 5 with Mockito.
+- Integration tests: Spock/Groovy in `src/integration-test/groovy/`.
+- Database integration tests: Testcontainers for PostgreSQL.
+- Security-sensitive endpoint changes need authorization coverage.
+- Kubernetes and Git integrations should be mocked unless explicitly running integration tests.
+
+---
+
+## Context to Confirm When Needed
+
+Stop and ask, or inspect current source, when a task depends on:
+
+- workspace lifecycle and expiry behaviour;
+- Kubernetes resource shape or naming contracts;
+- Git repository naming and branch/merge rules;
+- authentication, logout, or fine-grained authorization behaviour.