@adobe/spacecat-shared-data-access 3.3.0 → 3.4.1

package/CHANGELOG.md

@@ -1,3 +1,15 @@
+## [@adobe/spacecat-shared-data-access-v3.4.1](https://github.com/adobe/spacecat-shared/compare/@adobe/spacecat-shared-data-access-v3.4.0...@adobe/spacecat-shared-data-access-v3.4.1) (2026-03-01)
+
+### Bug Fixes
+
+* **data-access:** chunk batchGetByKeys to avoid 414 URI Too Large ([#1391](https://github.com/adobe/spacecat-shared/issues/1391)) ([25b0c0d](https://github.com/adobe/spacecat-shared/commit/25b0c0dab34b4e8e8b5fef9f4033cbcc698652f5)), closes [#1390](https://github.com/adobe/spacecat-shared/issues/1390)
+
+## [@adobe/spacecat-shared-data-access-v3.4.0](https://github.com/adobe/spacecat-shared/compare/@adobe/spacecat-shared-data-access-v3.3.0...@adobe/spacecat-shared-data-access-v3.4.0) (2026-02-26)
+
+### Features
+
+* register commerce-product-enrichments-yearly audit type ([#1381](https://github.com/adobe/spacecat-shared/issues/1381)) ([6d2990f](https://github.com/adobe/spacecat-shared/commit/6d2990f691d3c9837772d092efb72cd7069a8024))
+
 ## [@adobe/spacecat-shared-data-access-v3.3.0](https://github.com/adobe/spacecat-shared/compare/@adobe/spacecat-shared-data-access-v3.2.0...@adobe/spacecat-shared-data-access-v3.3.0) (2026-02-26)
 
 ### Features

package/CLAUDE.md

@@ -0,0 +1,204 @@
+# spacecat-shared-data-access
+
+Shared data-access layer for SpaceCat services. Provides entity models/collections backed by PostgreSQL via PostgREST.
+
+## Architecture
+
+```
+Lambda/ECS service
+  -> this package (@adobe/spacecat-shared-data-access)
+       -> @supabase/postgrest-js
+            -> mysticat-data-service (PostgREST + Aurora PostgreSQL)
+```
+
+- **Database schema**: lives in [mysticat-data-service](https://github.com/adobe/mysticat-data-service) as dbmate SQL migrations
+- **This package**: JavaScript model/collection layer mapping camelCase entities to snake_case PostgREST API
+- **v2 (retired)**: ElectroDB -> DynamoDB. Published as `@adobe/spacecat-shared-data-access-v2`
+- **v3 (current)**: PostgREST client -> mysticat-data-service
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `src/index.js` | Default export: `dataAccessWrapper(fn)` for Helix/Lambda handlers |
+| `src/service/index.js` | `createDataAccess(config, log?, client?)` factory |
+| `src/models/base/schema.builder.js` | DSL for defining entity schemas (attributes, references, indexes) |
+| `src/models/base/base.model.js` | Base entity class (auto-generated getters/setters, save, remove) |
+| `src/models/base/base.collection.js` | Base collection class (findById, all, query, count) |
+| `src/models/base/entity.registry.js` | Registers all entity collections |
+| `src/util/postgrest.utils.js` | camelCase<->snake_case field mapping, query builders, cursor pagination |
+| `src/models/index.js` | Barrel export of all entity models |
+
+## Entity Structure
+
+Each entity lives in `src/models/<entity>/` with 4 files:
+
+```
+src/models/site/
+  site.schema.js       # SchemaBuilder definition (attributes, references, indexes)
+  site.model.js        # Extends BaseModel (business logic, constants)
+  site.collection.js   # Extends BaseCollection (custom queries)
+  index.js             # Re-exports model, collection, schema
+```
+
+### Schema Definition Pattern
+
+```js
+const schema = new SchemaBuilder(Site, SiteCollection)
+  .addReference('belongs_to', 'Organization')   // FK -> organizations.id
+  .addReference('has_many', 'Audits')            // One-to-many relationship
+  .addAttribute('baseURL', {
+    type: 'string',
+    required: true,
+    validate: (value) => isValidUrl(value),
+  })
+  .addAttribute('deliveryType', {
+    type: Object.values(Site.DELIVERY_TYPES),     // Enum validation
+    default: Site.DEFAULT_DELIVERY_TYPE,
+    required: true,
+  })
+  .addAttribute('config', {
+    type: 'any',
+    default: DEFAULT_CONFIG,
+    get: (value) => Config(value),                // Transform on read
+  })
+  .addAllIndex(['imsOrgId'])                      // Query index
+  .build();
+```
+
+### Attribute Options
+
+| Option | Purpose |
+|--------|---------|
+| `type` | `'string'`, `'number'`, `'boolean'`, `'any'`, `'map'`, or array of enum values |
+| `required` | Validation on save |
+| `default` | Default value or factory function |
+| `validate` | Custom validation function |
+| `readOnly` | No setter generated |
+| `postgrestField` | Custom DB column name (default: `camelToSnake(name)`) |
+| `postgrestIgnore` | Virtual attribute, not sent to DB |
+| `hidden` | Excluded from `toJSON()` |
+| `watch` | Array of field names that trigger this attribute's setter |
+| `set` | Custom setter `(value, allAttrs) => transformedValue` |
+| `get` | Custom getter `(value) => transformedValue` |
+
+### Field Mapping
+
+Models use camelCase, database uses snake_case. Mapping is automatic:
+
+| Model field | DB column | Notes |
+|-------------|-----------|-------|
+| `siteId` (idName) | `id` | Primary key always maps to `id` |
+| `baseURL` | `base_url` | Auto camelToSnake |
+| `organizationId` | `organization_id` | FK from `belongs_to` reference |
+| `isLive` | `is_live` | Auto camelToSnake |
+
+Override with `postgrestField: 'custom_name'` on the attribute.
+
+## Changing Entities
+
+Changes require **two repos**:
+
+### 1. Database schema — [mysticat-data-service](https://github.com/adobe/mysticat-data-service)
+
+```bash
+make migrate-new name=add_foo_to_sites
+# Edit the migration SQL (table, columns, indexes, grants, comments)
+make migrate && make test
+```
+
+Every migration must include: indexes on FKs, `GRANT` to `postgrest_anon`/`postgrest_writer`, `COMMENT ON` for OpenAPI docs. See [mysticat-data-service CLAUDE.md](https://github.com/adobe/mysticat-data-service/blob/main/CLAUDE.md).
+
+### 2. Model layer — this package
+
+- Add attribute in `<entity>.schema.js` -> auto-generates getter/setter
+- Add business logic in `<entity>.model.js`
+- Add custom queries in `<entity>.collection.js`
+- New entity: create 4 files + register in `src/models/index.js`
+
+### 3. Integration test
+
+```bash
+npm run test:it    # Spins up PostgREST via Docker, runs mocha suite
+```
+
+## Testing
+
+```bash
+npm test              # Unit tests (mocha + sinon + chai)
+npm run test:debug    # Unit tests with debugger
+npm run test:it       # Integration tests (Docker: Postgres + PostgREST)
+npm run lint          # ESLint
+npm run lint:fix      # Auto-fix lint issues
+```
+
+### Integration Test Setup
+
+Integration tests pull the `mysticat-data-service` Docker image from ECR:
+
+```bash
+# ECR login (one-time)
+aws ecr get-login-password --profile spacecat-dev --region us-east-1 \
+  | docker login --username AWS --password-stdin 682033462621.dkr.ecr.us-east-1.amazonaws.com
+
+# Override image tag
+export MYSTICAT_DATA_SERVICE_TAG=v1.13.0
+npm run test:it
+```
+
+### Unit Test Conventions
+
+- Tests in `test/unit/models/<entity>/`
+- PostgREST calls are stubbed via sinon
+- Each entity model and collection has its own test file
+
+## Common Patterns
+
+### Collection query with WHERE clause
+
+```js
+// In a collection method
+async findByStatus(status) {
+  return this.all(
+    (attrs, op) => op.eq(attrs.status, status),
+    { limit: 100, order: { field: 'createdAt', direction: 'desc' } }
+  );
+}
+```
+
+### Reference traversal
+
+```js
+// belongs_to: site.getOrganization() -> fetches parent org
+// has_many: organization.getSites() -> fetches child sites
+const site = await dataAccess.Site.findById(id);
+const org = await site.getOrganization();
+const audits = await site.getAudits();
+```
+
+### PostgREST WHERE operators
+
+`eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `is`, `in`, `contains`, `like`, `ilike`
+
+```js
+// Usage in collection.all()
+const liveSites = await dataAccess.Site.all(
+  (attrs, op) => op.eq(attrs.isLive, true)
+);
+```
+
+## Environment Variables
+
+| Variable | Required | Purpose |
+|----------|----------|---------|
+| `POSTGREST_URL` | Yes | PostgREST base URL (e.g. `http://data-svc.internal`) |
+| `POSTGREST_SCHEMA` | No | Schema name (default: `public`) |
+| `POSTGREST_API_KEY` | No | JWT for `postgrest_writer` role (enables UPDATE/DELETE) |
+| `S3_CONFIG_BUCKET` | No | Only for `Configuration` entity |
+| `AWS_REGION` | No | Only for `Configuration` entity |
+
+## Special Entities
+
+- **Configuration**: S3-backed (not PostgREST). Requires `S3_CONFIG_BUCKET`.
+- **KeyEvent**: Deprecated in v3. All methods throw.
+- **LatestAudit**: Virtual entity computed from `Audit` queries (no dedicated table).

package/README.md

@@ -134,12 +134,88 @@ Current exported entities include:
 - `TrialUser`
 - `TrialUserActivity`
 
+## Architecture
+
+```
+Lambda / ECS service
+  -> @adobe/spacecat-shared-data-access (this package)
+       -> @supabase/postgrest-js
+            -> mysticat-data-service (PostgREST + Aurora PostgreSQL)
+                 https://github.com/adobe/mysticat-data-service
+```
+
+**v2 (retired):** ElectroDB -> DynamoDB (direct, schema-in-code)
+**v3 (current):** PostgREST client -> [mysticat-data-service](https://github.com/adobe/mysticat-data-service) (schema-in-database)
+
+The database schema (tables, indexes, enums, grants) lives in **mysticat-data-service** as dbmate migrations.
+This package provides the JavaScript model/collection layer that maps camelCase entities to the snake_case PostgREST API.
+
 ## V3 Behavior Notes
 
 - `Configuration` remains S3-backed in v3.
 - `KeyEvent` is deprecated in v3 and intentionally throws on access/mutation methods.
 - `LatestAudit` is virtual in v3 and derived from `Audit` queries (no dedicated table required).
 
+## Changing Entities
+
+Adding or modifying an entity now requires changes in **two repositories**:
+
+### 1. Database schema — [mysticat-data-service](https://github.com/adobe/mysticat-data-service)
+
+Create a dbmate migration for the schema change (table, columns, indexes, grants, enums):
+
+```bash
+# In mysticat-data-service
+make migrate-new name=add_foo_column_to_sites
+# Edit db/migrations/YYYYMMDDHHMMSS_add_foo_column_to_sites.sql
+make migrate
+docker compose -f docker/docker-compose.yml restart postgrest
+make test
+```
+
+See the [mysticat-data-service CLAUDE.md](https://github.com/adobe/mysticat-data-service/blob/main/CLAUDE.md) for migration conventions (required grants, indexes, comments, etc.).
+
+### 2. Model/collection layer — this package
+
+Update the entity schema, model, and/or collection in `src/models/<entity>/`:
+
+| File | What to change |
+|------|---------------|
+| `<entity>.schema.js` | Add/modify attributes, references, indexes |
+| `<entity>.model.js` | Add business logic methods |
+| `<entity>.collection.js` | Add custom query methods |
+
+**Adding a new attribute example:**
+
+```js
+// In <entity>.schema.js, add to the SchemaBuilder chain:
+.addAttribute('myNewField', {
+  type: 'string',
+  required: false,
+  // Optional: custom DB column name (default: camelToSnake)
+  // postgrestField: 'custom_column_name',
+})
+```
+
+This automatically generates `getMyNewField()` and `setMyNewField()` on the model.
+
+**Adding a new entity:** Create 4 files following the pattern in any existing entity folder:
+- `<entity>.schema.js` — SchemaBuilder definition
+- `<entity>.model.js` — extends `BaseModel`
+- `<entity>.collection.js` — extends `BaseCollection`
+- `index.js` — re-exports model, collection, schema
+
+Then register the entity in `src/models/index.js`.
+
+### 3. Integration test the full stack
+
+```bash
+# In this package — runs PostgREST in Docker
+npm run test:it
+```
+
+Integration tests pull the `mysticat-data-service` Docker image from ECR, so new schema changes must be published as a new image tag first (or test against a local PostgREST).
+
 ## Migrating from V2
 
 If you are upgrading from DynamoDB/ElectroDB-based v2:
@@ -154,6 +230,7 @@ If you are upgrading from DynamoDB/ElectroDB-based v2:
 
 - Backing store is now Postgres via PostgREST, not DynamoDB/ElectroDB.
 - You must provide `postgrestUrl` (or `POSTGREST_URL` via wrapper env).
+- Schema changes now go through [mysticat-data-service](https://github.com/adobe/mysticat-data-service) migrations, not code.
 - `Configuration` remains S3-backed (requires `s3Bucket`/`S3_CONFIG_BUCKET` when used).
 - `KeyEvent` is deprecated in v3 and now throws.
 - `LatestAudit` is no longer a dedicated table and is computed from `Audit` queries.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@adobe/spacecat-shared-data-access",
-  "version": "3.3.0",
+  "version": "3.4.1",
   "description": "Shared modules of the Spacecat Services - Data Access",
   "type": "module",
   "engines": {

package/src/models/audit/audit.model.js

@@ -84,6 +84,7 @@ class Audit extends BaseModel {
     TOC: 'toc',
     WIKIPEDIA_ANALYSIS: 'wikipedia-analysis',
     COMMERCE_PRODUCT_ENRICHMENTS: 'commerce-product-enrichments',
+    COMMERCE_PRODUCT_ENRICHMENTS_YEARLY: 'commerce-product-enrichments-yearly',
     COMMERCE_PRODUCT_PAGE_ENRICHMENT: 'commerce-product-page-enrichment',
     COMMERCE_PRODUCT_CATALOG_ENRICHMENT: 'commerce-product-catalog-enrichment',
   };

package/src/models/base/base.collection.js

@@ -617,21 +617,37 @@ class BaseCollection {
         const dbField = this.#toDbField(bulkKeyField);
         const values = keys.map((key) => key[bulkKeyField]);
         const select = this.#buildSelect(options.attributes);
-        const { data, error } = await this.postgrestService
-          .from(this.tableName)
-          .select(select)
-          .in(dbField, values);
 
-        if (!error) {
+        // Chunk values to avoid 414 URI Too Large from PostgREST GET URLs.
+        // Each UUID is ~36 chars; 50 × 36 ≈ 1,800 chars, well under the 8KB URL limit.
+        const CHUNK_SIZE = 50;
+        const allData = [];
+        let hadInvalidInput = false;
+
+        for (let i = 0; i < values.length; i += CHUNK_SIZE) {
+          const chunk = values.slice(i, i + CHUNK_SIZE);
+          // eslint-disable-next-line no-await-in-loop
+          const { data, error } = await this.postgrestService
+            .from(this.tableName)
+            .select(select)
+            .in(dbField, chunk);
+
+          if (error) {
+            if (!this.#isInvalidInputError(error)) {
+              throw error;
+            }
+            hadInvalidInput = true;
+            break;
+          }
+          allData.push(...(data || []));
+        }
+
+        if (!hadInvalidInput) {
           return {
-            data: this.#createInstances((data || []).map((record) => this.#toModelRecord(record))),
+            data: this.#createInstances(allData.map((record) => this.#toModelRecord(record))),
             unprocessed: [],
           };
         }
-
-        if (!this.#isInvalidInputError(error)) {
-          throw error;
-        }
       }
 
       const records = await Promise.all(