@hartl-services/medusa-payload-sync 0.1.0

package/README.md

@@ -0,0 +1,507 @@
+# Medusa Payload Sync
+
+This Medusa shop component mirrors the small set of Medusa product
+and variant facts needed by Payload projects. Medusa remains the source of truth
+for catalog and commerce data. Payload remains the source of truth for editorial
+content.
+
+The plugin writes only two hidden, protected mirror collections. Editorial
+documents live in separate, project-defined collections, so publishing a Medusa
+title or handle can never publish an editorial draft.
+
+## Support
+
+- Medusa: `^2.19.0`
+- Payload: `3.x` REST API
+- Node.js: `>=20`
+- Package manager: `pnpm` (`pnpm@11.5.3` in this workspace)
+
+All Medusa 2.x peer dependencies accept `^2.19.0`; development dependencies
+remain pinned to `2.19.0`.
+
+## Installation in a shop
+
+```sh
+pnpm add @hartl-services/medusa-payload-sync
+```
+
+```ts
+// medusa-config.ts
+const plugins = [
+  {
+    resolve: "@hartl-services/medusa-payload-sync",
+    options: {
+      payloadUrl: process.env.PAYLOAD_URL,
+      payloadApiKey: process.env.PAYLOAD_SERVICE_API_KEY,
+      // optional, defaults to "users"
+      payloadAuthCollection: process.env.PAYLOAD_AUTH_COLLECTION,
+    },
+  },
+  // ...other plugins
+];
+```
+
+Required options and environment variables:
+
+- `payloadUrl` — Payload origin, `http`/`https` (`PAYLOAD_URL`).
+- `payloadApiKey` — API key of the dedicated Payload service user
+  (`PAYLOAD_SERVICE_API_KEY`).
+
+Everything else (`payloadAuthCollection`, `collections.product`,
+`collections.variant`, `requestTimeoutMs`, `backfillBatchSize`,
+`backfillBatchDelayMs`) is optional; see
+[Options and exact defaults](#options-and-exact-defaults) below. Keep
+`payloadApiKey` out of any public/browser environment-variable prefix and
+never forward it to a Storefront or Admin client. Also complete
+[Configure Payload authentication](#configure-payload-authentication) below,
+which the plugin itself does not automate.
+
+## Configure Medusa
+
+This repository's demo/test host, `apps/backend`, configures Payload
+Sync as a local workspace dependency directly in `apps/backend/medusa-config.ts`,
+following the same direct `medusa-config.ts` entry shown above; it does not
+ship a Payload runtime package.
+
+Feature selection and non-secret defaults belong directly in the plugin's
+options in `apps/backend/medusa-config.ts`. It is enabled by being listed
+there, not by a separate runtime feature flag. `medusa-config.ts` selects the
+default Payload auth collection `users`, overridable via
+`PAYLOAD_AUTH_COLLECTION`.
+
+Dokploy must provide the server-only runtime values `PAYLOAD_URL` and
+`PAYLOAD_SERVICE_API_KEY`; `PAYLOAD_AUTH_COLLECTION` can override the
+default. `apps/backend/medusa-config.ts` validates and passes those values to
+the local plugin. Never use a public/browser environment-variable prefix for
+the API key or forward it to a Storefront or Admin client.
+
+The backend configuration owns these server-only options directly; Payload
+Sync exports no consumer-facing option or HTTP contract.
+
+### Options and exact defaults
+
+| Option                  | Required | Implemented default       | Purpose                                                                                        |
+| ----------------------- | -------- | ------------------------- | ---------------------------------------------------------------------------------------------- |
+| `payloadUrl`            | yes      | none                      | Payload origin using `http` or `https`; one trailing slash is removed. The client adds `/api`. |
+| `payloadApiKey`         | yes      | none                      | API key of the dedicated Payload service user.                                                 |
+| `payloadAuthCollection` | no       | `users`                   | Payload auth collection slug used in the API-key Authorization header.                         |
+| `collections.product`   | no       | `medusa-products`         | Product mirror collection slug.                                                                |
+| `collections.variant`   | no       | `medusa-product-variants` | Variant mirror collection slug.                                                                |
+| `requestTimeoutMs`      | no       | `10000`                   | Timeout for one Payload REST request; accepted range `100`–`120000`.                           |
+| `backfillBatchSize`     | no       | `50`                      | Medusa/Payload page size for reconciliation; accepted range `1`–`500`.                         |
+| `backfillBatchDelayMs`  | no       | `250`                     | Delay between non-final Medusa product pages; accepted range `0`–`60000`.                      |
+
+Collection and auth slugs must be lowercase Payload-style slugs.
+`apps/backend/medusa-config.ts` currently forwards only the resolved URL,
+API key, and auth collection; changes to non-secret shop defaults must be
+modeled directly in the plugin's options there.
+
+The REST client rejects any successful JSON response larger than 8 MiB with
+`PAYLOAD_RESPONSE_TOO_LARGE` and cancels the response stream. This fixed bound
+keeps a malformed or unexpectedly expanded Payload response from consuming
+unbounded memory while leaving ample room for `/api/access` and the maximum
+configured page of 500 deliberately minimal mirror documents. Keep editorial
+and other large fields out of the mirror collections rather than relying on the
+limit as storage capacity.
+
+## Configure Payload authentication
+
+Create a dedicated user in one Payload auth collection and enable Payload's
+standard API-key strategy on that collection. For the default auth slug:
+
+```ts
+import type { CollectionConfig } from "payload";
+
+export const Users: CollectionConfig = {
+  slug: "users",
+  auth: {
+    useAPIKey: true,
+  },
+  fields: [
+    // project-specific user fields
+  ],
+};
+```
+
+After restarting Payload with `useAPIKey: true`:
+
+1. Create one account dedicated to this integration in the configured auth
+   collection through Payload's normal Admin or Local API.
+2. Enable API-key authentication for that account and copy the generated key
+   once into the Medusa server's secret store.
+3. Copy the account's Payload document ID into the Payload server environment.
+
+Keep the resulting values server-side:
+
+- its document ID as `PAYLOAD_SYNC_SERVICE_USER_ID` in the Payload process;
+- its API key as `PAYLOAD_SERVICE_API_KEY` in the Medusa process.
+
+Set `payloadAuthCollection` to the auth collection's actual slug. The plugin
+sends the standard header
+`Authorization: <payloadAuthCollection> API-Key <payloadApiKey>` on every
+Payload request. It never logs this header or the key. Do not put either secret
+in a public environment variable, client bundle, URL, or checked-in `.env`
+file.
+
+No custom Payload endpoint, hook, field factory, or companion package is
+required.
+
+## Payload mirror collection contract
+
+Both mirror collections enable Payload drafts, stay hidden from the main Admin
+navigation, deny every delete, and give only the dedicated service user create
+and update access. Any authenticated Payload user may read them. Anonymous
+reads receive only published documents through an explicit `_status` filter.
+
+Place the shared helpers in a server-only Payload source file:
+
+```ts
+// payload-sync-access.ts
+import type { Access, FieldAccess } from "payload";
+
+export const isPayloadSyncService: Access = ({ req }) =>
+  Boolean(
+    req.user?.id &&
+    String(req.user.id) === process.env.PAYLOAD_SYNC_SERVICE_USER_ID,
+  );
+
+export const syncFieldUpdate: FieldAccess = ({ req }) =>
+  Boolean(
+    req.user?.id &&
+    String(req.user.id) === process.env.PAYLOAD_SYNC_SERVICE_USER_ID,
+  );
+```
+
+`PAYLOAD_SYNC_SERVICE_USER_ID` must be set before Payload starts. A missing or
+incorrect ID denies mirror writes.
+
+### Product mirror
+
+```ts
+// collections/MedusaProducts.ts
+import type { CollectionConfig } from "payload";
+
+import { isPayloadSyncService, syncFieldUpdate } from "../payload-sync-access";
+
+export const MedusaProducts: CollectionConfig = {
+  slug: "medusa-products",
+  admin: { hidden: true, useAsTitle: "title" },
+  versions: { drafts: true },
+  access: {
+    read: ({ req }) => (req.user ? true : { _status: { equals: "published" } }),
+    create: isPayloadSyncService,
+    update: isPayloadSyncService,
+    delete: () => false,
+  },
+  fields: [
+    {
+      name: "medusaId",
+      type: "text",
+      required: true,
+      unique: true,
+      index: true,
+      admin: { readOnly: true },
+      access: { update: syncFieldUpdate },
+    },
+    {
+      name: "title",
+      type: "text",
+      required: true,
+      admin: { readOnly: true },
+      access: { update: syncFieldUpdate },
+    },
+    {
+      name: "handle",
+      type: "text",
+      required: true,
+      index: true,
+      admin: { readOnly: true },
+      access: { update: syncFieldUpdate },
+    },
+  ],
+};
+```
+
+`medusaId` is the only identity key and must stay unique. `handle` is
+intentionally **not unique**. If a deleted Medusa product is later replaced by
+a new product using the old handle, its new `medusaId` identifies a new mirror
+document. A title or handle change with the same `medusaId` updates the existing
+document and Payload records the revision.
+
+Field-level access does not replace collection access: the service user still
+needs collection-level `update` permission for field updates to succeed.
+
+### Variant mirror
+
+```ts
+// collections/MedusaProductVariants.ts
+import type { CollectionConfig } from "payload";
+
+import { isPayloadSyncService, syncFieldUpdate } from "../payload-sync-access";
+
+export const MedusaProductVariants: CollectionConfig = {
+  slug: "medusa-product-variants",
+  admin: { hidden: true, useAsTitle: "title" },
+  versions: { drafts: true },
+  access: {
+    read: ({ req }) => (req.user ? true : { _status: { equals: "published" } }),
+    create: isPayloadSyncService,
+    update: isPayloadSyncService,
+    delete: () => false,
+  },
+  fields: [
+    {
+      name: "medusaId",
+      type: "text",
+      required: true,
+      unique: true,
+      index: true,
+      admin: { readOnly: true },
+      access: { update: syncFieldUpdate },
+    },
+    {
+      name: "product",
+      type: "relationship",
+      relationTo: "medusa-products",
+      required: true,
+      index: true,
+      admin: { readOnly: true },
+      access: { update: syncFieldUpdate },
+    },
+    {
+      name: "title",
+      type: "text",
+      required: true,
+      admin: { readOnly: true },
+      access: { update: syncFieldUpdate },
+    },
+  ],
+};
+```
+
+When `collections.product` or `collections.variant` uses a custom slug, change
+the corresponding Payload `slug` values and change the variant `relationTo` to
+the configured product mirror slug as well.
+
+Register `Users`, `MedusaProducts`, and `MedusaProductVariants` in the project's
+normal `payload.config.ts`. The plugin does not create or migrate these
+collections.
+
+## Editorial isolation
+
+Mirror documents contain only the plugin-owned facts shown above. Do not add
+editorial Rich Text, SEO, media, compliance, or other mutable content fields to
+them. Create any editorial collections your project needs and relate them to
+the product or variant mirror using normal Payload relationships.
+
+The plugin neither requires nor inspects those editorial collection names,
+fields, access rules, relationships, drafts, or publishing workflow. It never
+reads or writes editorial documents. This is the key editorial safety property:
+Medusa immediately updates and publishes only hidden mirror facts, so it cannot
+publish or overwrite an in-progress editorial draft.
+
+## State and publishing invariant
+
+Every mirror write sends `_status: "draft"` or `_status: "published"`
+explicitly. The plugin does not use `draft=true` as a status transition and
+never hard-deletes a Payload document.
+
+| Medusa state/action     | Product mirror        | Current variant mirrors       | Historical/deleted variant mirrors | Editorial documents |
+| ----------------------- | --------------------- | ----------------------------- | ---------------------------------- | ------------------- |
+| Published create/update | published immediately | published immediately         | draft                              | untouched           |
+| draft/proposed/rejected | draft immediately     | draft immediately             | draft                              | untouched           |
+| Product delete          | draft                 | draft                         | draft                              | untouched           |
+| Product republish       | published immediately | published immediately         | draft                              | untouched           |
+| Variant delete          | unchanged             | deleted variant becomes draft | draft                              | untouched           |
+
+On every product sync, the plugin queries the current Medusa variants. A
+published product publishes exactly those current variants and drafts mirror
+variants no longer present in Medusa. Republish therefore restores all current
+variants but never resurrects historical or deleted variants. A draft,
+proposed, rejected, or deleted product drafts its product mirror and all related
+variant mirrors immediately.
+
+A standalone variant sync first ensures the parent product mirror exists and
+uses its Payload document ID for the required relationship. The variant follows
+the current Medusa parent product status.
+
+Repeated events are idempotent by `medusaId`. Payload's required unique index is
+the concurrency guard for parallel creates. Payload 3.87 reports that race as a
+structured HTTP 400 `ValidationError`; the client recognizes only the exact
+configured-collection `medusaId` shape, reads the winner once, and updates it
+with the requesting snapshot. Legacy HTTP 409 conflicts remain compatible.
+Other validation errors remain ordinary permanent 4xx failures. Safe repeat
+processing may create additional Payload versions, but not additional mirror
+identities.
+
+### Ordering and locking contract
+
+Product sync, variant sync, product deletion, variant deletion, and both orphan
+reconciliation page types share one coarse `payload-sync:catalog` lock. After
+acquiring it, the workflow reads the authoritative Medusa entity again and
+then performs the required Payload writes before releasing the lock. A delayed
+old event therefore converges to the current Medusa state instead of winning
+merely because its Payload request completed last.
+
+The plugin uses Medusa 2.19.0's public owner-aware `acquire` and `release`
+operations rather than `locking.execute`. Every attempt has a random owner ID,
+a 15-second acquisition deadline, and a 30-second expiring lease. While the
+authoritative callback is active, the same owner renews the lease every 10
+seconds. Lease continuity is tracked conservatively from the start of each
+successful provider call. A heartbeat that starts at or after the last
+confirmed deadline is not sent, and a renewal first observed at or after that
+deadline is not accepted as continuity even if the provider reports success.
+
+Individual acquire, renew, and release provider calls are locally observed for
+at most their configured deadline. A timed-out provider promise cannot be
+cancelled; if it later acquires or renews a lease, cleanup uses only that
+attempt's original owner ID and is itself locally observed for a finite period.
+An unresolved cleanup can therefore remain in flight, but it cannot delete a
+successor owner's lease and the provider-side lease remains finite. The
+heartbeat timer is cancelled before the normal owner-specific release,
+including error paths. A stopped process therefore leaves a finite lease rather
+than an ownerless permanent lock.
+
+If renewal or release ownership is lost while a callback is already active,
+the callback is allowed to settle, its stale owner cannot release a successor's
+lease, and the plugin acquires a fresh owner for one more complete authoritative
+read-and-write pass. The original retryable lifecycle error is still propagated
+to the workflow retry boundary even when that recovery pass encounters a
+permanent Payload error. This means an effect that ran after lease loss is not
+accepted as the final completer; operators can still observe a transient state
+until the recovery/retry pass finishes. Lock lifecycle failures use redacted
+structured error codes and are never treated as permanent Payload contract
+failures.
+
+Medusa's default in-memory locking provider coordinates only one backend
+process. A deployment with multiple workers or backend replicas must configure
+a shared provider that implements Medusa's owner-ID checks, same-owner lease
+renewal, expiry takeover, and owner-specific release. Medusa 2.19.0's Redis
+locking provider has those semantics when every process uses the same Redis
+instance. The installed Medusa 2.19.0 PostgreSQL locking provider does not let a
+different owner take over its expired owner row through `acquire`, so it does
+not satisfy this plugin's process-death recovery contract. Reconciliation must
+use the same compatible provider and lock domain as event workers. Otherwise
+the ordering guarantee is only process-local or does not apply.
+
+## Reduced startup capability check
+
+At normal startup, the plugin calls Payload's standard authenticated
+`/api/access` endpoint. It validates:
+
+- reachable authentication;
+- both configured collection slugs;
+- collection-level `read`, `create`, and `update` capabilities;
+- required field capabilities only when Payload exposes those field shapes.
+
+This is intentionally a reduced preflight. With no custom Payload code, the
+standard endpoint cannot prove every field type, `unique`/index rule,
+relationship `relationTo`, drafts/version setting, delete ACL, or omitted field
+shape. The copyable schema above is the authoritative configuration contract;
+runtime writes and the real Payload contract test are the authoritative
+protocol gate.
+
+Recognizable auth, schema, permission, and other non-retryable contract errors
+fail startup with the affected collection or capability. Temporary network
+failures, timeouts, HTTP 408, 429, and 5xx responses produce a structured,
+redacted warning and allow startup in degraded mode; later workflows retry.
+Remote preflight I/O is skipped in Medusa's data-loader-only/migration context.
+
+## Initial and repeat reconciliation
+
+Reconciliation is the initial backfill and the repair tool after missed or
+permanently failed events. Run the package's built script; do not invoke a raw
+TypeScript source path when the same plugin is enabled in the backend, because
+that can register the workflows twice.
+
+From the shop's root, run:
+
+```sh
+pnpm exec medusa exec \
+  node_modules/@hartl-services/medusa-payload-sync/.medusa/server/src/scripts/reconcile-payload-catalog.js
+```
+
+The command uses the shop's configured database and server-only Payload
+credentials.
+
+Every run:
+
+1. scans Medusa products in ascending, bounded pages;
+2. invokes the normal product workflow sequentially, including current
+   variants;
+3. waits `backfillBatchDelayMs` only between non-final product pages;
+4. scans draft-inclusive product and variant mirrors in stable `medusaId`
+   order;
+5. drafts published mirror orphans whose `medusaId` no longer exists in
+   Medusa;
+6. logs one machine-readable summary.
+
+The summary fields are `scannedProducts`, `syncedProducts`, `syncedVariants`,
+`draftedOrphanProducts`, `draftedOrphanVariants`, and `failedProductIds`.
+Individual product failures do not stop later products or orphan scans. After
+all work is attempted, any non-empty `failedProductIds` makes the command exit
+with an error. Automation must require an empty list.
+
+If a multi-ID orphan page partially succeeds and then retries, that run's
+drafted-orphan count can undercount mirrors that already converged on the first
+attempt. Mirror state and repeat-run convergence are authoritative; this is a
+metrics-only limitation.
+
+The runner is restartable and idempotent: safe repeat runs converge on the
+current Medusa state without duplicate mirror documents. Pagination is not a
+database snapshot, so rerun after concurrent catalog changes. V1 deliberately
+has no automatic schedule; operators decide when to rerun it.
+
+## Retries, logs, and incident recovery
+
+External workflow steps use Medusa retries (`maxRetries: 3`,
+`retryInterval: 2`) and idempotent writes. Transient network failures, timeouts,
+HTTP 408, 429, and 5xx responses are retryable. Auth, validation, missing
+capability, duplicate-identity, invalid-response, and other ordinary 4xx
+contract errors are permanent, except for the handled create unique conflict.
+Failed event processing is propagated rather than silently acknowledged.
+
+Structured JSON logs include the operation and classified `code`, plus the
+entity type, `medusaId`, event name, workflow/run identifiers, collection, HTTP
+status, retryability, and attempt where available. Authorization values,
+API-key values, and response headers are redacted. Do not add secrets to log
+context or paste unredacted request data into incident records.
+
+V1 has no plugin-owned dead-letter table. Recover an incident in this order:
+
+1. Inspect structured logs by `medusaId` and error `code`.
+2. Repair the authentication, collection schema/ACL, or network cause.
+3. Rerun the full reconciliation command.
+4. Require `failedProductIds: []` and confirm the expected draft/published
+   product and variant states.
+
+Because reconciliation also drafts orphans, it repairs both missing updates and
+missed delete events without deleting Payload history.
+
+## V1 out of scope
+
+- Payload editorial/content collections, Rich Text, media, SEO, or compliance
+  schemas
+- a Payload package, hook, custom endpoint, field factory, or automatic schema
+  creation
+- Storefront integration or merging Medusa and Payload responses
+- Medusa categories, collections, prices, inventory, Sales Channels, or other
+  commerce facts
+- Medusa Module Links or `defineLink`
+- Medusa Admin UI, an Admin reconciliation route, or an automatic schedule
+- Payload hard delete or a configurable delete mode
+- a plugin-owned dead-letter database/table
+
+Projects are free to add any separate Payload collections and relationships
+needed for their editorial model; those remain outside this plugin's contract.
+
+## Plugin development contract
+
+Payload Sync is server-to-server integration code and intentionally exposes no
+Store or Admin HTTP routes. It has no public DTO, schema, path, Storefront
+client, or runtime Payload-credential contract.
+
+If this component gains a Medusa HTTP route, define its request/query/response
+DTOs with `@medusajs/framework/zod`, derive public types from those schemas,
+add them and a small path object to `@hartl-services/medusa-food-supplements-contracts`, and
+let consumers use their own Medusa SDK with `sdk.client.fetch<T>`.

package/package.json

@@ -0,0 +1,52 @@
+{
+  "name": "@hartl-services/medusa-payload-sync",
+  "version": "0.1.0",
+  "description": "Mirrors Medusa product and variant facts into Payload CMS.",
+  "license": "MIT",
+  "files": [
+    ".medusa/server",
+    "README.md",
+    "LICENSE"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "git+https://gitlab.com/hartl-services-gmbh/services/food-supplements-e-commerce.git",
+    "directory": "plugins/medusa-payload-sync"
+  },
+  "publishConfig": {
+    "access": "public",
+    "registry": "https://registry.npmjs.org/"
+  },
+  "sideEffects": [
+    "./.medusa/**"
+  ],
+  "exports": {
+    "./package.json": "./package.json",
+    "./.medusa/server/src/modules/*": "./.medusa/server/src/modules/*/index.js",
+    "./.medusa/server/src/*": "./.medusa/server/src/*"
+  },
+  "peerDependencies": {
+    "@medusajs/framework": "^2.19.0",
+    "@medusajs/medusa": "^2.19.0"
+  },
+  "devDependencies": {
+    "@medusajs/cli": "2.19.0",
+    "@medusajs/framework": "2.19.0",
+    "@medusajs/medusa": "2.19.0",
+    "@swc/core": "^1.7.28",
+    "@swc/jest": "^0.2.39",
+    "@types/jest": "^29.5.14",
+    "@types/node": "^20.12.11",
+    "jest": "^29.7.0",
+    "typescript": "^5.6.2"
+  },
+  "engines": {
+    "node": ">=20"
+  },
+  "scripts": {
+    "build": "medusa plugin:build",
+    "typecheck": "tsc --noEmit",
+    "test:unit": "jest --config jest.config.js",
+    "test:contract": "jest --config jest.contract.config.js --runInBand"
+  }
+}