@clidey/dory 0.1.0 → 0.2.0

package/docs/integrations/shopify-integration/shopify-setup.mdx

@@ -1,226 +0,0 @@
----
-title: 'Shopify Connection & Setup'
-description: 'How to set up and configure Shopify connections including manual and automated sync options.'
----
-
-# Shopify Connection & Setup Service
-
-## 1. Purpose and Scope
-
-The **ShopifyConnectService** is responsible for managing the lifecycle of Shopify store connections within the Genie platform. This includes initiating both automatic and manual OAuth connection flows, handling callbacks from Shopify, and setting up associated data structures such as teams, stores, users, and onboarding state.
-
-This service acts as a key integration point between the Genie backend and Shopify's OAuth and API endpoints. It is essential for enabling Genie users to link their Shopify stores, provision required data entities, and track onboarding progress.
-
-### Architectural Context
-
-- Operates within the Genie API backend under the `setup` module.
-- Collaborates closely with the `ShopifyApiService`, `ShopifyStoreService`, `UserManagement` services (Teams and User repositories), and onboarding and reporting subsystems.
-- Controls multi-step connection flows with both automated and manual connection support.
-
-
-## 2. Technical Overview
-
-### High-Level Flow
-
-```mermaid
-flowchart TD
-  subgraph Frontend
-    UI[Shopify Connect UI]
-  end
-
-  subgraph Genie_Backend["Genie Backend"]
-    SCS["ShopifyConnectService"]
-    SAS["ShopifyApiService"]
-    SSS["ShopifyStoreService"]
-    TS["TeamsService"]
-    OR["OnboardingService & ReportsService"]
-    JWT["JWT Service"]
-    RepoUser["User Repository"]
-    RepoTeam["Team Repository"]
-    ReqRepo["ShopifyConnectRequest Repository"]
-    Tracking["TrackingService"]
-    SavedView["SavedViewService"]
-  end
-
-  UI -->|Begin Connection| SCS
-  SCS -->|Begin OAuth| SAS
-  SAS -->|OAuth Redirect| UI
-  UI -->|OAuth Callback| SCS
-  SCS -->|Get/Setup Store & Team| SSS
-  SSS -->|Persist Team & Store| RepoTeam
-  SCS -->|Setup Admin & Business| TS
-  SCS -->|Initialize Onboarding & Reports| OR
-  SCS -->|Generate Token| JWT
-  SCS -->|Track Events| Tracking
-  SCS -->|Create Default Templates| SavedView
-  SCS -->|Update User Role & Details| RepoUser
-  ReqRepo -->|Handle Manual Requests| SCS
-
-  click SCS "apps/api/src/setup/services/shopify-connect.service.ts"
-  click SAS "apps/api/src/shopify/services/shopify-api.service.ts"
-  click SSS "apps/api/src/shopify/services/shopify-store.service.ts"
-  click TS "apps/api/src/user-management/services/teams.service.ts"
-  click OR "apps/api/src/setup/services/onboarding.service.ts"
-  click RepoUser "apps/api/src/user-management/entities/user.entity.ts"
-  click RepoTeam "apps/api/src/user-management/entities/team.entity.ts"
-  click ReqRepo "apps/api/src/setup/entities/shopify-connect-request.entity.ts"
-  click Tracking "apps/api/src/tracking/services/tracking.service.ts"
-  click SavedView "apps/api/src/pim/services/saved-view.service.ts"
-
-  classDef service fill:#f9f,stroke:#333,stroke-width:1px
-  class SCS,SAS,SSS,TS,OR,Tracking,SavedView service
-
-  classDef repository fill:#bbf,stroke:#333,stroke-width:1px
-  class RepoUser,RepoTeam,ReqRepo repository
-```
-
-### Key Technologies & Patterns
-
-- **NestJS Framework:** The service is an injectable singleton leveraging NestJS's dependency injection.
-- **TypeORM:** For database interactions managing entities like User, Team, Shopify stores, and manual connection requests.
-- **JWT:** Used to sign session tokens post successful Shopify connect.
-- **Shopify API Library:** Wraps Shopify OAuth and API interactions.
-- **Event Tracking:** Tracks onboarding progress events.
-- **Transactional Operations:** Ensures atomic creation/updating of Shopify Store and Team entities.
-- **Manual vs Automatic Connection Flows:** Supports two modes of connection to cater to different UX models.
-
-
-## 3. Core Concepts and Responsibilities
-
-### Connection Types
-
-Defined in shared types (`ShopifyConnectionType`):
-- `Setup`: Full setup connection with admin/business details capturing.
-- `GrantPermissions`: Connection flow specifically for approving additional Shopify permissions (e.g., after new features rollout).
-
-### Main Responsibilities
-
-- **beginConnecting:** Starts the automatic OAuth flow for a Shopify domain.
-- **beginManualConnection:** Starts OAuth flow based on a saved manual connection request.
-- **finishConnecting:** Handles Shopify OAuth callback, establishes Shopify store and Genie team, assigns roles, initiates onboarding.
-- **saveManualConnectionRequest:** Persists manual connection request data for later usage.
-- **assignAdminAndBusinessDetailsToTeam:** Assigns the connecting user as team admin and updates team business name.
-- **createDefaultTemplatesIfNeeded:** Adds default saved views/templates for new teams as part of setup.
-
-### Ownership
-
-- Maintained by the backend team responsible for Shopify integration.
-- Logs and tracks exceptions via GenieException handling and Sentry integration.
-
-### Non-Functional Requirements
-
-- **Reliability:** Uses transactions to guarantee consistent creation of team and store.
-- **Security:** Ensures proper JWT token generation and validates Shopify domains.
-- **Scalability:** Supports multiple connection requests and re-authorization workflows.
-
-
-## 4. Code Structure
-
-### Major Files
-
-- `shopify-connect.service.ts`: Core service implementing connection logic.
-- `shopify-store.service.ts`: Supports store domain sanitization and transactional setup.
-- `shopify-connect.controller.ts`: Exposes REST endpoints for front-end interaction with OAuth flows.
-
-### Key Methods Overview
-
-```ts
-// Starts automatic OAuth flow
-beginConnecting({ shopDomain, callbackPath, req, res })
-
-// Starts manual OAuth flow using saved request ID
-async beginManualConnection({ connectionRequestId, callbackPath, req, res })
-
-// Handles OAuth callback, sets up team/store, onboarding, and returns token
-async finishConnecting({ req, res, manualConnectionRequestId? })
-
-// Saves manual connection request from UI input
-async saveManualConnectionRequest(userId, data: ManualConnectShopifyRequestDto)
-
-// Assigns admin role and business details to team
-private async assignAdminAndBusinessDetailsToTeam({ teamId, conenctRequest })
-```
-
-### Example: Initiating OAuth Flow (Automatic)
-
-```ts
-beginConnecting({ shopDomain, callbackPath, req, res }) {
-  this.trackingService.trackEvent({
-    name: EVENTS.Onboarding,
-    properties: {
-      onboarding_progress: "Not Started",
-      onboarding_connection_type: "Automatic",
-    },
-  });
-
-  return this.shopifyApiService.getAuthApi().begin({
-    shop: this.shopifyStoreService.sanitizeShopDomainOrThrow(shopDomain),
-    callbackPath,
-    isOnline: false,
-    rawRequest: req,
-    rawResponse: res,
-  });
-}
-```
-
-
-## 5. API and Interfaces
-
-The service itself is consumed primarily by the NestJS controller `ShopifyConnectController` which exposes the following endpoints:
-
-| Endpoint                      | HTTP Method | Description                                      |
-|-------------------------------|-------------|------------------------------------------------|
-| `/shopify/connect/begin`       | GET         | Begins automatic OAuth flow with shop domain   |
-| `/shopify/connect/manual`      | POST        | Saves manual connect request and returns ID    |
-| `/shopify/connect/manual/begin`| GET         | Begins manual OAuth flow using request ID      |
-| `/shopify/connect/callback`    | GET         | Handles OAuth callback and finalizes connection|
-
-### ManualConnectShopifyRequestDto
-
-Defines input for manual connect:
-- `shopifyUrl`: Shopify domain/url string.
-- `type`: Connection type (`SETUP` or `GRANT_PERMISSIONS`).
-- For `SETUP` type, additional fields: `firstName`, `surname`, `businessName`.
-
-### Expected Returns
-
-- `finishConnecting` returns an install session token (JWT) unless it was a manual connection flow.
-
-### Error Handling
-
-- Invalid Shopify URL detected during manual connect throws `GenieException` with code `ErrorCode.InvalidShopifyUrl`.
-- Controller maps this to 400 BadRequest with error code `ApiErrorCode.InvalidShopifyUrl`.
-
-
-## 6. Environment & Configuration
-
-This service relies on core platform config for:
-- `APP_URL`: Used for forming callback and redirect URLs.
-- Shopify API credentials and tokens are managed through `ShopifyApiService`.
-- Cookie setup for manual connect request tracking (30 minutes expiration).
-
-No additional explicit environment variables or secrets are defined directly in this service.
-
-
-## 7. Testing, Monitoring, and Deployment
-
-No explicit test information is included in the provided context.
-
-
-## 8. Troubleshooting & Known Issues
-
-- Manual connection requests must be valid with correct Shopify URLs; invalid URLs raise errors.
-- The service tracks and logs exceptions via `GenieException` and `captureException`.
-- Manual connection requests are persisted to prevent unauthorized or invalid OAuth flow starts.
-
-
-## 9. References
-
-- See the service code: [apps/api/src/setup/services/shopify-connect.service.ts](apps/api/src/setup/services/shopify-connect.service.ts)
-- Controller usage: [apps/api/src/setup/controllers/shopify-connect.controller.ts](apps/api/src/setup/controllers/shopify-connect.controller.ts)
-- Store setup logic: [apps/api/src/shopify/services/shopify-store.service.ts](apps/api/src/shopify/services/shopify-store.service.ts)
-
-
----
-
-This documentation covers the internal workings and integration points of the Shopify connection and setup service within Genie. It is crucial for maintaining and extending Shopify integration capabilities.

package/docs/integrations/shopify-integration/shopify-webhooks.mdx

@@ -1,207 +0,0 @@
----
-title: 'Shopify Webhooks'
-description: 'Webhooks handling for inventory, compliance, and shopify events and how the system processes these.'
----
-
-## Shopify Webhooks
-
-### Purpose and Scope
-
-The **Shopify Webhooks** integration in the Genie app is responsible for subscribing to, receiving, storing, and processing webhook events generated by Shopify stores connected to the platform. This component enables the system to stay synchronized with store data changes such as Products, Locations, Inventory Levels, Collections, Orders, and Shop metadata updates.
-
-It bridges Shopify's asynchronous event system with Genie’s internal data models, ensuring real-time or near-real-time reflections of Shopify changes in Genie’s system. The Shopify Webhooks service owns the lifecycle of webhook registration per team (client) and manages persistent storage and processing of incoming webhook events.
-
-**System boundaries:**
-- Receives webhook HTTP callbacks from Shopify (via a public API endpoint).
-- Stores raw and processed events in a dedicated Postgres table.
-- Processes and orchestrates updates to Genie internal entities like Product, Variant, Location, Collection, Sales Order, and Settings.
-- Registers and unregisters subscriptions for Shopify webhooks via Shopify's API.
-
-
-### Technical Overview
-
-This component fits within the Genie integration layer for Shopify:
-- The **WebhooksController** exposes a public HTTP POST endpoint `/shopify/webhooks` to receive Shopify webhook payloads, secured by `ShopifyWebhooksGuard` verification.
-- The **ShopifyWebhooksService** handles storage, registration, deregistration, and fetching logic for webhooks with Shopify.
-- The **ShopifyWebhooksConsumer** asynchronously processes webhook events from a Redis-backed queue (BullMQ), delegating business logic to the **WebhooksInsightsSyncService**.
-- Sync service manages domain-specific sync flows: Products, Variants, Locations, Inventory, Collections, Orders, Shop settings.
-
-[Mermaid architecture diagram omitted here; see separate Mermaid.js output]
-
-This component depends on:
-- ShopifyApiService: For authenticated Shopify API calls.
-- TeamsService: To manage active/inactive team states and billing plan checks.
-- InsightsSyncService: For importing and syncing Shopify data.
-- TypeORM repositories for persistence.
-- BullMQ queue for asynchronous event handling.
-
-Key technologies:
-- NestJS for service and controller abstractions.
-- TypeORM for database entity management.
-- BullMQ for queued, concurrent processing.
-- Zod for schema validation of webhook payloads.
-- Shopify SDK (`@shopify/shopify-api`) for API and webhook registrations.
-
-
-### Core Concepts and Responsibilities
-
-- **Webhook Event:** Represents a single Shopify event with payload, topic, timestamps, and status.
-- **Team:** Represents a customer or client; webhooks are registered per team.
-- **Webhook Topics:** Shopify event types such as ProductsCreate, LocationsUpdate, InventoryLevelsUpdate, OrdersCreate, ShopUpdate, etc.
-- **Event Status:** Tracks processing states - PENDING, PROCESSED, SKIPPED, FAILED.
-
-Responsibility boundaries:
-- Register/unregister webhooks dynamically based on active team status and plan type.
-- Store raw webhook events reliably with input sanitization.
-- Process webhooks in order, skipping obsolete or duplicate events.
-- Mark events as processed or failed to manage retry logic.
-- Coordinate syncing of Shopify entities to Genie internal tables.
-
-
-### Code Structure
-
-| File/Folder | Description |
-|-------------|-------------|
-| [apps/api/src/shopify/controllers/webhooks.controller.ts](apps/api/src/shopify/controllers/webhooks.controller.ts) | REST controller endpoint receiving raw webhook POST requests from Shopify and forwarding to service. |
-| [apps/api/src/shopify/services/shopify-webhooks.service.ts](apps/api/src/shopify/services/shopify-webhooks.service.ts) | Core service for Shopify webhook registering, unregistering, event persistence, and queueing processing tasks. |
-| [apps/api/src/pim/services/webhooks-insights-sync.service.ts](apps/api/src/pim/services/webhooks-insights-sync.service.ts) | Business logic service to interpret and sync webhook events to internal domain (products, orders, collections, etc). |
-| [apps/api/src/shopify/shopify-webhook-event.ts](apps/api/src/shopify/shopify-webhook-event.ts) | Class representing a typed and validated Shopify webhook event instance with status setters. |
-| [apps/api/src/shopify/schemas.ts](apps/api/src/shopify/schemas.ts) | Zod schemas for validating the structure of Shopify webhook payloads by topic.
-
-
-#### Example: Receiving and Storing a Webhook Event
-
-```ts
-// In WebhooksController
-@Post('webhooks')
-async processShopifyEventForSync(
-  @Request() { webhookContext },
-  @Body() body: any
-) {
-  await this.shopifyWebhooksService.storeEvent(webhookContext, body);
-}
-```
-
-```ts
-// In ShopifyWebhooksService
-async storeEvent(context, body) {
-  // Sanitize and validate payload
-  // Resolve team by store domain
-  // Insert event record with PENDING status
-}
-```
-
-
-### API and Interfaces
-
-- **Webhook Reception Endpoint:** `/shopify/webhooks` (POST)
-  - Protected by `ShopifyWebhooksGuard` to verify Shopify request authenticity.
-  - Accepts any webhook event payload, mapped to the internal team by domain.
-
-- **ShopifyWebhooksService** key methods:
-  - `storeEvent(context, body)`: Persist a webhook event.
-  - `registerWebhooksForTeam(teamId, topics)`: Registers webhook subscriptions on Shopify for specified topics.
-  - `unregisterWebhooksForTeam(teamId)`: Deletes existing webhook subscriptions from Shopify for a team.
-  - `markEventsAsProcessed(ids)`, `markEventsAsFailed(ids)`: Update statuses in DB.
-  - `getEventsByTopics(events, topics)`: Filter events by topic.
-
-- **Event Status Workflow:**
-  - NEW events start with status `PENDING`.
-  - Successfully processed events are marked as `PROCESSED`.
-  - Duplicate or obsolete older events are marked as `SKIPPED`.
-  - Errors mark events as `FAILED` for retry.
-
-
-### Environment & Configuration
-
-- **Configuartion keys used:**
-  - `API_URL`: Base URL for webhook callback URL registration.
-  - `SHOPIFY_APP_CLIENT_ID` and `SHOPIFY_APP_CLIENT_SECRET`: Shopify app credentials.
-  - Relevant env/config management handled globally.
-
-- Webhook callback endpoint hardcoded as `/shopify/webhooks` relative to `API_URL`.
-
-
-### Testing, Monitoring, and Deployment
-
-- Webhook events are queued on receipt and processed asynchronously for scalability.
-- Processing errors are captured and logged via Sentry integration.
-- Batch processing and retry logic ensure robustness.
-- Logging throughout the WebhooksService and InsightsSyncService supports operational debugging.
-
-
-### Troubleshooting & Known Issues
-
-- Webhook registration failures for a team are logged with detailed error messages.
-- The system includes automatic retry for failed webhook registrations.
-- Event processing failures are marked `FAILED` and retried later.
-- Duplicate or older webhook events are identified and skipped.
-
-
----
-
-## Mermaid Diagram: Shopify Webhooks Architecture
-
-```mermaid
-flowchart TD
-  subgraph "Shopify System"
-    ShopifyWebhookAPI["Shopify Webhook API"]:::external
-  end
-
-  subgraph "Genie API Backend"
-    direction TB
-    WebhooksController["WebhooksController /shopify/webhooks"]:::controller
-    ShopifyWebhooksService["ShopifyWebhooksService"]:::service
-    ShopifyWebhooksQueue["BullMQ Queue"]:::queue
-    ShopifyWebhooksConsumer["ShopifyWebhooksConsumer"]:::worker
-    WebhooksInsightsSyncService["WebhooksInsightsSyncService"]:::service
-    ProductRepo["Product Repository":::db]
-    VariantRepo["ProductVariant Repository":::db]
-    LocationRepo["Location Repository":::db]
-    CollectionRepo["Collection Repository":::db]
-    SalesOrderRepo["SalesOrder Repository":::db]
-    SalesOrderItemRepo["SalesOrderItem Repository":::db]
-    SettingsService["SettingsService"]:::service
-    TeamsService["TeamsService"]:::service
-    ShopifyApiService["ShopifyApiService"]:::service
-  end
-
-  ShopifyWebhookAPI -->|"HTTP Post Webhook Calls"| WebhooksController
-  WebhooksController -->|"storeEvent()"| ShopifyWebhooksService
-  ShopifyWebhooksService -->|"Insert Event DB"| DB["ShopifyWebhookEventEntity Table"]:::db
-  ShopifyWebhooksService -->|"Add jobs to Queue"| ShopifyWebhooksQueue
-  ShopifyWebhooksConsumer -->|"Fetch Event"| DB
-  ShopifyWebhooksConsumer -->|"Process Event"| WebhooksInsightsSyncService
-
-  WebhooksInsightsSyncService --> ProductRepo
-  WebhooksInsightsSyncService --> VariantRepo
-  WebhooksInsightsSyncService --> LocationRepo
-  WebhooksInsightsSyncService --> CollectionRepo
-  WebhooksInsightsSyncService --> SalesOrderRepo
-  WebhooksInsightsSyncService --> SalesOrderItemRepo
-  WebhooksInsightsSyncService --> SettingsService
-
-  ShopifyWebhooksService --> ShopifyApiService
-  ShopifyWebhooksService --> TeamsService
-
-  classDef external fill:#D6EAF8,stroke:#2E86C1,color:#154360,font-weight:bold;
-  classDef controller fill:#D5F5E3,stroke:#27AE60,color:#145A32,font-weight:bold;
-  classDef service fill:#FCF3CF,stroke:#F1C40F,color:#7D6608,font-weight:bold;
-  classDef queue fill:#FADBD8,stroke:#E74C3C,color:#78281F,font-weight:bold;
-  classDef worker fill:#F5EEF8,stroke:#8E44AD,color:#4A235A,font-weight:bold;
-  classDef db fill:#D4E6F1,stroke:#2980B9,color:#1B4F72,font-weight:bold;
-
-  click WebhooksController "apps/api/src/shopify/controllers/webhooks.controller.ts"
-  click ShopifyWebhooksService "apps/api/src/shopify/services/shopify-webhooks.service.ts"
-  click ShopifyWebhooksConsumer "apps/api/src/shopify/services/shopify-webhooks.consumer.ts"
-  click WebhooksInsightsSyncService "apps/api/src/pim/services/webhooks-insights-sync.service.ts"
-  click ProductRepo "apps/api/src/pim/entities/product.entity.ts"
-  click VariantRepo "apps/api/src/pim/entities/product-variant.entity.ts"
-  click LocationRepo "apps/api/src/pim/entities/location.entity.ts"
-  click CollectionRepo "apps/api/src/pim/entities/collection.entity.ts"
-  click SalesOrderRepo "apps/api/src/sales-order/entities/sales-order.entity.ts"
-  click SalesOrderItemRepo "apps/api/src/sales-order/entities/sales-order-item.entity.ts"
-  click SettingsService "apps/api/src/user-management/services/settings.service.ts"
-  click TeamsService "apps/api/src/user-management/services/teams.service.ts"
-  click ShopifyApiService "apps/api/src/shopify/services/shopify-api.service.ts"
-```